data_redactor 0.8.0 → 0.10.0

data/readme.md

@@ -8,7 +8,33 @@ A Ruby gem with a C extension for high-performance regex-based redaction of sens
 
 ## What it does
 
-DataRedactor scans text for sensitive patterns and replaces matches with `[REDACTED]`. It uses a C extension backed by POSIX `regex.h` so the heavy lifting happens outside the Ruby VM, making it fast enough for large payloads.
+DataRedactor scans text for sensitive data — API keys and cloud secrets, IBANs,
+credit cards, national IDs, emails, phone numbers, IPs, and more — and replaces
+each match with a placeholder. The scanning runs in a C extension backed by a
+zero-dependency Thompson NFA → lazy-DFA multi-pattern engine (v19) that scans
+all 88 built-in patterns in a single pass — 2–2.5× faster than pure-Ruby `gsub`
+on large payloads, with no external library dependencies.
+
+It ships **88 built-in patterns** across 15+ countries, grouped into tags
+(`:credentials`, `:financial`, `:contact`, ...) so you can redact only what you
+care about. Beyond plain strings it can walk nested Hashes, Arrays, and JSON,
+audit a payload without mutating it (`scan`), and plug into Logger, Rails, and
+Rack. You can also register your own patterns at boot.
+
+### Use cases
+
+- **Log scrubbing** — drop the `Logger` formatter in so no secret or PII ever
+  reaches disk or your log aggregator.
+- **Rails parameter filtering** — feed `filter_parameters` a redactor-backed proc
+  to keep request params out of logs and error reports.
+- **HTTP request/response sanitising** — Rack middleware scrubs response bodies
+  and sensitive headers in flight.
+- **Sanitising LLM / API payloads** — run `redact_deep` over a params hash or
+  `redact_json` over a JSON body before it leaves the process.
+- **Compliance & auditing** — `scan` reports every match with byte offsets, tag,
+  and pattern name without changing the text, for false-positive tuning.
+- **Internal identifiers** — register company-specific patterns (`add_pattern`)
+  or generate them from a person's name (`name_pattern`).
 
 ## Usage
 
@@ -158,6 +184,46 @@ DataRedactor.clear_custom_patterns!               # mostly for test suites
 
 **`boundary: true`** — wraps the pattern with `(^|[^0-9A-Za-z])(PATTERN)([^0-9A-Za-z]|$)` so it only fires when the token is not embedded in a longer alphanumeric string. Incompatible with patterns that contain capture groups.
 
+### Name patterns
+
+Personal names can't ship as built-ins — every team has different ones — but the regex
+boilerplate to match a name across its written variations is the same every time.
+`name_pattern` generates that regex for you, ready to hand to `add_pattern`:
+
+```ruby
+DataRedactor.add_pattern(
+  name:  "person_mario_rossi",
+  regex: DataRedactor.name_pattern("Mario", "Rossi"),
+  tag:   :contact
+)
+
+DataRedactor.redact("ticket from Mario Rossi about ...")
+# => "ticket from [REDACTED] about ..."
+```
+
+A single generated pattern matches all of these:
+
+- **Case** — `Mario Rossi`, `mario rossi`, `MARIO ROSSI`
+- **Order** — `Mario Rossi`, `Rossi Mario`, `Rossi, Mario`, `Rossi,Mario`
+- **Initials** — `M. Rossi`, `M Rossi`, `Mario R.`, `M.R.`, `MR`
+- **Diacritics** — `name_pattern("Jose", "Munoz")` also matches `José Muñoz` (and vice versa)
+- **Separators** — spaces and hyphens are interchangeable. `name_pattern("Anne-Marie", "Berg")`
+  matches `Anne-Marie Berg`, `Anne Marie Berg`, `AnneMarie Berg`, and each half alone
+  (`Anne Berg`, `Marie Berg`). Multi-word parts like `"Van der Berg"` tolerate any
+  space/hyphen separator between words.
+
+It does **not** match a name embedded in a longer word — `Mario` will not fire inside
+`Mariolino` — because the generated pattern is boundary-wrapped. For that reason, register
+it with the default `boundary: false` (the wrapper is already baked into the returned
+string; `boundary: true` would double-wrap and reject its capture groups).
+
+Pass `middle:` to also cover a middle name — both the no-middle and with-middle forms match:
+
+```ruby
+DataRedactor.name_pattern("Mario", "Rossi", middle: "Luigi")
+# matches "Mario Rossi" AND "Mario Luigi Rossi" AND "Rossi Mario Luigi"
+```
+
 ## Integrations
 
 Optional adapters for Logger, Rails, and Rack. None are loaded automatically — `require` only what you use, and the gem adds zero runtime dependencies in the gemspec.
@@ -306,7 +372,9 @@ redactor/
 ├── lib/
 │   ├── data_redactor.rb          # Ruby entry point, loads the .so
 │   └── data_redactor/
-│       └── version.rb
+│       ├── version.rb
+│       ├── name_pattern.rb        # name_pattern helper — generates a name regex for add_pattern
+│       └── integrations/          # soft-required Logger / Rails / Rack adapters
 ├── ext/
 │   └── data_redactor/
 │       ├── extconf.rb            # Checks for C headers, generates Makefile (globs *.c)
@@ -317,8 +385,18 @@ redactor/
 │       ├── scan.{c,h}            # _scan + byte-offset replacement-log macros
 │       ├── custom_patterns.{c,h} # Dynamic registry: add/remove/clear/list
 │       └── tags.h                # TAG_* bit constants
-└── spec/
-    └── data_redactor_spec.rb     # RSpec tests — at least one example per pattern, plus filter / placeholder / custom-pattern coverage
+├── spec/
+│   └── data_redactor_spec.rb     # RSpec tests — at least one example per pattern, plus filter / placeholder / custom-pattern coverage
+├── benchmark/                    # Repo-only perf scripts (not packaged in the gem)
+│   ├── README.md                 # How to run, what each script measures
+│   ├── support/corpus.rb         # Shared payload builders + pure-Ruby baseline redactor
+│   ├── throughput.rb             # MB/s on representative payloads
+│   ├── vs_pure_ruby.rb           # C extension vs pure-Ruby gsub (same 88 patterns)
+│   ├── scaling.rb                # Runtime vs input size 1KB → 50MB
+│   └── per_pattern.rb            # Per-pattern scan cost
+└── docs/                         # Design and execution docs for future work
+    ├── standalone_matcher_design.md
+    └── combined_matcher_plan.md
 ```
 
 ## Requirements
@@ -393,6 +471,45 @@ Or compile and test in one step:
 bundle exec rake
 ```
 
+## Benchmarks
+
+The `benchmark/` directory holds four scripts that measure the C engine under
+different angles. They are **not** packaged with the gem.
+
+```bash
+bundle install                                   # pulls benchmark-ips, benchmark-memory (dev deps)
+bundle exec rake compile
+bundle exec ruby benchmark/vs_pure_ruby.rb       # head-to-head vs pure-Ruby gsub, same 88 patterns
+bundle exec ruby benchmark/throughput.rb         # MB/s on a log line, JSON, 1MB and 10MB log files
+bundle exec ruby benchmark/scaling.rb            # runtime vs input size (1KB → 50MB), confirms linear scaling
+bundle exec ruby benchmark/per_pattern.rb        # per-pattern scan cost over a 1MB payload
+```
+
+See [`benchmark/README.md`](benchmark/README.md) for what each script measures
+and how the pure-Ruby baseline is kept honest (it reads the same patterns the
+C engine uses, via `DataRedactor::BUILTIN_PATTERN_SOURCES`).
+
+### Performance (0.10.0 — v19 multi-pattern engine)
+
+As of 0.10.0 the C extension runs a **Thompson NFA → lazy-DFA multi-pattern
+engine** (v19) that scans the input once across all 88 built-in patterns,
+with two selective-merge passes (pure-digit group + IBAN union) that further
+reduce work for the most common pattern classes. Custom patterns (`add_pattern`)
+still use the glibc path (required for correct UTF-8 diacritic matching).
+
+| Payload               | v19 engine (0.10.0) | Pure-Ruby `gsub` | Ratio           |
+|-----------------------|---------------------|------------------|-----------------|
+| log line (168 B)      | 41 µs / call        | 71 µs / call     | **1.7× faster** |
+| JSON blob (~580 B)    | 81 µs / call        | 132 µs / call    | **1.6× faster** |
+| 8 log lines (1.3 KB)  | 175 µs / call       | 399 µs / call    | **2.3× faster** |
+| 100 log lines (17 KB) | 2.0 ms / call       | 4.6 ms / call    | **2.3× faster** |
+| 1 MB log              | 138 ms / call       | 294 ms / call    | **2.1× faster** |
+| 10 MB log             | 1.44 s / call       | —                | 6.9 MB/s        |
+
+All payload sizes pass a correctness check (redaction count matches pure-Ruby `gsub`).
+The previous engine (per-pattern `regexec`) was **4.25× slower** than pure Ruby on the
+1 MB payload — a ~9× swing. Old numbers are in git history (`CHANGELOG.md` [0.9.0]).
+
 ## How it works
 
 1. At load time, `Init_data_redactor` compiles all 85 regex patterns once using `regcomp` (POSIX ERE) and stores them as static `regex_t` structs. Patterns marked as boundary-wrapped are expanded with `wrap_boundary()` before compilation.
@@ -423,3 +540,4 @@ Released under the [MIT License](LICENSE).
 - **Pattern ordering matters** — patterns run sequentially. An early broad pattern (e.g. the 9-digit passport) may consume digits that a later pattern (e.g. credit card) depends on. Boundary wrapping mitigates this for pure-digit patterns.
 - **AWS Secret Key (pattern 1)** — 40 consecutive base64 characters is a broad match. It can produce false positives in base64-encoded content such as embedded images or binary blobs.
 - **Duplicate digit patterns** — several national ID formats share the same digit-length (11 digits: PESEL, Norwegian Fødselsnummer, Belgian National Number). They are kept as separate slots for clarity but the practical effect is that any 11-digit boundary-delimited number will be redacted.
+- **Performance is currently slower than pure-Ruby `gsub`.** A May 2026 investigation found the C extension is 3–5× slower than a pure-Ruby `gsub` loop running the same 88 patterns, across input sizes from 168 bytes to 1 MB. The root cause is glibc's POSIX `regexec()`: each call allocates an O(input-length) state buffer before any matching begins, and the gem calls it once per pattern in sequence. Ruby's Onigmo engine wins by using a built-in Boyer-Moore literal pre-filter that this gem can only approximate. Two perf fixes have shipped (buffer-sizing in `replace_all_matches`, a `strstr` literal pre-filter, and input chunking for large payloads), which gave ~25-30% improvement and made scaling linear, but the absolute gap remains. Use the gem on small payloads where the absolute latency is still acceptable (< 1 ms for typical log lines); for high-throughput pipelines, hold off until the next major release. See `docs/standalone_matcher_design.md` for the long-term plan.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_redactor
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 0.10.0
 platform: ruby
 authors:
 - Daniele Frisanco
@@ -79,6 +79,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: benchmark-ips
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.13'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.13'
+- !ruby/object:Gem::Dependency
+  name: benchmark-memory
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
 description: A Ruby gem with a C extension for high-performance scanning and redaction
   of 85 sensitive patterns — API keys, tokens, credentials, IBANs, national IDs, emails,
   phone numbers, and PII from 15+ countries. Optional Logger formatter, Rails filter_parameters
@@ -97,6 +125,8 @@ files:
 - ext/data_redactor/custom_patterns.h
 - ext/data_redactor/data_redactor.c
 - ext/data_redactor/extconf.rb
+- ext/data_redactor/matcher.c
+- ext/data_redactor/matcher.h
 - ext/data_redactor/patterns.c
 - ext/data_redactor/patterns.h
 - ext/data_redactor/placeholder.c
@@ -110,6 +140,7 @@ files:
 - lib/data_redactor/integrations/logger.rb
 - lib/data_redactor/integrations/rack.rb
 - lib/data_redactor/integrations/rails.rb
+- lib/data_redactor/name_pattern.rb
 - lib/data_redactor/version.rb
 - readme.md
 homepage: https://github.com/danielefrisanco/data_redactor