data_redactor 0.8.0-x86_64-linux → 0.10.0-x86_64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e7441d6d0ed53c590a8cffabedfadde8c7eb8dd0eff856ecb91010680613aea6
-  data.tar.gz: f0e0380592eca17b2411e28c59318d03ffdcca5e5f46b8cc3442811dbdea2c5d
+  metadata.gz: 8c68f704fce41662e1b7829085e9fadffa4931dae4323f528916897164cfdf5b
+  data.tar.gz: 1d91d3f55059e579cf36967eb06294ed18e7a54c3c1396971a749d30570118e4
 SHA512:
-  metadata.gz: 98f986da3e6ef37b8a17fe7321b9612b2dd088ffe88942ef3fcf8ab3f90ad02839dd8ac6996e2b03fcf75405d7a0dacd4b7204ce7a86a8c7f5d1a355c6fb84d9
-  data.tar.gz: f5e34041da64c3ec4ca917dc605cae268a4c88f2299ec80d90266522bf60c467bf9db1dc60f1ed1e526a9fc789f0d52653a343a862d161ad1c240948622eae6e
+  metadata.gz: c5df30474632d5026ca2a79e8be61fc020a4cb72cd7c3fb9333134d03a7d8430380185a0d209eabab33b2f27b6502ca7e9e34fb5c8fdd6df9ac24b2d71e33bd5
+  data.tar.gz: b0fd12a2666dfa50e9dd18ecb3a6733ca712ea5f1a8ea31305bd0cf788fa766cba9720c3891deec67c3eace73dafca2cc5fc566712d8bdb627b2bc9eac818b81

data/CHANGELOG.md

@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.10.0] - 2026-06-09
+
+### Changed
+- **Engine rewrite (v19 hybrid)** — `redact` and `scan` now run through a
+  Thompson NFA → bytecode → lazy-DFA multi-pattern engine (v19) for all 88
+  built-in patterns, replacing the previous per-pattern POSIX `regexec` loop.
+  Custom patterns (`add_pattern`) continue to use the glibc path (hybrid split
+  — required for correct UTF-8 multibyte character-class matching in user regex).
+- Throughput on a 1 MB log: **~8.4× faster** than the previous C engine
+  (0.87 i/s → 7.27 i/s); **2.25× faster** than pure-Ruby `gsub` (was 4×
+  slower). Small per-call strings: 1.7–2.3× faster (was 3–4.6× slower).
+- Overlap resolution: built-in matches are now resolved by an index-order
+  greedy claim (`mm_resolve`) that reproduces today's sequential per-pattern
+  rewrite semantics exactly. The one accepted divergence (rewrite-created
+  boundary when two secrets abut with no separator) is documented in
+  `TODO.md §1d` and pinned by `DIVERGENCE` specs.
+- `rb_data_redactor_scan`: coordinate mapping (`repl_log` / `WORKING_TO_ORIG`)
+  replaced by direct original-frame offset emission from the v19 engine; custom
+  patterns use a lightweight offset-walk over the built-in event list.
+
+### Fixed
+- **Swiss AHV false-negative** — boundary-wrapped patterns with a
+  start-anchored required literal now correctly set `max_back = 1` (not 0) so
+  the literal-skip does not overshoot the boundary byte. `756.1234.5678.90`
+  now matches as expected. (Pre-existing bug in the old engine, caught by
+  going live.)
+
+## [0.9.0] - 2026-05-22
+
+### Added
+- `DataRedactor.name_pattern(first, last, middle:)` — generates a POSIX ERE that matches a person's name across common written variations (case-insensitivity, First/Last order swaps, `Last, First`, initials, diacritics, and interchangeable space/hyphen separators). Returns a String ready to pass to `add_pattern`. The pattern is boundary-wrapped, so `"Mario"` matches as a word but not inside `"Mariolino"`. When `middle:` is given, both the no-middle and with-middle forms match.
+
+## [0.8.0] - 2026-05-21
+
 ### Added
 - `DataRedactor.redact_deep(data, only:, except:, placeholder:)` — recursively redacts every String value in a nested Hash/Array structure. Non-string scalars (Integer, Float, nil, Boolean) and Hash keys are passed through unchanged. Returns a deep copy; never mutates the input. Raises `ArgumentError` on circular references.
 - `DataRedactor.redact_json(json_string, only:, except:, placeholder:)` — parses JSON, redacts via `redact_deep`, and returns valid JSON. Raises `JSON::ParserError` on invalid input.
@@ -170,7 +204,9 @@ features as 0.7.1 plus the pipeline fix.
 - `DataRedactor.redact(text)` module function returning the input with every match replaced by `[REDACTED]`.
 - RSpec suite with one example per pattern.
 
-[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.7.2...HEAD
+[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.9.0...HEAD
+[0.9.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.8.0...v0.9.0
+[0.8.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.7.2...v0.8.0
 [0.7.2]: https://github.com/danielefrisanco/data_redactor/compare/v0.7.1...v0.7.2
 [0.7.1]: https://github.com/danielefrisanco/data_redactor/compare/v0.7.0...v0.7.1
 [0.7.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.6.1...v0.7.0

data/lib/data_redactor/integrations/rack.rb

@@ -1,6 +1,12 @@
 require "data_redactor"
 
 module DataRedactor
+  # Namespace for the optional framework adapters under
+  # +lib/data_redactor/integrations/+ ({Logger}, +Rails+, {Rack}).
+  #
+  # Each adapter is soft-required — none load with +require "data_redactor"+;
+  # +require+ only the one you need. They add no runtime gem dependencies and
+  # all redaction is delegated to {DataRedactor.redact}.
   module Integrations
     # Rack middleware that scrubs sensitive data from selectable surfaces of
     # the response (and request headers, for downstream loggers to see scrubbed
@@ -23,8 +29,13 @@ module DataRedactor
     #   the env hash so any downstream middleware that logs them sees scrubbed
     #   values.
     class Rack
+      # Surfaces scrubbed when +scrub:+ is not given to {#initialize}.
+      # @return [Array<Symbol>]
       DEFAULT_SCRUB = [:body, :headers].freeze
 
+      # Request-header env keys redacted in place when +:headers+ is scrubbed,
+      # so downstream middleware that logs the env sees scrubbed values.
+      # @return [Array<String>] Rack env keys (HTTP_-prefixed, upper-case).
       SENSITIVE_REQUEST_HEADERS = %w[
         HTTP_AUTHORIZATION
         HTTP_PROXY_AUTHORIZATION
@@ -34,6 +45,9 @@ module DataRedactor
         HTTP_X_ACCESS_TOKEN
       ].freeze
 
+      # Response headers whose values are redacted when +:headers+ is scrubbed.
+      # Matched case-insensitively (Rack 2 capitalises, Rack 3 lower-cases).
+      # @return [Array<String>]
       SENSITIVE_RESPONSE_HEADERS = %w[
         Set-Cookie
         Authorization
@@ -60,6 +74,13 @@ module DataRedactor
         @placeholder = placeholder
       end
 
+      # Rack entry point. Scrubs the configured surfaces of the request and
+      # response and returns the standard Rack response triple.
+      #
+      # @param env [Hash] the Rack environment.
+      # @return [Array(Integer, Hash, #each)] the +[status, headers, body]+
+      #   triple, with sensitive data redacted from the surfaces named in
+      #   +scrub:+. When +:body+ is scrubbed, +Content-Length+ is dropped.
       def call(env)
         scrub_request_headers(env) if @scrub.include?(:headers)
         status, headers, body = @app.call(env)

data/lib/data_redactor/name_pattern.rb

@@ -0,0 +1,170 @@
+# frozen_string_literal: true
+
+module DataRedactor
+  # Maps a base ASCII letter to the set of accented characters that should
+  # also match it. Used to make generated name patterns diacritic-tolerant:
+  # an input "Jose" still matches "José", and "Munoz" matches "Muñoz".
+  #
+  # @api private
+  DIACRITIC_FOLD = {
+    "a" => "àáâãäåāăą",
+    "c" => "çćĉċč",
+    "e" => "èéêëēĕėęě",
+    "i" => "ìíîïĩīĭįı",
+    "n" => "ñńņňʼn",
+    "o" => "òóôõöøōŏő",
+    "u" => "ùúûüũūŭůűų",
+    "y" => "ýÿŷ",
+    "s" => "śŝşš",
+    "z" => "źżž",
+    "g" => "ĝğġģ",
+    "l" => "ĺļľŀł",
+    "r" => "ŕŗř",
+    "t" => "ţťŧ"
+  }.freeze
+
+  module_function
+
+  # Build a POSIX ERE that matches a person's name across common written
+  # variations, ready to hand to {add_pattern}.
+  #
+  # The returned pattern is **boundary-wrapped** — it embeds
+  # +(^|[^A-Za-z])+ ... +([^A-Za-z]|$)+ so that +"Mario"+ matches as a whole
+  # word but not inside +"Mariolino"+. Because the wrapper uses capture
+  # groups, register the pattern with the default +boundary: false+ (do
+  # **not** pass +boundary: true+ — that would double-wrap and reject the
+  # groups).
+  #
+  # Variations covered:
+  # - **Case** — every letter becomes a case-insensitive character class
+  #   (+[Mm][Aa]...+), since POSIX ERE has no +/i+ flag.
+  # - **Order** — +"First Last"+, +"Last First"+, +"Last, First"+,
+  #   +"Last,First"+.
+  # - **Initials** — +"M. Last"+, +"M Last"+, +"First R."+, +"First R"+,
+  #   +"M.R."+, +"M R"+, +"MR"+.
+  # - **Diacritics** — an ASCII letter with a {DIACRITIC_FOLD} entry also
+  #   matches its accented forms (+"Jose"+ matches +"José"+). An accented
+  #   input letter also matches its bare ASCII form.
+  # - **Separators** — spaces and hyphens are interchangeable between and
+  #   within name parts. A hyphenated part like +"Anne-Marie"+ also matches
+  #   +"Anne Marie"+, +"AnneMarie"+, and each half on its own (+"Anne"+,
+  #   +"Marie"+). Multi-word parts like +"Van der Berg"+ tolerate any
+  #   space/hyphen separator between words.
+  #
+  # @param first [String] the given name. May contain hyphens or spaces.
+  # @param last [String] the family name. May contain hyphens or spaces.
+  # @param middle [String, nil] optional middle name. When given, the pattern
+  #   matches **both** the no-middle forms and the with-middle forms.
+  # @return [String] a POSIX ERE source string.
+  # @raise [ArgumentError] if +first+ or +last+ is not a non-empty String,
+  #   or +middle+ is given but is not a non-empty String.
+  #
+  # @example Register a name pattern
+  #   DataRedactor.add_pattern(
+  #     name:  "person_mario_rossi",
+  #     regex: DataRedactor.name_pattern("Mario", "Rossi"),
+  #     tag:   :contact
+  #   )
+  #
+  # @example With a middle name
+  #   DataRedactor.name_pattern("Mario", "Rossi", middle: "Luigi")
+  def name_pattern(first, last, middle: nil)
+    _validate_name_arg!(first, "first")
+    _validate_name_arg!(last, "last")
+    _validate_name_arg!(middle, "middle") unless middle.nil?
+
+    first_tok  = _part_token(first)
+    last_tok   = _part_token(last)
+    middle_tok = middle && _part_token(middle)
+
+    # Separator between name parts. Optional so initial-only forms collapse
+    # ("MR", "M.R.") and so "First,Last" with no space still matches.
+    sep = "[ ,-]*"
+
+    bodies = []
+    bodies << "#{first_tok}#{sep}#{last_tok}"            # First Last
+    bodies << "#{last_tok}#{sep}#{first_tok}"            # Last First / Last, First
+
+    if middle_tok
+      bodies << "#{first_tok}#{sep}#{middle_tok}#{sep}#{last_tok}" # First Middle Last
+      bodies << "#{last_tok}#{sep}#{first_tok}#{sep}#{middle_tok}" # Last First Middle
+    end
+
+    "(^|[^A-Za-z])(#{bodies.join('|')})([^A-Za-z]|$)"
+  end
+
+  # @api private
+  # Build the alternation for one name part: the full case-insensitive name,
+  # or its initial (with optional dot). Hyphenated/multi-word parts also
+  # match each sub-word alone and tolerant separators between sub-words.
+  #
+  # @param part [String] a single name part, e.g. "Mario" or "Anne-Marie".
+  # @return [String] a parenthesised POSIX ERE alternation.
+  def _part_token(part)
+    words = part.split(/[ -]+/).reject(&:empty?)
+
+    word_alts = words.map { |w| _word_alternatives(w) }
+
+    forms = []
+    # whole part with tolerant separators between its words
+    forms << word_alts.map { |alts| "(#{alts.join('|')})" }.join("[ -]?")
+    # each word on its own (covers "Anne" / "Marie" from "Anne-Marie")
+    if words.length > 1
+      word_alts.each { |alts| forms << "(#{alts.join('|')})" }
+    end
+
+    "(#{forms.uniq.join('|')})"
+  end
+
+  # @api private
+  # Alternatives for a single whitespace-free word: the full name (each
+  # letter as a case-insensitive, diacritic-folded class) and its initial.
+  #
+  # @param word [String] a single word with no spaces or hyphens.
+  # @return [Array<String>] alternation members for this word.
+  def _word_alternatives(word)
+    full    = word.chars.map { |ch| _letter_class(ch) }.join
+    initial = "#{_letter_class(word[0])}\\.?"
+    [full, initial]
+  end
+
+  # @api private
+  # Build a POSIX bracket expression matching one letter case-insensitively
+  # and, where applicable, its accented variants.
+  #
+  # @param char [String] a single character.
+  # @return [String] a bracket expression, e.g. "[Mm]" or "[EeÈÉÊËèéêë]".
+  def _letter_class(char)
+    down = char.downcase
+    up   = char.upcase
+    members = [down]
+    members << up unless up == down
+
+    base = DIACRITIC_FOLD.key?(down) ? down : _ascii_base(down)
+    if base && DIACRITIC_FOLD.key?(base)
+      accented = DIACRITIC_FOLD[base]
+      members << accented << accented.upcase
+      members << base << base.upcase # accented input still matches bare ASCII
+    end
+
+    "[#{members.join}]"
+  end
+
+  # @api private
+  # If +char+ is an accented letter, return the bare ASCII letter it folds
+  # to; otherwise nil.
+  #
+  # @param char [String] a single lowercase character.
+  # @return [String, nil]
+  def _ascii_base(char)
+    DIACRITIC_FOLD.each { |ascii, accents| return ascii if accents.include?(char) }
+    nil
+  end
+
+  # @api private
+  def _validate_name_arg!(value, label)
+    return if value.is_a?(String) && !value.strip.empty?
+
+    raise ArgumentError, "#{label} must be a non-empty String, got #{value.inspect}"
+  end
+end

data/lib/data_redactor/version.rb

@@ -1,4 +1,4 @@
 module DataRedactor
   # Current gem version. Follows {https://semver.org Semantic Versioning 2.0.0}.
-  VERSION = "0.8.0"
+  VERSION = "0.10.0"
 end

data/lib/data_redactor.rb

@@ -2,6 +2,7 @@ require "set"
 require "json"
 require_relative "data_redactor/version"
 require_relative "data_redactor/data_redactor" # loads the compiled .so
+require_relative "data_redactor/name_pattern"
 
 # High-performance regex-based redactor for sensitive data.
 #
@@ -73,6 +74,15 @@ module DataRedactor
   # Default placeholder used when +placeholder:+ is not given to {redact}.
   PLACEHOLDER_DEFAULT = "[REDACTED]"
 
+  # @api private
+  # Inputs larger than this (bytes) are split into newline-bounded chunks before
+  # being handed to the C engine. Bounds the per-call O(N) cost glibc regexec
+  # pays for state-log allocation, turning total redaction cost from O(N²) (one
+  # giant pass) into O(N × CHUNK_SIZE) (many bounded passes). 64 KB is a
+  # compromise: small enough to keep per-call cost low, large enough that
+  # typical log/JSON inputs use few chunks. See option G in TODO.md.
+  CHUNK_SIZE = 64 * 1024
+
   module_function
 
   # List of supported tag symbols.
@@ -131,6 +141,11 @@ module DataRedactor
   def redact(text, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT)
     enable_bits = build_enable_bits(only, except)
     ph_mode, ph_str = resolve_placeholder(placeholder)
+    # Defer to the C layer's TypeError for non-Strings; only chunk if the input
+    # is a String big enough to benefit (avoid bytesize on non-Strings).
+    if text.is_a?(String) && text.bytesize > CHUNK_SIZE
+      return _chunk_bytes(text).map { |c| _redact(c, ph_mode, ph_str, enable_bits) }.join
+    end
     _redact(text, ph_mode, ph_str, enable_bits)
   end
 
@@ -156,7 +171,12 @@ module DataRedactor
   #   #                 value: "user@example.com", start: 0, length: 16}] }
   def scan(text, only: nil, except: nil)
     enable_bits = build_enable_bits(only, except)
-    result = _scan(text, enable_bits)
+    result =
+      if text.is_a?(String) && text.bytesize > CHUNK_SIZE
+        _chunked_scan(text, enable_bits)
+      else
+        _scan(text, enable_bits)
+      end
     # Normalise: convert tag string from C (uppercase) back to the Symbol used in TAGS
     result[:matches].each { |m| m[:tag] = m[:tag].to_s.downcase.to_sym }
     result
@@ -418,4 +438,59 @@ module DataRedactor
         "placeholder must be a String, :tagged, or :hash — got #{placeholder.inspect}"
     end
   end
+
+  # @api private
+  # Split +text+ into byte-bounded chunks for the chunked redact/scan path.
+  # Chunks end at a +\n+ when possible so no match straddles a boundary; if a
+  # single line exceeds {CHUNK_SIZE} (rare in real inputs), it becomes one
+  # oversized chunk and pays the per-pattern O(N) cost — documented limitation.
+  # Returns an Array of byte-Strings whose concatenation equals +text+ exactly
+  # (including the original newline separators).
+  #
+  # @param text [String]
+  # @return [Array<String>]
+  def _chunk_bytes(text)
+    chunks = []
+    pos = 0
+    len = text.bytesize
+    while pos < len
+      remaining = len - pos
+      if remaining <= CHUNK_SIZE
+        chunks << text.byteslice(pos, remaining)
+        break
+      end
+      # Find the last \n in [pos, pos+CHUNK_SIZE). If none, chunk is one long
+      # line — take CHUNK_SIZE bytes as a fallback (boundary-split risk).
+      window = text.byteslice(pos, CHUNK_SIZE)
+      nl = window.rindex("\n")
+      take = nl ? nl + 1 : CHUNK_SIZE
+      chunks << text.byteslice(pos, take)
+      pos += take
+    end
+    chunks
+  end
+
+  # @api private
+  # Chunked variant of +_scan+: runs the C scanner on each chunk, then offsets
+  # each match's +:start+ by the chunk's base byte-position in the original
+  # input so the byteslice invariant holds end-to-end.
+  #
+  # @param text [String]
+  # @param enable_bits [Array<Integer>]
+  # @return [Hash{Symbol => Object}] +{ redacted: String, matches: Array<Hash> }+
+  def _chunked_scan(text, enable_bits)
+    redacted = +""
+    matches = []
+    base = 0
+    _chunk_bytes(text).each do |chunk|
+      part = _scan(chunk, enable_bits)
+      redacted << part[:redacted]
+      part[:matches].each do |m|
+        m[:start] += base
+        matches << m
+      end
+      base += chunk.bytesize
+    end
+    { redacted: redacted, matches: matches }
+  end
 end

data/readme.md

@@ -8,7 +8,33 @@ A Ruby gem with a C extension for high-performance regex-based redaction of sens
 
 ## What it does
 
-DataRedactor scans text for sensitive patterns and replaces matches with `[REDACTED]`. It uses a C extension backed by POSIX `regex.h` so the heavy lifting happens outside the Ruby VM, making it fast enough for large payloads.
+DataRedactor scans text for sensitive data — API keys and cloud secrets, IBANs,
+credit cards, national IDs, emails, phone numbers, IPs, and more — and replaces
+each match with a placeholder. The scanning runs in a C extension backed by a
+zero-dependency Thompson NFA → lazy-DFA multi-pattern engine (v19) that scans
+all 88 built-in patterns in a single pass — 2–2.5× faster than pure-Ruby `gsub`
+on large payloads, with no external library dependencies.
+
+It ships **88 built-in patterns** across 15+ countries, grouped into tags
+(`:credentials`, `:financial`, `:contact`, ...) so you can redact only what you
+care about. Beyond plain strings it can walk nested Hashes, Arrays, and JSON,
+audit a payload without mutating it (`scan`), and plug into Logger, Rails, and
+Rack. You can also register your own patterns at boot.
+
+### Use cases
+
+- **Log scrubbing** — drop the `Logger` formatter in so no secret or PII ever
+  reaches disk or your log aggregator.
+- **Rails parameter filtering** — feed `filter_parameters` a redactor-backed proc
+  to keep request params out of logs and error reports.
+- **HTTP request/response sanitising** — Rack middleware scrubs response bodies
+  and sensitive headers in flight.
+- **Sanitising LLM / API payloads** — run `redact_deep` over a params hash or
+  `redact_json` over a JSON body before it leaves the process.
+- **Compliance & auditing** — `scan` reports every match with byte offsets, tag,
+  and pattern name without changing the text, for false-positive tuning.
+- **Internal identifiers** — register company-specific patterns (`add_pattern`)
+  or generate them from a person's name (`name_pattern`).
 
 ## Usage
 
@@ -158,6 +184,46 @@ DataRedactor.clear_custom_patterns!               # mostly for test suites
 
 **`boundary: true`** — wraps the pattern with `(^|[^0-9A-Za-z])(PATTERN)([^0-9A-Za-z]|$)` so it only fires when the token is not embedded in a longer alphanumeric string. Incompatible with patterns that contain capture groups.
 
+### Name patterns
+
+Personal names can't ship as built-ins — every team has different ones — but the regex
+boilerplate to match a name across its written variations is the same every time.
+`name_pattern` generates that regex for you, ready to hand to `add_pattern`:
+
+```ruby
+DataRedactor.add_pattern(
+  name:  "person_mario_rossi",
+  regex: DataRedactor.name_pattern("Mario", "Rossi"),
+  tag:   :contact
+)
+
+DataRedactor.redact("ticket from Mario Rossi about ...")
+# => "ticket from [REDACTED] about ..."
+```
+
+A single generated pattern matches all of these:
+
+- **Case** — `Mario Rossi`, `mario rossi`, `MARIO ROSSI`
+- **Order** — `Mario Rossi`, `Rossi Mario`, `Rossi, Mario`, `Rossi,Mario`
+- **Initials** — `M. Rossi`, `M Rossi`, `Mario R.`, `M.R.`, `MR`
+- **Diacritics** — `name_pattern("Jose", "Munoz")` also matches `José Muñoz` (and vice versa)
+- **Separators** — spaces and hyphens are interchangeable. `name_pattern("Anne-Marie", "Berg")`
+  matches `Anne-Marie Berg`, `Anne Marie Berg`, `AnneMarie Berg`, and each half alone
+  (`Anne Berg`, `Marie Berg`). Multi-word parts like `"Van der Berg"` tolerate any
+  space/hyphen separator between words.
+
+It does **not** match a name embedded in a longer word — `Mario` will not fire inside
+`Mariolino` — because the generated pattern is boundary-wrapped. For that reason, register
+it with the default `boundary: false` (the wrapper is already baked into the returned
+string; `boundary: true` would double-wrap and reject its capture groups).
+
+Pass `middle:` to also cover a middle name — both the no-middle and with-middle forms match:
+
+```ruby
+DataRedactor.name_pattern("Mario", "Rossi", middle: "Luigi")
+# matches "Mario Rossi" AND "Mario Luigi Rossi" AND "Rossi Mario Luigi"
+```
+
 ## Integrations
 
 Optional adapters for Logger, Rails, and Rack. None are loaded automatically — `require` only what you use, and the gem adds zero runtime dependencies in the gemspec.
@@ -306,7 +372,9 @@ redactor/
 ├── lib/
 │   ├── data_redactor.rb          # Ruby entry point, loads the .so
 │   └── data_redactor/
-│       └── version.rb
+│       ├── version.rb
+│       ├── name_pattern.rb        # name_pattern helper — generates a name regex for add_pattern
+│       └── integrations/          # soft-required Logger / Rails / Rack adapters
 ├── ext/
 │   └── data_redactor/
 │       ├── extconf.rb            # Checks for C headers, generates Makefile (globs *.c)
@@ -317,8 +385,18 @@ redactor/
 │       ├── scan.{c,h}            # _scan + byte-offset replacement-log macros
 │       ├── custom_patterns.{c,h} # Dynamic registry: add/remove/clear/list
 │       └── tags.h                # TAG_* bit constants
-└── spec/
-    └── data_redactor_spec.rb     # RSpec tests — at least one example per pattern, plus filter / placeholder / custom-pattern coverage
+├── spec/
+│   └── data_redactor_spec.rb     # RSpec tests — at least one example per pattern, plus filter / placeholder / custom-pattern coverage
+├── benchmark/                    # Repo-only perf scripts (not packaged in the gem)
+│   ├── README.md                 # How to run, what each script measures
+│   ├── support/corpus.rb         # Shared payload builders + pure-Ruby baseline redactor
+│   ├── throughput.rb             # MB/s on representative payloads
+│   ├── vs_pure_ruby.rb           # C extension vs pure-Ruby gsub (same 88 patterns)
+│   ├── scaling.rb                # Runtime vs input size 1KB → 50MB
+│   └── per_pattern.rb            # Per-pattern scan cost
+└── docs/                         # Design and execution docs for future work
+    ├── standalone_matcher_design.md
+    └── combined_matcher_plan.md
 ```
 
 ## Requirements
@@ -393,6 +471,45 @@ Or compile and test in one step:
 bundle exec rake
 ```
 
+## Benchmarks
+
+The `benchmark/` directory holds four scripts that measure the C engine under
+different angles. They are **not** packaged with the gem.
+
+```bash
+bundle install                                   # pulls benchmark-ips, benchmark-memory (dev deps)
+bundle exec rake compile
+bundle exec ruby benchmark/vs_pure_ruby.rb       # head-to-head vs pure-Ruby gsub, same 88 patterns
+bundle exec ruby benchmark/throughput.rb         # MB/s on a log line, JSON, 1MB and 10MB log files
+bundle exec ruby benchmark/scaling.rb            # runtime vs input size (1KB → 50MB), confirms linear scaling
+bundle exec ruby benchmark/per_pattern.rb        # per-pattern scan cost over a 1MB payload
+```
+
+See [`benchmark/README.md`](benchmark/README.md) for what each script measures
+and how the pure-Ruby baseline is kept honest (it reads the same patterns the
+C engine uses, via `DataRedactor::BUILTIN_PATTERN_SOURCES`).
+
+### Performance (0.10.0 — v19 multi-pattern engine)
+
+As of 0.10.0 the C extension runs a **Thompson NFA → lazy-DFA multi-pattern
+engine** (v19) that scans the input once across all 88 built-in patterns,
+with two selective-merge passes (pure-digit group + IBAN union) that further
+reduce work for the most common pattern classes. Custom patterns (`add_pattern`)
+still use the glibc path (required for correct UTF-8 diacritic matching).
+
+| Payload               | v19 engine (0.10.0) | Pure-Ruby `gsub` | Ratio           |
+|-----------------------|---------------------|------------------|-----------------|
+| log line (168 B)      | 41 µs / call        | 71 µs / call     | **1.7× faster** |
+| JSON blob (~580 B)    | 81 µs / call        | 132 µs / call    | **1.6× faster** |
+| 8 log lines (1.3 KB)  | 175 µs / call       | 399 µs / call    | **2.3× faster** |
+| 100 log lines (17 KB) | 2.0 ms / call       | 4.6 ms / call    | **2.3× faster** |
+| 1 MB log              | 138 ms / call       | 294 ms / call    | **2.1× faster** |
+| 10 MB log             | 1.44 s / call       | —                | 6.9 MB/s        |
+
+All payload sizes pass a correctness check (redaction count matches pure-Ruby `gsub`).
+The previous engine (per-pattern `regexec`) was **4.25× slower** than pure Ruby on the
+1 MB payload — a ~9× swing. Old numbers are in git history (`CHANGELOG.md` [0.9.0]).
+
 ## How it works
 
 1. At load time, `Init_data_redactor` compiles all 85 regex patterns once using `regcomp` (POSIX ERE) and stores them as static `regex_t` structs. Patterns marked as boundary-wrapped are expanded with `wrap_boundary()` before compilation.
@@ -423,3 +540,4 @@ Released under the [MIT License](LICENSE).
 - **Pattern ordering matters** — patterns run sequentially. An early broad pattern (e.g. the 9-digit passport) may consume digits that a later pattern (e.g. credit card) depends on. Boundary wrapping mitigates this for pure-digit patterns.
 - **AWS Secret Key (pattern 1)** — 40 consecutive base64 characters is a broad match. It can produce false positives in base64-encoded content such as embedded images or binary blobs.
 - **Duplicate digit patterns** — several national ID formats share the same digit-length (11 digits: PESEL, Norwegian Fødselsnummer, Belgian National Number). They are kept as separate slots for clarity but the practical effect is that any 11-digit boundary-delimited number will be redacted.
+- **Performance is currently slower than pure-Ruby `gsub`.** A May 2026 investigation found the C extension is 3–5× slower than a pure-Ruby `gsub` loop running the same 88 patterns, across input sizes from 168 bytes to 1 MB. The root cause is glibc's POSIX `regexec()`: each call allocates an O(input-length) state buffer before any matching begins, and the gem calls it once per pattern in sequence. Ruby's Onigmo engine wins by using a built-in Boyer-Moore literal pre-filter that this gem can only approximate. Two perf fixes have shipped (buffer-sizing in `replace_all_matches`, a `strstr` literal pre-filter, and input chunking for large payloads), which gave ~25-30% improvement and made scaling linear, but the absolute gap remains. Use the gem on small payloads where the absolute latency is still acceptable (< 1 ms for typical log lines); for high-throughput pipelines, hold off until the next major release. See `docs/standalone_matcher_design.md` for the long-term plan.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_redactor
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 0.10.0
 platform: x86_64-linux
 authors:
 - Daniele Frisanco
@@ -65,6 +65,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: benchmark-ips
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.13'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.13'
+- !ruby/object:Gem::Dependency
+  name: benchmark-memory
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
 description: A Ruby gem with a C extension for high-performance scanning and redaction
   of 85 sensitive patterns — API keys, tokens, credentials, IBANs, national IDs, emails,
   phone numbers, and PII from 15+ countries. Optional Logger formatter, Rails filter_parameters
@@ -88,6 +116,7 @@ files:
 - lib/data_redactor/integrations/logger.rb
 - lib/data_redactor/integrations/rack.rb
 - lib/data_redactor/integrations/rails.rb
+- lib/data_redactor/name_pattern.rb
 - lib/data_redactor/version.rb
 - readme.md
 homepage: https://github.com/danielefrisanco/data_redactor