data_redactor 0.9.0-x86_64-linux → 0.10.0-x86_64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 951318e73616f9a47b5c031355973888342e11255c06daf406510a6c5e6aabd8
-  data.tar.gz: 6dfd488dbe21ef3de75f14b596369fd6839f0635c8ffb26827551c80c3ec18e1
+  metadata.gz: 8c68f704fce41662e1b7829085e9fadffa4931dae4323f528916897164cfdf5b
+  data.tar.gz: 1d91d3f55059e579cf36967eb06294ed18e7a54c3c1396971a749d30570118e4
 SHA512:
-  metadata.gz: 6a41b8e7855c55d8d123a28849a1988edc35fd97742c8d6ae19e9c57ddb30277e4d0421875d393948d4ba5e07321b7294c5d3fee45248d8a86ee79e00762f5da
-  data.tar.gz: 9105f60eae3236711e6d05554298442ecac8ffa9cf11889514b36748e14f43c96f4d38f1ca94898eededda45fa8689ad7ffcb23da68b386bd9260ec01a78e188
+  metadata.gz: c5df30474632d5026ca2a79e8be61fc020a4cb72cd7c3fb9333134d03a7d8430380185a0d209eabab33b2f27b6502ca7e9e34fb5c8fdd6df9ac24b2d71e33bd5
+  data.tar.gz: b0fd12a2666dfa50e9dd18ecb3a6733ca712ea5f1a8ea31305bd0cf788fa766cba9720c3891deec67c3eace73dafca2cc5fc566712d8bdb627b2bc9eac818b81

data/CHANGELOG.md

@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.10.0] - 2026-06-09
+
+### Changed
+- **Engine rewrite (v19 hybrid)** — `redact` and `scan` now run through a
+  Thompson NFA → bytecode → lazy-DFA multi-pattern engine (v19) for all 88
+  built-in patterns, replacing the previous per-pattern POSIX `regexec` loop.
+  Custom patterns (`add_pattern`) continue to use the glibc path (hybrid split
+  — required for correct UTF-8 multibyte character-class matching in user regex).
+- Throughput on a 1 MB log: **~8.4× faster** than the previous C engine
+  (0.87 i/s → 7.27 i/s); **2.25× faster** than pure-Ruby `gsub` (was 4×
+  slower). Small per-call strings: 1.7–2.3× faster (was 3–4.6× slower).
+- Overlap resolution: built-in matches are now resolved by an index-order
+  greedy claim (`mm_resolve`) that reproduces today's sequential per-pattern
+  rewrite semantics exactly. The one accepted divergence (rewrite-created
+  boundary when two secrets abut with no separator) is documented in
+  `TODO.md §1d` and pinned by `DIVERGENCE` specs.
+- `rb_data_redactor_scan`: coordinate mapping (`repl_log` / `WORKING_TO_ORIG`)
+  replaced by direct original-frame offset emission from the v19 engine; custom
+  patterns use a lightweight offset-walk over the built-in event list.
+
+### Fixed
+- **Swiss AHV false-negative** — boundary-wrapped patterns with a
+  start-anchored required literal now correctly set `max_back = 1` (not 0) so
+  the literal-skip does not overshoot the boundary byte. `756.1234.5678.90`
+  now matches as expected. (Pre-existing bug in the old engine, caught by
+  going live.)
+
 ## [0.9.0] - 2026-05-22
 
 ### Added

data/lib/data_redactor/version.rb

@@ -1,4 +1,4 @@
 module DataRedactor
   # Current gem version. Follows {https://semver.org Semantic Versioning 2.0.0}.
-  VERSION = "0.9.0"
+  VERSION = "0.10.0"
 end

data/lib/data_redactor.rb

@@ -74,6 +74,15 @@ module DataRedactor
   # Default placeholder used when +placeholder:+ is not given to {redact}.
   PLACEHOLDER_DEFAULT = "[REDACTED]"
 
+  # @api private
+  # Inputs larger than this (bytes) are split into newline-bounded chunks before
+  # being handed to the C engine. Bounds the per-call O(N) cost glibc regexec
+  # pays for state-log allocation, turning total redaction cost from O(N²) (one
+  # giant pass) into O(N × CHUNK_SIZE) (many bounded passes). 64 KB is a
+  # compromise: small enough to keep per-call cost low, large enough that
+  # typical log/JSON inputs use few chunks. See option G in TODO.md.
+  CHUNK_SIZE = 64 * 1024
+
   module_function
 
   # List of supported tag symbols.
@@ -132,6 +141,11 @@ module DataRedactor
   def redact(text, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT)
     enable_bits = build_enable_bits(only, except)
     ph_mode, ph_str = resolve_placeholder(placeholder)
+    # Defer to the C layer's TypeError for non-Strings; only chunk if the input
+    # is a String big enough to benefit (avoid bytesize on non-Strings).
+    if text.is_a?(String) && text.bytesize > CHUNK_SIZE
+      return _chunk_bytes(text).map { |c| _redact(c, ph_mode, ph_str, enable_bits) }.join
+    end
     _redact(text, ph_mode, ph_str, enable_bits)
   end
 
@@ -157,7 +171,12 @@ module DataRedactor
   #   #                 value: "user@example.com", start: 0, length: 16}] }
   def scan(text, only: nil, except: nil)
     enable_bits = build_enable_bits(only, except)
-    result = _scan(text, enable_bits)
+    result =
+      if text.is_a?(String) && text.bytesize > CHUNK_SIZE
+        _chunked_scan(text, enable_bits)
+      else
+        _scan(text, enable_bits)
+      end
     # Normalise: convert tag string from C (uppercase) back to the Symbol used in TAGS
     result[:matches].each { |m| m[:tag] = m[:tag].to_s.downcase.to_sym }
     result
@@ -419,4 +438,59 @@ module DataRedactor
         "placeholder must be a String, :tagged, or :hash — got #{placeholder.inspect}"
     end
   end
+
+  # @api private
+  # Split +text+ into byte-bounded chunks for the chunked redact/scan path.
+  # Chunks end at a +\n+ when possible so no match straddles a boundary; if a
+  # single line exceeds {CHUNK_SIZE} (rare in real inputs), it becomes one
+  # oversized chunk and pays the per-pattern O(N) cost — documented limitation.
+  # Returns an Array of byte-Strings whose concatenation equals +text+ exactly
+  # (including the original newline separators).
+  #
+  # @param text [String]
+  # @return [Array<String>]
+  def _chunk_bytes(text)
+    chunks = []
+    pos = 0
+    len = text.bytesize
+    while pos < len
+      remaining = len - pos
+      if remaining <= CHUNK_SIZE
+        chunks << text.byteslice(pos, remaining)
+        break
+      end
+      # Find the last \n in [pos, pos+CHUNK_SIZE). If none, chunk is one long
+      # line — take CHUNK_SIZE bytes as a fallback (boundary-split risk).
+      window = text.byteslice(pos, CHUNK_SIZE)
+      nl = window.rindex("\n")
+      take = nl ? nl + 1 : CHUNK_SIZE
+      chunks << text.byteslice(pos, take)
+      pos += take
+    end
+    chunks
+  end
+
+  # @api private
+  # Chunked variant of +_scan+: runs the C scanner on each chunk, then offsets
+  # each match's +:start+ by the chunk's base byte-position in the original
+  # input so the byteslice invariant holds end-to-end.
+  #
+  # @param text [String]
+  # @param enable_bits [Array<Integer>]
+  # @return [Hash{Symbol => Object}] +{ redacted: String, matches: Array<Hash> }+
+  def _chunked_scan(text, enable_bits)
+    redacted = +""
+    matches = []
+    base = 0
+    _chunk_bytes(text).each do |chunk|
+      part = _scan(chunk, enable_bits)
+      redacted << part[:redacted]
+      part[:matches].each do |m|
+        m[:start] += base
+        matches << m
+      end
+      base += chunk.bytesize
+    end
+    { redacted: redacted, matches: matches }
+  end
 end

data/readme.md

@@ -10,9 +10,10 @@ A Ruby gem with a C extension for high-performance regex-based redaction of sens
 
 DataRedactor scans text for sensitive data — API keys and cloud secrets, IBANs,
 credit cards, national IDs, emails, phone numbers, IPs, and more — and replaces
-each match with a placeholder. The scanning runs in a C extension backed by POSIX
-`regex.h`, so the heavy lifting happens outside the Ruby VM and stays fast enough
-to run inline on large payloads.
+each match with a placeholder. The scanning runs in a C extension backed by a
+zero-dependency Thompson NFA → lazy-DFA multi-pattern engine (v19) that scans
+all 88 built-in patterns in a single pass — 2–2.5× faster than pure-Ruby `gsub`
+on large payloads, with no external library dependencies.
 
 It ships **88 built-in patterns** across 15+ countries, grouped into tags
 (`:credentials`, `:financial`, `:contact`, ...) so you can redact only what you
@@ -384,8 +385,18 @@ redactor/
 │       ├── scan.{c,h}            # _scan + byte-offset replacement-log macros
 │       ├── custom_patterns.{c,h} # Dynamic registry: add/remove/clear/list
 │       └── tags.h                # TAG_* bit constants
-└── spec/
-    └── data_redactor_spec.rb     # RSpec tests — at least one example per pattern, plus filter / placeholder / custom-pattern coverage
+├── spec/
+│   └── data_redactor_spec.rb     # RSpec tests — at least one example per pattern, plus filter / placeholder / custom-pattern coverage
+├── benchmark/                    # Repo-only perf scripts (not packaged in the gem)
+│   ├── README.md                 # How to run, what each script measures
+│   ├── support/corpus.rb         # Shared payload builders + pure-Ruby baseline redactor
+│   ├── throughput.rb             # MB/s on representative payloads
+│   ├── vs_pure_ruby.rb           # C extension vs pure-Ruby gsub (same 88 patterns)
+│   ├── scaling.rb                # Runtime vs input size 1KB → 50MB
+│   └── per_pattern.rb            # Per-pattern scan cost
+└── docs/                         # Design and execution docs for future work
+    ├── standalone_matcher_design.md
+    └── combined_matcher_plan.md
 ```
 
 ## Requirements
@@ -460,6 +471,45 @@ Or compile and test in one step:
 bundle exec rake
 ```
 
+## Benchmarks
+
+The `benchmark/` directory holds four scripts that measure the C engine under
+different angles. They are **not** packaged with the gem.
+
+```bash
+bundle install                                   # pulls benchmark-ips, benchmark-memory (dev deps)
+bundle exec rake compile
+bundle exec ruby benchmark/vs_pure_ruby.rb       # head-to-head vs pure-Ruby gsub, same 88 patterns
+bundle exec ruby benchmark/throughput.rb         # MB/s on a log line, JSON, 1MB and 10MB log files
+bundle exec ruby benchmark/scaling.rb            # runtime vs input size (1KB → 50MB), confirms linear scaling
+bundle exec ruby benchmark/per_pattern.rb        # per-pattern scan cost over a 1MB payload
+```
+
+See [`benchmark/README.md`](benchmark/README.md) for what each script measures
+and how the pure-Ruby baseline is kept honest (it reads the same patterns the
+C engine uses, via `DataRedactor::BUILTIN_PATTERN_SOURCES`).
+
+### Performance (0.10.0 — v19 multi-pattern engine)
+
+As of 0.10.0 the C extension runs a **Thompson NFA → lazy-DFA multi-pattern
+engine** (v19) that scans the input once across all 88 built-in patterns,
+with two selective-merge passes (pure-digit group + IBAN union) that further
+reduce work for the most common pattern classes. Custom patterns (`add_pattern`)
+still use the glibc path (required for correct UTF-8 diacritic matching).
+
+| Payload               | v19 engine (0.10.0) | Pure-Ruby `gsub` | Ratio           |
+|-----------------------|---------------------|------------------|-----------------|
+| log line (168 B)      | 41 µs / call        | 71 µs / call     | **1.7× faster** |
+| JSON blob (~580 B)    | 81 µs / call        | 132 µs / call    | **1.6× faster** |
+| 8 log lines (1.3 KB)  | 175 µs / call       | 399 µs / call    | **2.3× faster** |
+| 100 log lines (17 KB) | 2.0 ms / call       | 4.6 ms / call    | **2.3× faster** |
+| 1 MB log              | 138 ms / call       | 294 ms / call    | **2.1× faster** |
+| 10 MB log             | 1.44 s / call       | —                | 6.9 MB/s        |
+
+All payload sizes pass a correctness check (redaction count matches pure-Ruby `gsub`).
+The previous engine (per-pattern `regexec`) was **4.25× slower** than pure Ruby on the
+1 MB payload — a ~9× swing. Old numbers are in git history (`CHANGELOG.md` [0.9.0]).
+
 ## How it works
 
 1. At load time, `Init_data_redactor` compiles all 85 regex patterns once using `regcomp` (POSIX ERE) and stores them as static `regex_t` structs. Patterns marked as boundary-wrapped are expanded with `wrap_boundary()` before compilation.
@@ -490,3 +540,4 @@ Released under the [MIT License](LICENSE).
 - **Pattern ordering matters** — patterns run sequentially. An early broad pattern (e.g. the 9-digit passport) may consume digits that a later pattern (e.g. credit card) depends on. Boundary wrapping mitigates this for pure-digit patterns.
 - **AWS Secret Key (pattern 1)** — 40 consecutive base64 characters is a broad match. It can produce false positives in base64-encoded content such as embedded images or binary blobs.
 - **Duplicate digit patterns** — several national ID formats share the same digit-length (11 digits: PESEL, Norwegian Fødselsnummer, Belgian National Number). They are kept as separate slots for clarity but the practical effect is that any 11-digit boundary-delimited number will be redacted.
+- **Performance is currently slower than pure-Ruby `gsub`.** A May 2026 investigation found the C extension is 3–5× slower than a pure-Ruby `gsub` loop running the same 88 patterns, across input sizes from 168 bytes to 1 MB. The root cause is glibc's POSIX `regexec()`: each call allocates an O(input-length) state buffer before any matching begins, and the gem calls it once per pattern in sequence. Ruby's Onigmo engine wins by using a built-in Boyer-Moore literal pre-filter that this gem can only approximate. Two perf fixes have shipped (buffer-sizing in `replace_all_matches`, a `strstr` literal pre-filter, and input chunking for large payloads), which gave ~25-30% improvement and made scaling linear, but the absolute gap remains. Use the gem on small payloads where the absolute latency is still acceptable (< 1 ms for typical log lines); for high-throughput pipelines, hold off until the next major release. See `docs/standalone_matcher_design.md` for the long-term plan.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_redactor
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 0.10.0
 platform: x86_64-linux
 authors:
 - Daniele Frisanco
@@ -65,6 +65,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: benchmark-ips
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.13'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.13'
+- !ruby/object:Gem::Dependency
+  name: benchmark-memory
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
 description: A Ruby gem with a C extension for high-performance scanning and redaction
   of 85 sensitive patterns — API keys, tokens, credentials, IBANs, national IDs, emails,
   phone numbers, and PII from 15+ countries. Optional Logger formatter, Rails filter_parameters