data_redactor 0.15.0 → 0.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dfda0b2a543fc9b415c0816dd08a3fb727c1abcfb082c4c0b6d04362c83ee4d2
-  data.tar.gz: 6e1b528c5ce1759ebbefff1f3f2e72bdfc2b2fa2959e5c523945752bebb999f2
+  metadata.gz: 62a1726acf7948f272e0e04bc44dac5068063731953cc7500803f2e1f8c052ed
+  data.tar.gz: 64d9f0fb75ef197facaf6538a4c46c5dad11a37e196e8b37de83ba2e629aeb24
 SHA512:
-  metadata.gz: a6a3e64351089a1b69b94a9ed33ee50eeb7339a0d359d357ce0ae6ca57c722d3b9fc5c92a4cb3b5fe8d7cb0fa40d0f3e50529e5b27226465294df78a30872714
-  data.tar.gz: 9f4082c693c639a8f4ab211bbbf92d4c9e6d79422f1696971937d95539670171cac662d64f2a3dd05185d5946b9c79f3b6a35d4ec1c4c4376d8853ecaaea719c
+  metadata.gz: ae2923599925002c945f3a488e8eafac91037075a88815975a3c920cddc195732f4186fca51dd6b98485a60e84d9ef2d55fb1fa439e9d88a3203c8f4cd6b9f75
+  data.tar.gz: c8f6691b41a419dd5adfbe89b0399d4624f3ae06b7c5a10fdffcbaee774b2009567ba687ab65b46aa0554dcbb31408212c3ebe7bef66dfb892f19191776fa8fe

data/CHANGELOG.md

@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.16.0] - 2026-06-21
+
+### Added
+- **Opt-in `#redact` refinements.** `require "data_redactor/refinements"` then
+  `using DataRedactor::Refinements` adds `#redact` to `String` (→
+  `DataRedactor.redact`) and to `Hash`/`Array` (→ `DataRedactor.redact_deep`), e.g.
+  `"email a@b.com".redact` and `chat.ask(user_input.redact)`. Refinements are
+  lexically scoped, so they never pollute the core classes globally — apps that
+  don't opt in are unaffected and there is no collision risk. Forwards
+  `only:`/`except:`/`placeholder:`; never mutates the receiver.
+  `DataRedactor.redact` remains the primary API.
+- **Length-aware placeholder modes.** `placeholder: :length` replaces each match
+  with `[REDACTED:N]` and `placeholder: :tagged_length` with `[REDACTED:TAGNAME:N]`,
+  where `N` is the **byte length** of the redacted value. Readers can gauge what
+  was there without seeing it. Both compose with `only:`/`except:` and are
+  forwarded by `redact_deep`, `redact_json`, and the integrations. Additive —
+  two new values for the existing `placeholder:` keyword; no behaviour changes.
+- **CI: ASan/UBSan memory-safety gate.** New job builds the matcher engine
+  standalone under `-fsanitize=address,undefined` and drives it over an
+  adversarial corpus + seeded fuzz loop (catches the `OP_EOL`-class OOB read).
+- **CI: throughput-trend history + PR comment.** New job records the C/pure-Ruby
+  ratio over time (history in `actions/cache`, no gh-pages), posts a sticky PR
+  comment comparing each run to the previous point, and fails on a >10% drop.
+
 ## [0.15.0] - 2026-06-17
 
 ### Changed
@@ -312,7 +336,8 @@ features as 0.7.1 plus the pipeline fix.
 - `DataRedactor.redact(text)` module function returning the input with every match replaced by `[REDACTED]`.
 - RSpec suite with one example per pattern.
 
-[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.15.0...HEAD
+[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.16.0...HEAD
+[0.16.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.15.0...v0.16.0
 [0.15.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.1...v0.15.0
 [0.14.1]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.0...v0.14.1
 [0.14.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.13.0...v0.14.0

data/README.md

@@ -6,6 +6,12 @@
 
 A Ruby gem with a C extension for high-performance regex-based redaction of sensitive data from strings.
 
+> 📄 The engineering behind the v19 matching engine is written up as an experience
+> report, *"The Fastest Engine Is Not the Shippable Engine: Replacing a Regex Engine
+> for Data Redaction Under Production Constraints,"* currently under review at
+> *Software: Practice and Experience* (Manuscript ID 7985366). Source and the
+> reproducibility bundle are in [`paper/`](paper/).
+
 ## What it does
 
 DataRedactor scans text for sensitive data — API keys and cloud secrets, IBANs,
@@ -103,9 +109,18 @@ DataRedactor.redact(text, placeholder: :hash)
 # "user@example.com"  → "[CONTACT_3d7a]"
 # "user@example.com"  → "[CONTACT_3d7a]"  (same every time)
 # "other@example.com" → "[CONTACT_91fc]"  (different value, different hash)
+
+# Length — embeds the byte length of the redacted value, so readers can
+# gauge what was there without seeing it.
+DataRedactor.redact(text, placeholder: :length)
+# "user@example.com"  → "[REDACTED:16]"
+
+# Tagged length — tag name plus byte length.
+DataRedactor.redact(text, placeholder: :tagged_length)
+# "user@example.com"  → "[REDACTED:CONTACT:16]"
 ```
 
-All three modes compose with `only:` and `except:`:
+All modes compose with `only:` and `except:`:
 
 ```ruby
 DataRedactor.redact(text, only: :contact, placeholder: :tagged)
@@ -165,6 +180,24 @@ safe_json = DataRedactor.redact_json('{"email":"alice@example.com","count":3}')
 DataRedactor.redact_json("not json")  # => JSON::ParserError
 ```
 
+### `#redact` refinements (opt-in)
+
+Prefer `"text".redact` over `DataRedactor.redact("text")`? Opt into the refinement. It adds `#redact` to `String` (via `redact`) and to `Hash`/`Array` (via `redact_deep`) **only in the files that `using` it** — refinements are lexically scoped, so the core classes are never monkey-patched globally and there is no collision risk for apps that don't opt in. `DataRedactor.redact` remains the primary API.
+
+```ruby
+require "data_redactor/refinements"
+using DataRedactor::Refinements
+
+"email alice@example.com".redact            # => "email [REDACTED]"
+{ token: "AKIAIOSFODNN7EXAMPLE" }.redact    # => { token: "[REDACTED]" }
+["a@b.com", 3].redact                       # => ["[REDACTED]", 3]
+
+# Handy right before sending text to an LLM:
+chat.ask(user_input.redact)
+```
+
+`#redact` forwards `only:`/`except:`/`placeholder:` and never mutates the receiver. Without `using DataRedactor::Refinements` in the current file, `#redact` is not defined.
+
 ### Custom patterns
 
 Teams often have internal IDs that the gem can't ship. Register them at boot — or at runtime from any thread (registration is thread-safe, see [Thread safety](#thread-safety)):
@@ -309,6 +342,24 @@ safe_response = DataRedactor::Integrations::OpenAI.redact_response(response)
 
 `content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works.
 
+### RubyLLM
+
+[RubyLLM](https://rubyllm.com) is a unified Ruby client for every major LLM provider — and a perfect match for `data_redactor`: anything you send to a model is exactly the kind of free text that leaks secrets and PII. Because RubyLLM takes plain strings, you can scrub them with `DataRedactor.redact` before they leave the process — no extra integration required:
+
+```ruby
+require "ruby_llm"
+require "data_redactor"
+
+chat = RubyLLM.chat(model: "claude-opus-4-8")
+chat.with_instructions(DataRedactor.redact("You are a support agent for ACME Corp."))
+
+user_input = "My card is 4111 1111 1111 1111 and my email is alice@example.com"
+chat.ask(DataRedactor.redact(user_input))
+# the model receives: "My card is [REDACTED] and my email is [REDACTED]"
+```
+
+Wrap each prompt (and any `with_instructions` system prompt) in `DataRedactor.redact` before passing it to `ask`. This is a per-call step you opt into — RubyLLM does not yet expose a request hook for automatic, transparent redaction of every outbound call ([crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765) tracks the connection-middleware hook that would enable it).
+
 ## Detected patterns (89 total)
 
 The table below is a representative sample. Use `DataRedactor.pattern_names` for the canonical, machine-readable list — it stays in sync with the C extension automatically.

data/ext/data_redactor/data_redactor.c

@@ -68,9 +68,11 @@ void Init_data_redactor(void) {
     rb_define_const(mDataRedactor, "BUILTIN_PATTERN_BOUNDARY", rb_ary_freeze(builtin_boundary));
 
     /* Placeholder mode constants. */
-    rb_define_const(mDataRedactor, "PH_MODE_PLAIN",  INT2NUM(PLACEHOLDER_MODE_PLAIN));
-    rb_define_const(mDataRedactor, "PH_MODE_TAGGED", INT2NUM(PLACEHOLDER_MODE_TAGGED));
-    rb_define_const(mDataRedactor, "PH_MODE_HASH",   INT2NUM(PLACEHOLDER_MODE_HASH));
+    rb_define_const(mDataRedactor, "PH_MODE_PLAIN",         INT2NUM(PLACEHOLDER_MODE_PLAIN));
+    rb_define_const(mDataRedactor, "PH_MODE_TAGGED",        INT2NUM(PLACEHOLDER_MODE_TAGGED));
+    rb_define_const(mDataRedactor, "PH_MODE_HASH",          INT2NUM(PLACEHOLDER_MODE_HASH));
+    rb_define_const(mDataRedactor, "PH_MODE_LENGTH",        INT2NUM(PLACEHOLDER_MODE_LENGTH));
+    rb_define_const(mDataRedactor, "PH_MODE_TAGGED_LENGTH", INT2NUM(PLACEHOLDER_MODE_TAGGED_LENGTH));
 
     /* Tag bitmask values used by the Ruby wrapper to build only/except masks. */
     rb_define_const(mDataRedactor, "TAG_CREDENTIALS", INT2NUM(TAG_CREDENTIALS));

data/ext/data_redactor/placeholder.c

@@ -20,6 +20,10 @@ size_t write_placeholder(char *buf, const placeholder_t *ph,
             unsigned int h = djb2(match, match_len) & 0xFFFF;
             return (size_t)sprintf(buf, "[%s_%04x]", ph->str, h);
         }
+        case PLACEHOLDER_MODE_LENGTH:
+            return (size_t)sprintf(buf, "[REDACTED:%zu]", match_len);
+        case PLACEHOLDER_MODE_TAGGED_LENGTH:
+            return (size_t)sprintf(buf, "[REDACTED:%s:%zu]", ph->str, match_len);
         default: /* PLACEHOLDER_MODE_PLAIN */
             {
                 size_t len = strlen(ph->str);
@@ -29,11 +33,17 @@ size_t write_placeholder(char *buf, const placeholder_t *ph,
     }
 }
 
+/* Widest decimal a size_t byte-length can print to (UINT64_MAX is 20 digits). */
+#define MAX_LEN_DIGITS 20
+
 size_t max_placeholder_len(const placeholder_t *ph) {
     size_t tag_len = strlen(ph->str);
     switch (ph->mode) {
         case PLACEHOLDER_MODE_TAGGED: return 2 + 9 + tag_len + 1; /* "[REDACTED:" + tag + "]" */
         case PLACEHOLDER_MODE_HASH:   return 1 + tag_len + 1 + 4 + 1; /* "[" + tag + "_" + 4hex + "]" */
+        case PLACEHOLDER_MODE_LENGTH: return 10 + MAX_LEN_DIGITS + 1; /* "[REDACTED:" + digits + "]" */
+        case PLACEHOLDER_MODE_TAGGED_LENGTH:
+            return 10 + tag_len + 1 + MAX_LEN_DIGITS + 1; /* "[REDACTED:" + tag + ":" + digits + "]" */
         default:                      return tag_len;
     }
 }

data/ext/data_redactor/placeholder.h

@@ -3,9 +3,11 @@
 
 #include <stddef.h>
 
-#define PLACEHOLDER_MODE_PLAIN  0  /* use ph.str verbatim                  */
-#define PLACEHOLDER_MODE_TAGGED 1  /* "[REDACTED:TAGNAME]"                 */
-#define PLACEHOLDER_MODE_HASH   2  /* "[TAGNAME_xxxx]" (4-hex djb2 suffix) */
+#define PLACEHOLDER_MODE_PLAIN         0  /* use ph.str verbatim                  */
+#define PLACEHOLDER_MODE_TAGGED        1  /* "[REDACTED:TAGNAME]"                 */
+#define PLACEHOLDER_MODE_HASH          2  /* "[TAGNAME_xxxx]" (4-hex djb2 suffix) */
+#define PLACEHOLDER_MODE_LENGTH        3  /* "[REDACTED:16]" (byte-length)        */
+#define PLACEHOLDER_MODE_TAGGED_LENGTH 4  /* "[REDACTED:TAGNAME:16]"              */
 
 typedef struct {
     int         mode;

data/lib/data_redactor/refinements.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+require "data_redactor"
+
+module DataRedactor
+  # Opt-in refinements that add a `#redact` method to `String`, `Hash`, and
+  # `Array` as sugar over {DataRedactor.redact} / {DataRedactor.redact_deep}.
+  #
+  # Refinements are lexically scoped: `#redact` exists only in files that
+  # `using DataRedactor::Refinements`, so loading this file never pollutes the
+  # core classes globally. Apps that don't opt in are unaffected, and there is
+  # no collision risk with other libraries' `String#redact`.
+  #
+  # `DataRedactor.redact` remains the primary API; this is convenience only.
+  #
+  # @example
+  #   require "data_redactor/refinements"
+  #   using DataRedactor::Refinements
+  #
+  #   "email alice@example.com".redact            #=> "email [REDACTED]"
+  #   { token: "AKIAIOSFODNN7EXAMPLE" }.redact    #=> { token: "[REDACTED]" }
+  #   chat.ask(user_input.redact)                 # scrub before sending to an LLM
+  module Refinements
+    refine String do
+      # Redact this String via {DataRedactor.redact}. Returns a new String; the
+      # receiver is not mutated.
+      #
+      # @param only forwarded to {DataRedactor.redact}
+      # @param except forwarded to {DataRedactor.redact}
+      # @param placeholder forwarded to {DataRedactor.redact}
+      # @return [String] the redacted copy.
+      # @example
+      #   "ssn 123-45-6789".redact #=> "ssn [REDACTED]"
+      def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
+        DataRedactor.redact(self, only: only, except: except, placeholder: placeholder)
+      end
+    end
+
+    refine Hash do
+      # Deep-redact this Hash's String values via {DataRedactor.redact_deep}.
+      # Returns a deep copy; the receiver is not mutated and keys are untouched.
+      #
+      # @param only forwarded to {DataRedactor.redact_deep}
+      # @param except forwarded to {DataRedactor.redact_deep}
+      # @param placeholder forwarded to {DataRedactor.redact_deep}
+      # @return [Hash] a deep copy with String leaves redacted.
+      # @example
+      #   { email: "a@b.com" }.redact #=> { email: "[REDACTED]" }
+      def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
+        DataRedactor.redact_deep(self, only: only, except: except, placeholder: placeholder)
+      end
+    end
+
+    refine Array do
+      # Deep-redact this Array's String elements via {DataRedactor.redact_deep}.
+      # Returns a deep copy; the receiver is not mutated.
+      #
+      # @param only forwarded to {DataRedactor.redact_deep}
+      # @param except forwarded to {DataRedactor.redact_deep}
+      # @param placeholder forwarded to {DataRedactor.redact_deep}
+      # @return [Array] a deep copy with String leaves redacted.
+      # @example
+      #   ["a@b.com", 3].redact #=> ["[REDACTED]", 3]
+      def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
+        DataRedactor.redact_deep(self, only: only, except: except, placeholder: placeholder)
+      end
+    end
+  end
+end

data/lib/data_redactor/version.rb

@@ -1,4 +1,4 @@
 module DataRedactor
   # Current gem version. Follows {https://semver.org Semantic Versioning 2.0.0}.
-  VERSION = "0.15.0"
+  VERSION = "0.16.0"
 end

data/lib/data_redactor.rb

@@ -23,8 +23,10 @@ require_relative "data_redactor/name_pattern"
 #
 # @example Custom placeholder
 #   DataRedactor.redact(text, placeholder: "***")
-#   DataRedactor.redact(text, placeholder: :tagged) # => "[REDACTED:CONTACT]"
-#   DataRedactor.redact(text, placeholder: :hash)   # => "[CONTACT_a3f9]"
+#   DataRedactor.redact(text, placeholder: :tagged)        # => "[REDACTED:CONTACT]"
+#   DataRedactor.redact(text, placeholder: :hash)          # => "[CONTACT_a3f9]"
+#   DataRedactor.redact(text, placeholder: :length)        # => "[REDACTED:16]"
+#   DataRedactor.redact(text, placeholder: :tagged_length) # => "[REDACTED:CONTACT:16]"
 #
 # @example Audit / dry-run
 #   DataRedactor.scan(text)
@@ -125,12 +127,15 @@ module DataRedactor
   #   and/or pattern name(s).
   # @param except [Symbol, String, Array, nil] exclude the given tag(s)
   #   and/or pattern name(s). May be combined with +only:+.
-  # @param placeholder [String, :tagged, :hash] replacement strategy.
-  #   A String is used verbatim. +:tagged+ produces +[REDACTED:TAGNAME]+.
-  #   +:hash+ produces a deterministic +[TAGNAME_xxxx]+ token (4-hex djb2)
-  #   so the same input value always maps to the same token.
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length]
+  #   replacement strategy. A String is used verbatim. +:tagged+ produces
+  #   +[REDACTED:TAGNAME]+. +:hash+ produces a deterministic +[TAGNAME_xxxx]+
+  #   token (4-hex djb2) so the same input value always maps to the same token.
+  #   +:length+ produces +[REDACTED:N]+ and +:tagged_length+ produces
+  #   +[REDACTED:TAGNAME:N]+, where +N+ is the byte length of the redacted value.
   # @return [String] a new string with every match replaced.
-  # @raise [ArgumentError] if +placeholder:+ is not a String/:tagged/:hash.
+  # @raise [ArgumentError] if +placeholder:+ is not a String/:tagged/:hash/
+  #   :length/:tagged_length.
   # @raise [UnknownTagError] if any Symbol in +only:+/+except:+ is not in {TAGS}.
   # @raise [UnknownPatternError] if any String in +only:+/+except:+ is not in {pattern_names}.
   #
@@ -194,7 +199,7 @@ module DataRedactor
   #   Any type is accepted; non-String scalars are returned as-is.
   # @param only [Symbol, String, Array, nil] forwarded to {redact}.
   # @param except [Symbol, String, Array, nil] forwarded to {redact}.
-  # @param placeholder [String, :tagged, :hash] forwarded to {redact}.
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
   # @return [Hash, Array, String, Object] a new structure of the same shape
   #   with all String leaves redacted.
   # @raise [ArgumentError] if the structure contains a circular reference.
@@ -217,7 +222,7 @@ module DataRedactor
   # @param json_string [String] valid JSON input.
   # @param only [Symbol, String, Array, nil] forwarded to {redact}.
   # @param except [Symbol, String, Array, nil] forwarded to {redact}.
-  # @param placeholder [String, :tagged, :hash] forwarded to {redact}.
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
   # @return [String] a JSON string with all String values redacted.
   # @raise [JSON::ParserError] if +json_string+ is not valid JSON.
   #
@@ -425,17 +430,20 @@ module DataRedactor
   # Translate the user-facing +placeholder:+ value into the +(mode_int, str)+
   # pair the C layer expects.
   #
-  # @param placeholder [String, :tagged, :hash]
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length]
   # @return [Array(Integer, String)]
   # @raise [ArgumentError] if +placeholder+ is none of the accepted values.
   def resolve_placeholder(placeholder)
     case placeholder
-    when :tagged then [PH_MODE_TAGGED, ""]
-    when :hash   then [PH_MODE_HASH,   ""]
-    when String  then [PH_MODE_PLAIN,  placeholder]
+    when :tagged        then [PH_MODE_TAGGED,        ""]
+    when :hash          then [PH_MODE_HASH,          ""]
+    when :length        then [PH_MODE_LENGTH,        ""]
+    when :tagged_length then [PH_MODE_TAGGED_LENGTH, ""]
+    when String         then [PH_MODE_PLAIN,         placeholder]
     else
       raise ArgumentError,
-        "placeholder must be a String, :tagged, or :hash — got #{placeholder.inspect}"
+        "placeholder must be a String, :tagged, :hash, :length, or :tagged_length " \
+        "— got #{placeholder.inspect}"
     end
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_redactor
 version: !ruby/object:Gem::Version
-  version: 0.15.0
+  version: 0.16.0
 platform: ruby
 authors:
 - Daniele Frisanco
@@ -145,6 +145,7 @@ files:
 - lib/data_redactor/integrations/rack.rb
 - lib/data_redactor/integrations/rails.rb
 - lib/data_redactor/name_pattern.rb
+- lib/data_redactor/refinements.rb
 - lib/data_redactor/version.rb
 homepage: https://github.com/danielefrisanco/data_redactor
 licenses: