top_secret 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d570b8ecd9cb5ab35f59dc688a6a13a749cebf1abfcbce36906f12d1d8452189
-  data.tar.gz: 8b5d639506a7ecd6bbb548e245db54384a2d16c0b47bbd564230c4140639c05d
+  metadata.gz: 7b57ce12774e37584e4d7a25a2606799b642ddff1eaaeb6f8fa53c2c1e4cae58
+  data.tar.gz: 60b70028bdf30ea474c1987ea15a27df87c7d5614743b69e9f8a20e00a358401
 SHA512:
-  metadata.gz: fef5120cc93ac1772270816788ee9bd3a7141779f300a839ecb7d8d35d228c7539e33bb55cac1ff97d975b35251d46913020879d985dad75655de385a47b31ca
-  data.tar.gz: 5acc9bb4f77210d0ae804ee50472ec75952f12812892c6d304feb4209ee152a5d3a3740fe938d17dd47cb664945d90dc73ddfa1578b2c4f1400a1ef23d8e93bb
+  metadata.gz: de3cb592711f29551bdcb81c8e2c4effb03d7d0e47a1c995adb7a4c828652a98050f133918b63c9a5cbdd3e03aba1e15c0abd06e969f9c7f711b5ce9f22247de
+  data.tar.gz: cb08a6e173106fa653f16f9abbf8ee7b1dd6018fe4097b6841701004ada46ab37b4a9be2079da7398233e35a6c334ee1a445c8d5f0faf7c82eb4c35b0343b79c

data/CHANGELOG.md

@@ -1,5 +1,32 @@
 ## [Unreleased]
 
-## [0.1.0] - 2025-07-23
+## [0.2.0] - 2025-08-18
 
-- Initial release
+### Added
+
+-   Added `TopSecret::Text.filter_all` for batch processing multiple messages with globally consistent redaction labels
+-   Added `TopSecret::Text::BatchResult` class to hold results from batch operations
+-   Added `TopSecret::FilteredText` class for restoring filtered text by substituting placeholders with original values
+-   Added `TopSecret::FilteredText::Result` class to track restoration success and failures
+
+### Changed
+
+-   **BREAKING:** Moved `TopSecret::Result` to `TopSecret::Text::Result` and `TopSecret::BatchResult` to `TopSecret::Text::BatchResult` for better namespace organization
+-   **BREAKING:** Refactored configuration system to use individual filter accessors instead of nested `default_filters`
+-   Updated `TopSecret::Text.filter` to accept keyword arguments for filter overrides and `custom_filters` array
+-   Each default filter now has its own configuration accessor (e.g., `TopSecret.email_filter`, `TopSecret.people_filter`)
+
+### Migration Guide
+
+-   Replace `TopSecret::Result` with `TopSecret::Text::Result` and `TopSecret::BatchResult` with `TopSecret::Text::BatchResult`
+-   Replace `TopSecret.configure { |c| c.default_filters.email_filter = filter }` with `TopSecret.configure { |c| c.email_filter = filter }`
+-   Replace `TopSecret::Text.filter(text, filters: { email_filter: filter })` with `TopSecret::Text.filter(text, email_filter: filter)`
+-   For new filters, use `TopSecret::Text.filter(text, custom_filters: [filter])` instead of adding to `default_filters`
+
+## [0.1.1] - 2025-08-08
+
+-   Ensure `TopSecret.min_confidence_score` is respected
+
+## [0.1.0] - 2025-08-08
+
+-   Initial release

data/CODEOWNERS

@@ -0,0 +1,15 @@
+# Lines starting with '#' are comments.
+# Each line is a file pattern followed by one or more owners.
+
+# More details are here: https://help.github.com/articles/about-codeowners/
+
+# The '*' pattern is global owners.
+
+# Order is important. The last matching pattern has the most precedence.
+# The folders are ordered as follows:
+
+# In each subsection folders are ordered first by depth, then alphabetically.
+# This should make it easy to add new rules without breaking existing ones.
+
+# Global rule:
+* @stevepolitodesign

data/README.md

@@ -1,6 +1,8 @@
 # Top Secret
 
-Filter sensitive information from free text before sending it to external services or APIs, such as Chatbots.
+[![Ruby](https://github.com/thoughtbot/top_secret/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/thoughtbot/top_secret/actions/workflows/main.yml)
+
+Filter sensitive information from free text before sending it to external services or APIs, such as chatbots and LLMs.
 
 By default it filters the following:
 
@@ -123,7 +125,7 @@ TopSecret::Text.filter("Ralph can be reached at ralph@thoughtbot.com")
 This will return
 
 ```ruby
-<TopSecret::Result
+<TopSecret::Text::Result
   @input="Ralph can be reached at ralph@thoughtbot.com",
   @mapping={:EMAIL_1=>"ralph@thoughtbot.com", :PERSON_1=>"Ralph"},
   @output="[PERSON_1] can be reached at [EMAIL_1]"
@@ -154,26 +156,123 @@ result.mapping
 # => {:EMAIL_1=>"ralph@thoughtbot.com", :PERSON_1=>"Ralph"}
 ```
 
+### Batch Processing
+
+When processing multiple messages, use `filter_all` to ensure consistent redaction labels across all messages:
+
+```ruby
+messages = [
+  "Contact ralph@thoughtbot.com for details",
+  "Email ralph@thoughtbot.com again if needed",
+  "Also CC ruby@thoughtbot.com on the thread"
+]
+
+result = TopSecret::Text.filter_all(messages)
+```
+
+This will return
+
+```ruby
+<TopSecret::Text::BatchResult
+  @mapping={:EMAIL_1=>"ralph@thoughtbot.com", :EMAIL_2=>"ruby@thoughtbot.com"},
+  @items=[
+    <TopSecret::Text::BatchResult::Item @input="Contact ralph@thoughtbot.com for details", @output="Contact [EMAIL_1] for details">,
+    <TopSecret::Text::BatchResult::Item @input="Email ralph@thoughtbot.com again if needed", @output="Email [EMAIL_1] again if needed">,
+    <TopSecret::Text::BatchResult::Item @input="Also CC ruby@thoughtbot.com on the thread", @output="Also CC [EMAIL_2] on the thread">
+  ]
+>
+```
+
+Access the global mapping
+
+```ruby
+result.mapping
+
+# => {:EMAIL_1=>"ralph@thoughtbot.com", :EMAIL_2=>"ruby@thoughtbot.com"}
+```
+
+Access individual items
+
+```ruby
+result.items[0].input
+# => "Contact ralph@thoughtbot.com for details"
+
+result.items[0].output
+# => "Contact [EMAIL_1] for details"
+```
+
+The key benefit is that identical values receive the same labels across all messages - notice how `ralph@thoughtbot.com` becomes `[EMAIL_1]` in both the first and second messages.
+
+### Restoring Filtered Text
+
+When external services (like LLMs) return responses containing filter placeholders, use `TopSecret::FilteredText.restore` to substitute them back with original values:
+
+```ruby
+# Filter messages before sending to LLM
+messages = ["Contact ralph@thoughtbot.com for details"]
+batch_result = TopSecret::Text.filter_all(messages)
+
+# Send filtered text to LLM: "Contact [EMAIL_1] for details"
+# LLM responds with: "I'll email [EMAIL_1] about this request"
+llm_response = "I'll email [EMAIL_1] about this request"
+
+# Restore the original values
+restore_result = TopSecret::FilteredText.restore(llm_response, mapping: batch_result.mapping)
+```
+
+This will return
+
+```ruby
+<TopSecret::FilteredText::Result
+  @output="I'll email ralph@thoughtbot.com about this request",
+  @restored=["[EMAIL_1]"],
+  @unrestored=[]
+>
+```
+
+Access the restored text
+
+```ruby
+restore_result.output
+# => "I'll email ralph@thoughtbot.com about this request"
+```
+
+Track which placeholders were restored
+
+```ruby
+restore_result.restored
+# => ["[EMAIL_1]"]
+
+restore_result.unrestored
+# => []
+```
+
+The restoration process tracks both successful and failed placeholder substitutions, allowing you to handle cases where the LLM response contains placeholders not found in your mapping.
+
 ### Advanced Examples
 
 #### Overriding the default filters
 
 When overriding or [disabling](#disabling-a-default-filter-1) a [default filter](#default-filters), you must map to the correct key.
 
+> [!IMPORTANT]
+> Invalid filter keys will raise an `ArgumentError`. Only the following keys are valid:
+> `credit_card_filter`, `email_filter`, `phone_number_filter`, `ssn_filter`, `people_filter`, `location_filter`
+
 ```ruby
 regex_filter = TopSecret::Filters::Regex.new(label: "EMAIL_ADDRESS", regex: /\b\w+\[at\]\w+\.\w+\b/)
 ner_filter = TopSecret::Filters::NER.new(label: "NAME", tag: :person, min_confidence_score: 0.25)
 
-TopSecret::Text.filter("Ralph can be reached at ralph[at]thoughtbot.com", filters: {
+TopSecret::Text.filter("Ralph can be reached at ralph[at]thoughtbot.com",
   email_filter: regex_filter,
   people_filter: ner_filter
-})
+)
 ```
 
 This will return
 
 ```ruby
-<TopSecret::Result
+<TopSecret::Text::Result
   @input="Ralph can be reached at ralph[at]thoughtbot.com",
   @mapping={:EMAIL_ADDRESS_1=>"ralph[at]thoughtbot.com", :NAME_1=>"Ralph", :NAME_2=>"ralph["},
   @output="[NAME_1] can be reached at [EMAIL_ADDRESS_1]"
@@ -183,22 +282,29 @@ This will return
 #### Disabling a default filter
 
 ```ruby
-TopSecret::Text.filter("Ralph can be reached at ralph@thoughtbot.com", filters: {
+TopSecret::Text.filter("Ralph can be reached at ralph@thoughtbot.com",
   email_filter: nil,
   people_filter: nil
-})
+)
 ```
 
 This will return
 
 ```ruby
-<TopSecret::Result
+<TopSecret::Text::Result
   @input="Ralph can be reached at ralph@thoughtbot.com",
   @mapping={},
   @output="Ralph can be reached at ralph@thoughtbot.com"
 >
 ```
 
+#### Error handling for invalid filter keys
+
+```ruby
+# This will raise ArgumentError: Unknown key: :invalid_filter. Valid keys are: ...
+TopSecret::Text.filter("some text", invalid_filter: some_filter)
+```
+
 ### Custom Filters
 
 #### Adding new [Regex filters][]
@@ -209,15 +315,15 @@ ip_address_filter = TopSecret::Filters::Regex.new(
   regex: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/
 )
 
-TopSecret::Text.filter("Ralph's IP address is 192.168.1.1", filters: {
-  ip_address_filter: ip_address_filter
-})
+TopSecret::Text.filter("Ralph's IP address is 192.168.1.1",
+  custom_filters: [ip_address_filter]
+)
 ```
 
 This will return
 
 ```ruby
-<TopSecret::Result
+<TopSecret::Text::Result
   @input="Ralph's IP address is 192.168.1.1",
   @mapping={:PERSON_1=>"Ralph", :IP_ADDRESS_1=>"192.168.1.1"},
   @output="[PERSON_1]'s IP address is [IP_ADDRESS_1]"
@@ -235,15 +341,15 @@ language_filter = TopSecret::Filters::NER.new(
   min_confidence_score: 0.75
 )
 
-TopSecret::Text.filter("Ralph's favorite programming language is Ruby.", filters: {
-  language_filter: language_filter
-})
+TopSecret::Text.filter("Ralph's favorite programming language is Ruby.",
+  custom_filters: [language_filter]
+)
 ```
 
 This will return
 
 ```ruby
-<TopSecret::Result
+<TopSecret::Text::Result
   @input="Ralph's favorite programming language is Ruby.",
   @mapping={:PERSON_1=>"Ralph", :LANGUAGE_1=>"Ruby"},
   @output="[PERSON_1]'s favorite programming language is [LANGUAGE_1]"
@@ -265,9 +371,9 @@ regex_filter = TopSecret::Filters::Regex.new(
   regex: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/
 )
 
-result = TopSecret::Text.filter("Server IP: 192.168.1.1", filters: {
-  ip_address_filter: regex_filter
-})
+result = TopSecret::Text.filter("Server IP: 192.168.1.1",
+  custom_filters: [regex_filter]
+)
 
 result.output
 # => "Server IP: [IP_ADDRESS_1]"
@@ -285,9 +391,9 @@ ner_filter = TopSecret::Filters::NER.new(
   min_confidence_score: 0.25
 )
 
-result = TopSecret::Text.filter("Ralph and Ruby work at thoughtbot.", filters: {
+result = TopSecret::Text.filter("Ralph and Ruby work at thoughtbot.",
   people_filter: ner_filter
-})
+)
 
 result.output
 # => "[PERSON_1] and [PERSON_2] work at thoughtbot."
@@ -326,7 +432,7 @@ end
 
 ```ruby
 TopSecret.configure do |config|
-  config.default_filters.email_filter = TopSecret::Filters::Regex.new(
+  config.email_filter = TopSecret::Filters::Regex.new(
     label: "EMAIL_ADDRESS",
     regex: /\b\w+\[at\]\w+\.\w+\b/
   )
@@ -337,18 +443,20 @@ end
 
 ```ruby
 TopSecret.configure do |config|
-  config.default_filters.email_filter = nil
+  config.email_filter = nil
 end
 ```
 
-### Adding new default filters
+### Adding custom filters globally
 
 ```ruby
+ip_address_filter = TopSecret::Filters::Regex.new(
+  label: "IP_ADDRESS",
+  regex: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/
+)
+
 TopSecret.configure do |config|
-  config.default_filters.ip_address_filter = TopSecret::Filters::Regex.new(
-    label: "IP_ADDRESS",
-    regex: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/
-  )
+  config.custom_filters << ip_address_filter
 end
 ```
 
@@ -361,11 +469,26 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
 >
 > You'll need to download and extract [ner_model.dat][] first, and place it in the root of this project.
 
+### Performance Benchmarks
+
+Run `bin/benchmark` to test performance and catch regressions:
+
+```bash
+bin/benchmark  # CI-optimized benchmark with pass/fail thresholds
+```
+
+> [!NOTE]
+> When adding new public methods to the API, ensure they are included in the benchmark script to catch performance regressions.
+
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at [https://github.com/thoughtbot/top_secret](https://github.com/thoughtbot/top_secret). This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/thoughtbot/top_secret/blob/main/CODE_OF_CONDUCT.md).
+[Bug reports](https://github.com/thoughtbot/top_secret/issues/new?template=bug_report.md) and [pull requests](https://github.com/thoughtbot/top_secret/pulls) are welcome on GitHub at [https://github.com/thoughtbot/top_secret](https://github.com/thoughtbot/top_secret).
+
+Please create a [new discussion](https://github.com/thoughtbot/top_secret/discussions/new?category=ideas) if you want to share ideas for new features.
+
+This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/thoughtbot/top_secret/blob/main/CODE_OF_CONDUCT.md).
 
 ## License
 

data/lib/top_secret/filtered_text/result.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module TopSecret
+  class FilteredText
+    # Result object returned by FilteredText restoration operations.
+    #
+    # Contains the restored text along with tracking information about which
+    # placeholders were successfully restored and which remain unrestored.
+    class Result
+      # @return [String] The text with placeholders restored to original values
+      attr_reader :output
+
+      # @return [Array<String>] Array of placeholder strings that could not be restored
+      attr_reader :unrestored
+
+      # @return [Array<String>] Array of placeholder strings that were successfully restored
+      attr_reader :restored
+
+      # @param output [String] The restored text
+      # @param unrestored [Array<String>] Placeholders that could not be restored
+      # @param restored [Array<String>] Placeholders that were successfully restored
+      def initialize(output, unrestored, restored)
+        @output = output
+        @unrestored = unrestored
+        @restored = restored
+      end
+    end
+  end
+end

data/lib/top_secret/filtered_text.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+require_relative "filtered_text/result"
+
+module TopSecret
+  # Restores filtered text by substituting placeholders with original values.
+  #
+  # This class is used to reverse the filtering process, typically when processing
+  # responses from external services like LLMs that may contain filtered placeholders.
+  class FilteredText
+    # @return [String] The text being processed for restoration
+    attr_reader :output
+
+    # @param filtered_text [String] Text containing filter placeholders like [EMAIL_1]
+    # @param mapping [Hash] Hash mapping filter symbols to original values
+    def initialize(filtered_text, mapping:)
+      @mapping = mapping
+      @output = filtered_text.dup
+    end
+
+    # Convenience method to restore filtered text in one call
+    #
+    # @param filtered_text [String] Text containing filter placeholders
+    # @param mapping [Hash] Hash mapping filter symbols to original values
+    # @return [Result] Contains restored text and tracking information
+    #
+    # @example Basic restoration
+    #   mapping = {EMAIL_1: "john@example.com"}
+    #   result = TopSecret::FilteredText.restore("Contact [EMAIL_1]", mapping: mapping)
+    #   result.output # => "Contact john@example.com"
+    #   result.restored # => ["[EMAIL_1]"]
+    #   result.unrestored # => []
+    def self.restore(filtered_text, mapping:)
+      new(filtered_text, mapping:).restore
+    end
+
+    # Performs the restoration process
+    #
+    # Substitutes all found placeholders with their mapped values and tracks
+    # which placeholders were successfully restored vs those that remain unrestored.
+    #
+    # @return [Result] Contains the restored text and tracking arrays
+    def restore
+      restored = []
+
+      mapping.each do |filter, value|
+        placeholder = build_placeholder(filter)
+
+        if output.include? placeholder
+          restored << placeholder
+          output.gsub! placeholder, value
+        end
+      end
+
+      unrestored = output.scan(/\[\w*_\d\]/)
+
+      Result.new(output, unrestored, restored)
+    end
+
+    private
+
+    # @return [Hash] Mapping from filter symbols to original values
+    attr_reader :mapping
+
+    # Builds a placeholder string from a filter symbol
+    #
+    # @param filter [Symbol] The filter symbol (e.g., :EMAIL_1)
+    # @return [String] The placeholder string (e.g., "[EMAIL_1]")
+    def build_placeholder(filter)
+      "[#{filter}]"
+    end
+  end
+end

data/lib/top_secret/filters/ner.rb

@@ -13,7 +13,7 @@ module TopSecret
       def initialize(label:, tag:, min_confidence_score: nil)
         @label = label
         @tag = tag.upcase.to_s
-        @min_confidence_score = min_confidence_score || TopSecret.min_confidence_score
+        @min_confidence_score = min_confidence_score
       end
 
       # Filters and extracts entity texts matching the tag and score threshold.
@@ -21,7 +21,7 @@ module TopSecret
       # @param entities [Array<Hash>] List of entity hashes with keys :tag, :score, and :text
       # @return [Array<String>] Matched entity texts
       def call(entities)
-        tags = entities.filter { _1.fetch(:tag) == tag && _1.fetch(:score) >= min_confidence_score }
+        tags = entities.filter { _1.fetch(:tag) == tag && _1.fetch(:score) >= (min_confidence_score || TopSecret.min_confidence_score) }
         tags.map { _1.fetch(:text) }
       end
 

data/lib/top_secret/text/batch_result.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module TopSecret
+  class Text
+    # Holds the result of a batch redaction operation on multiple messages.
+    # Contains a global mapping that ensures consistent labeling across all messages
+    # and a collection of individual input/output pairs.
+    class BatchResult
+      # @return [Hash] Global mapping of redaction labels to original values across all messages
+      attr_reader :mapping
+
+      # @return [Array<Item>] Array of input/output pairs for each processed message
+      attr_reader :items
+
+      # Creates a new BatchResult instance
+      #
+      # @param mapping [Hash] Global mapping of redaction labels to original values
+      # @param items [Array<Item>] Array of input/output pairs
+      def initialize(mapping: {}, items: [])
+        @mapping = mapping
+        @items = items
+      end
+
+      # Represents a single message within a batch redaction operation.
+      # Contains only the input and output text, without individual mappings.
+      # The mapping is maintained at the BatchResult level for global consistency.
+      class Item
+        # @return [String] The original unredacted input
+        attr_reader :input
+
+        # @return [String] The redacted output
+        attr_reader :output
+
+        # Creates a new Item instance
+        #
+        # @param input [String] The original text
+        # @param output [String] The redacted text
+        def initialize(input, output)
+          @input = input
+          @output = output
+        end
+      end
+    end
+  end
+end

data/lib/top_secret/text/result.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module TopSecret
+  class Text
+    # Holds the result of a redaction operation.
+    class Result
+      # @return [String] The original unredacted input
+      attr_reader :input
+
+      # @return [String] The redacted output
+      attr_reader :output
+
+      # @return [Hash] Mapping of redacted labels to matched values
+      attr_reader :mapping
+
+      # @param input [String] The original text
+      # @param output [String] The redacted text
+      # @param mapping [Hash] Map of labels to matched values
+      def initialize(input, output, mapping)
+        @input = input
+        @output = output
+        @mapping = mapping
+      end
+    end
+  end
+end

data/lib/top_secret/text.rb

@@ -1,37 +1,110 @@
 # frozen_string_literal: true
 
+require "active_support/core_ext/hash/keys"
+require_relative "text/result"
+require_relative "text/batch_result"
+
 module TopSecret
   # Processes text to identify and redact sensitive information using configured filters.
   class Text
     # @param input [String] The original text to be filtered
     # @param filters [Hash, nil] Optional set of filters to override the defaults
-    def initialize(input, filters: TopSecret.default_filters)
+    # @param custom_filters [Array] Additional custom filters to apply
+    # @param model [Mitie::NER, nil] Optional pre-loaded MITIE model for performance
+    def initialize(input, custom_filters: [], filters: {}, model: nil)
       @input = input
       @output = input.dup
       @mapping = {}
 
-      @model = Mitie::NER.new(TopSecret.model_path)
+      @model = model || Mitie::NER.new(TopSecret.model_path)
       @doc = @model.doc(@output)
       @entities = @doc.entities
 
       @filters = filters
+      @custom_filters = custom_filters
     end
 
     # Convenience method to create an instance and filter input
     #
     # @param input [String] The text to filter
-    # @param filters [Hash] Optional filters to override defaults
+    # @param filters [Hash] Optional filters to override defaults (only valid filter keys accepted)
+    # @param custom_filters [Array] Additional custom filters to apply
     # @return [Result] The filtered result
-    def self.filter(input, filters: {})
-      new(input, filters:).filter
+    # @raise [ArgumentError] If invalid filter keys are provided
+    def self.filter(input, custom_filters: [], **filters)
+      new(input, filters:, custom_filters:).filter
+    end
+
+    # Filters multiple messages with globally consistent redaction labels
+    #
+    # Processes a collection of messages and ensures that identical sensitive values
+    # receive the same redaction labels across all messages. This is useful when
+    # processing conversation threads or document collections where consistency matters.
+    #
+    # @param messages [Array<String>] Array of text messages to filter
+    # @param custom_filters [Array] Additional custom filters to apply
+    # @param filters [Hash] Optional filters to override defaults (only valid filter keys accepted)
+    # @return [BatchResult] Contains global mapping and array of input/output pairs
+    # @raise [ArgumentError] If invalid filter keys are provided
+    #
+    # @example Basic usage
+    #   messages = ["Contact john@test.com", "Email john@test.com again"]
+    #   result = TopSecret::Text.filter_all(messages)
+    #   result.items[0].output # => "Contact [EMAIL_1]"
+    #   result.items[1].output # => "Email [EMAIL_1] again"
+    #   result.mapping # => { EMAIL_1: "john@test.com" }
+    #
+    # @example With custom filters
+    #   ip_filter = TopSecret::Filters::Regex.new(label: "IP", regex: /\d+\.\d+\.\d+\.\d+/)
+    #   result = TopSecret::Text.filter_all(messages, custom_filters: [ip_filter])
+    def self.filter_all(messages, custom_filters: [], **filters)
+      shared_model = Mitie::NER.new(TopSecret.model_path)
+
+      individual_results = messages.map do |message|
+        new(message, filters:, custom_filters:, model: shared_model).filter
+      end
+
+      global_mapping = {}
+      label_counters = {}
+
+      individual_results.each do |result|
+        result.mapping.each do |individual_key, value|
+          next if global_mapping.key?(value)
+
+          # TODO: This assumes labels are formatted consistently.
+          # We need to account for the following for the case where a label could begin with an "_"
+          label_type = individual_key.to_s.rpartition("_").first
+
+          label_counters[label_type] ||= 0
+          label_counters[label_type] += 1
+          global_key = :"#{label_type}_#{label_counters[label_type]}"
+
+          global_mapping[value] = global_key
+        end
+      end
+
+      inverted_global_mapping = global_mapping.invert
+
+      items = individual_results.map do |result|
+        output = result.input.dup
+        inverted_global_mapping.each { |filter, value| output.gsub!(value, "[#{filter}]") }
+        Text::BatchResult::Item.new(result.input, output)
+      end
+
+      Text::BatchResult.new(mapping: global_mapping.invert, items:)
     end
 
     # Applies configured filters to the input, redacting matches and building a mapping.
     #
     # @return [Result] Contains original input, redacted output, and mapping of labels to values
     # @raise [Error] If an unsupported filter is encountered
+    # @raise [ArgumentError] If invalid filter keys are provided
     def filter
-      TopSecret.default_filters.merge(filters).compact.each_value do |filter|
+      validate_filters!
+
+      all_filters.each do |filter|
+        next if filter.nil?
+
         values = case filter
         when TopSecret::Filters::Regex
           filter.call(input)
@@ -45,7 +118,7 @@ module TopSecret
 
       substitute_text
 
-      Result.new(input, output, mapping)
+      Text::Result.new(input, output, mapping)
     end
 
     private
@@ -65,6 +138,9 @@ module TopSecret
     # @return [Hash] Active filters used for redaction
     attr_reader :filters
 
+    # @return [Array] Custom filters to apply
+    attr_reader :custom_filters
+
     # Builds the mapping of label keys to matched values, indexed uniquely.
     #
     # @param values [Array<String>] Values matched by a filter
@@ -85,5 +161,43 @@ module TopSecret
         output.gsub! value, "[#{filter}]"
       end
     end
+
+    # Collects all filters to apply: default filters with overrides plus custom filters
+    #
+    # @return [Array] Array of filter objects to apply
+    def all_filters
+      merged_filters.values.compact + TopSecret.custom_filters + custom_filters
+    end
+
+    # Merges default filters with user-provided filter overrides
+    #
+    # @return [Hash] Hash containing default filters with any user overrides applied
+    # @private
+    def merged_filters
+      default_filters.merge(filters)
+    end
+
+    # Validates that all provided filter keys are recognized
+    #
+    # @return [void]
+    # @raise [ArgumentError] If invalid filter keys are provided
+    def validate_filters!
+      merged_filters.assert_valid_keys(*default_filters.keys)
+    end
+
+    # Returns the default filters configuration hash
+    #
+    # @return [Hash] Hash containing all configured default filters, keyed by filter name
+    # @private
+    def default_filters
+      {
+        credit_card_filter: TopSecret.credit_card_filter,
+        email_filter: TopSecret.email_filter,
+        phone_number_filter: TopSecret.phone_number_filter,
+        ssn_filter: TopSecret.ssn_filter,
+        people_filter: TopSecret.people_filter,
+        location_filter: TopSecret.location_filter
+      }
+    end
   end
 end

data/lib/top_secret/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module TopSecret
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/top_secret.rb

@@ -11,10 +11,10 @@ require_relative "top_secret/constants"
 require_relative "top_secret/filters/ner"
 require_relative "top_secret/filters/regex"
 require_relative "top_secret/error"
-require_relative "top_secret/result"
 require_relative "top_secret/text"
+require_relative "top_secret/filtered_text"
 
-# TopSecret filters sensitive information from free text before it's sent to external services or APIs, such as Chatbots.
+# TopSecret filters sensitive information from free text before it's sent to external services or APIs, such as chatbots and LLMs.
 #
 # @!attribute [rw] model_path
 #   @return [String] the path to the MITIE NER model
@@ -22,23 +22,38 @@ require_relative "top_secret/text"
 # @!attribute [rw] min_confidence_score
 #   @return [Float] the minimum confidence score required for NER matches
 #
-# @!attribute [rw] default_filters
-#   @return [ActiveSupport::OrderedOptions] a set of default filters used to identify sensitive data
+# @!attribute [rw] custom_filters
+#   @return [Array] array of custom filters that can be configured
+#
+# @!attribute [rw] credit_card_filter
+#   @return [TopSecret::Filters::Regex] filter for credit card numbers
+#
+# @!attribute [rw] email_filter
+#   @return [TopSecret::Filters::Regex] filter for email addresses
+#
+# @!attribute [rw] phone_number_filter
+#   @return [TopSecret::Filters::Regex] filter for phone numbers
+#
+# @!attribute [rw] ssn_filter
+#   @return [TopSecret::Filters::Regex] filter for social security numbers
+#
+# @!attribute [rw] people_filter
+#   @return [TopSecret::Filters::NER] filter for person names
+#
+# @!attribute [rw] location_filter
+#   @return [TopSecret::Filters::NER] filter for location names
 module TopSecret
   include ActiveSupport::Configurable
 
   config_accessor :model_path, default: "ner_model.dat"
   config_accessor :min_confidence_score, default: MIN_CONFIDENCE_SCORE
 
-  config_accessor :default_filters do
-    options = ActiveSupport::OrderedOptions.new
-    options.credit_card_filter = TopSecret::Filters::Regex.new(label: "CREDIT_CARD", regex: CREDIT_CARD_REGEX)
-    options.email_filter = TopSecret::Filters::Regex.new(label: "EMAIL", regex: EMAIL_REGEX)
-    options.phone_number_filter = TopSecret::Filters::Regex.new(label: "PHONE_NUMBER", regex: PHONE_REGEX)
-    options.ssn_filter = TopSecret::Filters::Regex.new(label: "SSN", regex: SSN_REGEX)
-    options.people_filter = TopSecret::Filters::NER.new(label: "PERSON", tag: :person)
-    options.location_filter = TopSecret::Filters::NER.new(label: "LOCATION", tag: :location)
+  config_accessor :custom_filters, default: []
 
-    options
-  end
+  config_accessor :credit_card_filter, default: TopSecret::Filters::Regex.new(label: "CREDIT_CARD", regex: CREDIT_CARD_REGEX)
+  config_accessor :email_filter, default: TopSecret::Filters::Regex.new(label: "EMAIL", regex: EMAIL_REGEX)
+  config_accessor :phone_number_filter, default: TopSecret::Filters::Regex.new(label: "PHONE_NUMBER", regex: PHONE_REGEX)
+  config_accessor :ssn_filter, default: TopSecret::Filters::Regex.new(label: "SSN", regex: SSN_REGEX)
+  config_accessor :people_filter, default: TopSecret::Filters::NER.new(label: "PERSON", tag: :person)
+  config_accessor :location_filter, default: TopSecret::Filters::NER.new(label: "LOCATION", tag: :location)
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: top_secret
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Steve Polito
@@ -44,7 +44,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 0.3.2
 description: Filter sensitive information from free text before sending it to external
-  services or APIs, such as Chatbots.
+  services or APIs, such as chatbots and LLMs.
 email:
 - stevepolito@hey.com
 executables: []
@@ -52,6 +52,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - CHANGELOG.md
+- CODEOWNERS
 - CODE_OF_CONDUCT.md
 - LICENSE.txt
 - README.md
@@ -59,10 +60,13 @@ files:
 - lib/top_secret.rb
 - lib/top_secret/constants.rb
 - lib/top_secret/error.rb
+- lib/top_secret/filtered_text.rb
+- lib/top_secret/filtered_text/result.rb
 - lib/top_secret/filters/ner.rb
 - lib/top_secret/filters/regex.rb
-- lib/top_secret/result.rb
 - lib/top_secret/text.rb
+- lib/top_secret/text/batch_result.rb
+- lib/top_secret/text/result.rb
 - lib/top_secret/version.rb
 - sig/top_secret.rbs
 homepage: https://github.com/thoughtbot/top_secret

data/lib/top_secret/result.rb

@@ -1,24 +0,0 @@
-# frozen_string_literal: true
-
-module TopSecret
-  # Holds the result of a redaction operation.
-  class Result
-    # @return [String] The original unredacted input
-    attr_reader :input
-
-    # @return [String] The redacted output
-    attr_reader :output
-
-    # @return [Hash] Mapping of redacted labels to matched values
-    attr_reader :mapping
-
-    # @param input [String] The original text
-    # @param output [String] The redacted text
-    # @param mapping [Hash] Map of labels to matched values
-    def initialize(input, output, mapping)
-      @input = input
-      @output = output
-      @mapping = mapping
-    end
-  end
-end