prompt-sanitizer 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 7e864f26f679a72a9bad801c49bbe351e708553593dc67441f122e79c6d292b6
+  data.tar.gz: d29b0754dfd1c75caebd1ffd854ae19ae1e4016b9820933a20fa64ddbf87f976
+SHA512:
+  metadata.gz: 9db1e6f1aa3b42cd67cf72368c45c5fb20af4573b9f1f129e36eb83d4ebf80a438b4c9b0e1056065fcdb692eb650175516b2e2782c465bb66d4c90bdd5293577
+  data.tar.gz: 7493b5d3c26df78d25173f53082d92f70c59475eed3f53a6721d5137ca2864e66dbbb1cf02b5c8813f6b553e51bde5c45db42ff00db9d082c42b0e414df12434

data/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog — prompt-sanitizer (Ruby)
+
+All notable changes to this project will be documented in this file.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [Unreleased]
+
+### Added
+- **Rails integrations**: Railtie, Rack middleware, `ActionControllerConcern`,
+  `ActiveJobConcern`, and an install generator (`rails g prompt_sanitizer:install`)
+- **`Session`** — multi-turn vault with `anonymize` / `deanonymize` / `anonymize_with_result`,
+  block form (`session.use { |s| … }`) with automatic vault cleanup
+- **`Sanitizer`** — full pipeline: regex → secrets → NER (mode-gated) → dedup →
+  on_detect callback → replace → reconstruct → audit
+- **`SyntheticEngine`** — Faker-backed realistic replacements for all 27 entity types;
+  graceful fallback to `[TYPE_N]` tokens when Faker is not installed
+- **`MemoryAuditLog`** — thread-safe, Mutex-backed audit log; JSON/CSV export;
+  `since:` / `session_id:` filtering
+- **`AuditEvent`** — struct capturing timestamp, session, entity type, confidence,
+  mode, and SHA-256 hash of the original value (never stores raw PII)
+- **`PIIDetectedError`** — raised in `:block` mode; carries `entities` array
+- **`RegexEngine`** — 27 built-in patterns; `add_pattern` accepts String or Regexp
+- **`SecretsEngine`** — API keys, JWTs, bearer tokens, AWS credentials, private keys,
+  DB connection strings
+- **`NEREngine`** — pluggable backend (`informers` distilbert / `mitie`); lazy load
+
+### Technical notes
+- Zero runtime dependencies in FAST mode
+- Thread-safe throughout (Mutex-backed vault and audit log)
+- Ruby ≥ 3.1 required (Ruby ≥ 3.3 needed for `informers` / `mitie` NER backends)

data/README.md

@@ -0,0 +1,269 @@
+# prompt-sanitizer — Ruby
+
+**Bidirectional PII sanitizer for LLM pipelines. Zero cloud calls. GDPR & HIPAA ready.**
+
+Strips PII from prompts before they reach any model API, then optionally restores
+original values in the response — all in-process, with no third-party telemetry.
+
+---
+
+## Table of contents
+
+- [Installation](#installation)
+- [Quick start](#quick-start)
+- [Modes](#modes)
+- [Multi-turn sessions](#multi-turn-sessions)
+- [Rails integration](#rails-integration)
+  - [Rack middleware](#rack-middleware)
+  - [ActionController concern](#actioncontroller-concern)
+  - [ActiveJob concern](#activejob-concern)
+  - [Install generator](#install-generator)
+- [Audit log](#audit-log)
+- [Custom patterns](#custom-patterns)
+- [Entity types detected](#entity-types-detected)
+- [Optional dependencies](#optional-dependencies)
+- [License](#license)
+
+---
+
+## Installation
+
+```ruby
+# Gemfile
+gem "prompt-sanitizer"
+```
+
+```bash
+bundle install
+```
+
+---
+
+## Quick start
+
+```ruby
+require "prompt_sanitizer"
+
+sanitizer = PromptSanitizer::Sanitizer.new   # FAST mode — zero dependencies
+
+result = sanitizer.sanitize("Hi, I'm John Doe. Reach me at john@acme.com or 555-867-5309")
+puts result.text
+# => "Hi, I'm [PERSON_1]. Reach me at [EMAIL_1] or [PHONE_1]"
+
+puts result.entities.map { |e| [e.type, e.original] }.inspect
+# => [[:person, "John Doe"], [:email, "john@acme.com"], [:phone, "555-867-5309"]]
+```
+
+---
+
+## Modes
+
+| Mode | Engines | Latency | Catches |
+|------|---------|---------|---------|
+| `:fast` *(default)* | Regex + Secrets | < 1 ms | Email, phone, SSN, CC, IBAN, IP, MAC, URL, ZIP, dates, crypto, bank, passport, DL, API keys, JWTs, AWS keys, DB strings |
+| `:smart` | Fast + NER | ~25–50 ms | + Names, organisations, locations, miscellaneous entities |
+| `:full` | Smart + Synthetic + Audit | ~25–50 ms | + Realistic fake replacements, compliance audit trail |
+
+```ruby
+# SMART mode — requires `gem "informers"` (or `gem "mitie"`)
+sanitizer = PromptSanitizer::Sanitizer.new(mode: :smart)
+
+# FULL mode — also requires `gem "faker"`
+sanitizer = PromptSanitizer::Sanitizer.new(mode: :full)
+```
+
+### on_detect callbacks
+
+```ruby
+# :redact (default) — replace PII with tokens
+sanitizer = PromptSanitizer::Sanitizer.new(on_detect: :redact)
+
+# :warn — replace AND call a warning handler
+sanitizer = PromptSanitizer::Sanitizer.new(
+  on_detect: :warn,
+  on_detect_callback: ->(entities) { Rails.logger.warn "PII: #{entities.map(&:type)}" }
+)
+
+# :block — raise PIIDetectedError immediately
+sanitizer = PromptSanitizer::Sanitizer.new(on_detect: :block)
+begin
+  sanitizer.sanitize("SSN: 123-45-6789")
+rescue PromptSanitizer::PIIDetectedError => e
+  puts e.entities.first.type   # => :ssn
+end
+```
+
+---
+
+## Multi-turn sessions
+
+Sessions share a vault across conversation turns so the original values can be
+restored from the model's reply:
+
+```ruby
+sanitizer = PromptSanitizer::Sanitizer.new
+session   = sanitizer.session
+
+# Turn 1
+clean_prompt = session.anonymize("Book a flight for Alice Chen, alice@example.com")
+# => "Book a flight for [PERSON_1], [EMAIL_1]"
+
+llm_reply = YourLLMClient.chat(clean_prompt)
+# => "Sure! I've booked a flight for [PERSON_1] ([EMAIL_1])."
+
+final_reply = session.deanonymize(llm_reply)
+# => "Sure! I've booked a flight for Alice Chen (alice@example.com)."
+
+# Block form — vault is cleared automatically on exit
+sanitizer.session do |s|
+  clean = s.anonymize(user_prompt)
+  s.deanonymize(llm_client.chat(clean))
+end
+```
+
+---
+
+## Rails integration
+
+### Install generator
+
+```bash
+rails generate prompt_sanitizer:install
+```
+
+This creates `config/initializers/prompt_sanitizer.rb` with all options commented.
+
+### Initializer
+
+```ruby
+# config/initializers/prompt_sanitizer.rb
+PromptSanitizer.configure do |config|
+  config.mode        = :smart           # :fast | :smart | :full
+  config.ner_backend = :informers       # :informers | :mitie
+  config.on_detect   = :redact          # :redact | :warn | :block
+  config.audit_log   = :memory          # :memory (more backends coming)
+end
+```
+
+### Rack middleware
+
+Automatically sanitizes JSON request bodies before they hit your controllers.
+Supports `prompt`, `messages[].content` (OpenAI format), `input`, `text`, `query`.
+
+```ruby
+# config/initializers/prompt_sanitizer.rb
+PromptSanitizer.configure do |config|
+  config.use_middleware     = true
+  config.middleware_routes  = ["/api/"]       # only sanitize these path prefixes
+  config.restore_response   = false           # set true to deanonymize JSON responses
+end
+```
+
+Alternatively, insert manually:
+
+```ruby
+# config/application.rb
+config.middleware.use PromptSanitizer::Integrations::SanitizerMiddleware,
+  routes: ["/api/v1/chat"],
+  restore_response: false
+```
+
+### ActionController concern
+
+Fine-grained control inside individual actions:
+
+```ruby
+class ChatController < ApplicationController
+  include PromptSanitizer::Integrations::ActionControllerConcern
+
+  def create
+    # Sanitize specific params in-place
+    sanitize_params!(:prompt, :message)
+
+    # Or use a multi-turn session scoped to this request
+    with_pii_session do |session|
+      clean   = session.anonymize(params[:prompt])
+      raw     = LLMClient.chat(clean)
+      @reply  = session.deanonymize(raw)
+    end
+  end
+end
+```
+
+### ActiveJob concern
+
+Scrubs PII from job arguments before the job performs:
+
+```ruby
+class LLMJob < ApplicationJob
+  include PromptSanitizer::Integrations::ActiveJobConcern
+
+  sanitize_argument :prompt   # sanitized in-place before perform
+
+  def perform(prompt:, user_id:)
+    LLMClient.chat(prompt)    # prompt is already clean
+  end
+end
+```
+
+---
+
+## Audit log
+
+The audit log records every sanitization event — entity type, confidence, session ID,
+and a SHA-256 hash of the original value. **Raw PII is never stored.**
+
+```ruby
+PromptSanitizer.configure do |c|
+  c.audit_log = :memory
+end
+
+sanitizer = PromptSanitizer::Sanitizer.new(mode: :full)
+sanitizer.sanitize("Call Jane at 555-123-4567")
+
+log = PromptSanitizer.audit_log
+puts log.count                          # => 1
+puts log.export(format: :json)          # JSON array of events
+puts log.export(since: "1h")            # events in the last hour
+```
+
+---
+
+## Custom patterns
+
+```ruby
+sanitizer = PromptSanitizer::Sanitizer.new
+sanitizer.add_pattern(/EMP-\d{6}/, :custom)   # employee IDs
+sanitizer.sanitize("Assigned to EMP-004821")
+# => "Assigned to [CUSTOM_1]"
+```
+
+---
+
+## Entity types detected
+
+`EMAIL` · `PHONE` · `SSN` · `CREDIT_CARD` · `IBAN` · `IP_ADDRESS` ·
+`MAC_ADDRESS` · `URL` · `ZIP_CODE` · `DATE_OF_BIRTH` · `DATE` ·
+`CRYPTO_ADDRESS` · `BANK_ACCOUNT` · `PASSPORT` · `DRIVING_LICENSE` ·
+`API_KEY` · `JWT` · `BEARER_TOKEN` · `AWS_ACCESS_KEY` · `AWS_SECRET_KEY` ·
+`PRIVATE_KEY` · `DB_CONNECTION` · `PERSON` · `ORGANIZATION` · `LOCATION` ·
+`AGE` · `CUSTOM`
+
+---
+
+## Optional dependencies
+
+| Gem | Version | Required for |
+|-----|---------|-------------|
+| [`informers`](https://github.com/ankane/informers) | `>= 1.3` | SMART / FULL mode NER (recommended) |
+| [`mitie`](https://github.com/ankane/mitie) | `>= 0.4` | SMART / FULL mode NER (alternative) |
+| [`faker`](https://github.com/faker-ruby/faker) | `>= 2.0` | FULL mode synthetic replacements |
+
+> **Note:** `informers` and `mitie` require Ruby ≥ 3.3.
+
+---
+
+## License
+
+MIT
+

data/lib/generators/prompt_sanitizer/install_generator.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module PromptSanitizer
+  module Generators
+    # Generates a prompt_sanitizer initializer.
+    #
+    # Usage:
+    #   rails g prompt_sanitizer:install
+    #
+    # Creates:
+    #   config/initializers/prompt_sanitizer.rb
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Creates a PromptSanitizer initializer in config/initializers/."
+
+      def create_initializer
+        template "initializer.rb", "config/initializers/prompt_sanitizer.rb"
+      end
+
+      def show_instructions
+        say ""
+        say "✅  prompt_sanitizer initializer created.", :green
+        say ""
+        say "Next steps:", :bold
+        say "  1. Review config/initializers/prompt_sanitizer.rb"
+        say "  2. Choose a mode: :fast (default), :smart (+ NER), or :full (+ audit log)"
+        say "  3. For :smart/:full mode, add to your Gemfile:"
+        say '       gem "informers", ">= 1.3"   # downloads distilbert-NER (~66 MB on first run)'
+        say ""
+        say "  Full docs: https://github.com/jeslor/prompt-sanitizer/tree/main/packages/ruby"
+        say ""
+      end
+    end
+  end
+end

data/lib/generators/prompt_sanitizer/templates/initializer.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+PromptSanitizer.configure do |config|
+  # Detection mode:
+  #   :fast  — regex + secrets patterns only. Zero ML deps. Best for production
+  #            where latency matters and NER is not required. (default)
+  #   :smart — adds NER via the `informers` gem (distilbert-NER, ~66 MB).
+  #            Catches names, orgs, and locations missed by regex alone.
+  #   :full  — SMART + in-memory audit log. Every detection event is recorded
+  #            (hashed, never raw PII) for compliance export.
+  config.mode = :fast
+
+  # BCP-47 locale used by the synthetic replacement engine (Faker).
+  # e.g. "en", "fr", "de", "es", "ja"
+  config.locale = "en"
+
+  # NER backend — used only when mode is :smart or :full.
+  #   :informers — distilbert-NER (ONNX, int8, ~66 MB). Downloads once to
+  #                ~/.cache/huggingface/ on first call. Recommended.
+  #   :mitie     — MITIE C++ library (~600 MB model). Faster than informers
+  #                but requires a separate model file and the `mitie` gem.
+  config.ner_backend = :informers
+
+  # Optional: supply a custom audit log backend (must inherit from
+  # PromptSanitizer::Audit::Base). When nil, a MemoryAuditLog is used
+  # automatically in :full mode.
+  # config.audit_log = MyActiveRecordAuditLog.new
+end
+
+# ── Optional: Rack middleware ──────────────────────────────────────────────────
+# Auto-sanitize JSON request bodies (messages[].content, prompt, input, …)
+# before they reach your controllers. Uncomment to enable.
+#
+# Rails.application.config.prompt_sanitizer.middleware        = true
+# Rails.application.config.prompt_sanitizer.middleware_routes = ["/api/llm", "/chat"]
+# Rails.application.config.prompt_sanitizer.restore_response  = false

data/lib/prompt_sanitizer/audit/base.rb

@@ -0,0 +1,105 @@
+# frozen_string_literal: true
+
+require "digest"
+require "time"
+require "json"
+
+module PromptSanitizer
+  module Audit
+    # Immutable value object representing a single PII-detection event.
+    #
+    # The original PII text is *never* stored — only a 16-char SHA-256 prefix
+    # hash, making the log itself safe to persist or export for compliance.
+    AuditEvent = Struct.new(
+      :timestamp,        # String — ISO-8601 UTC
+      :entity_type,      # Symbol — EntityType constant
+      :confidence,       # Float  — detection confidence 0.0..1.0
+      :layer,            # Symbol — :regex | :secrets | :ner
+      :redaction_method, # Symbol — :synthetic | :placeholder
+      :text_hash,        # String — SHA-256[:16] of original PII (NOT the value)
+      :session_id,       # String | nil — caller-supplied session identifier
+      keyword_init: true
+    ) do
+      def to_h
+        super.transform_keys(&:to_s)
+      end
+    end
+
+    # Compute a SHA-256 prefix hash of a PII value for safe audit storage.
+    def self.hash_value(value)
+      Digest::SHA256.hexdigest(value.to_s)[0, 16]
+    end
+
+    # Return the current UTC time as an ISO-8601 string.
+    def self.now_iso
+      Time.now.utc.iso8601
+    end
+
+    # Parse a "since" argument into a comparable UTC Time.
+    #
+    # Accepts:
+    # - +nil+       → no cutoff
+    # - Integer     → seconds ago
+    # - "7d"        → 7 days ago
+    # - "12h"       → 12 hours ago
+    # - Time        → as-is
+    # - ISO-8601 String → parsed
+    def self.parse_since(since)
+      return nil if since.nil?
+      return since if since.is_a?(Time)
+
+      if since.is_a?(String)
+        if since =~ /\A(\d+)d\z/
+          Time.now.utc - (Regexp.last_match(1).to_i * 86_400) - 1
+        elsif since =~ /\A(\d+)h\z/
+          Time.now.utc - (Regexp.last_match(1).to_i * 3_600) - 1
+        else
+          Time.parse(since).utc
+        end
+      elsif since.is_a?(Integer)
+        Time.now.utc - since
+      end
+    end
+
+    # Abstract base class for audit log backends.
+    #
+    # Subclass and implement +record+, +export+, +count+, and +clear+.
+    #
+    # Example custom backend:
+    #
+    #   class MyAuditLog < PromptSanitizer::Audit::Base
+    #     def record(event) = MyDB.insert(event.to_h)
+    #     def export(format: :json, since: nil, session_id: nil) = "..."
+    #     def count(since: nil) = MyDB.count
+    #     def clear = MyDB.truncate
+    #   end
+    class Base
+      # Record a detection event. Must not store the original PII value.
+      # @param event [AuditEvent]
+      def record(_event)
+        raise NotImplementedError, "#{self.class}#record is not implemented"
+      end
+
+      # Export events as a formatted string.
+      # @param format [Symbol] :json or :csv
+      # @param since  [String, Time, nil] cutoff (e.g. "7d", "1h", Time object)
+      # @param session_id [String, nil] filter to a specific session
+      # @return [String]
+      def export(format: :json, since: nil, session_id: nil)
+        raise NotImplementedError, "#{self.class}#export is not implemented"
+      end
+
+      # Count matching events.
+      # @param since [String, Time, nil]
+      # @return [Integer]
+      def count(since: nil)
+        raise NotImplementedError, "#{self.class}#count is not implemented"
+      end
+
+      # Remove all stored events.
+      def clear
+        raise NotImplementedError, "#{self.class}#clear is not implemented"
+      end
+    end
+  end
+end

data/lib/prompt_sanitizer/audit/memory_audit_log.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+module PromptSanitizer
+  module Audit
+    # Thread-safe in-memory audit log backend.
+    #
+    # Events are stored in a plain Array protected by a Mutex.
+    # Data is lost on process restart — use this for development,
+    # testing, or short-lived jobs. For persistence, implement a
+    # custom backend (e.g. ActiveRecord, SQLite) using Audit::Base.
+    #
+    # Usage:
+    #
+    #   log = PromptSanitizer::Audit::MemoryAuditLog.new
+    #   sanitizer = PromptSanitizer.sanitizer(audit_log: log)
+    #   sanitizer.sanitize("contact john@acme.com")
+    #   log.count          # => 1
+    #   log.export         # => "[{\"entity_type\":\"email\", ...}]"
+    class MemoryAuditLog < Base
+      def initialize
+        super
+        @mutex  = Mutex.new
+        @events = []
+      end
+
+      # @param event [AuditEvent]
+      def record(event)
+        @mutex.synchronize { @events << event }
+        nil
+      end
+
+      # @return [Array<AuditEvent>] a frozen snapshot (thread-safe copy)
+      def events
+        @mutex.synchronize { @events.dup }
+      end
+
+      # @param format  [:json, :csv]
+      # @param since   [String, Time, nil] e.g. "7d", "1h", Time.now - 3600
+      # @param session_id [String, nil]
+      # @return [String]
+      def export(format: :json, since: nil, session_id: nil)
+        rows = _filter(since: since, session_id: session_id).map(&:to_h)
+        case format.to_sym
+        when :json
+          JSON.generate(rows)
+        when :csv
+          return "" if rows.empty?
+
+          fields = rows.first.keys
+          lines  = [fields.join(",")]
+          rows.each { |r| lines << fields.map { |f| r[f].to_s }.join(",") }
+          lines.join("\n")
+        else
+          raise ArgumentError, "Unknown format: #{format.inspect}. Use :json or :csv"
+        end
+      end
+
+      # @param since [String, Time, nil]
+      # @return [Integer]
+      def count(since: nil)
+        _filter(since: since).length
+      end
+
+      def clear
+        @mutex.synchronize { @events.clear }
+        nil
+      end
+
+      private
+
+      def _filter(since: nil, session_id: nil)
+        cutoff = Audit.parse_since(since)
+        evts   = @mutex.synchronize { @events.dup }
+
+        if cutoff
+          evts = evts.select do |e|
+            Time.parse(e.timestamp).utc >= cutoff
+          end
+        end
+
+        evts = evts.select { |e| e.session_id == session_id } if session_id
+        evts
+      end
+    end
+  end
+end