better_translate 0.5.0 → 1.0.0

data/docs/implementation/03.5-variable_preservation.md

@@ -0,0 +1,509 @@
+# 03.5 - Variable Preservation (Critical Feature)
+
+[← Previous: 03-Core Components](./03-core_components.md) | [Back to Index](../../IMPLEMENTATION_PLAN.md) | [Next: 04-Provider Architecture →](./04-provider_architecture.md)
+
+---
+
+## Variable Preservation System
+
+**PRIORITY: CRITICAL** - This feature prevents bugs by ensuring interpolation variables are not translated or modified.
+
+### Overview
+
+When translating text that contains variables (e.g., `%{name}`, `{{user}}`), we must ensure these variables remain unchanged. This system extracts variables before translation, replaces them with safe placeholders, and restores them after translation.
+
+---
+
+## 3.5.1 `lib/better_translate/variable_extractor.rb`
+
+```ruby
+# frozen_string_literal: true
+
+module BetterTranslate
+  # Extracts and preserves interpolation variables during translation
+  #
+  # Supports multiple variable formats:
+  # - Rails I18n: %{name}, %{count}
+  # - I18n.js: {{user}}, {{email}}
+  # - ES6 templates: ${var}
+  # - Simple braces: {name}
+  #
+  # @example
+  #   extractor = VariableExtractor.new("Hello %{name}, you have {{count}} messages")
+  #   safe_text = extractor.extract
+  #   #=> "Hello __VAR_0__, you have __VAR_1__ messages"
+  #
+  #   translated = translate(safe_text)  # "Ciao __VAR_0__, hai __VAR_1__ messaggi"
+  #   final = extractor.restore(translated)
+  #   #=> "Ciao %{name}, hai {{count}} messaggi"
+  #
+  class VariableExtractor
+    # Variable patterns to detect and preserve
+    VARIABLE_PATTERNS = {
+      rails: /%\{[^}]+\}/,              # %{name}, %{count}
+      i18n_js: /\{\{[^}]+\}\}/,         # {{user}}, {{email}}
+      es6: /\$\{[^}]+\}/,               # ${var}
+      simple: /\{[a-zA-Z_][a-zA-Z0-9_]*\}/  # {name} but not {1,2,3}
+    }.freeze
+
+    # Combined pattern to match any variable format
+    COMBINED_PATTERN = Regexp.union(*VARIABLE_PATTERNS.values)
+
+    # Placeholder prefix
+    PLACEHOLDER_PREFIX = "__VAR_"
+    PLACEHOLDER_SUFFIX = "__"
+
+    # @return [String] Original text with variables
+    attr_reader :original_text
+
+    # @return [Array<String>] Extracted variables in order
+    attr_reader :variables
+
+    # @return [Hash<String, String>] Mapping of placeholders to original variables
+    attr_reader :placeholder_map
+
+    # Initialize extractor with text
+    #
+    # @param text [String] Text containing variables
+    def initialize(text)
+      @original_text = text
+      @variables = []
+      @placeholder_map = {}
+      @reverse_map = {}
+    end
+
+    # Extract variables and replace with placeholders
+    #
+    # @return [String] Text with variables replaced by placeholders
+    def extract
+      return original_text if original_text.nil? || original_text.empty?
+
+      result = original_text.dup
+      index = 0
+
+      # Find and replace all variables
+      result.gsub!(COMBINED_PATTERN) do |match|
+        placeholder = "#{PLACEHOLDER_PREFIX}#{index}#{PLACEHOLDER_SUFFIX}"
+        @variables << match
+        @placeholder_map[placeholder] = match
+        @reverse_map[match] = placeholder
+        index += 1
+        placeholder
+      end
+
+      result
+    end
+
+    # Restore variables from placeholders in translated text
+    #
+    # @param translated_text [String] Translated text with placeholders
+    # @param strict [Boolean] If true, raises error if variables are missing
+    # @return [String] Translated text with original variables restored
+    # @raise [ValidationError] if strict mode and variables are missing
+    def restore(translated_text, strict: true)
+      return translated_text if translated_text.nil? || translated_text.empty?
+      return translated_text if @placeholder_map.empty?
+
+      result = translated_text.dup
+
+      # Restore all placeholders
+      @placeholder_map.each do |placeholder, original_var|
+        result.gsub!(placeholder, original_var)
+      end
+
+      # Validate all variables are present
+      if strict
+        validate_variables!(result)
+      end
+
+      result
+    end
+
+    # Check if text contains variables
+    #
+    # @return [Boolean] true if variables are present
+    def variables?
+      !@variables.empty?
+    end
+
+    # Get count of variables
+    #
+    # @return [Integer] Number of variables
+    def variable_count
+      @variables.size
+    end
+
+    # Validate that all original variables are present in text
+    #
+    # @param text [String] Text to validate
+    # @raise [ValidationError] if variables are missing or modified
+    # @return [true] if all variables are present
+    def validate_variables!(text)
+      missing = []
+      extra = []
+
+      # Check for missing variables
+      @variables.each do |var|
+        unless text.include?(var)
+          missing << var
+        end
+      end
+
+      # Check for extra/unknown variables (potential corruption)
+      text.scan(COMBINED_PATTERN).each do |var|
+        unless @variables.include?(var)
+          extra << var
+        end
+      end
+
+      if missing.any? || extra.any?
+        error_msg = []
+        error_msg << "Missing variables: #{missing.join(', ')}" if missing.any?
+        error_msg << "Unexpected variables: #{extra.join(', ')}" if extra.any?
+
+        raise ValidationError.new(
+          "Variable validation failed: #{error_msg.join('; ')}",
+          context: {
+            original_variables: @variables,
+            missing: missing,
+            extra: extra,
+            text: text
+          }
+        )
+      end
+
+      true
+    end
+
+    # Extract variables from text without creating instance
+    #
+    # @param text [String] Text to analyze
+    # @return [Array<String>] List of variables found
+    def self.find_variables(text)
+      return [] if text.nil? || text.empty?
+      text.scan(COMBINED_PATTERN)
+    end
+
+    # Check if text contains variables
+    #
+    # @param text [String] Text to check
+    # @return [Boolean] true if variables are present
+    def self.contains_variables?(text)
+      return false if text.nil? || text.empty?
+      text.match?(COMBINED_PATTERN)
+    end
+  end
+end
+```
+
+---
+
+## 3.5.2 Integration with BaseHttpProvider
+
+Update `lib/better_translate/providers/base_http_provider.rb` to use VariableExtractor:
+
+```ruby
+# Add to BaseHttpProvider class
+
+# Translate text with variable preservation
+#
+# @param text [String] Text to translate
+# @param target_lang_code [String] Target language code
+# @param target_lang_name [String] Target language name
+# @param preserve_variables [Boolean] Whether to preserve variables (default: true)
+# @return [String] Translated text
+# @api private
+def translate_with_variable_preservation(text, target_lang_code, target_lang_name, preserve_variables: true)
+  # If no variables or preservation disabled, translate directly
+  unless preserve_variables && VariableExtractor.contains_variables?(text)
+    return translate_text_raw(text, target_lang_code, target_lang_name)
+  end
+
+  # Extract variables
+  extractor = VariableExtractor.new(text)
+  safe_text = extractor.extract
+
+  if config.verbose
+    puts "[VariablePreservation] Found #{extractor.variable_count} variables: #{extractor.variables.join(', ')}"
+  end
+
+  # Translate text with placeholders
+  translated = translate_text_raw(safe_text, target_lang_code, target_lang_name)
+
+  # Restore variables
+  begin
+    final_text = extractor.restore(translated, strict: true)
+
+    if config.verbose
+      puts "[VariablePreservation] ✓ All variables preserved"
+    end
+
+    final_text
+  rescue ValidationError => e
+    # Log warning but return translation anyway (non-strict mode fallback)
+    warn "[VariablePreservation] WARNING: #{e.message}"
+
+    # Attempt non-strict restore
+    extractor.restore(translated, strict: false)
+  end
+end
+
+# Raw translation without variable preservation (used internally)
+#
+# @param text [String] Text to translate
+# @param target_lang_code [String] Target language code
+# @param target_lang_name [String] Target language name
+# @return [String] Translated text
+# @api private
+def translate_text_raw(text, target_lang_code, target_lang_name)
+  # Original translate_text implementation
+  # Subclasses implement this method
+  raise NotImplementedError, "#{self.class} must implement #translate_text_raw"
+end
+```
+
+**Note:** Each provider (ChatGPT, Gemini, Anthropic) should rename their `translate_text` to `translate_text_raw` and use `translate_with_variable_preservation` as the public interface.
+
+---
+
+## 3.5.3 Test: `spec/better_translate/variable_extractor_spec.rb`
+
+```ruby
+# frozen_string_literal: true
+
+RSpec.describe BetterTranslate::VariableExtractor do
+  describe "#extract" do
+    it "extracts Rails I18n variables" do
+      extractor = described_class.new("Hello %{name}, you have %{count} messages")
+      safe_text = extractor.extract
+
+      expect(safe_text).to eq("Hello __VAR_0__, you have __VAR_1__ messages")
+      expect(extractor.variables).to eq(["%{name}", "%{count}"])
+      expect(extractor.variable_count).to eq(2)
+    end
+
+    it "extracts I18n.js variables" do
+      extractor = described_class.new("Welcome {{user}}!")
+      safe_text = extractor.extract
+
+      expect(safe_text).to eq("Welcome __VAR_0__!")
+      expect(extractor.variables).to eq(["{{user}}"])
+    end
+
+    it "extracts ES6 template variables" do
+      extractor = described_class.new("Total: ${amount}")
+      safe_text = extractor.extract
+
+      expect(safe_text).to eq("Total: __VAR_0__")
+      expect(extractor.variables).to eq(["${amount}"])
+    end
+
+    it "extracts simple brace variables" do
+      extractor = described_class.new("Hello {name}")
+      safe_text = extractor.extract
+
+      expect(safe_text).to eq("Hello __VAR_0__")
+      expect(extractor.variables).to eq(["{name}"])
+    end
+
+    it "extracts mixed variable formats" do
+      text = "Hi %{name}, you have {{count}} items (${total})"
+      extractor = described_class.new(text)
+      safe_text = extractor.extract
+
+      expect(extractor.variables).to eq(["%{name}", "{{count}}", "${total}"])
+      expect(safe_text).to eq("Hi __VAR_0__, you have __VAR_1__ items (__VAR_2__)")
+    end
+
+    it "handles text without variables" do
+      extractor = described_class.new("Hello world")
+      safe_text = extractor.extract
+
+      expect(safe_text).to eq("Hello world")
+      expect(extractor.variables).to be_empty
+    end
+
+    it "handles empty text" do
+      extractor = described_class.new("")
+      expect(extractor.extract).to eq("")
+      expect(extractor.variables).to be_empty
+    end
+  end
+
+  describe "#restore" do
+    it "restores variables to original format" do
+      extractor = described_class.new("Hello %{name}")
+      safe_text = extractor.extract
+      translated = "Ciao __VAR_0__"
+
+      restored = extractor.restore(translated)
+      expect(restored).to eq("Ciao %{name}")
+    end
+
+    it "restores multiple variables in correct positions" do
+      extractor = described_class.new("You have %{count} messages from {{user}}")
+      safe_text = extractor.extract
+      # Simulated translation that reorders placeholders
+      translated = "{{user}} ti ha inviato %{count} messaggi"
+      # Replace placeholders
+      translated = "__VAR_1__ ti ha inviato __VAR_0__ messaggi"
+
+      restored = extractor.restore(translated)
+      expect(restored).to eq("{{user}} ti ha inviato %{count} messaggi")
+    end
+
+    it "raises error in strict mode if variable is missing" do
+      extractor = described_class.new("Hello %{name}")
+      extractor.extract
+      translated = "Ciao"  # Missing variable
+
+      expect {
+        extractor.restore(translated, strict: true)
+      }.to raise_error(BetterTranslate::ValidationError, /Missing variables: %\{name\}/)
+    end
+
+    it "does not raise in non-strict mode if variable is missing" do
+      extractor = described_class.new("Hello %{name}")
+      extractor.extract
+      translated = "Ciao"
+
+      expect {
+        extractor.restore(translated, strict: false)
+      }.not_to raise_error
+
+      expect(extractor.restore(translated, strict: false)).to eq("Ciao")
+    end
+
+    it "detects unexpected variables" do
+      extractor = described_class.new("Hello world")
+      extractor.extract
+      translated = "Ciao %{name}"  # Unexpected variable added
+
+      expect {
+        extractor.restore(translated, strict: true)
+      }.to raise_error(BetterTranslate::ValidationError, /Unexpected variables: %\{name\}/)
+    end
+  end
+
+  describe ".find_variables" do
+    it "finds all variables in text" do
+      text = "Hi %{name}, {{count}} items"
+      variables = described_class.find_variables(text)
+
+      expect(variables).to contain_exactly("%{name}", "{{count}}")
+    end
+
+    it "returns empty array for text without variables" do
+      expect(described_class.find_variables("Hello world")).to eq([])
+    end
+  end
+
+  describe ".contains_variables?" do
+    it "returns true if variables are present" do
+      expect(described_class.contains_variables?("Hello %{name}")).to be true
+      expect(described_class.contains_variables?("Hi {{user}}")).to be true
+    end
+
+    it "returns false if no variables" do
+      expect(described_class.contains_variables?("Hello world")).to be false
+    end
+  end
+
+  describe "#validate_variables!" do
+    it "passes validation if all variables are present" do
+      extractor = described_class.new("Hello %{name}")
+      extractor.extract
+
+      expect {
+        extractor.validate_variables!("Ciao %{name}")
+      }.not_to raise_error
+    end
+
+    it "fails validation if variables are missing" do
+      extractor = described_class.new("Hello %{name} and %{title}")
+      extractor.extract
+
+      expect {
+        extractor.validate_variables!("Ciao %{name}")
+      }.to raise_error(BetterTranslate::ValidationError)
+    end
+  end
+end
+```
+
+---
+
+## 3.5.4 Configuration Option
+
+Add to `lib/better_translate/configuration.rb`:
+
+```ruby
+# @return [Boolean] Preserve interpolation variables during translation (default: true)
+attr_accessor :preserve_variables
+
+# In initialize method:
+@preserve_variables = true
+```
+
+---
+
+## Usage Examples
+
+### Example 1: Automatic Variable Preservation
+
+```ruby
+# Input YAML
+en:
+  welcome: "Welcome %{name}!"
+  messages: "You have {{count}} new messages"
+
+# After translation (Italian)
+it:
+  welcome: "Benvenuto %{name}!"
+  messages: "Hai {{count}} nuovi messaggi"
+```
+
+### Example 2: Direct Translation with Variables
+
+```ruby
+text = "Hello %{user}, your balance is ${amount}"
+result = BetterTranslate.translate_text(text, from: "en", to: "it")
+#=> "Ciao %{user}, il tuo saldo è ${amount}"
+
+# Variables are automatically preserved!
+```
+
+### Example 3: Error Detection
+
+```ruby
+# If translation removes a variable
+text = "Total: %{amount}"
+# Translation accidentally returns: "Totale:" (missing %{amount})
+#=> ValidationError raised with details about missing variable
+```
+
+---
+
+## Benefits
+
+1. **Bug Prevention**: Eliminates broken interpolations in translations
+2. **Automatic**: Works transparently in all translation flows
+3. **Multiple Formats**: Supports Rails, I18n.js, ES6, and simple braces
+4. **Validation**: Detects missing or corrupted variables
+5. **Configurable**: Can be disabled if needed
+
+---
+
+## Implementation Checklist
+
+- [ ] Create `lib/better_translate/variable_extractor.rb`
+- [ ] Update `BaseHttpProvider` to use variable preservation
+- [ ] Update all providers (ChatGPT, Gemini, Anthropic) to use `translate_text_raw`
+- [ ] Add `preserve_variables` configuration option
+- [ ] Create comprehensive test suite
+- [ ] Update YARD documentation
+- [ ] Add usage examples to README
+
+---
+
+[← Previous: 03-Core Components](./03-core_components.md) | [Back to Index](../../IMPLEMENTATION_PLAN.md) | [Next: 04-Provider Architecture →](./04-provider_architecture.md)