familia 2.0.0.pre5 → 2.0.0.pre6

data/docs/wiki/Transient-Fields-Guide.md

@@ -0,0 +1,280 @@
+# Transient Fields Guide
+
+## Overview
+
+Transient fields provide secure handling of sensitive runtime data that should never be persisted to Redis/Valkey. Unlike encrypted fields, transient fields exist only in memory and are automatically wrapped in `RedactedString` for security.
+
+## When to Use Transient Fields
+
+Use transient fields for:
+- API keys and tokens that change frequently
+- Temporary passwords or passphrases
+- Session-specific secrets
+- Any sensitive data that should never touch persistent storage
+- Debug or development secrets that need secure handling
+
+## Basic Usage
+
+### Define Transient Fields
+
+```ruby
+class ApiClient < Familia::Horreum
+  feature :transient_fields
+
+  field :endpoint          # Regular persistent field
+  transient_field :token   # Transient field (not persisted)
+  transient_field :secret, as: :api_secret  # Custom accessor name
+end
+```
+
+### Working with Transient Fields
+
+```ruby
+client = ApiClient.new(
+  endpoint: 'https://api.example.com',
+  token: ENV['API_TOKEN'],
+  secret: ENV['API_SECRET']
+)
+
+# Regular field persists
+client.save
+client.endpoint  # => "https://api.example.com"
+
+# Transient fields are RedactedString instances
+puts client.token  # => "[REDACTED]"
+
+# Access the actual value safely
+client.token.expose do |token|
+  response = HTTP.post(client.endpoint,
+    headers: { 'Authorization' => "Bearer #{token}" }
+  )
+  # Token value is only available within this block
+end
+
+# Explicit cleanup when done
+client.token.clear!
+```
+
+## RedactedString Security
+
+### Automatic Wrapping
+
+All transient field values are automatically wrapped in `RedactedString`:
+
+```ruby
+client = ApiClient.new(token: 'secret123')
+client.token.class  # => RedactedString
+```
+
+### Safe Access Pattern
+
+```ruby
+# ✅ Recommended: Use .expose block
+client.token.expose do |token|
+  # Use token directly without creating copies
+  HTTP.auth("Bearer #{token}")  # Safe
+end
+
+# ✅ Direct access (use carefully)
+raw_token = client.token.value
+# Remember to clear original source if needed
+
+# ❌ Avoid: These create uncontrolled copies
+token_copy = client.token.value.dup      # Creates copy in memory
+interpolated = "Bearer #{client.token}"  # Creates copy via to_s
+```
+
+### Memory Management
+
+```ruby
+# Clear individual fields
+client.token.clear!
+
+# Check if cleared
+client.token.cleared?  # => true
+
+# Accessing cleared values raises error
+client.token.value  # => SecurityError: Value already cleared
+```
+
+## Advanced Features
+
+### Custom Accessor Names
+
+```ruby
+class Service < Familia::Horreum
+  transient_field :api_key, as: :secret_key
+end
+
+service = Service.new(api_key: 'secret123')
+service.secret_key.expose { |key| use_api_key(key) }
+```
+
+### Integration with Encrypted Fields
+
+```ruby
+class SecureService < Familia::Horreum
+  feature :transient_fields
+
+  encrypted_field :long_term_secret    # Persisted, encrypted
+  transient_field :session_token       # Runtime only, not persisted
+  field :public_endpoint               # Normal field
+end
+
+service = SecureService.new(
+  long_term_secret: 'stored encrypted in Redis',
+  session_token: 'temporary runtime secret',
+  public_endpoint: 'https://api.example.com'
+)
+
+service.save
+# Only long_term_secret and public_endpoint are saved to Redis
+# session_token exists only in memory
+```
+
+## RedactedString API Reference
+
+### Core Methods
+
+```ruby
+# Create (usually automatic via transient_field)
+secret = RedactedString.new('sensitive_value')
+
+# Safe access
+secret.expose { |value| use_value(value) }
+
+# Direct access (use with caution)
+value = secret.value
+
+# Cleanup
+secret.clear!
+
+# Status
+secret.cleared?  # => true/false
+```
+
+### Security Methods
+
+```ruby
+# Logging/debugging protection
+puts secret.to_s     # => "[REDACTED]"
+puts secret.inspect  # => "[REDACTED]"
+
+# Equality (object identity only)
+secret1 == secret2   # => false (unless same object)
+
+# Hash (constant for all instances)
+secret.hash  # => Same for all RedactedString instances
+```
+
+## Security Considerations
+
+### Ruby Memory Limitations
+
+**⚠️ Important**: Ruby provides no memory safety guarantees:
+
+- **No secure wiping**: `.clear!` is best-effort only
+- **GC copying**: Garbage collector may duplicate secrets
+- **String operations**: Every manipulation creates copies
+- **Memory persistence**: Secrets may remain in memory indefinitely
+
+### Best Practices
+
+```ruby
+# ✅ Wrap immediately after input
+password = RedactedString.new(params[:password])
+params[:password] = nil  # Clear original reference
+
+# ✅ Use .expose for short operations
+token.expose { |t| api_call(t) }
+
+# ✅ Clear explicitly when done
+token.clear!
+
+# ✅ Avoid string operations that create copies
+token.expose { |t| "Bearer #{t}" }  # Creates copy
+# Better: Pass token directly to methods that need it
+
+# ❌ Don't pass RedactedString to logging
+logger.info "Token: #{token}"  # Still logs "[REDACTED]" but safer to avoid
+
+# ❌ Don't store in instance variables outside field system
+@raw_token = token.value  # Creates uncontrolled copy
+```
+
+### Production Recommendations
+
+For highly sensitive applications, consider:
+- External secrets management (HashiCorp Vault, AWS Secrets Manager)
+- Hardware Security Modules (HSMs)
+- Languages with secure memory handling
+- Encrypted swap and memory protection at OS level
+
+## Integration Examples
+
+### Rails Controller
+
+```ruby
+class ApiController < ApplicationController
+  def authenticate
+    service = ApiService.new(
+      endpoint: params[:endpoint],
+      token: params[:token]  # Auto-wrapped in RedactedString
+    )
+
+    result = service.token.expose do |token|
+      # Token only accessible within this block
+      ExternalAPI.authenticate(token)
+    end
+
+    # Clear token when request is done
+    service.token.clear!
+
+    render json: { status: result }
+  end
+end
+```
+
+### Background Job
+
+```ruby
+class ApiSyncJob
+  def perform(user_id, token)
+    user = User.find(user_id)
+
+    # Wrap external token securely
+    secure_token = RedactedString.new(token)
+    token.clear if token.respond_to?(:clear)  # Clear original
+
+    client = ApiClient.new(token: secure_token)
+
+    begin
+      sync_data(client)
+    ensure
+      client.token.clear!  # Always cleanup
+    end
+  end
+
+  private
+
+  def sync_data(client)
+    client.token.expose do |token|
+      # Use token for API calls
+      fetch_and_process_data(token)
+    end
+  end
+end
+```
+
+## Comparison with Encrypted Fields
+
+| Feature | Encrypted Fields | Transient Fields |
+|---------|------------------|------------------|
+| **Persistence** | Saved to Redis/Valkey | Memory only |
+| **Encryption** | AES/XChaCha20 | None (not stored) |
+| **Use Case** | Long-term secrets | Runtime secrets |
+| **Access** | Automatic decrypt | RedactedString wrapper |
+| **Performance** | Crypto overhead | No crypto operations |
+| **Lifecycle** | Survives restarts | Cleared on restart |
+
+Choose encrypted fields for data that must persist across sessions. Choose transient fields for sensitive runtime data that should never be stored.

data/lib/familia/base.rb

@@ -35,7 +35,7 @@ module Familia
 
       def add_feature(klass, feature_name, depends_on: [])
         @features_available ||= {}
-        Familia.ld "[#{self}] Adding feature #{klass} as #{feature_name.inspect}"
+        Familia.trace :ADD_FEATURE, klass, feature_name, caller(1..1) if Familia.debug?
 
         # Create field definition object
         feature_def = FeatureDefinition.new(

data/lib/familia/data_type/types/counter.rb

@@ -0,0 +1,38 @@
+# lib/familia/data_type/types/counter.rb
+
+module Familia
+  class Counter < String
+    def initialize(*args)
+      super
+      @opts[:default] ||= 0
+    end
+
+    # Enhanced counter semantics
+    def reset(val = 0)
+      set(val).to_s.eql?('OK')
+    end
+
+    def increment_if_less_than(threshold, amount = 1)
+      current = to_i
+      return false if current >= threshold
+
+      incrementby(amount)
+      true
+    end
+
+    def atomic_increment_and_get(amount = 1)
+      incrementby(amount)
+    end
+
+    # Override to ensure integer serialization
+    def value=(val)
+      super(val.to_i)
+    end
+
+    def value
+      super.to_i
+    end
+  end
+end
+
+Familia::DataType.register Familia::Counter, :counter

data/lib/familia/data_type/types/hashkey.rb

@@ -61,6 +61,24 @@ module Familia
     end
     alias all hgetall
 
+    # Sets field in the hash stored at key to value, only if field does not yet exist.
+    # If field already exists, this operation has no effect.
+    # @param field [String] The field name
+    # @param val [Object] The value to set
+    # @return [Integer] 1 if field is a new field and value was set, 0 if field already exists
+    def hsetnx(field, val)
+      ret = dbclient.hsetnx dbkey, field.to_s, serialize_value(val)
+      update_expiration if ret == 1
+      ret
+    rescue TypeError => e
+      Familia.le "[hsetnx] #{e.message}"
+      Familia.ld "[hsetnx] #{dbkey} #{field}=#{val}" if Familia.debug
+      echo :hsetnx, caller(1..1).first if Familia.debug # logs via echo to the db and back
+      klass = val.class
+      msg = "Cannot store #{field} => #{val.inspect} (#{klass}) in #{dbkey}"
+      raise e.class, msg
+    end
+
     def key?(field)
       dbclient.hexists dbkey, field.to_s
     end

data/lib/familia/data_type/types/lock.rb

@@ -0,0 +1,43 @@
+# lib/familia/data_type/types/lock.rb
+
+module Familia
+  class Lock < String
+    def initialize(*args)
+      super
+      @opts[:default] = nil
+    end
+
+    # Acquire a lock with optional TTL
+    # @param token [String] Unique token to identify lock holder (auto-generated if nil)
+    # @param ttl [Integer, nil] Time-to-live in seconds. nil = no expiration, <=0 rejected
+    # @return [String, false] Returns token if acquired successfully, false otherwise
+    def acquire(token = SecureRandom.uuid, ttl: 10)
+      success = setnx(token)
+      # Handle both integer (1/0) and boolean (true/false) return values
+      return false unless success == 1 || success == true
+      return del && false if ttl&.<=(0)
+      return del && false if ttl&.positive? && !expire(ttl)
+      token
+    end
+
+    def release(token)
+      # Lua script to atomically check token and delete
+      script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end"
+      dbclient.eval(script, [dbkey], [token]) == 1
+    end
+
+    def locked?
+      !value.nil?
+    end
+
+    def held_by?(token)
+      value == token
+    end
+
+    def force_unlock!
+      del
+    end
+  end
+end
+
+Familia::DataType.register Familia::Lock, :lock

data/lib/familia/data_type/types/string.rb

@@ -108,12 +108,19 @@ module Familia
       ret
     end
 
+    def del
+      ret = dbclient.del dbkey
+      ret.positive?
+    end
+
     def nil?
       value.nil?
     end
 
     Familia::DataType.register self, :string
-    Familia::DataType.register self, :counter
-    Familia::DataType.register self, :lock
   end
 end
+
+# Both subclass String
+require_relative 'lock'
+require_relative 'counter'

data/lib/familia/data_type.rb

@@ -32,7 +32,7 @@ module Familia
       # +methname+ is the term used for the class and instance methods
       # that are created for the given +klass+ (e.g. set, list, etc)
       def register(klass, methname)
-        Familia.ld "[#{self}] Registering #{klass} as #{methname.inspect}"
+        Familia.trace :REGISTER, nil, "[#{self}] Registering #{klass} as #{methname.inspect}", caller(1..1) if Familia.debug?
 
         @registered_types[methname] = klass
       end
@@ -115,7 +115,7 @@ module Familia
         # this point. This would result in a Familia::Problem being raised. So
         # to be on the safe-side here until we have a better understanding of
         # the issue, we'll just log the class name for each key-value pair.
-        Familia.ld " [setting] #{k} #{v.class}"
+        Familia.trace :SETTING, nil, " [setting] #{k} #{v.class}", caller(1..1) if Familia.debug?
         send(:"#{k}=", v) if respond_to? :"#{k}="
       end
 

data/lib/familia/encryption/encrypted_data.rb

@@ -0,0 +1,137 @@
+# lib/familia/encryption/encrypted_data.rb
+
+module Familia
+  module Encryption
+    EncryptedData = Data.define(:algorithm, :nonce, :ciphertext, :auth_tag, :key_version) do
+      # Class methods for parsing and validation
+      def self.valid?(json_string)
+        return true if json_string.nil?  # Allow nil values
+        return false unless json_string.kind_of?(::String)
+
+        begin
+          parsed = JSON.parse(json_string, symbolize_names: true)
+          return false unless parsed.is_a?(Hash)
+
+          # Check for required fields
+          required_fields = [:algorithm, :nonce, :ciphertext, :auth_tag, :key_version]
+          result = required_fields.all? { |field| parsed.key?(field) }
+          Familia.ld "[valid?] result: #{result}, parsed: #{parsed}, required: #{required_fields}"
+          result
+        rescue JSON::ParserError => e
+          Familia.ld "[valid?] JSON error: #{e.message}"
+          false
+        end
+      end
+
+      def self.validate!(json_string)
+        return nil if json_string.nil?
+
+        unless json_string.kind_of?(::String)
+          raise EncryptionError, "Expected JSON string, got #{json_string.class}"
+        end
+
+        begin
+          parsed = JSON.parse(json_string, symbolize_names: true)
+        rescue JSON::ParserError => e
+          raise EncryptionError, "Invalid JSON structure: #{e.message}"
+        end
+
+        unless parsed.is_a?(Hash)
+          raise EncryptionError, "Expected JSON object, got #{parsed.class}"
+        end
+
+        required_fields = [:algorithm, :nonce, :ciphertext, :auth_tag, :key_version]
+        missing_fields = required_fields.reject { |field| parsed.key?(field) }
+
+        unless missing_fields.empty?
+          raise EncryptionError, "Missing required fields: #{missing_fields.join(', ')}"
+        end
+
+        new(**parsed)
+      end
+
+      def self.from_json(json_string)
+        validate!(json_string)
+      end
+
+      # Instance methods for decryptability validation
+      def decryptable?
+        return false unless algorithm && nonce && ciphertext && auth_tag && key_version
+
+        # Ensure Registry is set up before checking algorithms
+        Registry.setup! if Registry.providers.empty?
+
+        # Check if algorithm is supported
+        return false unless Registry.providers.key?(algorithm)
+
+        # Validate Base64 encoding of binary fields
+        begin
+          Base64.strict_decode64(nonce)
+          Base64.strict_decode64(ciphertext)
+          Base64.strict_decode64(auth_tag)
+        rescue ArgumentError
+          return false
+        end
+
+        true
+      end
+
+      def validate_decryptable!
+        unless algorithm
+          raise EncryptionError, "Missing algorithm field"
+        end
+
+        # Ensure Registry is set up before checking algorithms
+        Registry.setup! if Registry.providers.empty?
+
+        unless Registry.providers.key?(algorithm)
+          raise EncryptionError, "Unsupported algorithm: #{algorithm}"
+        end
+
+        unless nonce && ciphertext && auth_tag && key_version
+          missing = []
+          missing << 'nonce' unless nonce
+          missing << 'ciphertext' unless ciphertext
+          missing << 'auth_tag' unless auth_tag
+          missing << 'key_version' unless key_version
+          raise EncryptionError, "Missing required fields: #{missing.join(', ')}"
+        end
+
+        # Get the provider for size validation
+        provider = Registry.providers[algorithm]
+
+        # Validate Base64 encoding and sizes
+        begin
+          decoded_nonce = Base64.strict_decode64(nonce)
+          if decoded_nonce.bytesize != provider.nonce_size
+            raise EncryptionError, "Invalid nonce size: expected #{provider.nonce_size}, got #{decoded_nonce.bytesize}"
+          end
+        rescue ArgumentError
+          raise EncryptionError, "Invalid Base64 encoding in nonce field"
+        end
+
+        begin
+          Base64.strict_decode64(ciphertext)  # ciphertext can be variable size
+        rescue ArgumentError
+          raise EncryptionError, "Invalid Base64 encoding in ciphertext field"
+        end
+
+        begin
+          decoded_auth_tag = Base64.strict_decode64(auth_tag)
+          if decoded_auth_tag.bytesize != provider.auth_tag_size
+            raise EncryptionError, "Invalid auth_tag size: expected #{provider.auth_tag_size}, got #{decoded_auth_tag.bytesize}"
+          end
+        rescue ArgumentError
+          raise EncryptionError, "Invalid Base64 encoding in auth_tag field"
+        end
+
+        # Validate that the key version exists
+        unless Familia.config.encryption_keys&.key?(key_version.to_sym)
+          raise EncryptionError, "No key for version: #{key_version}"
+        end
+
+        self
+      end
+    end
+  end
+end

data/lib/familia/encryption/manager.rb

@@ -33,23 +33,28 @@ module Familia
       def decrypt(encrypted_json, context:, additional_data: nil)
         return nil if encrypted_json.nil? || encrypted_json.empty?
 
+        # Increment counter immediately to track all decryption attempts, even failed ones
+        Familia::Encryption.derivation_count.increment
+
         begin
           data = Familia::Encryption::EncryptedData.new(**JSON.parse(encrypted_json, symbolize_names: true))
 
           # Validate algorithm support
           provider = Registry.get(data.algorithm)
-          key = derive_key(context, version: data.key_version, provider: provider)
+          key = derive_key_without_increment(context, version: data.key_version, provider: provider)
 
           # Safely decode and validate sizes
           nonce = decode_and_validate(data.nonce, provider.nonce_size, 'nonce')
-          ciphertext = Base64.strict_decode64(data.ciphertext)
+          ciphertext = decode_and_validate_ciphertext(data.ciphertext)
           auth_tag = decode_and_validate(data.auth_tag, provider.auth_tag_size, 'auth_tag')
 
           provider.decrypt(ciphertext, key, nonce, auth_tag, additional_data)
         rescue EncryptionError
           raise
-        rescue StandardError
-          raise EncryptionError, 'Decryption failed'
+        rescue JSON::ParserError => e
+          raise EncryptionError, "Invalid JSON structure: #{e.message}"
+        rescue StandardError => e
+          raise EncryptionError, "Decryption failed: #{e.message}"
         end
       ensure
         Familia::Encryption.secure_wipe(key) if key
@@ -61,12 +66,24 @@ module Familia
         decoded = Base64.strict_decode64(encoded)
         raise EncryptionError, 'Invalid encrypted data' unless decoded.bytesize == expected_size
         decoded
+      rescue ArgumentError => e
+        raise EncryptionError, "Invalid Base64 encoding in #{component} field"
+      end
+
+      def decode_and_validate_ciphertext(encoded)
+        Base64.strict_decode64(encoded)
+      rescue ArgumentError => e
+        raise EncryptionError, "Invalid Base64 encoding in ciphertext field"
       end
 
       def derive_key(context, version: nil, provider: nil)
         # Increment counter to prove no caching is happening
         Familia::Encryption.derivation_count.increment
 
+        derive_key_without_increment(context, version: version, provider: provider)
+      end
+
+      def derive_key_without_increment(context, version: nil, provider: nil)
         # Use provided provider or fall back to instance provider
         provider ||= @provider
 

data/lib/familia/encryption/providers/aes_gcm_provider.rb

@@ -87,6 +87,26 @@ module Familia
           key&.clear
         end
 
+        def self.nonce_size
+          NONCE_SIZE
+        end
+
+        def self.auth_tag_size
+          AUTH_TAG_SIZE
+        end
+
+        def nonce_size
+          NONCE_SIZE
+        end
+
+        def auth_tag_size
+          AUTH_TAG_SIZE
+        end
+
+        def algorithm
+          ALGORITHM
+        end
+
         private
 
         def create_cipher(mode)

data/lib/familia/encryption/providers/xchacha20_poly1305_provider.rb

@@ -106,6 +106,26 @@ module Familia
           key&.clear
         end
 
+        def self.nonce_size
+          NONCE_SIZE
+        end
+
+        def self.auth_tag_size
+          AUTH_TAG_SIZE
+        end
+
+        def nonce_size
+          NONCE_SIZE
+        end
+
+        def auth_tag_size
+          AUTH_TAG_SIZE
+        end
+
+        def algorithm
+          ALGORITHM
+        end
+
         private
 
         def validate_key_length!(key)

data/lib/familia/encryption.rb

@@ -10,12 +10,12 @@ require_relative 'encryption/providers/xchacha20_poly1305_provider'
 require_relative 'encryption/providers/aes_gcm_provider'
 require_relative 'encryption/registry'
 require_relative 'encryption/manager'
+require_relative 'encryption/encrypted_data'
 
 module Familia
   class EncryptionError < StandardError; end
 
   module Encryption
-    EncryptedData = Data.define(:algorithm, :nonce, :ciphertext, :auth_tag, :key_version)
 
     # Smart facade with provider selection and field-specific encryption
     #