familia 2.0.0.pre14 → 2.0.0.pre16

data/docs/guides/feature-relationships.md

@@ -0,0 +1,200 @@
+# Relationships Feature Guide
+
+The Relationships feature transforms how you manage object associations in Familia applications. Instead of manually maintaining foreign keys and indexes, relationships provide automatic bidirectional links, efficient queries, and Ruby-like collection syntax that makes working with related objects feel natural and intuitive.
+
+This guide provides a breadth-first introduction to the four core relationship capabilities: **participation**, **indexing**, **querying**, and **cascading operations**.
+
+> [!TIP]
+> Enable relationships with `feature :relationships`, define associations with `participates_in`, and use clean Ruby syntax like `customer.domains << domain` to manage relationships.
+
+## What Are Relationships?
+
+Relationships automate object associations in Familia, eliminating manual foreign key management:
+
+```ruby
+# Without relationships - manual and error-prone
+customer.domain_ids.add(domain.identifier)
+domain.customer_id = customer.identifier
+
+# With relationships - automatic and clean
+customer.domains << domain  # Updates both sides automatically
+```
+
+**Key Benefits:**
+- **Automatic bidirectional updates** - no manual synchronization
+- **Ruby-like syntax** - familiar `<<` and collection operations
+- **O(1) lookups** - efficient Valkey/Redis-backed indexing
+- **Lifecycle management** - automatic cleanup and maintenance
+
+## Core Relationship Capabilities
+
+### 1. Participation - Bidirectional Object Links
+
+Connect objects with automatic synchronization:
+
+```ruby
+class User < Familia::Horreum
+  feature :relationships
+  set :teams  # Collection holder
+end
+
+class Team < Familia::Horreum
+  feature :relationships
+  participates_in User, :teams  # Declares participation
+end
+
+# Usage - automatic bidirectional updates
+user.teams << team
+team.in_user_teams?(user)  # => true
+```
+
+**Many-to-Many Example:**
+```ruby
+class Project < Familia::Horreum
+  set :contributors, :reviewers
+end
+
+class Developer < Familia::Horreum
+  participates_in Project, :contributors
+  participates_in Project, :reviewers
+end
+
+project.contributors << alice  # Alice contributes
+project.reviewers << bob       # Bob reviews
+```
+
+### 2. Indexing - Automatic Object Tracking
+
+Enable O(1) lookups with automatic index management:
+
+```ruby
+class User < Familia::Horreum
+  feature :relationships
+  field :email, :created_at
+
+  # Global unique lookups
+  class_indexed_by :email, :email_lookup
+
+  # Scored tracking collections
+  class_participates_in :all_users, score: :created_at
+end
+
+# Automatic on save/destroy
+User.find_by_email("alice@example.com")    # O(1) lookup
+User.all_users.range(0, 9)                 # Most recent 10 users
+```
+
+**Relationship-Scoped Indexing:**
+```ruby
+class Domain < Familia::Horreum
+  participates_in Customer, :domains
+  indexed_by :name, :domain_index, target: Customer  # Unique per customer
+end
+
+customer.find_by_name("example.com")  # Find domain within this customer
+```
+
+### 3. Querying - Ruby-like Collection Operations
+
+Work with relationships like standard Ruby collections:
+
+```ruby
+# Standard collection operations
+org.members << alice                    # Add relationship
+org.members.merge([id1, id2, id3])     # Bulk additions
+org.members.size                       # Count relationships
+org.members.empty?                     # Check if any exist
+
+# Set operations
+common = org1.members & org2.members   # Intersection
+all = org1.members | org2.members      # Union
+
+# Load actual objects when needed
+member_ids = org.members.to_a
+members = Person.multiget(*member_ids)  # Efficient bulk loading
+```
+
+> [!WARNING]
+> Collections store identifiers, not objects. Use `multiget` for efficient bulk loading.
+
+### 4. Cascading Operations - Scored Relationships
+
+Use scores for time-based tracking and priority systems:
+
+```ruby
+class Timeline < Familia::Horreum
+  feature :relationships
+  sorted_set :events  # Scored collection
+end
+
+class Event < Familia::Horreum
+  feature :relationships
+  field :timestamp, :priority
+  participates_in Timeline, :events, score: :timestamp
+end
+
+# Automatic scoring when relationships established
+timeline.events << event  # Uses event.timestamp as score
+
+# Time-based and priority queries
+recent = timeline.events.range_by_score((Time.now - 1.hour).to_i, '+inf')
+top_priority = project.tasks.range(0, 4, order: 'DESC')  # Highest priority first
+```
+
+**Common Scoring Patterns:**
+- **Timestamps** for chronological ordering
+- **Priority levels** for ranking systems
+- **User ratings** for recommendation systems
+- **Custom lambdas** for complex scoring logic
+
+## Best Practices
+
+**Performance:**
+- Use `merge([id1, id2, id3])` for bulk additions
+- Use `multiget(*ids)` for efficient bulk loading
+- Use pagination: `collection.range(0, 9)` instead of loading all
+
+**Lifecycle Management:**
+```ruby
+def destroy
+  cleanup_relationships  # Remove from all relationships first
+  super
+end
+```
+
+**Validation:**
+```ruby
+def add_member(user_id)
+  raise "Team is full" if members.size >= max_members
+  members << user_id
+end
+```
+
+## Common Patterns
+
+**Conditional Scoring:**
+```ruby
+class_participates_in :active_users,
+  score: ->(user) { user.active? ? user.last_activity : 0 }
+```
+
+**Bidirectional Updates:**
+```ruby
+# ✅ Automatic bidirectional
+customer.domains << domain
+
+# ❌ Manual (avoid)
+customer.domains.add(domain.identifier)
+```
+
+---
+
+## See Also
+
+- **[Technical Reference](../reference/api-technical.md#relationships-feature-v200-pre7)** - Implementation details and advanced patterns
+- **[Relationship Methods Guide](feature-relationships-methods.md)** - Complete method reference
+- **[Feature System Guide](feature-system.md)** - Understanding Familia's feature architecture
+- **[Implementation Guide](implementation.md)** - Production deployment and configuration patterns
+
+> [!NOTE]
+> **Next Steps:** Once you're comfortable with basic relationships, explore the [Relationship Methods Guide](feature-relationships-methods.md) for detailed method references, or check the [Technical Reference](../reference/api-technical.md#relationships-feature-v200-pre7) for advanced patterns like permission encoding and performance optimization.

data/docs/guides/{Features-System-Developer-Guide.md → feature-system-devs.md}

@@ -776,11 +776,11 @@ module Familia::Features::DebugLogging
 
     base.define_singleton_method(:feature) do |name|
       Familia.ld "[DEBUG] Loading feature #{name} on #{self}"
-      start_time = Time.now
+      start_time = Familia.now
 
       result = original_feature_method.call(name)
 
-      load_time = (Time.now - start_time) * 1000
+      load_time = (Familia.now - start_time) * 1000
       Familia.ld "[DEBUG] Feature #{name} loaded in #{load_time.round(2)}ms"
 
       result
@@ -792,11 +792,11 @@ module Familia::Features::DebugLogging
       return block.call unless Familia.debug?
 
       Familia.ld "[DEBUG] Calling #{method_name} on #{self}"
-      start_time = Time.now
+      start_time = Familia.now
 
       result = block.call
 
-      duration = (Time.now - start_time) * 1000
+      duration = (Familia.now - start_time) * 1000
       Familia.ld "[DEBUG] #{method_name} completed in #{duration.round(2)}ms"
 
       result

data/docs/guides/{Feature-System-Guide.md → feature-system.md}

@@ -113,7 +113,7 @@ class Customer < Familia::Horreum
   field :custid, :name, :email
 
   # Define relationship collections
-  tracked_in :active_users, type: :sorted_set
+  participates_in :active_users, type: :sorted_set
   indexed_by :email_lookup, field: :email
   set :domains
 end
@@ -125,7 +125,7 @@ class Domain < Familia::Horreum
   field :domain_id, :name
 
   # Declare membership in customer collections
-  member_of Customer, :domains, type: :set
+  participates_in Customer, :domains, type: :set
 end
 
 # Usage
@@ -165,7 +165,7 @@ module Familia
   module Features
     module MyCustomFeature
       def self.included(base)
-        Familia.trace :LOADED, self, base, caller(1..1) if Familia.debug?
+        Familia.trace :LOADED, self, base if Familia.debug?
         base.extend ClassMethods
         base.prepend InstanceMethods  # Use prepend for method interception
       end
@@ -213,12 +213,12 @@ module Familia
 
       module ClassMethods
         def enable_audit_for(*field_names)
-          @audited_fields ||= Set.new
+          @audited_fields ||= ::Set.new
           @audited_fields.merge(field_names.map(&:to_sym))
         end
 
         def audited_fields
-          @audited_fields || Set.new
+          @audited_fields || ::Set.new
         end
       end
 

data/docs/guides/{Transient-Fields-Guide.md → feature-transient-fields.md}

@@ -2,7 +2,7 @@
 
 ## Overview
 
-Transient fields provide secure handling of sensitive runtime data that should never be persisted to Redis/Valkey. Unlike encrypted fields, transient fields exist only in memory and are automatically wrapped in `RedactedString` for security.
+Transient fields provide secure handling of sensitive runtime data that should never be persisted to Valkey/Redis. Unlike encrypted fields, transient fields exist only in memory and are automatically wrapped in `RedactedString` for security.
 
 ## When to Use Transient Fields
 
@@ -270,7 +270,7 @@ end
 
 | Feature | Encrypted Fields | Transient Fields |
 |---------|------------------|------------------|
-| **Persistence** | Saved to Redis/Valkey | Memory only |
+| **Persistence** | Saved to Valkey/Redis | Memory only |
 | **Encryption** | AES/XChaCha20 | None (not stored) |
 | **Use Case** | Long-term secrets | Runtime secrets |
 | **Access** | Automatic decrypt | RedactedString wrapper |

data/docs/guides/{Implementation-Guide.md → implementation.md}

@@ -5,8 +5,8 @@
 The encrypted fields feature uses a modular provider system with field transformation hooks:
 
 ```
-User Input → Field Setter → Provider Selection → Encryption → Redis/Valkey
-Redis/Valkey → Algorithm Detection → Decryption → Field Getter → User Output
+User Input → Field Setter → Provider Selection → Encryption → Valkey/Redis
+Valkey/Redis → Algorithm Detection → Decryption → Field Getter → User Output
 ```
 
 ### Provider Architecture
@@ -247,7 +247,7 @@ it "encrypts sensitive fields", :encryption do
   user = User.create(favorite_snack: "leftover pizza")
 
   # Verify encryption in Redis
-  raw_value = redis.hget(user.rediskey, "favorite_snack")
+  raw_value = redis.hget(user.dbkey, "favorite_snack")
   expect(raw_value).not_to include("leftover pizza")
   expect(JSON.parse(raw_value)).to have_key("ciphertext")
 end

data/docs/guides/index.md

@@ -0,0 +1,176 @@
+# Familia v2.0 Documentation
+
+Welcome to the comprehensive documentation for Familia v2.0. This guide collection provides detailed explanations of all major features including security, connection management, architecture, and object relationships.
+
+> **📖 Documentation Layers**
+> - **[Overview](../overview.md)** - Conceptual introduction and getting started
+> - **[Technical Reference](../reference/api-technical.md)** - Implementation patterns and technical details
+> - **This Guide Collection** - Deep-dive topic guides with detailed prose and examples
+
+## 📚 Guide Structure
+
+### 🏗️ Architecture & System Design
+
+4. **[Feature System](feature-system.md)** - Modular architecture with dependencies and autoloader patterns
+5. **[Feature System for Developers](feature-system-devs.md)** - Advanced feature development patterns
+3. **[Security Model](security-model.md)** - Cryptographic design and Ruby memory considerations
+6. **[Connection Pooling](config-connection-pooling.md)** - Provider pattern for efficient Redis/Valkey pooling
+7. **[Core Field System](core-field-system.md)** - Field definitions and data type mappings
+
+
+### 🔐 Special Field Types
+
+1. **[Encrypted Fields](feature-encrypted-fields.md)** - Persistent encrypted storage with modular providers
+2. **[Transient Fields](feature-transient-fields.md)** - Non-persistent secure data handling with RedactedString
+10. **[Object Identifiers](feature-object-identifiers.md)** - Automatic ID generation with configurable strategies _(new!)_
+11. **[External Identifiers](feature-external-identifiers.md)** - Integration with external systems and legacy data _(new!)_
+
+
+### 🔗 Object Relationships & Identifiers
+
+8. **[Relationships](feature-relationships.md)** - Object relationships and membership system
+9. **[Relationship Methods](feature-relationships-methods.md)** - Detailed method reference for relationships
+
+### ⏱️ Time & Analytics Features
+
+12. **[Expiration](feature-expiration.md)** - TTL management and cascading expiration
+13. **[Quantization](feature-quantization.md)** - Time-based data bucketing for analytics
+14. **[Time Utilities](time-utilities.md)** - Time manipulation and formatting utilities
+
+### 🛠️ Implementation & Usage
+
+15. **[Implementation Guide](implementation.md)** - Advanced configuration and usage patterns
+
+## 🚀 Quick Start Examples
+
+### Encrypted Fields (Persistent)
+```ruby
+class User < Familia::Horreum
+  feature :encrypted_fields
+  encrypted_field :secret_recipe
+end
+
+# Configure encryption
+Familia.configure do |config|
+  config.encryption_keys = { v1: ENV['FAMILIA_ENCRYPTION_KEY'] }
+  config.current_key_version = :v1
+end
+
+user = User.new(secret_recipe: "donna's cookies")
+user.save
+user.secret_recipe  # => "donna's cookies" (automatically decrypted)
+```
+
+### Feature System (Modular)
+```ruby
+class Customer < Familia::Horreum
+  feature :safe_dump       # API-safe serialization
+  feature :expiration      # TTL support
+  feature :encrypted_fields # Secure storage
+
+  field :name, :email
+  encrypted_field :api_key
+  default_expiration 24.hours
+  safe_dump_fields :name, :email
+end
+```
+
+### Connection Pooling (Performance)
+```ruby
+# Configure connection provider for multi-database pooling
+Familia.connection_provider = lambda do |uri|
+  parsed = URI.parse(uri) # => URI::Redis
+  pool_key = "#{parsed.host}:#{parsed.port}/#{parsed.db || 0}"
+
+  @pools[pool_key] ||= ConnectionPool.new(size: 10) do
+    Redis.new(host: parsed.host, port: parsed.port, db: parsed.db || 0)
+  end
+
+  @pools[pool_key].with { |conn| conn }
+end
+```
+
+### Object Relationships
+```ruby
+class Customer < Familia::Horreum
+  feature :relationships
+  identifier_field :custid
+  field :custid, :name, :email
+  set :domains  # Customer collections
+end
+
+class Domain < Familia::Horreum
+  feature :relationships
+  identifier_field :domain_id
+  field :domain_id, :name, :dns_zone
+  participates_in Customer, :domains  # Bidirectional membership
+end
+
+# Create objects and establish relationships
+customer = Customer.new(custid: "cust123", name: "Acme Corp")
+domain = Domain.new(domain_id: "dom456", name: "acme.com")
+
+# Ruby-like syntax for relationships
+customer.domains << domain  # Clean collection syntax
+
+# Query relationships
+domain.in_customer_domains?(customer.custid)  # => true
+customer.domains.member?(domain.identifier)   # => true
+```
+
+### Object Identifiers (Auto-generation)
+```ruby
+class Document < Familia::Horreum
+  feature :object_identifier, generator: :uuid_v4
+  field :title, :content
+end
+
+class Session < Familia::Horreum
+  feature :object_identifier, generator: :hex
+  field :user_id, :data
+end
+
+# Automatic ID generation
+doc = Document.create(title: "My Document")
+doc.objid  # => "f47ac10b-58cc-4372-a567-0e02b2c3d479"
+
+session = Session.create(user_id: "123")
+session.objid  # => "a1b2c3d4e5f6"
+```
+
+### External Identifiers (Legacy Integration)
+```ruby
+class ExternalUser < Familia::Horreum
+  feature :external_identifier
+  field :internal_id, :external_id, :name
+end
+
+# Map external system IDs to internal objects
+user = ExternalUser.create(
+  internal_id: SecureRandom.uuid,
+  external_id: "ext_12345",
+  name: "Legacy User"
+)
+
+# Find by external ID
+found = ExternalUser.find_by_external_id("ext_12345")
+```
+
+### Quantization (Analytics)
+```ruby
+class MetricsBucket < Familia::Horreum
+  feature :quantization
+  field :metric_key, :value_count
+  string :counter, quantize: [10.minutes, '%H:%M']
+end
+
+# Automatic time bucketing for analytics
+MetricsBucket.record_event("page_view")  # Groups into 10-min buckets
+```
+
+
+## Related Resources
+
+- [Familia README](https://github.com/delano/familia) - Main project documentation
+- [Issue #57](https://github.com/delano/familia/issues/57) - Original feature proposal
+- [Issue #58](https://github.com/delano/familia/issues/58) - Wiki documentation tracking

data/docs/guides/{Security-Model.md → security-model.md}

@@ -138,7 +138,7 @@ Ruby provides **NO** memory safety guarantees for cryptographic secrets. This af
 
 Both providers attempt best-effort memory clearing:
 - Call `.clear` on sensitive strings after use
-- Set variables to `nil` when done
+- UnsortedSet variables to `nil` when done
 - Use finalizers for cleanup (no guarantees)
 
 **Recommendation**: For production systems with high-security requirements, consider:

data/docs/migrating/v2.0.0-pre.md

@@ -6,7 +6,7 @@ This guide covers migrating from Familia v1.x to the v2.0.0-pre foundation relea
 
 The v2.0.0-pre series represents a major modernization of Familia with:
 - Complete API redesign for clarity and consistency
-- Valkey compatibility alongside Redis support
+- Valkey compatibility alongside Valkey/Redis support
 - Ruby 3.4+ modernization with improved thread safety
 - New connection pooling architecture
 

data/docs/migrating/v2.0.0-pre11.md

@@ -124,7 +124,7 @@ end
 
 class Customer < Familia::Horreum
   module Features
-    include Familia::Autoloader
+    include Familia::Features::Autoloader
     # Automatically discovers and loads all *.rb files from customer/features/
   end
 end
@@ -172,7 +172,7 @@ module CommonFields
     field :version
 
     def touch_updated
-      self.updated = Time.now.to_i
+      self.updated = Familia.now.to_i
     end
   end
 
@@ -202,7 +202,7 @@ safe_dump_field :field2
 safe_dump_field :field3, ->(obj) { ... }
 ```
 
-### 2. Set Up Auto-loading (Optional)
+### 2. UnsortedSet Up Auto-loading (Optional)
 If you have project-specific features, set up auto-loading:
 
 ```ruby
@@ -210,7 +210,7 @@ If you have project-specific features, set up auto-loading:
 module YourProject
   class ModelName < Familia::Horreum
     module Features
-      include Familia::Autoloader
+      include Familia::Features::Autoloader
     end
   end
 end

data/docs/migrating/v2.0.0-pre12.md

@@ -14,7 +14,7 @@ The new `Familia::VerifiableIdentifier` module allows applications to create and
 class Customer < Familia::Horreum
   feature :verifiable_identifier
 
-  # Required: Set the HMAC secret (do this once in your app initialization)
+  # Required: UnsortedSet the HMAC secret (do this once in your app initialization)
   # Generate with: SecureRandom.hex(64)
   ENV['VERIFIABLE_ID_HMAC_SECRET'] = 'your_64_character_hex_secret'
 end
@@ -85,7 +85,7 @@ puts "VERIFIABLE_ID_HMAC_SECRET=#{secret}"
 #### Environment Configuration
 ```ruby
 # config/application.rb or equivalent
-# Set this BEFORE any VerifiableIdentifier usage
+# UnsortedSet this BEFORE any VerifiableIdentifier usage
 ENV['VERIFIABLE_ID_HMAC_SECRET'] = Rails.application.credentials.verifiable_id_secret
 
 # Or configure programmatically

data/docs/migrating/v2.0.0-pre13.md

@@ -68,7 +68,7 @@ Common issues:
 
 The Feature Autoloading System consists of two key components:
 
-### Familia::Autoloader
+### Familia::Features::Autoloader
 A utility module providing shared file loading functionality:
 - Handles Dir.glob pattern matching and file loading
 - Provides consistent debug logging across all autoloading scenarios

data/docs/migrating/v2.0.0-pre5.md

@@ -51,7 +51,7 @@ end
 
 ### 3. Update Serialization Code
 
-Handle `RedactedString` in serialization:
+Handle `ConcealedString` in serialization:
 
 **Before:**
 ```ruby
@@ -63,16 +63,16 @@ end
 **After:**
 ```ruby
 def to_json
-  # RedactedString automatically excluded from serialization
+  # ConcealedString automatically excluded from serialization
   { name: name }.to_json  # password field omitted if transient
 end
 ```
 
-**Manual RedactedString Handling:**
+**Manual ConcealedString Handling:**
 ```ruby
 # Access original value when needed
 password.reveal  # Returns actual string value
-password.redacted?  # Returns true if redacted
+password.cleared?  # Returns true if cleared from memory
 ```
 
 ### 4. Implement Key Rotation Procedures
@@ -80,19 +80,38 @@ password.redacted?  # Returns true if redacted
 **Rotation Process:**
 1. Add new key version to configuration
 2. Update `current_key_version`
-3. Re-encrypt existing data gradually
-4. Remove old keys after migration complete
+3. Re-encrypt existing data using `re_encrypt_fields!`
+4. Verify migration completion
+5. Remove old keys after migration complete
 
 **Example Rotation Script:**
 ```ruby
-# Add new key version
-Familia.config.encryption_keys[:v3] = 'new-key'
-Familia.config.current_key_version = :v3
+# Step 1: Add new key version
+Familia.configure do |config|
+  config.encryption_keys = {
+    v2: ENV['OLD_ENCRYPTION_KEY'],
+    v3: ENV['NEW_ENCRYPTION_KEY']
+  }
+  config.current_key_version = :v3
+end
+
+# Step 2: Validate configuration
+Familia::Encryption.validate_configuration!
 
-# Re-encrypt existing records
+# Step 3: Re-encrypt existing records
 Vault.all.each do |vault|
-  vault.save  # Automatically uses new key version
+  vault.re_encrypt_fields!  # Re-encrypts with current key
+  vault.save
 end
+
+# Step 4: Verify migration
+Vault.all.each do |vault|
+  status = vault.encrypted_fields_status
+  puts "Vault #{vault.identifier}: #{status}"
+end
+
+# Step 5: Remove old key (after verification)
+Familia.config.encryption_keys.delete(:v2)
 ```
 
 ## Security Best Practices
@@ -101,7 +120,9 @@ end
 - **Key Rotation:** Rotate encryption keys regularly (quarterly/annually)
 - **Field Selection:** Only encrypt fields that truly need protection
 - **Memory Clearing:** Use transient fields for temporary sensitive data
-- **Logging:** Verify RedactedString prevents accidental logging
+- **Logging:** Verify ConcealedString prevents accidental logging
+- **Configuration Validation:** Use `validate_configuration!` before production
+- **Monitoring:** Use `encrypted_fields_status` to track encryption state
 
 ## Next Steps
 

data/docs/migrating/v2.0.0-pre6.md

@@ -35,7 +35,7 @@ The Horreum class structure was reorganized for better maintainability:
 - `Familia::Horreum::Core` - Essential functionality
 - `Familia::Horreum::ClassMethods` - Class-level methods
 - `Familia::Horreum::Serialization` - Object serialization
-- `Familia::Horreum::Commands` - Redis command wrappers
+- `Familia::Horreum::Commands` - Valkey/Redis command wrappers
 
 **Feature System Improvements:**
 - Dependency management between features
@@ -56,7 +56,7 @@ end
 ```
 
 **Improved Data Consistency:**
-- Automatic retry for transient Redis connection issues
+- Automatic retry for transient Valkey/Redis connection issues
 - Better handling of concurrent modifications
 - Enhanced validation before persistence operations