familia 2.0.0.pre7 → 2.0.0.pre10

data/docs/archive/FAMILIA_UPDATE.md

@@ -0,0 +1,226 @@
+# Familia v2.0.0-pre Series Update Overview
+
+**Familia** is a Ruby ORM for Redis/Valkey that provides object mapping and persistence capabilities. This document summarizes the major updates from v2.0.0-pre through v2.0.0-pre7, representing a significant evolution with new features, security enhancements, and architectural improvements.
+
+## Version Summary
+
+| Version | Focus | Key Features |
+|---------|-------|-------------|
+| v2.0.0-pre | Foundation | Modern API, Valkey support, connection pooling |
+| v2.0.0-pre5 | Security | Encrypted fields, transient fields, RedactedString |
+| v2.0.0-pre6 | Architecture | Horreum reorganization, enhanced persistence |
+| v2.0.0-pre7 | Relationships | Comprehensive relationship system, permissions |
+
+---
+
+## v2.0.0-pre - Foundation Release
+
+### Major API Modernization
+- **Complete API redesign** for clarity and modern Ruby conventions
+- **Valkey compatibility** alongside traditional Redis support
+- **Ruby 3.4+ modernization** with fiber and thread safety improvements
+- **Connection pooling foundation** with provider pattern architecture
+
+### Security & Dependency Management
+- **Critical security fixes** in Ruby workflow vulnerabilities
+- **Systematic dependency resolution** via multi-constraint optimization
+- **GitHub Actions security hardening** with matrix optimization
+
+### Documentation Infrastructure
+- **YARD documentation workflow** with automated GitHub Pages deployment
+- **Comprehensive wiki system** with structured documentation
+- **Developer-focused guides** for implementation and usage
+
+---
+
+## v2.0.0-pre5 - Security Enhancement Release
+
+### Encrypted Fields System
+- **Field-level encryption** with transparent access patterns
+- **Multiple encryption providers**:
+  - XChaCha20-Poly1305 (preferred, requires rbnacl)
+  - AES-256-GCM (fallback, OpenSSL-based)
+- **Field-specific key derivation** for cryptographic domain separation
+- **Configurable key versioning** supporting key rotation
+
+```ruby
+class Vault < Familia::Horreum
+  feature :encrypted_fields
+
+  field :name                    # Plaintext
+  encrypted_field :secret_key    # Encrypted at rest
+  encrypted_field :api_token     # Transparent access
+end
+```
+
+### Transient Fields & RedactedString
+- **Non-persistent field storage** for sensitive runtime data
+- **RedactedString wrapper** preventing accidental logging/serialization
+- **Memory-safe handling** of sensitive data in Ruby objects
+- **API-safe serialization** excluding transient fields
+
+```ruby
+class User < Familia::Horreum
+  feature :transient_fields
+
+  field :email
+  transient_field :password      # Never persisted
+  transient_field :session_token # Runtime only
+end
+```
+
+### Connection Pooling Enhancement
+- **Connection provider pattern** for flexible pooling strategies
+- **Multi-database support** with intelligent pool management
+- **Thread-safe connection handling** for concurrent applications
+- **Configurable pool sizing** and timeout management
+
+---
+
+## v2.0.0-pre6 - Architecture Enhancement Release
+
+### Horreum Architecture Reorganization
+- **Modular class structure** with cleaner separation of concerns
+- **Enhanced feature system** with dependency management
+- **Improved inheritance patterns** for better code organization
+- **Streamlined base class functionality**
+
+### Enhanced Persistence Operations
+- **New `save_if_not_exists` method** for conditional persistence
+- **Atomic persistence operations** with transaction support
+- **Enhanced error handling** for persistence failures
+- **Improved data consistency** guarantees
+
+### Security Improvements
+- **Encryption field security hardening** with additional validation
+- **Enhanced memory protection** for sensitive data handling
+- **Improved key management** patterns and best practices
+- **Security test suite expansion** with comprehensive coverage
+
+---
+
+## v2.0.0-pre7 - Relationships & Permissions Release
+
+### Comprehensive Relationships System
+- **Three relationship types** optimized for different use cases:
+  - `tracked_in` - Multi-presence tracking with score encoding
+  - `indexed_by` - O(1) hash-based lookups
+  - `member_of` - Bidirectional membership with collision-free naming
+
+```ruby
+class Customer < Familia::Horreum
+  feature :relationships
+
+  identifier_field :custid
+  field :custid, :name, :email
+
+  # Define collections
+  set :domains
+  tracked_in :active_users, type: :sorted_set
+end
+
+class Domain < Familia::Horreum
+  feature :relationships
+
+  identifier_field :domain_id
+  field :domain_id, :name
+
+  # Bidirectional relationship
+  member_of Customer, :domains, type: :set
+end
+```
+
+### Categorical Permission System
+- **Bit-encoded permissions** for efficient storage and querying
+- **Time-based permission scoring** for temporal access control
+- **Permission tier hierarchies** with inheritance patterns
+- **Scalable permission management** for large object collections
+
+### Advanced Relationship Features
+- **Score-based sorting** with custom scoring functions
+- **Permission-aware queries** filtering by access levels
+- **Relationship validation framework** ensuring data integrity
+- **Performance optimizations** for large-scale relationship operations
+
+---
+
+## Breaking Changes & Migration
+
+### v2.0.0-pre Series
+- **API method renaming** for consistency and clarity
+- **Configuration changes** in connection management
+- **Feature activation syntax** updates for the new system
+- **Identifier field declaration** syntax modernization
+
+### Security Considerations
+- **Encryption key configuration** required for encrypted fields
+- **Memory handling changes** for sensitive data protection
+- **Permission system migration** for existing relationship data
+
+---
+
+## Performance Improvements
+
+### Connection Management
+- **Pool-based connections** reducing connection overhead
+- **Intelligent connection reuse** across operations
+- **Concurrent operation support** with thread-safe pooling
+
+### Relationship Operations
+- **O(1) indexed lookups** for field-based queries
+- **Optimized sorted set operations** for scored relationships
+- **Batch relationship operations** for bulk updates
+- **Memory-efficient bit encoding** for permission storage
+
+### Feature System
+- **Lazy feature loading** reducing memory footprint
+- **Dependency-aware activation** preventing conflicts
+- **Optimized method dispatch** for feature methods
+
+---
+
+## Migration Guide Summary
+
+### From v1.x to v2.0.0-pre
+1. **Update connection configuration** to use new pooling system
+2. **Migrate identifier declarations** to new syntax
+3. **Update feature activations** to use `feature :name` syntax
+4. **Review method calls** for renamed API methods
+
+### Security Feature Adoption
+1. **Configure encryption keys** for encrypted fields
+2. **Identify sensitive fields** for encryption/transient marking
+3. **Update serialization code** to handle RedactedString
+4. **Implement key rotation procedures** for production systems
+
+### Relationship System Migration
+1. **Analyze existing relationships** for optimization opportunities
+2. **Choose appropriate relationship types** based on usage patterns
+3. **Implement permission systems** for access-controlled relationships
+4. **Update queries** to use new relationship methods
+
+---
+
+## Next Steps & Roadmap
+
+### Immediate (v2.0.0 Final)
+- **Production stability** testing and bug fixes
+- **Performance benchmarking** and optimization
+- **Documentation completion** for all features
+- **Migration tooling** for existing applications
+
+### Future Releases
+- **Advanced encryption features** with hardware security modules
+- **Extended relationship types** for specialized use cases
+- **Performance analytics** and monitoring integration
+- **Cloud-native deployment** patterns and examples
+
+---
+
+## Resources
+
+- [GitHub Repository](https://github.com/delano/familia)
+- [Wiki Documentation](https://github.com/delano/familia/wiki)
+- [API Reference](docs/wiki/API-Reference.md)
+- [Implementation Guide](docs/wiki/Implementation-Guide.md)
+- [Security Model](docs/wiki/Security-Model.md)

data/docs/archive/README.md

@@ -0,0 +1,67 @@
+# Archived Documentation
+
+This directory contains original documentation files that have been migrated to the new Scriv-based changelog system and reorganized documentation structure.
+
+## Migration Date
+**September 1, 2025** - As part of implementing [Issue #84: Scriv-based changelog system](https://github.com/delano/familia/issues/84)
+
+## Archived Files
+
+### FAMILIA_UPDATE.md
+**Original Purpose:** Version summary table and detailed release notes for v2.0.0-pre series
+
+**Migration Destinations:**
+- **Changelog entries** → Extracted to Scriv fragments, aggregated into `CHANGELOG.md`
+- **Migration guides** → Reorganized into `docs/migration/v2.0.0-pre*.md`
+- **Feature descriptions** → Cross-referenced with existing feature guides in `docs/guides/`
+
+### FAMILIA_RELATIONSHIPS.md
+**Original Purpose:** Auto-generated relationship methods reference
+
+**Migration Destination:**
+- **Complete content** → Moved to `docs/guides/relationships-methods.md`
+- **Cross-referenced** with existing `docs/guides/Relationships-Guide.md`
+
+### FAMILIA_TECHNICAL.md
+**Original Purpose:** Technical API reference for v2.0.0-pre series classes and methods
+
+**Migration Destination:**
+- **Core technical content** → Moved to `docs/reference/api-technical.md`
+- **Cross-referenced** with existing `docs/guides/API-Reference.md`
+
+## New Documentation Structure
+
+```
+docs/
+├── migration/           # Version-specific migration guides
+│   ├── v2.0.0-pre.md   # Foundation migration
+│   ├── v2.0.0-pre5.md  # Security features
+│   ├── v2.0.0-pre6.md  # Architecture improvements
+│   └── v2.0.0-pre7.md  # Relationships system
+├── guides/              # Feature guides (moved from wiki/)
+│   ├── relationships-methods.md  # From FAMILIA_RELATIONSHIPS.md
+│   └── [other guides...]
+├── reference/           # Technical reference
+│   └── api-technical.md # From FAMILIA_TECHNICAL.md
+└── archive/            # This directory
+```
+
+## Why These Files Were Archived
+
+1. **Sustainability** - The original files accumulated overlapping content without clear organization
+2. **Maintainability** - Scriv fragment system prevents documentation bloat
+3. **Clarity** - Separation of changelog, guides, and reference improves findability
+4. **Workflow** - Fragment-based workflow scales better with development
+
+## Finding Migrated Content
+
+- **Version changes** → Check `CHANGELOG.md` (generated from fragments)
+- **How to upgrade** → See `docs/migration/` for version-specific guides
+- **Feature usage** → See `docs/guides/` for implementation examples
+- **API reference** → See `docs/reference/` for technical details
+
+## References
+
+- [Issue #84](https://github.com/delano/familia/issues/84) - Original migration plan
+- [Scriv Documentation](https://scriv.readthedocs.io/) - Changelog management tool
+- [Keep a Changelog](https://keepachangelog.com/) - Changelog format standard

data/docs/guides/.gitignore

@@ -0,0 +1,2 @@
+# Even all wiki markdown docs get the blues
+!*.md

data/docs/{wiki → guides}/Feature-System-Guide.md

@@ -517,21 +517,6 @@ class ConfigurableModel < Familia::Horreum
 end
 ```
 
-### Runtime Feature Checking
-
-```ruby
-class Model < Familia::Horreum
-  feature :expiration
-
-  def available_features
-    self.class.features_enabled
-  end
-end
-
-model = Model.new
-model.available_features  # => [:expiration]
-```
-
 ## Testing Features
 
 ### Feature Testing

data/docs/{wiki → guides}/Relationships-Guide.md

@@ -27,9 +27,13 @@ class Customer < Familia::Horreum
   identifier_field :custid
   field :custid, :name, :email
 
-  # Define relationship collections
-  tracked_in :active_users, type: :sorted_set
-  indexed_by :email_lookup, type: :hash
+  # Collections for relationships
+  set :domains           # Simple set of domain IDs
+  sorted_set :activity   # Activity feed with timestamps
+
+  # Class-level relationship collections (automatically managed)
+  class_tracked_in :all_customers, score: :created_at
+  class_indexed_by :email, :email_lookup
 end
 
 class Domain < Familia::Horreum
@@ -39,7 +43,7 @@ class Domain < Familia::Horreum
   field :domain_id, :name, :dns_zone
 
   # Define bidirectional membership
-  member_of Customer, :domains, type: :set
+  member_of Customer, :domains
 end
 ```
 
@@ -57,27 +61,26 @@ class User < Familia::Horreum
   field :user_id, :name, :score_value
 
   # Simple sorted set tracking
-  tracked_in :leaderboard, type: :sorted_set, score: :score_value
+  class_tracked_in :leaderboard, score: :score_value
 
   # Time-based tracking with automatic timestamps
-  tracked_in :activity_log, type: :sorted_set
+  class_tracked_in :activity_log, score: :created_at
 
   # Proc-based scoring for complex calculations
-  tracked_in :performance_metrics, type: :sorted_set,
-    score: ->(user) { (user.score_value || 0) * 2 }
+  class_tracked_in :performance_metrics, score: -> { (score_value || 0) * 2 }
 end
 
 # Usage
 user = User.new(user_id: 'user123', score_value: 85)
 
 # Add to collections
-User.add_to_leaderboard(user)      # Uses score_value (85)
-User.add_to_activity_log(user)     # Uses current timestamp
-User.add_to_performance_metrics(user) # Uses proc result (170)
+User.add_to_leaderboard(user)              # Uses score_value (85)
+User.add_to_activity_log(user)             # Uses created_at timestamp
+User.add_to_performance_metrics(user)      # Uses proc result (170)
 
 # Query collections
 User.leaderboard.score('user123')           # => 85.0
-User.activity_log.range_by_score('-inf', '+inf')  # All users by time
+User.activity_log.rangebyscore('-inf', '+inf')  # All users by time
 User.performance_metrics.rank('user123')    # User's rank by performance
 ```
 
@@ -93,8 +96,7 @@ class Document < Familia::Horreum
   field :doc_id, :title, :content
 
   # Permission-based tracking with 8-bit encoding
-  tracked_in :authorized_users, type: :sorted_set,
-    score: :encode_permissions
+  class_tracked_in :authorized_users, score: :encode_permissions
 
   private
 
@@ -183,7 +185,7 @@ end
 
 ### Hash-Based Lookups
 
-The `indexed_by` relationship creates O(1) hash-based indexes for field values:
+The `indexed_by` relationship creates O(1) hash-based indexes for field values. The `context` parameter determines index ownership and scope:
 
 ```ruby
 class User < Familia::Horreum
@@ -191,55 +193,102 @@ class User < Familia::Horreum
 
   field :email, :username, :department
 
-  # Create indexes for fast lookups
-  indexed_by :email_index, field: :email
-  indexed_by :username_index, field: :username
-  indexed_by :department_index, field: :department
+  # Global indexes for system-wide unique lookups
+  class_indexed_by :email, :email_index
+  class_indexed_by :username, :username_index
+
+  # Scoped indexes for values unique within a context
+  indexed_by :department, :department_index, parent: Organization
 end
 
-# Usage
+# Usage for Global Context
 user = User.new(email: 'john@example.com', username: 'johndoe')
-User.add_to_email_index(user)
-User.add_to_username_index(user)
 
-# Fast O(1) lookups
+# Add to global indexes (instance methods)
+user.add_to_global_email_index
+user.add_to_global_username_index
+
+# Fast O(1) lookups (class methods)
 user_id = User.email_index.get('john@example.com')    # => user.identifier
 user_id = User.username_index.get('johndoe')         # => user.identifier
 
 # Batch operations
 users = [user1, user2, user3]
-users.each { |u| User.add_to_email_index(u) }
+users.each { |u| u.add_to_global_email_index }
 
 # Check if indexed
 User.email_index.exists?('john@example.com')  # => true
+
+# Usage for Scoped Context
+organization = Organization.new(org_id: 'acme_corp')
+organization.find_by_department('engineering')        # Find user by department within this org
 ```
 
-### Multi-Field Indexing
+### Context Parameter Usage Patterns
+
+Understanding when to use global vs class context:
 
 ```ruby
 class Product < Familia::Horreum
   feature :relationships
 
-  field :category, :brand, :status
+  field :sku, :category, :brand
 
-  # Composite indexing for complex queries
-  indexed_by :category_brand_index, field: ->(product) {
-    "#{product.category}:#{product.brand}"
-  }
+  # Global context: SKUs must be unique system-wide
+  class_indexed_by :sku, :sku_index
 
-  indexed_by :active_products, field: ->(product) {
-    product.status == 'active' ? product.identifier : nil
-  }
+  # Class context: Categories are unique per brand
+  indexed_by :category, :category_index, parent: Brand
 end
 
-# Usage
-product = Product.new(category: 'electronics', brand: 'apple', status: 'active')
-Product.add_to_category_brand_index(product)
-Product.add_to_active_products(product)
+class Brand < Familia::Horreum
+  feature :relationships
+
+  identifier_field :brand_id
+  field :brand_id, :name
+  sorted_set :products
+end
+
+# Usage patterns:
+product = Product.new(sku: 'ELEC001', category: 'laptops', brand: 'apple')
+
+# Global indexing (system-wide unique SKUs)
+product.add_to_global_sku_index
+Product.sku_index.get('ELEC001')  # => product.identifier
 
-# Query by composite key
-Product.category_brand_index.get('electronics:apple')  # => product.identifier
-Product.active_products.exists?(product.identifier)    # => true
+# Scoped indexing (categories unique per brand)
+brand = Brand.new(brand_id: 'apple', name: 'Apple Inc.')
+brand.find_by_category('laptops')         # Find products in this brand's laptop category
+```
+
+### Context Parameter Reference
+
+The `context` parameter is a **required** architectural decision that determines index scope:
+
+| Context Type | Usage | Redis Key Pattern | When to Use |
+|--------------|--------|------------------|-------------|
+| `:global` | `context: :global` | `global:index_name` | Field values unique system-wide (emails, usernames, API keys) |
+| Class | `context: SomeClass` | `someclass:123:index_name` | Field values unique within parent object scope (project names per team) |
+
+#### Generated Methods
+
+**Global Context** (`context: :global`):
+- **Instance methods**: `object.add_to_global_index_name`, `object.remove_from_global_index_name`
+- **Class methods**: `Class.index_name` (returns hash), `Class.find_by_field`
+
+**Class Context** (`context: Customer`):
+- **Instance methods**: `object.add_to_customer_index_name(customer)`, `object.remove_from_customer_index_name(customer)`
+- **Class methods on context**: `customer.find_by_field(value)`, `customer.find_all_by_field(values)`
+
+#### Migration from Incorrect Syntax
+
+```ruby
+# ❌ Old incorrect syntax (will cause ArgumentError)
+indexed_by :email_lookup, field: :email
+
+# ✅ New correct syntax
+class_indexed_by :email, :email_lookup                  # Global scope
+indexed_by :email, :customer_lookup, parent: Customer   # Scoped per customer
 ```
 
 ## Member Of Relationships
@@ -444,8 +493,8 @@ class User < Familia::Horreum
 
   # Multi-tenant membership
   member_of Organization, :members, type: :set
-  tracked_in :global_activity, type: :sorted_set
-  indexed_by :email_lookup, field: :email
+  class_tracked_in :global_activity, score: :created_at
+  class_indexed_by :email, :email_lookup
 end
 
 class Project < Familia::Horreum
@@ -455,7 +504,7 @@ class Project < Familia::Horreum
   field :project_id, :name, :status
 
   member_of Organization, :projects, type: :set
-  tracked_in :status_timeline, type: :sorted_set,
+  class_tracked_in :status_timeline,
     score: ->(proj) { "#{Time.now.to_i}.#{proj.status.hash}" }
 end
 
@@ -464,17 +513,21 @@ org = Organization.new(org_id: 'org123', name: 'Acme Corp')
 user = User.new(user_id: 'user456', email: 'john@acme.com')
 project = Project.new(project_id: 'proj789', name: 'Website')
 
-# Establish relationships
-user.add_to_organization_members(org.org_id)
-project.add_to_organization_projects(org.org_id)
+# Establish relationships with clean syntax
+org.members << user    # Clean Ruby-like syntax
+org.projects << project
+
+# Save objects to trigger automatic indexing
+user.save    # Automatically added to class-level indexes
+project.save # Automatically added to class-level tracking
 
 # Query organization structure
 org.members.size      # Number of organization members
 org.projects.members  # All project IDs in organization
 
-# Global indexes
-User.add_to_email_lookup(user)
+# Automatic class-level indexing (no manual calls needed)
 user_id = User.email_lookup.get('john@acme.com')  # Fast email lookup
+found_user = User.find_by_email('john@acme.com')  # Convenience method
 ```
 
 ### Analytics and Reporting
@@ -485,7 +538,7 @@ class AnalyticsService
     since = (Time.now - days.days).to_f.floor
 
     # Get all users active in time period
-    active_users = User.global_activity.range_by_score(since, '+inf')
+    active_users = User.activity.range_by_score(since, '+inf')
 
     # Analyze permission levels
     permission_breakdown = Document.authorized_users
@@ -495,7 +548,7 @@ class AnalyticsService
     {
       total_active_users: active_users.size,
       permission_breakdown: permission_breakdown.transform_values(&:size),
-      top_contributors: User.global_activity.range(0, 9, with_scores: true)
+      top_contributors: User.activity.range(0, 9, with_scores: true)
     }
   end