familia 2.0.0.pre15 → 2.0.0.pre17

data/README.md

@@ -1,8 +1,56 @@
 # Familia - 2.0
 
-**Organize and store Ruby objects in Valkey/Redis. A powerful Ruby ORM (of sorts) for Valkey/Redis.**
+**Organize and store Ruby objects in Valkey/Redis using native database types (an ORM of sorts).**
 
-Familia provides a flexible and feature-rich way to interact with Valkey using Ruby objects. It's designed to make working with Valkey as natural as working with Ruby classes, while offering advanced features for complex data management.
+Familia provides object-oriented access to Valkey/Redis using their database types. Unlike traditional ORMs that map objects to relational tables, Familia maps Ruby objects directly to Valkey's native data structures (strings, lists, sets, sorted sets, hashes) as instance variables.
+
+> [!CAUTION]
+> Familia 2 is in pre-release and not ready for production use. (October 2025)
+## Traditional ORM vs Familia
+
+**Traditional ORMs** convert your objects to SQL tables. A product with categories becomes two tables with a join table. Checking if a tag exists requires a query with joins.
+
+**Familia** stores your objects using Valkey/Redis data structures directly. A product with categories uses an actual Valkey/Redis list. Checking if a tag exists is a native O(1) Valkey/Redis operation.
+
+```ruby
+# Traditional ORM - everything becomes SQL tables
+class Product < ActiveRecord::Base
+  has_and_belongs_to_many :tags  # Creates products_tags junction table
+end
+
+product.tags.include?(tag)  # SELECT * FROM products_tags WHERE ...
+
+# Familia - uses Valkey/Redis data types directly
+class Product < Familia::Horreum
+  set :tags                  # Actual Valkey/Redis set
+end
+
+product.tags.include?("electronics")  # Valkey/Redis SISMEMBER - O(1) operation
+```
+
+### What This Means in Practice
+
+When you define a Familia model, each data type declaration creates the corresponding Valkey/Redis structure:
+
+```ruby
+class Product < Familia::Horreum
+  identifier_field :sku
+
+  field :name, :price        # Stored in Valkey/Redis hash
+  list :categories           # Actual Valkey/Redis list
+  set :tags                  # Actual Valkey/Redis set
+  zset :ratings              # Actual Valkey/Redis sorted set
+  counter :views             # Valkey/Redis string with atomic increment
+end
+
+# These are Valkey/Redis native operations, not ORM abstractions
+product.categories.push("electronics")   # LPUSH
+product.tags.add("popular")              # SADD
+product.ratings.add(4.5, "user123")      # ZADD with score
+product.views.increment                  # INCR (atomic)
+```
+
+The performance characteristics you rely on in Valkey/Redis remain unchanged. Set membership is O(1). Sorted sets maintain order automatically. Counters increment atomically without read-modify-write cycles.
 
 ## Quick Start
 
@@ -10,7 +58,7 @@ Familia provides a flexible and feature-rich way to interact with Valkey using R
 
 ```bash
 # Add to Gemfile
-gem 'familia', '>= 2.0.0'
+gem 'familia', '>= 2.0'
 
 # Or install directly
 gem install familia
@@ -45,19 +93,49 @@ end
 ### 4. Basic Operations
 
 ```ruby
-# Create
-user = User.new(email: 'alice@example.com', name: 'Alice')
-user.save
+# Create and save
+user = User.create(email: 'alice@example.com', name: 'Alice', created_at: Time.now.to_i)
 
-# Find
+# Find by identifier
 user = User.load('alice@example.com')
 
-# Update
-user.name = 'Alice Smith'
+# Update and save
+user.name = 'Alice Windows'
 user.save
 
+# Fast update (immediate persistence)
+user.name!('Alice Smith')  # Sets and saves immediately
+
 # Check existence
 User.exists?('alice@example.com')  #=> true
+
+# Delete
+user.destroy
+
+# Conditional save
+user.save_if_not_exists  # Only saves if object doesn't exist yet
+```
+
+### 5. Generated Method Patterns
+
+Familia automatically generates methods for fields and data types:
+
+```ruby
+class User < Familia::Horreum
+  field :name                    # → name, name=, name!
+  set :tags                      # → tags, tags=, tags?
+  list :history                  # → history, history=, history?
+end
+
+# Field methods
+user.name                        # Get field value
+user.name = 'Alice'              # Set field value
+user.name!('Alice')              # Set and save immediately
+
+# Data type methods
+user.tags                        # Get Set instance
+user.tags = new_set              # Replace Set instance
+user.tags?                       # Check if it's a Set type
 ```
 
 ## Prerequisites
@@ -66,13 +144,104 @@ User.exists?('alice@example.com')  #=> true
 - **Valkey/Redis**: 6.0+
 - **Gems**: `redis` (automatically installed)
 
+---
+
+## Core Concepts
+
+### Data Types
+
+Familia provides direct mappings to Valkey/Redis native data structures:
+
+```ruby
+class BlogPost < Familia::Horreum
+  identifier_field :slug
+
+  # Basic fields
+  field :slug, :title, :content, :published_at
+
+  # Valkey/Redis data types as instance variables
+  string :view_count, default: '0'           # Atomic counters
+  list :comments                             # Ordered, allows duplicates
+  set :tags                                  # Unique values
+  zset :popularity_scores                    # Scored/ranked data
+  hashkey :metadata                          # Key-value pairs
+
+  # Advanced field types
+  counter :likes                             # Specialized atomic counter
+end
+
+post = BlogPost.create(slug: "hello-world", title: "Hello World")
+
+# Work with Valkey/Redis data types naturally
+post.view_count.increment                    # INCR view_count
+post.comments.push("Great post!")           # LPUSH comments
+post.tags.add("ruby", "programming")        # SADD tags
+post.popularity_scores.add(4.5, "user123") # ZADD popularity_scores
+post.metadata["author"] = "Alice"           # HSET metadata
+post.likes.increment(5)                     # INCRBY likes 5
+```
+
+### Features System
+
+Enable advanced functionality with Familia's modular feature system:
+
+```ruby
+class User < Familia::Horreum
+  # Enable features as needed
+  feature :expiration                        # TTL management
+  feature :safe_dump                         # API-safe serialization
+  feature :encrypted_fields                  # Field-level encryption
+  feature :relationships                     # Object relationships
+  feature :object_identifier                 # Auto-generated IDs
+  feature :quantization                      # Time-based data bucketing
+
+  identifier_field :email
+  field :email, :name, :created_at
+
+  # Feature-specific functionality
+  encrypted_field :api_key                   # Automatically encrypted
+  safe_dump_field :email                     # Include in safe_dump
+  safe_dump_field :name                      # Include in safe_dump
+  default_expiration 30.days                # Auto-expire inactive users
+end
+
+user = User.create(email: "alice@example.com", api_key: "secret123")
+user.api_key.class                          # => ConcealedString
+user.api_key.to_s                           # => "[CONCEALED]" (safe for logs)
+user.safe_dump                              # => {email: "...", name: "..."}
+```
+
+### Querying and Finding
+
+```ruby
+# Primary key lookup
+user = User.load("alice@example.com")
+
+# Existence checks
+User.exists?("alice@example.com")           # => true/false
+
+# Bulk operations
+users = User.multiget("alice@example.com", "bob@example.com")
+
+# With relationships feature - indexed lookups
+class User < Familia::Horreum
+  feature :relationships
+  field :email, :username
+  class_indexed_by :username, :username_lookup
+end
+
+# O(1) indexed finding
+user = User.find_by_username("alice_doe")   # Fast hash lookup
+```
+
+---
 
 ## Usage Examples
 
 ### Creating and Saving Objects
 
 ```ruby
-flower = Flower.create(name: "Red Rose", token: "rrose")
+flower = Flower.create(name: "Red Rose", token: "prose")
 flower.owners.push("Alice", "Bob")
 flower.tags.add("romantic")
 flower.metrics.increment("views", 1)
@@ -83,7 +252,7 @@ flower.save
 ### Retrieving and Updating Objects
 
 ```ruby
-rose = Flower.find_by_id("rrose")
+rose = Flower.load("prose")
 rose.name = "Pink Rose"
 rose.save
 ```
@@ -91,9 +260,9 @@ rose.save
 ### Using Safe Dump
 
 ```ruby
-user = User.create(username: "rosedog", first_name: "Rose", last_name: "Dog")
+user = User.create(username: "prosedog", first_name: "Rose", last_name: "Dog")
 user.safe_dump
-# => {id: "user:rosedog", username: "rosedog", full_name: "Rose Dog"}
+# => {id: "user:prosedog", username: "prosedog", full_name: "Rose Dog"}
 ```
 
 ### Working with Time-based Data
@@ -106,7 +275,7 @@ metric.counter.increment  # Increments the counter for the current hour
 ### Bulk Operations
 
 ```ruby
-Flower.multiget("rrose", "tulip", "daisy")
+Flower.multiget("prose", "tulip", "daisy")
 ```
 
 ### Transactional Operations
@@ -114,113 +283,125 @@ Flower.multiget("rrose", "tulip", "daisy")
 ```ruby
 user.transaction do |conn|
   conn.set("user:#{user.id}:status", "active")
-  conn.zadd("active_users", Time.now.to_i, user.id)
+  conn.zadd("active_users", Familia.now.to_i, user.id)
 end
 ```
 
-### Object Relationships
-
-Familia includes a powerful relationships system for managing object associations:
+### Advanced Patterns
 
+**Time-based Expiration:**
 ```ruby
-class Customer < Familia::Horreum
-  feature :relationships
-
-  identifier_field :custid
-  field :custid, :name, :email
-  set :domains  # Collection for related objects
+class Session < Familia::Horreum
+  feature :expiration
+  default_expiration 24.hours
 
-  # Automatic indexing and tracking
-  class_indexed_by :email, :email_lookup
-  class_tracked_in :all_customers, score: :created_at
+  field :user_id, :token
 end
 
-class Domain < Familia::Horreum
-  feature :relationships
+session = Session.create(user_id: "123", token: "abc123")
+session.ttl                                 # Check remaining time
+session.expire_in(1.hour)                  # Custom expiration
+```
 
-  identifier_field :domain_id
-  field :domain_id, :name, :status
+**Encrypted Fields:**
+```ruby
+class SecureData < Familia::Horreum
+  feature :encrypted_fields
 
-  # Bidirectional membership
-  member_of Customer, :domains
+  field :name
+  encrypted_field :credit_card, :ssn
 end
 
-# Clean, Ruby-like syntax
-customer = Customer.new(custid: "cust123", email: "admin@acme.com")
-customer.save  # Automatically indexed and tracked
-
-domain = Domain.new(domain_id: "dom456", name: "acme.com")
-customer.domains << domain  # Clean collection syntax
-
-# Fast O(1) lookups
-found_customer = Customer.find_by_email("admin@acme.com")
+data = SecureData.create(name: "Alice", credit_card: "4111-1111-1111-1234")
+data.credit_card.reveal                     # => "4111-1111-1111-1234"
+data.credit_card.to_s                       # => "[CONCEALED]"
 ```
 
-## Advanced Features
+## Configuration
 
-### Relationships and Associations
+### Basic Setup
 
-Familia provides three types of relationships with automatic management:
+```ruby
+# config/initializers/familia.rb (Rails)
+require 'familia'
 
-- **`member_of`** - Bidirectional membership with clean `<<` operator support
-- **`indexed_by`** - O(1) hash-based field lookups (class-level or relationship-scoped)
-- **`tracked_in`** - Scored collections for rankings, time-series, and analytics
+# Basic configuration
+Familia.uri = 'redis://localhost:6379/0'
 
-All relationships support automatic indexing and tracking - objects are automatically added to class-level collections when saved, with no manual management required.
+# Production configuration
+Familia.configure do |config|
+  config.redis_uri = ENV['REDIS_URL']
+  config.debug = ENV['FAMILIA_DEBUG'] == 'true'
+end
+```
 
-## Organizing Complex Models
+### Connection Pooling
 
-For large applications, you can organize model complexity using custom features and the Feature Autoloading System:
+```ruby
+require 'connection_pool'
 
-### Feature Autoloading System
+Familia.connection_provider = lambda do |uri|
+  ConnectionPool.new(size: 10, timeout: 5) do
+    Redis.new(url: uri)
+  end.with { |conn| yield conn if block_given?; conn }
+end
+```
 
-Familia automatically discovers and loads feature-specific configuration files, enabling clean separation between core model definitions and feature configurations:
+### Encryption Setup
 
 ```ruby
-# app/models/user.rb - Clean model definition
-class User < Familia::Horreum
-  field :name, :email, :password
-  feature :safe_dump  # Configuration auto-loaded
-end
-
-# app/models/user/safe_dump_extensions.rb - Automatically discovered
-class User
-  safe_dump_fields :name, :email  # password excluded for security
+Familia.configure do |config|
+  config.encryption_keys = {
+    v1: ENV['FAMILIA_ENCRYPTION_KEY_V1'],
+    v2: ENV['FAMILIA_ENCRYPTION_KEY_V2']
+  }
+  config.current_key_version = :v2
 end
 ```
 
-Extension files follow the pattern: `{model_name}/{feature_name}_*.rb`
+## Organizing Complex Models
 
-### Self-Registering Features
+For large applications, you can organize model complexity using custom features and the Feature Autoloading System:
 
-```ruby
-# app/features/customer_management.rb
-module MyApp::Features::CustomerManagement
-  Familia::Base.add_feature(self, :customer_management)
+### Feature Organization with Autoloader
 
-  def self.included(base)
-    base.extend(ClassMethods)
-  end
+For large applications, organize features into modular files using the autoloader:
 
-  module ClassMethods
-    def create_with_validation(attrs)
-      # Complex creation logic
-    end
+```ruby
+# app/models/customer.rb - Main model file
+class Customer < Familia::Horreum
+  module Features
+    include Familia::Features::Autoloader
+    # Automatically loads all .rb files from app/models/customer/features/
   end
 
-  def complex_business_method
-    # Instance methods
+  identifier_field :custid
+  field :custid, :name, :email
+  feature :safe_dump                        # Feature configuration loaded automatically
+end
+
+# app/models/customer/features/notifications.rb - Automatically loaded
+module Customer::Features::Notifications
+  def send_welcome_email
+    NotificationService.send_template(
+      email: email,
+      template: 'customer_welcome',
+      variables: { name: name, custid: custid }
+    )
   end
 end
 
-# models/customer.rb
-class Customer < Familia::Horreum
-  field :email, :name
-  feature :customer_management  # Clean model definition
+# app/models/customer/features/safe_dump_extensions.rb - Feature-specific config
+module Customer::Features::SafeDumpExtensions
+  def self.included(base)
+    base.safe_dump_field :custid
+    base.safe_dump_field :name
+    base.safe_dump_field :email
+  end
 end
 ```
 
-These approaches keep complex models organized while maintaining Familia's clean, declarative style. For detailed migration information, see the [migration guides](docs/migrating/).
+This approach keeps complex models organized while maintaining clean, declarative style.
 
 ## AI Development Assistance
 
@@ -234,10 +415,29 @@ This version of Familia was developed with assistance from AI tools. The followi
 
 I remain responsible for all design decisions and the final code. I believe in being transparent about development tools, especially as AI becomes more integrated into our workflows as developers.
 
-## Epilogue
+## Links
 
-For more information, visit:
 - [Github Repository](https://github.com/delano/familia)
 - [RubyGems Page](https://rubygems.org/gems/familia)
 
-Contributions are welcome! Feel free to submit a Pull Request.
+## Documentation
+
+For comprehensive guides and detailed technical information:
+
+- **[Overview Guide](docs/overview.md)** - Conceptual understanding and getting started
+- **[Technical Reference](docs/reference/api-technical.md)** - Implementation details and advanced patterns
+- **[Migration Guides](docs/migrating/)** - Upgrading from previous versions
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

data/changelog.d/README.md

@@ -53,8 +53,8 @@ git commit
 - **One Fragment Per Change:** Keep each fragment focused on a single feature, fix, or improvement.
 - **Documenting AI Assistance:** If a change involved significant AI assistance, place it in its own fragment. This ensures the `### AI Assistance` section clearly corresponds to the single change described in that fragment.
 - **Write for a Human Audience:** Describe the *impact* of the change, not just the implementation details.
-    - **Good:** "Improved the performance and stability of Redis connections under high load."
-    - **Bad:** "Refactored the `RedisManager`."
+    - **Good:** "Improved the performance and stability of Database connections under high load."
+    - **Bad:** "Refactored the `DatabaseManager`."
 - **Be Specific:** Avoid generic messages like "fixed a bug." Clearly state what was fixed.
 - **Include Context:** Reference issue or pull request numbers to provide a link to the discussion and implementation details. `scriv` will automatically create links for them.
     - **Example:** `- Fixed a bug where users could not reset their passwords. PR #123`

data/docs/archive/FAMILIA_RELATIONSHIPS.md

@@ -16,11 +16,11 @@ Generated methods on Domain instances:
 
 The method names follow the pattern: {action}_to_{lowercase_class_name}_{collection_name}
 
-tracked_in Relationships
+participates_in Relationships
 
 When you declare:
 class Customer < Familia::Horreum
-  tracked_in :all_customers, type: :sorted_set, score: :created_at
+  participates_in :all_customers, type: :sorted_set, score: :created_at
 end
 
 Generated class methods:
@@ -29,18 +29,18 @@ Generated class methods:
 - Customer.all_customers - Access the sorted set collection directly
 
 For scoped tracking (with class prefix):
-tracked_in :global, :active_users, score: :last_seen
+participates_in :global, :active_users, score: :last_seen
 Generates: Customer.add_to_active_users(customer) and Customer.active_users
 
 indexed_by Relationships
 
-The `indexed_by` method creates Redis hash-based indexes for O(1) field lookups. The `context` parameter determines index ownership and scope.
+The `indexed_by` method creates Valkey/Redis hash-based indexes for O(1) field lookups. The `context` parameter determines index ownership and scope.
 
 **Global Context (Shared Index)**
 When you declare:
 ```ruby
 class Customer < Familia::Horreum
-  indexed_by :email, :email_lookup, context: :global
+  indexed_by :email, :email_lookup, target: :global
 end
 ```
 
@@ -49,13 +49,13 @@ Generated class methods:
 - Customer.remove_from_email_lookup(customer) - Remove customer from global email index
 - Customer.email_lookup - Access the global hash index directly (supports .get(email))
 
-Redis key pattern: `global:email_lookup`
+Valkey/Redis key pattern: `global:email_lookup`
 
 **Class Context (Per-Instance Index)**
 When you declare:
 ```ruby
 class Domain < Familia::Horreum
-  indexed_by :name, :domain_index, context: Customer
+  indexed_by :name, :domain_index, target: Customer
 end
 ```
 
@@ -63,7 +63,7 @@ Generated class methods on Customer:
 - Customer.find_by_name(domain_name) - Find domain by name within this customer
 - Customer.find_all_by_name(domain_names) - Find multiple domains by names
 
-Redis key pattern: `customer:123:domain_index` (per customer instance)
+Valkey/Redis key pattern: `customer:123:domain_index` (per customer instance)
 
 **When to Use Each Context**
 - **Global context (`:global`)**: Use for system-wide lookups where the field value should be unique across all instances
@@ -85,7 +85,7 @@ domain.add_to_customer_domains(customer.custid)     # Add to relationship
 domain.remove_from_customer_domains(customer.custid) # Remove from relationship
 domain.in_customer_domains?(customer.custid)       # Query membership
 
-# For tracked_in relationships:
+# For participates_in relationships:
 Customer.add_to_all_customers(customer)    # Class method
 Customer.all_customers.range(0, -1)        # Direct collection access
 
@@ -97,10 +97,10 @@ Method Naming Conventions
 
 The relationship system uses consistent naming patterns:
 - member_of: {add_to|remove_from|in}_#{parent_class.downcase}_#{collection_name}
-- tracked_in: {add_to|remove_from}_#{collection_name} (class methods)
+- participates_in: {add_to|remove_from}_#{collection_name} (class methods)
 - indexed_by: {add_to|remove_from}_#{index_name} (class methods)
 
-This automatic method generation creates a clean, predictable API that handles both the Redis operations and maintains referential consistency
+This automatic method generation creates a clean, predictable API that handles both the db operations and maintains referential consistency
 across related objects.
 
 
@@ -119,8 +119,8 @@ class User < Familia::Horreum
   field :user_id, :email, :username
 
   # System-wide unique email lookup
-  indexed_by :email, :email_lookup, context: :global
-  indexed_by :username, :username_lookup, context: :global
+  indexed_by :email, :email_lookup, target: :global
+  indexed_by :username, :username_lookup, target: :global
 end
 
 # Usage:
@@ -128,7 +128,7 @@ User.add_to_email_lookup(user)
 found_user_id = User.email_lookup.get("john@example.com")
 ```
 
-**Redis keys generated**: `global:email_lookup`, `global:username_lookup`
+**Valkey/Redis keys generated**: `global:email_lookup`, `global:username_lookup`
 
 ### Class Context Pattern
 Use `context: SomeClass` when field values are unique within a specific parent context:
@@ -149,8 +149,8 @@ class Domain < Familia::Horreum
   field :domain_id, :name, :subdomain
 
   # Domains are unique per customer (customer can't have duplicate domain names)
-  indexed_by :name, :domain_index, context: Customer
-  indexed_by :subdomain, :subdomain_index, context: Customer
+  indexed_by :name, :domain_index, target: Customer
+  indexed_by :subdomain, :subdomain_index, target: Customer
 end
 
 # Usage:
@@ -159,7 +159,7 @@ customer.find_by_name("example.com")           # Find domain within this custome
 customer.find_all_by_subdomain(["www", "api"]) # Find multiple subdomains
 ```
 
-**Redis keys generated**: `customer:cust_123:domain_index`, `customer:cust_123:subdomain_index`
+**Valkey/Redis keys generated**: `customer:cust_123:domain_index`, `customer:cust_123:subdomain_index`
 
 ### Mixed Pattern Example
 A real-world example showing both patterns:
@@ -172,11 +172,11 @@ class ApiKey < Familia::Horreum
   field :key_id, :key_hash, :name, :scope
 
   # API key hashes must be globally unique
-  indexed_by :key_hash, :global_key_lookup, context: :global
+  indexed_by :key_hash, :global_key_lookup, target: :global
 
   # But key names can be reused across different customers
-  indexed_by :name, :customer_key_lookup, context: Customer
-  indexed_by :scope, :scope_lookup, context: Customer
+  indexed_by :name, :customer_key_lookup, target: Customer
+  indexed_by :scope, :scope_lookup, target: Customer
 end
 
 # Usage examples:
@@ -197,10 +197,10 @@ If you have existing code with incorrect syntax, here's how to fix it:
 indexed_by :email_lookup, field: :email
 
 # ✅ New correct syntax - Global scope
-indexed_by :email, :email_lookup, context: :global
+indexed_by :email, :email_lookup, target: :global
 
 # ✅ New correct syntax - Class scope
-indexed_by :email, :customer_email_lookup, context: Customer
+indexed_by :email, :customer_email_lookup, target: Customer
 ```
 
 **Key Differences**: