synthra 0.1.0

data/tech_docs/advanced/thread_safety.md

@@ -0,0 +1,253 @@
+# Thread Safety Guide
+
+FakeDataDSL is designed for safe use in multi-threaded applications. This guide explains what is and isn't thread-safe.
+
+## Quick Reference
+
+| Operation | Thread-Safe? | Notes |
+|-----------|--------------|-------|
+| `schema.generate()` | ✅ Yes | Each call is independent |
+| `schema.generate_many()` | ✅ Yes | Batch is atomic |
+| `schema.generate_stream()` | ✅ Yes | Iterator is independent |
+| `registry.load_file()` | ✅ Yes | Mutex-protected |
+| `registry.schema()` | ✅ Yes | Mutex-protected |
+| `shared()` within batch | ✅ Yes | Context is per-batch |
+| `shared()` across batches | ⚠️ N/A | Each batch has own context |
+| Custom functions | ⚠️ Depends | Your code must be thread-safe |
+| `FakeDataDSL.configure` | ⚠️ No | Configure before multi-threading |
+
+---
+
+## Safe Patterns
+
+### Pattern 1: Concurrent Generation with Same Schema
+
+```ruby
+schema = registry.schema("User")
+
+# Safe: Each thread generates independently
+threads = 10.times.map do
+  Thread.new do
+    100.times { schema.generate(registry: registry) }
+  end
+end
+
+threads.each(&:join)
+```
+
+### Pattern 2: Different Seeds per Thread
+
+```ruby
+schema = registry.schema("User")
+
+# Safe: Different seeds ensure different outputs
+threads = 10.times.map do |i|
+  Thread.new do
+    seed = i * 1000
+    schema.generate_many(100, seed: seed, registry: registry)
+  end
+end
+
+results = threads.map(&:value)
+```
+
+### Pattern 3: Shared Registry
+
+```ruby
+# Load schemas once
+registry = FakeDataDSL::Registry.new
+registry.load_dir("schemas/")
+
+# Safe: Registry is thread-safe for reads
+threads = 10.times.map do
+  Thread.new do
+    user_schema = registry.schema("User")
+    order_schema = registry.schema("Order")
+    
+    user_schema.generate(registry: registry)
+    order_schema.generate(registry: registry)
+  end
+end
+```
+
+---
+
+## What About `shared()`?
+
+The `shared()` type is designed for use within a **single batch** generated by `generate_many()`:
+
+```ruby
+# Safe: shared() context is isolated to this batch
+webhooks = schema.generate_many(10, registry: registry)
+# All 10 webhooks have the same shared recipient
+```
+
+### Important: `shared()` Does NOT Share Across Batches
+
+```ruby
+# Each batch has its OWN shared context
+batch1 = schema.generate_many(10, registry: registry)  # Recipient A
+batch2 = schema.generate_many(10, registry: registry)  # Recipient B (different!)
+
+# This is correct behavior - batches are independent
+```
+
+### Concurrent Batches with `shared()`
+
+```ruby
+schema = registry.schema("TwitterDMWebhook")
+
+# Safe: Each batch has independent shared context
+threads = 5.times.map do
+  Thread.new do
+    schema.generate_many(100, registry: registry)
+  end
+end
+
+batches = threads.map(&:value)
+# Each batch has consistent shared values within itself
+```
+
+---
+
+## Unsafe Patterns
+
+### Anti-Pattern 1: Modifying Configuration During Generation
+
+```ruby
+# ❌ UNSAFE: Don't modify config while generating
+Thread.new do
+  1000.times { schema.generate }
+end
+
+Thread.new do
+  FakeDataDSL.configure { |c| c.default_mode = :edge }  # Race condition!
+end
+```
+
+**Fix**: Configure before spawning threads.
+
+```ruby
+# ✅ Safe: Configure first
+FakeDataDSL.configure { |c| c.default_mode = :edge }
+
+threads = 10.times.map { Thread.new { schema.generate } }
+```
+
+### Anti-Pattern 2: Sharing Mutable State in Custom Functions
+
+```ruby
+# ❌ UNSAFE: Mutable shared state
+counter = 0
+
+FakeDataDSL.register_function(:next_id) do |context|
+  counter += 1  # Race condition!
+end
+```
+
+**Fix**: Use thread-local storage or atomic operations.
+
+```ruby
+# ✅ Safe: Thread-local counter
+FakeDataDSL.register_function(:next_id) do |context|
+  Thread.current[:counter] ||= 0
+  Thread.current[:counter] += 1
+end
+
+# ✅ Safe: Atomic counter
+require 'concurrent'
+counter = Concurrent::AtomicFixnum.new(0)
+
+FakeDataDSL.register_function(:next_id) do |context|
+  counter.increment
+end
+```
+
+### Anti-Pattern 3: Loading Schemas While Generating
+
+```ruby
+# ⚠️ Technically safe but not recommended
+Thread.new { registry.load_file("new_schema.dsl") }
+Thread.new { registry.schema("User").generate }
+```
+
+**Fix**: Load all schemas before starting generation.
+
+---
+
+## Determinism in Concurrent Scenarios
+
+### Same Seed = Same Output (Even in Threads)
+
+```ruby
+# Results are deterministic when using the same seed
+result1 = schema.generate(seed: 12345)
+result2 = schema.generate(seed: 12345)
+
+result1 == result2  # true, regardless of threading
+```
+
+### Batch Determinism
+
+```ruby
+# Same seed produces same batch
+batch1 = schema.generate_many(100, seed: 42)
+batch2 = schema.generate_many(100, seed: 42)
+
+batch1 == batch2  # true
+```
+
+---
+
+## Performance Considerations
+
+### RNG Isolation
+
+Each generation creates its own Random Number Generator (RNG) instance, so there's no contention on shared RNG state.
+
+### Registry Caching
+
+The Registry uses mutex-protected caching for parsed schemas:
+
+```ruby
+# First load: parses and caches
+registry.load_file("user.dsl")
+
+# Subsequent loads: uses cache (fast, thread-safe)
+registry.load_file("user.dsl")
+```
+
+### Parallel Batch Generation
+
+For maximum throughput, use parallel batches:
+
+```ruby
+require 'parallel'
+
+# Generate 1 million users across all CPU cores
+results = Parallel.map(1..100, in_processes: 8) do |batch_num|
+  seed = batch_num * 10000
+  schema.generate_many(10000, seed: seed, registry: registry)
+end
+
+all_users = results.flatten
+```
+
+---
+
+## Summary
+
+1. **Generation is thread-safe** - Call `generate()`, `generate_many()`, `generate_stream()` from any thread
+2. **Registry is thread-safe** - Share a single registry across threads
+3. **Configure once** - Set configuration before multi-threaded use
+4. **Custom functions must be thread-safe** - Avoid shared mutable state
+5. **`shared()` is batch-scoped** - Each `generate_many()` call has independent shared context
+
+---
+
+## Next Steps
+
+- [Streaming Generation](streaming.md) - Memory-efficient generation
+- [Custom Functions](../api/custom_functions.md) - Writing thread-safe custom functions
+- [Configuration](../api/configuration.md) - Global configuration options
+

data/tech_docs/api/overview.md

@@ -0,0 +1,485 @@
+# Ruby API Overview
+
+Complete reference for the FakeDataDSL Ruby API.
+
+## Table of Contents
+
+- [Quick Reference](#quick-reference)
+- [Main Module](#main-module)
+- [Core Classes](#core-classes)
+- [Loading Schemas](#loading-schemas)
+- [Generating Data](#generating-data)
+- [Configuration](#configuration)
+- [Custom Extensions](#custom-extensions)
+
+---
+
+## Quick Reference
+
+```ruby
+require 'fake_data_dsl'
+
+# Load and generate
+schema = FakeDataDSL.load('schema.dsl')
+data = schema.generate
+data = schema.generate(seed: 12345, mode: :edge)
+many = schema.generate_many(100)
+
+# Registry for multiple schemas
+registry = FakeDataDSL::Registry.new
+registry.load_dir('schemas/')
+user = registry.schema('User').generate(registry: registry)
+
+# Custom types and functions
+FakeDataDSL.register_type(:custom_type) { |rng, ctx, args, mode| "value" }
+FakeDataDSL.register_function(:compute) { |ctx| ctx['a'] + ctx['b'] }
+
+# Configuration
+FakeDataDSL.configure do |config|
+  config.default_mode = :random
+  config.limits.max_array_size = 1000
+end
+```
+
+---
+
+## Main Module
+
+### FakeDataDSL
+
+The main entry point for the gem.
+
+```ruby
+# Load schema from file
+schema = FakeDataDSL.load(path)
+
+# Parse schema from string
+schema = FakeDataDSL.parse(dsl_string)
+
+# Register custom type
+FakeDataDSL.register_type(name, &block)
+FakeDataDSL.register_type(name, class)
+
+# Register custom function
+FakeDataDSL.register_function(name, &block)
+
+# Register custom behavior
+FakeDataDSL.register_behavior(name, &block)
+
+# Configure globally
+FakeDataDSL.configure { |config| ... }
+
+# Reset configuration
+FakeDataDSL.reset_configuration!
+```
+
+---
+
+## Core Classes
+
+### Class Overview
+
+| Class | Purpose |
+|-------|---------|
+| `Schema` | Represents a parsed schema |
+| `Registry` | Manages multiple schemas |
+| `Generator::Engine` | Core generation logic |
+| `Generator::RNG` | Random number generation |
+| `Generator::Context` | Field generation context |
+| `Configuration` | Global settings |
+| `Limits` | Resource limits |
+
+### Schema
+
+```ruby
+# Load schema
+schema = FakeDataDSL.load('user.dsl')
+
+# Access properties
+schema.name          # => "User"
+schema.fields        # => [Field, Field, ...]
+schema.behaviors     # => [Behavior, ...]
+schema.metadata      # => { seed: nil, ... }
+
+# Generate data
+schema.generate
+schema.generate(seed: 123)
+schema.generate(mode: :edge)
+schema.generate_many(100)
+schema.generate_stream(count: 1000)
+schema.generate_json
+schema.generate_json(pretty: true)
+```
+
+### Registry
+
+```ruby
+# Create registry
+registry = FakeDataDSL::Registry.new
+
+# Load schemas
+registry.load_file('user.dsl')
+registry.load_dir('schemas/')
+registry.load_string(dsl_content, name: 'inline')
+
+# Access schemas
+schema = registry.schema('User')
+schemas = registry.schemas  # => { 'User' => Schema, ... }
+registry.schema?('User')    # => true
+
+# Generate with registry
+registry.schema('Order').generate(registry: registry)
+```
+
+### Generator::Engine
+
+```ruby
+# Usually not used directly - Schema wraps this
+
+engine = FakeDataDSL::Generator::Engine.new(
+  schema: schema,
+  registry: registry,
+  seed: 12345,
+  mode: :random
+)
+
+data = engine.generate
+```
+
+### Generator::RNG
+
+```ruby
+# Seedable random number generator
+rng = FakeDataDSL::Generator::RNG.new(seed: 12345)
+
+rng.int(1, 100)        # => 42
+rng.float(0.0, 1.0)    # => 0.7234
+rng.bool               # => true
+rng.sample([1, 2, 3])  # => 2
+rng.shuffle([1, 2, 3]) # => [3, 1, 2]
+rng.string(10)         # => "aBcDeFgHiJ"
+rng.hex(16)            # => "a1b2c3d4..."
+```
+
+---
+
+## Loading Schemas
+
+### From File
+
+```ruby
+# Load single file
+schema = FakeDataDSL.load('schemas/user.dsl')
+
+# File with multiple schemas returns first
+schema = FakeDataDSL.load('schemas/all.dsl')  # Returns first schema
+
+# Use registry for multiple schemas
+registry = FakeDataDSL::Registry.new
+registry.load_file('schemas/all.dsl')
+user = registry.schema('User')
+order = registry.schema('Order')
+```
+
+### From String
+
+```ruby
+dsl = <<~DSL
+  User:
+    id: uuid
+    name: name
+DSL
+
+schema = FakeDataDSL.parse(dsl)
+```
+
+### From Directory
+
+```ruby
+registry = FakeDataDSL::Registry.new
+registry.load_dir('schemas/')  # Loads all .dsl files
+
+# Recursive loading
+registry.load_dir('schemas/', recursive: true)
+```
+
+---
+
+## Generating Data
+
+### Single Record
+
+```ruby
+# Basic generation
+user = schema.generate
+
+# With options
+user = schema.generate(
+  seed: 12345,           # Reproducible output
+  mode: :random,         # Generation mode
+  registry: registry,    # For cross-references
+  overrides: {           # Override specific fields
+    'status' => 'active'
+  }
+)
+```
+
+### Multiple Records
+
+```ruby
+# Generate array
+users = schema.generate_many(100)
+
+# With options
+users = schema.generate_many(100,
+  seed: 12345,
+  mode: :edge
+)
+
+# Each record has different data
+users.map { |u| u['id'] }.uniq.size  # => 100
+```
+
+### Streaming
+
+```ruby
+# Memory-efficient for large batches
+schema.generate_stream(count: 1_000_000).each do |record|
+  database.insert(record)
+end
+
+# With options
+schema.generate_stream(
+  count: 1_000_000,
+  seed: 12345,
+  mode: :random
+).each do |record|
+  process(record)
+end
+```
+
+### JSON Output
+
+```ruby
+# Compact JSON
+json = schema.generate_json
+
+# Pretty printed
+json = schema.generate_json(pretty: true)
+
+# Multiple records
+json = schema.generate_many_json(100)
+json = schema.generate_many_json(100, pretty: true)
+```
+
+### Generation Modes
+
+```ruby
+# Random - typical realistic data (default)
+schema.generate(mode: :random)
+
+# Edge - boundary values
+schema.generate(mode: :edge)
+
+# Invalid - invalid data for validation testing
+schema.generate(mode: :invalid)
+
+# Mixed - combination of modes
+schema.generate(mode: :mixed)
+```
+
+---
+
+## Configuration
+
+### Global Configuration
+
+```ruby
+FakeDataDSL.configure do |config|
+  # Default generation mode
+  config.default_mode = :random
+  
+  # Maximum retries for unique values
+  config.max_unique_retries = 1000
+  
+  # Enable/disable behaviors
+  config.behaviors_enabled = true
+  
+  # Registry cache size
+  config.registry_cache_size = 100
+end
+```
+
+### Resource Limits
+
+```ruby
+FakeDataDSL.configure do |config|
+  # Maximum latency from @latency behavior
+  config.limits.max_latency_ms = 10_000
+  
+  # Maximum recursion depth
+  config.limits.max_recursion = 10
+  
+  # Maximum array size
+  config.limits.max_array_size = 1000
+  
+  # Maximum text length
+  config.limits.max_text_length = 10_000
+end
+```
+
+### Accessing Configuration
+
+```ruby
+config = FakeDataDSL.configuration
+
+config.default_mode          # => :random
+config.limits.max_array_size # => 1000
+```
+
+### Reset
+
+```ruby
+FakeDataDSL.reset_configuration!
+```
+
+---
+
+## Custom Extensions
+
+### Custom Types
+
+```ruby
+# Block-based
+FakeDataDSL.register_type(:ssn) do |rng, context, args, mode|
+  case mode
+  when :random
+    "#{rng.int(100, 999)}-#{rng.int(10, 99)}-#{rng.int(1000, 9999)}"
+  when :edge
+    "000-00-0000"
+  when :invalid
+    "invalid-ssn"
+  end
+end
+
+# Class-based
+class SSN < FakeDataDSL::Types::Base
+  def generate_random(rng, args)
+    "#{rng.int(100, 999)}-#{rng.int(10, 99)}-#{rng.int(1000, 9999)}"
+  end
+  
+  def generate_edge(rng, args)
+    "000-00-0000"
+  end
+  
+  def generate_invalid(rng, args)
+    "invalid-ssn"
+  end
+end
+
+FakeDataDSL.register_type(:ssn, SSN)
+```
+
+### Custom Functions
+
+```ruby
+# Simple computed field
+FakeDataDSL.register_function(:full_name) do |context|
+  "#{context['first_name']} #{context['last_name']}"
+end
+
+# Complex logic
+FakeDataDSL.register_function(:order_total) do |context|
+  items = context[:items] || []
+  subtotal = items.sum { |i| i['price'] * i['quantity'] }
+  tax = subtotal * 0.08
+  shipping = subtotal > 100 ? 0 : 9.99
+  (subtotal + tax + shipping).round(2)
+end
+```
+
+### Custom Behaviors
+
+```ruby
+# Block-based
+FakeDataDSL.register_behavior(:log) do |result, context, value|
+  puts "[DEBUG] Generated: #{result.inspect}"
+  result
+end
+
+# Class-based
+class AuditBehavior < FakeDataDSL::Behaviors::Base
+  def apply(result, context, value)
+    log_generation(result, context)
+    result
+  end
+  
+  private
+  
+  def log_generation(result, context)
+    # Custom logging logic
+  end
+end
+
+FakeDataDSL.register_behavior(:audit, AuditBehavior)
+```
+
+---
+
+## Error Handling
+
+### Error Types
+
+```ruby
+begin
+  data = schema.generate
+rescue FakeDataDSL::Errors::ParseError => e
+  # DSL syntax error
+  puts "Parse error: #{e.message}"
+  puts "Line #{e.line}, column #{e.column}"
+  
+rescue FakeDataDSL::Errors::UnknownTypeError => e
+  # Unknown type in schema
+  puts "Unknown type: #{e.type_name}"
+  
+rescue FakeDataDSL::Errors::UniquenessError => e
+  # Couldn't generate unique value
+  puts "Uniqueness failed for: #{e.field_name}"
+  
+rescue FakeDataDSL::Errors::CyclicReferenceError => e
+  # Circular schema reference
+  puts "Cycle detected: #{e.cycle_path}"
+  
+rescue FakeDataDSL::Errors::SimulatedFailure
+  # @failure behavior triggered
+  
+rescue FakeDataDSL::Errors::SimulatedError => e
+  # @simulate_error behavior triggered
+  puts "HTTP #{e.status_code}"
+end
+```
+
+### Error Codes
+
+```ruby
+# Each error has a code for programmatic handling
+rescue FakeDataDSL::Errors::ParseError => e
+  case e.class::ERROR_CODE
+  when "SYNTAX_ERROR"
+    # Handle syntax error
+  when "UNKNOWN_TYPE_ERROR"
+    # Handle unknown type
+  end
+end
+```
+
+---
+
+## Next Steps
+
+- [Schema Class](schema.md) - Detailed Schema API
+- [Registry Class](registry.md) - Multi-schema management
+- [Custom Types](custom_types.md) - Creating type generators
+- [Custom Functions](custom_functions.md) - Computed fields
+- [Error Handling](error_handling.md) - Complete error reference
+