synthra 0.1.0

data/docs/QUALITY_METRICS.md

@@ -0,0 +1,555 @@
+# Data Quality Metrics
+
+FakeDataDSL's Quality Metrics feature analyzes your generated data for **realism, uniqueness, distribution, and completeness**. Get actionable insights into how realistic and diverse your test data is.
+
+## Quick Start
+
+```ruby
+# Generate data with quality report
+result = FakeDataDSL.generate_with_quality("User", count: 1000)
+
+# Access the data
+result.data  # => Array of 1000 user records
+
+# Access the quality report
+result.quality_report
+# => {
+#   uniqueness: { id: 100%, email: 100%, name: 85% },
+#   distribution: { role: { admin: 5%, user: 95% } },
+#   completeness: 98.5%,
+#   realism_score: 94
+# }
+```
+
+## Basic Usage
+
+### Generate with Quality Analysis
+
+```ruby
+result = FakeDataDSL::QualityMetrics.generate_with_quality("Order",
+  count: 500,
+  seed: 42
+)
+
+puts result.quality_report.summary
+# Uniqueness:   98.2%
+# Completeness: 96.5%
+# Distribution: balanced
+# Realism:      92/100
+```
+
+### Analyze Existing Data
+
+```ruby
+# Analyze data you've already generated
+data = FakeDataDSL.generate_many("Product", 1000)
+report = FakeDataDSL::QualityMetrics.analyze(data, schema: "Product")
+
+report.issues  # => Array of quality issues
+report.suggestions  # => How to improve data quality
+```
+
+## Quality Metrics
+
+### Uniqueness
+
+Measures how unique values are across records:
+
+```ruby
+report.uniqueness
+# => {
+#   id: 100.0,           # 100% unique (good for IDs)
+#   email: 100.0,        # 100% unique (as expected)
+#   name: 85.3,          # 85% unique (some duplicates)
+#   country: 12.5,       # 12.5% unique (expected for enums)
+#   status: 3.3          # 3.3% unique (only 3 possible values)
+# }
+
+# Check specific field
+report.uniqueness_for(:email)  # => 100.0
+
+# Fields that should be unique
+report.uniqueness_violations
+# => [:name]  # name has @unique annotation but has duplicates
+```
+
+### Distribution
+
+Analyzes value distribution for enum-like fields:
+
+```ruby
+report.distribution
+# => {
+#   role: {
+#     "user" => 0.85,      # 85% are regular users
+#     "admin" => 0.10,     # 10% are admins
+#     "moderator" => 0.05  # 5% are moderators
+#   },
+#   status: {
+#     "active" => 0.70,
+#     "pending" => 0.20,
+#     "suspended" => 0.10
+#   }
+# }
+
+# Check if distribution matches expected
+report.distribution_balanced?(:role)  # => true/false
+
+# Get distribution statistics
+report.distribution_stats(:role)
+# => {
+#   entropy: 0.85,         # Information entropy (higher = more diverse)
+#   chi_squared: 2.34,     # Chi-squared statistic
+#   is_uniform: false,     # Whether uniformly distributed
+#   dominant_value: "user" # Most common value
+# }
+```
+
+### Completeness
+
+Measures how many optional fields are populated:
+
+```ruby
+report.completeness
+# => 98.5  # 98.5% of optional fields have values
+
+report.completeness_by_field
+# => {
+#   bio: 95.0,           # 95% of records have bio
+#   avatar_url: 88.2,    # 88.2% have avatar
+#   phone: 72.0,         # 72% have phone
+#   middle_name: 45.5    # 45.5% have middle name
+# }
+
+# Fields with low completeness
+report.sparse_fields(threshold: 50)
+# => [:middle_name]  # Less than 50% populated
+```
+
+### Realism Score
+
+Overall assessment of how realistic the data looks:
+
+```ruby
+report.realism_score  # => 94 (out of 100)
+
+report.realism_breakdown
+# => {
+#   email_format: 100,      # Emails look real
+#   name_format: 95,        # Names are realistic
+#   phone_format: 90,       # Phones follow patterns
+#   date_consistency: 98,   # Dates are logically consistent
+#   numeric_ranges: 85,     # Numbers in expected ranges
+#   text_quality: 92        # Text is coherent
+# }
+
+# Specific realism checks
+report.realism_issues
+# => [
+#   { field: :age, issue: "5% of values are negative" },
+#   { field: :price, issue: "Values exceed expected range" }
+# ]
+```
+
+## Quality Thresholds
+
+### Define Quality Standards
+
+```ruby
+FakeDataDSL::QualityMetrics.configure do |config|
+  config.thresholds = {
+    uniqueness: {
+      id: 100,        # Must be 100% unique
+      email: 100,     # Must be 100% unique
+      name: 80        # At least 80% unique
+    },
+    completeness: {
+      minimum: 90,    # At least 90% of optional fields filled
+      required: 100   # Required fields must always be present
+    },
+    realism: {
+      minimum: 85     # Minimum realism score
+    }
+  }
+end
+```
+
+### Validate Against Thresholds
+
+```ruby
+result = FakeDataDSL.generate_with_quality("User", count: 1000)
+
+if result.quality_report.meets_thresholds?
+  puts "Data quality is acceptable"
+else
+  puts "Quality issues:"
+  result.quality_report.threshold_violations.each do |violation|
+    puts "  - #{violation[:field]}: #{violation[:message]}"
+  end
+end
+```
+
+### CI Integration
+
+```ruby
+# In your test suite
+RSpec.describe "Data Quality" do
+  it "generates high-quality user data" do
+    result = FakeDataDSL.generate_with_quality("User", count: 1000)
+    
+    expect(result.quality_report.realism_score).to be >= 90
+    expect(result.quality_report.uniqueness[:email]).to eq(100)
+    expect(result.quality_report.completeness).to be >= 95
+  end
+end
+```
+
+## Detailed Analysis
+
+### Field-Level Analysis
+
+```ruby
+report = FakeDataDSL::QualityMetrics.analyze(data, schema: "User")
+
+# Analyze specific field
+field_report = report.field_analysis(:email)
+# => {
+#   type: "email",
+#   total_values: 1000,
+#   unique_values: 1000,
+#   uniqueness: 100.0,
+#   null_count: 0,
+#   format_valid: 1000,
+#   format_invalid: 0,
+#   sample_values: ["john@example.com", "jane@test.org", ...],
+#   patterns: {
+#     "gmail.com" => 230,
+#     "yahoo.com" => 180,
+#     "example.com" => 590
+#   }
+# }
+```
+
+### Correlation Analysis
+
+```ruby
+# Find correlations between fields
+report.correlations
+# => {
+#   [:age, :retirement_status] => 0.95,  # Strong correlation
+#   [:country, :phone_prefix] => 0.88,   # Expected correlation
+#   [:name, :email] => 0.02              # No correlation (good)
+# }
+
+# Check for unexpected correlations
+report.suspicious_correlations
+# => [
+#   { fields: [:id, :created_at], correlation: 0.99, 
+#     message: "ID and created_at are highly correlated - may indicate sequential generation" }
+# ]
+```
+
+### Outlier Detection
+
+```ruby
+# Find outliers in numeric fields
+report.outliers
+# => {
+#   age: {
+#     outliers: [150, -5, 999],
+#     outlier_percentage: 0.3,
+#     expected_range: 0..120
+#   },
+#   price: {
+#     outliers: [0.001, 999999.99],
+#     outlier_percentage: 0.1,
+#     expected_range: 0.01..10000
+#   }
+# }
+```
+
+## Quality Reports
+
+### Summary Report
+
+```ruby
+report = FakeDataDSL.generate_with_quality("User", count: 1000).quality_report
+
+puts report.summary
+# ═══════════════════════════════════════════════
+# Data Quality Report: User (1000 records)
+# ═══════════════════════════════════════════════
+# 
+# Overall Score: 94/100 ✓
+# 
+# Uniqueness:    98.2% ✓
+#   - id:        100%  ✓
+#   - email:     100%  ✓
+#   - name:      85.3% ⚠ (expected: 80%)
+# 
+# Completeness:  96.5% ✓
+#   - bio:       95.0%
+#   - avatar:    88.2%
+# 
+# Distribution:  Balanced ✓
+#   - role:      { user: 85%, admin: 10%, mod: 5% }
+# 
+# Realism:       92/100 ✓
+#   - 2 minor issues detected
+# ═══════════════════════════════════════════════
+```
+
+### JSON Report
+
+```ruby
+report.to_json
+# => {
+#   "schema": "User",
+#   "record_count": 1000,
+#   "generated_at": "2026-01-24T10:30:00Z",
+#   "overall_score": 94,
+#   "metrics": {
+#     "uniqueness": { ... },
+#     "completeness": { ... },
+#     "distribution": { ... },
+#     "realism": { ... }
+#   },
+#   "issues": [ ... ],
+#   "suggestions": [ ... ]
+# }
+```
+
+### HTML Report
+
+```ruby
+# Generate visual HTML report
+report.to_html("quality_report.html")
+
+# Or get HTML string
+html = report.to_html_string
+```
+
+### Export for Monitoring
+
+```ruby
+# Export metrics for monitoring systems
+report.to_prometheus
+# => [
+#   'fake_data_quality_uniqueness{schema="User",field="email"} 100.0',
+#   'fake_data_quality_completeness{schema="User"} 96.5',
+#   ...
+# ]
+
+report.to_datadog
+# => { metrics: [...], tags: [...] }
+```
+
+## Quality Improvement Suggestions
+
+```ruby
+report.suggestions
+# => [
+#   {
+#     field: :name,
+#     issue: "Low uniqueness (85%)",
+#     suggestion: "Consider using more diverse name patterns or adding middle names",
+#     priority: :medium
+#   },
+#   {
+#     field: :phone,
+#     issue: "28% null values",
+#     suggestion: "Increase phone generation probability or make required",
+#     priority: :low
+#   }
+# ]
+
+# Auto-apply suggestions
+improved_schema = FakeDataDSL::QualityMetrics.improve_schema(
+  FakeDataDSL.schema("User"),
+  based_on: report
+)
+```
+
+## Comparison Reports
+
+### Compare Generations
+
+```ruby
+# Compare quality across different generations
+report1 = FakeDataDSL.generate_with_quality("User", count: 1000, seed: 1)
+report2 = FakeDataDSL.generate_with_quality("User", count: 1000, seed: 2)
+
+comparison = FakeDataDSL::QualityMetrics.compare(
+  report1.quality_report,
+  report2.quality_report
+)
+
+comparison.differences
+# => {
+#   uniqueness: { name: [-2.3, "seed 1 had higher name uniqueness"] },
+#   realism: [+3, "seed 2 produced more realistic data"]
+# }
+```
+
+### Compare Against Baseline
+
+```ruby
+# Save a baseline
+baseline = FakeDataDSL.generate_with_quality("User", count: 1000)
+baseline.quality_report.save_as_baseline("user_baseline")
+
+# Later, compare against baseline
+current = FakeDataDSL.generate_with_quality("User", count: 1000)
+diff = current.quality_report.compare_to_baseline("user_baseline")
+
+diff.regressions
+# => [{ metric: :uniqueness, field: :email, change: -5.0 }]
+```
+
+## API Reference
+
+### QualityMetrics.generate_with_quality
+
+```ruby
+FakeDataDSL::QualityMetrics.generate_with_quality(schema_name, options = {})
+```
+
+**Parameters:**
+- `schema_name` - Name of the schema
+- `options[:count]` - Number of records (default: 100)
+- `options[:seed]` - Random seed
+- `options[:mode]` - Generation mode
+- `options[:analyze_fields]` - Specific fields to analyze (default: all)
+
+**Returns:** `QualityResult` with `data` and `quality_report`
+
+### QualityMetrics.analyze
+
+```ruby
+FakeDataDSL::QualityMetrics.analyze(data, options = {})
+```
+
+**Parameters:**
+- `data` - Array of records to analyze
+- `options[:schema]` - Schema name for context
+- `options[:thresholds]` - Quality thresholds to validate against
+
+**Returns:** `QualityReport`
+
+### QualityReport Methods
+
+```ruby
+report.uniqueness           # Hash of field => uniqueness percentage
+report.completeness         # Overall completeness percentage
+report.distribution         # Distribution analysis for enum fields
+report.realism_score        # Overall realism score (0-100)
+report.issues               # Array of detected issues
+report.suggestions          # Array of improvement suggestions
+report.meets_thresholds?    # Boolean - passes all thresholds?
+report.summary              # Human-readable summary string
+report.to_json              # JSON representation
+report.to_html(path)        # Generate HTML report file
+```
+
+## Configuration
+
+```ruby
+FakeDataDSL::QualityMetrics.configure do |config|
+  # Default thresholds
+  config.default_thresholds = {
+    uniqueness: { default: 80, id: 100, email: 100 },
+    completeness: { minimum: 90 },
+    realism: { minimum: 85 }
+  }
+  
+  # Analysis options
+  config.analyze_distributions = true
+  config.detect_outliers = true
+  config.correlation_threshold = 0.8
+  
+  # Sampling for large datasets
+  config.max_sample_size = 10000
+  config.sample_strategy = :random  # :random, :stratified
+  
+  # Report options
+  config.include_sample_values = true
+  config.max_sample_values = 5
+end
+```
+
+## Best Practices
+
+### 1. Set Appropriate Thresholds
+
+```ruby
+# Strict for critical fields
+config.thresholds[:uniqueness][:id] = 100
+config.thresholds[:uniqueness][:email] = 100
+
+# Relaxed for non-unique fields
+config.thresholds[:uniqueness][:name] = 70
+config.thresholds[:uniqueness][:city] = 10
+```
+
+### 2. Monitor Quality Over Time
+
+```ruby
+# Track quality metrics in CI
+after(:suite) do
+  report = aggregate_quality_reports
+  QualityMetricsDashboard.record(report)
+end
+```
+
+### 3. Use Quality Gates
+
+```ruby
+# Fail CI if quality drops
+RSpec.configure do |config|
+  config.after(:suite) do
+    report = FakeDataDSL::QualityMetrics.session_report
+    if report.realism_score < 85
+      raise "Data quality below threshold: #{report.realism_score}"
+    end
+  end
+end
+```
+
+## Troubleshooting
+
+### Low Uniqueness
+
+```ruby
+# Issue: email uniqueness is only 95%
+# Solution: Increase domain variety
+override "User", email: -> { Faker::Internet.email(domain: random_domain) }
+
+# Or use @unique annotation
+# User:
+#   email: email @unique
+```
+
+### Poor Distribution
+
+```ruby
+# Issue: role distribution is skewed (99% user, 1% admin)
+# Solution: Use weighted enum
+# role: enum(user:70%, admin:20%, moderator:10%)
+```
+
+### Low Realism
+
+```ruby
+# Issue: phone numbers don't look realistic
+# Solution: Use locale-specific formats
+FakeDataDSL.configure do |c|
+  c.locale = :en_US  # US phone format
+end
+```
+
+## See Also
+
+- [Generation Modes](generation_modes.md)
+- [Behaviors](behaviors.md)
+- [Testing Best Practices](best_practices.md)
+- [Snapshot Testing](SNAPSHOT_TESTING.md)

data/docs/QUICK_REFERENCE.md

@@ -0,0 +1,88 @@
+# Quick Reference
+
+Quick reference card for FakeDataDSL.
+
+## Basic Syntax
+
+```ruby
+SchemaName:
+  field_name: type
+  field_name: type(args)
+  field_name: type?          # Nullable
+  field_name: type optional   # Optional
+  field_name: type unique     # Unique
+  field_name: type if condition  # Conditional
+```
+
+## Common Types
+
+| Type | Example | Description |
+|------|---------|-------------|
+| `uuid` | `id: uuid` | UUID identifier |
+| `name` | `name: name` | Person name |
+| `email` | `email: email` | Email address |
+| `number` | `age: number(18..65)` | Number in range |
+| `boolean` | `active: boolean` | True/false |
+| `text` | `bio: text(100..500)` | Text with length |
+| `date` | `birth_date: date` | Calendar date |
+| `array` | `tags: array(text, 3..10)` | Array of values |
+| `enum` | `status: enum(active, inactive)` | Enumeration |
+
+## Behaviors
+
+```ruby
+@latency 100ms        # Add delay
+@failure 5%           # Failure probability
+@partial_data 20%     # Omit fields sometimes
+@randomize_order      # Randomize field order
+```
+
+## Generation Modes
+
+```ruby
+schema.generate(mode: :random)   # Default: realistic data
+schema.generate(mode: :edge)     # Edge cases
+schema.generate(mode: :invalid)  # Invalid data
+schema.generate(mode: :mixed)    # Mix of all
+```
+
+## Code Examples
+
+### Basic Usage
+
+```ruby
+require 'fake_data_dsl'
+
+registry = FakeDataDSL::Registry.new
+registry.load_file('schema.dsl')
+schema = registry.schema('User')
+record = schema.generate
+```
+
+### Multiple Records
+
+```ruby
+records = schema.generate_many(10)
+```
+
+### Streaming
+
+```ruby
+schema.generate_many_stream(1000).each do |record|
+  process(record)
+end
+```
+
+### Deterministic
+
+```ruby
+FakeDataDSL.configure { |c| c.seed = 12345 }
+record = schema.generate  # Same data every time
+```
+
+## See Also
+
+- [Getting Started](getting_started.md) - Full guide
+- [DSL Reference](dsl_reference.md) - Complete syntax
+- [Type Reference](type_reference.md) - All types
+