activerecord-graph-extractor 0.1.0

data/docs/usage.md

@@ -0,0 +1,363 @@
+# Usage Guide
+
+This guide covers common usage patterns and best practices for the ActiveRecord Graph Extractor gem.
+
+## Getting Started
+
+### Installation
+
+Add the gem to your application's Gemfile:
+
+```ruby
+gem 'activerecord-graph-extractor'
+```
+
+Or install directly:
+
+```bash
+gem install activerecord-graph-extractor
+```
+
+### Basic Usage
+
+#### Extracting Data
+
+```ruby
+require 'activerecord_graph_extractor'
+
+# Find your root object
+order = Order.find(12345)
+
+# Create extractor with default configuration
+extractor = ActiveRecordGraphExtractor::Extractor.new
+
+# Extract to file
+result = extractor.extract_to_file(order, 'order_export.json')
+puts "Extracted #{result['metadata']['total_records']} records"
+```
+
+#### Importing Data
+
+```ruby
+# Create importer
+importer = ActiveRecordGraphExtractor::Importer.new
+
+# Import from file
+result = importer.import_from_file('order_export.json')
+puts "Imported #{result['imported_records']} records"
+```
+
+## Configuration
+
+### Global Configuration
+
+```ruby
+ActiveRecordGraphExtractor.configure do |config|
+  config.max_depth = 3                           # Limit relationship depth
+  config.include_relationship('user')            # Only include specific relationships
+  config.include_relationship('products')
+  config.exclude_model('History')                # Skip these models
+  config.exclude_model('AuditLog')
+  config.batch_size = 500                       # Process in batches
+  config.stream_json = true                     # Use streaming for large files
+end
+```
+
+### Per-Extraction Configuration
+
+```ruby
+extractor = ActiveRecordGraphExtractor::Extractor.new
+result = extractor.extract(order, {
+  max_depth: 3,
+  custom_serializers: {
+    'User' => ->(user) { { id: user.id, email: user.email } }
+  }
+})
+```
+
+## Progress Monitoring
+
+### Programmatic Progress Tracking
+
+```ruby
+ActiveRecordGraphExtractor.configure do |config|
+  config.progress_enabled = true
+end
+
+extractor = ActiveRecordGraphExtractor::Extractor.new
+result = extractor.extract(order)
+```
+
+### CLI Progress Visualization
+
+```bash
+# Extract with beautiful progress bars
+arge extract Order 12345 \
+  --output export.json \
+  --progress \
+  --show-graph
+
+# Import with progress
+arge import export.json \
+  --progress \
+  --batch-size 500
+```
+
+## Advanced Configuration
+
+### Handling Circular References
+
+```ruby
+ActiveRecordGraphExtractor.configure do |config|
+  config.handle_circular_references = true
+  config.max_depth = 5
+end
+
+extractor = ActiveRecordGraphExtractor::Extractor.new
+result = extractor.extract(order)
+```
+
+### Memory Optimization
+
+```ruby
+# For large datasets
+ActiveRecordGraphExtractor.configure do |config|
+  config.stream_json = true          # Stream JSON writing/reading
+  config.batch_size = 250           # Smaller batches
+  config.progress_enabled = true    # Monitor progress
+  config.exclude_model('History')
+  config.exclude_model('AuditLog')
+  config.exclude_model('UserEmailHistory')
+end
+```
+
+## Error Handling
+
+### Extraction Errors
+
+```ruby
+begin
+  result = extractor.extract_to_file(order, 'export.json')
+rescue ActiveRecordGraphExtractor::ExtractionError => e
+  puts "Extraction failed: #{e.message}"
+end
+```
+
+### Import Errors
+
+```ruby
+begin
+  result = importer.import_from_file('export.json')
+rescue ActiveRecordGraphExtractor::ImportError => e
+  puts "Import failed: #{e.message}"
+end
+```
+
+## Performance Considerations
+
+### Extraction Performance
+
+1. **Limit Depth**: Use `max_depth` to prevent deep traversal
+2. **Filter Models**: Use `exclude_model` to skip unnecessary tables
+3. **Filter Relationships**: Use `include_relationship` for specific paths
+4. **Batch Size**: Adjust `batch_size` based on memory constraints
+5. **Streaming**: Enable `stream_json` for large datasets
+
+### Import Performance
+
+1. **Skip Validations**: Use `validate_records: false` for trusted data
+2. **Larger Batches**: Increase `batch_size` for faster imports
+3. **Transaction Strategy**: Use `use_transactions: true` for consistency
+
+## Best Practices
+
+### 1. Start Small
+
+```ruby
+# Test with limited scope first
+ActiveRecordGraphExtractor.configure do |config|
+  config.max_depth = 2
+  config.include_relationship('user')
+  config.include_relationship('products')
+end
+
+extractor = ActiveRecordGraphExtractor::Extractor.new
+result = extractor.extract(order)
+```
+
+### 2. Use Dry Runs
+
+```ruby
+# Always test imports with validation first
+ActiveRecordGraphExtractor.configure do |config|
+  config.validate_records = true
+end
+
+importer = ActiveRecordGraphExtractor::Importer.new
+result = importer.import_from_file('export.json')
+puts "Imported #{result['imported_records']} records"
+```
+
+### 3. Monitor Memory Usage
+
+```ruby
+ActiveRecordGraphExtractor.configure do |config|
+  config.progress_enabled = true
+  config.batch_size = 500  # Adjust based on memory constraints
+end
+```
+
+### 4. Handle Large Datasets
+
+```ruby
+# For datasets > 10k records
+ActiveRecordGraphExtractor.configure do |config|
+  config.stream_json = true
+  config.batch_size = 250
+  config.progress_enabled = true
+  config.exclude_model('History')
+  config.exclude_model('AuditLog')
+end
+```
+
+### 5. Version Your Exports
+
+```ruby
+timestamp = Time.now.strftime('%Y%m%d_%H%M%S')
+filename = "order_#{order.id}_#{timestamp}.json"
+result = extractor.extract_to_file(order, filename)
+```
+
+## Common Patterns
+
+### Environment Migration
+
+```ruby
+# Production -> Staging
+production_order = Order.find(id)
+
+ActiveRecordGraphExtractor.configure do |config|
+  config.exclude_model('History')
+  config.exclude_model('AuditLog')
+  config.max_depth = 4
+end
+
+extractor = ActiveRecordGraphExtractor::Extractor.new
+
+# Export from production
+result = extractor.extract_to_file(production_order, 'prod_export.json')
+
+# Import to staging (different environment)
+ActiveRecordGraphExtractor.configure do |config|
+  config.validate_records = true
+end
+
+importer = ActiveRecordGraphExtractor::Importer.new
+staging_result = importer.import_from_file('prod_export.json')
+```
+
+### Test Data Setup
+
+```ruby
+# Extract test fixtures
+test_orders = Order.joins(:user)
+  .where(users: { email: 'test@example.com' })
+  .limit(3)
+
+extractor = ActiveRecordGraphExtractor::Extractor.new
+
+test_orders.each_with_index do |order, i|
+  ActiveRecordGraphExtractor.configure do |config|
+    config.max_depth = 2
+  end
+  
+  extractor.extract_to_file(order, "test_order_#{i + 1}.json")
+end
+```
+
+### Debugging Data Issues
+
+```ruby
+# Extract with maximum detail for debugging
+ActiveRecordGraphExtractor.configure do |config|
+  config.max_depth = 10
+  config.handle_circular_references = true
+  config.progress_enabled = true
+end
+
+extractor = ActiveRecordGraphExtractor::Extractor.new
+result = extractor.extract(problematic_record)
+```
+
+## CLI Quick Reference
+
+```bash
+# Basic extraction
+arge extract Order 12345 -o export.json
+
+# Advanced extraction
+arge extract Order 12345 \
+  --output export.json \
+  --max-depth 3 \
+  --include-relationships user,products,partner \
+  --exclude-models History,AuditLog \
+  --batch-size 500 \
+  --progress \
+  --show-graph \
+  --stream
+
+# Basic import
+arge import export.json
+
+# Advanced import
+arge import export.json \
+  --batch-size 1000 \
+  --skip-validations \
+  --progress
+
+# Dry run import
+arge import export.json --dry-run
+
+# Analyze export file
+arge analyze export.json
+
+# Get help
+arge help extract
+arge help import
+```
+
+## Troubleshooting
+
+### Memory Issues
+
+If you encounter memory issues:
+
+1. Reduce `batch_size`
+2. Enable `stream_json`
+3. Increase progress monitoring frequency
+4. Use `exclude_model` to skip large tables
+
+### Performance Issues
+
+For slow extraction/import:
+
+1. Increase `batch_size` (if memory allows)
+2. Use `validate_records: false` for imports
+3. Limit `max_depth`
+4. Filter relationships with `include_relationship`
+
+### Circular Reference Errors
+
+If you hit circular references:
+
+1. Use `handle_circular_references: true`
+2. Set appropriate `max_depth`
+3. Use `include_relationship` to avoid problematic paths
+
+### Validation Errors
+
+If import validation fails:
+
+1. Check for missing required fields
+2. Verify foreign key relationships
+3. Consider `validate_records: false` if data is trusted 

data/examples/dry_run_example.rb

@@ -0,0 +1,227 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Example: Dry Run Analysis
+# This script demonstrates how to use the dry run feature to analyze
+# what would be extracted without performing the actual extraction.
+
+require 'bundler/setup'
+require 'activerecord_graph_extractor'
+
+# This would typically be in your Rails application
+# For this example, we'll assume you have User and Order models
+
+puts "🔍 ActiveRecord Graph Extractor - Dry Run Examples"
+puts "=" * 60
+puts
+
+# Example 1: Basic Dry Run Analysis
+puts "1. Basic Dry Run Analysis"
+puts "-" * 30
+
+begin
+  # Find a user to analyze
+  user = User.first
+  
+  if user
+    extractor = ActiveRecordGraphExtractor::Extractor.new
+    
+    puts "Analyzing User ##{user.id}..."
+    analysis = extractor.dry_run(user)
+    
+    puts "✅ Analysis completed in #{analysis['analysis_time']} seconds"
+    puts
+    puts "📊 Summary:"
+    puts "   Models involved: #{analysis['extraction_scope']['total_models']}"
+    puts "   Estimated records: #{analysis['extraction_scope']['total_estimated_records']}"
+    puts "   Estimated file size: #{analysis['estimated_file_size']['human_readable']}"
+    puts "   Estimated extraction time: #{analysis['performance_estimates']['estimated_extraction_time_human']}"
+    puts
+    
+    # Show model breakdown
+    puts "📋 Records by Model:"
+    analysis['estimated_counts_by_model'].each do |model, count|
+      percentage = (count.to_f / analysis['extraction_scope']['total_estimated_records'] * 100).round(1)
+      puts "   #{model.ljust(20)} #{count.to_s.rjust(8)} (#{percentage}%)"
+    end
+    puts
+  else
+    puts "❌ No users found in database"
+  end
+rescue => e
+  puts "❌ Error: #{e.message}"
+end
+
+# Example 2: Dry Run with Custom Depth
+puts "2. Dry Run with Custom Max Depth"
+puts "-" * 35
+
+begin
+  user = User.first
+  
+  if user
+    extractor = ActiveRecordGraphExtractor::Extractor.new
+    
+    # Compare different depths
+    [1, 2, 3].each do |depth|
+      analysis = extractor.dry_run(user, max_depth: depth)
+      
+      puts "Depth #{depth}:"
+      puts "   Models: #{analysis['extraction_scope']['total_models']}"
+      puts "   Records: #{analysis['extraction_scope']['total_estimated_records']}"
+      puts "   File size: #{analysis['estimated_file_size']['human_readable']}"
+      puts "   Time: #{analysis['performance_estimates']['estimated_extraction_time_human']}"
+      puts
+    end
+  end
+rescue => e
+  puts "❌ Error: #{e.message}"
+end
+
+# Example 3: Analyzing Multiple Objects
+puts "3. Multiple Objects Analysis"
+puts "-" * 30
+
+begin
+  users = User.limit(3).to_a
+  
+  if users.any?
+    extractor = ActiveRecordGraphExtractor::Extractor.new
+    
+    analysis = extractor.dry_run(users)
+    
+    puts "Analyzing #{users.size} users..."
+    puts "✅ Analysis completed"
+    puts
+    puts "📊 Summary:"
+    puts "   Root objects: #{analysis['root_objects']['count']}"
+    puts "   Total estimated records: #{analysis['extraction_scope']['total_estimated_records']}"
+    puts "   Estimated file size: #{analysis['estimated_file_size']['human_readable']}"
+    puts
+  end
+rescue => e
+  puts "❌ Error: #{e.message}"
+end
+
+# Example 4: Analyzing Warnings and Recommendations
+puts "4. Warnings and Recommendations"
+puts "-" * 35
+
+begin
+  # Try to find a user with many relationships to trigger warnings
+  user = User.joins(:orders).group('users.id').having('COUNT(orders.id) > 0').first
+  
+  if user
+    extractor = ActiveRecordGraphExtractor::Extractor.new
+    
+    analysis = extractor.dry_run(user, max_depth: 5)
+    
+    # Show warnings
+    if analysis['warnings'].any?
+      puts "⚠️  Warnings:"
+      analysis['warnings'].each do |warning|
+        puts "   #{warning['type'].upcase} (#{warning['severity']}): #{warning['message']}"
+      end
+      puts
+    else
+      puts "✅ No warnings detected"
+      puts
+    end
+    
+    # Show recommendations
+    if analysis['recommendations'].any?
+      puts "💡 Recommendations:"
+      analysis['recommendations'].each do |rec|
+        puts "   #{rec['type'].upcase}: #{rec['message']}"
+        puts "   → #{rec['action']}"
+        puts
+      end
+    else
+      puts "✅ No specific recommendations"
+      puts
+    end
+  end
+rescue => e
+  puts "❌ Error: #{e.message}"
+end
+
+# Example 5: Saving Analysis Report
+puts "5. Saving Analysis Report"
+puts "-" * 28
+
+begin
+  user = User.first
+  
+  if user
+    extractor = ActiveRecordGraphExtractor::Extractor.new
+    
+    analysis = extractor.dry_run(user)
+    
+    # Save to file
+    report_file = "dry_run_analysis_#{Time.now.strftime('%Y%m%d_%H%M%S')}.json"
+    File.write(report_file, JSON.pretty_generate(analysis))
+    
+    puts "📄 Analysis report saved to: #{report_file}"
+    puts "   File size: #{File.size(report_file)} bytes"
+    puts
+    
+    # Show a sample of the JSON structure
+    puts "📋 Report structure:"
+    analysis.keys.each do |key|
+      puts "   #{key}"
+    end
+    puts
+  end
+rescue => e
+  puts "❌ Error: #{e.message}"
+end
+
+# Example 6: Decision Making Based on Analysis
+puts "6. Decision Making Example"
+puts "-" * 28
+
+begin
+  user = User.first
+  
+  if user
+    extractor = ActiveRecordGraphExtractor::Extractor.new
+    
+    analysis = extractor.dry_run(user)
+    
+    total_records = analysis['extraction_scope']['total_estimated_records']
+    file_size_mb = analysis['estimated_file_size']['bytes'] / (1024.0 * 1024)
+    extraction_time = analysis['performance_estimates']['estimated_extraction_time_seconds']
+    
+    puts "📊 Analysis Results:"
+    puts "   Records: #{total_records}"
+    puts "   File size: #{file_size_mb.round(1)} MB"
+    puts "   Estimated time: #{extraction_time.round(1)} seconds"
+    puts
+    
+    # Make decisions based on analysis
+    if total_records > 10_000
+      puts "🚨 Large dataset detected!"
+      puts "   Recommendation: Consider using batch processing or reducing max_depth"
+    elsif file_size_mb > 50
+      puts "📦 Large file expected!"
+      puts "   Recommendation: Consider uploading directly to S3"
+    elsif extraction_time > 300 # 5 minutes
+      puts "⏰ Long extraction time expected!"
+      puts "   Recommendation: Run during off-peak hours"
+    else
+      puts "✅ Extraction looks manageable - proceed with confidence!"
+    end
+    puts
+  end
+rescue => e
+  puts "❌ Error: #{e.message}"
+end
+
+puts "🎉 Dry run examples completed!"
+puts
+puts "💡 Tips:"
+puts "   • Use dry run before large extractions to understand scope"
+puts "   • Pay attention to warnings and recommendations"
+puts "   • Save analysis reports for documentation"
+puts "   • Compare different max_depth values to optimize performance"
+puts "   • Use dry run results to make informed decisions about extraction strategy"