fact_db 0.0.1

data/docs/guides/batch-processing.md

@@ -0,0 +1,325 @@
+# Batch Processing
+
+FactDb uses the `simple_flow` gem to provide concurrent pipeline processing for efficient batch operations.
+
+## Overview
+
+Batch processing is useful for:
+
+- Processing multiple documents at once
+- Resolving many entity names
+- Detecting conflicts across entities
+- Bulk fact extraction
+
+## Batch Extraction
+
+### Sequential Processing
+
+Process content one at a time:
+
+```ruby
+facts = FactDb.new
+
+content_ids = [content1.id, content2.id, content3.id]
+
+results = facts.batch_extract(
+  content_ids,
+  extractor: :llm,
+  parallel: false
+)
+```
+
+### Parallel Processing
+
+Process content concurrently (default):
+
+```ruby
+results = facts.batch_extract(
+  content_ids,
+  extractor: :llm,
+  parallel: true  # default
+)
+
+results.each do |result|
+  puts "Content #{result[:content_id]}:"
+  puts "  Facts extracted: #{result[:facts].count}"
+  puts "  Error: #{result[:error]}" if result[:error]
+end
+```
+
+### Result Structure
+
+```ruby
+result = {
+  content_id: 123,
+  facts: [<Fact>, <Fact>, ...],  # Extracted facts
+  error: nil                      # Error message if failed
+}
+```
+
+## Batch Entity Resolution
+
+Resolve multiple names at once:
+
+```ruby
+names = [
+  "Paula Chen",
+  "John Smith",
+  "Microsoft",
+  "Acme Corporation",
+  "Seattle"
+]
+
+results = facts.batch_resolve_entities(names, type: nil)
+
+results.each do |result|
+  case result[:status]
+  when :resolved
+    puts "#{result[:name]} -> #{result[:entity].canonical_name}"
+  when :not_found
+    puts "#{result[:name]} -> Not found"
+  when :error
+    puts "#{result[:name]} -> Error: #{result[:error]}"
+  end
+end
+```
+
+### With Type Filtering
+
+```ruby
+# Only resolve as person entities
+results = facts.batch_resolve_entities(names, type: :person)
+```
+
+## Conflict Detection
+
+Check multiple entities for conflicting facts:
+
+```ruby
+entity_ids = [paula.id, john.id, microsoft.id]
+
+results = facts.detect_fact_conflicts(entity_ids)
+
+results.each do |result|
+  if result[:conflict_count] > 0
+    puts "Entity #{result[:entity_id]} has #{result[:conflict_count]} conflicts:"
+    result[:conflicts].each do |conflict|
+      puts "  #{conflict[:fact1].fact_text}"
+      puts "  vs"
+      puts "  #{conflict[:fact2].fact_text}"
+      puts "  Similarity: #{conflict[:similarity]}"
+    end
+  end
+end
+```
+
+## Using Pipelines Directly
+
+For more control, use the pipeline classes directly:
+
+### Extraction Pipeline
+
+```ruby
+pipeline = FactDb::Pipeline::ExtractionPipeline.new(FactDb.config)
+
+# Sequential
+results = pipeline.process(contents, extractor: :llm)
+
+# Parallel
+results = pipeline.process_parallel(contents, extractor: :llm)
+```
+
+### Resolution Pipeline
+
+```ruby
+pipeline = FactDb::Pipeline::ResolutionPipeline.new(FactDb.config)
+
+# Resolve entities
+results = pipeline.resolve_entities(names, type: :person)
+
+# Detect conflicts
+results = pipeline.detect_conflicts(entity_ids)
+```
+
+## SimpleFlow Integration
+
+FactDb's pipelines are built on SimpleFlow:
+
+```ruby
+require 'simple_flow'
+
+# Create custom pipeline
+pipeline = SimpleFlow::Pipeline.new do
+  # Step 1: Validate
+  step ->(result) {
+    content = result.value
+    if content.raw_text.blank?
+      result.halt("Empty content")
+    else
+      result.continue(content)
+    end
+  }
+
+  # Step 2: Extract
+  step ->(result) {
+    facts = extractor.extract(result.value)
+    result.continue(facts)
+  }
+
+  # Step 3: Validate facts
+  step ->(result) {
+    valid_facts = result.value.select(&:valid?)
+    result.continue(valid_facts)
+  }
+end
+
+# Execute
+result = pipeline.call(SimpleFlow::Result.new(content))
+```
+
+## Error Handling
+
+### Graceful Degradation
+
+```ruby
+results = facts.batch_extract(content_ids, extractor: :llm)
+
+successful = results.select { |r| r[:error].nil? }
+failed = results.reject { |r| r[:error].nil? }
+
+puts "Successful: #{successful.count}"
+puts "Failed: #{failed.count}"
+
+# Retry failed items with different extractor
+if failed.any?
+  retry_ids = failed.map { |r| r[:content_id] }
+  retry_results = facts.batch_extract(retry_ids, extractor: :rule_based)
+end
+```
+
+### Logging Errors
+
+```ruby
+results.each do |result|
+  if result[:error]
+    logger.error(
+      "Extraction failed",
+      content_id: result[:content_id],
+      error: result[:error]
+    )
+  end
+end
+```
+
+## Performance Considerations
+
+### Optimal Batch Size
+
+```ruby
+# Process in batches of 10-50 for optimal performance
+content_ids.each_slice(25) do |batch|
+  results = facts.batch_extract(batch, parallel: true)
+  process_results(results)
+end
+```
+
+### Rate Limiting
+
+For LLM extraction, add delays between batches:
+
+```ruby
+content_ids.each_slice(10) do |batch|
+  results = facts.batch_extract(batch, extractor: :llm)
+  process_results(results)
+  sleep(2)  # Rate limit
+end
+```
+
+### Memory Management
+
+```ruby
+# Process results immediately to avoid memory buildup
+content_ids.each_slice(50) do |batch|
+  results = facts.batch_extract(batch)
+
+  results.each do |result|
+    # Process and discard
+    save_facts(result[:facts])
+  end
+
+  # Force garbage collection if needed
+  GC.start if batch_count % 10 == 0
+end
+```
+
+## Monitoring
+
+Track batch processing metrics:
+
+```ruby
+start_time = Time.now
+
+results = facts.batch_extract(content_ids, parallel: true)
+
+duration = Time.now - start_time
+success_rate = results.count { |r| r[:error].nil? }.to_f / results.count
+
+puts "Processed #{results.count} items in #{duration}s"
+puts "Success rate: #{(success_rate * 100).round(1)}%"
+puts "Items/second: #{(results.count / duration).round(2)}"
+```
+
+## Best Practices
+
+### 1. Use Parallel for Large Batches
+
+```ruby
+# Sequential for small batches (< 5 items)
+if content_ids.count < 5
+  results = facts.batch_extract(content_ids, parallel: false)
+else
+  results = facts.batch_extract(content_ids, parallel: true)
+end
+```
+
+### 2. Handle Partial Failures
+
+```ruby
+def process_batch(content_ids)
+  results = facts.batch_extract(content_ids)
+
+  {
+    successful: results.select { |r| r[:error].nil? },
+    failed: results.reject { |r| r[:error].nil? }
+  }
+end
+
+batch_result = process_batch(content_ids)
+retry_failed(batch_result[:failed]) if batch_result[:failed].any?
+```
+
+### 3. Log Progress
+
+```ruby
+total = content_ids.count
+processed = 0
+
+content_ids.each_slice(25) do |batch|
+  results = facts.batch_extract(batch)
+  processed += batch.count
+
+  logger.info "Progress: #{processed}/#{total} (#{(processed.to_f/total*100).round(1)}%)"
+end
+```
+
+### 4. Use Appropriate Extractors
+
+```ruby
+# LLM for complex documents
+complex_docs = contents.select { |c| c.raw_text.length > 1000 }
+facts.batch_extract(complex_docs.map(&:id), extractor: :llm)
+
+# Rule-based for simple, structured content
+simple_docs = contents.select { |c| c.raw_text.length <= 1000 }
+facts.batch_extract(simple_docs.map(&:id), extractor: :rule_based)
+```

data/docs/guides/configuration.md

@@ -0,0 +1,243 @@
+# Configuration
+
+FactDb uses the `anyway_config` gem for flexible configuration via environment variables, YAML files, or Ruby code.
+
+## Configuration Methods
+
+### Environment Variables
+
+All settings can be configured via environment variables with the `EVENT_CLOCK_` prefix:
+
+```bash
+export EVENT_CLOCK_DATABASE_URL="postgresql://localhost/fact_db"
+export EVENT_CLOCK_DATABASE_POOL_SIZE=10
+export EVENT_CLOCK_LLM_PROVIDER="openai"
+export EVENT_CLOCK_LLM_MODEL="gpt-4o-mini"
+export EVENT_CLOCK_LLM_API_KEY="sk-..."
+export EVENT_CLOCK_FUZZY_MATCH_THRESHOLD=0.85
+```
+
+### YAML Configuration
+
+Create `config/fact_db.yml`:
+
+```yaml
+# Database
+database_url: postgresql://localhost/fact_db
+database_pool_size: 10
+database_timeout: 30000
+
+# Embeddings
+embedding_dimensions: 1536
+
+# LLM
+llm_provider: openai
+llm_model: gpt-4o-mini
+llm_api_key: <%= ENV['OPENAI_API_KEY'] %>
+
+# Extraction
+default_extractor: manual
+
+# Entity Resolution
+fuzzy_match_threshold: 0.85
+auto_merge_threshold: 0.95
+
+# Logging
+log_level: info
+```
+
+### Ruby Block
+
+```ruby
+FactDb.configure do |config|
+  # Database
+  config.database_url = "postgresql://localhost/fact_db"
+  config.database_pool_size = 10
+  config.database_timeout = 30_000
+
+  # Embeddings
+  config.embedding_dimensions = 1536
+  config.embedding_generator = ->(text) {
+    # Your embedding generation logic
+    OpenAI::Client.new.embeddings(input: text)
+  }
+
+  # LLM
+  config.llm_provider = :openai
+  config.llm_model = "gpt-4o-mini"
+  config.llm_api_key = ENV['OPENAI_API_KEY']
+
+  # Or provide a pre-configured client
+  config.llm_client = FactDb::LLM::Adapter.new(
+    provider: :anthropic,
+    model: "claude-sonnet-4-20250514"
+  )
+
+  # Extraction
+  config.default_extractor = :llm
+
+  # Entity Resolution
+  config.fuzzy_match_threshold = 0.85
+  config.auto_merge_threshold = 0.95
+
+  # Logging
+  config.logger = Rails.logger
+  config.log_level = :debug
+end
+```
+
+## Configuration Options
+
+### Database Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `database_url` | String | nil | PostgreSQL connection URL (required) |
+| `database_pool_size` | Integer | 5 | Connection pool size |
+| `database_timeout` | Integer | 30000 | Query timeout in milliseconds |
+
+### Embedding Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `embedding_dimensions` | Integer | 1536 | Vector dimensions (match your model) |
+| `embedding_generator` | Proc | nil | Custom embedding generation function |
+
+### LLM Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `llm_client` | Object | nil | Pre-configured LLM client |
+| `llm_provider` | Symbol | nil | Provider name (:openai, :anthropic, etc.) |
+| `llm_model` | String | varies | Model name |
+| `llm_api_key` | String | nil | API key |
+
+### Extraction Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `default_extractor` | Symbol | :manual | Default extraction method |
+
+### Resolution Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `fuzzy_match_threshold` | Float | 0.85 | Minimum similarity for fuzzy matching |
+| `auto_merge_threshold` | Float | 0.95 | Similarity threshold for auto-merge |
+
+### Logging Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `logger` | Logger | STDOUT | Logger instance |
+| `log_level` | Symbol | :info | Log level |
+
+## LLM Provider Configuration
+
+### OpenAI
+
+```ruby
+FactDb.configure do |config|
+  config.llm_provider = :openai
+  config.llm_model = "gpt-4o-mini"  # or "gpt-4o", "gpt-4-turbo"
+  config.llm_api_key = ENV['OPENAI_API_KEY']
+end
+```
+
+### Anthropic
+
+```ruby
+FactDb.configure do |config|
+  config.llm_provider = :anthropic
+  config.llm_model = "claude-sonnet-4-20250514"
+  config.llm_api_key = ENV['ANTHROPIC_API_KEY']
+end
+```
+
+### Google Gemini
+
+```ruby
+FactDb.configure do |config|
+  config.llm_provider = :gemini
+  config.llm_model = "gemini-2.0-flash"
+  config.llm_api_key = ENV['GEMINI_API_KEY']
+end
+```
+
+### Ollama (Local)
+
+```ruby
+FactDb.configure do |config|
+  config.llm_provider = :ollama
+  config.llm_model = "llama3.2"
+  # No API key needed for local Ollama
+end
+```
+
+### AWS Bedrock
+
+```ruby
+FactDb.configure do |config|
+  config.llm_provider = :bedrock
+  config.llm_model = "claude-sonnet-4"
+  # Uses AWS credentials from environment
+end
+```
+
+### OpenRouter
+
+```ruby
+FactDb.configure do |config|
+  config.llm_provider = :openrouter
+  config.llm_model = "anthropic/claude-sonnet-4"
+  config.llm_api_key = ENV['OPENROUTER_API_KEY']
+end
+```
+
+## Environment-Specific Configuration
+
+Use YAML anchors for shared settings:
+
+```yaml
+# config/fact_db.yml
+defaults: &defaults
+  embedding_dimensions: 1536
+  fuzzy_match_threshold: 0.85
+
+development:
+  <<: *defaults
+  database_url: postgresql://localhost/fact_db_dev
+  log_level: debug
+
+test:
+  <<: *defaults
+  database_url: postgresql://localhost/fact_db_test
+  log_level: warn
+
+production:
+  <<: *defaults
+  database_url: <%= ENV['DATABASE_URL'] %>
+  log_level: info
+```
+
+## Validation
+
+Validate configuration at startup:
+
+```ruby
+FactDb.configure do |config|
+  config.database_url = ENV['DATABASE_URL']
+end
+
+# Raises ConfigurationError if invalid
+FactDb.config.validate!
+```
+
+## Reset Configuration
+
+For testing, reset configuration between tests:
+
+```ruby
+# In test setup
+FactDb.reset_configuration!
+```