fact_db 0.0.1

data/docs/guides/ingesting-content.md

@@ -0,0 +1,252 @@
+# Ingesting Content
+
+Content is the foundation of FactDb - immutable source documents from which facts are extracted.
+
+## Basic Ingestion
+
+```ruby
+facts = FactDb.new
+
+content = facts.ingest(
+  "Paula Chen joined Microsoft as Principal Engineer on January 10, 2024.",
+  type: :announcement
+)
+```
+
+## Full Options
+
+```ruby
+content = facts.ingest(
+  raw_text,
+  type: :email,
+  title: "RE: Offer Letter - Paula Chen",
+  source_uri: "mailto:hr@company.com/msg/12345",
+  captured_at: Time.parse("2024-01-08 10:30:00"),
+  metadata: {
+    from: "hr@company.com",
+    to: "hiring@company.com",
+    cc: ["manager@company.com"],
+    subject: "RE: Offer Letter - Paula Chen",
+    thread_id: "THR-12345"
+  }
+)
+```
+
+## Content Types
+
+Choose a type that best describes the source:
+
+| Type | Use Case |
+|------|----------|
+| `:email` | Email messages |
+| `:document` | General documents, PDFs |
+| `:article` | News articles, blog posts |
+| `:transcript` | Meeting transcripts, interviews |
+| `:report` | Reports, analysis documents |
+| `:announcement` | Official announcements |
+| `:social` | Social media posts |
+| `:form` | Structured forms, surveys |
+| `:note` | Notes, memos |
+
+```ruby
+# Custom types are also allowed
+content = facts.ingest(text, type: :slack_message)
+```
+
+## Metadata
+
+Store additional context in metadata:
+
+```ruby
+# Email metadata
+metadata: {
+  from: "sender@example.com",
+  to: "recipient@example.com",
+  subject: "Important Update",
+  message_id: "<abc123@mail.example.com>"
+}
+
+# Document metadata
+metadata: {
+  author: "Jane Smith",
+  version: "2.1",
+  department: "Engineering",
+  classification: "internal"
+}
+
+# Article metadata
+metadata: {
+  author: "John Doe",
+  publication: "Tech News",
+  url: "https://technews.com/article/123",
+  published_at: "2024-01-15T14:30:00Z"
+}
+```
+
+## Deduplication
+
+Content is automatically deduplicated by SHA256 hash:
+
+```ruby
+# First ingestion - creates new record
+content1 = facts.ingest("Hello world", type: :note)
+
+# Second ingestion - returns existing record
+content2 = facts.ingest("Hello world", type: :note)
+
+content1.id == content2.id  # => true
+```
+
+## Timestamps
+
+### captured_at
+
+When the content was captured/received (defaults to current time):
+
+```ruby
+# Email received yesterday
+content = facts.ingest(
+  email_body,
+  type: :email,
+  captured_at: Time.parse("2024-01-14 09:00:00")
+)
+```
+
+### created_at
+
+Automatically set when record is created (system timestamp).
+
+## Batch Ingestion
+
+For multiple documents:
+
+```ruby
+documents = [
+  { text: "Doc 1 content", type: :document, title: "Doc 1" },
+  { text: "Doc 2 content", type: :document, title: "Doc 2" },
+  { text: "Doc 3 content", type: :document, title: "Doc 3" }
+]
+
+contents = documents.map do |doc|
+  facts.ingest(doc[:text], type: doc[:type], title: doc[:title])
+end
+```
+
+## Content Service
+
+For advanced operations, use the content service directly:
+
+```ruby
+# Create content
+content = facts.content_service.create(
+  raw_text,
+  type: :document,
+  title: "Annual Report"
+)
+
+# Find by ID
+content = facts.content_service.find(content_id)
+
+# Find by hash
+content = facts.content_service.find_by_hash(sha256_hash)
+
+# Search by text
+results = facts.content_service.search("quarterly earnings")
+
+# Semantic search (requires embedding)
+results = facts.content_service.semantic_search(
+  "financial performance",
+  limit: 10
+)
+```
+
+## Embeddings
+
+If you configure an embedding generator, content embeddings are created automatically:
+
+```ruby
+FactDb.configure do |config|
+  config.embedding_generator = ->(text) {
+    # Your embedding logic
+    client.embeddings(input: text)
+  }
+end
+
+# Embeddings generated on ingest
+content = facts.ingest(text, type: :document)
+content.embedding  # => [0.123, -0.456, ...]
+```
+
+## Source URIs
+
+Track original locations with source_uri:
+
+```ruby
+# Email
+source_uri: "mailto:sender@example.com/msg/12345"
+
+# Web page
+source_uri: "https://example.com/articles/123"
+
+# File
+source_uri: "file:///path/to/document.pdf"
+
+# Database record
+source_uri: "db://crm/contacts/12345"
+
+# API
+source_uri: "api://salesforce/leads/ABC123"
+```
+
+## Best Practices
+
+### 1. Preserve Original Text
+
+```ruby
+# Good - preserve original formatting
+facts.ingest(original_email_body, type: :email)
+
+# Avoid - don't pre-process
+facts.ingest(cleaned_text.strip.downcase, type: :email)
+```
+
+### 2. Include Context in Metadata
+
+```ruby
+content = facts.ingest(
+  transcript,
+  type: :transcript,
+  title: "Q4 2024 Earnings Call",
+  metadata: {
+    participants: ["CEO", "CFO", "Analysts"],
+    duration_minutes: 60,
+    recording_url: "https://..."
+  }
+)
+```
+
+### 3. Use Consistent Types
+
+```ruby
+# Define content types for your organization
+module ContentTypes
+  EMAIL = :email
+  SLACK = :slack_message
+  MEETING = :meeting_transcript
+  # ...
+end
+
+facts.ingest(text, type: ContentTypes::EMAIL)
+```
+
+### 4. Track Source
+
+```ruby
+# Always include source information for audit trails
+content = facts.ingest(
+  text,
+  type: :document,
+  source_uri: "sharepoint://documents/annual-report-2024.pdf",
+  metadata: { uploaded_by: "jane@company.com" }
+)
+```

data/docs/guides/llm-integration.md

@@ -0,0 +1,299 @@
+# LLM Integration
+
+FactDb integrates with multiple LLM providers via the `ruby_llm` gem for AI-powered fact extraction.
+
+## Setup
+
+### Install ruby_llm
+
+Add to your Gemfile:
+
+```ruby
+gem 'ruby_llm'
+```
+
+### Configure Provider
+
+=== "OpenAI"
+
+    ```ruby
+    FactDb.configure do |config|
+      config.llm_provider = :openai
+      config.llm_model = "gpt-4o-mini"
+      config.llm_api_key = ENV['OPENAI_API_KEY']
+    end
+    ```
+
+=== "Anthropic"
+
+    ```ruby
+    FactDb.configure do |config|
+      config.llm_provider = :anthropic
+      config.llm_model = "claude-sonnet-4-20250514"
+      config.llm_api_key = ENV['ANTHROPIC_API_KEY']
+    end
+    ```
+
+=== "Google Gemini"
+
+    ```ruby
+    FactDb.configure do |config|
+      config.llm_provider = :gemini
+      config.llm_model = "gemini-2.0-flash"
+      config.llm_api_key = ENV['GEMINI_API_KEY']
+    end
+    ```
+
+=== "Ollama (Local)"
+
+    ```ruby
+    FactDb.configure do |config|
+      config.llm_provider = :ollama
+      config.llm_model = "llama3.2"
+    end
+    ```
+
+## Supported Providers
+
+| Provider | Models | Config Key |
+|----------|--------|------------|
+| OpenAI | gpt-4o, gpt-4o-mini, gpt-4-turbo | `OPENAI_API_KEY` |
+| Anthropic | claude-sonnet-4, claude-3-haiku | `ANTHROPIC_API_KEY` |
+| Google Gemini | gemini-2.0-flash, gemini-pro | `GEMINI_API_KEY` |
+| Ollama | llama3.2, mistral, codellama | (local) |
+| AWS Bedrock | claude-sonnet-4, titan | AWS credentials |
+| OpenRouter | Various | `OPENROUTER_API_KEY` |
+
+## Default Models
+
+If no model is specified, these defaults are used:
+
+```ruby
+PROVIDER_DEFAULTS = {
+  openai: "gpt-4o-mini",
+  anthropic: "claude-sonnet-4-20250514",
+  gemini: "gemini-2.0-flash",
+  ollama: "llama3.2",
+  bedrock: "claude-sonnet-4",
+  openrouter: "anthropic/claude-sonnet-4"
+}
+```
+
+## Using LLM Extraction
+
+```ruby
+facts = FactDb.new
+
+# Ingest content
+content = facts.ingest(
+  "Paula Chen joined Microsoft as Principal Engineer on January 10, 2024. She previously worked at Google for 5 years.",
+  type: :announcement
+)
+
+# Extract facts using LLM
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+extracted.each do |fact|
+  puts "Fact: #{fact.fact_text}"
+  puts "  Valid: #{fact.valid_at}"
+  puts "  Confidence: #{fact.confidence}"
+  fact.entity_mentions.each do |m|
+    puts "  Entity: #{m.entity.canonical_name} (#{m.mention_role})"
+  end
+end
+```
+
+## Extraction Prompts
+
+The LLM extractor uses carefully designed prompts to extract:
+
+1. **Facts** - Temporal assertions about entities
+2. **Entities** - People, organizations, places mentioned
+3. **Dates** - When facts became valid
+4. **Relationships** - How entities relate to facts
+
+### Example Prompt Structure
+
+```
+Extract temporal facts from this content. For each fact:
+1. Identify the assertion (what is being stated)
+2. Identify entities mentioned (people, organizations, places)
+3. Determine when the fact became valid
+4. Assess confidence level
+
+Content:
+{content.raw_text}
+
+Return JSON:
+{
+  "facts": [
+    {
+      "text": "...",
+      "valid_at": "YYYY-MM-DD",
+      "entities": [
+        {"name": "...", "type": "person|organization|place", "role": "subject|object|..."}
+      ],
+      "confidence": 0.0-1.0
+    }
+  ]
+}
+```
+
+## Custom LLM Client
+
+Provide a pre-configured client:
+
+```ruby
+# Create custom adapter
+adapter = FactDb::LLM::Adapter.new(
+  provider: :openai,
+  model: "gpt-4o",
+  api_key: ENV['OPENAI_API_KEY']
+)
+
+FactDb.configure do |config|
+  config.llm_client = adapter
+end
+```
+
+## Direct LLM Usage
+
+Use the adapter directly:
+
+```ruby
+adapter = FactDb::LLM::Adapter.new(
+  provider: :anthropic,
+  model: "claude-sonnet-4-20250514"
+)
+
+response = adapter.chat("Extract facts from: Paula joined Microsoft on Jan 10, 2024")
+puts response
+```
+
+## Error Handling
+
+```ruby
+begin
+  extracted = facts.extract_facts(content.id, extractor: :llm)
+rescue FactDb::ConfigurationError => e
+  # LLM not configured or ruby_llm missing
+  puts "LLM Error: #{e.message}"
+  # Fall back to rule-based
+  extracted = facts.extract_facts(content.id, extractor: :rule_based)
+rescue StandardError => e
+  # API error, rate limit, etc.
+  puts "Extraction failed: #{e.message}"
+end
+```
+
+## Batch Processing with LLM
+
+Process multiple documents efficiently:
+
+```ruby
+content_ids = [content1.id, content2.id, content3.id]
+
+# Parallel processing (uses simple_flow pipeline)
+results = facts.batch_extract(content_ids, extractor: :llm, parallel: true)
+
+results.each do |result|
+  if result[:error]
+    puts "Error for #{result[:content_id]}: #{result[:error]}"
+  else
+    puts "Extracted #{result[:facts].count} facts from #{result[:content_id]}"
+  end
+end
+```
+
+## Cost Optimization
+
+### Use Appropriate Models
+
+```ruby
+# For simple extractions, use smaller models
+config.llm_model = "gpt-4o-mini"  # Cheaper than gpt-4o
+
+# For complex documents, use larger models
+config.llm_model = "gpt-4o"
+```
+
+### Batch Processing
+
+```ruby
+# Process in batches to reduce API calls
+content_ids.each_slice(10) do |batch|
+  facts.batch_extract(batch, extractor: :llm)
+  sleep(1)  # Rate limiting
+end
+```
+
+### Local Models
+
+```ruby
+# Use Ollama for development/testing
+FactDb.configure do |config|
+  config.llm_provider = :ollama
+  config.llm_model = "llama3.2"
+end
+```
+
+## Testing
+
+Mock LLM responses in tests:
+
+```ruby
+class MockLLMClient
+  def chat(prompt)
+    # Return predictable test data
+    '{"facts": [{"text": "Test fact", "valid_at": "2024-01-01", "entities": [], "confidence": 0.9}]}'
+  end
+end
+
+FactDb.configure do |config|
+  config.llm_client = MockLLMClient.new
+end
+```
+
+## Best Practices
+
+### 1. Validate Extractions
+
+```ruby
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+extracted.each do |fact|
+  # Flag low-confidence extractions
+  if fact.confidence < 0.7
+    fact.update!(metadata: { needs_review: true })
+  end
+end
+```
+
+### 2. Use Caching
+
+```ruby
+# Cache LLM responses for repeated content
+cache_key = "llm_extraction:#{content.content_hash}"
+extracted = Rails.cache.fetch(cache_key) do
+  facts.extract_facts(content.id, extractor: :llm)
+end
+```
+
+### 3. Handle Rate Limits
+
+```ruby
+require 'retryable'
+
+Retryable.retryable(tries: 3, sleep: 5) do
+  facts.extract_facts(content.id, extractor: :llm)
+end
+```
+
+### 4. Monitor Usage
+
+```ruby
+# Track extraction statistics
+extracted = facts.extract_facts(content.id, extractor: :llm)
+StatsD.increment('fact_db.llm_extractions')
+StatsD.histogram('fact_db.facts_per_content', extracted.count)
+```