fact_db 0.0.1

data/docs/api/extractors/llm.md

@@ -0,0 +1,162 @@
+# LLMExtractor
+
+AI-powered fact extraction using large language models.
+
+## Class: `FactDb::Extractors::LLMExtractor`
+
+```ruby
+extractor = FactDb::Extractors::LLMExtractor.new(config)
+```
+
+## Requirements
+
+- `ruby_llm` gem installed
+- LLM provider configured (API key, model)
+
+## Configuration
+
+```ruby
+FactDb.configure do |config|
+  config.llm_provider = :openai
+  config.llm_model = "gpt-4o-mini"
+  config.llm_api_key = ENV['OPENAI_API_KEY']
+end
+```
+
+## Methods
+
+### extract
+
+```ruby
+def extract(content)
+```
+
+Extract facts from content using LLM.
+
+**Parameters:**
+
+- `content` (Models::Content) - Content to process
+
+**Returns:** `Array<Models::Fact>`
+
+**Example:**
+
+```ruby
+extractor = LLMExtractor.new(config)
+facts = extractor.extract(content)
+
+facts.each do |fact|
+  puts fact.fact_text
+  puts "  Valid: #{fact.valid_at}"
+  puts "  Confidence: #{fact.confidence}"
+end
+```
+
+## Extraction Process
+
+1. **Prompt Construction** - Build prompt with content text
+2. **LLM Call** - Send to configured LLM provider
+3. **Response Parsing** - Parse JSON response
+4. **Fact Creation** - Create fact records
+5. **Entity Resolution** - Resolve mentioned entities
+6. **Source Linking** - Link facts to source content
+
+## Prompt Structure
+
+The extractor uses a structured prompt:
+
+```
+Extract temporal facts from this content. For each fact:
+1. Identify the assertion (what is being stated)
+2. Identify entities mentioned (people, organizations, places)
+3. Determine when the fact became valid
+4. Assess confidence level
+
+Content:
+{content.raw_text}
+
+Return JSON:
+{
+  "facts": [
+    {
+      "text": "...",
+      "valid_at": "YYYY-MM-DD",
+      "entities": [
+        {"name": "...", "type": "person|organization|place", "role": "subject|object|..."}
+      ],
+      "confidence": 0.0-1.0
+    }
+  ]
+}
+```
+
+## Supported Providers
+
+| Provider | Models | Config |
+|----------|--------|--------|
+| OpenAI | gpt-4o, gpt-4o-mini | `llm_provider: :openai` |
+| Anthropic | claude-sonnet-4, claude-3-haiku | `llm_provider: :anthropic` |
+| Google | gemini-2.0-flash | `llm_provider: :gemini` |
+| Ollama | llama3.2, mistral | `llm_provider: :ollama` |
+| AWS Bedrock | claude-sonnet-4 | `llm_provider: :bedrock` |
+| OpenRouter | Various | `llm_provider: :openrouter` |
+
+## Error Handling
+
+```ruby
+begin
+  facts = extractor.extract(content)
+rescue FactDb::ConfigurationError => e
+  # LLM not configured
+  puts "Config error: #{e.message}"
+rescue FactDb::ExtractionError => e
+  # Extraction failed
+  puts "Extraction error: #{e.message}"
+end
+```
+
+## Advantages
+
+- Handles unstructured text
+- Understands context and nuance
+- Identifies implicit facts
+- Resolves entities automatically
+
+## Disadvantages
+
+- API costs
+- Latency
+- Occasional errors
+- Requires validation
+
+## Best Practices
+
+### 1. Validate Results
+
+```ruby
+facts = extractor.extract(content)
+facts.each do |fact|
+  if fact.confidence < 0.7
+    fact.update!(metadata: { needs_review: true })
+  end
+end
+```
+
+### 2. Cache Responses
+
+```ruby
+cache_key = "llm:#{content.content_hash}"
+facts = Rails.cache.fetch(cache_key) do
+  extractor.extract(content)
+end
+```
+
+### 3. Handle Rate Limits
+
+```ruby
+require 'retryable'
+
+Retryable.retryable(tries: 3, sleep: lambda { |n| 2**n }) do
+  extractor.extract(content)
+end
+```

data/docs/api/extractors/manual.md

@@ -0,0 +1,92 @@
+# ManualExtractor
+
+API-driven fact creation for maximum control and accuracy.
+
+## Class: `FactDb::Extractors::ManualExtractor`
+
+The ManualExtractor doesn't automatically extract facts - instead it provides a structured interface for creating facts programmatically.
+
+## Usage
+
+```ruby
+extractor = FactDb::Extractors::ManualExtractor.new(config)
+```
+
+## Methods
+
+### extract
+
+```ruby
+def extract(content)
+```
+
+Returns an empty array - manual extraction is done via direct fact creation.
+
+**Returns:** `[]`
+
+## When to Use
+
+- High-stakes facts that require human verification
+- Structured data import from external systems
+- Fact correction or adjustment
+- Initial seeding of the system
+
+## Creating Facts Manually
+
+Instead of using the extractor, create facts directly:
+
+```ruby
+facts = FactDb.new
+
+# Create entities
+paula = facts.entity_service.create("Paula Chen", type: :person)
+microsoft = facts.entity_service.create("Microsoft", type: :organization)
+
+# Create fact with explicit links
+fact = facts.fact_service.create(
+  "Paula Chen joined Microsoft as Principal Engineer",
+  valid_at: Date.parse("2024-01-10"),
+  mentions: [
+    { entity: paula, role: "subject", text: "Paula Chen" },
+    { entity: microsoft, role: "organization", text: "Microsoft" }
+  ],
+  sources: [
+    { content: announcement, type: "primary", excerpt: "...accepted the offer..." }
+  ],
+  confidence: 1.0
+)
+```
+
+## Bulk Import Pattern
+
+```ruby
+# Import from structured data
+data = [
+  { text: "Fact 1", date: "2024-01-01", entity: "Paula" },
+  { text: "Fact 2", date: "2024-01-15", entity: "Paula" }
+]
+
+data.each do |item|
+  entity = facts.resolve_entity(item[:entity])
+
+  facts.fact_service.create(
+    item[:text],
+    valid_at: Date.parse(item[:date]),
+    mentions: [{ entity: entity, role: "subject", text: item[:entity] }],
+    extraction_method: "manual"
+  )
+end
+```
+
+## Advantages
+
+- Complete control over fact creation
+- Highest accuracy (human-verified)
+- No LLM costs
+- Works without external dependencies
+
+## Disadvantages
+
+- Labor intensive
+- Not scalable for large volumes
+- Requires domain expertise

data/docs/api/extractors/rule-based.md

@@ -0,0 +1,165 @@
+# RuleBasedExtractor
+
+Pattern-based fact extraction using regular expressions.
+
+## Class: `FactDb::Extractors::RuleBasedExtractor`
+
+```ruby
+extractor = FactDb::Extractors::RuleBasedExtractor.new(config)
+```
+
+## Methods
+
+### extract
+
+```ruby
+def extract(content)
+```
+
+Extract facts using pattern matching.
+
+**Returns:** `Array<Models::Fact>`
+
+## Built-in Patterns
+
+The extractor includes patterns for common fact types:
+
+### Employment Events
+
+```ruby
+# "X joined Y"
+/(?<person>\w+(?:\s+\w+)*)\s+joined\s+(?<org>\w+(?:\s+\w+)*)/i
+
+# "X left Y"
+/(?<person>\w+(?:\s+\w+)*)\s+left\s+(?<org>\w+(?:\s+\w+)*)/i
+
+# "X was hired by Y"
+/(?<person>\w+(?:\s+\w+)*)\s+was\s+hired\s+by\s+(?<org>\w+(?:\s+\w+)*)/i
+```
+
+### Title Changes
+
+```ruby
+# "X is/was the Y"
+/(?<person>\w+(?:\s+\w+)*)\s+(?:is|was)\s+(?:the\s+)?(?<title>[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)/
+
+# "X promoted to Y"
+/(?<person>\w+(?:\s+\w+)*)\s+(?:was\s+)?promoted\s+to\s+(?<title>[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)/i
+```
+
+### Date Patterns
+
+```ruby
+# "on January 10, 2024"
+/on\s+(?<month>\w+)\s+(?<day>\d{1,2}),?\s+(?<year>\d{4})/i
+
+# "in Q4 2024"
+/in\s+Q(?<quarter>\d)\s+(?<year>\d{4})/i
+
+# ISO dates
+/(?<date>\d{4}-\d{2}-\d{2})/
+```
+
+## Usage Example
+
+```ruby
+extractor = RuleBasedExtractor.new(config)
+
+content = Models::Content.create!(
+  raw_text: "Paula Chen joined Microsoft on January 10, 2024 as Principal Engineer.",
+  content_type: "announcement",
+  captured_at: Time.current
+)
+
+facts = extractor.extract(content)
+# Returns facts about:
+# - Paula joining Microsoft
+# - Paula's title as Principal Engineer
+# - Date: January 10, 2024
+```
+
+## Adding Custom Patterns
+
+Extend the extractor with custom patterns:
+
+```ruby
+class CustomRuleExtractor < FactDb::Extractors::RuleBasedExtractor
+  CUSTOM_PATTERNS = [
+    {
+      pattern: /revenue\s+of\s+\$(?<amount>[\d,]+)/i,
+      type: :financial,
+      handler: :extract_revenue
+    }
+  ]
+
+  def extract(content)
+    facts = super(content)
+    facts + extract_custom_patterns(content)
+  end
+
+  private
+
+  def extract_custom_patterns(content)
+    facts = []
+    CUSTOM_PATTERNS.each do |rule|
+      content.raw_text.scan(rule[:pattern]) do |match|
+        facts << send(rule[:handler], match, content)
+      end
+    end
+    facts
+  end
+
+  def extract_revenue(match, content)
+    Models::Fact.create!(
+      fact_text: "Revenue of $#{match[:amount]}",
+      valid_at: content.captured_at,
+      extraction_method: "rule_based",
+      # ...
+    )
+  end
+end
+```
+
+## Advantages
+
+- Fast execution
+- No external dependencies
+- Predictable results
+- Works offline
+- Zero cost
+
+## Disadvantages
+
+- Limited to defined patterns
+- Misses implicit facts
+- Requires pattern maintenance
+- May produce false positives
+
+## Best Practices
+
+### 1. Combine with LLM
+
+```ruby
+# Use rule-based for structured content
+if content.content_type == "form"
+  facts = rule_extractor.extract(content)
+else
+  facts = llm_extractor.extract(content)
+end
+```
+
+### 2. Validate Matches
+
+```ruby
+facts = extractor.extract(content)
+facts.select { |f| f.confidence > 0.8 }
+```
+
+### 3. Log Unmatched Content
+
+```ruby
+facts = extractor.extract(content)
+if facts.empty?
+  logger.info "No patterns matched for content #{content.id}"
+end
+```

data/docs/api/facts.md

@@ -0,0 +1,300 @@
+# Facts
+
+The main interface for FactDb operations.
+
+## Class: `FactDb::Facts`
+
+```ruby
+facts = FactDb.new
+# or
+facts = FactDb::Facts.new(config: custom_config)
+```
+
+## Attributes
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `config` | Config | Configuration instance |
+| `content_service` | ContentService | Service for content operations |
+| `entity_service` | EntityService | Service for entity operations |
+| `fact_service` | FactService | Service for fact operations |
+| `extraction_pipeline` | ExtractionPipeline | Pipeline for batch extraction |
+| `resolution_pipeline` | ResolutionPipeline | Pipeline for batch resolution |
+
+## Methods
+
+### initialize
+
+```ruby
+def initialize(config: nil)
+```
+
+Create a new Facts instance.
+
+**Parameters:**
+
+- `config` (Config, optional) - Configuration instance. Uses `FactDb.config` if not provided.
+
+**Example:**
+
+```ruby
+# Use default configuration
+facts = FactDb.new
+
+# Use custom configuration
+config = FactDb::Config.new
+config.database_url = "postgresql://localhost/my_db"
+facts = FactDb.new(config: config)
+```
+
+---
+
+### ingest
+
+```ruby
+def ingest(raw_text, type:, captured_at: Time.current, metadata: {}, title: nil, source_uri: nil)
+```
+
+Ingest raw content into the fact database.
+
+**Parameters:**
+
+- `raw_text` (String) - The content text
+- `type` (Symbol) - Content type (:email, :document, :article, etc.)
+- `captured_at` (Time, optional) - When content was captured
+- `metadata` (Hash, optional) - Additional metadata
+- `title` (String, optional) - Content title
+- `source_uri` (String, optional) - Original location
+
+**Returns:** `Models::Content`
+
+**Example:**
+
+```ruby
+content = facts.ingest(
+  "Paula joined Microsoft on Jan 10, 2024",
+  type: :announcement,
+  title: "New Hire",
+  captured_at: Time.current
+)
+```
+
+---
+
+### extract_facts
+
+```ruby
+def extract_facts(content_id, extractor: @config.default_extractor)
+```
+
+Extract facts from content.
+
+**Parameters:**
+
+- `content_id` (Integer) - Content ID
+- `extractor` (Symbol, optional) - Extraction method (:manual, :llm, :rule_based)
+
+**Returns:** `Array<Models::Fact>`
+
+**Example:**
+
+```ruby
+extracted = facts.extract_facts(content.id, extractor: :llm)
+```
+
+---
+
+### query_facts
+
+```ruby
+def query_facts(topic: nil, at: nil, entity: nil, status: :canonical)
+```
+
+Query facts with temporal and entity filtering.
+
+**Parameters:**
+
+- `topic` (String, optional) - Text search query
+- `at` (Date/Time, optional) - Point in time (nil = current)
+- `entity` (Integer, optional) - Entity ID filter
+- `status` (Symbol, optional) - Fact status filter
+
+**Returns:** `ActiveRecord::Relation<Models::Fact>`
+
+**Example:**
+
+```ruby
+# Current facts about Paula
+results = facts.query_facts(entity: paula.id)
+
+# Facts on a topic
+results = facts.query_facts(topic: "engineering")
+
+# Historical query
+results = facts.query_facts(at: Date.parse("2023-06-15"))
+```
+
+---
+
+### resolve_entity
+
+```ruby
+def resolve_entity(name, type: nil)
+```
+
+Resolve a name to an entity.
+
+**Parameters:**
+
+- `name` (String) - Name to resolve
+- `type` (Symbol, optional) - Entity type filter
+
+**Returns:** `Models::Entity` or `nil`
+
+**Example:**
+
+```ruby
+entity = facts.resolve_entity("Paula Chen", type: :person)
+```
+
+---
+
+### timeline_for
+
+```ruby
+def timeline_for(entity_id, from: nil, to: nil)
+```
+
+Build a timeline for an entity.
+
+**Parameters:**
+
+- `entity_id` (Integer) - Entity ID
+- `from` (Date/Time, optional) - Start of range
+- `to` (Date/Time, optional) - End of range
+
+**Returns:** `Array<Models::Fact>`
+
+**Example:**
+
+```ruby
+timeline = facts.timeline_for(paula.id, from: "2023-01-01", to: "2024-12-31")
+```
+
+---
+
+### current_facts_for
+
+```ruby
+def current_facts_for(entity_id)
+```
+
+Get currently valid facts about an entity.
+
+**Parameters:**
+
+- `entity_id` (Integer) - Entity ID
+
+**Returns:** `ActiveRecord::Relation<Models::Fact>`
+
+**Example:**
+
+```ruby
+current = facts.current_facts_for(paula.id)
+```
+
+---
+
+### facts_at
+
+```ruby
+def facts_at(at, entity: nil, topic: nil)
+```
+
+Get facts valid at a specific point in time.
+
+**Parameters:**
+
+- `at` (Date/Time) - Point in time
+- `entity` (Integer, optional) - Entity ID filter
+- `topic` (String, optional) - Text search query
+
+**Returns:** `ActiveRecord::Relation<Models::Fact>`
+
+**Example:**
+
+```ruby
+historical = facts.facts_at(Date.parse("2023-06-15"), entity: paula.id)
+```
+
+---
+
+### batch_extract
+
+```ruby
+def batch_extract(content_ids, extractor: @config.default_extractor, parallel: true)
+```
+
+Batch extract facts from multiple content items.
+
+**Parameters:**
+
+- `content_ids` (Array<Integer>) - Content IDs to process
+- `extractor` (Symbol, optional) - Extraction method
+- `parallel` (Boolean, optional) - Use parallel processing (default: true)
+
+**Returns:** `Array<Hash>` - Results per content
+
+**Example:**
+
+```ruby
+results = facts.batch_extract([c1.id, c2.id, c3.id], parallel: true)
+results.each do |r|
+  puts "#{r[:content_id]}: #{r[:facts].count} facts"
+end
+```
+
+---
+
+### batch_resolve_entities
+
+```ruby
+def batch_resolve_entities(names, type: nil)
+```
+
+Batch resolve entity names.
+
+**Parameters:**
+
+- `names` (Array<String>) - Names to resolve
+- `type` (Symbol, optional) - Entity type filter
+
+**Returns:** `Array<Hash>` - Resolution results
+
+**Example:**
+
+```ruby
+results = facts.batch_resolve_entities(["Paula", "Microsoft"])
+```
+
+---
+
+### detect_fact_conflicts
+
+```ruby
+def detect_fact_conflicts(entity_ids)
+```
+
+Detect fact conflicts for multiple entities.
+
+**Parameters:**
+
+- `entity_ids` (Array<Integer>) - Entity IDs to check
+
+**Returns:** `Array<Hash>` - Conflict detection results
+
+**Example:**
+
+```ruby
+conflicts = facts.detect_fact_conflicts([paula.id, john.id])
+```