fact_db 0.0.1

data/docs/api/models/index.md

@@ -0,0 +1,77 @@
+# Models
+
+FactDb uses ActiveRecord models for data persistence.
+
+## Core Models
+
+- [Content](content.md) - Immutable source documents
+- [Entity](entity.md) - Resolved identities with aliases
+- [Fact](fact.md) - Temporal assertions
+
+## Supporting Models
+
+### EntityAlias
+
+Stores alternative names for entities.
+
+```ruby
+class EntityAlias < ActiveRecord::Base
+  belongs_to :entity
+end
+```
+
+| Column | Type | Description |
+|--------|------|-------------|
+| entity_id | bigint | Parent entity |
+| alias_text | string | Alternative name |
+| alias_type | string | Type (nickname, abbreviation, etc.) |
+| confidence | float | Match confidence |
+
+### EntityMention
+
+Links facts to mentioned entities.
+
+```ruby
+class EntityMention < ActiveRecord::Base
+  belongs_to :fact
+  belongs_to :entity
+end
+```
+
+| Column | Type | Description |
+|--------|------|-------------|
+| fact_id | bigint | Parent fact |
+| entity_id | bigint | Referenced entity |
+| mention_text | string | How entity was mentioned |
+| mention_role | string | Role (subject, object, etc.) |
+| confidence | float | Resolution confidence |
+
+### FactSource
+
+Links facts to source content.
+
+```ruby
+class FactSource < ActiveRecord::Base
+  belongs_to :fact
+  belongs_to :content
+end
+```
+
+| Column | Type | Description |
+|--------|------|-------------|
+| fact_id | bigint | Parent fact |
+| content_id | bigint | Source content |
+| source_type | string | Type (primary, supporting, contradicting) |
+| excerpt | text | Relevant text excerpt |
+| confidence | float | Source confidence |
+
+## Model Relationships
+
+```mermaid
+erDiagram
+    Content ||--o{ FactSource : "sourced by"
+    Entity ||--o{ EntityAlias : "has"
+    Entity ||--o{ EntityMention : "mentioned in"
+    Fact ||--o{ EntityMention : "mentions"
+    Fact ||--o{ FactSource : "sourced from"
+```

data/docs/api/pipeline/extraction.md

@@ -0,0 +1,175 @@
+# ExtractionPipeline
+
+Concurrent fact extraction from multiple content items.
+
+## Class: `FactDb::Pipeline::ExtractionPipeline`
+
+```ruby
+pipeline = FactDb::Pipeline::ExtractionPipeline.new(config)
+```
+
+## Methods
+
+### process
+
+```ruby
+def process(contents, extractor: config.default_extractor)
+```
+
+Process content items sequentially.
+
+**Parameters:**
+
+- `contents` (Array<Content>) - Content records
+- `extractor` (Symbol) - Extraction method
+
+**Returns:** `Array<Hash>`
+
+**Example:**
+
+```ruby
+contents = Models::Content.where(id: [1, 2, 3])
+results = pipeline.process(contents, extractor: :llm)
+```
+
+---
+
+### process_parallel
+
+```ruby
+def process_parallel(contents, extractor: config.default_extractor)
+```
+
+Process content items concurrently.
+
+**Parameters:**
+
+- `contents` (Array<Content>) - Content records
+- `extractor` (Symbol) - Extraction method
+
+**Returns:** `Array<Hash>`
+
+**Example:**
+
+```ruby
+results = pipeline.process_parallel(contents, extractor: :llm)
+
+results.each do |result|
+  puts "Content #{result[:content_id]}:"
+  puts "  Facts: #{result[:facts].count}"
+  puts "  Error: #{result[:error]}" if result[:error]
+end
+```
+
+## Pipeline Steps
+
+### Sequential Pipeline
+
+```mermaid
+graph LR
+    A[Content] --> B[Validate]
+    B --> C[Extract]
+    C --> D[Validate Facts]
+    D --> E[Results]
+
+    style A fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style B fill:#B45309,stroke:#92400E,color:#FFFFFF
+    style C fill:#047857,stroke:#065F46,color:#FFFFFF
+    style D fill:#B45309,stroke:#92400E,color:#FFFFFF
+    style E fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+```
+
+1. **Validate** - Check content is not empty
+2. **Extract** - Run extractor
+3. **Validate Facts** - Filter valid facts
+4. **Results** - Return extracted facts
+
+### Parallel Pipeline
+
+```mermaid
+graph TB
+    subgraph Parallel
+        A1[Content 1] --> E1[Extract 1]
+        A2[Content 2] --> E2[Extract 2]
+        A3[Content 3] --> E3[Extract 3]
+    end
+    E1 --> Aggregate
+    E2 --> Aggregate
+    E3 --> Aggregate
+
+    style A1 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style A2 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style A3 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style E1 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style E2 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style E3 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style Aggregate fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+```
+
+## Result Structure
+
+```ruby
+{
+  content_id: 123,
+  facts: [<Fact>, <Fact>, ...],  # Extracted facts
+  error: nil                      # Error message if failed
+}
+```
+
+## Usage via Facts
+
+```ruby
+facts = FactDb.new
+
+# Sequential
+results = facts.batch_extract(content_ids, parallel: false)
+
+# Parallel (default)
+results = facts.batch_extract(content_ids, parallel: true)
+```
+
+## Error Handling
+
+The pipeline catches errors per-item:
+
+```ruby
+results = pipeline.process_parallel(contents)
+
+results.each do |result|
+  if result[:error]
+    logger.error "Content #{result[:content_id]}: #{result[:error]}"
+  else
+    logger.info "Content #{result[:content_id]}: #{result[:facts].count} facts"
+  end
+end
+```
+
+## Performance
+
+### Batch Size
+
+Optimal batch size depends on:
+
+- Extractor type (LLM has rate limits)
+- Content length
+- System resources
+
+```ruby
+# Process in optimal batches
+contents.each_slice(25) do |batch|
+  results = pipeline.process_parallel(batch)
+  process_results(results)
+end
+```
+
+### Memory
+
+For large batches, process and discard:
+
+```ruby
+contents.each_slice(50) do |batch|
+  results = pipeline.process_parallel(batch)
+  save_facts(results.flat_map { |r| r[:facts] })
+  # Results discarded after each batch
+end
+```

data/docs/api/pipeline/index.md

@@ -0,0 +1,72 @@
+# Pipeline
+
+Pipelines provide concurrent processing for batch operations using SimpleFlow.
+
+## Available Pipelines
+
+- [ExtractionPipeline](extraction.md) - Concurrent fact extraction
+- [ResolutionPipeline](resolution.md) - Parallel entity resolution
+
+## SimpleFlow Integration
+
+Pipelines are built on the `simple_flow` gem:
+
+```ruby
+require 'simple_flow'
+
+pipeline = SimpleFlow::Pipeline.new do
+  step ->(result) { result.continue(transformed_value) }
+  step ->(result) { result.continue(more_transformation) }
+end
+
+result = pipeline.call(SimpleFlow::Result.new(initial_value))
+```
+
+## Pipeline Pattern
+
+All pipelines follow a common structure:
+
+```ruby
+class SomePipeline
+  attr_reader :config
+
+  def initialize(config = FactDb.config)
+    @config = config
+  end
+
+  def process(items, **options)
+    # Sequential processing
+  end
+
+  def process_parallel(items, **options)
+    # Parallel processing
+  end
+end
+```
+
+## Result Structure
+
+Pipeline results follow a consistent format:
+
+```ruby
+{
+  content_id: 123,          # Item identifier
+  facts: [<Fact>, ...],     # Extracted/resolved items
+  error: nil                # Error message if failed
+}
+```
+
+## Error Handling
+
+Pipelines handle errors gracefully:
+
+```ruby
+results = pipeline.process_parallel(items)
+
+successful = results.select { |r| r[:error].nil? }
+failed = results.reject { |r| r[:error].nil? }
+
+failed.each do |result|
+  logger.error "Failed: #{result[:error]}"
+end
+```

data/docs/api/pipeline/resolution.md

@@ -0,0 +1,209 @@
+# ResolutionPipeline
+
+Parallel entity resolution and conflict detection.
+
+## Class: `FactDb::Pipeline::ResolutionPipeline`
+
+```ruby
+pipeline = FactDb::Pipeline::ResolutionPipeline.new(config)
+```
+
+## Methods
+
+### resolve_entities
+
+```ruby
+def resolve_entities(names, type: nil)
+```
+
+Resolve multiple entity names in parallel.
+
+**Parameters:**
+
+- `names` (Array<String>) - Names to resolve
+- `type` (Symbol) - Optional entity type filter
+
+**Returns:** `Array<Hash>`
+
+**Example:**
+
+```ruby
+names = ["Paula Chen", "Microsoft", "Seattle"]
+results = pipeline.resolve_entities(names)
+
+results.each do |result|
+  puts "#{result[:name]}: #{result[:status]}"
+  puts "  Entity: #{result[:entity]&.canonical_name}"
+end
+```
+
+---
+
+### detect_conflicts
+
+```ruby
+def detect_conflicts(entity_ids)
+```
+
+Find fact conflicts for multiple entities in parallel.
+
+**Parameters:**
+
+- `entity_ids` (Array<Integer>) - Entity IDs to check
+
+**Returns:** `Array<Hash>`
+
+**Example:**
+
+```ruby
+results = pipeline.detect_conflicts([paula.id, john.id])
+
+results.each do |result|
+  puts "Entity #{result[:entity_id]}: #{result[:conflict_count]} conflicts"
+  result[:conflicts].each do |c|
+    puts "  - #{c[:fact1].fact_text} vs #{c[:fact2].fact_text}"
+  end
+end
+```
+
+## Result Structures
+
+### Resolution Result
+
+```ruby
+{
+  name: "Paula Chen",
+  entity: <Entity>,           # Resolved entity or nil
+  status: :resolved,          # :resolved, :not_found, :error
+  error: nil                  # Error message if failed
+}
+```
+
+### Conflict Detection Result
+
+```ruby
+{
+  entity_id: 123,
+  conflicts: [
+    {
+      fact1: <Fact>,
+      fact2: <Fact>,
+      similarity: 0.75
+    }
+  ],
+  conflict_count: 1
+}
+```
+
+## Pipeline Steps
+
+### Entity Resolution Pipeline
+
+```mermaid
+graph TB
+    subgraph Parallel
+        N1[Name 1] --> R1[Resolve 1]
+        N2[Name 2] --> R2[Resolve 2]
+        N3[Name 3] --> R3[Resolve 3]
+    end
+    R1 --> Aggregate
+    R2 --> Aggregate
+    R3 --> Aggregate
+
+    style N1 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style N2 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style N3 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style R1 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style R2 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style R3 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style Aggregate fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+```
+
+### Conflict Detection Pipeline
+
+```mermaid
+graph TB
+    subgraph Parallel
+        E1[Entity 1] --> C1[Find Conflicts 1]
+        E2[Entity 2] --> C2[Find Conflicts 2]
+    end
+    C1 --> Aggregate
+    C2 --> Aggregate
+
+    style E1 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style E2 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style C1 fill:#B45309,stroke:#92400E,color:#FFFFFF
+    style C2 fill:#B45309,stroke:#92400E,color:#FFFFFF
+    style Aggregate fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+```
+
+## Usage via Facts
+
+```ruby
+facts = FactDb.new
+
+# Resolve entities
+results = facts.batch_resolve_entities(["Paula", "Microsoft"])
+
+# Detect conflicts
+results = facts.detect_fact_conflicts([entity1.id, entity2.id])
+```
+
+## Resolution Strategies
+
+The pipeline uses the EntityResolver which tries:
+
+1. **Exact match** on canonical name
+2. **Alias match** on registered aliases
+3. **Fuzzy match** using Levenshtein distance
+
+```ruby
+FactDb.configure do |config|
+  config.fuzzy_match_threshold = 0.85
+end
+```
+
+## Error Handling
+
+```ruby
+results = pipeline.resolve_entities(names)
+
+# Handle unresolved names
+unresolved = results.select { |r| r[:status] == :not_found }
+unresolved.each do |result|
+  # Optionally create new entities
+  entity = facts.entity_service.create(
+    result[:name],
+    type: :person,
+    metadata: { needs_review: true }
+  )
+end
+
+# Handle errors
+errors = results.select { |r| r[:status] == :error }
+errors.each do |result|
+  logger.error "Resolution failed for #{result[:name]}: #{result[:error]}"
+end
+```
+
+## Performance Tips
+
+### Batch Size
+
+```ruby
+# Process in batches for large name lists
+names.each_slice(100) do |batch|
+  results = pipeline.resolve_entities(batch)
+  process_results(results)
+end
+```
+
+### Pre-warm Cache
+
+```ruby
+# Load entities into memory first
+Entity.where(entity_type: 'person').to_a
+
+# Then resolve
+results = pipeline.resolve_entities(person_names, type: :person)
+```

data/docs/api/services/content-service.md

@@ -0,0 +1,166 @@
+# ContentService
+
+Service for ingesting and managing source content.
+
+## Class: `FactDb::Services::ContentService`
+
+```ruby
+service = FactDb::Services::ContentService.new(config)
+```
+
+## Methods
+
+### create
+
+```ruby
+def create(raw_text, type:, captured_at: Time.current, metadata: {}, title: nil, source_uri: nil)
+```
+
+Create new content with automatic deduplication.
+
+**Parameters:**
+
+- `raw_text` (String) - Content text
+- `type` (Symbol) - Content type
+- `captured_at` (Time) - Capture timestamp
+- `metadata` (Hash) - Additional metadata
+- `title` (String) - Optional title
+- `source_uri` (String) - Original location
+
+**Returns:** `Models::Content`
+
+**Example:**
+
+```ruby
+content = service.create(
+  "Email body text...",
+  type: :email,
+  title: "RE: Important",
+  metadata: { from: "sender@example.com" }
+)
+```
+
+---
+
+### find
+
+```ruby
+def find(id)
+```
+
+Find content by ID.
+
+**Returns:** `Models::Content`
+
+---
+
+### find_by_hash
+
+```ruby
+def find_by_hash(hash)
+```
+
+Find content by SHA256 hash.
+
+**Returns:** `Models::Content` or `nil`
+
+**Example:**
+
+```ruby
+hash = Digest::SHA256.hexdigest(text)
+content = service.find_by_hash(hash)
+```
+
+---
+
+### search
+
+```ruby
+def search(query, limit: 20)
+```
+
+Full-text search content.
+
+**Parameters:**
+
+- `query` (String) - Search query
+- `limit` (Integer) - Max results
+
+**Returns:** `Array<Models::Content>`
+
+**Example:**
+
+```ruby
+results = service.search("quarterly report", limit: 10)
+```
+
+---
+
+### semantic_search
+
+```ruby
+def semantic_search(query, limit: 10)
+```
+
+Semantic similarity search using embeddings.
+
+**Parameters:**
+
+- `query` (String) - Search query
+- `limit` (Integer) - Max results
+
+**Returns:** `Array<Models::Content>`
+
+**Example:**
+
+```ruby
+results = service.semantic_search("financial performance")
+```
+
+---
+
+### by_type
+
+```ruby
+def by_type(type)
+```
+
+Filter content by type.
+
+**Returns:** `ActiveRecord::Relation`
+
+**Example:**
+
+```ruby
+emails = service.by_type(:email)
+```
+
+---
+
+### recent
+
+```ruby
+def recent(limit: 20)
+```
+
+Get recently captured content.
+
+**Returns:** `Array<Models::Content>`
+
+---
+
+### mentioning_entity
+
+```ruby
+def mentioning_entity(entity_id)
+```
+
+Find content that mentions an entity (via facts).
+
+**Returns:** `Array<Models::Content>`
+
+**Example:**
+
+```ruby
+paula_content = service.mentioning_entity(paula.id)
+```