fact_db 0.0.1

data/docs/architecture/index.md

@@ -0,0 +1,149 @@
+# Architecture
+
+FactDb is designed around the Event Clock concept - a system for capturing organizational reasoning through temporal facts with full audit trails.
+
+## Design Principles
+
+### Immutable Content
+
+Source content is never modified after ingestion. This ensures:
+
+- Complete audit trails
+- Reproducible fact extraction
+- Historical accuracy
+
+### Temporal First
+
+Every fact has temporal bounds (`valid_at`, `invalid_at`). This enables:
+
+- Point-in-time queries
+- Change tracking
+- Historical analysis
+
+### Entity Resolution
+
+Names and mentions are resolved to canonical entities:
+
+- Reduces ambiguity
+- Enables cross-reference
+- Supports alias matching
+
+### Provenance
+
+Every fact links back to source content:
+
+- Verifiable assertions
+- Confidence tracking
+- Contradiction detection
+
+## System Architecture
+
+```mermaid
+graph TB
+    subgraph Input
+        Email[Email]
+        Doc[Documents]
+        News[News]
+        API[APIs]
+    end
+
+    subgraph FactDb["FactDb Core"]
+        CS[ContentService]
+        ES[EntityService]
+        FS[FactService]
+
+        subgraph Extractors
+            ME[ManualExtractor]
+            LE[LLMExtractor]
+            RE[RuleBasedExtractor]
+        end
+
+        subgraph Resolution
+            ER[EntityResolver]
+            FR[FactResolver]
+        end
+
+        subgraph Pipeline
+            EP[ExtractionPipeline]
+            RP[ResolutionPipeline]
+        end
+    end
+
+    subgraph Storage["PostgreSQL + pgvector"]
+        Contents[(Contents)]
+        Entities[(Entities)]
+        Facts[(Facts)]
+    end
+
+    Email --> CS
+    Doc --> CS
+    News --> CS
+    API --> CS
+
+    CS --> Contents
+    ES --> Entities
+    FS --> Facts
+
+    Extractors --> FS
+    Resolution --> ES
+    Resolution --> FS
+    Pipeline --> Extractors
+
+    style Email fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style Doc fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style News fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style API fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style CS fill:#B45309,stroke:#92400E,color:#FFFFFF
+    style ES fill:#B45309,stroke:#92400E,color:#FFFFFF
+    style FS fill:#B45309,stroke:#92400E,color:#FFFFFF
+    style ME fill:#047857,stroke:#065F46,color:#FFFFFF
+    style LE fill:#047857,stroke:#065F46,color:#FFFFFF
+    style RE fill:#047857,stroke:#065F46,color:#FFFFFF
+    style ER fill:#C2410C,stroke:#9A3412,color:#FFFFFF
+    style FR fill:#C2410C,stroke:#9A3412,color:#FFFFFF
+    style EP fill:#7C3AED,stroke:#6D28D9,color:#FFFFFF
+    style RP fill:#7C3AED,stroke:#6D28D9,color:#FFFFFF
+    style Contents fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+    style Entities fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+    style Facts fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+```
+
+## Components
+
+### Services
+
+- **ContentService** - Ingests and manages source content
+- **EntityService** - Creates and resolves entities
+- **FactService** - Extracts, creates, and queries facts
+
+### Extractors
+
+- **ManualExtractor** - API-driven fact creation
+- **LLMExtractor** - AI-powered extraction using OpenAI, Anthropic, etc.
+- **RuleBasedExtractor** - Pattern matching with regular expressions
+
+### Resolution
+
+- **EntityResolver** - Matches mentions to canonical entities
+- **FactResolver** - Handles supersession, synthesis, and conflicts
+
+### Pipeline
+
+- **ExtractionPipeline** - Concurrent fact extraction using SimpleFlow
+- **ResolutionPipeline** - Parallel entity resolution
+
+## Data Flow
+
+1. **Ingest** - Raw content enters the system
+2. **Store** - Content is hashed, deduplicated, and stored
+3. **Extract** - Facts are extracted via chosen method
+4. **Resolve** - Entity mentions are resolved to canonical entities
+5. **Link** - Facts are linked to sources and entities
+6. **Query** - Temporal queries retrieve relevant facts
+
+## Documentation
+
+- [Three-Layer Model](three-layer-model.md) - Content, Entities, Facts
+- [Temporal Facts](temporal-facts.md) - Validity periods and status
+- [Entity Resolution](entity-resolution.md) - Resolving mentions
+- [Database Schema](database-schema.md) - Table structure

data/docs/architecture/temporal-facts.md

@@ -0,0 +1,268 @@
+# Temporal Facts
+
+Temporal facts are the core innovation of FactDb - assertions with explicit validity periods that enable point-in-time queries.
+
+## Validity Periods
+
+Every fact has two timestamps:
+
+```ruby
+{
+  valid_at: "2024-01-10T00:00:00Z",   # When fact became true
+  invalid_at: "2024-06-01T00:00:00Z"  # When fact stopped being true (or nil)
+}
+```
+
+### Currently Valid
+
+Facts with `invalid_at: nil` are currently valid:
+
+```ruby
+# Paula is currently a Principal Engineer
+{
+  fact_text: "Paula Chen is Principal Engineer",
+  valid_at: "2024-01-10",
+  invalid_at: nil
+}
+```
+
+### Historical
+
+Facts with both dates represent historical periods:
+
+```ruby
+# Paula was Senior Engineer before promotion
+{
+  fact_text: "Paula Chen is Senior Engineer",
+  valid_at: "2022-03-15",
+  invalid_at: "2024-01-10"
+}
+```
+
+## Temporal Queries
+
+### Current Facts
+
+```ruby
+# Get facts valid right now
+current = facts.query_facts(
+  entity: paula.id,
+  status: :canonical
+)
+
+# Shorthand
+current = facts.current_facts_for(paula.id)
+```
+
+### Point-in-Time
+
+```ruby
+# What was true on a specific date?
+queried = facts.facts_at(
+  Date.parse("2023-06-15"),
+  entity: paula.id
+)
+```
+
+### Time Range
+
+```ruby
+# Facts active during a period
+queried = facts.fact_service.query(
+  entity: paula.id,
+  from: Date.parse("2023-01-01"),
+  to: Date.parse("2023-12-31")
+)
+```
+
+## Fact Status
+
+Facts move through different statuses as information evolves:
+
+```mermaid
+stateDiagram-v2
+    [*] --> canonical: Extracted
+    canonical --> corroborated: 2+ sources confirm
+    canonical --> superseded: New info replaces
+    corroborated --> superseded: Later replaced
+    canonical --> synthesized: Combined
+    superseded --> [*]
+
+    classDef blue fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    classDef green fill:#047857,stroke:#065F46,color:#FFFFFF
+    classDef red fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+    classDef yellow fill:#B45309,stroke:#92400E,color:#FFFFFF
+
+    class canonical blue
+    class corroborated green
+    class superseded red
+    class synthesized yellow
+```
+
+### Canonical
+
+The current authoritative version of a fact:
+
+```ruby
+fact = facts.fact_service.create(
+  "Paula Chen is Principal Engineer",
+  valid_at: Date.parse("2024-01-10"),
+  mentions: [{ entity: paula, role: "subject" }]
+)
+# fact.status => "canonical"
+```
+
+### Superseded
+
+When information changes, old facts are superseded:
+
+```ruby
+# Paula gets promoted
+new_fact = facts.fact_service.resolver.supersede(
+  old_fact.id,
+  "Paula Chen is Senior Principal Engineer",
+  valid_at: Date.parse("2024-06-01")
+)
+
+# old_fact.status => "superseded"
+# old_fact.invalid_at => "2024-06-01"
+# old_fact.superseded_by_id => new_fact.id
+```
+
+### Corroborated
+
+Facts confirmed by multiple independent sources:
+
+```ruby
+# Two sources say the same thing
+facts.fact_service.resolver.corroborate(fact.id, other_fact.id)
+facts.fact_service.resolver.corroborate(fact.id, third_fact.id)
+
+# After 2+ corroborations
+fact.reload
+# fact.status => "corroborated"
+# fact.corroborated_by_ids => [other_fact.id, third_fact.id]
+```
+
+### Synthesized
+
+Derived facts combine information from multiple sources:
+
+```ruby
+synthesized = facts.fact_service.resolver.synthesize(
+  [fact1.id, fact2.id, fact3.id],
+  "Paula worked at Microsoft from Jan 2024 to present, starting as Principal Engineer",
+  valid_at: Date.parse("2024-01-10")
+)
+
+# synthesized.status => "synthesized"
+# synthesized.derived_from_ids => [fact1.id, fact2.id, fact3.id]
+```
+
+## Timelines
+
+Build complete timelines for entities:
+
+```ruby
+timeline = facts.timeline_for(paula.id)
+
+# Returns chronological list of facts
+timeline.each do |entry|
+  puts "#{entry.valid_at}: #{entry.fact_text}"
+  puts "  Until: #{entry.invalid_at || 'present'}"
+end
+```
+
+### Timeline Example
+
+```
+2022-03-15: Paula Chen joined Company as Software Engineer
+  Until: 2023-01-10
+
+2023-01-10: Paula Chen promoted to Senior Engineer
+  Until: 2024-01-10
+
+2024-01-10: Paula Chen is Principal Engineer at Microsoft
+  Until: present
+```
+
+## Conflict Detection
+
+FactDb can detect potentially conflicting facts:
+
+```ruby
+conflicts = facts.fact_service.resolver.find_conflicts(
+  entity_id: paula.id,
+  topic: "title"
+)
+
+conflicts.each do |conflict|
+  puts "Potential conflict:"
+  puts "  Fact 1: #{conflict[:fact1].fact_text}"
+  puts "  Fact 2: #{conflict[:fact2].fact_text}"
+  puts "  Similarity: #{conflict[:similarity]}"
+end
+```
+
+### Resolving Conflicts
+
+```ruby
+# Keep one fact, supersede the others
+facts.fact_service.resolver.resolve_conflict(
+  keep_fact_id: correct_fact.id,
+  supersede_fact_ids: [wrong_fact.id],
+  reason: "Verified with HR records"
+)
+```
+
+## Best Practices
+
+### 1. Always Set valid_at
+
+```ruby
+# Good - explicit date
+facts.fact_service.create(
+  "Paula joined the team",
+  valid_at: Date.parse("2024-01-10")
+)
+
+# Avoid - implicit current time
+# valid_at defaults to Time.current if not provided
+```
+
+### 2. Supersede Don't Delete
+
+```ruby
+# Good - supersede old fact
+facts.fact_service.resolver.supersede(
+  old_fact.id,
+  "Updated information",
+  valid_at: Date.today
+)
+
+# Avoid - deleting facts
+# old_fact.destroy  # Loses audit trail
+```
+
+### 3. Track Confidence
+
+```ruby
+fact = facts.fact_service.create(
+  "Paula may be promoted soon",
+  valid_at: Date.today,
+  confidence: 0.6  # Lower confidence for speculation
+)
+```
+
+### 4. Link Sources
+
+```ruby
+fact = facts.fact_service.create(
+  "Paula is Principal Engineer",
+  valid_at: Date.parse("2024-01-10"),
+  sources: [
+    { content: email, type: "primary", excerpt: "...accepted the offer..." },
+    { content: announcement, type: "supporting" }
+  ]
+)
+```

data/docs/architecture/three-layer-model.md

@@ -0,0 +1,242 @@
+# Three-Layer Model
+
+FactDb organizes information into three distinct layers, each with specific responsibilities.
+
+## Overview
+
+```mermaid
+graph TB
+    subgraph Layer1["Layer 1: Content"]
+        C1[Immutable Documents]
+        C2[Source Evidence]
+        C3[Captured Timestamps]
+    end
+
+    subgraph Layer2["Layer 2: Entities"]
+        E1[Canonical Names]
+        E2[Aliases]
+        E3[Types]
+    end
+
+    subgraph Layer3["Layer 3: Facts"]
+        F1[Temporal Assertions]
+        F2[Validity Periods]
+        F3[Status Tracking]
+    end
+
+    Layer1 --> Layer3
+    Layer2 --> Layer3
+
+    style C1 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style C2 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style C3 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style E1 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style E2 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style E3 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style F1 fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+    style F2 fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+    style F3 fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+```
+
+## Layer 1: Content
+
+The content layer stores raw source material that serves as evidence for facts.
+
+### Characteristics
+
+| Property | Description |
+|----------|-------------|
+| Immutable | Content never changes after ingestion |
+| Deduplicated | SHA256 hash prevents duplicate storage |
+| Timestamped | `captured_at` records when content was obtained |
+| Typed | Categories like email, document, article |
+| Searchable | Full-text and semantic vector search |
+
+### Content Types
+
+```ruby
+# Common content types
+:email        # Email messages
+:document     # General documents
+:article      # News articles
+:transcript   # Meeting transcripts
+:report       # Reports and analysis
+:announcement # Official announcements
+:social       # Social media posts
+```
+
+### Example
+
+```ruby
+content = facts.ingest(
+  "Paula Chen accepted the offer for Principal Engineer...",
+  type: :email,
+  title: "RE: Offer Letter - Paula Chen",
+  source_uri: "mailto:hr@company.com/12345",
+  captured_at: Time.current,
+  metadata: {
+    from: "hr@company.com",
+    to: "hiring@company.com",
+    subject: "RE: Offer Letter - Paula Chen"
+  }
+)
+```
+
+## Layer 2: Entities
+
+Entities represent real-world things mentioned in content.
+
+### Entity Types
+
+| Type | Description | Examples |
+|------|-------------|----------|
+| `person` | Individual people | Paula Chen, John Smith |
+| `organization` | Companies, teams, groups | Microsoft, Platform Team |
+| `place` | Locations | San Francisco, Building A |
+| `product` | Products and services | Windows 11, Azure |
+| `event` | Named events | Q4 Earnings, Annual Review |
+
+### Resolution Status
+
+```mermaid
+stateDiagram-v2
+    [*] --> unresolved: Created
+    unresolved --> resolved: Confirmed identity
+    resolved --> merged: Duplicate found
+    merged --> [*]: Points to canonical
+
+    classDef blue fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    classDef green fill:#047857,stroke:#065F46,color:#FFFFFF
+    classDef red fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+
+    class unresolved blue
+    class resolved green
+    class merged red
+```
+
+- **unresolved** - Entity created but not confirmed
+- **resolved** - Entity identity confirmed
+- **merged** - Entity merged into another (canonical) entity
+
+### Aliases
+
+Entities can have multiple aliases for flexible matching:
+
+```ruby
+entity = facts.entity_service.create(
+  "Paula Chen",
+  type: :person,
+  aliases: [
+    "Paula",
+    "P. Chen",
+    "Chen, Paula"
+  ]
+)
+```
+
+## Layer 3: Facts
+
+Facts are temporal assertions about entities, extracted from content.
+
+### Fact Structure
+
+```ruby
+fact = Models::Fact.new(
+  fact_text: "Paula Chen is Principal Engineer at Microsoft",
+  valid_at: Date.parse("2024-01-10"),
+  invalid_at: nil,  # Still valid
+  status: "canonical",
+  confidence: 0.95,
+  extraction_method: "llm"
+)
+```
+
+### Temporal Bounds
+
+Every fact has:
+
+- `valid_at` - When the fact became true (required)
+- `invalid_at` - When the fact stopped being true (nil if current)
+
+```ruby
+# Currently valid fact
+fact1 = { valid_at: "2024-01-10", invalid_at: nil }
+
+# Historical fact
+fact2 = { valid_at: "2023-01-01", invalid_at: "2024-01-09" }
+
+# Point-in-time query
+facts.facts_at(Date.parse("2023-06-15"))  # Returns fact2
+facts.facts_at(Date.parse("2024-02-01"))  # Returns fact1
+```
+
+### Fact Status
+
+| Status | Description |
+|--------|-------------|
+| `canonical` | Current authoritative version |
+| `superseded` | Replaced by newer information |
+| `corroborated` | Confirmed by multiple sources |
+| `synthesized` | Derived from multiple facts |
+
+### Relationships
+
+Facts connect to both content and entities:
+
+```mermaid
+graph LR
+    C[Content] -->|fact_sources| F[Fact]
+    F -->|entity_mentions| E1[Entity 1]
+    F -->|entity_mentions| E2[Entity 2]
+
+    style C fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style F fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+    style E1 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style E2 fill:#047857,stroke:#065F46,color:#FFFFFF
+```
+
+## Layer Interactions
+
+### Content to Facts
+
+Facts are extracted from content and maintain source links:
+
+```ruby
+# Extract facts from content
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+# Each fact links back to source
+extracted.first.fact_sources.each do |source|
+  puts source.content.title
+  puts source.excerpt
+end
+```
+
+### Entities to Facts
+
+Facts mention entities with specific roles:
+
+```ruby
+fact.entity_mentions.each do |mention|
+  puts "#{mention.entity.canonical_name}: #{mention.mention_role}"
+end
+# Output:
+# Paula Chen: subject
+# Microsoft: organization
+# Principal Engineer: role
+```
+
+### Cross-Layer Queries
+
+Query across all layers:
+
+```ruby
+# Find all content about an entity
+contents = facts.content_service.mentioning_entity(paula.id)
+
+# Find all entities mentioned in content
+entities = facts.entity_service.in_content(content.id)
+
+# Find all facts from a specific source
+source_facts = facts.fact_service.from_content(content.id)
+```