fact_db 0.0.1

data/docs/guides/temporal-queries.md

@@ -0,0 +1,315 @@
+# Temporal Queries
+
+FactDb's temporal query system lets you retrieve facts across time - what's true now, what was true then, and how things changed.
+
+## Current Facts
+
+Get facts that are valid right now:
+
+```ruby
+facts = FactDb.new
+
+# All currently valid canonical facts
+current = facts.query_facts(status: :canonical)
+
+# Current facts about a specific entity
+current_about_paula = facts.current_facts_for(paula.id)
+
+# Current facts on a topic
+engineering_facts = facts.query_facts(topic: "engineering", status: :canonical)
+```
+
+## Point-in-Time Queries
+
+What was true at a specific moment:
+
+```ruby
+# What did we know about Paula on June 15, 2023?
+historical = facts.facts_at(
+  Date.parse("2023-06-15"),
+  entity: paula.id
+)
+
+# What did we know about Microsoft on Jan 1, 2024?
+microsoft_facts = facts.facts_at(
+  Date.parse("2024-01-01"),
+  entity: microsoft.id
+)
+```
+
+## Time Range Queries
+
+Facts active during a period:
+
+```ruby
+# Facts valid during Q4 2023
+q4_facts = facts.fact_service.query(
+  from: Date.parse("2023-10-01"),
+  to: Date.parse("2023-12-31")
+)
+
+# Paula's employment history for 2023
+paula_2023 = facts.fact_service.query(
+  entity: paula.id,
+  from: Date.parse("2023-01-01"),
+  to: Date.parse("2023-12-31")
+)
+```
+
+## Timelines
+
+Build complete timelines for entities:
+
+```ruby
+# Full timeline
+timeline = facts.timeline_for(paula.id)
+
+timeline.each do |fact|
+  range = fact.invalid_at ? "#{fact.valid_at} - #{fact.invalid_at}" : "#{fact.valid_at} - present"
+  puts "#{range}: #{fact.fact_text}"
+end
+
+# Timeline for specific period
+timeline = facts.timeline_for(
+  paula.id,
+  from: Date.parse("2023-01-01"),
+  to: Date.parse("2024-12-31")
+)
+```
+
+### Timeline Output Example
+
+```
+2022-03-15 - 2023-01-09: Paula Chen is Software Engineer at Company
+2023-01-10 - 2024-01-09: Paula Chen is Senior Engineer at Company
+2024-01-10 - present: Paula Chen is Principal Engineer at Microsoft
+```
+
+## Filtering by Status
+
+Query facts by their status:
+
+```ruby
+# Only canonical (current authoritative) facts
+canonical = facts.query_facts(status: :canonical)
+
+# Only corroborated (confirmed by multiple sources) facts
+corroborated = facts.query_facts(status: :corroborated)
+
+# Include both canonical and corroborated
+trusted = facts.query_facts(status: [:canonical, :corroborated])
+
+# Superseded facts (historical)
+superseded = facts.query_facts(status: :superseded)
+
+# Synthesized facts (derived)
+synthesized = facts.query_facts(status: :synthesized)
+```
+
+## Topic Search
+
+Search facts by text content:
+
+```ruby
+# Full-text search
+engineering_facts = facts.query_facts(topic: "engineering")
+
+# Combined with entity filter
+paula_engineering = facts.query_facts(
+  entity: paula.id,
+  topic: "promotion"
+)
+
+# Combined with time filter
+recent_engineering = facts.query_facts(
+  topic: "engineering",
+  at: Date.today
+)
+```
+
+## Advanced Queries
+
+### Using Scopes
+
+```ruby
+# Direct ActiveRecord queries on Fact model
+facts = FactDb::Models::Fact
+  .canonical
+  .currently_valid
+  .mentioning_entity(paula.id)
+  .search_text("engineer")
+  .order(valid_at: :desc)
+```
+
+### Available Scopes
+
+| Scope | Description |
+|-------|-------------|
+| `canonical` | Status is 'canonical' |
+| `currently_valid` | invalid_at is nil |
+| `valid_at(date)` | Valid at specific date |
+| `valid_during(from, to)` | Valid during range |
+| `mentioning_entity(id)` | Mentions specific entity |
+| `search_text(query)` | Full-text search |
+| `by_extraction_method(method)` | Filter by extractor |
+| `high_confidence` | Confidence > 0.8 |
+
+### Combining Scopes
+
+```ruby
+# High-confidence facts about Paula currently valid
+facts = FactDb::Models::Fact
+  .mentioning_entity(paula.id)
+  .canonical
+  .currently_valid
+  .high_confidence
+
+# LLM-extracted facts from last month
+facts = FactDb::Models::Fact
+  .by_extraction_method('llm')
+  .where('created_at > ?', 1.month.ago)
+```
+
+## Semantic Search
+
+Search by meaning using embeddings:
+
+```ruby
+# Find facts semantically similar to a query
+similar_facts = facts.fact_service.semantic_search(
+  "Paula's career progression",
+  limit: 10
+)
+
+# Combined with entity filter
+similar_about_paula = facts.fact_service.semantic_search(
+  "job title changes",
+  entity: paula.id,
+  limit: 5
+)
+```
+
+## Query Results
+
+### Fact Attributes
+
+```ruby
+fact = facts.query_facts(entity: paula.id).first
+
+fact.fact_text        # The assertion text
+fact.valid_at         # When it became true
+fact.invalid_at       # When it stopped (nil if current)
+fact.status           # canonical, superseded, etc.
+fact.confidence       # 0.0 to 1.0
+fact.extraction_method # manual, llm, rule_based
+fact.metadata         # Additional data
+```
+
+### Related Data
+
+```ruby
+# Entity mentions
+fact.entity_mentions.each do |mention|
+  puts "#{mention.entity.canonical_name} (#{mention.mention_role})"
+end
+
+# Source content
+fact.fact_sources.each do |source|
+  puts "Source: #{source.content.title}"
+  puts "Excerpt: #{source.excerpt}"
+end
+
+# Superseding fact
+if fact.superseded?
+  new_fact = fact.superseded_by
+  puts "Superseded by: #{new_fact.fact_text}"
+end
+
+# Source facts (for synthesized)
+if fact.synthesized?
+  fact.derived_from_ids.each do |id|
+    source = FactDb::Models::Fact.find(id)
+    puts "Derived from: #{source.fact_text}"
+  end
+end
+```
+
+## Performance Tips
+
+### Use Indexes
+
+The temporal indexes are optimized for:
+
+```ruby
+# These queries are fast
+facts.facts_at(Date.today)
+facts.query_facts(entity: id, status: :canonical)
+```
+
+### Limit Results
+
+```ruby
+# Always limit when possible
+queried = facts.fact_service.query(
+  entity: paula.id,
+  limit: 100
+)
+```
+
+### Eager Load Associations
+
+```ruby
+facts = FactDb::Models::Fact
+  .includes(:entity_mentions, :fact_sources)
+  .mentioning_entity(paula.id)
+```
+
+### Use Count for Totals
+
+```ruby
+# Don't load all records just to count
+total = FactDb::Models::Fact.canonical.currently_valid.count
+```
+
+## Common Patterns
+
+### Before/After Comparison
+
+```ruby
+# What changed for Paula?
+before = facts.facts_at(Date.parse("2023-12-31"), entity: paula.id)
+after = facts.facts_at(Date.parse("2024-01-31"), entity: paula.id)
+
+# Find differences
+new_facts = after - before
+```
+
+### Audit Trail
+
+```ruby
+# Get complete history of a topic
+all_facts = FactDb::Models::Fact
+  .mentioning_entity(paula.id)
+  .search_text("title")
+  .order(valid_at: :asc)
+
+all_facts.each do |fact|
+  status_info = fact.superseded? ? "(superseded)" : "(current)"
+  puts "#{fact.valid_at}: #{fact.fact_text} #{status_info}"
+end
+```
+
+### Change Detection
+
+```ruby
+# Find facts that changed recently
+recently_superseded = FactDb::Models::Fact
+  .where(status: 'superseded')
+  .where('invalid_at > ?', 1.week.ago)
+  .includes(:superseded_by)
+
+recently_superseded.each do |old_fact|
+  puts "Changed: #{old_fact.fact_text}"
+  puts "To: #{old_fact.superseded_by.fact_text}"
+end
+```

data/docs/index.md

@@ -0,0 +1,121 @@
+# FactDb
+
+> [!CAUTION]
+> This gem is under active development. APIs and features may change without notice. See the [CHANGELOG](https://github.com/MadBomber/fact_db/blob/main/CHANGELOG.md) for details.
+
+<table>
+<tr>
+<td width="50%" align="center" valign="top">
+<img src="assets/images/fact_db.jpg" alt="FactDb"><br>
+<em>"Do you swear to add the facts and only the facts?"</em>
+</td>
+<td width="50%" valign="top">
+FactDb implements the Event Clock concept - a powerful approach to capturing organizational knowledge through temporal facts. Every fact has explicit validity periods (<code>valid_at</code>/<code>invalid_at</code>) so you always know when information became true and when it changed.<br><br>
+The system resolves entity mentions to canonical identities, supports aliases and fuzzy matching, and maintains complete audit trails linking every fact back to its source content. Whether you're tracking employee roles, organizational changes, or any evolving information, FactDb gives you a queryable timeline of truth.
+</td>
+</tr>
+</table>
+
+## Key Features
+
+- **Temporal Facts** - Track facts with validity periods, knowing when information became true and when it changed
+- **Entity Resolution** - Resolve mentions to canonical entities with alias support and fuzzy matching
+- **Audit Trails** - Every fact links back to source content for full provenance
+- **Multiple Extractors** - Extract facts manually, via LLM, or with rule-based patterns
+- **Semantic Search** - Built on PostgreSQL with pgvector for vector similarity search
+- **Concurrent Processing** - Batch process content with parallel pipelines
+
+## Quick Example
+
+```ruby
+require 'fact_db'
+
+# Initialize the facts instance
+facts = FactDb.new
+
+# Ingest content
+content = facts.ingest(
+  "Paula Chen joined Microsoft as Principal Engineer on January 10, 2024.",
+  type: :email,
+  captured_at: Time.current
+)
+
+# Extract facts using LLM
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+# Query current facts about Paula
+current_facts = facts.query_facts(entity: paula.id)
+
+# Query facts valid at a specific point in time
+historical_facts = facts.facts_at(Date.parse("2023-06-15"), entity: paula.id)
+
+# Build a timeline
+timeline = facts.timeline_for(paula.id)
+```
+
+## The Event Clock Concept
+
+The Event Clock model organizes information into three layers:
+
+```mermaid
+graph TB
+    subgraph Content["Content Layer (Immutable)"]
+        C1[Email]
+        C2[Document]
+        C3[News Article]
+    end
+
+    subgraph Entities["Entity Layer"]
+        E1[Paula Chen]
+        E2[Microsoft]
+    end
+
+    subgraph Facts["Fact Layer (Temporal)"]
+        F1["Paula is Principal Engineer<br/>valid: 2024-01-10 to present"]
+        F2["Paula works at Microsoft<br/>valid: 2024-01-10 to present"]
+    end
+
+    C1 --> F1
+    C1 --> F2
+    F1 --> E1
+    F2 --> E1
+    F2 --> E2
+
+    style C1 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style C2 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style C3 fill:#1E40AF,stroke:#1E3A8A,color:#FFFFFF
+    style E1 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style E2 fill:#047857,stroke:#065F46,color:#FFFFFF
+    style F1 fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+    style F2 fill:#B91C1C,stroke:#991B1B,color:#FFFFFF
+```
+
+1. **Content** - Immutable source documents that serve as evidence
+2. **Entities** - Resolved identities (people, organizations, places)
+3. **Facts** - Temporal assertions with validity periods
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem 'fact_db'
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+See the [Installation Guide](getting-started/installation.md) for detailed setup instructions.
+
+## Requirements
+
+- Ruby >= 3.0
+- PostgreSQL with pgvector extension
+- Optional: ruby_llm gem for LLM-powered extraction
+
+## License
+
+MIT License - Copyright (c) 2025 Dewayne VanHoozer

data/examples/README.md

@@ -0,0 +1,130 @@
+# FactDb Examples
+
+This directory contains demonstration programs showcasing the capabilities of the FactDb gem.
+
+## Prerequisites
+
+1. PostgreSQL database with pgvector extension
+2. Set the `DATABASE_URL` environment variable or use the default `postgres://localhost/fact_db_demo`
+3. Run migrations to set up the schema
+
+```bash
+bundle install
+DATABASE_URL=postgres://localhost/fact_db_demo rake db:migrate
+```
+
+## Examples
+
+### basic_usage.rb
+
+**Foundational introduction to FactDb**
+
+Demonstrates:
+- Configuring FactDb
+- Ingesting content (emails, documents)
+- Creating entities (people, organizations)
+- Creating facts with entity mentions
+- Basic fact queries
+- Getting system statistics
+
+```bash
+ruby examples/basic_usage.rb
+```
+
+### entity_management.rb
+
+**Deep dive into entity operations**
+
+Demonstrates:
+- Creating entities with multiple types (person, organization, place)
+- Managing aliases (names, emails, abbreviations)
+- Entity resolution using fuzzy matching
+- Merging duplicate entities
+- Searching entities by name and type
+- Building entity timelines
+
+```bash
+ruby examples/entity_management.rb
+```
+
+### temporal_queries.rb
+
+**Working with time-based data**
+
+Demonstrates:
+- Creating facts with temporal bounds (valid_at, invalid_at)
+- Point-in-time queries ("What was true on date X?")
+- Distinguishing current vs historical facts
+- Superseding facts (replacing old information with new)
+- Building temporal timelines
+- Computing diffs between time periods
+- Querying facts by entity role
+
+```bash
+ruby examples/temporal_queries.rb
+```
+
+### rule_based_extraction.rb
+
+**Automatic fact extraction from text**
+
+Demonstrates:
+- Using the rule-based extractor
+- Pattern detection for employment, relationships, locations
+- Processing extraction results
+- Saving extracted facts to the database
+- Entity auto-creation from extracted text
+- Testing individual extraction patterns
+
+```bash
+ruby examples/rule_based_extraction.rb
+```
+
+### hr_system.rb
+
+**Practical HR knowledge management system**
+
+A comprehensive real-world example demonstrating:
+- Organizational hierarchy (company, departments, locations)
+- Employee profile management
+- Recording employment history
+- Processing promotions with fact supersession
+- Recording employee transfers
+- Historical queries ("What was X's role in 2024?")
+- Organization chart queries
+- Complete audit trails with source documents
+- HR statistics and reporting
+
+```bash
+ruby examples/hr_system.rb
+```
+
+## Key Concepts
+
+### The Event Clock Pattern
+
+FactDb implements the Event Clock concept where:
+- Every fact has a `valid_at` timestamp (when it became true)
+- Facts may have an `invalid_at` timestamp (when they stopped being true)
+- This enables temporal queries at any point in time
+
+### Entity Resolution
+
+Entities can have multiple aliases and are resolved using fuzzy matching:
+- "Bob Johnson" and "Robert Johnson" can resolve to the same entity
+- Duplicates can be merged while preserving audit history
+
+### Fact Lifecycle
+
+Facts progress through states:
+- `canonical` - Currently accepted as true
+- `superseded` - Replaced by a newer fact
+- `corroborated` - Supported by other evidence
+- `synthesized` - Derived from multiple sources
+
+### Source Tracking
+
+All facts link back to source content:
+- Primary sources (direct evidence)
+- Supporting sources (additional evidence)
+- Corroborating sources (independent confirmation)