fact_db 0.0.1

data/docs/guides/entity-management.md

@@ -0,0 +1,364 @@
+# Entity Management
+
+Entities represent real-world things mentioned in facts - people, organizations, places, and more.
+
+## Creating Entities
+
+### Basic Creation
+
+```ruby
+facts = FactDb.new
+
+person = facts.entity_service.create(
+  "Paula Chen",
+  type: :person
+)
+```
+
+### With Aliases
+
+```ruby
+person = facts.entity_service.create(
+  "Paula Chen",
+  type: :person,
+  aliases: ["Paula", "P. Chen", "Chen, Paula"]
+)
+```
+
+### With Metadata
+
+```ruby
+person = facts.entity_service.create(
+  "Paula Chen",
+  type: :person,
+  aliases: ["Paula"],
+  metadata: {
+    employee_id: "E12345",
+    department: "Engineering",
+    start_date: "2024-01-10"
+  }
+)
+```
+
+## Entity Types
+
+| Type | Description | Examples |
+|------|-------------|----------|
+| `:person` | Individual people | Paula Chen, John Smith |
+| `:organization` | Companies, teams | Microsoft, Platform Team |
+| `:place` | Locations | San Francisco, Building A |
+| `:product` | Products, services | Windows 11, Azure |
+| `:event` | Named events | Q4 Earnings, Annual Review |
+
+```ruby
+# Custom types are also supported
+entity = facts.entity_service.create(
+  "TPS Report",
+  type: :document_type
+)
+```
+
+## Managing Aliases
+
+### Add Alias
+
+```ruby
+facts.entity_service.add_alias(
+  entity.id,
+  "P. Chen",
+  type: :abbreviation,
+  confidence: 0.95
+)
+```
+
+### Alias Types
+
+| Type | Description |
+|------|-------------|
+| `nickname` | Informal names |
+| `abbreviation` | Shortened forms |
+| `formal` | Formal/legal names |
+| `maiden_name` | Previous names |
+| `trading_name` | Business aliases |
+
+### List Aliases
+
+```ruby
+entity.entity_aliases.each do |alias_record|
+  puts "#{alias_record.alias_text} (#{alias_record.alias_type})"
+  puts "  Confidence: #{alias_record.confidence}"
+end
+```
+
+### Remove Alias
+
+```ruby
+facts.entity_service.remove_alias(entity.id, "Old Name")
+```
+
+## Entity Resolution
+
+### Basic Resolution
+
+```ruby
+# Resolve a name to an entity
+entity = facts.resolve_entity("Paula Chen")
+
+# Returns existing entity or nil if not found
+```
+
+### Type-Constrained Resolution
+
+```ruby
+# Only match person entities
+person = facts.resolve_entity("Paula", type: :person)
+
+# Only match organizations
+org = facts.resolve_entity("Microsoft", type: :organization)
+```
+
+### Resolution Strategies
+
+The resolver tries in order:
+
+1. **Exact match** on canonical name
+2. **Alias match** on registered aliases
+3. **Fuzzy match** using Levenshtein distance
+
+```ruby
+# Configure fuzzy matching
+FactDb.configure do |config|
+  config.fuzzy_match_threshold = 0.85  # 85% similarity required
+end
+```
+
+### Batch Resolution
+
+```ruby
+names = ["Paula Chen", "John Smith", "Microsoft", "Seattle"]
+
+results = facts.batch_resolve_entities(names)
+
+results.each do |result|
+  status = result[:status]  # :resolved, :not_found, :error
+  entity = result[:entity]
+  puts "#{result[:name]}: #{status} -> #{entity&.canonical_name}"
+end
+```
+
+## Merging Entities
+
+When duplicate entities are discovered:
+
+```ruby
+# Merge entity2 into entity1 (entity1 is kept)
+facts.entity_service.merge(entity1.id, entity2.id)
+
+# After merge:
+entity2.reload
+entity2.resolution_status  # => "merged"
+entity2.merged_into_id     # => entity1.id
+```
+
+### What Happens on Merge
+
+1. Entity2's status changes to "merged"
+2. Entity2 points to entity1 via `merged_into_id`
+3. Entity2's aliases are copied to entity1
+4. All facts mentioning entity2 now also reference entity1
+
+### Auto-Merge
+
+Configure automatic merging for high-confidence matches:
+
+```ruby
+FactDb.configure do |config|
+  config.auto_merge_threshold = 0.95  # Auto-merge at 95% similarity
+end
+```
+
+## Updating Entities
+
+### Update Canonical Name
+
+```ruby
+facts.entity_service.update(
+  entity.id,
+  canonical_name: "Paula M. Chen"
+)
+```
+
+### Update Metadata
+
+```ruby
+facts.entity_service.update(
+  entity.id,
+  metadata: entity.metadata.merge(title: "Senior Principal Engineer")
+)
+```
+
+### Change Type
+
+```ruby
+# Reclassify entity type
+facts.entity_service.update(
+  entity.id,
+  entity_type: :organization
+)
+```
+
+## Resolution Status
+
+| Status | Description |
+|--------|-------------|
+| `unresolved` | Entity created but not confirmed |
+| `resolved` | Entity identity confirmed |
+| `merged` | Entity merged into another |
+
+### Mark as Resolved
+
+```ruby
+facts.entity_service.update(
+  entity.id,
+  resolution_status: :resolved
+)
+```
+
+### Find Unresolved
+
+```ruby
+unresolved = FactDb::Models::Entity
+  .where(resolution_status: 'unresolved')
+  .order(created_at: :desc)
+```
+
+## Querying Entities
+
+### Find by ID
+
+```ruby
+entity = facts.entity_service.find(entity_id)
+```
+
+### Search by Name
+
+```ruby
+entities = facts.entity_service.search("Paula")
+```
+
+### Filter by Type
+
+```ruby
+people = FactDb::Models::Entity
+  .where(entity_type: 'person')
+  .where.not(resolution_status: 'merged')
+```
+
+### Find Entities in Content
+
+```ruby
+# Find all entities mentioned in a content
+entities = facts.entity_service.in_content(content.id)
+```
+
+### Find Related Entities
+
+```ruby
+# Entities mentioned in facts about Paula
+related = facts.entity_service.related_to(paula.id)
+```
+
+## Semantic Search
+
+Search entities by meaning:
+
+```ruby
+# Find entities similar to a description
+similar = facts.entity_service.semantic_search(
+  "software engineering leadership",
+  type: :person,
+  limit: 10
+)
+```
+
+## Best Practices
+
+### 1. Use Comprehensive Aliases
+
+```ruby
+entity = facts.entity_service.create(
+  "International Business Machines Corporation",
+  type: :organization,
+  aliases: [
+    "IBM",
+    "Big Blue",
+    "International Business Machines",
+    "IBM Corp",
+    "IBM Corporation"
+  ]
+)
+```
+
+### 2. Store Relevant Metadata
+
+```ruby
+person = facts.entity_service.create(
+  "Paula Chen",
+  type: :person,
+  metadata: {
+    # Stable identifiers
+    employee_id: "E12345",
+    linkedin_url: "linkedin.com/in/paulachen",
+
+    # Useful context
+    department: "Engineering",
+    location: "San Francisco"
+  }
+)
+```
+
+### 3. Review Unresolved Entities
+
+```ruby
+# Periodically review unresolved entities
+unresolved = FactDb::Models::Entity
+  .where(resolution_status: 'unresolved')
+  .where('created_at < ?', 1.week.ago)
+
+unresolved.each do |entity|
+  # Try to find duplicates
+  similar = facts.entity_service.search(entity.canonical_name)
+  if similar.count > 1
+    puts "Potential duplicate: #{entity.canonical_name}"
+  end
+end
+```
+
+### 4. Handle Merged Entities
+
+```ruby
+# When querying, exclude merged entities
+active_entities = FactDb::Models::Entity
+  .where.not(resolution_status: 'merged')
+
+# Or follow the merge chain
+def canonical_entity(entity)
+  while entity.merged_into_id
+    entity = FactDb::Models::Entity.find(entity.merged_into_id)
+  end
+  entity
+end
+```
+
+### 5. Validate Entity Types
+
+```ruby
+VALID_TYPES = %i[person organization place product event].freeze
+
+def create_entity(name, type:)
+  unless VALID_TYPES.include?(type.to_sym)
+    raise ArgumentError, "Invalid entity type: #{type}"
+  end
+  facts.entity_service.create(name, type: type)
+end
+```

data/docs/guides/extracting-facts.md

@@ -0,0 +1,299 @@
+# Extracting Facts
+
+Facts are extracted from content using one of three methods: manual, LLM-powered, or rule-based.
+
+## Extraction Methods
+
+### Manual Extraction
+
+Create facts directly via the API:
+
+```ruby
+facts = FactDb.new
+
+# Create entities first
+paula = facts.entity_service.create("Paula Chen", type: :person)
+microsoft = facts.entity_service.create("Microsoft", type: :organization)
+
+# Create fact with explicit links
+fact = facts.fact_service.create(
+  "Paula Chen joined Microsoft as Principal Engineer",
+  valid_at: Date.parse("2024-01-10"),
+  mentions: [
+    { entity: paula, role: "subject", text: "Paula Chen" },
+    { entity: microsoft, role: "organization", text: "Microsoft" }
+  ],
+  sources: [
+    { content: content, type: "primary", excerpt: "...accepted the offer..." }
+  ]
+)
+```
+
+### LLM Extraction
+
+Use AI to automatically extract facts:
+
+```ruby
+# Configure LLM
+FactDb.configure do |config|
+  config.llm_provider = :openai
+  config.llm_api_key = ENV['OPENAI_API_KEY']
+end
+
+facts = FactDb.new
+
+# Extract facts from content
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+extracted.each do |fact|
+  puts fact.fact_text
+  puts "  Valid from: #{fact.valid_at}"
+  puts "  Entities: #{fact.entity_mentions.map(&:entity).map(&:canonical_name)}"
+end
+```
+
+### Rule-Based Extraction
+
+Use regex patterns for structured content:
+
+```ruby
+extracted = facts.extract_facts(content.id, extractor: :rule_based)
+```
+
+The rule-based extractor includes patterns for:
+
+- Dates and time references
+- Employment events (joined, promoted, left)
+- Title/role changes
+- Location references
+- Organizational relationships
+
+## Setting Default Extractor
+
+```ruby
+FactDb.configure do |config|
+  config.default_extractor = :llm  # or :manual, :rule_based
+end
+
+# Uses configured default
+extracted = facts.extract_facts(content.id)
+```
+
+## Fact Structure
+
+Every extracted fact includes:
+
+```ruby
+fact = Models::Fact.new(
+  fact_text: "Paula Chen is Principal Engineer at Microsoft",
+  fact_hash: "sha256...",           # For deduplication
+  valid_at: Time.parse("2024-01-10"),
+  invalid_at: nil,                   # nil = currently valid
+  status: "canonical",               # canonical, superseded, corroborated, synthesized
+  confidence: 0.95,                  # Extraction confidence
+  extraction_method: "llm",          # manual, llm, rule_based
+  metadata: {}                       # Additional data
+)
+```
+
+## Entity Mentions
+
+Facts link to entities via mentions:
+
+```ruby
+fact.add_mention(
+  entity: paula,
+  text: "Paula Chen",    # How entity was mentioned
+  role: "subject",       # Role in the fact
+  confidence: 0.95       # Resolution confidence
+)
+```
+
+### Mention Roles
+
+| Role | Description | Example |
+|------|-------------|---------|
+| `subject` | Primary actor | "Paula joined..." |
+| `object` | Target | "...hired Paula" |
+| `organization` | Company/team | "...at Microsoft" |
+| `location` | Place | "...in Seattle" |
+| `role` | Title/position | "...as Engineer" |
+| `temporal` | Time reference | "...in Q4 2024" |
+| `attribute` | Property | "...with 10 years experience" |
+
+## Source Links
+
+Facts link to source content:
+
+```ruby
+fact.add_source(
+  content: email_content,
+  type: "primary",
+  excerpt: "Paula has accepted our offer to join as Principal Engineer...",
+  confidence: 0.95
+)
+```
+
+### Source Types
+
+| Type | Description |
+|------|-------------|
+| `primary` | Direct source of the fact |
+| `supporting` | Confirms the fact |
+| `contradicting` | Contradicts the fact |
+
+## Batch Extraction
+
+Process multiple content items:
+
+```ruby
+content_ids = [content1.id, content2.id, content3.id]
+
+# Sequential processing
+results = facts.batch_extract(content_ids, parallel: false)
+
+# Parallel processing (default)
+results = facts.batch_extract(content_ids, parallel: true)
+
+results.each do |result|
+  puts "Content #{result[:content_id]}:"
+  puts "  Facts: #{result[:facts].count}"
+  puts "  Error: #{result[:error]}" if result[:error]
+end
+```
+
+## Custom Extractors
+
+Create custom extractors by extending the base class:
+
+```ruby
+class MyExtractor < FactDb::Extractors::Base
+  def extract(content)
+    extracted = []
+
+    # Your extraction logic here
+    # Parse content.raw_text
+    # Create fact records
+
+    extracted
+  end
+end
+
+# Register and use
+facts.fact_service.extract_from_content(
+  content.id,
+  extractor: MyExtractor.new(config)
+)
+```
+
+## Extraction Confidence
+
+Track confidence levels:
+
+```ruby
+# High confidence - direct statement
+fact = facts.fact_service.create(
+  "Paula is Principal Engineer",
+  confidence: 0.95
+)
+
+# Medium confidence - inferred
+fact = facts.fact_service.create(
+  "Paula likely works in Engineering",
+  confidence: 0.7
+)
+
+# Low confidence - speculation
+fact = facts.fact_service.create(
+  "Paula may be promoted soon",
+  confidence: 0.4
+)
+```
+
+## Post-Extraction Processing
+
+After extraction, you may want to:
+
+### Resolve Entities
+
+```ruby
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+extracted.each do |fact|
+  fact.entity_mentions.each do |mention|
+    if mention.entity.nil?
+      # Resolve unlinked mention
+      entity = facts.resolve_entity(mention.mention_text)
+      mention.update!(entity: entity) if entity
+    end
+  end
+end
+```
+
+### Detect Conflicts
+
+```ruby
+conflicts = facts.fact_service.resolver.find_conflicts(
+  entity_id: paula.id
+)
+
+conflicts.each do |conflict|
+  puts "Conflict between:"
+  puts "  #{conflict[:fact1].fact_text}"
+  puts "  #{conflict[:fact2].fact_text}"
+end
+```
+
+### Corroborate Facts
+
+```ruby
+# If multiple sources say the same thing
+if fact1.fact_text.similar_to?(fact2.fact_text)
+  facts.fact_service.resolver.corroborate(fact1.id, fact2.id)
+end
+```
+
+## Best Practices
+
+### 1. Review LLM Extractions
+
+```ruby
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+extracted.select { |f| f.confidence < 0.8 }.each do |fact|
+  # Flag for human review
+  fact.update!(metadata: fact.metadata.merge(needs_review: true))
+end
+```
+
+### 2. Validate Temporal Information
+
+```ruby
+# Ensure valid_at is reasonable
+if fact.valid_at > Time.current
+  logger.warn "Future date detected: #{fact.valid_at}"
+end
+```
+
+### 3. Link Sources
+
+```ruby
+# Always link facts to their sources
+fact = facts.fact_service.create(
+  "Important fact",
+  valid_at: Date.today,
+  sources: [{ content: source_content, type: "primary" }]
+)
+```
+
+### 4. Handle Extraction Errors
+
+```ruby
+begin
+  extracted = facts.extract_facts(content.id, extractor: :llm)
+rescue FactDb::ExtractionError => e
+  logger.error "Extraction failed: #{e.message}"
+  # Fall back to manual or rule-based
+  extracted = facts.extract_facts(content.id, extractor: :rule_based)
+end
+```

data/docs/guides/index.md

@@ -0,0 +1,22 @@
+# Guides
+
+Practical guides for using FactDb in your applications.
+
+## Configuration
+
+- [Configuration](configuration.md) - Configure database, LLM, and extraction settings
+
+## Working with Data
+
+- [Ingesting Content](ingesting-content.md) - Import documents, emails, and other content
+- [Extracting Facts](extracting-facts.md) - Extract facts using different methods
+- [LLM Integration](llm-integration.md) - Set up AI-powered extraction
+- [Temporal Queries](temporal-queries.md) - Query facts across time
+
+## Entity Management
+
+- [Entity Management](entity-management.md) - Create, resolve, and merge entities
+
+## Performance
+
+- [Batch Processing](batch-processing.md) - Process content in parallel