fact_db 0.0.1

data/docs/getting-started/database-setup.md

@@ -0,0 +1,170 @@
+# Database Setup
+
+FactDb uses PostgreSQL with the pgvector extension for storing content, entities, and facts with semantic search capabilities.
+
+## Create Database
+
+```bash
+createdb fact_db
+```
+
+## Enable pgvector
+
+Connect to your database and enable the extension:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+```
+
+## Run Migrations
+
+FactDb provides migrations that create all necessary tables:
+
+```ruby
+require 'fact_db'
+
+FactDb.configure do |config|
+  config.database_url = "postgresql://localhost/fact_db"
+end
+
+FactDb::Database.migrate!
+```
+
+## Schema Overview
+
+The migrations create six tables:
+
+### contents
+
+Stores immutable source documents.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| id | bigint | Primary key |
+| content_hash | string | SHA256 hash for deduplication |
+| content_type | string | Type (email, document, article) |
+| raw_text | text | Original content |
+| title | string | Optional title |
+| source_uri | string | Original location |
+| source_metadata | jsonb | Additional metadata |
+| embedding | vector(1536) | Semantic search vector |
+| captured_at | timestamptz | When content was captured |
+
+### entities
+
+Stores resolved identities.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| id | bigint | Primary key |
+| canonical_name | string | Authoritative name |
+| entity_type | string | person, organization, place, etc. |
+| resolution_status | string | unresolved, resolved, merged |
+| merged_into_id | bigint | Points to canonical entity if merged |
+| metadata | jsonb | Additional attributes |
+| embedding | vector(1536) | Semantic search vector |
+
+### entity_aliases
+
+Stores alternative names for entities.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| id | bigint | Primary key |
+| entity_id | bigint | Foreign key to entities |
+| alias_text | string | Alternative name |
+| alias_type | string | nickname, abbreviation, etc. |
+| confidence | float | Match confidence (0-1) |
+
+### facts
+
+Stores temporal assertions.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| id | bigint | Primary key |
+| fact_text | text | The assertion |
+| fact_hash | string | For deduplication |
+| valid_at | timestamptz | When fact became true |
+| invalid_at | timestamptz | When fact stopped being true |
+| status | string | canonical, superseded, corroborated, synthesized |
+| superseded_by_id | bigint | Points to replacing fact |
+| derived_from_ids | bigint[] | Source facts for synthesized |
+| corroborated_by_ids | bigint[] | Corroborating facts |
+| confidence | float | Extraction confidence |
+| extraction_method | string | manual, llm, rule_based |
+| metadata | jsonb | Additional data |
+| embedding | vector(1536) | Semantic search vector |
+
+### entity_mentions
+
+Links facts to entities.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| id | bigint | Primary key |
+| fact_id | bigint | Foreign key to facts |
+| entity_id | bigint | Foreign key to entities |
+| mention_text | string | Text that mentioned entity |
+| mention_role | string | subject, object, location, etc. |
+| confidence | float | Resolution confidence |
+
+### fact_sources
+
+Links facts to source content.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| id | bigint | Primary key |
+| fact_id | bigint | Foreign key to facts |
+| content_id | bigint | Foreign key to contents |
+| source_type | string | primary, supporting, contradicting |
+| excerpt | text | Relevant text excerpt |
+| confidence | float | Source confidence |
+
+## Indexes
+
+The migrations create indexes for:
+
+- Content hash (unique)
+- Content type
+- Full-text search on raw_text
+- Entity canonical name
+- Entity type
+- Fact status
+- Temporal range queries (valid_at, invalid_at)
+- HNSW indexes for vector similarity search
+
+## Custom Migration
+
+If you need to integrate with an existing database or customize the schema:
+
+```ruby
+# Copy migration files to your project
+FileUtils.cp_r(
+  FactDb.root.join('db/migrate'),
+  Rails.root.join('db/migrate')
+)
+
+# Or run standalone
+FactDb::Database.migrate!(
+  migrations_path: '/custom/path/to/migrations'
+)
+```
+
+## Connection Pool
+
+Configure the connection pool for your workload:
+
+```ruby
+FactDb.configure do |config|
+  config.database_url = ENV['DATABASE_URL']
+  config.database_pool_size = 10  # Default: 5
+  config.database_timeout = 60_000  # Default: 30000ms
+end
+```
+
+## Next Steps
+
+- [Quick Start](quick-start.md) - Start using FactDb
+- [Configuration](../guides/configuration.md) - Full configuration options

data/docs/getting-started/index.md

@@ -0,0 +1,71 @@
+# Getting Started
+
+This section will help you get FactDb up and running in your Ruby application.
+
+## Prerequisites
+
+Before installing FactDb, ensure you have:
+
+- **Ruby 3.0+** - FactDb requires Ruby 3.0 or later
+- **PostgreSQL 14+** - With the pgvector extension installed
+- **Bundler** - For dependency management
+
+## Quick Navigation
+
+<div class="grid cards" markdown>
+
+-   :material-download:{ .lg .middle } **Installation**
+
+    ---
+
+    Install FactDb and its dependencies
+
+    [:octicons-arrow-right-24: Installation Guide](installation.md)
+
+-   :material-rocket-launch:{ .lg .middle } **Quick Start**
+
+    ---
+
+    Get up and running in 5 minutes
+
+    [:octicons-arrow-right-24: Quick Start](quick-start.md)
+
+-   :material-database:{ .lg .middle } **Database Setup**
+
+    ---
+
+    Configure PostgreSQL and run migrations
+
+    [:octicons-arrow-right-24: Database Setup](database-setup.md)
+
+</div>
+
+## Overview
+
+Getting started with FactDb involves three steps:
+
+1. **Install the gem** - Add FactDb to your Gemfile
+2. **Set up the database** - Create tables and enable pgvector
+3. **Configure** - Set database URL and optional LLM settings
+
+Once configured, you can start ingesting content and extracting facts:
+
+```ruby
+require 'fact_db'
+
+# Configure
+FactDb.configure do |config|
+  config.database_url = ENV['DATABASE_URL']
+end
+
+# Create a facts instance
+facts = FactDb.new
+
+# Ingest content
+content = facts.ingest("Important information...", type: :document)
+
+# Extract and query facts
+extracted = facts.extract_facts(content.id)
+```
+
+Continue to the [Installation Guide](installation.md) to begin.

data/docs/getting-started/installation.md

@@ -0,0 +1,98 @@
+# Installation
+
+## Requirements
+
+- Ruby >= 3.0.0
+- PostgreSQL >= 14 with pgvector extension
+- Bundler
+
+## Install the Gem
+
+Add FactDb to your Gemfile:
+
+```ruby
+gem 'fact_db'
+```
+
+Then install:
+
+```bash
+bundle install
+```
+
+Or install directly:
+
+```bash
+gem install fact_db
+```
+
+## Install pgvector
+
+FactDb uses pgvector for semantic search. Install the PostgreSQL extension:
+
+=== "macOS (Homebrew)"
+
+    ```bash
+    brew install pgvector
+    ```
+
+=== "Ubuntu/Debian"
+
+    ```bash
+    sudo apt install postgresql-14-pgvector
+    ```
+
+=== "From Source"
+
+    ```bash
+    git clone https://github.com/pgvector/pgvector.git
+    cd pgvector
+    make
+    sudo make install
+    ```
+
+Then enable the extension in your database:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+```
+
+## Optional Dependencies
+
+### LLM Extraction
+
+For LLM-powered fact extraction, add the ruby_llm gem:
+
+```ruby
+gem 'ruby_llm'
+```
+
+### Async Processing
+
+For parallel pipeline processing with async fibers:
+
+```ruby
+gem 'async', '~> 2.0'
+```
+
+## Verify Installation
+
+Create a simple test script:
+
+```ruby
+require 'fact_db'
+
+puts "FactDb version: #{FactDb::VERSION}"
+puts "Installation successful!"
+```
+
+Run it:
+
+```bash
+ruby test_install.rb
+```
+
+## Next Steps
+
+- [Database Setup](database-setup.md) - Configure your database
+- [Quick Start](quick-start.md) - Start using FactDb

data/docs/getting-started/quick-start.md

@@ -0,0 +1,191 @@
+# Quick Start
+
+Get FactDb running in 5 minutes.
+
+## 1. Configure
+
+Create a configuration file or use environment variables:
+
+=== "Environment Variables"
+
+    ```bash
+    export EVENT_CLOCK_DATABASE_URL="postgresql://localhost/fact_db"
+    export EVENT_CLOCK_LLM_PROVIDER="openai"
+    export EVENT_CLOCK_LLM_API_KEY="sk-..."
+    ```
+
+=== "YAML Config"
+
+    ```yaml
+    # config/fact_db.yml
+    database_url: postgresql://localhost/fact_db
+    llm_provider: openai
+    llm_api_key: <%= ENV['OPENAI_API_KEY'] %>
+    ```
+
+=== "Ruby Block"
+
+    ```ruby
+    FactDb.configure do |config|
+      config.database_url = "postgresql://localhost/fact_db"
+      config.llm_provider = :openai
+      config.llm_api_key = ENV['OPENAI_API_KEY']
+    end
+    ```
+
+## 2. Set Up Database
+
+Run the migrations:
+
+```ruby
+require 'fact_db'
+
+FactDb.configure do |config|
+  config.database_url = ENV['DATABASE_URL']
+end
+
+# Run migrations
+FactDb::Database.migrate!
+```
+
+## 3. Create Your First Facts Instance
+
+```ruby
+require 'fact_db'
+
+facts = FactDb.new
+```
+
+## 4. Ingest Content
+
+```ruby
+# Ingest an email
+content = facts.ingest(
+  "Hi team, Paula Chen has accepted our offer and will join as Principal Engineer starting January 10, 2024. She'll be reporting to Sarah in the Platform team.",
+  type: :email,
+  title: "New Hire Announcement",
+  captured_at: Time.current
+)
+
+puts "Ingested content: #{content.id}"
+```
+
+## 5. Create Entities
+
+```ruby
+# Create entities for people and organizations
+paula = facts.entity_service.create(
+  "Paula Chen",
+  type: :person,
+  aliases: ["Paula", "P. Chen"]
+)
+
+sarah = facts.entity_service.create(
+  "Sarah Johnson",
+  type: :person,
+  aliases: ["Sarah"]
+)
+
+platform_team = facts.entity_service.create(
+  "Platform Team",
+  type: :organization
+)
+```
+
+## 6. Extract Facts
+
+### Manual Extraction
+
+```ruby
+fact = facts.fact_service.create(
+  "Paula Chen joined as Principal Engineer",
+  valid_at: Date.parse("2024-01-10"),
+  mentions: [
+    { entity: paula, role: "subject", text: "Paula Chen" }
+  ],
+  sources: [
+    { content: content, type: "primary" }
+  ]
+)
+```
+
+### LLM Extraction
+
+```ruby
+# Extract facts automatically using LLM
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+extracted.each do |fact|
+  puts "Extracted: #{fact.fact_text}"
+  puts "  Valid from: #{fact.valid_at}"
+end
+```
+
+## 7. Query Facts
+
+```ruby
+# Get current facts about Paula
+current = facts.current_facts_for(paula.id)
+current.each { |f| puts f.fact_text }
+
+# Get facts valid at a specific date
+historical = facts.facts_at(
+  Date.parse("2023-12-01"),
+  entity: paula.id
+)
+
+# Search by topic
+team_facts = facts.query_facts(topic: "Platform Team")
+```
+
+## 8. Build Timelines
+
+```ruby
+timeline = facts.timeline_for(paula.id)
+
+timeline.each do |entry|
+  puts "#{entry[:date]}: #{entry[:fact].fact_text}"
+end
+```
+
+## Complete Example
+
+```ruby
+require 'fact_db'
+
+# Configure
+FactDb.configure do |config|
+  config.database_url = ENV['DATABASE_URL']
+  config.llm_provider = :openai
+  config.llm_api_key = ENV['OPENAI_API_KEY']
+end
+
+# Create facts instance
+facts = FactDb.new
+
+# Ingest content
+content = facts.ingest(
+  "Paula Chen joined Microsoft as Principal Engineer on January 10, 2024.",
+  type: :announcement,
+  captured_at: Time.current
+)
+
+# Create entities
+paula = facts.entity_service.create("Paula Chen", type: :person)
+microsoft = facts.entity_service.create("Microsoft", type: :organization)
+
+# Extract facts via LLM
+extracted = facts.extract_facts(content.id, extractor: :llm)
+
+# Query
+puts "Current facts about Paula:"
+facts.current_facts_for(paula.id).each do |fact|
+  puts "  - #{fact.fact_text}"
+end
+```
+
+## Next Steps
+
+- [Configuration Guide](../guides/configuration.md) - Detailed configuration options
+- [Ingesting Content](../guides/ingesting-content.md) - Learn about content types
+- [LLM Integration](../guides/llm-integration.md) - Set up LLM providers