fact_db 0.0.2 → 0.0.3

data/examples/README.md

@@ -1,21 +1,39 @@
 # FactDb Examples
 
-This directory contains demonstration programs showcasing the capabilities of the FactDb gem.
+This directory contains demonstration programs showcasing the capabilities of the FactDb gem. Files are numbered to indicate a progressive learning path from basic to advanced.
 
 ## Prerequisites
 
 1. PostgreSQL database with pgvector extension
-2. Set the `DATABASE_URL` environment variable or use the default `postgres://localhost/fact_db_demo`
+2. Set the `FDB_DATABASE__NAME` environment variable or use the default `fact_db_demo`
 3. Run migrations to set up the schema
 
 ```bash
 bundle install
-DATABASE_URL=postgres://localhost/fact_db_demo rake db:migrate
+FDB_DATABASE__NAME=fact_db_demo rake db:migrate
 ```
 
 ## Examples
 
-### basic_usage.rb
+### 001_configuration.rb
+
+**Complete guide to FactDb configuration**
+
+Demonstrates all configuration methods:
+- Bundled defaults (shipped with the gem)
+- Environment variables (FDB_*)
+- XDG user config files (~/.config/fact_db/fact_db.yml)
+- Project config files (./config/fact_db.yml)
+- Local overrides (./config/fact_db.local.yml)
+- Programmatic configuration blocks
+- Database URL reconciliation
+- Configuration validation
+
+```bash
+ruby examples/001_configuration.rb
+```
+
+### 010_basic_usage.rb
 
 **Foundational introduction to FactDb**
 
@@ -28,26 +46,26 @@ Demonstrates:
 - Getting system statistics
 
 ```bash
-ruby examples/basic_usage.rb
+ruby examples/010_basic_usage.rb
 ```
 
-### entity_management.rb
+### 020_entity_management.rb
 
 **Deep dive into entity operations**
 
 Demonstrates:
-- Creating entities with multiple types (person, organization, place)
+- Creating entities with multiple kinds (person, organization, place)
 - Managing aliases (names, emails, abbreviations)
 - Entity resolution using fuzzy matching
 - Merging duplicate entities
-- Searching entities by name and type
+- Searching entities by name and kind
 - Building entity timelines
 
 ```bash
-ruby examples/entity_management.rb
+ruby examples/020_entity_management.rb
 ```
 
-### temporal_queries.rb
+### 030_temporal_queries.rb
 
 **Working with time-based data**
 
@@ -61,10 +79,26 @@ Demonstrates:
 - Querying facts by entity role
 
 ```bash
-ruby examples/temporal_queries.rb
+ruby examples/030_temporal_queries.rb
+```
+
+### 040_output_formats.rb
+
+**LLM-optimized output transformers**
+
+Demonstrates the new output format system:
+- JSON format (default) for structured data
+- Triples format (Subject-Predicate-Object) for knowledge graphs
+- Cypher format for graph database notation
+- Text format for human-readable output
+- Raw format for ActiveRecord access
+- Using formats with queries
+
+```bash
+ruby examples/040_output_formats.rb
 ```
 
-### rule_based_extraction.rb
+### 050_rule_based_extraction.rb
 
 **Automatic fact extraction from text**
 
@@ -77,10 +111,43 @@ Demonstrates:
 - Testing individual extraction patterns
 
 ```bash
-ruby examples/rule_based_extraction.rb
+ruby examples/050_rule_based_extraction.rb
+```
+
+### 060_fluent_temporal_api.rb
+
+**Fluent query builder for temporal analysis**
+
+Demonstrates the new chainable API:
+- `facts.at(date)` - Query at a specific point in time
+- `facts.at(date).facts_for(entity_id)` - Entity state at a date
+- `facts.at(date).compare_to(other_date)` - Compare two dates
+- `facts.diff(topic, from:, to:)` - Compute temporal diffs
+- Career timeline snapshots across multiple dates
+
+```bash
+ruby examples/060_fluent_temporal_api.rb
 ```
 
-### hr_system.rb
+### 070_introspection.rb
+
+**Schema and data introspection**
+
+Demonstrates discovery capabilities:
+- `facts.introspect()` - Discover system capabilities
+- `facts.introspect(topic)` - Examine specific entities
+- `facts.suggest_queries(topic)` - Get query suggestions
+- `facts.suggest_strategies(query)` - Strategy recommendations
+- `entity_service.relationship_types` - All relationship types
+- `entity_service.timespan_for(id)` - Fact date range
+- `fact_service.fact_stats(id)` - Fact status breakdown
+- Building LLM context with introspection
+
+```bash
+ruby examples/070_introspection.rb
+```
+
+### 080_hr_system.rb
 
 **Practical HR knowledge management system**
 
@@ -96,7 +163,107 @@ A comprehensive real-world example demonstrating:
 - HR statistics and reporting
 
 ```bash
-ruby examples/hr_system.rb
+ruby examples/080_hr_system.rb
+```
+
+### 090_ingest_demo.rb
+
+**Building a fact database from markdown files**
+
+Demonstrates LLM-based ingestion:
+- Parsing markdown files with optional YAML frontmatter
+- Using LLM-based extraction to identify entities and facts
+- Automatic entity resolution and deduplication
+- Progressive entity discovery from text
+
+```bash
+ruby examples/090_ingest_demo.rb <directory>           # Build/update database
+ruby examples/090_ingest_demo.rb <file.md>             # Process single file
+ruby examples/090_ingest_demo.rb <path> --rebuild      # Drop and rebuild
+ruby examples/090_ingest_demo.rb --stats               # Show statistics only
+```
+
+### 100_query_context.rb
+
+**Natural language query to context generation**
+
+Takes a natural language query and generates context from the facts database suitable for LLM consumption. Features multi-signal relevance ranking with configurable weights.
+
+Demonstrates:
+- Extracting entity candidates from natural language queries
+- Resolving entities from query text (proper nouns, possessives)
+- Multi-strategy fact gathering (entity mentions, full-text search, semantic search)
+- **Multi-signal relevance ranking** with PostgreSQL ts_rank, pgvector similarity, and heuristics
+- Generating context in multiple output formats
+- Building LLM-ready context from a facts database
+
+```bash
+# Basic usage
+ruby examples/100_query_context.rb "Who is Sapphira's husband?"
+
+# With verbose output showing processing steps and signal breakdown
+ruby examples/100_query_context.rb --verbose "What happened to Ananias?"
+
+# Different output formats
+ruby examples/100_query_context.rb --format triples "Tell me about Peter"
+ruby examples/100_query_context.rb --format json "Who are the apostles?"
+ruby examples/100_query_context.rb --format cypher "Where was Stephen martyred?"
+
+# Disable ranking (return facts in database order)
+ruby examples/100_query_context.rb --no-rank "Tell me about the apostles"
+```
+
+**Ranking Signals** (configurable via FactDb.config.ranking):
+| Signal | Default Weight | Description |
+|--------|----------------|-------------|
+| ts_rank | 0.25 | PostgreSQL full-text search relevance |
+| vector_similarity | 0.25 | Semantic similarity via pgvector embeddings |
+| entity_mentions | 0.15 | Facts mentioning query entities |
+| direct_answer | 0.15 | Pattern match for query intent |
+| term_overlap | 0.10 | Query word matches |
+| relationship_match | 0.05 | Relationship words (husband, wife, etc.) |
+| confidence | 0.05 | Fact's stored confidence score |
+
+**Note:** Run `090_ingest_demo.rb acts_esv/` first to populate the database with biblical facts.
+
+## Utilities
+
+### 110_prove_it.rb
+
+**Source evidence viewer**
+
+Displays fact records along with their original source text evidence.
+
+```bash
+ruby examples/110_prove_it.rb <fact_id> [fact_id...]    # Show facts with evidence
+ruby examples/110_prove_it.rb --last <n>                # Show last n facts
+ruby examples/110_prove_it.rb --search <term>           # Search facts and show evidence
+```
+
+### 120_dump_database.rb
+
+**Database export utility**
+
+Dumps the contents of the fact_db database in a structured format for verification and inspection.
+
+```bash
+ruby examples/120_dump_database.rb                # Full dump
+ruby examples/120_dump_database.rb --summary      # Summary only
+ruby examples/120_dump_database.rb --entities     # Entities only
+ruby examples/120_dump_database.rb --facts        # Facts only
+ruby examples/120_dump_database.rb --content      # Content only
+ruby examples/120_dump_database.rb --search TERM  # Search facts/entities
+```
+
+### 130_cleanup_aliases.rb
+
+**Alias maintenance utility**
+
+Cleanup script to remove invalid aliases (pronouns, generic terms, etc.) from existing entities.
+
+```bash
+ruby examples/130_cleanup_aliases.rb           # Dry run
+ruby examples/130_cleanup_aliases.rb --execute # Actually delete
 ```
 
 ## Key Concepts
@@ -128,3 +295,50 @@ All facts link back to source content:
 - Primary sources (direct evidence)
 - Supporting sources (additional evidence)
 - Corroborating sources (independent confirmation)
+
+### Output Formats
+
+Facts can be transformed into multiple formats optimized for different consumers:
+- `json` - Structured data with entities and metadata
+- `triples` - Subject-Predicate-Object for semantic reasoning
+- `cypher` - Graph notation for Neo4j-style visualization
+- `text` - Human-readable markdown format
+- `raw` - Direct ActiveRecord objects
+
+### Fluent Temporal API
+
+The chainable `at()` method provides an intuitive way to query temporal data:
+```ruby
+facts.at("2024-01-15").facts_for(entity_id)          # What did we know?
+facts.at("2024-01-15").compare_to("2024-06-15")      # What changed?
+facts.diff(nil, from: "2024-01", to: "2024-06")      # Detailed diff
+```
+
+### Introspection
+
+Discover what the system knows before querying:
+```ruby
+facts.introspect              # Schema, capabilities, statistics
+facts.introspect("Paula")     # Entity coverage, relationships
+facts.suggest_queries("Paula") # What can I ask about Paula?
+```
+
+### Ranking Configuration
+
+Tune relevance ranking weights for your use case:
+```ruby
+FactDb.configure do |config|
+  # Boost semantic search for concept-heavy domains
+  config.ranking.vector_similarity_weight = 0.35
+  config.ranking.ts_rank_weight = 0.15
+
+  # Or boost entity mentions for person-centric queries
+  config.ranking.entity_mention_weight = 0.30
+end
+```
+
+Or via environment variables:
+```bash
+export FDB_RANKING__VECTOR_SIMILARITY_WEIGHT=0.35
+export FDB_RANKING__TS_RANK_WEIGHT=0.15
+```

data/examples/data/lincoln_associates.md

@@ -0,0 +1,201 @@
+# Abraham Lincoln's Associates, Friends, and Colleagues
+
+## Law Partners
+
+### John Todd Stuart (First Law Partner, 1837-1841)
+
+| Field | Value |
+|-------|-------|
+| Full Name | John Todd Stuart |
+| Partnership Period | 1837-1841 |
+| Relationship | Cousin of Mary Todd Lincoln |
+| Role | Senior partner |
+| Political Office | U.S. Representative from Illinois |
+| How They Met | Illinois militia during Black Hawk War |
+| Contribution | Encouraged Lincoln to study law |
+
+Stuart was Lincoln's first law partner and mentor. He encouraged Lincoln to pursue a legal career and lent him law books.
+
+### Stephen T. Logan (Second Law Partner, 1841-1844)
+
+| Field | Value |
+|-------|-------|
+| Full Name | Stephen Trigg Logan |
+| Partnership Period | 1841-1844 |
+| Role | Senior partner |
+| Reputation | One of the best lawyers in Illinois |
+| Political Role | Member of "the Junto" political group |
+| Impact | Helped refine Lincoln's legal skills |
+
+### William H. Herndon (Third and Final Law Partner, 1844-1865)
+
+| Field | Value |
+|-------|-------|
+| Full Name | William Henry Herndon |
+| Birth Date | December 25, 1818 |
+| Birth Place | Green County, Kentucky |
+| Partnership Period | 1844-1865 (until Lincoln's death) |
+| How They Met | At Joshua Speed's store; debates and poetry readings |
+| Lincoln's Quote | "He was my man always above all other men on the globe" |
+| Political Role | Active in founding Republican Party in Illinois |
+| Public Service | Mayor of Springfield (1855) |
+| Later Work | Wrote "Herndon's Lincoln: The True Story of a Great Life" (1888, with Jesse Weik) |
+
+Herndon was Lincoln's longest-serving and most loyal law partner. Their office was located in downtown Springfield on Sixth Street in the Tinsley building. Lincoln invited the younger Herndon to form a partnership in October 1844. Unlike Lincoln, Herndon did not travel the Eighth Judicial Circuit but argued cases in court while Lincoln rode the circuit. They split fees equally.
+
+After Lincoln's assassination, Herndon spent nearly 20 years compiling histories and interviews for his biography of Lincoln.
+
+---
+
+## Close Friends
+
+### Joshua F. Speed
+
+| Field | Value |
+|-------|-------|
+| Full Name | Joshua Fry Speed |
+| Relationship | Close friend and roommate |
+| Home State | Kentucky (transplant to Illinois) |
+| Occupation | General store owner |
+| Meeting | Lincoln roomed with Speed when first arriving in Springfield |
+| Duration of Friendship | Lifelong |
+| Significance | Perhaps Lincoln's closest personal friend |
+
+Speed and Lincoln shared living quarters above Speed's store in Springfield. Their friendship is considered one of the closest of Lincoln's life.
+
+### Edward Dickinson Baker
+
+| Field | Value |
+|-------|-------|
+| Full Name | Edward Dickinson Baker |
+| Relationship | Close friend and political ally |
+| Occupation | Politician, Lawyer, Military officer |
+| Lincoln's Honor | Lincoln's son Edward ("Eddie") was named after him |
+| Political Role | Member of "the Junto" political group |
+| Death | Killed at Battle of Ball's Bluff (October 21, 1861) |
+| Significance | Only sitting U.S. Senator to die in combat |
+
+---
+
+## Political Associates
+
+### "The Junto" Political Group
+
+A political group in Springfield that included Lincoln and other prominent Whig politicians:
+
+| Member | Role/Position |
+|--------|---------------|
+| Abraham Lincoln | Politician, Lawyer |
+| Edward D. Baker | Politician, Lawyer |
+| Stephen T. Logan | Lawyer, Politician |
+| Ninian W. Edwards | Politician (brother-in-law of Lincoln) |
+
+### Stephen A. Douglas (Political Rival)
+
+| Field | Value |
+|-------|-------|
+| Full Name | Stephen Arnold Douglas |
+| Nickname | "The Little Giant" |
+| Relationship | Political rival |
+| Notable | Lincoln-Douglas Debates (1858) |
+| 1858 Senate Race | Douglas defeated Lincoln |
+| 1860 Presidential Race | Lincoln defeated Douglas |
+| Death | June 3, 1861 |
+
+Douglas and Lincoln debated slavery and popular sovereignty in the famous 1858 Senate race debates. Though rivals, Douglas ultimately supported Lincoln's efforts to preserve the Union.
+
+---
+
+## Springfield Connections
+
+### William Butler
+
+| Field | Value |
+|-------|-------|
+| Full Name | William Butler |
+| Occupation | Attorney |
+| Relationship | Lincoln often took meals with the Butler family |
+
+### Ninian W. Edwards
+
+| Field | Value |
+|-------|-------|
+| Full Name | Ninian Wirt Edwards |
+| Relationship | Brother-in-law (married to Mary's sister Elizabeth) |
+| Occupation | Politician, Lawyer |
+| Political Role | Member of "the Junto" |
+| Significance | Lincoln met Mary Todd at the Edwards home |
+
+---
+
+## Presidential Secretaries
+
+### John George Nicolay
+
+| Field | Value |
+|-------|-------|
+| Full Name | John George Nicolay |
+| Position | Private Secretary to President Lincoln |
+| Term | 1861-1865 |
+| Birth | Bavaria, Germany |
+| Later Work | Co-authored "Abraham Lincoln: A History" (10 volumes) with Hay |
+
+### John Milton Hay
+
+| Field | Value |
+|-------|-------|
+| Full Name | John Milton Hay |
+| Position | Assistant Private Secretary to President Lincoln |
+| Term | 1861-1865 |
+| Later Career | U.S. Secretary of State (1898-1905) |
+| Later Work | Co-authored "Abraham Lincoln: A History" with Nicolay |
+
+Nicolay and Hay lived in the White House and worked closely with Lincoln throughout his presidency. They later co-authored the authoritative 10-volume biography of Lincoln.
+
+---
+
+## Military Associates
+
+(See separate file: lincoln_military.md for detailed information)
+
+Key military associates included:
+- Ulysses S. Grant
+- William T. Sherman
+- George B. McClellan
+- Edwin M. Stanton (Secretary of War)
+
+---
+
+## Legal Circuit Associates
+
+Lincoln practiced on the Eighth Judicial Circuit in Illinois, where he developed relationships with other lawyers and judges:
+
+### David Davis
+
+| Field | Value |
+|-------|-------|
+| Full Name | David Davis |
+| Occupation | Judge, Politician |
+| Role | Presiding judge on Eighth Judicial Circuit |
+| 1860 Campaign | Managed Lincoln's presidential campaign |
+| Later Position | U.S. Supreme Court Justice (appointed by Lincoln) |
+
+---
+
+## Key Relationships Summary
+
+| Category | Notable Names |
+|----------|---------------|
+| Law Partners | Stuart, Logan, Herndon |
+| Close Friends | Joshua Speed, Edward Baker |
+| Political Allies | Davis, Baker, Edwards |
+| Political Rivals | Stephen Douglas |
+| Presidential Staff | Nicolay, Hay |
+| Family Connections | Edwards (brother-in-law), Stuart (Mary's cousin) |
+
+## Sources
+
+- Mr. Lincoln and Friends (mrlincolnandfriends.org)
+- Abraham Lincoln's Classroom
+- Illinois Historic Preservation Division
+- Papers of Abraham Lincoln

data/examples/data/lincoln_biography.md

@@ -0,0 +1,66 @@
+# Abraham Lincoln - Biography
+
+## Basic Information
+
+| Field | Value |
+|-------|-------|
+| Full Name | Abraham Lincoln |
+| Birth Date | February 12, 1809 |
+| Birth Place | Hardin County (now LaRue County), Kentucky |
+| Birth Location | One-room log cabin on Sinking Spring Farm near Hodgenville |
+| Death Date | April 15, 1865 |
+| Death Place | Petersen House, Washington, D.C. |
+| Cause of Death | Assassination (gunshot wound to the head) |
+| Age at Death | 56 years |
+| Height | 6 feet 4 inches |
+| Occupation | Lawyer, Politician |
+| Political Party | Whig (early career), Republican |
+| Religion | No formal church membership; spiritual beliefs |
+| Presidency | 16th President of the United States |
+| Presidential Term | March 4, 1861 - April 15, 1865 |
+| Burial Place | Oak Ridge Cemetery, Springfield, Illinois |
+
+## Physical Description
+
+Abraham Lincoln was notably tall at 6 feet 4 inches, making him the tallest U.S. president. He was described as "tall and lanky" with distinctive features that made him stand out in any crowd.
+
+## Early Life Summary
+
+Lincoln was born into humble circumstances on the Kentucky frontier. His family moved to Indiana when he was seven years old, and later to Illinois when he was twenty-one. Despite limited formal education (less than 12 months total), Lincoln was a voracious reader and self-taught himself law.
+
+## Career Progression
+
+1. **Farmer/Laborer** - Early years helping family on frontier farms
+2. **Rail Splitter** - Manual labor in his youth
+3. **Store Clerk** - New Salem, Illinois (1831)
+4. **Militia Captain** - Black Hawk War (1832)
+5. **Postmaster** - New Salem (1833-1836)
+6. **Surveyor** - Sangamon County
+7. **Illinois State Legislator** - Four terms (1834-1842)
+8. **Lawyer** - Springfield, Illinois (1836-1861)
+9. **U.S. Representative** - Illinois 7th District (1847-1849)
+10. **Presidential Candidate** - Lost Senate race to Douglas (1858)
+11. **President of the United States** - (1861-1865)
+
+## Key Accomplishments
+
+- Preserved the Union during the Civil War
+- Issued the Emancipation Proclamation (January 1, 1863)
+- Delivered the Gettysburg Address (November 19, 1863)
+- Promoted passage of the 13th Amendment abolishing slavery
+- Strengthened the federal government
+- Modernized the economy through the Homestead Act, transcontinental railroad, and national banking system
+
+## Famous Quotes
+
+- "A house divided against itself cannot stand."
+- "Four score and seven years ago..."
+- "Government of the people, by the people, for the people, shall not perish from the earth."
+- "With malice toward none, with charity for all..."
+
+## Sources
+
+- White House Historical Association
+- National Archives Presidential Resources
+- Library of Congress Abraham Lincoln Papers
+- Miller Center, University of Virginia