fact_db 0.0.2 → 0.0.3

data/examples/data/lincoln_todd_family.md

@@ -0,0 +1,136 @@
+# Mary Todd Lincoln's Family (The Todd Family)
+
+## Mary's Father
+
+### Robert Smith Todd
+
+| Field | Value |
+|-------|-------|
+| Full Name | Robert Smith Todd |
+| Birth Year | 1791 |
+| Education | Studied law at Transylvania College |
+| Career | Businessman (chose business over law) |
+| Position | President of the Lexington branch of the Bank of Kentucky |
+| Political Offices | Justice of the Peace, Sheriff, Clerk of State House of Representatives |
+| First Marriage | Elizabeth "Eliza" Parker (1812) |
+| Second Marriage | Elizabeth "Betsy" Humphreys (1826) |
+
+---
+
+## Mary's Mother
+
+### Elizabeth "Eliza" Parker Todd
+
+| Field | Value |
+|-------|-------|
+| Full Name | Elizabeth "Eliza" Parker Todd |
+| Birth Year | 1794 |
+| Family Background | Wealthy merchant family |
+| Education | Likely attended local female academy |
+| Marriage | Robert Smith Todd (1812) |
+| Death Year | 1825 |
+| Cause of Death | Complications following childbirth |
+| Children | 7 children |
+
+Mary was only 6 years old when her mother died.
+
+---
+
+## Mary's Stepmother
+
+### Elizabeth "Betsy" Humphreys Todd
+
+| Field | Value |
+|-------|-------|
+| Full Name | Elizabeth "Betsy" Humphreys Todd |
+| Marriage to Robert | 1826 |
+| Children with Robert | 9 children |
+
+---
+
+## Children from First Marriage (Eliza Parker Todd)
+
+| Name | Notes |
+|------|-------|
+| Elizabeth Todd Edwards | Mary's older sister; Mary lived with her in Springfield |
+| Frances Todd | Sister |
+| Levi Todd | Brother |
+| **Mary Todd Lincoln** | Fourth child; married Abraham Lincoln |
+| Robert Todd | Died in infancy |
+| Ann Todd | Sister |
+| George Todd | Brother |
+
+---
+
+## Children from Second Marriage (Betsy Humphreys Todd)
+
+| Name | Notes |
+|------|-------|
+| Margaret Todd | Half-sister |
+| Samuel Todd | Half-brother |
+| David Todd | Half-brother |
+| Martha Todd | Half-sister |
+| Emilie Todd Helm | Half-sister; particularly close to Mary |
+| Alexander Todd | Half-brother |
+| Elodie Todd | Half-sister |
+| Katherine "Kittie" Todd | Half-sister |
+
+---
+
+## Mary's Sister: Elizabeth Todd Edwards
+
+Elizabeth was Mary's older sister who lived in Springfield, Illinois with her husband Ninian W. Edwards. Mary moved to Springfield in 1839 to live with Elizabeth, and it was at Elizabeth's home that Mary met Abraham Lincoln.
+
+### Ninian W. Edwards (Brother-in-law)
+
+| Field | Value |
+|-------|-------|
+| Full Name | Ninian Wirt Edwards |
+| Relationship | Husband of Elizabeth Todd |
+| Occupation | Politician, Lawyer |
+| Political Role | Member of "the Junto" political group with Lincoln |
+
+---
+
+## Civil War Divided Loyalties
+
+The Todd family was deeply divided during the Civil War, reflecting the broader national conflict:
+
+### Union Supporters (5 siblings)
+The Todds who lived in Northern states, including Mary, supported the Union cause.
+
+### Confederate Supporters (8 siblings)
+The majority of Mary's siblings sided with the Confederacy, creating painful divisions within the family.
+
+### Geographic Distribution
+When Abraham and Mary Lincoln entered the White House, Todd family members were scattered across:
+- Kentucky
+- Illinois
+- Ohio
+- Virginia
+- Alabama
+- Louisiana
+
+This geographic dispersal mirrored the political divide that split the nation.
+
+---
+
+## Emilie Todd Helm
+
+Mary's half-sister Emilie was particularly close to her. Emilie married Benjamin Hardin Helm, who became a Confederate general and was killed at the Battle of Chickamauga in 1863. After his death, Emilie visited Mary at the White House, causing political controversy.
+
+---
+
+## Total Todd Family Size
+
+| Category | Count |
+|----------|-------|
+| Children from first marriage | 7 |
+| Children from second marriage | 9 |
+| **Total children of Robert S. Todd** | **16** |
+
+## Sources
+
+- Mary Todd Lincoln House (mtlhouse.org)
+- White House Historical Association
+- Lincoln Collection

data/examples/ingest_reporter.rb

@@ -0,0 +1,335 @@
+# frozen_string_literal: true
+
+# IngestReporter - Progress reporting for document ingestion
+#
+# This class provides clear, non-blocking feedback during long ingestion
+# processes. It uses a single-line status approach that updates in place,
+# showing users that work is happening without cluttering the terminal.
+#
+# Usage:
+#   reporter = IngestReporter.new
+#   reporter.start_ingestion(total_files: 10)
+#   reporter.file_started("document.md", 1, 10)
+#   reporter.section_started("Introduction", 1, 5)
+#   reporter.extraction_started
+#   reporter.extraction_progress  # call periodically during LLM calls
+#   reporter.extraction_completed(facts_count: 3, entities_count: 5)
+#   reporter.section_completed
+#   reporter.file_completed(facts: 8, entities: 12, errors: 0, skipped: 1)
+#   reporter.finish_ingestion
+#
+# Customization:
+#   Subclass and override methods to customize output format, or set
+#   output: to a different IO object for logging.
+
+class IngestReporter
+  SPINNER_CHARS = %w[⠋ ⠙ ⠹ ⠸ ⠼ ⠴ ⠦ ⠧ ⠇ ⠏].freeze
+
+  attr_reader :output, :total_files, :total_facts, :total_entities, :total_errors
+
+  def initialize(output: $stdout, color: true)
+    @output = output
+    @color = color && output.respond_to?(:tty?) && output.tty?
+    @spinner_index = 0
+    @extraction_start_time = nil
+    @file_start_time = nil
+    @ingestion_start_time = nil
+    @current_file = nil
+    @current_file_index = 0
+    @current_section = nil
+    @current_section_index = 0
+    @total_sections = 0
+    @total_files = 0
+    @total_facts = 0
+    @total_entities = 0
+    @total_errors = 0
+    @line_length = 0
+  end
+
+  # Called once at the start of the ingestion process
+  def start_ingestion(total_files:, source_path:)
+    @total_files = total_files
+    @ingestion_start_time = Time.now
+    @total_facts = 0
+    @total_entities = 0
+    @total_errors = 0
+
+    write_line ""
+    write_line "Starting ingestion: #{total_files} file(s) from #{source_path}"
+    write_line ""
+  end
+
+  # Called when a new file begins processing
+  def file_started(filename, index, total)
+    @current_file = filename
+    @current_file_index = index
+    @total_files = total
+    @file_start_time = Time.now
+    @file_facts = 0
+    @file_entities = 0
+    @file_errors = 0
+    @file_skipped = 0
+
+    update_status
+  end
+
+  # Called when a new section within a file begins
+  def section_started(section_ref, index, total)
+    @current_section = section_ref
+    @current_section_index = index
+    @total_sections = total
+
+    update_status
+  end
+
+  # Called when a section is skipped (already processed)
+  def section_skipped(section_ref)
+    @file_skipped += 1
+  end
+
+  # Called when LLM extraction begins
+  def extraction_started
+    @extraction_start_time = Time.now
+    update_status
+  end
+
+  # Called periodically during LLM extraction to show activity
+  # Returns immediately - call this in a loop or timer
+  def extraction_progress
+    @spinner_index = (@spinner_index + 1) % SPINNER_CHARS.length
+    update_status(extracting: true)
+  end
+
+  # Called when extraction completes for a section
+  def extraction_completed(facts_count:, entities_count:)
+    @file_facts += facts_count
+    @file_entities += entities_count
+    @total_facts += facts_count
+    @total_entities += entities_count
+    @extraction_start_time = nil
+
+    update_status
+  end
+
+  # Called when a section finishes processing
+  def section_completed
+    @current_section = nil
+  end
+
+  # Called when an error occurs
+  def error_occurred(error, context: nil)
+    @file_errors += 1
+    @total_errors += 1
+
+    clear_line
+    error_msg = context ? "#{context}: #{error.message}" : error.message
+    write_line colorize("  ✗ Error: #{truncate(error_msg, 70)}", :red)
+  end
+
+  # Called when a file finishes processing
+  def file_completed(facts:, entities:, errors:, skipped:)
+    elapsed = Time.now - @file_start_time
+    clear_line
+
+    status_parts = ["#{facts} facts", "#{entities} entities"]
+    status_parts << colorize("#{errors} errors", :red) if errors > 0
+    status_parts << "#{skipped} skipped" if skipped > 0
+    status_parts << format_duration(elapsed)
+
+    symbol = errors > 0 ? colorize("✗", :red) : colorize("✓", :green)
+    write_line "#{symbol} #{@current_file}: #{status_parts.join(", ")}"
+
+    @current_file = nil
+    @current_section = nil
+  end
+
+  # Called when all files are processed
+  def finish_ingestion
+    elapsed = Time.now - @ingestion_start_time
+    clear_line
+
+    write_line ""
+    write_line "─" * 50
+    write_line "Ingestion complete in #{format_duration(elapsed)}"
+    write_line "  Files processed: #{@current_file_index}"
+    write_line "  Facts extracted: #{@total_facts}"
+    write_line "  Entities found:  #{@total_entities}"
+    write_line colorize("  Errors: #{@total_errors}", :red) if @total_errors > 0
+    write_line ""
+  end
+
+  # Called to report files that were already processed
+  def report_already_processed(count)
+    return if count == 0
+
+    write_line colorize("  (#{count} file(s) already processed, skipping)", :dim)
+  end
+
+  # Called when no files need processing
+  def no_files_to_process
+    write_line colorize("  All files already processed. Use --rebuild to reprocess.", :dim)
+  end
+
+  private
+
+  def update_status(extracting: false)
+    clear_line
+
+    parts = []
+
+    # File progress
+    parts << "[#{@current_file_index}/#{@total_files}]"
+
+    # Current file (truncated)
+    if @current_file
+      parts << truncate(@current_file, 25)
+    end
+
+    # Section progress
+    if @current_section && @total_sections > 0
+      parts << "#{@current_section_index}/#{@total_sections} sections"
+    end
+
+    # Extraction indicator with spinner and elapsed time
+    if extracting && @extraction_start_time
+      elapsed = Time.now - @extraction_start_time
+      spinner = SPINNER_CHARS[@spinner_index]
+      parts << "#{spinner} extracting (#{format_duration(elapsed)})"
+    end
+
+    # Running totals
+    if @total_facts > 0 || @file_facts > 0
+      parts << "#{@total_facts} facts"
+      parts << "#{@total_entities} entities"
+    end
+
+    status = parts.join(" │ ")
+    write_status(status)
+  end
+
+  def write_status(text)
+    @line_length = text.length
+    @output.print "\r#{text}"
+    @output.flush
+  end
+
+  def clear_line
+    return if @line_length == 0
+
+    @output.print "\r#{" " * @line_length}\r"
+    @output.flush
+    @line_length = 0
+  end
+
+  def write_line(text)
+    clear_line
+    @output.puts text
+  end
+
+  def format_duration(seconds)
+    if seconds < 60
+      format("%.1fs", seconds)
+    elsif seconds < 3600
+      minutes = (seconds / 60).to_i
+      secs = (seconds % 60).to_i
+      "#{minutes}m #{secs}s"
+    else
+      hours = (seconds / 3600).to_i
+      minutes = ((seconds % 3600) / 60).to_i
+      "#{hours}h #{minutes}m"
+    end
+  end
+
+  def truncate(text, length)
+    return text if text.length <= length
+
+    text[0, length - 1] + "…"
+  end
+
+  def colorize(text, color)
+    return text unless @color
+
+    codes = {
+      red: "\e[31m",
+      green: "\e[32m",
+      yellow: "\e[33m",
+      dim: "\e[2m",
+      reset: "\e[0m"
+    }
+
+    "#{codes[color]}#{text}#{codes[:reset]}"
+  end
+end
+
+
+# QuietReporter - Minimal output for scripting/automation
+#
+# Use this when you want minimal output, e.g., in CI pipelines
+# or when redirecting output to a file.
+class QuietReporter < IngestReporter
+  def start_ingestion(total_files:, source_path:)
+    @total_files = total_files
+    @ingestion_start_time = Time.now
+    @total_facts = 0
+    @total_entities = 0
+    @total_errors = 0
+  end
+
+  def file_started(filename, index, total)
+    @current_file = filename
+    @current_file_index = index
+    @file_start_time = Time.now
+  end
+
+  def section_started(section_ref, index, total); end
+  def section_skipped(section_ref); end
+  def extraction_started; end
+  def extraction_progress; end
+  def extraction_completed(facts_count:, entities_count:)
+    @total_facts += facts_count
+    @total_entities += entities_count
+  end
+  def section_completed; end
+
+  def error_occurred(error, context: nil)
+    @total_errors += 1
+    error_msg = context ? "#{context}: #{error.message}" : error.message
+    @output.puts "ERROR: #{error_msg}"
+  end
+
+  def file_completed(facts:, entities:, errors:, skipped:)
+    @output.puts "#{@current_file}: #{facts} facts, #{entities} entities"
+  end
+
+  def finish_ingestion
+    elapsed = Time.now - @ingestion_start_time
+    @output.puts "Completed: #{@total_facts} facts, #{@total_entities} entities in #{format_duration(elapsed)}"
+  end
+
+  def report_already_processed(count); end
+  def no_files_to_process
+    @output.puts "All files already processed."
+  end
+end
+
+
+# VerboseReporter - Detailed output for debugging
+#
+# Shows all details including section names and extraction timing.
+class VerboseReporter < IngestReporter
+  def section_started(section_ref, index, total)
+    super
+    write_line "    Section #{index}/#{total}: #{section_ref}"
+  end
+
+  def extraction_completed(facts_count:, entities_count:)
+    elapsed = @extraction_start_time ? Time.now - @extraction_start_time : 0
+    super
+    write_line "      Extracted #{facts_count} facts, #{entities_count} entities (#{format_duration(elapsed)})"
+  end
+
+  def error_occurred(error, context: nil)
+    super
+    write_line "      #{error.backtrace&.first}" if error.backtrace
+  end
+end

data/examples/utilities.rb

@@ -0,0 +1,182 @@
+# frozen_string_literal: true
+
+# Common utilities for FactDb demo programs
+#
+# Usage: require_relative "utilities"
+#
+# This file provides:
+# - Environment setup (ensures FDB_ENV=demo)
+# - Database reset before each demo
+# - Common output formatting methods
+
+module DemoUtilities
+  SEPARATOR_WIDTH = 60
+
+  class << self
+    # Setup the demo environment and reset the database
+    # Call this at the start of each demo
+    def setup!(demo_name = nil)
+      ensure_demo_environment!
+      reset_demo_database!
+      require_fact_db!
+
+      if demo_name
+        header(demo_name)
+      end
+    end
+
+    # Setup for CLI tools - sets environment but does NOT reset database
+    # Use this for utility scripts that inspect/query existing data
+    def cli_setup!(tool_name = nil)
+      ensure_demo_environment!
+      require_fact_db!
+      FactDb::Database.establish_connection!
+
+      if tool_name
+        header(tool_name)
+      end
+    end
+
+    # Ensure FDB_ENV is set to "demo"
+    def ensure_demo_environment!
+      ENV["FDB_ENV"] = "demo"
+    end
+
+    # Reset the demo database using rake task
+    def reset_demo_database!
+      project_root = File.expand_path("..", __dir__)
+
+      # Run rake db:reset:demo quietly
+      Dir.chdir(project_root) do
+        system("bundle exec rake db:reset:demo > /dev/null 2>&1")
+      end
+    end
+
+    # Require fact_db after environment is set
+    def require_fact_db!
+      require "bundler/setup"
+      require "fact_db"
+    end
+
+    # Print a major header (demo title)
+    def header(title)
+      puts separator
+      puts title
+      puts separator
+    end
+
+    # Print a section header
+    def section(title)
+      str     = "--- #{title} ---"
+      wrapper = str[0] * str.length
+      puts "\n#{wrapper}\n#{str}\n#{wrapper}"
+    end
+
+    # Print a subsection header
+    def subsection(title)
+      puts "\n#{title}:"
+    end
+
+    # Print the demo completion footer
+    def footer(title = "Demo Complete!")
+      puts "\n" + separator
+      puts title
+      puts separator
+    end
+
+    # Print a separator line
+    def separator
+      "=" * SEPARATOR_WIDTH
+    end
+
+    # Print a list item
+    def list_item(text, indent: 2)
+      puts "#{" " * indent}- #{text}"
+    end
+
+    # Print an indented line
+    def indent(text, level: 1)
+      puts "#{"  " * level}#{text}"
+    end
+
+    # Print a key-value pair
+    def kv(key, value, indent_level: 1)
+      puts "#{"  " * indent_level}#{key}: #{value}"
+    end
+
+    # Print multiple lines with consistent formatting
+    def block(lines)
+      puts lines.map { |line| "  #{line}" }.join("\n")
+    end
+
+    # Configure logging to a file based on demo filename
+    # Overwrites the log file on each run (does not append)
+    def configure_logging(demo_file)
+      log_path = File.join(File.dirname(demo_file), "#{File.basename(demo_file, '.rb')}.log")
+
+      FactDb.configure do |config|
+        config.logger = Logger.new(File.open(log_path, 'w'))
+      end
+    end
+
+    # Create a new FactDb instance with optional logging setup
+    def create_fact_db(demo_file = nil)
+      configure_logging(demo_file) if demo_file
+      FactDb.new
+    end
+  end
+end
+
+##########################################################
+# Convenience methods at top level for cleaner demo code
+def demo_setup!(demo_name = nil)
+  DemoUtilities.setup!(demo_name)
+end
+
+def demo_header(title)
+  DemoUtilities.header(title)
+end
+
+def demo_section(title)
+  DemoUtilities.section(title)
+end
+
+def demo_subsection(title)
+  DemoUtilities.subsection(title)
+end
+
+def demo_footer(title = "Demo Complete!")
+  DemoUtilities.footer(title)
+end
+
+def demo_separator
+  DemoUtilities.separator
+end
+
+def demo_list_item(text, indent: 2)
+  DemoUtilities.list_item(text, indent: indent)
+end
+
+def demo_indent(text, level: 1)
+  DemoUtilities.indent(text, level: level)
+end
+
+def demo_kv(key, value, indent_level: 1)
+  DemoUtilities.kv(key, value, indent_level: indent_level)
+end
+
+def demo_block(lines)
+  DemoUtilities.block(lines)
+end
+
+def demo_configure_logging(demo_file)
+  DemoUtilities.configure_logging(demo_file)
+end
+
+def demo_create_fact_db(demo_file = nil)
+  DemoUtilities.create_fact_db(demo_file)
+end
+
+def cli_setup!(tool_name = nil)
+  DemoUtilities.cli_setup!(tool_name)
+end