agent-tome 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5ea76c93f40a8d4e0a643df1ed73eef19954938d1e2b28977564a82a58284c93
+  data.tar.gz: 801d26d865bbe49e1975dd4f8039d008c37de5a12822850a8e32041c8e912a99
+SHA512:
+  metadata.gz: 81fc0b5eedfc6126e23d553abbf8a2359c5e98a118bbba065b4c847be56eb9303d6b3f4885ee2649d11ef02f2c1fae6370f78c205e4ecd884bdbfb1ecb79a27e
+  data.tar.gz: 62c870a453dde2d2d06b8e26b2361267f211691a3962a6e477a30e96cec749a69f438f2f4dbd44fe5cdd3e3bf6e0ad4c8df0a2b46449c66c3dc20bbc2489d652

data/CHANGELOG.md

@@ -0,0 +1,22 @@
+## [Unreleased]
+
+## [1.0.0] - 2026-04-03
+
+### Added
+
+- **Article management**: Create articles with body, description, keywords, sources, and related article references. Add content addenda to existing articles.
+- **Search**: Keyword-based article search with `--match any` (default) and `--match all` modes, results ordered by keyword relevance, capped at 1000.
+- **Fetch**: Retrieve full article content including all entries, sources, and consolidation history.
+- **Related articles**: Discover related articles via shared keywords, article references (both directions), and consolidation links (both directions).
+- **Consolidation**: Merge all addenda into a single consolidated entry, preserving keywords, sources, and original article IDs for continued lookup.
+- **Keyword discovery**: List keywords matching a prefix/substring for vocabulary exploration.
+- **Source search**: Find articles by web URL or file path, with optional `--system` flag for scoped file path lookups.
+- **Append-only SQLite store**: WAL mode and busy timeout for concurrent access. Automatic migration on first run.
+- **Opaque identifiers**: All user-facing entities use randomly generated 7-character base58 global IDs. Internal integer IDs are never exposed.
+- **Input validation**: Structured JSON error responses for missing fields, invalid URLs, blank bodies, oversized descriptions (350 char limit), self-references, empty stdin, and invalid JSON.
+- **Data integrity**: Keyword singularisation and downcasing, web URL normalisation (strips tracking params), source deduplication, duplicate keyword and reference handling.
+- **CLI**: `agent-tome` executable with JSON stdin/stdout interface, exit code 0 on success and non-zero on error.
+- **Claude Code skills**: Companion `tome-lookup` and `tome-capture` skills for AI agent integration.
+
+[Unreleased]: https://github.com/beatmadsen/agent-tome/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/beatmadsen/agent-tome/releases/tag/v1.0.0

data/CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Contributing
+
+Issues and pull requests welcome. Run `bin/setup` then `rake test`.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Erik T. Madsen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,81 @@
+# Agent Tome
+
+Build a personal encyclopedia of accumulated knowledge, designed for AI agents.
+
+AI agents are brilliant researchers, and terrible at remembering what they found. Every new session starts from scratch: the same APIs re-discovered, the same docs re-read. Agent Tome gives your agent a place to store what it learns, so you can build the habits and tooling to stop repeating that work.
+
+## Installation
+
+```bash
+gem install agent-tome
+```
+
+On first run, a config directory is created at `~/.agent-tome/` with a SQLite database at `~/.agent-tome/tome.db`. To use a different database location, edit `~/.agent-tome/config.yml` and change `db_path`.
+
+## Quick Start
+
+```bash
+# Your agent just researched Ruby GC internals. Capture what it learned:
+echo '{"description": "Ruby GC internals", "body": "Ruby uses a generational mark-and-sweep garbage collector.", "keywords": ["ruby", "garbage-collection"]}' | agent-tome create
+# => {"global_id": "3xK9mWp", ...}
+
+# Two weeks later, a different session hits the same topic.
+# Instead of re-reading the docs, your agent checks the tome:
+agent-tome search ruby gc
+
+# The article exists. The agent adds what it learned today:
+echo '{"body": "GC compaction was added in Ruby 2.7 via GC.compact."}' | agent-tome addend 3xK9mWp
+
+# Fetch the full article with all entries
+agent-tome fetch 3xK9mWp
+```
+
+## Knowledge Evolution
+
+Articles grow over time as your agent adds findings from different sessions:
+
+    Session 1: "Ruby uses mark-and-sweep GC"
+    Session 3: "GC compaction added in Ruby 2.7"
+    Session 7: "Tuning GC with RUBY_GC_HEAP_INIT_SLOTS"
+
+Eventually, the article has redundant or overlapping entries. `consolidate` lets your agent (or you) synthesize them into a single authoritative entry:
+
+    "Ruby uses generational mark-and-sweep GC with optional compaction
+    (since 2.7). Key tuning env vars: RUBY_GC_HEAP_INIT_SLOTS, ..."
+
+The original entries are preserved. The article ID still works. Keywords and sources are carried forward. This is how a tome stays useful over months instead of becoming a pile of notes.
+
+## Commands
+
+| Command | Description |
+|---------|-------------|
+| `agent-tome create` | Create a new article with its first entry (JSON via stdin) |
+| `agent-tome addend <id>` | Add an addendum to an existing article (JSON via stdin) |
+| `agent-tome search <keywords...>` | Search articles by keywords (`--match any` or `--match all`) |
+| `agent-tome fetch <id>` | Retrieve full article content with all entries and sources |
+| `agent-tome related <id>` | Find articles related through shared keywords or references |
+| `agent-tome consolidate <id>` | Merge all addenda into a single consolidated article (JSON via stdin) |
+| `agent-tome keywords <prefix>` | List keywords matching a prefix for vocabulary discovery |
+| `agent-tome source-search <url-or-path>` | Find articles referencing a given source |
+
+## Agent Integration
+
+Agent Tome works with any tool that can call a CLI. JSON in, JSON out, exit code 0 on success:
+
+- **Claude Code**: Use the companion [tome-lookup](https://github.com/beatmadsen/claude-skills/tree/main/skills/tome-lookup) and [tome-capture](https://github.com/beatmadsen/claude-skills/tree/main/skills/tome-capture) skills from the [claude-skills](https://github.com/beatmadsen/claude-skills) repository
+- **Cursor / Windsurf**: Call `agent-tome` as a custom tool or shell command
+- **MCP servers**: Wrap the CLI as a tool definition
+- **Any agent framework**: If it can `exec` a process and read stdout, it works
+
+No SDK, no API keys, no server. The CLI is the interface.
+
+## Design Principles
+
+- **Append-only**: Your agent can never corrupt or lose knowledge. There are no updates or deletions. When information evolves, `consolidate` synthesizes a clean entry while keeping the full history.
+- **Agent-first**: JSON in, JSON out. Simple commands with minimal flags. Every response is machine-parseable, every error is structured.
+- **Keyword-based discovery**: Retrieval is deterministic. You always know exactly why a result came back and what keywords will find it again. No embeddings to configure, no similarity thresholds to tune.
+- **Opaque identifiers**: All user-facing entities use randomly generated 7-character base58 global IDs. Internal details never leak into your agent's context.
+
+## License
+
+MIT

data/db/migrate/20250601000001_create_initial_schema.rb

@@ -0,0 +1,85 @@
+class CreateInitialSchema < ActiveRecord::Migration[7.1]
+  def change
+    create_table :articles do |t|
+      t.string :global_id, limit: 7, null: false
+      t.text :description, null: false
+      t.datetime :created_at, null: false
+    end
+    add_index :articles, :global_id, unique: true
+
+    create_table :entries do |t|
+      t.string :global_id, limit: 7, null: false
+      t.references :article, null: false, foreign_key: true
+      t.text :body
+      t.datetime :created_at, null: false
+    end
+    add_index :entries, :global_id, unique: true
+
+    create_table :keywords do |t|
+      t.string :term, null: false
+      t.datetime :created_at, null: false
+    end
+    add_index :keywords, :term, unique: true
+
+    create_table :article_keywords do |t|
+      t.references :article, null: false, foreign_key: true
+      t.references :keyword, null: false, foreign_key: true
+      t.datetime :created_at, null: false
+    end
+    add_index :article_keywords, [:article_id, :keyword_id], unique: true
+
+    create_table :web_sources do |t|
+      t.string :global_id, limit: 7, null: false
+      t.text :url, null: false
+      t.string :title
+      t.datetime :fetched_at
+      t.datetime :created_at, null: false
+    end
+    add_index :web_sources, :global_id, unique: true
+    add_index :web_sources, :url, unique: true
+
+    create_table :file_sources do |t|
+      t.string :global_id, limit: 7, null: false
+      t.text :path, null: false
+      t.string :system_name, null: false
+      t.datetime :created_at, null: false
+    end
+    add_index :file_sources, :global_id, unique: true
+    add_index :file_sources, [:path, :system_name], unique: true
+
+    create_table :entry_web_sources do |t|
+      t.references :entry, null: false, foreign_key: true
+      t.references :web_source, null: false, foreign_key: true
+      t.datetime :created_at, null: false
+    end
+    add_index :entry_web_sources, [:entry_id, :web_source_id], unique: true
+
+    create_table :entry_file_sources do |t|
+      t.references :entry, null: false, foreign_key: true
+      t.references :file_source, null: false, foreign_key: true
+      t.datetime :created_at, null: false
+    end
+    add_index :entry_file_sources, [:entry_id, :file_source_id], unique: true
+
+    create_table :article_references do |t|
+      t.bigint :source_article_id, null: false
+      t.bigint :target_article_id, null: false
+      t.datetime :created_at, null: false
+    end
+    add_index :article_references, :source_article_id
+    add_index :article_references, :target_article_id
+    add_index :article_references, [:source_article_id, :target_article_id], unique: true, name: "idx_article_refs_unique"
+    add_foreign_key :article_references, :articles, column: :source_article_id
+    add_foreign_key :article_references, :articles, column: :target_article_id
+
+    create_table :consolidation_links do |t|
+      t.bigint :new_article_id, null: false
+      t.bigint :old_article_id, null: false
+      t.datetime :created_at, null: false
+    end
+    add_index :consolidation_links, :new_article_id
+    add_index :consolidation_links, :old_article_id
+    add_foreign_key :consolidation_links, :articles, column: :new_article_id
+    add_foreign_key :consolidation_links, :articles, column: :old_article_id
+  end
+end

data/exe/agent-tome

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "agent/tome"
+
+Agent::Tome::CLI.run(ARGV)

data/lib/agent/tome/cli.rb

@@ -0,0 +1,178 @@
+require "json"
+require "optparse"
+
+module Agent
+  module Tome
+    class CLI
+      def self.run(argv)
+        new(argv).run
+      end
+
+      def initialize(argv)
+        @argv = argv.dup
+      end
+
+      def run
+        command = @argv.shift
+
+        if command.nil? || command == "--help" || command == "-h"
+          print_usage
+          exit 0
+        end
+
+        if command == "--version" || command == "version"
+          puts VERSION
+          exit 0
+        end
+
+        config = Config.new
+        begin
+          config.load!
+        rescue ConfigError => e
+          output_error(e.message)
+          exit 1
+        end
+
+        begin
+          Database.connect!(config.db_path)
+        rescue DatabaseError => e
+          output_error(e.message)
+          exit 1
+        end
+
+        dispatch(command)
+      rescue ValidationError => e
+        output_error(e.message)
+        exit 1
+      rescue NotFoundError => e
+        output_error(e.message)
+        exit 1
+      rescue => e
+        output_error(e.message)
+        exit 1
+      end
+
+      private
+
+      def dispatch(command)
+        case command
+        when "create"
+          input = read_stdin_json
+          result = Commands::Create.new.call(input)
+          puts JSON.generate(result)
+        when "addend"
+          article_id = @argv.shift
+          raise ValidationError, "article_global_id is required" if article_id.nil? || article_id.strip.empty?
+
+          input = read_stdin_json
+          result = Commands::Addend.new(article_global_id: article_id).call(input)
+          puts JSON.generate(result)
+        when "search"
+          keywords, match = parse_search_args
+          result = Commands::Search.new(keywords: keywords, match: match).call
+          puts JSON.generate(result)
+        when "fetch"
+          global_id = @argv.shift
+          raise ValidationError, "global_id is required" if global_id.nil? || global_id.strip.empty?
+
+          result = Commands::Fetch.new(global_id: global_id).call
+          puts JSON.generate(result)
+        when "consolidate"
+          global_id = @argv.shift
+          raise ValidationError, "global_id is required" if global_id.nil? || global_id.strip.empty?
+
+          input = read_stdin_json
+          result = Commands::Consolidate.new(global_id: global_id).call(input)
+          puts JSON.generate(result)
+        when "related"
+          global_id = @argv.shift
+          raise ValidationError, "global_id is required" if global_id.nil? || global_id.strip.empty?
+
+          result = Commands::Related.new(global_id: global_id).call
+          puts JSON.generate(result)
+        when "keywords"
+          prefix = @argv.shift
+          result = Commands::KeywordsList.new(prefix: prefix).call
+          puts JSON.generate(result)
+        when "source-search"
+          source, system = parse_source_search_args
+          result = Commands::SourceSearch.new(source: source, system: system).call
+          puts JSON.generate(result)
+        else
+          output_error("Unknown command: #{command}")
+          exit 1
+        end
+
+        exit 0
+      end
+
+      def read_stdin_json
+        if $stdin.tty?
+          $stderr.puts "Reading JSON from stdin (Ctrl-D to finish)..."
+        end
+        raw = $stdin.read
+        raise ValidationError, "Empty input" if raw.nil? || raw.strip.empty?
+
+        JSON.parse(raw)
+      rescue JSON::ParserError => e
+        raise ValidationError, "Invalid JSON input: #{e.message}"
+      end
+
+      def parse_search_args
+        match = "any"
+        keywords = []
+
+        i = 0
+        while i < @argv.length
+          if @argv[i] == "--match" && i + 1 < @argv.length
+            match = @argv[i + 1]
+            i += 2
+          else
+            keywords << @argv[i]
+            i += 1
+          end
+        end
+
+        [keywords, match]
+      end
+
+      def parse_source_search_args
+        source = @argv.shift
+        system = nil
+
+        if @argv[0] == "--system" && @argv[1]
+          @argv.shift
+          system = @argv.shift
+        end
+
+        [source, system]
+      end
+
+      def print_usage
+        $stderr.puts <<~USAGE
+          Usage: agent-tome <command> [options]
+
+          Commands:
+            create                    Create a new article (JSON via stdin)
+            addend <id>               Add an addendum to an article (JSON via stdin)
+            search <keywords...>      Search articles by keywords
+            fetch <id>                Retrieve full article content
+            related <id>              Find related articles
+            consolidate <id>          Merge addenda into one article (JSON via stdin)
+            keywords <prefix>         List keywords matching a prefix
+            source-search <url|path>  Find articles by source reference
+
+          Options:
+            --help, -h                Show this help
+            --version                 Show version
+
+          See https://github.com/beatmadsen/agent-tome for full documentation.
+        USAGE
+      end
+
+      def output_error(message)
+        puts JSON.generate({ "error" => message })
+      end
+    end
+  end
+end

data/lib/agent/tome/commands/addend.rb

@@ -0,0 +1,158 @@
+require "active_support/core_ext/string/inflections"
+
+module Agent
+  module Tome
+    module Commands
+      class Addend
+        def initialize(article_global_id:)
+          @article_global_id = article_global_id
+        end
+
+        def call(input)
+          article = Article.find_by(global_id: @article_global_id)
+          raise NotFoundError, "Article not found: #{@article_global_id}" unless article
+
+          validate!(input)
+
+          result = {}
+
+          ActiveRecord::Base.transaction do
+            entry = Entry.create!(
+              article: article,
+              body: input["body"].to_s.strip.empty? ? nil : input["body"],
+              created_at: Time.now
+            )
+
+            process_keywords!(article, input["keywords"] || [])
+            web_source_ids = process_web_sources!(entry, input["web_sources"] || [])
+            file_source_ids = process_file_sources!(entry, input["file_sources"] || [])
+            process_related_articles!(article, input["related_article_ids"] || [])
+
+            result = {
+              "entry_global_id" => entry.global_id,
+              "web_source_global_ids" => web_source_ids,
+              "file_source_global_ids" => file_source_ids
+            }
+          end
+
+          result
+        end
+
+        private
+
+        def validate!(input)
+          body = input["body"]
+          if body && !body.to_s.strip.empty?
+            return
+          end
+
+          keywords = input["keywords"] || []
+          web_sources = input["web_sources"] || []
+          file_sources = input["file_sources"] || []
+          related = input["related_article_ids"] || []
+
+          if body.is_a?(String) && body.strip.empty?
+            raise ValidationError, "body cannot be blank"
+          end
+
+          if keywords.empty? && web_sources.empty? && file_sources.empty? && related.empty?
+            raise ValidationError, "At least one field must be substantively present"
+          end
+
+          validate_keywords!(keywords) if keywords.any?
+          validate_web_sources!(web_sources) if web_sources.any?
+          validate_file_sources!(file_sources) if file_sources.any?
+          validate_related_ids!(input["related_article_ids"]) if input.key?("related_article_ids")
+        end
+
+        def validate_keywords!(keywords)
+          keywords.each do |kw|
+            raise ValidationError, "keyword must be a non-empty string" unless kw.is_a?(String) && !kw.strip.empty?
+          end
+        end
+
+        def validate_web_sources!(sources)
+          sources.each do |src|
+            raise ValidationError, "invalid URL: #{src["url"]}" unless UrlNormalizer.valid?(src["url"].to_s)
+          end
+        end
+
+        def validate_file_sources!(sources)
+          sources.each do |src|
+            raise ValidationError, "file_source path cannot be empty" if src["path"].to_s.strip.empty?
+            raise ValidationError, "file_source system_name cannot be empty" if src["system_name"].to_s.strip.empty?
+          end
+        end
+
+        def validate_related_ids!(ids)
+          return unless ids
+
+          ids.each do |id|
+            raise ValidationError, "Referenced article not found: #{id}" unless Article.exists?(global_id: id)
+          end
+        end
+
+        def process_keywords!(article, keywords)
+          keywords.each do |kw|
+            normalized = normalize_keyword(kw)
+            keyword = Keyword.find_or_create_by!(term: normalized) do |k|
+              k.created_at = Time.now
+            end
+            ArticleKeyword.find_or_create_by!(article: article, keyword: keyword) do |ak|
+              ak.created_at = Time.now
+            end
+          end
+        end
+
+        def process_web_sources!(entry, sources)
+          sources.map do |src|
+            normalized_url = UrlNormalizer.normalize(src["url"])
+            ws = WebSource.find_or_create_by!(url: normalized_url) do |w|
+              w.global_id = GlobalId.generate
+              w.title = src["title"]
+              w.fetched_at = src["fetched_at"] ? Time.parse(src["fetched_at"]) : nil
+              w.created_at = Time.now
+            end
+            EntryWebSource.find_or_create_by!(entry: entry, web_source: ws) do |ews|
+              ews.created_at = Time.now
+            end
+            ws.global_id
+          end
+        end
+
+        def process_file_sources!(entry, sources)
+          sources.map do |src|
+            fs = FileSource.find_or_create_by!(path: src["path"], system_name: src["system_name"]) do |f|
+              f.global_id = GlobalId.generate
+              f.created_at = Time.now
+            end
+            EntryFileSource.find_or_create_by!(entry: entry, file_source: fs) do |efs|
+              efs.created_at = Time.now
+            end
+            fs.global_id
+          end
+        end
+
+        def process_related_articles!(article, related_ids)
+          related_ids.each do |target_id|
+            raise ValidationError, "An article cannot reference itself" if target_id == article.global_id
+
+            target = Article.find_by!(global_id: target_id)
+            ArticleReference.find_or_create_by!(
+              source_article: article,
+              target_article: target
+            ) do |ref|
+              ref.created_at = Time.now
+            end
+          end
+        end
+
+        def normalize_keyword(kw)
+          words = kw.downcase.split("-")
+          words[-1] = ActiveSupport::Inflector.singularize(words[-1])
+          words.join("-")
+        end
+      end
+    end
+  end
+end

data/lib/agent/tome/commands/consolidate.rb

@@ -0,0 +1,90 @@
+module Agent
+  module Tome
+    module Commands
+      class Consolidate
+        def initialize(global_id:)
+          @global_id = global_id
+        end
+
+        def call(input)
+          article = Article.find_by(global_id: @global_id)
+          raise NotFoundError, "Article not found: #{@global_id}" unless article
+
+          validate!(input)
+
+          result = {}
+
+          ActiveRecord::Base.transaction do
+            original_global_id = article.global_id
+            new_description = input["description"] || article.description
+
+            # Assign old article a new global_id
+            old_global_id = GlobalId.generate
+            article.update_columns(global_id: old_global_id)
+
+            # Create the new consolidated article with the original global_id
+            new_article = Article.create!(
+              global_id: original_global_id,
+              description: new_description,
+              created_at: Time.now
+            )
+
+            # Create the first (and only) entry for the consolidated article
+            new_entry = Entry.create!(
+              article: new_article,
+              body: input["body"],
+              created_at: Time.now
+            )
+
+            # Copy all sources from old article's entries to the consolidated entry
+            article.entries.each do |old_entry|
+              old_entry.web_sources.each do |ws|
+                EntryWebSource.find_or_create_by!(entry: new_entry, web_source: ws)
+              end
+              old_entry.file_sources.each do |fs|
+                EntryFileSource.find_or_create_by!(entry: new_entry, file_source: fs)
+              end
+            end
+
+            # Copy keywords from old article to new article
+            article.keywords.each do |keyword|
+              ArticleKeyword.find_or_create_by!(article: new_article, keyword: keyword) do |ak|
+                ak.created_at = Time.now
+              end
+            end
+
+            # Create consolidation link
+            ConsolidationLink.create!(
+              new_article: new_article,
+              old_article: article,
+              created_at: Time.now
+            )
+
+            result = {
+              "new_article_global_id" => new_article.global_id,
+              "old_article_global_id" => article.global_id
+            }
+          end
+
+          result
+        end
+
+        private
+
+        def validate!(input)
+          raise ValidationError, "body is required" unless input.key?("body")
+
+          body = input["body"]
+          raise ValidationError, "body must be a string" unless body.is_a?(String)
+          raise ValidationError, "body cannot be blank" if body.strip.empty?
+
+          if input.key?("description")
+            desc = input["description"]
+            raise ValidationError, "description must be a string" unless desc.is_a?(String)
+            raise ValidationError, "description must be 350 characters or fewer" if desc.length > 350
+          end
+        end
+      end
+    end
+  end
+end