htm 0.0.11 → 0.0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c6e41109b86fd5c51c8f96cec8a11ea9455035da54d72ae461b3b2f0a7ab7a8d
-  data.tar.gz: 1fe15917933aefa827a12c6e92406711ab95449097cb78bdab4c0f6510f595c1
+  metadata.gz: 9ae4db5f168c495f5afef19f740f16ee9b39136cd54403f54090921cb6952275
+  data.tar.gz: 454816f53e620a72efeae7bd8b6c0a30c459403b4f1b6a3d0ca7adf30f3cda7c
 SHA512:
-  metadata.gz: c02e326797a6986e1a5b987bec948f09c0ffc7cb45045898e3fc6faf5c57c9bf3ca4d399324e88528f994c7e3e1989bb71f462aae8adbb270b7c99654aa135d3
-  data.tar.gz: 4ee90ea84f36ab2d85c72cae4f5a1382e9a174d354f2d976d73b8915cb5dc8df68f6ae799ec082076293ce9b386500b94b4fe12d291903c0804351f5fd027151
+  metadata.gz: 52adbf798f4004961936f9112878d245c6872bf812611efa47bc9b7808ba39510d79d0902f109db8169e1cd0d5420338094edd7719f995f67c1b3def07a2a4d2
+  data.tar.gz: 7ac9e6a8913b9b836ddfdfdb909857ac62bdd2c5d6c6c904c2a257224fc80499a118afd3d63a3a14d0daed838e11c02b2f2149eddb8c0385ebb2d5be9ab258f8

data/.dictate.toml

@@ -0,0 +1,46 @@
+# Dictator Configuration for HTM
+# Decree-based structural enforcement
+# https://github.com/seuros/dictator
+
+[decree.supreme]
+# Universal structural rules (applies to ALL files)
+trailing_whitespace   = "deny"
+tabs_vs_spaces        = "spaces"
+tab_width             = 2
+final_newline         = "require"
+line_endings          = "lf"
+blank_line_whitespace = "deny"
+
+[decree.ruby]
+# Ruby-specific structural enforcement
+max_line_length         = 120
+max_lines               = 300
+ignore_comments         = true
+ignore_blank_lines      = true
+method_visibility_order = ["public", "protected", "private"]
+comment_spacing         = true
+
+[decree.ruby.linter]
+# External linter - Dictator adds: -A --format json
+command = "rubocop"
+
+[decree.frontmatter]
+# YAML frontmatter ordering for .md files
+order    = ["title", "slug", "pubDate", "description", "tags", "draft"]
+required = ["title"]
+
+# Disable unused language decrees
+[decree.typescript]
+enabled = false
+
+[decree.golang]
+enabled = false
+
+[decree.rust]
+enabled = false
+
+[decree.python]
+enabled = false
+
+[decree.kjr]
+enabled = false

data/.envrc

@@ -2,6 +2,8 @@
 
 export RR=`pwd`
 
+export HTM_EXTRACT_PROPOSITIONS=true
+
 # Database connection - Localhost PostgreSQL
 export HTM_DBHOST=localhost
 export HTM_DBPORT=5432

data/CHANGELOG.md

@@ -7,6 +7,89 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.0.15] - 2025-12-06
+
+### Added
+- **"Why Robots Instead of Agents?" README section** - Explains terminology choice
+  - Opens with Shakespeare quote from Romeo and Juliet
+  - Discusses semantic clarity, honest capabilities, avoiding hype cycle
+  - Critiques "agent" frameworks that are just prompt wrappers
+  - References rich heritage (Čapek's R.U.R., Asimov's Three Laws)
+  - Notes cultural personality of robots (R2-D2, Wall-E, Bender)
+- **MCP Server documentation** - Comprehensive guide for AI assistant integration
+  - Added MCP Server section to README.md with tools/resources reference
+  - Created `docs/guides/mcp-server.md` with detailed configuration for Claude Desktop, Claude Code, and AIA
+  - Covers all 11 tools (SetRobot, Remember, Recall, Forget, Restore, ListTags, SearchTags, FindByTopic, Stats, GetRobot, GetWorkingMemory)
+  - Covers all 3 resources (statistics, tags/hierarchy, memories/recent)
+  - Usage examples, session management, and troubleshooting guide
+- **HTM Rake Tasks Reference** - Complete documentation for all 44 rake tasks
+  - Added rake tasks summary table to README.md Development section
+  - Created `docs/development/rake-tasks.md` with detailed reference organized by namespace:
+    - Database Tasks (`htm:db:*`) - 16 tasks
+    - Documentation Tasks (`htm:doc:*`) - 9 tasks
+    - File Loading Tasks (`htm:files:*`) - 7 tasks
+    - Job Tasks (`htm:jobs:*`) - 7 tasks
+    - Tag Tasks (`htm:tags:*`) - 5 tasks
+  - Common workflows, environment variables, and troubleshooting
+
+### Fixed
+- **Documentation code examples** - Fixed incorrect API usage in docs/guides/*.md
+  - `add_node()` → `remember()`
+  - `topic:` named parameter → positional argument in `recall()`
+  - `retrieve(key)` → `long_term_memory.retrieve(node_id)`
+  - `create_context()` → `working_memory.assemble_context()`
+  - Removed invalid parameters: `type:`, `importance:`, `category:`, `related_to:`
+
+## [0.0.14] - 2025-12-05
+
+### Added
+- **OpenTelemetry metrics** - Optional observability with zero overhead when disabled
+  - New `HTM::Telemetry` module with null object pattern
+  - Metrics: `htm.jobs` (counter), `htm.embedding.latency`, `htm.tag.latency`, `htm.search.latency` (histograms), `htm.cache.operations` (counter)
+  - Instrumented: `GenerateEmbeddingJob`, `GenerateTagsJob`, search methods, `QueryCache`
+  - Enable via `HTM_TELEMETRY_ENABLED=true` or `config.telemetry_enabled = true`
+  - Works with 50+ OTLP-compatible backends (Jaeger, Prometheus, Datadog, etc.)
+  - Comprehensive documentation in `docs/telemetry.md`
+  - 18 new telemetry tests
+- **Telemetry demo with Grafana visualization** - `examples/telemetry/`
+  - Live dashboard showing HTM metrics (job counts, latencies, cache hit rates)
+  - Uses Homebrew-installed Prometheus and Grafana (no Docker required)
+  - Auto-configures Prometheus scrape target
+  - Pre-built Grafana dashboard JSON for easy import
+  - Interactive shutdown prompt for service management
+- **`htm:db:create` rake task** - Create database if it doesn't exist (respects `RAILS_ENV`)
+
+### Changed
+- **All `htm:db:*` tasks now respect `RAILS_ENV`** - Following Rails conventions
+  - Database selection based on environment: `htm_development`, `htm_test`, `htm_production`
+  - `rake test` automatically sets `RAILS_ENV=test`
+  - Example: `RAILS_ENV=test rake htm:db:setup` operates on `htm_test`
+- **`config/database.yml` refactored** - Extracts base name from `HTM_DBURL` and appends environment suffix
+  - `HTM_DBURL=postgresql://...htm_development` + `RAILS_ENV=test` → connects to `htm_test`
+- **`HTM::Database.default_config` now respects `RAILS_ENV`** - Uses `ActiveRecordConfig.load_database_config`
+- **Renamed `htm:db:test` to `htm:db:verify`** - Avoids naming collision with test database namespace
+  - `htm:db:verify` verifies database connection
+  - `RAILS_ENV=test rake htm:db:*` operates on test database
+
+## [0.0.13] - 2025-12-04
+
+### Changed
+- **MarkdownChunker now uses Baran gem** - Replaced custom ParagraphChunker with Baran's `MarkdownSplitter`
+  - Respects markdown structure (headers, code blocks, horizontal rules)
+  - Configurable `chunk_size` and `chunk_overlap` settings
+  - Returns cursor positions for each chunk
+- **Fuzzy tag search with trigram matching** - Tags now searchable with fuzzy matching via pg_trgm
+- **LongTermMemory modularization** - Refactored into separate concerns for better maintainability
+  - Search optimizations for vector, fulltext, and hybrid strategies
+
+### Fixed
+- **Configurable limits** - The following are now configurable (previously hard-coded):
+  - `max_embedding_dimension` (default: 2000)
+  - `max_tag_depth` (default: 4)
+  - Circuit breaker settings: `failure_threshold`, `reset_timeout`, `half_open_max_calls`
+  - Relevance scoring weights: `semantic_weight`, `tag_weight`, `recency_weight`, `access_weight`
+  - `relevance_recency_half_life_hours` (default: 168 = 1 week)
+
 ## [0.0.11] - 2025-12-02
 
 ### Added
@@ -260,7 +343,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Markdown file loader** - Load markdown files into long-term memory
   - `FileSource` model to track loaded files with metadata and sync status
   - `MarkdownLoader` with YAML frontmatter extraction
-  - `ParagraphChunker` for splitting content into semantic chunks
+  - `MarkdownChunker` for splitting content into semantic chunks (uses Baran gem)
   - DELTA_TIME tolerance (5 seconds) for reliable file change detection
 - **New HTM API methods** for file operations:
   - `htm.load_file(path, force: false)` - Load single markdown file
@@ -347,7 +430,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [0.0.1] - 2025-10-25
 
 ### Added
-- Initial release of HTM (Hierarchical Temporary Memory)
+- Initial release of HTM (Hierarchical Temporal Memory)
 - Two-tier memory system:
   - Working memory: Token-limited, in-memory active context
   - Long-term memory: Durable PostgreSQL/TimescaleDB storage

data/README.md

@@ -51,6 +51,12 @@
     - Cross-robot context awareness
     - Track which robot said what
 
+- **Robot Groups**
+    - Coordinate multiple robots with shared working memory
+    - Real-time synchronization via PostgreSQL LISTEN/NOTIFY
+    - Active/passive roles for instant failover
+    - Dynamic scaling (add/remove robots at runtime)
+
 - **LLM-Driven Tag Extraction**
     - Automatic hierarchical tag extraction from content
     - Tags in colon-delimited format (e.g., `database:postgresql:performance`)
@@ -68,6 +74,12 @@
     - Source file tracking with re-sync support
     - YAML frontmatter extraction as metadata
 
+- **Telemetry (OpenTelemetry)**
+    - Optional metrics collection via OpenTelemetry
+    - Zero overhead when disabled (null object pattern)
+    - Works with 50+ backends (Jaeger, Prometheus, Datadog, etc.)
+    - Tracks job latency, search performance, and cache effectiveness
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -306,6 +318,82 @@ rake htm:files:sync                     # Sync all files (reload changed)
 rake htm:files:stats                    # Show file loading statistics
 ```
 
+### Robot Groups
+
+Robot Groups enable coordination of multiple robots with shared working memory and real-time synchronization. This is useful for high-availability setups, load balancing, and collaborative AI agents.
+
+**Key Classes:**
+- `HTM::RobotGroup` - Coordinates multiple robots with active/passive roles and shared working memory
+- `HTM::WorkingMemoryChannel` - PostgreSQL LISTEN/NOTIFY pub/sub for real-time cross-process synchronization
+
+**Basic Usage:**
+
+```ruby
+require 'htm'
+
+HTM.configure do |config|
+  config.embedding_provider = :ollama
+  config.embedding_model = 'nomic-embed-text'
+end
+
+# Create a robot group with primary + standby
+group = HTM::RobotGroup.new(
+  name: 'customer-support-ha',
+  active: ['support-primary'],
+  passive: ['support-standby'],
+  max_tokens: 8000
+)
+
+# Add memories to the shared working memory
+group.remember("Customer #123 prefers email communication", originator: 'support-primary')
+
+# All robots in the group can recall shared memories
+memories = group.recall('customer', limit: 5, strategy: :fulltext)
+
+# Simulate failover when primary fails
+group.failover!  # Promotes standby to active
+
+# Dynamic scaling - add more robots at runtime
+group.add_active('support-secondary')
+group.sync_robot('support-secondary')  # Sync existing memories
+
+# Check group status
+status = group.status
+# => { name: "customer-support-ha", active: [...], passive: [...], in_sync: true, ... }
+
+# Cleanup
+group.shutdown
+```
+
+**Cross-Process Synchronization:**
+
+For multi-process deployments, use `HTM::WorkingMemoryChannel` directly:
+
+```ruby
+# In each worker process
+channel = HTM::WorkingMemoryChannel.new('my-group', HTM::Database.default_config)
+
+# Listen for memory changes from other processes
+channel.on_change do |event, node_id, origin_robot_id|
+  case event
+  when :added    # Another robot added a memory
+  when :evicted  # Another robot evicted a memory
+  when :cleared  # Another robot cleared working memory
+  end
+end
+
+channel.start_listening
+
+# Notify other processes when you add a memory
+channel.notify(:added, node_id: node.id, robot_id: my_robot_id)
+```
+
+**Demo Applications:**
+
+See `examples/robot_groups/` for complete examples:
+- `same_process.rb` - Single-process demo with multiple robots
+- `multi_process.rb` - Multi-process demo with real-time sync
+
 ### Automatic Tag Extraction
 
 HTM automatically extracts hierarchical tags from content using LLM analysis. Tags are inferred from the content itself - you never specify them manually.
@@ -373,6 +461,137 @@ rake htm:jobs:clear_all
 
 See `rake -T htm:jobs` for complete list of job management tasks.
 
+## Telemetry (OpenTelemetry)
+
+HTM includes optional OpenTelemetry-based metrics for production observability. Telemetry is **disabled by default** with zero overhead when off.
+
+### Enabling Telemetry
+
+```ruby
+HTM.configure do |config|
+  config.telemetry_enabled = true
+end
+
+# Or via environment variable
+# HTM_TELEMETRY_ENABLED=true
+```
+
+### Configuring the Destination
+
+HTM emits metrics via standard OpenTelemetry protocols. Configure your destination using environment variables:
+
+```bash
+# Export to any OTLP-compatible backend
+export OTEL_METRICS_EXPORTER="otlp"
+export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
+```
+
+### Available Metrics
+
+| Metric | Type | Attributes | Description |
+|--------|------|------------|-------------|
+| `htm.jobs` | Counter | `job`, `status` | Job execution counts (embedding, tags) |
+| `htm.embedding.latency` | Histogram | `provider`, `status` | Embedding generation time (ms) |
+| `htm.tag.latency` | Histogram | `provider`, `status` | Tag extraction time (ms) |
+| `htm.search.latency` | Histogram | `strategy` | Search operation time (ms) |
+| `htm.cache.operations` | Counter | `operation` | Cache hits/misses |
+
+### Compatible Backends
+
+HTM works with any OTLP-compatible observability platform:
+
+**Open Source:** Jaeger, Prometheus, Grafana Tempo/Mimir, SigNoz, Uptrace
+
+**Commercial:** Datadog, New Relic, Honeycomb, Splunk, Dynatrace, AWS X-Ray, Google Cloud Trace, Azure Monitor
+
+### Optional Dependencies
+
+Users who want telemetry should add these gems:
+
+```ruby
+gem 'opentelemetry-sdk'
+gem 'opentelemetry-metrics-sdk'
+gem 'opentelemetry-exporter-otlp'  # For OTLP export
+```
+
+### Design
+
+HTM uses a **null object pattern** for telemetry. When disabled or when the SDK is not installed:
+- All metric operations are no-ops
+- Zero runtime overhead
+- No errors or exceptions
+
+See [docs/telemetry.md](docs/telemetry.md) for detailed configuration and usage examples.
+
+## MCP Server (Model Context Protocol)
+
+HTM includes an MCP server that exposes memory capabilities to AI assistants like Claude Desktop, Claude Code, and AIA. This enables any MCP-compatible client to store, recall, and manage memories through a standardized protocol.
+
+### Available Tools
+
+| Tool | Description |
+|------|-------------|
+| `SetRobotTool` | Set the robot identity for the session (call first) |
+| `GetRobotTool` | Get current robot information |
+| `GetWorkingMemoryTool` | Get working memory contents for session restore |
+| `RememberTool` | Store information with optional tags and metadata |
+| `RecallTool` | Search memories using vector, fulltext, or hybrid strategies |
+| `ForgetTool` | Soft-delete a memory (recoverable) |
+| `RestoreTool` | Restore a soft-deleted memory |
+| `ListTagsTool` | List tags with optional prefix filtering |
+| `SearchTagsTool` | Fuzzy tag search with typo tolerance |
+| `FindByTopicTool` | Find nodes by topic with optional fuzzy matching |
+| `StatsTool` | Get memory usage statistics |
+
+### Available Resources
+
+| URI | Description |
+|-----|-------------|
+| `htm://statistics` | Memory statistics as JSON |
+| `htm://tags/hierarchy` | Tag hierarchy as text tree |
+| `htm://memories/recent` | Last 20 memories |
+
+### Client Configuration
+
+**Claude Desktop** (`~/.config/claude/claude_desktop_config.json`):
+```json
+{
+  "mcpServers": {
+    "htm-memory": {
+      "command": "htm_mcp.rb",
+      "env": {
+        "HTM_DBURL": "postgresql://user@localhost:5432/htm_development"
+      }
+    }
+  }
+}
+```
+
+**Claude Code** (`~/.claude/claude_code_config.json`):
+```json
+{
+  "mcpServers": {
+    "htm-memory": {
+      "command": "htm_mcp.rb",
+      "env": {
+        "HTM_DBURL": "postgresql://user@localhost:5432/htm_development"
+      }
+    }
+  }
+}
+```
+
+**AIA** (`~/.config/aia/config.yml`):
+```yaml
+mcp_servers:
+  htm-memory:
+    command: htm_mcp.rb
+    env:
+      HTM_DBURL: postgresql://user@localhost:5432/htm_development
+```
+
+See [docs/guides/mcp-server.md](docs/guides/mcp-server.md) for detailed configuration, usage examples, and troubleshooting.
+
 ## Configuration
 
 HTM uses dependency injection for LLM access, allowing you to configure embedding generation, tag extraction, logging, and token counting.
@@ -1160,6 +1379,17 @@ export HTM_LOG_LEVEL="INFO"   # Default
 
 This is used by the default logger when `HTM.configure` is called without a custom logger.
 
+#### HTM_TELEMETRY_ENABLED
+
+Enable OpenTelemetry metrics collection:
+
+```bash
+export HTM_TELEMETRY_ENABLED="true"   # Enable telemetry
+export HTM_TELEMETRY_ENABLED="false"  # Disable (default)
+```
+
+When enabled, HTM emits metrics to configured OpenTelemetry collectors. See [Telemetry](#telemetry-opentelemetry) for details.
+
 ### Quick Setup Examples
 
 #### Local Development (PostgreSQL)
@@ -1239,50 +1469,86 @@ rake test
 ruby examples/basic_usage.rb
 ```
 
-### Database Management
-
-HTM provides comprehensive rake tasks under the `htm:db` namespace for managing the database:
-
-```bash
-# List all database tasks
-rake -T htm:db
-
-# Set up database (create schema + run migrations)
-rake htm:db:setup
-
-# Run pending migrations only
-rake htm:db:migrate
-
-# Show migration status (which migrations are applied)
-rake htm:db:status
-
-# Show database info (size, tables, extensions, row counts)
-rake htm:db:info
-
-# Test database connection
-rake htm:db:test
-
-# Open PostgreSQL console (interactive psql session)
-rake htm:db:console
-
-# Seed database with sample data
-rake htm:db:seed
-
-# Drop all HTM tables (WARNING: destructive!)
-rake htm:db:drop
-
-# Drop and recreate database (WARNING: destructive!)
-rake htm:db:reset
-```
-
 **Important**: Make sure `direnv allow` has been run once in the project directory to load database environment variables from `.envrc`. Alternatively, you can manually export the environment variables:
 
 ```bash
-# Manually export HTM_DBURL
 export HTM_DBURL="postgresql://user:password@host:port/dbname?sslmode=require"
 ```
 
-**Common Workflows**:
+### HTM Rake Tasks Reference
+
+HTM provides comprehensive rake tasks for database management, documentation, file loading, job processing, and tag management. Use `rake -T htm` to see all available tasks.
+
+#### Database Tasks (`htm:db:*`)
+
+| Task | Description |
+|------|-------------|
+| `rake htm:db:setup` | Set up HTM database schema and run migrations (sets up from scratch) |
+| `rake htm:db:create` | Create database if it doesn't exist (respects RAILS_ENV) |
+| `rake htm:db:migrate` | Run pending database migrations |
+| `rake htm:db:status` | Show migration status |
+| `rake htm:db:verify` | Verify database connection (respects RAILS_ENV) |
+| `rake htm:db:info` | Show database info (size, tables, extensions) |
+| `rake htm:db:stats` | Show record counts for all HTM tables |
+| `rake htm:db:console` | Open PostgreSQL console (respects RAILS_ENV) |
+| `rake htm:db:seed` | Seed database with sample data |
+| `rake htm:db:schema:dump` | Dump current schema to db/schema.sql |
+| `rake htm:db:schema:load` | Load schema from db/schema.sql |
+| `rake htm:db:rebuild:embeddings` | Rebuild embeddings for all nodes |
+| `rake htm:db:rebuild:propositions` | Rebuild propositions for all non-proposition nodes |
+| `rake htm:db:tags:cleanup` | Soft delete orphaned tags and stale node_tags entries |
+| `rake htm:db:drop` | Drop all HTM tables (WARNING: destructive!) |
+| `rake htm:db:reset` | Drop and recreate database (WARNING: destructive!) |
+
+#### Documentation Tasks (`htm:doc:*`)
+
+| Task | Description |
+|------|-------------|
+| `rake htm:doc:all` | Generate DB docs, YARD API docs, build site, and serve |
+| `rake htm:doc:yard` | Build YARD API documentation (markdown format for MkDocs) |
+| `rake htm:doc:db` | Generate/update database documentation in docs/database/ |
+| `rake htm:doc:build` | Build documentation site with MkDocs |
+| `rake htm:doc:serve` | Serve documentation site locally with MkDocs |
+| `rake htm:doc:server[port]` | Start YARD documentation server (live reload) |
+| `rake htm:doc:fix_anchors` | Fix YARD anchor links for MkDocs compatibility |
+| `rake htm:doc:stats` | Show documentation coverage statistics |
+| `rake htm:doc:clean` | Clean generated documentation |
+
+#### File Loading Tasks (`htm:files:*`)
+
+| Task | Description |
+|------|-------------|
+| `rake htm:files:load[path]` | Load a markdown file into long-term memory |
+| `rake htm:files:load_dir[path,pattern]` | Load all markdown files from a directory |
+| `rake htm:files:list` | List all loaded file sources |
+| `rake htm:files:info[path]` | Show details for a loaded file |
+| `rake htm:files:sync` | Sync all loaded files (reload changed files) |
+| `rake htm:files:unload[path]` | Unload a file from memory |
+| `rake htm:files:stats` | Show file loading statistics |
+
+#### Job Processing Tasks (`htm:jobs:*`)
+
+| Task | Description |
+|------|-------------|
+| `rake htm:jobs:stats` | Show statistics for nodes and async job processing |
+| `rake htm:jobs:process_embeddings` | Process pending embedding jobs for nodes without embeddings |
+| `rake htm:jobs:process_tags` | Process pending tag extraction jobs for nodes without tags |
+| `rake htm:jobs:process_all` | Process all pending jobs (embeddings and tags) |
+| `rake htm:jobs:reprocess_embeddings` | Reprocess embeddings for all nodes (force regeneration) |
+| `rake htm:jobs:failed` | Show nodes that failed async processing |
+| `rake htm:jobs:clear_all` | Clear all embeddings and tags (for testing/development) |
+
+#### Tag Management Tasks (`htm:tags:*`)
+
+| Task | Description |
+|------|-------------|
+| `rake htm:tags:tree[prefix]` | Display tags as a hierarchical tree (text format) |
+| `rake htm:tags:mermaid[prefix]` | Export tags as Mermaid flowchart to tags.md |
+| `rake htm:tags:svg[prefix]` | Export tags as SVG visualization to tags.svg |
+| `rake htm:tags:export[prefix]` | Export tags in all formats (tags.txt, tags.md, tags.svg) |
+| `rake htm:tags:rebuild` | Rebuild all tags from node content |
+
+### Common Workflows
 
 ```bash
 # Initial setup (first time)
@@ -1295,57 +1561,35 @@ rake htm:db:migrate      # Run pending migrations
 # Check database state
 rake htm:db:status       # See migration status
 rake htm:db:info         # See database details
+rake htm:db:stats        # See record counts
 
-# Reset database (development only!)
-rake htm:db:reset        # Drops and recreates everything
-
-# Open database console for debugging
-rake htm:db:console      # Opens psql
-```
-
-### Async Job Management
-
-HTM provides rake tasks under the `htm:jobs` namespace for managing async embedding and tag extraction jobs:
-
-```bash
-# List all job management tasks
-rake -T htm:jobs
-
-# Show statistics for async processing
+# Monitor async processing
 rake htm:jobs:stats
+rake htm:jobs:failed     # Show stuck jobs
 
-# Process pending jobs
-rake htm:jobs:process_embeddings    # Process nodes without embeddings
-rake htm:jobs:process_tags          # Process nodes without tags
-rake htm:jobs:process_all           # Process both
-
-# Reprocess all nodes (force regeneration)
-rake htm:jobs:reprocess_embeddings  # WARNING: Prompts for confirmation
-
-# Debugging and maintenance
-rake htm:jobs:failed                # Show stuck jobs (>1 hour old)
-rake htm:jobs:clear_all            # Clear all embeddings/tags (testing only)
-```
+# Process pending jobs manually
+rake htm:jobs:process_all
 
-**Common Workflows**:
+# Generate documentation
+rake htm:doc:all         # Full documentation build
+rake htm:doc:serve       # Local preview
 
-```bash
-# Monitor async processing
-rake htm:jobs:stats
+# Load files into memory
+rake 'htm:files:load[docs/guide.md]'
+rake 'htm:files:load_dir[docs/,**/*.md]'
 
-# Manually process pending jobs (if async job runner is not running)
-rake htm:jobs:process_all
+# View tag hierarchy
+rake htm:tags:tree
+rake 'htm:tags:svg[database]'  # Export filtered tags
 
-# Debug stuck jobs
-rake htm:jobs:failed
-rake htm:jobs:process_embeddings    # Retry failed embeddings
+# Reset database (development only!)
+rake htm:db:reset        # Drops and recreates everything
 
-# Development/testing: force regeneration
-rake htm:jobs:reprocess_embeddings  # Regenerate all embeddings
-rake htm:jobs:clear_all             # Clear everything and start fresh
+# Open database console for debugging
+rake htm:db:console      # Opens psql
 ```
 
-See the [Async Job Processing](#async-job-processing) section for more details.
+See the [Async Job Processing](#async-job-processing) section for more details on background jobs.
 
 ## Testing
 
@@ -1409,6 +1653,31 @@ This separation allows you to provide any LLM implementation while HTM handles r
 - [x] Phase 8: Production-ready gem
 
 
+## Why "Robots" Instead of "Agents"?
+
+> "What's in a name? That which we call a rose
+> By any other name would smell as sweet."
+> — Shakespeare, *Romeo and Juliet*
+
+Shakespeare argues names are arbitrary. In software, we respectfully disagree—names shape expectations and understanding.
+
+HTM uses "robots" rather than the fashionable "agents" for several reasons:
+
+- **Semantic clarity**: "Agent" is overloaded—user agents, software agents, real estate agents, secret agents. "Robot" is specific to automated workers.
+
+- **Honest about capabilities**: "Agent" implies autonomy and genuine decision-making. These systems follow instructions—they don't have agency. "Robot" acknowledges what they actually are: tools.
+
+- **Avoiding the hype cycle**: "AI Agent" and "Agentic AI" became buzzwords in 2023-2024, often meaning nothing more than "LLM with a prompt." We prefer terminology that will age well.
+
+- **Many "agent" frameworks are prompt wrappers**: Look under the hood of popular agent libraries and you'll often find a single prompt in a for-loop. Calling that an "agent" sets false expectations.
+
+- **Rich heritage**: "Robot" comes from Karel Čapek's 1920 play *R.U.R.* (from Czech *robota*, meaning labor). Isaac Asimov gave us the Three Laws. There's decades of thoughtful writing about robot ethics and behavior. "Agent" has no comparable tradition.
+
+- **Personality**: Robots are endearing—R2-D2, Wall-E, Bender. They have cultural weight that "agent" lacks.
+
+Honest terminology leads to clearer thinking. These are robots: tireless workers with perfect memory, executing instructions on our behalf. That's exactly what HTM helps them do better.
+
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/madbomber/htm.