claude_swarm 1.0.2 → 1.0.5

data/docs/v2/reference/swarm_memory_technical_details.md

@@ -0,0 +1,2090 @@
+# SwarmMemory: Comprehensive Technical Reference
+
+**Version:** 2.1.1
+**Purpose:** Persistent memory system with semantic search for SwarmSDK agents
+**Location:** `lib/swarm_memory/`
+
+---
+
+## Table of Contents
+
+1. [Overview & Architecture](#overview--architecture)
+2. [Core Components](#core-components)
+3. [Adapters](#adapters)
+4. [Embeddings & Search](#embeddings--search)
+5. [Memory Tools](#memory-tools)
+6. [Optimization System](#optimization-system)
+7. [Integration with SwarmSDK](#integration-with-swarmsdk)
+8. [DSL & Configuration](#dsl--configuration)
+9. [CLI Commands](#cli-commands)
+10. [Skills System](#skills-system)
+11. [Data Flow & Lifecycle](#data-flow--lifecycle)
+12. [Testing Strategy](#testing-strategy)
+
+---
+
+## Overview & Architecture
+
+### What SwarmMemory Does
+
+SwarmMemory provides **hierarchical persistent memory with semantic search** for SwarmSDK agents. It enables agents to:
+
+- **Store knowledge** across sessions in 4 fixed categories: `concept/`, `fact/`, `skill/`, `experience/`
+- **Search semantically** using ONNX-based embeddings (Informers gem)
+- **Load skills** that dynamically swap agent tools
+- **Optimize memory** through defragmentation, duplicate detection, and quality analysis
+- **Maintain isolation** - each agent has its own memory storage
+
+### High-Level Architecture
+
+```mermaid
+graph TB
+    subgraph "SwarmSDK Layer"
+        Agent[Agent]
+        Chat[Agent::Chat]
+        ToolConfig[ToolConfigurator]
+    end
+
+    subgraph "SwarmMemory Plugin"
+        SDKPlugin[SDKPlugin]
+        MemConfig[MemoryConfig DSL]
+        Registration[Auto-Registration]
+    end
+
+    subgraph "Memory Tools"
+        MemWrite[MemoryWrite]
+        MemRead[MemoryRead]
+        MemEdit[MemoryEdit]
+        MemGlob[MemoryGlob]
+        MemGrep[MemoryGrep]
+        MemDelete[MemoryDelete]
+        MemDefrag[MemoryDefrag]
+        LoadSkill[LoadSkill]
+    end
+
+    subgraph "Core System"
+        Storage[Storage]
+        SemanticIndex[SemanticIndex]
+        Adapter[FilesystemAdapter]
+    end
+
+    subgraph "Embeddings & Search"
+        Embedder[InformersEmbedder]
+        ONNX[ONNX Model<br/>sentence-transformers]
+        TextSim[TextSimilarity]
+    end
+
+    subgraph "Optimization"
+        Analyzer[Analyzer]
+        Defrag[Defragmenter]
+    end
+
+    Agent --> Chat
+    Chat --> MemWrite
+    Chat --> MemRead
+    MemWrite --> Storage
+    MemRead --> Storage
+    Storage --> SemanticIndex
+    Storage --> Adapter
+    SemanticIndex --> Embedder
+    Embedder --> ONNX
+    SDKPlugin --> Registration
+    MemDefrag --> Defrag
+    Defrag --> Analyzer
+    LoadSkill --> ToolConfig
+
+    style Agent fill:#e1f5ff
+    style Storage fill:#fff4e1
+    style Embedder fill:#e8f5e9
+    style SDKPlugin fill:#f3e5f5
+```
+
+### Key Design Principles
+
+1. **Single Process** - No inter-process communication, all in-memory
+2. **Fiber-Safe Concurrency** - Uses `Async::Semaphore` for safe concurrent access
+3. **Plugin Architecture** - Integrates with SwarmSDK via plugin system
+4. **Adapter Pattern** - Storage backends are pluggable (filesystem, Redis, etc.)
+5. **Hybrid Search** - Combines semantic similarity + keyword matching
+6. **Tool Isolation** - Each tool is self-contained with clear responsibilities
+7. **Zero Configuration** - Works out-of-box with sensible defaults
+
+---
+
+## Core Components
+
+### 1. Entry
+
+**File:** `lib/swarm_memory/core/entry.rb`
+
+The fundamental data structure representing a single memory entry.
+
+```ruby
+Entry = Struct.new(
+  :content,      # String - Markdown content
+  :title,        # String - Brief description
+  :updated_at,   # Time - Last modification
+  :size,         # Integer - Bytes
+  :embedding,    # Array<Float> - 384-dim vector (optional)
+  :metadata,     # Hash - Structured metadata
+  keyword_init: true
+)
+```
+
+**Key Methods:**
+- `embedded?` - Checks if entry has an embedding vector
+- `has_metadata?` - Checks if entry has metadata
+
+**Metadata Structure:**
+```ruby
+{
+  "type" => "concept|fact|skill|experience",  # REQUIRED
+  "confidence" => "high|medium|low",          # Default: medium
+  "tags" => ["keyword1", "keyword2"],         # Searchable keywords
+  "related" => ["memory://path/to/entry.md"], # Cross-references
+  "domain" => "programming/ruby",             # Category
+  "source" => "user|documentation|...",       # Origin
+  "tools" => ["Read", "Edit"],                # For skills only
+  "permissions" => { ... }                    # For skills only
+}
+```
+
+---
+
+### 2. Storage
+
+**File:** `lib/swarm_memory/core/storage.rb`
+
+High-level orchestrator for all storage operations. Coordinates adapter, embedder, and semantic index.
+
+```mermaid
+graph LR
+    Storage[Storage] --> Adapter[FilesystemAdapter]
+    Storage --> SemanticIndex[SemanticIndex]
+    SemanticIndex --> Embedder[InformersEmbedder]
+
+    style Storage fill:#fff4e1
+```
+
+**Initialization:**
+```ruby
+adapter = Adapters::FilesystemAdapter.new(directory: ".swarm/memory")
+embedder = Embeddings::InformersEmbedder.new
+storage = Storage.new(adapter: adapter, embedder: embedder)
+```
+
+**Key Methods:**
+
+| Method | Purpose | Returns |
+|--------|---------|---------|
+| `write(file_path:, content:, title:, metadata:)` | Create/update entry | Entry |
+| `read(file_path:)` | Read content only | String |
+| `read_entry(file_path:)` | Read full entry with metadata | Entry |
+| `delete(file_path:)` | Remove entry | void |
+| `list(prefix: nil)` | List all entries | Array<Hash> |
+| `glob(pattern:)` | Pattern matching | Array<Hash> |
+| `grep(pattern:, options...)` | Content search | Array<Hash> |
+
+**Embedding Generation:**
+
+When writing an entry with embedder enabled, Storage:
+
+1. Builds **searchable text** (not full content):
+   - Title (most important)
+   - Tags (critical keywords)
+   - Domain (context)
+   - First paragraph (summary, configurable length)
+
+2. Generates **384-dimension vector** via Informers
+
+3. Emits **LogStream event** for observability:
+   ```ruby
+   SwarmSDK::LogStream.emit(
+     type: "memory_embedding_generated",
+     file_path: normalized_path,
+     searchable_text: text,
+     metadata_tags: tags
+   )
+   ```
+
+**Searchable Text Construction:**
+
+```ruby
+# Environment variable control:
+SWARM_MEMORY_EMBEDDING_MAX_CHARS=1200  # Default
+SWARM_MEMORY_EMBEDDING_MAX_CHARS=-1    # Unlimited
+SWARM_MEMORY_EMBEDDING_MAX_CHARS=500   # Custom
+
+# Builds: Title + Tags + Domain + First Paragraph
+parts = [
+  "Title: #{title}",
+  "Tags: #{tags.join(", ")}",
+  "Domain: #{domain}",
+  "Summary: #{first_paragraph}"  # Capped at max_chars
+]
+```
+
+---
+
+### 3. SemanticIndex
+
+**File:** `lib/swarm_memory/core/semantic_index.rb`
+
+Provides **hybrid search** combining semantic similarity and keyword matching.
+
+**Hybrid Scoring Formula:**
+```ruby
+hybrid_score = (semantic_weight * semantic_similarity) +
+               (keyword_weight * keyword_score)
+
+# Default: 50/50 (discovered via systematic evaluation)
+# Configurable via ENV:
+SWARM_MEMORY_SEMANTIC_WEIGHT=0.5  # Default
+SWARM_MEMORY_KEYWORD_WEIGHT=0.5   # Default
+```
+
+**Search Process:**
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant SemanticIndex
+    participant Embedder
+    participant Adapter
+
+    User->>SemanticIndex: search(query, top_k, threshold)
+    SemanticIndex->>SemanticIndex: extract_keywords(query)
+    SemanticIndex->>Embedder: embed(query)
+    Embedder-->>SemanticIndex: query_embedding
+    SemanticIndex->>Adapter: semantic_search(embedding, top_k*3)
+    Adapter-->>SemanticIndex: results (semantic only)
+    SemanticIndex->>SemanticIndex: calculate_hybrid_scores(results, keywords)
+    SemanticIndex->>SemanticIndex: apply_filters(filter)
+    SemanticIndex->>SemanticIndex: filter by threshold
+    SemanticIndex-->>User: top_k results
+```
+
+**Keyword Matching:**
+- Extracts keywords from query (removes stop words)
+- Matches against entry **tags** (fuzzy matching)
+- Normalizes score to 0-1 scale
+- Uses `min(keywords.size, 5)` to avoid penalizing long queries
+
+**Key Methods:**
+- `search(query:, top_k:, threshold:, filter:)` - Hybrid search
+- `find_similar(embedding:, top_k:, threshold:)` - Embedding-based only
+
+---
+
+### 4. PathNormalizer
+
+**File:** `lib/swarm_memory/core/path_normalizer.rb`
+
+Validates and normalizes memory paths for safety and consistency.
+
+**Validation Rules:**
+```ruby
+INVALID_PATTERNS = [
+  %r{\A/},        # No absolute paths
+  /\.\./,         # No parent references
+  %r{//},         # No double slashes
+  /\A\s/,         # No leading whitespace
+  /\s\z/,         # No trailing whitespace
+  /[<>:"|?*]/     # No invalid filesystem chars
+]
+```
+
+**Normalization:**
+```ruby
+# Input:  "//concepts///ruby//classes.md"
+# Output: "concepts/ruby/classes.md"
+
+# Input:  "../secrets/api-key.md"
+# Raises: ArgumentError (security violation)
+```
+
+---
+
+### 5. StorageReadTracker
+
+**File:** `lib/swarm_memory/core/storage_read_tracker.rb`
+
+**Thread-safe** global tracker enforcing **read-before-edit** pattern.
+
+```ruby
+# Each agent has independent read history
+@read_entries = {
+  agent_1: #<Set: {"concept/ruby.md", "fact/john.md"}>,
+  agent_2: #<Set: {"skill/debug.md"}>
+}
+```
+
+**Purpose:**
+- Prevents editing entries without reading them first
+- Ensures agents have current content for exact matching
+- Thread-safe with `Mutex`
+
+**API:**
+```ruby
+StorageReadTracker.register_read(:agent_1, "concept/ruby.md")
+StorageReadTracker.entry_read?(:agent_1, "concept/ruby.md") # => true
+StorageReadTracker.clear(:agent_1)  # Clear agent's history
+```
+
+---
+
+### 6. FrontmatterParser & MetadataExtractor
+
+**Files:**
+- `lib/swarm_memory/core/frontmatter_parser.rb`
+- `lib/swarm_memory/core/metadata_extractor.rb`
+
+Parse YAML frontmatter from markdown content.
+
+**Format:**
+```markdown
+---
+type: concept
+confidence: high
+tags: [ruby, oop]
+---
+
+# Ruby Classes
+
+Content here...
+```
+
+**Parsing:**
+```ruby
+parsed = FrontmatterParser.parse(content)
+# => {
+#   frontmatter: { type: "concept", confidence: "high", ... },
+#   body: "# Ruby Classes\n\nContent...",
+#   error: nil
+# }
+```
+
+**Quality Scoring:**
+```ruby
+score = MetadataExtractor.quality_score(content)
+# Scoring:
+# - Has type: 20 points
+# - Has confidence: 20 points
+# - Has tags: 15 points
+# - Has related links: 15 points
+# - Has domain: 10 points
+# - Has last_verified: 10 points
+# - Confidence=high: +10 bonus
+# Max: 100 points
+```
+
+---
+
+## Adapters
+
+### Base Adapter Interface
+
+**File:** `lib/swarm_memory/adapters/base.rb`
+
+Abstract base class defining the storage adapter interface.
+
+**Required Methods:**
+```ruby
+class MyAdapter < Base
+  def write(file_path:, content:, title:, embedding:, metadata:)
+    # Store entry
+  end
+
+  def read(file_path:)
+    # Return content string
+  end
+
+  def read_entry(file_path:)
+    # Return full Entry object
+  end
+
+  def delete(file_path:)
+    # Remove entry
+  end
+
+  def list(prefix: nil)
+    # Return array of entry metadata
+  end
+
+  def glob(pattern:)
+    # Pattern matching
+  end
+
+  def grep(pattern:, case_insensitive:, output_mode:, path:)
+    # Content search
+  end
+
+  def semantic_search(embedding:, top_k:, threshold:)
+    # Semantic similarity search
+  end
+
+  def clear
+    # Delete all entries
+  end
+
+  def total_size
+    # Total bytes
+  end
+
+  def size
+    # Entry count
+  end
+end
+```
+
+**Size Limits:**
+```ruby
+MAX_ENTRY_SIZE = 1_000_000   # 1MB per entry
+MAX_TOTAL_SIZE = 100_000_000 # 100MB total
+```
+
+---
+
+### FilesystemAdapter
+
+**File:** `lib/swarm_memory/adapters/filesystem_adapter.rb`
+
+Real filesystem storage using `.md/.yml/.emb` file triplets.
+
+**On-Disk Structure:**
+```
+.swarm/memory/
+├── concept/
+│   └── ruby/
+│       ├── classes.md       # Content (markdown)
+│       ├── classes.yml      # Metadata (title, tags, etc.)
+│       └── classes.emb      # Embedding (binary float array)
+├── fact/
+│   ├── api.md
+│   ├── api.yml
+│   └── api.emb
+├── skill/
+│   └── debugging/
+│       ├── api-errors.md
+│       ├── api-errors.yml
+│       └── api-errors.emb
+└── .lock                    # File lock for cross-process sync
+```
+
+**Key Features:**
+
+1. **Hierarchical Storage** - Paths stored as actual directories (not flattened)
+
+2. **Cross-Process Locking** - Uses `flock` for exclusive write access:
+   ```ruby
+   def with_write_lock
+     File.open(@lock_file_path, File::RDWR | File::CREAT) do |lock_file|
+       lock_file.flock(File::LOCK_EX)  # Block until acquired
+       yield
+     ensure
+       lock_file.flock(File::LOCK_UN)
+     end
+   end
+   ```
+
+3. **Fiber-Safe Concurrency** - `Async::Semaphore` for in-process:
+   ```ruby
+   @semaphore = Async::Semaphore.new(1)
+
+   @semaphore.acquire do
+     # Exclusive access
+   end
+   ```
+
+4. **In-Memory Index** - Built on boot for fast lookups:
+   ```ruby
+   @index = {
+     "concept/ruby/classes.md" => {
+       disk_path: "concept/ruby/classes",
+       title: "Ruby Classes",
+       size: 1024,
+       updated_at: Time.now
+     }
+   }
+   ```
+
+5. **Hit Tracking** - Tracks access count in `.yml` files:
+   ```yaml
+   # concept/ruby/classes.yml
+   title: Ruby Classes
+   hits: 42  # Incremented on read
+   ```
+
+6. **Stub/Redirect System** - For merged/moved entries:
+   ```markdown
+   # merged → concept/ruby/advanced-classes.md
+
+   This entry was merged into concept/ruby/advanced-classes.md
+   ```
+
+7. **Virtual Entries** - Built-in skills that don't consume storage:
+   ```ruby
+   VIRTUAL_ENTRIES = {
+     "skill/meta/deep-learning.md" => "meta/deep-learning"
+   }
+   ```
+
+**Semantic Search Implementation:**
+```ruby
+def semantic_search(embedding:, top_k:, threshold:)
+  results = []
+
+  @index.each do |logical_path, index_data|
+    emb_file = "#{index_data[:disk_path]}.emb"
+    next unless File.exist?(emb_file)
+
+    entry_embedding = File.read(emb_file).unpack("f*")
+    similarity = cosine_similarity(embedding, entry_embedding)
+    next if similarity < threshold
+
+    results << {
+      path: logical_path,
+      similarity: similarity,
+      title: index_data[:title],
+      metadata: load_metadata(logical_path)
+    }
+  end
+
+  results.sort_by { |r| -r[:similarity] }.take(top_k)
+end
+```
+
+**Glob Implementation:**
+```ruby
+# Uses native Ruby Dir.glob with hierarchical paths
+def glob(pattern:)
+  # Smart normalization:
+  # "fact/**"  → "fact/**/*.md"  (recursive)
+  # "fact/*"   → "fact/*.md"     (direct children)
+  # "fact"     → "fact.md"       (specific file)
+
+  Dir.glob(File.join(@directory, normalized_pattern))
+    .reject { |f| stub_file?(f) }
+    .map { |f| build_entry_metadata(f) }
+end
+```
+
+**Grep Implementation:**
+```ruby
+def grep(pattern:, case_insensitive:, output_mode:, path:)
+  regex = Regexp.new(pattern, case_insensitive ? Regexp::IGNORECASE : 0)
+
+  case output_mode
+  when "files_with_matches"
+    # Fast path: search .yml first (metadata), then .md
+    grep_files_with_matches(regex, path)
+  when "content"
+    # Return matching lines with line numbers
+    grep_with_content(regex, path)
+  when "count"
+    # Return match counts per file
+    grep_with_count(regex, path)
+  end
+end
+```
+
+---
+
+## Embeddings & Search
+
+### InformersEmbedder
+
+**File:** `lib/swarm_memory/embeddings/informers_embedder.rb`
+
+Fast, local embedding generation using ONNX models via the Informers gem.
+
+**Key Specifications:**
+```ruby
+DEFAULT_MODEL = "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
+EMBEDDING_DIMENSIONS = 384
+MODEL_SIZE = ~90MB (unquantized ONNX)
+```
+
+**Model Options:**
+```ruby
+# Q&A optimized (default, 512 token context)
+SWARM_MEMORY_EMBEDDING_MODEL=sentence-transformers/multi-qa-MiniLM-L6-cos-v1
+
+# General purpose (256 token context)
+SWARM_MEMORY_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
+```
+
+**Usage:**
+```ruby
+embedder = InformersEmbedder.new
+
+# Check if cached locally
+embedder.cached? # => false
+
+# Preload model (triggers download if needed)
+embedder.preload!  # Downloads ~90MB on first call
+
+# Generate embeddings
+vector = embedder.embed("This is a test sentence")
+# => [0.123, -0.456, 0.789, ...] (384 dimensions)
+
+# Batch generation
+vectors = embedder.embed_batch(["text1", "text2", "text3"])
+# => [[...], [...], [...]]
+```
+
+**Cache Location:**
+```ruby
+Informers.cache_dir
+# => "~/.cache/huggingface/hub" (default)
+
+# Custom cache:
+Informers.cache_dir = "/custom/path"
+```
+
+---
+
+### TextSimilarity
+
+**File:** `lib/swarm_memory/search/text_similarity.rb`
+
+Provides similarity calculations for text and vectors.
+
+**Jaccard Similarity (Word Overlap):**
+```ruby
+# J(A,B) = |A ∩ B| / |A ∪ B|
+sim = TextSimilarity.jaccard("ruby classes", "ruby modules")
+# => 0.33 (1 shared word out of 3 unique)
+```
+
+**Cosine Similarity (Vector Angle):**
+```ruby
+# cos(θ) = (A · B) / (||A|| * ||B||)
+vec1 = [0.1, 0.2, 0.3]
+vec2 = [0.2, 0.3, 0.4]
+sim = TextSimilarity.cosine(vec1, vec2)
+# => 0.99 (very similar)
+```
+
+**Tokenization:**
+```ruby
+# Lowercase, alphanumeric only, min 3 chars, remove stop words
+tokenize("The quick brown fox") # => #<Set: {"quick", "brown", "fox"}>
+```
+
+---
+
+### Semantic Search Architecture
+
+```mermaid
+graph TB
+    subgraph "User Query"
+        Query[/"Search: debugging api errors"/]
+    end
+
+    subgraph "SemanticIndex"
+        ExtractKW[Extract Keywords<br/>debug, api, error]
+        Embed[Generate Query Embedding]
+        Search[Adapter Search<br/>top_k * 3]
+        Hybrid[Calculate Hybrid Scores]
+        Filter[Apply Filters & Threshold]
+        TopK[Return Top K]
+    end
+
+    subgraph "Results"
+        R1["skill/debugging/api-errors.md<br/>hybrid: 0.82"]
+        R2["concept/debugging/traces.md<br/>hybrid: 0.68"]
+        R3["experience/api-bug-2024.md<br/>hybrid: 0.61"]
+    end
+
+    Query --> ExtractKW
+    ExtractKW --> Embed
+    Embed --> Search
+    Search --> Hybrid
+    Hybrid --> Filter
+    Filter --> TopK
+    TopK --> R1
+    TopK --> R2
+    TopK --> R3
+
+    style Query fill:#e1f5ff
+    style Hybrid fill:#fff4e1
+    style R1 fill:#c8e6c9
+```
+
+**Hybrid Scoring Details:**
+```ruby
+# For each result:
+semantic_score = cosine_similarity(query_emb, entry_emb)  # 0-1
+
+# Keyword matching (fuzzy on tags):
+keyword_score = matches / min(keywords.size, 5)  # 0-1
+
+# Hybrid (default 50/50):
+hybrid_score = (0.5 * semantic_score) + (0.5 * keyword_score)
+
+# Results sorted by hybrid_score descending
+```
+
+**Adaptive Thresholds:**
+```ruby
+# Short queries (< 10 words) use lower threshold
+word_count = query.split.size
+word_cutoff = ENV["SWARM_MEMORY_ADAPTIVE_WORD_CUTOFF"] || "10"
+
+threshold = if word_count < word_cutoff
+  ENV["SWARM_MEMORY_DISCOVERY_THRESHOLD_SHORT"] || "0.25"  # Lower
+else
+  ENV["SWARM_MEMORY_DISCOVERY_THRESHOLD"] || "0.35"        # Normal
+end
+```
+
+---
+
+## Memory Tools
+
+All tools inherit from `RubyLLM::Tool` and are registered with SwarmSDK's plugin system.
+
+### 1. MemoryWrite
+
+**File:** `lib/swarm_memory/tools/memory_write.rb`
+
+Creates new memory entries with structured metadata.
+
+**Required Parameters:**
+```ruby
+file_path:  "concept/ruby/classes.md"  # Where to store
+content:    "Pure markdown content"    # No frontmatter
+title:      "Ruby Classes"             # Brief description
+type:       "concept"                  # concept|fact|skill|experience
+tags:       '["ruby", "oop"]'          # JSON string of keywords
+related:    '["memory://..."]'         # JSON string of cross-refs
+domain:     "programming/ruby"         # Category
+```
+
+**Optional Parameters:**
+```ruby
+confidence: "high|medium|low"          # Default: medium
+source:     "user|documentation|..."   # Default: user
+tools:      '["Read", "Edit"]'         # For skills only
+permissions: { ... }                   # For skills only
+```
+
+**Content Length Limit:**
+```ruby
+WORD_LIMIT = 250
+
+# If exceeded, tool returns error:
+"Content exceeds 250-word limit (X words).
+Extract key entities and split into multiple memories."
+```
+
+**Validation:**
+```ruby
+# Path must start with one of 4 categories:
+- concept/
+- fact/
+- skill/
+- experience/
+
+# INVALID: documentation/, reference/, notes/, etc.
+```
+
+---
+
+### 2. MemoryRead
+
+**File:** `lib/swarm_memory/tools/memory_read.rb`
+
+Reads memory entries with metadata.
+
+**Output Format (JSON):**
+```json
+{
+  "content": "    1→# Ruby Classes\n    2→\n    3→Classes in Ruby...",
+  "metadata": {
+    "title": "Ruby Classes",
+    "type": "concept",
+    "confidence": "high",
+    "tags": ["ruby", "oop"],
+    "related": ["memory://concept/ruby/modules.md"],
+    "domain": "programming/ruby"
+  }
+}
+```
+
+**Side Effect:**
+```ruby
+# Registers read in StorageReadTracker
+Core::StorageReadTracker.register_read(agent_name, file_path)
+```
+
+---
+
+### 3. MemoryEdit
+
+**File:** `lib/swarm_memory/tools/memory_edit.rb`
+
+Exact string replacement (like Edit tool for files).
+
+**Parameters:**
+```ruby
+file_path:   "concept/ruby/classes.md"
+old_string:  "def initialize"         # Exact match required
+new_string:  "def setup"              # Replacement
+replace_all: false                    # Replace all occurrences?
+```
+
+**Enforcement:**
+```ruby
+# MUST read entry first:
+unless StorageReadTracker.entry_read?(agent_name, file_path)
+  return error("Cannot edit without reading first")
+end
+
+# old_string must exist:
+unless content.include?(old_string)
+  return error("old_string not found")
+end
+
+# old_string must be unique (unless replace_all=true):
+if content.scan(old_string).count > 1 && !replace_all
+  return error("Found N occurrences. Use replace_all or make unique.")
+end
+```
+
+---
+
+### 4. MemoryMultiEdit
+
+**File:** `lib/swarm_memory/tools/memory_multi_edit.rb`
+
+Apply multiple edits sequentially to one entry.
+
+**Input (JSON array):**
+```json
+[
+  {"old_string": "status: pending", "new_string": "status: resolved"},
+  {"old_string": "confidence: medium", "new_string": "confidence: high"}
+]
+```
+
+**Behavior:**
+- Edits applied **sequentially** (each sees previous results)
+- **All-or-nothing** - if any edit fails, NO changes saved
+- Shows which edits succeeded before failure
+
+---
+
+### 5. MemoryGlob
+
+**File:** `lib/swarm_memory/tools/memory_glob.rb`
+
+Pattern-based file discovery (like filesystem glob).
+
+**Glob Syntax:**
+```ruby
+# Single level (direct children only):
+"fact/*"        # → fact/*.md (not nested)
+
+# Recursive (all descendants):
+"fact/**"       # → fact/**/*.md (all nested)
+"fact/**/*"     # → same as above
+
+# Specific pattern:
+"skill/debug*"  # → skill/debug*.md
+"**/api-*.md"   # → any path ending with api-*.md
+```
+
+**Examples:**
+```ruby
+MemoryGlob(pattern: "concept/ruby/*")
+# Returns: concept/ruby/classes.md, concept/ruby/modules.md
+
+MemoryGlob(pattern: "skill/**")
+# Returns: All skills recursively
+
+MemoryGlob(pattern: "**/*")
+# Returns: All entries across all categories
+```
+
+**Limits:**
+```ruby
+MAX_RESULTS = 500  # Truncates with system reminder if exceeded
+```
+
+---
+
+### 6. MemoryGrep
+
+**File:** `lib/swarm_memory/tools/memory_grep.rb`
+
+Regex-based content search.
+
+**Parameters:**
+```ruby
+pattern:          "TODO|FIXME"              # Regex pattern
+path:             "concept/"                # Optional filter
+case_insensitive: true                      # Default: false
+output_mode:      "files_with_matches"      # See below
+```
+
+**Output Modes:**
+
+1. **files_with_matches** (default) - Just paths
+2. **content** - Matching lines with line numbers
+3. **count** - Match counts per file
+
+**Path Filtering:**
+```ruby
+# Directory filter:
+path: "concept/"           # All in concept/
+
+# Subdirectory filter:
+path: "fact/api"           # Only fact/api/* (not fact/api-design/)
+
+# Specific file:
+path: "skill/debug.md"     # Just that file
+```
+
+---
+
+### 7. MemoryDelete
+
+**File:** `lib/swarm_memory/tools/memory_delete.rb`
+
+Permanently removes entries.
+
+**Usage:**
+```ruby
+MemoryDelete(file_path: "experience/temp-experiment.md")
+```
+
+**Warning:** Deletion is permanent. Consider using `MemoryDefrag` to identify candidates first.
+
+---
+
+### 8. MemoryDefrag
+
+**File:** `lib/swarm_memory/tools/memory_defrag.rb`
+
+Analyzes and optimizes memory storage.
+
+**Actions:**
+
+#### Read-Only Analysis
+
+```ruby
+# 1. Overall health report
+MemoryDefrag(action: "analyze")
+
+# 2. Find duplicates (similarity > threshold)
+MemoryDefrag(action: "find_duplicates", similarity_threshold: 0.85)
+
+# 3. Find low-quality entries
+MemoryDefrag(action: "find_low_quality", confidence_filter: "low")
+
+# 4. Find old, unused entries
+MemoryDefrag(action: "find_archival_candidates", age_days: 90)
+
+# 5. Find related entries that should be linked
+MemoryDefrag(
+  action: "find_related",
+  min_similarity: 0.60,  # Lower bound
+  max_similarity: 0.85   # Upper bound (above = duplicates)
+)
+```
+
+#### Active Optimization (Modifies Memory)
+
+```ruby
+# 6. Create bidirectional links between related entries
+MemoryDefrag(action: "link_related", dry_run: true)   # Preview
+MemoryDefrag(action: "link_related", dry_run: false)  # Execute
+
+# 7. Merge duplicate entries
+MemoryDefrag(
+  action: "merge_duplicates",
+  similarity_threshold: 0.85,
+  merge_strategy: "keep_newer",  # or "keep_larger" or "combine"
+  dry_run: true
+)
+
+# 8. Delete old stub files
+MemoryDefrag(
+  action: "cleanup_stubs",
+  age_days: 30,
+  max_hits: 3,  # Max access count to delete
+  dry_run: false
+)
+
+# 9. Delete low-value entries
+MemoryDefrag(
+  action: "compact",
+  min_quality_score: 20,
+  min_age_days: 30,
+  max_hits: 0,
+  dry_run: false
+)
+
+# 10. Full optimization workflow
+MemoryDefrag(action: "full", dry_run: true)  # Preview all
+```
+
+**Safety:** All active operations default to `dry_run: true`
+
+---
+
+### 9. LoadSkill
+
+**File:** `lib/swarm_memory/tools/load_skill.rb`
+
+Loads a skill from memory and **dynamically swaps agent tools**.
+
+**Requirements:**
+```ruby
+# Skill must:
+- Be in skill/ hierarchy
+- Have type: "skill" in metadata
+- Optionally specify tools array
+- Optionally specify permissions hash
+```
+
+**Tool Swapping Process:**
+```mermaid
+sequenceDiagram
+    participant Agent
+    participant LoadSkill
+    participant Storage
+    participant Chat
+    participant ToolConfig
+
+    Agent->>LoadSkill: LoadSkill(skill/debug-api.md)
+    LoadSkill->>Storage: read_entry(skill/debug-api.md)
+    Storage-->>LoadSkill: Entry with metadata
+    LoadSkill->>LoadSkill: Validate type=skill
+    LoadSkill->>Chat: remove_mutable_tools()
+    LoadSkill->>ToolConfig: create_tool_instance(Read)
+    LoadSkill->>ToolConfig: create_tool_instance(Edit)
+    LoadSkill->>ToolConfig: create_tool_instance(Bash)
+    LoadSkill->>Chat: add_tool(Read)
+    LoadSkill->>Chat: add_tool(Edit)
+    LoadSkill->>Chat: add_tool(Bash)
+    LoadSkill->>Chat: mark_skill_loaded(path)
+    LoadSkill-->>Agent: Skill content + system reminder
+
+    Note over Agent,Chat: Agent now has skill's toolset
+```
+
+**Immutable Tools:**
+```ruby
+# These tools NEVER removed during skill loading:
+- MemoryWrite, MemoryRead, MemoryEdit, MemoryMultiEdit
+- MemoryDelete, MemoryGlob, MemoryGrep, MemoryDefrag
+- LoadSkill (self)
+```
+
+**System Reminder:**
+```
+<system-reminder>
+Your available tools have been updated.
+
+New tools loaded from skill:
+  - Read
+  - Edit
+  - Bash
+
+Your complete toolset is now:
+  - Bash
+  - Edit
+  - LoadSkill
+  - MemoryEdit
+  - MemoryRead
+  - MemoryWrite
+  - Read
+
+Only use tools from this list.
+</system-reminder>
+```
+
+---
+
+## Optimization System
+
+### Analyzer
+
+**File:** `lib/swarm_memory/optimization/analyzer.rb`
+
+Generates health metrics and statistics.
+
+**Health Score Components (0-100):**
+```ruby
+# Frontmatter coverage (30 points)
+>80% entries: +30
+>50% entries: +20
+>20% entries: +10
+
+# Tags coverage (20 points)
+>60% entries: +20
+>30% entries: +10
+
+# Links coverage (20 points)
+>40% entries: +20
+>20% entries: +10
+
+# Embedding coverage (15 points)
+>80% entries: +15
+>50% entries: +8
+
+# High confidence ratio (15 points)
+>50% entries: +15
+>25% entries: +8
+```
+
+**Health Report:**
+```
+# Memory Health Report
+
+## Overview
+- Total entries: 42
+- Total size: 156.3KB
+- Entries with frontmatter: 38 (90%)
+- Entries with embeddings: 42 (100%)
+- Entries with tags: 35 (83%)
+- Entries with related links: 28 (67%)
+- Average quality score: 72/100
+
+## By Type
+- concept: 15 (36%)
+- fact: 18 (43%)
+- skill: 6 (14%)
+- experience: 3 (7%)
+
+## By Confidence
+- high: 25 (60%)
+- medium: 13 (31%)
+- low: 4 (10%)
+
+## Health Score: 85/100
+Excellent - Memory is well-organized and high-quality
+```
+
+---
+
+### Defragmenter
+
+**File:** `lib/swarm_memory/optimization/defragmenter.rb`
+
+Performs optimization operations.
+
+**Duplicate Detection:**
+```ruby
+# Uses both text and semantic similarity:
+text_sim = TextSimilarity.jaccard(entry1.content, entry2.content)
+semantic_sim = TextSimilarity.cosine(entry1.embedding, entry2.embedding)
+similarity = [text_sim, semantic_sim].max
+
+# Pairs above threshold are candidates
+```
+
+**Merge Strategies:**
+
+1. **keep_newer** - Keep most recently updated entry
+2. **keep_larger** - Keep entry with most content
+3. **combine** - Merge content from both entries
+
+**Stub Creation:**
+```markdown
+# merged → concept/ruby/advanced-classes.md
+
+This entry was merged into concept/ruby/advanced-classes.md
+```
+
+**Related Entry Linking:**
+```ruby
+# Find pairs with 60-85% semantic similarity
+# (not duplicates, but related content)
+
+# Creates bidirectional links:
+entry1.metadata["related"] << "memory://path/to/entry2.md"
+entry2.metadata["related"] << "memory://path/to/entry1.md"
+```
+
+---
+
+## Integration with SwarmSDK
+
+### SDKPlugin
+
+**File:** `lib/swarm_memory/integration/sdk_plugin.rb`
+
+Bridges SwarmMemory with SwarmSDK's plugin system.
+
+**Plugin Interface:**
+```ruby
+class SDKPlugin < SwarmSDK::Plugin
+  def name
+    :memory
+  end
+
+  def tools
+    [:MemoryRead, :MemoryWrite, ...] # All memory tools
+  end
+
+  def create_tool(tool_name, context)
+    SwarmMemory.create_tool(tool_name, ...)
+  end
+
+  def create_storage(agent_name:, config:)
+    # Create adapter and storage
+  end
+
+  def system_prompt_contribution(agent_definition:, storage:)
+    # Return memory prompt based on mode
+  end
+
+  def storage_enabled?(agent_definition)
+    agent_definition.memory_enabled?
+  end
+end
+```
+
+**Lifecycle Hooks:**
+
+```mermaid
+sequenceDiagram
+    participant SwarmSDK
+    participant SDKPlugin
+    participant Agent
+    participant Storage
+
+    SwarmSDK->>SDKPlugin: on_agent_initialized
+    SDKPlugin->>Storage: create_storage(config)
+    SDKPlugin->>Agent: filter tools by mode
+    SDKPlugin->>Agent: register LoadSkill (special)
+    SDKPlugin->>Agent: mark tools immutable
+
+    Note over SwarmSDK,Agent: Agent ready with memory
+
+    SwarmSDK->>SDKPlugin: on_user_message(prompt)
+    SDKPlugin->>Storage: semantic search (skills)
+    SDKPlugin->>Storage: semantic search (memories)
+    SDKPlugin-->>SwarmSDK: system reminders
+
+    Note over SwarmSDK,Storage: Auto-discovery of relevant content
+```
+
+**Memory Modes:**
+
+| Mode | Tools | Use Case |
+|------|-------|----------|
+| **assistant** (default) | Read, Glob, Grep, Write, Edit | Learning & retrieval |
+| **retrieval** | Read, Glob, Grep | Read-only Q&A |
+| **researcher** | All tools | Knowledge extraction |
+
+**Mode-Based Tool Filtering:**
+```ruby
+def on_agent_initialized(agent_name:, agent:, context:)
+  mode = extract_mode(context[:agent_definition].memory)
+  allowed_tools = tools_for_mode(mode)
+
+  # Remove tools not allowed in this mode
+  all_memory_tools.each do |tool_name|
+    agent.remove_tool(tool_name) unless allowed_tools.include?(tool_name)
+  end
+
+  # Register LoadSkill (unless retrieval mode)
+  unless mode == :retrieval
+    load_skill = create_load_skill_tool(context)
+    agent.with_tool(load_skill)
+  end
+
+  # Mark mode-specific tools as immutable
+  agent.mark_tools_immutable(immutable_tools_for_mode(mode))
+end
+```
+
+**Semantic Discovery on User Messages:**
+```ruby
+def on_user_message(agent_name:, prompt:, is_first_message:)
+  # Adaptive threshold based on query length
+  word_count = prompt.split.size
+  threshold = word_count < 10 ? 0.25 : 0.35
+
+  # Search 1: Skills (type="skill")
+  skills = storage.semantic_index.search(
+    query: prompt,
+    top_k: 3,
+    threshold: threshold,
+    filter: { "type" => "skill" }
+  )
+
+  # Search 2: Memories (concept/fact/experience)
+  memories = storage.semantic_index.search(
+    query: prompt,
+    top_k: 3,
+    threshold: threshold,
+    filter: { "type" => ["concept", "fact", "experience"] }
+  )
+
+  reminders = []
+  reminders << build_skill_reminder(skills) if skills.any?
+  reminders << build_memory_reminder(memories) if memories.any?
+  reminders
+end
+```
+
+**System Reminders:**
+```
+<system-reminder>
+🎯 Found 2 skill(s) in memory that may be relevant:
+
+**Debug API Errors** (82% match)
+Path: `skill/debugging/api-errors.md`
+To use: `LoadSkill(file_path: "skill/debugging/api-errors.md")`
+
+**API Troubleshooting** (68% match)
+Path: `skill/debugging/api-troubleshooting.md`
+To use: `LoadSkill(file_path: "skill/debugging/api-troubleshooting.md")`
+
+If a skill matches your task: Load it to get instructions.
+If none match: Ignore and proceed normally.
+</system-reminder>
+
+<system-reminder>
+📚 Found 3 memory entries that may provide context:
+
+💡 **API Error Handling** (concept, 75% match)
+Path: `concept/api/error-handling.md`
+Read with: `MemoryRead(file_path: "concept/api/error-handling.md")`
+
+📋 **API Documentation** (fact, 70% match)
+Path: `fact/api/endpoints.md`
+Read with: `MemoryRead(file_path: "fact/api/endpoints.md")`
+
+🔍 **Fixed API Bug 2024** (experience, 65% match)
+Path: `experience/api-bug-2024.md`
+Read with: `MemoryRead(file_path: "experience/api-bug-2024.md")`
+
+These entries may contain relevant knowledge for your task.
+</system-reminder>
+```
+
+---
+
+### Registration
+
+**File:** `lib/swarm_memory/integration/registration.rb`
+
+Auto-registers plugin with SwarmSDK when loaded.
+
+```ruby
+# In lib/swarm_memory.rb:
+require_relative "swarm_memory/integration/registration"
+SwarmMemory::Integration::Registration.register!
+
+# Registration.register! does:
+def register!
+  return unless defined?(SwarmSDK)
+  plugin = SDKPlugin.new
+  SwarmSDK::PluginRegistry.register(plugin)
+end
+```
+
+---
+
+## DSL & Configuration
+
+### MemoryConfig
+
+**File:** `lib/swarm_memory/dsl/memory_config.rb`
+
+Configuration object for agent memory.
+
+**DSL Usage:**
+```ruby
+# In agent definition:
+memory do
+  adapter :filesystem             # Storage backend
+  directory ".swarm/agent-memory" # Where to store
+  mode :assistant                 # Tool mode
+end
+```
+
+**Options:**
+```ruby
+# Adapter options (passed to adapter constructor):
+option :namespace, "my_agent"
+option :connection_pool_size, 5
+
+# For filesystem adapter:
+directory ".swarm/memory"  # Convenience for option :directory, ...
+```
+
+**Modes:**
+- `:assistant` (default) - Read + Write + Edit
+- `:retrieval` - Read-only
+- `:researcher` - All tools
+
+---
+
+### BuilderExtension
+
+**File:** `lib/swarm_memory/dsl/builder_extension.rb`
+
+Injects `memory` DSL method into `SwarmSDK::Agent::Builder`.
+
+```ruby
+module BuilderExtension
+  def memory(&block)
+    @memory_config = SwarmMemory::DSL::MemoryConfig.new
+    @memory_config.instance_eval(&block) if block_given?
+    @memory_config
+  end
+end
+
+# Automatically injected when SwarmMemory loads:
+SwarmSDK::Agent::Builder.include(SwarmMemory::DSL::BuilderExtension)
+```
+
+**Usage in Agent Definition:**
+```ruby
+SwarmSDK.define do
+  agent :researcher do
+    memory do
+      directory ".swarm/researcher-memory"
+      mode :researcher  # All tools
+    end
+
+    tools :Read, :WebFetch, :Bash
+    # Memory tools added automatically
+  end
+end
+```
+
+---
+
+## CLI Commands
+
+**File:** `lib/swarm_memory/cli/commands.rb`
+
+Provides CLI for memory management.
+
+**Commands:**
+
+### 1. Setup Embeddings
+```bash
+swarm memory setup
+
+# With custom model:
+SWARM_MEMORY_EMBEDDING_MODEL=sentence-transformers/multi-qa-MiniLM-L6-cos-v1 \
+  swarm memory setup
+```
+
+**What it does:**
+- Downloads ONNX embedding model (~90MB)
+- Caches to `~/.cache/huggingface/hub`
+- One-time setup, then reused
+
+---
+
+### 2. Check Status
+```bash
+swarm memory status
+
+# Output:
+# SwarmMemory Embedding Status
+# ==================================================
+#
+# Status: ✓ Model cached
+# Model: sentence-transformers/multi-qa-MiniLM-L6-cos-v1
+# Dimensions: 384
+# Cache: /Users/user/.cache/huggingface/hub
+#
+# Semantic search is available.
+```
+
+---
+
+### 3. Show Model Path
+```bash
+swarm memory model-path
+# /Users/user/.cache/huggingface/hub
+```
+
+---
+
+### 4. Defragment Memory
+```bash
+swarm memory defrag .swarm/assistant-memory
+
+# Runs full analysis:
+# - Health report
+# - Find duplicates
+# - Find low-quality entries
+# - Find archival candidates
+```
+
+---
+
+### 5. Rebuild Embeddings
+```bash
+swarm memory rebuild .swarm/assistant-memory
+
+# With custom settings:
+SWARM_MEMORY_EMBEDDING_MAX_CHARS=850 \
+  swarm memory rebuild .swarm/assistant-memory
+
+# Regenerates all embeddings (useful after model change)
+```
+
+---
+
+## Skills System
+
+### What are Skills?
+
+Skills are **special memory entries** that:
+- Provide step-by-step procedures
+- Specify required tools
+- Define tool permissions
+- Can be loaded to **swap agent tools dynamically**
+
+### Skill Structure
+
+**Content (Markdown):**
+```markdown
+# Debug React Performance
+
+## Prerequisites
+- React DevTools installed
+- Profiling enabled
+
+## Steps
+
+1. Open DevTools → Profiler tab
+2. Click "Record"
+3. Perform the slow interaction
+4. Stop recording
+5. Analyze flamegraph:
+   - Look for wide bars (long renders)
+   - Check component names
+   - Identify render causes
+
+## Common Causes
+- Missing useMemo/useCallback
+- Lifting state too high
+- Expensive calculations in render
+```
+
+**Metadata (YAML sidecar):**
+```yaml
+title: Debug React Performance
+type: skill
+confidence: high
+tags:
+  - react
+  - performance
+  - debugging
+  - profiling
+related:
+  - memory://concept/react/reconciliation.md
+  - memory://skill/debugging/performance-general.md
+domain: frontend/react
+source: experimentation
+tools:
+  - Read
+  - Edit
+  - Bash
+  - Grep
+permissions:
+  Bash:
+    allowed_commands:
+      - "^npm "
+      - "^yarn "
+  Write:
+    denied_paths:
+      - "src/secrets/**"
+```
+
+### Virtual Skills
+
+Built-in skills that don't consume storage:
+- `skill/meta/deep-learning.md` - Systematic learning protocol
+
+**Deep Learning Protocol:**
+```markdown
+# Deep Learning Protocol
+
+Use this when asked to learn deeply about a topic.
+
+## Steps
+1. Define Scope
+2. Research Broadly (Overview)
+3. Research Deeply (Specifics)
+4. Build Practical Skills
+5. Self-Test Understanding
+6. Check Coverage (MemoryGrep)
+7. Identify Knowledge Gaps
+8. Fill Gaps
+9. Verify Completion
+10. Report Completion
+
+## Success Criteria
+- Have concepts (how it works)
+- Have facts (specifics)
+- Have skills (how to use)
+- Can explain clearly
+- Can apply practically
+```
+
+---
+
+## Data Flow & Lifecycle
+
+### Write Flow
+
+```mermaid
+sequenceDiagram
+    participant Agent
+    participant MemoryWrite
+    participant Storage
+    participant Embedder
+    participant Adapter
+
+    Agent->>MemoryWrite: execute(file_path, content, title, metadata)
+    MemoryWrite->>MemoryWrite: Validate word count (≤250)
+    MemoryWrite->>MemoryWrite: Parse JSON params
+    MemoryWrite->>Storage: write(file_path, content, title, metadata)
+    Storage->>Storage: PathNormalizer.normalize(file_path)
+    Storage->>Storage: build_searchable_text(content, title, metadata)
+    Storage->>Embedder: embed(searchable_text)
+    Embedder-->>Storage: embedding[384]
+    Storage->>Storage: Emit LogStream event
+    Storage->>Adapter: write(file_path, content, title, embedding, metadata)
+    Adapter->>Adapter: with_write_lock (flock)
+    Adapter->>Adapter: @semaphore.acquire (fiber-safe)
+    Adapter->>Adapter: Check size limits
+    Adapter->>Adapter: Write .md file
+    Adapter->>Adapter: Write .yml file
+    Adapter->>Adapter: Write .emb file
+    Adapter->>Adapter: Update index
+    Adapter-->>Storage: Entry
+    Storage-->>MemoryWrite: Entry
+    MemoryWrite-->>Agent: "Stored at memory://path (size)"
+```
+
+### Read Flow
+
+```mermaid
+sequenceDiagram
+    participant Agent
+    participant MemoryRead
+    participant Storage
+    participant Adapter
+    participant Tracker
+
+    Agent->>MemoryRead: execute(file_path)
+    MemoryRead->>Tracker: register_read(agent_name, file_path)
+    MemoryRead->>Storage: read_entry(file_path)
+    Storage->>Adapter: read_entry(file_path)
+    Adapter->>Adapter: Check virtual entries
+    Adapter->>Adapter: Check stubs (follow redirects)
+    Adapter->>Adapter: Read .md file
+    Adapter->>Adapter: Read .yml file
+    Adapter->>Adapter: Read .emb file (if exists)
+    Adapter->>Adapter: increment_hits(file_path)
+    Adapter-->>Storage: Entry
+    Storage-->>MemoryRead: Entry
+    MemoryRead->>MemoryRead: format_as_json(entry)
+    MemoryRead-->>Agent: JSON{content, metadata}
+```
+
+### Edit Flow
+
+```mermaid
+sequenceDiagram
+    participant Agent
+    participant MemoryEdit
+    participant Tracker
+    participant Storage
+    participant Adapter
+
+    Agent->>MemoryEdit: execute(file_path, old_string, new_string)
+    MemoryEdit->>Tracker: entry_read?(agent_name, file_path)
+    Tracker-->>MemoryEdit: true/false
+    alt Not read
+        MemoryEdit-->>Agent: Error: Must read first
+    end
+    MemoryEdit->>Storage: read(file_path)
+    Storage-->>MemoryEdit: content
+    MemoryEdit->>MemoryEdit: Check old_string exists
+    MemoryEdit->>MemoryEdit: Count occurrences
+    alt Multiple occurrences && !replace_all
+        MemoryEdit-->>Agent: Error: Not unique
+    end
+    MemoryEdit->>MemoryEdit: Perform replacement
+    MemoryEdit->>Storage: read_entry(file_path)
+    Storage-->>MemoryEdit: Entry (for title)
+    MemoryEdit->>Storage: write(file_path, new_content, title)
+    Storage-->>MemoryEdit: Entry
+    MemoryEdit-->>Agent: "Replaced N occurrence(s)"
+```
+
+### Semantic Discovery Flow
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant SwarmSDK
+    participant SDKPlugin
+    participant SemanticIndex
+    participant Embedder
+    participant Agent
+
+    User->>SwarmSDK: Send message
+    SwarmSDK->>SDKPlugin: on_user_message(prompt)
+
+    par Parallel Search
+        SDKPlugin->>SemanticIndex: search(query, filter={type:"skill"})
+        SemanticIndex->>Embedder: embed(query)
+        Embedder-->>SemanticIndex: query_embedding
+        SemanticIndex->>SemanticIndex: semantic_search + keyword_match
+        SemanticIndex-->>SDKPlugin: skills[]
+    and
+        SDKPlugin->>SemanticIndex: search(query, filter={type:["concept","fact","experience"]})
+        SemanticIndex-->>SDKPlugin: memories[]
+    end
+
+    SDKPlugin->>SDKPlugin: Emit LogStream events
+    SDKPlugin->>SDKPlugin: build_skill_reminder(skills)
+    SDKPlugin->>SDKPlugin: build_memory_reminder(memories)
+    SDKPlugin-->>SwarmSDK: [skill_reminder, memory_reminder]
+    SwarmSDK->>Agent: Inject system reminders
+    Agent->>User: Response with context
+```
+
+### LoadSkill Flow
+
+```mermaid
+sequenceDiagram
+    participant Agent
+    participant LoadSkill
+    participant Storage
+    participant Chat
+    participant ToolConfig
+
+    Agent->>LoadSkill: LoadSkill(skill/debug-api.md)
+    LoadSkill->>LoadSkill: Validate path starts with "skill/"
+    LoadSkill->>Storage: read_entry(skill/debug-api.md)
+    Storage-->>LoadSkill: Entry
+    LoadSkill->>LoadSkill: Validate type=="skill"
+    LoadSkill->>LoadSkill: Extract tools & permissions
+
+    LoadSkill->>Chat: remove_mutable_tools()
+    Note over Chat: Keeps immutable tools:<br/>Memory*, LoadSkill
+
+    loop For each required tool
+        LoadSkill->>ToolConfig: create_tool_instance(tool_name)
+        ToolConfig-->>LoadSkill: tool_instance
+        LoadSkill->>ToolConfig: wrap_with_permissions(tool, perms)
+        ToolConfig-->>LoadSkill: wrapped_tool
+        LoadSkill->>Chat: add_tool(wrapped_tool)
+    end
+
+    LoadSkill->>Chat: mark_skill_loaded(path)
+    LoadSkill->>LoadSkill: format_with_line_numbers(content)
+    LoadSkill->>LoadSkill: build_toolset_reminder(tools)
+    LoadSkill-->>Agent: Content + System Reminder
+
+    Note over Agent: Agent now has skill's tools
+```
+
+---
+
+## Testing Strategy
+
+### Test Structure
+
+```
+test/swarm_memory/
+├── adapter_registry_test.rb
+├── adapters/
+│   ├── filesystem_adapter_test.rb
+│   ├── file_locking_test.rb
+│   └── virtual_entries_test.rb
+├── core/
+│   ├── entry_test.rb
+│   ├── frontmatter_parser_test.rb
+│   └── path_normalizer_test.rb
+├── tools/
+│   ├── memory_write_test.rb
+│   ├── memory_read_test.rb
+│   ├── memory_defrag_test.rb
+│   └── load_skill_test.rb
+├── integration/
+│   ├── registration_test.rb
+│   ├── swarm_sdk_integration_test.rb
+│   └── skills_integration_test.rb
+└── optimization/
+    └── defragmenter_test.rb
+```
+
+### Test Helpers
+
+**`swarm_memory_test_helper.rb`:**
+```ruby
+def create_temp_storage
+  temp_dir = File.join(Dir.tmpdir, "test-storage-#{SecureRandom.hex}")
+  adapter = SwarmMemory::Adapters::FilesystemAdapter.new(directory: temp_dir)
+  embedder = SwarmMemory::Embeddings::InformersEmbedder.new
+  SwarmMemory::Core::Storage.new(adapter: adapter, embedder: embedder)
+end
+
+def cleanup_storage(storage)
+  FileUtils.rm_rf(storage.adapter.instance_variable_get(:@directory))
+end
+```
+
+### Key Test Patterns
+
+**1. Tool Execution Tests:**
+```ruby
+def test_write_entry_with_minimal_metadata
+  result = @tool.execute(
+    file_path: "test/simple.md",
+    content: "Simple content",
+    title: "Simple Entry",
+    type: "fact",
+    tags: ["test"],
+    related: [],
+    domain: "testing"
+  )
+
+  assert_match(%r{Stored at memory://test/simple.md}, result)
+  entry = @storage.read_entry(file_path: "test/simple.md")
+  assert_equal("Simple content", entry.content)
+end
+```
+
+**2. Concurrency Tests:**
+```ruby
+def test_concurrent_writes
+  threads = 10.times.map do |i|
+    Thread.new do
+      @adapter.write(
+        file_path: "concurrent/entry-#{i}.md",
+        content: "content #{i}",
+        title: "Entry #{i}"
+      )
+    end
+  end
+
+  threads.each(&:join)
+  assert_equal(10, @adapter.list(prefix: "concurrent").size)
+end
+```
+
+**3. Integration Tests:**
+```ruby
+def test_swarm_sdk_integration
+  swarm = SwarmSDK.define do
+    agent :tester do
+      memory do
+        directory temp_memory_dir
+        mode :assistant
+      end
+      tools :Read
+    end
+  end
+
+  # Memory tools automatically added
+  assert_includes(swarm.agent(:tester).tools.keys, :MemoryWrite)
+end
+```
+
+---
+
+## Environment Variables Reference
+
+### Embedding Configuration
+
+```bash
+# Model selection (default: multi-qa-MiniLM-L6-cos-v1)
+SWARM_MEMORY_EMBEDDING_MODEL=sentence-transformers/multi-qa-MiniLM-L6-cos-v1
+
+# Searchable text length (default: 1200, -1 = unlimited)
+SWARM_MEMORY_EMBEDDING_MAX_CHARS=1200
+```
+
+### Search Configuration
+
+```bash
+# Hybrid search weights (default: 0.5 each)
+SWARM_MEMORY_SEMANTIC_WEIGHT=0.5
+SWARM_MEMORY_KEYWORD_WEIGHT=0.5
+
+# Adaptive thresholds (default: 0.35 normal, 0.25 short)
+SWARM_MEMORY_DISCOVERY_THRESHOLD=0.35         # Normal queries
+SWARM_MEMORY_DISCOVERY_THRESHOLD_SHORT=0.25   # Short queries
+SWARM_MEMORY_ADAPTIVE_WORD_CUTOFF=10          # Cutoff for "short"
+```
+
+---
+
+## Architecture Diagrams
+
+### Component Layers
+
+```mermaid
+graph TB
+    subgraph "Layer 5: Integration"
+        SDKPlugin[SDKPlugin]
+        DSL[DSL & Config]
+        CLI[CLI Commands]
+    end
+
+    subgraph "Layer 4: Tools"
+        MemWrite[MemoryWrite]
+        MemRead[MemoryRead]
+        MemEdit[MemoryEdit]
+        MemGlob[MemoryGlob]
+        MemGrep[MemoryGrep]
+        LoadSkill[LoadSkill]
+        MemDefrag[MemoryDefrag]
+    end
+
+    subgraph "Layer 3: Optimization"
+        Analyzer[Analyzer]
+        Defrag[Defragmenter]
+    end
+
+    subgraph "Layer 2: Core"
+        Storage[Storage]
+        SemanticIndex[SemanticIndex]
+        Entry[Entry]
+        PathNorm[PathNormalizer]
+        ReadTracker[ReadTracker]
+    end
+
+    subgraph "Layer 1: Adapters & Embeddings"
+        FSAdapter[FilesystemAdapter]
+        Embedder[InformersEmbedder]
+        TextSim[TextSimilarity]
+    end
+
+    SDKPlugin --> MemWrite
+    SDKPlugin --> LoadSkill
+    MemWrite --> Storage
+    MemRead --> Storage
+    LoadSkill --> Storage
+    MemDefrag --> Defrag
+    Defrag --> Analyzer
+    Storage --> SemanticIndex
+    Storage --> FSAdapter
+    SemanticIndex --> Embedder
+    SemanticIndex --> TextSim
+
+    style SDKPlugin fill:#f3e5f5
+    style Storage fill:#fff4e1
+    style Embedder fill:#e8f5e9
+```
+
+### File Organization
+
+```
+lib/swarm_memory/
+├── swarm_memory.rb              # Entry point, auto-registration
+├── version.rb                   # VERSION constant
+├── errors.rb                    # Error classes
+├── utils.rb                     # Utility methods
+│
+├── core/                        # Core system
+│   ├── entry.rb                 # Entry struct
+│   ├── storage.rb               # Orchestrator
+│   ├── semantic_index.rb        # Hybrid search
+│   ├── frontmatter_parser.rb    # YAML parsing
+│   ├── metadata_extractor.rb    # Quality scoring
+│   ├── path_normalizer.rb       # Path validation
+│   └── storage_read_tracker.rb  # Read enforcement
+│
+├── adapters/                    # Storage backends
+│   ├── base.rb                  # Abstract interface
+│   └── filesystem_adapter.rb    # Real filesystem
+│
+├── embeddings/                  # Embedding generation
+│   ├── embedder.rb              # Abstract interface
+│   └── informers_embedder.rb    # ONNX-based
+│
+├── search/                      # Search implementations
+│   ├── semantic_search.rb       # Embedding-based
+│   ├── text_search.rb           # Glob/grep wrapper
+│   └── text_similarity.rb       # Similarity metrics
+│
+├── tools/                       # Memory tools
+│   ├── memory_write.rb
+│   ├── memory_read.rb
+│   ├── memory_edit.rb
+│   ├── memory_multi_edit.rb
+│   ├── memory_delete.rb
+│   ├── memory_glob.rb
+│   ├── memory_grep.rb
+│   ├── memory_defrag.rb
+│   └── load_skill.rb
+│
+├── optimization/                # Memory optimization
+│   ├── analyzer.rb              # Health metrics
+│   └── defragmenter.rb          # Optimization ops
+│
+├── integration/                 # SwarmSDK/CLI integration
+│   ├── sdk_plugin.rb            # Plugin implementation
+│   ├── registration.rb          # Auto-registration
+│   ├── configuration.rb         # Config wrapper
+│   └── cli_registration.rb      # CLI registration
+│
+├── dsl/                         # DSL for agent config
+│   ├── memory_config.rb         # Config object
+│   └── builder_extension.rb     # DSL injection
+│
+├── cli/                         # CLI commands
+│   └── commands.rb              # swarm memory ...
+│
+├── chat_extension.rb            # Chat#remove_tool extension
+│
+├── prompts/                     # System prompts
+│   ├── memory_assistant.md.erb
+│   ├── memory_retrieval.md.erb
+│   └── memory_researcher.md.erb
+│
+└── skills/                      # Built-in skills
+    └── meta/
+        ├── deep-learning.md
+        └── deep-learning.yml
+```
+
+---
+
+## Summary
+
+SwarmMemory provides a complete **persistent memory system with semantic search** for SwarmSDK agents:
+
+✅ **Hierarchical Storage** - 4 fixed categories: concept/, fact/, skill/, experience/
+✅ **Hybrid Search** - Combines semantic similarity + keyword matching
+✅ **Tool Swapping** - LoadSkill dynamically adapts agent capabilities
+✅ **Memory Optimization** - Automated duplicate detection, linking, and cleanup
+✅ **Plugin Architecture** - Clean integration with SwarmSDK
+✅ **Semantic Discovery** - Auto-surfaces relevant skills and memories
+✅ **Multiple Modes** - assistant/retrieval/researcher with different tool sets
+✅ **CLI Management** - Setup, status, defrag, rebuild commands
+✅ **Production Ready** - Thread-safe, fiber-safe, cross-process locking
+
+**Key Innovations:**
+- **250-word memory limit** forces focused, searchable entries
+- **Read-before-edit enforcement** prevents blind modifications
+- **Immutable tools** ensure memory tools never removed by LoadSkill
+- **Adaptive thresholds** handle short and long queries differently
+- **Virtual skills** provide meta-capabilities without storage cost
+
+---
+
+**End of Technical Reference**