amalfa 1.0.1 → 1.0.2

package/docs/AGENT-METADATA-PATTERNS.md

@@ -0,0 +1,1021 @@
+# Agent-First Metadata: Auto-Augmentation Patterns
+
+**Date:** 2026-01-06  
+**Status:** Design Document  
+**Context:** Agent autonomy with human audit model
+
+---
+
+## Executive Summary
+
+Traditional knowledge management systems require humans to manually tag, link, and organize documents. This creates bottlenecks and doesn't scale. This document describes an **agent-first metadata system** where agents automatically augment documents with tags, links, and semantic relationships, while humans audit changes through git review rather than approving every decision.
+
+**Key Innovation:** Latent space tagging - tags and clusters emerge from vector embeddings rather than predefined taxonomies.
+
+---
+
+## Core Principles
+
+### 1. Agent Autonomy by Default
+
+**The inversion:**
+
+**❌ Old model:**
+```
+Agent: "Should I add this tag?"
+Human: *reviews, approves*
+Agent: "Should I link this document?"
+Human: *reviews, approves*
+```
+→ Human bottleneck on every decision
+
+**✅ New model:**
+```
+Agent: *adds tags, links, metadata automatically*
+Human: *reviews git diff occasionally*
+Human: *removes/modifies anything wrong*
+Daemon: *picks up changes, re-indexes*
+```
+→ Human audits, doesn't approve
+
+### 2. Git as Source of Truth
+
+**All agent augmentations are git commits:**
+- Atomic (each augmentation is one commit)
+- Auditable (see exactly what agent changed)
+- Reversible (revert any change)
+- Non-destructive (originals preserved in history)
+
+**Pattern:**
+```bash
+# Agent auto-augments
+[Amalfa: auto-tagged debrief-auth-refactor]
+
+# Human reviews
+$ git diff
+
+# If wrong, just edit
+$ vim debrief-auth-refactor.md
+$ git commit -m "Remove incorrect tag"
+
+# Daemon syncs automatically
+[Amalfa: detected manual edit, re-indexed]
+```
+
+### 3. Optimistic Metadata
+
+**Metadata is optimistic by default, corrected on audit:**
+
+- Agent adds metadata immediately (no approval needed)
+- Human reviews periodically (weekly, not per-decision)
+- Human corrects errors (removes/modifies as needed)
+- System adapts to corrections (learns from edits)
+
+**Scaling characteristic:** Human effort is O(log N), not O(N).
+
+### 4. Latent Space Organization
+
+**Tags and clusters emerge from vector space, not predefined taxonomy:**
+
+- No predefined tag list to maintain
+- Clusters form naturally from content similarity
+- Labels generated from cluster analysis
+- Adapts as knowledge base grows
+
+**Example:**
+```yaml
+tags:
+  latent:
+    - auth-state-patterns (0.91)  # cluster assignment
+    - ui-reactivity (0.78)        # secondary cluster
+```
+
+---
+
+## The Auto-Augmentation Workflow
+
+### On Document Save
+
+**Triggered automatically (pre-commit hook or daemon watch):**
+
+```bash
+[file saved: debrief-auth-refactor.md]
+
+$ amalfa auto-augment debrief-auth-refactor.md
+
+Processing...
+  [1/6] Entity extraction (0.3s)
+  [2/6] Auto-linking (0.5s)
+  [3/6] Clustering (0.8s)
+  [4/6] Similarity search (0.2s)
+  [5/6] Tag extraction (0.4s)
+  [6/6] Metadata generation (0.1s)
+  
+✓ Done. Modified front matter (15 lines changed).
+```
+
+**Agent augments document, commits, done.**
+
+### What Gets Added
+
+**Front matter is augmented with:**
+
+```yaml
+---
+type: debrief
+brief_id: brief-auth-refactor
+date: 2026-01-05
+author: claude-3.5
+
+# Auto-generated by Amalfa (edit freely, changes will sync)
+tags:
+  explicit: [alpine.js, state-management, localStorage]
+  latent:
+    - auth-state-patterns (0.91)
+    - ui-reactivity (0.78)
+  topics:
+    - authentication (0.45)
+    - state-patterns (0.38)
+
+links:
+  - playbook-alpine-patterns (uses-pattern, 0.89)
+  - playbook-state-persistence (extends, 0.81)
+
+suggested_reading:
+  - debrief-session-management (0.87)
+  - playbook-reactive-patterns (0.82)
+
+semantic_neighbors:
+  - debrief-session-management (0.87)
+  - debrief-login-flow (0.83)
+
+vector_id: vec_a7f3d2e
+embedding_model: all-MiniLM-L6-v2
+last_indexed: 2026-01-05T14:45:00Z
+---
+```
+
+**Body is augmented with wiki links:**
+
+```markdown
+## What Worked
+- [[playbook-alpine-patterns|Alpine's x-data pattern]] eliminated state tracking
+- Token refresh using [[debrief-token-refresh|$watch]] is reactive
+
+## Lessons Learned
+- Alpine for UI state, [[playbook-state-persistence|localStorage]] for persistence
+```
+
+---
+
+## The Pattern Library
+
+### Pattern 1: Latent Space Tagging
+
+**Purpose:** Organize documents without predefined taxonomy
+
+**How it works:**
+
+```python
+# Cluster all documents in embedding space
+docs = load_all_documents()
+embeddings = [doc.vector for doc in docs]
+
+# HDBSCAN or K-means clustering
+clusters = cluster_embeddings(embeddings, min_cluster_size=3)
+
+# Generate label for each cluster
+for cluster in clusters:
+    # Analyze cluster content to extract theme
+    label = generate_cluster_label(cluster.documents)
+    # Example: "auth-state-patterns"
+    
+    # Tag all docs in cluster
+    for doc in cluster.documents:
+        distance = doc.distance_to_centroid(cluster)
+        confidence = 1 - (distance / max_distance)
+        doc.add_tag(f"latent:{label}", confidence)
+```
+
+**Result in front matter:**
+
+```yaml
+tags:
+  latent:
+    - auth-state-patterns (0.91)  # strong cluster membership
+    - ui-reactivity (0.78)        # secondary cluster
+    - browser-persistence (0.65)  # weak membership
+```
+
+**Enables queries:**
+
+```bash
+# Find all docs in cluster
+$ amalfa search --cluster auth-state-patterns
+
+# Find docs near cluster boundary (potentially mis-clustered)
+$ amalfa search --cluster auth-state-patterns --confidence "<0.7"
+```
+
+**Advantages:**
+
+- No taxonomy to maintain
+- Clusters adapt as corpus grows
+- Multi-cluster membership (docs can be in multiple clusters)
+- Confidence scores expose uncertainty
+
+**Re-clustering:**
+
+```bash
+# Periodically re-cluster (weekly?)
+$ amalfa recluster --min-docs 15
+
+Analyzing 143 documents...
+  ✓ Found 12 clusters (was 10)
+  ✓ Created new cluster: api-integration-patterns (8 docs)
+  ✓ Merged clusters: css-layout + browser-layout → browser-layout (15 docs)
+  ✓ Relabeled cluster: state-mgmt → state-patterns (better fit)
+  ✓ Updated 143 document front matters
+  
+Commit? (Y/n)
+```
+
+**Human reviews cluster changes via git diff.**
+
+### Pattern 2: Entity Extraction & Auto-Linking
+
+**Purpose:** Link documents when concepts are mentioned
+
+**How it works:**
+
+```python
+# Agent writes: "Alpine's x-data pattern works well"
+
+# Entity extraction
+entities = extract_entities("Alpine's x-data pattern works well")
+# → ["Alpine", "x-data pattern"]
+
+# Search graph for matches
+for entity in entities:
+    matches = search_graph(entity, threshold=0.85)
+    # → playbook-alpine-patterns (0.89)
+    
+    # Rewrite content with wiki link
+    text = text.replace(
+        "Alpine's x-data pattern",
+        "[[playbook-alpine-patterns|Alpine's x-data pattern]]"
+    )
+```
+
+**Result:**
+
+```markdown
+Before:
+  - Alpine's x-data pattern works well
+
+After:
+  - [[playbook-alpine-patterns|Alpine's x-data pattern]] works well
+```
+
+**Advanced: Prevent over-linking**
+
+```markdown
+# First mention: linked
+[[playbook-alpine-patterns|Alpine's x-data pattern]] works well.
+
+# Subsequent mentions: not linked (avoid clutter)
+Later we used the x-data pattern again.
+
+# Unless in different section
+## Another Section
+We also applied [[playbook-alpine-patterns|Alpine's x-data]] here.
+```
+
+**Human can prevent linking:**
+
+```markdown
+<!-- amalfa-nolink: alpine -->
+We use Alpine here but don't link to playbook.
+```
+
+### Pattern 3: Topic Modeling
+
+**Purpose:** Extract high-level themes from content
+
+**How it works:**
+
+```python
+# Run LDA or BERTopic on corpus
+topics = extract_topics(all_documents, n_topics=10)
+
+# Topics emerge from content:
+Topic 1: [authentication, token, session, login] (coherence: 0.82)
+Topic 2: [layout, css, flexbox, grid, safari] (coherence: 0.79)
+Topic 3: [performance, debounce, throttle] (coherence: 0.75)
+
+# Compute topic distribution for each doc
+doc.topic_distribution = compute_distribution(doc, topics)
+# → {topic_1: 0.45, topic_2: 0.38, topic_3: 0.17}
+```
+
+**Result in front matter:**
+
+```yaml
+topics:
+  - authentication (0.45)      # strong topic
+  - css-layout (0.38)          # secondary topic
+  - performance (0.17)         # weak topic
+```
+
+**Enables queries:**
+
+```bash
+# Find all docs about authentication
+$ amalfa search --topic authentication --min-score 0.4
+
+# Find docs bridging two topics
+$ amalfa search --topics authentication,performance --min-both 0.3
+```
+
+**Topic evolution tracking:**
+
+```bash
+# Show how topics shift over time
+$ amalfa topic-timeline authentication
+
+2025-11:  5 docs (focus: OAuth patterns)
+2025-12:  8 docs (focus: Session management)
+2026-01: 12 docs (focus: Token refresh, Alpine integration)
+```
+
+### Pattern 4: Similarity-Based Suggested Reading
+
+**Purpose:** Help agents get context quickly
+
+**How it works:**
+
+```python
+# For new document, find k-nearest neighbors
+doc = load_document("debrief-auth-refactor.md")
+neighbors = find_knn(doc.vector, k=5, exclude=doc.id)
+
+# Rank by similarity
+results = [
+    ("debrief-session-management", 0.87),
+    ("playbook-state-patterns", 0.82),
+    ("brief-auth-system", 0.79),
+    ("debrief-token-refresh", 0.76),
+    ("playbook-alpine-patterns", 0.74)
+]
+```
+
+**Result in front matter:**
+
+```yaml
+suggested_reading:
+  - debrief-session-management (similar-patterns, 0.87)
+  - playbook-state-patterns (related-approach, 0.82)
+  - brief-auth-system (architectural-context, 0.79)
+```
+
+**Agent reads these first when resuming work:**
+
+```bash
+# New session starts
+Agent: "I'm working on brief-payment-refactor"
+
+Amalfa: "Here's the context:"
+  1. debrief-auth-refactor (0.87) - similar state patterns
+  2. playbook-session-management (0.82) - persistence approach
+  3. brief-auth-system (0.79) - architectural context
+
+Agent: *reads top 3 docs*
+Agent: *starts work with full context*
+```
+
+### Pattern 5: Temporal Sequences
+
+**Purpose:** Track work evolution over time
+
+**How it works:**
+
+```python
+# Detect brief → debrief → playbook → follow-up chains
+sequence = [
+    ("brief-auth-system", "2025-11-01"),
+    ("debrief-auth-system", "2025-11-05"),
+    ("playbook-alpine-patterns", "2025-11-05"),  # updated
+    ("brief-auth-refactor", "2025-12-01"),       # references playbook
+    ("debrief-auth-refactor", "2025-12-05"),
+    ("playbook-alpine-patterns", "2025-12-05"),  # updated again
+]
+
+# Tag docs with sequence metadata
+```
+
+**Result in front matter:**
+
+```yaml
+sequence:
+  chain: auth-system-evolution
+  predecessor: debrief-auth-system
+  successor: brief-auth-tests
+  position: 3/7
+```
+
+**Enables queries:**
+
+```bash
+# Show full evolution of auth work
+$ amalfa sequence auth-system-evolution
+
+Auth System Evolution (7 docs):
+  1. brief-auth-system (2025-11-01)
+  2. debrief-auth-system (2025-11-05)
+  3. playbook-alpine-patterns (updated 2025-11-05)
+  4. brief-auth-refactor (2025-12-01) ← references playbook
+  5. debrief-auth-refactor (2025-12-05)
+  6. playbook-alpine-patterns (updated 2025-12-05)
+  7. brief-auth-tests (2026-01-03)
+```
+
+### Pattern 6: Semantic Backlinks
+
+**Purpose:** Maintain bidirectional links automatically
+
+**How it works:**
+
+```python
+# When doc A links to doc B
+if "[[playbook-alpine-patterns]]" in doc_a.content:
+    # Update doc B's front matter automatically
+    doc_b.add_backlink(doc_a.id, similarity=0.89)
+```
+
+**Result in `playbook-alpine-patterns.md`:**
+
+```yaml
+backlinks:
+  - debrief-auth-refactor (2026-01-05, 0.89)
+  - debrief-session-management (2025-12-03, 0.82)
+  - brief-payment-refactor (2025-11-15, 0.78)
+  - debrief-login-flow (2025-11-08, 0.76)
+```
+
+**Human never maintains backlinks manually.**
+
+**Broken link detection:**
+
+```bash
+# If playbook-alpine-patterns is deleted/renamed
+$ amalfa check-links
+
+⚠️  Found 4 broken links:
+  - debrief-auth-refactor.md → [[playbook-alpine-patterns]] (deleted)
+  - debrief-session-management.md → [[playbook-alpine-patterns]] (deleted)
+  
+Suggested replacements:
+  - [[playbook-state-patterns]] (0.91 similar)
+  - [[playbook-reactive-patterns]] (0.85 similar)
+  
+Apply suggestions? (Y/n/individual)
+```
+
+### Pattern 7: Confidence-Based Tag Weighting
+
+**Purpose:** Express uncertainty in metadata
+
+**All tags have confidence scores:**
+
+```yaml
+tags:
+  explicit: 
+    - alpine.js (1.0)           # human-added, certain
+    - architecture-decision (1.0)
+  
+  extracted:
+    - state-management (0.87)   # mentioned 5 times, high confidence
+    - localStorage (0.78)       # mentioned 3 times
+    - token-refresh (0.45)      # mentioned once, low confidence
+  
+  latent:
+    - auth-state-patterns (0.91)  # strong cluster membership
+    - ui-reactivity (0.78)        # secondary cluster
+    - browser-persistence (0.65)  # weak membership
+```
+
+**Query by confidence:**
+
+```bash
+# Only high-confidence tags
+$ amalfa search --tags state-management --min-confidence 0.8
+
+# Find potentially mis-tagged docs
+$ amalfa search --tags --max-confidence 0.6
+```
+
+**Learning from human edits:**
+
+```python
+# Human removes tag "token-refresh" (was confidence 0.45)
+# System learns: tags below 0.5 confidence are often incorrect
+# Adjust threshold for future auto-tagging
+new_threshold = learn_from_removal(removed_tag, confidence=0.45)
+# → new_threshold = 0.55
+```
+
+---
+
+## The Daemon's Role
+
+### Continuous File Watching
+
+```bash
+$ amalfa daemon start
+
+Daemon started. Watching:
+  - /path/to/repo/docs/**/*.md
+  - /path/to/repo/briefs/**/*.md
+  - /path/to/repo/debriefs/**/*.md
+  - /path/to/repo/playbooks/**/*.md
+
+[2026-01-06 14:45:00] File changed: debriefs/2026-01-05-auth-refactor.md
+  ✓ Re-generated embedding
+  ✓ Updated cluster assignment (moved to cluster 1)
+  ✓ Re-computed similarity neighbors
+  ✓ Updated backlinks (3 docs reference this)
+  ✓ Committed changes
+
+[2026-01-06 14:46:30] File deleted: playbooks/old-pattern.md
+  ✓ Removed from graph database
+  ✓ Found 5 broken links
+  ✓ Suggested replacements
+  ✓ Updated front matter in referring docs
+  ✓ Committed changes
+
+[2026-01-06 14:48:15] Manual edit detected: debriefs/2026-01-05-auth-refactor.md
+  (User removed tag: token-refresh)
+  ✓ Re-indexed without removed tag
+  ✓ Updated confidence threshold (0.45 → 0.55)
+  ✓ No commit needed (human already committed)
+```
+
+### Git Integration
+
+**Daemon creates atomic commits:**
+
+```bash
+$ git log --oneline --grep="Amalfa:"
+
+a7f3d2e Amalfa: auto-tagged debrief-auth-refactor (added 4 tags, 3 links)
+8b2e4f1 Amalfa: re-clustered corpus (15 new docs, 12 clusters)
+c3d5a9f Amalfa: updated backlinks for playbook-alpine-patterns (2 new references)
+d4e6b2c Amalfa: detected broken links, suggested replacements (5 links fixed)
+```
+
+**Each commit is a unit of work that can be reviewed/reverted.**
+
+---
+
+## Human Audit Workflow
+
+### Weekly Review
+
+```bash
+# See what agent did this week
+$ git log --since="1 week ago" --grep="Amalfa:" --oneline
+
+a7f3d2e Amalfa: auto-tagged debrief-auth-refactor
+8b2e4f1 Amalfa: re-clustered corpus
+c3d5a9f Amalfa: updated backlinks
+d4e6b2c Amalfa: fixed broken links
+
+# Review specific commit
+$ git show a7f3d2e
+
+diff --git a/debriefs/2026-01-05-auth-refactor.md
++tags:
++  explicit: [alpine.js, state-management, localStorage, token-refresh]
++  latent:
++    - auth-state-patterns (0.91)
+
+# Looks good, move on
+```
+
+### Correction Workflow
+
+**If something is wrong:**
+
+```bash
+# Edit the document directly
+$ vim debriefs/2026-01-05-auth-refactor.md
+
+# Remove incorrect tag
+tags:
+  explicit: [alpine.js, state-management, localStorage]  # removed token-refresh
+  
+# Commit change
+$ git commit -m "Remove incorrect tag from auth-refactor debrief"
+
+# Daemon picks up change automatically
+[Amalfa daemon: detected manual edit]
+  ✓ Re-indexed without token-refresh tag
+  ✓ Updated graph query results
+  ✓ Learned: tags mentioning "refresh" once = low confidence
+```
+
+**No special commands needed.** Just edit markdown, commit, done.
+
+### Batch Corrections
+
+**If agent made systematic error:**
+
+```bash
+# Find all docs with questionable tag
+$ amalfa search --tags token-refresh --confidence "<0.6"
+
+Found 7 documents with low-confidence "token-refresh" tag:
+  - debrief-auth-refactor.md (0.45)
+  - debrief-session-management.md (0.52)
+  - debrief-login-flow.md (0.48)
+  ...
+
+# Remove tag from all
+$ amalfa untag --tag token-refresh --max-confidence 0.6
+
+Removing "token-refresh" from 7 documents...
+  ✓ Updated 7 files
+  ✓ Re-indexed 7 documents
+  ✓ Committed changes
+
+$ git show HEAD
+Amalfa: batch removed low-confidence tag "token-refresh" (7 docs)
+```
+
+---
+
+## Example: Full Lifecycle
+
+### 1. Agent Writes Document
+
+```markdown
+# Debrief: Auth Refactor
+
+## What Worked
+- Alpine's x-data pattern eliminated manual state tracking
+- Token refresh using $watch is reactive
+
+## What Failed
+- Storing token in Alpine state broke on reload
+
+## Lessons Learned
+- Alpine for UI state, localStorage for persistence
+```
+
+**No metadata yet. Just content.**
+
+### 2. Agent Saves → Auto-Augmentation
+
+```bash
+[pre-commit hook or daemon watch triggers]
+
+$ amalfa auto-augment debrief-auth-refactor.md
+
+Processing...
+  ✓ Entity extraction (found: Alpine, x-data, localStorage)
+  ✓ Auto-linking (3 links inserted)
+  ✓ Clustering (assigned to: auth-state-patterns, 0.91)
+  ✓ Similarity search (found 5 neighbors)
+  ✓ Tag extraction (6 tags)
+  ✓ Metadata generation (embedding, vector_id)
+  
+Commit? (Y/n) y
+
+[Amalfa: auto-tagged debrief-auth-refactor]
+```
+
+### 3. Result Document
+
+```markdown
+---
+type: debrief
+brief_id: brief-auth-refactor
+date: 2026-01-05
+author: claude-3.5
+
+# Auto-generated by Amalfa (edit freely)
+tags:
+  explicit: [alpine.js, state-management, localStorage, token-refresh]
+  latent:
+    - auth-state-patterns (0.91)
+    - ui-reactivity (0.78)
+  topics:
+    - authentication (0.45)
+    - state-patterns (0.38)
+
+links:
+  - playbook-alpine-patterns (uses-pattern, 0.89)
+  - playbook-state-persistence (extends, 0.81)
+  - debrief-token-refresh (similar-problem, 0.76)
+
+suggested_reading:
+  - debrief-session-management (0.87)
+  - playbook-reactive-patterns (0.82)
+
+semantic_neighbors:
+  - debrief-session-management (0.87)
+  - debrief-login-flow (0.83)
+
+vector_id: vec_a7f3d2e
+last_indexed: 2026-01-05T14:45:00Z
+---
+
+# Debrief: Auth Refactor
+
+## What Worked
+- [[playbook-alpine-patterns|Alpine's x-data pattern]] eliminated manual state tracking
+- Token refresh using [[debrief-token-refresh|$watch]] is reactive
+
+## What Failed
+- Storing token in Alpine state broke on reload
+
+## Lessons Learned
+- Alpine for UI state, [[playbook-state-persistence|localStorage]] for persistence
+```
+
+### 4. Human Reviews (Days Later)
+
+```bash
+$ git log --since="1 week ago" --oneline --grep="Amalfa:"
+
+a7f3d2e Amalfa: auto-tagged debrief-auth-refactor
+
+$ git show a7f3d2e
+
+# Human notices: "token-refresh" tag is wrong (not the focus of this doc)
+```
+
+### 5. Human Corrects
+
+```bash
+$ vim debriefs/2026-01-05-auth-refactor.md
+
+# Remove incorrect tag
+tags:
+  explicit: [alpine.js, state-management, localStorage]  # removed token-refresh
+
+$ git commit -m "Remove irrelevant token-refresh tag"
+```
+
+### 6. Daemon Syncs
+
+```bash
+[Amalfa daemon watches git commits]
+
+Detected manual edit: debrief-auth-refactor.md
+  ✓ Re-indexed without token-refresh tag
+  ✓ Updated search results
+  ✓ Learned: increase confidence threshold (0.45 → 0.55)
+  
+No commit needed (human already committed).
+```
+
+**System adapted to correction.**
+
+---
+
+## Configuration
+
+### `.amalfa.yaml`
+
+```yaml
+# Auto-augmentation settings
+auto_augment:
+  enabled: true
+  on_save: true           # Run on every save (vs manual trigger)
+  commit_changes: true    # Auto-commit augmentations
+  
+  # What to augment
+  features:
+    entity_linking: true
+    clustering: true
+    topic_modeling: true
+    similarity_search: true
+    tag_extraction: true
+    backlinks: true
+  
+  # Thresholds
+  thresholds:
+    tag_confidence: 0.55      # Learned from human corrections
+    link_similarity: 0.85     # Minimum similarity for auto-linking
+    cluster_confidence: 0.70  # Minimum for cluster assignment
+    
+  # Re-clustering
+  reclustering:
+    auto: true
+    trigger: 15               # Re-cluster after N new docs
+    min_cluster_size: 3
+
+# Daemon settings
+daemon:
+  watch_paths:
+    - docs/**/*.md
+    - briefs/**/*.md
+    - debriefs/**/*.md
+    - playbooks/**/*.md
+  
+  git_integration:
+    auto_commit: true
+    commit_prefix: "Amalfa:"
+    
+# Human audit
+audit:
+  weekly_digest: true         # Email summary of agent changes
+  confidence_alerts: true     # Alert on low-confidence tags
+  broken_link_fix: auto       # Auto-fix broken links
+```
+
+---
+
+## Implementation Phases
+
+### Phase 1: Basic Auto-Augmentation
+
+**Scope:** Tag extraction, basic linking
+
+**Deliverables:**
+- Entity extraction from content
+- Auto-insert wiki links (high similarity)
+- Extract explicit tags from content
+- Generate embeddings
+- Commit changes to git
+
+**Result:** Agent writes document → tags + links added automatically
+
+### Phase 2: Latent Space Tagging
+
+**Scope:** Clustering, topic modeling
+
+**Deliverables:**
+- Cluster documents in embedding space
+- Generate cluster labels automatically
+- Assign latent tags with confidence scores
+- Topic modeling (LDA or BERTopic)
+- Re-clustering when corpus grows
+
+**Result:** Documents auto-organize without predefined taxonomy
+
+### Phase 3: Semantic Relationships
+
+**Scope:** Similarity search, suggested reading
+
+**Deliverables:**
+- K-nearest neighbor search
+- Suggested reading lists
+- Semantic neighbor detection
+- Temporal sequence tracking
+- Backlink maintenance
+
+**Result:** Agents get context quickly when starting new sessions
+
+### Phase 4: Learning from Corrections
+
+**Scope:** Adapt to human edits
+
+**Deliverables:**
+- Track human removals (tags, links)
+- Adjust confidence thresholds
+- Improve entity extraction
+- Learn project-specific patterns
+- Weekly digest of changes
+
+**Result:** System gets better over time based on human feedback
+
+---
+
+## Success Metrics
+
+### Agent Productivity
+
+**Before auto-augmentation:**
+- Agent writes document: 15 minutes
+- Agent manually tags: 5 minutes
+- Agent manually links: 5 minutes
+- **Total: 25 minutes**
+
+**After auto-augmentation:**
+- Agent writes document: 15 minutes
+- Auto-augmentation runs: 2 seconds
+- **Total: 15 minutes (40% faster)**
+
+### Human Audit Overhead
+
+**Target: O(log N) effort**
+
+- 10 documents: 5 minutes weekly review
+- 100 documents: 15 minutes weekly review
+- 1000 documents: 30 minutes weekly review
+
+**Actual corrections: <5% of auto-augmentations need human fixes**
+
+### Knowledge Discovery
+
+**Measure: Time to find relevant context**
+
+**Before (manual search):**
+- Query: "authentication patterns"
+- Scan titles/filenames: 10 minutes
+- Read 5-10 docs to find relevant ones: 30 minutes
+- **Total: 40 minutes**
+
+**After (semantic search):**
+- Query: "authentication patterns"
+- Amalfa returns 5 most relevant: 5 seconds
+- Agent reads top 3: 10 minutes
+- **Total: 10 minutes (75% faster)**
+
+---
+
+## Advantages Over Manual Metadata
+
+### 1. Scales Automatically
+
+**Manual:**
+- N documents = N × human_tagging_time
+- Bottleneck
+
+**Auto-augmentation:**
+- N documents = N × 2 seconds + O(log N) human audit
+- No bottleneck
+
+### 2. Consistent
+
+**Manual:**
+- Different agents tag differently
+- Tags drift over time
+- Inconsistent naming
+
+**Auto-augmentation:**
+- Same algorithm tags all docs
+- Embeddings are comparable
+- Re-clustering normalizes tags
+
+### 3. Adaptive
+
+**Manual:**
+- Static taxonomy
+- Hard to reorganize
+- Tags become obsolete
+
+**Auto-augmentation:**
+- Latent space clusters adapt
+- Re-clustering as corpus grows
+- Old docs get new tags automatically
+
+### 4. Low Friction
+
+**Manual:**
+- Agent must remember to tag
+- Separate step after writing
+- Cognitive overhead
+
+**Auto-augmentation:**
+- Happens automatically on save
+- No extra effort
+- Agent just writes
+
+---
+
+## Conclusion
+
+**Agent-first metadata with git-based auditing enables:**
+
+1. **Agent autonomy** - No approval bottleneck
+2. **Human oversight** - Audit via git diff, O(log N) effort
+3. **Automatic organization** - Latent space clusters emerge
+4. **Fast context retrieval** - Agents get up to speed quickly
+5. **System learning** - Adapts to human corrections
+
+**The paradigm shift:** Metadata is **optimistically generated, occasionally corrected** rather than **pessimistically approved upfront**.
+
+**Result:** Knowledge base that scales with minimal human intervention while maintaining quality through periodic audits.
+
+---
+
+## References
+
+- **Latent Space Clustering:** HDBSCAN, K-means, Gaussian Mixture Models
+- **Topic Modeling:** LDA (Latent Dirichlet Allocation), BERTopic
+- **Entity Extraction:** spaCy, BERT NER, GPT-4 prompting
+- **Semantic Search:** Vector embeddings, FAISS, cosine similarity
+- **Git Integration:** Pre-commit hooks, file watchers (watchdog)
+
+---
+
+**Status:** Design document  
+**Next Steps:** Implement Phase 1 (basic auto-augmentation)  
+**Feedback:** Iterate based on PolyVis migration experience
+
+---
+
+_This document describes the agent-first metadata system for Amalfa. The goal: let agents do what they're good at (pattern recognition, semantic analysis) while humans do what they're good at (judgment, correction, strategic direction)._