amalfa 1.0.2 → 1.0.4

package/docs/AGENT-METADATA-PATTERNS.md

@@ -1,1021 +0,0 @@
-# Agent-First Metadata: Auto-Augmentation Patterns
-
-**Date:** 2026-01-06  
-**Status:** Design Document  
-**Context:** Agent autonomy with human audit model
-
----
-
-## Executive Summary
-
-Traditional knowledge management systems require humans to manually tag, link, and organize documents. This creates bottlenecks and doesn't scale. This document describes an **agent-first metadata system** where agents automatically augment documents with tags, links, and semantic relationships, while humans audit changes through git review rather than approving every decision.
-
-**Key Innovation:** Latent space tagging - tags and clusters emerge from vector embeddings rather than predefined taxonomies.
-
----
-
-## Core Principles
-
-### 1. Agent Autonomy by Default
-
-**The inversion:**
-
-**❌ Old model:**
-```
-Agent: "Should I add this tag?"
-Human: *reviews, approves*
-Agent: "Should I link this document?"
-Human: *reviews, approves*
-```
-→ Human bottleneck on every decision
-
-**✅ New model:**
-```
-Agent: *adds tags, links, metadata automatically*
-Human: *reviews git diff occasionally*
-Human: *removes/modifies anything wrong*
-Daemon: *picks up changes, re-indexes*
-```
-→ Human audits, doesn't approve
-
-### 2. Git as Source of Truth
-
-**All agent augmentations are git commits:**
-- Atomic (each augmentation is one commit)
-- Auditable (see exactly what agent changed)
-- Reversible (revert any change)
-- Non-destructive (originals preserved in history)
-
-**Pattern:**
-```bash
-# Agent auto-augments
-[Amalfa: auto-tagged debrief-auth-refactor]
-
-# Human reviews
-$ git diff
-
-# If wrong, just edit
-$ vim debrief-auth-refactor.md
-$ git commit -m "Remove incorrect tag"
-
-# Daemon syncs automatically
-[Amalfa: detected manual edit, re-indexed]
-```
-
-### 3. Optimistic Metadata
-
-**Metadata is optimistic by default, corrected on audit:**
-
-- Agent adds metadata immediately (no approval needed)
-- Human reviews periodically (weekly, not per-decision)
-- Human corrects errors (removes/modifies as needed)
-- System adapts to corrections (learns from edits)
-
-**Scaling characteristic:** Human effort is O(log N), not O(N).
-
-### 4. Latent Space Organization
-
-**Tags and clusters emerge from vector space, not predefined taxonomy:**
-
-- No predefined tag list to maintain
-- Clusters form naturally from content similarity
-- Labels generated from cluster analysis
-- Adapts as knowledge base grows
-
-**Example:**
-```yaml
-tags:
-  latent:
-    - auth-state-patterns (0.91)  # cluster assignment
-    - ui-reactivity (0.78)        # secondary cluster
-```
-
----
-
-## The Auto-Augmentation Workflow
-
-### On Document Save
-
-**Triggered automatically (pre-commit hook or daemon watch):**
-
-```bash
-[file saved: debrief-auth-refactor.md]
-
-$ amalfa auto-augment debrief-auth-refactor.md
-
-Processing...
-  [1/6] Entity extraction (0.3s)
-  [2/6] Auto-linking (0.5s)
-  [3/6] Clustering (0.8s)
-  [4/6] Similarity search (0.2s)
-  [5/6] Tag extraction (0.4s)
-  [6/6] Metadata generation (0.1s)
-  
-✓ Done. Modified front matter (15 lines changed).
-```
-
-**Agent augments document, commits, done.**
-
-### What Gets Added
-
-**Front matter is augmented with:**
-
-```yaml
----
-type: debrief
-brief_id: brief-auth-refactor
-date: 2026-01-05
-author: claude-3.5
-
-# Auto-generated by Amalfa (edit freely, changes will sync)
-tags:
-  explicit: [alpine.js, state-management, localStorage]
-  latent:
-    - auth-state-patterns (0.91)
-    - ui-reactivity (0.78)
-  topics:
-    - authentication (0.45)
-    - state-patterns (0.38)
-
-links:
-  - playbook-alpine-patterns (uses-pattern, 0.89)
-  - playbook-state-persistence (extends, 0.81)
-
-suggested_reading:
-  - debrief-session-management (0.87)
-  - playbook-reactive-patterns (0.82)
-
-semantic_neighbors:
-  - debrief-session-management (0.87)
-  - debrief-login-flow (0.83)
-
-vector_id: vec_a7f3d2e
-embedding_model: all-MiniLM-L6-v2
-last_indexed: 2026-01-05T14:45:00Z
----
-```
-
-**Body is augmented with wiki links:**
-
-```markdown
-## What Worked
-- [[playbook-alpine-patterns|Alpine's x-data pattern]] eliminated state tracking
-- Token refresh using [[debrief-token-refresh|$watch]] is reactive
-
-## Lessons Learned
-- Alpine for UI state, [[playbook-state-persistence|localStorage]] for persistence
-```
-
----
-
-## The Pattern Library
-
-### Pattern 1: Latent Space Tagging
-
-**Purpose:** Organize documents without predefined taxonomy
-
-**How it works:**
-
-```python
-# Cluster all documents in embedding space
-docs = load_all_documents()
-embeddings = [doc.vector for doc in docs]
-
-# HDBSCAN or K-means clustering
-clusters = cluster_embeddings(embeddings, min_cluster_size=3)
-
-# Generate label for each cluster
-for cluster in clusters:
-    # Analyze cluster content to extract theme
-    label = generate_cluster_label(cluster.documents)
-    # Example: "auth-state-patterns"
-    
-    # Tag all docs in cluster
-    for doc in cluster.documents:
-        distance = doc.distance_to_centroid(cluster)
-        confidence = 1 - (distance / max_distance)
-        doc.add_tag(f"latent:{label}", confidence)
-```
-
-**Result in front matter:**
-
-```yaml
-tags:
-  latent:
-    - auth-state-patterns (0.91)  # strong cluster membership
-    - ui-reactivity (0.78)        # secondary cluster
-    - browser-persistence (0.65)  # weak membership
-```
-
-**Enables queries:**
-
-```bash
-# Find all docs in cluster
-$ amalfa search --cluster auth-state-patterns
-
-# Find docs near cluster boundary (potentially mis-clustered)
-$ amalfa search --cluster auth-state-patterns --confidence "<0.7"
-```
-
-**Advantages:**
-
-- No taxonomy to maintain
-- Clusters adapt as corpus grows
-- Multi-cluster membership (docs can be in multiple clusters)
-- Confidence scores expose uncertainty
-
-**Re-clustering:**
-
-```bash
-# Periodically re-cluster (weekly?)
-$ amalfa recluster --min-docs 15
-
-Analyzing 143 documents...
-  ✓ Found 12 clusters (was 10)
-  ✓ Created new cluster: api-integration-patterns (8 docs)
-  ✓ Merged clusters: css-layout + browser-layout → browser-layout (15 docs)
-  ✓ Relabeled cluster: state-mgmt → state-patterns (better fit)
-  ✓ Updated 143 document front matters
-  
-Commit? (Y/n)
-```
-
-**Human reviews cluster changes via git diff.**
-
-### Pattern 2: Entity Extraction & Auto-Linking
-
-**Purpose:** Link documents when concepts are mentioned
-
-**How it works:**
-
-```python
-# Agent writes: "Alpine's x-data pattern works well"
-
-# Entity extraction
-entities = extract_entities("Alpine's x-data pattern works well")
-# → ["Alpine", "x-data pattern"]
-
-# Search graph for matches
-for entity in entities:
-    matches = search_graph(entity, threshold=0.85)
-    # → playbook-alpine-patterns (0.89)
-    
-    # Rewrite content with wiki link
-    text = text.replace(
-        "Alpine's x-data pattern",
-        "[[playbook-alpine-patterns|Alpine's x-data pattern]]"
-    )
-```
-
-**Result:**
-
-```markdown
-Before:
-  - Alpine's x-data pattern works well
-
-After:
-  - [[playbook-alpine-patterns|Alpine's x-data pattern]] works well
-```
-
-**Advanced: Prevent over-linking**
-
-```markdown
-# First mention: linked
-[[playbook-alpine-patterns|Alpine's x-data pattern]] works well.
-
-# Subsequent mentions: not linked (avoid clutter)
-Later we used the x-data pattern again.
-
-# Unless in different section
-## Another Section
-We also applied [[playbook-alpine-patterns|Alpine's x-data]] here.
-```
-
-**Human can prevent linking:**
-
-```markdown
-<!-- amalfa-nolink: alpine -->
-We use Alpine here but don't link to playbook.
-```
-
-### Pattern 3: Topic Modeling
-
-**Purpose:** Extract high-level themes from content
-
-**How it works:**
-
-```python
-# Run LDA or BERTopic on corpus
-topics = extract_topics(all_documents, n_topics=10)
-
-# Topics emerge from content:
-Topic 1: [authentication, token, session, login] (coherence: 0.82)
-Topic 2: [layout, css, flexbox, grid, safari] (coherence: 0.79)
-Topic 3: [performance, debounce, throttle] (coherence: 0.75)
-
-# Compute topic distribution for each doc
-doc.topic_distribution = compute_distribution(doc, topics)
-# → {topic_1: 0.45, topic_2: 0.38, topic_3: 0.17}
-```
-
-**Result in front matter:**
-
-```yaml
-topics:
-  - authentication (0.45)      # strong topic
-  - css-layout (0.38)          # secondary topic
-  - performance (0.17)         # weak topic
-```
-
-**Enables queries:**
-
-```bash
-# Find all docs about authentication
-$ amalfa search --topic authentication --min-score 0.4
-
-# Find docs bridging two topics
-$ amalfa search --topics authentication,performance --min-both 0.3
-```
-
-**Topic evolution tracking:**
-
-```bash
-# Show how topics shift over time
-$ amalfa topic-timeline authentication
-
-2025-11:  5 docs (focus: OAuth patterns)
-2025-12:  8 docs (focus: Session management)
-2026-01: 12 docs (focus: Token refresh, Alpine integration)
-```
-
-### Pattern 4: Similarity-Based Suggested Reading
-
-**Purpose:** Help agents get context quickly
-
-**How it works:**
-
-```python
-# For new document, find k-nearest neighbors
-doc = load_document("debrief-auth-refactor.md")
-neighbors = find_knn(doc.vector, k=5, exclude=doc.id)
-
-# Rank by similarity
-results = [
-    ("debrief-session-management", 0.87),
-    ("playbook-state-patterns", 0.82),
-    ("brief-auth-system", 0.79),
-    ("debrief-token-refresh", 0.76),
-    ("playbook-alpine-patterns", 0.74)
-]
-```
-
-**Result in front matter:**
-
-```yaml
-suggested_reading:
-  - debrief-session-management (similar-patterns, 0.87)
-  - playbook-state-patterns (related-approach, 0.82)
-  - brief-auth-system (architectural-context, 0.79)
-```
-
-**Agent reads these first when resuming work:**
-
-```bash
-# New session starts
-Agent: "I'm working on brief-payment-refactor"
-
-Amalfa: "Here's the context:"
-  1. debrief-auth-refactor (0.87) - similar state patterns
-  2. playbook-session-management (0.82) - persistence approach
-  3. brief-auth-system (0.79) - architectural context
-
-Agent: *reads top 3 docs*
-Agent: *starts work with full context*
-```
-
-### Pattern 5: Temporal Sequences
-
-**Purpose:** Track work evolution over time
-
-**How it works:**
-
-```python
-# Detect brief → debrief → playbook → follow-up chains
-sequence = [
-    ("brief-auth-system", "2025-11-01"),
-    ("debrief-auth-system", "2025-11-05"),
-    ("playbook-alpine-patterns", "2025-11-05"),  # updated
-    ("brief-auth-refactor", "2025-12-01"),       # references playbook
-    ("debrief-auth-refactor", "2025-12-05"),
-    ("playbook-alpine-patterns", "2025-12-05"),  # updated again
-]
-
-# Tag docs with sequence metadata
-```
-
-**Result in front matter:**
-
-```yaml
-sequence:
-  chain: auth-system-evolution
-  predecessor: debrief-auth-system
-  successor: brief-auth-tests
-  position: 3/7
-```
-
-**Enables queries:**
-
-```bash
-# Show full evolution of auth work
-$ amalfa sequence auth-system-evolution
-
-Auth System Evolution (7 docs):
-  1. brief-auth-system (2025-11-01)
-  2. debrief-auth-system (2025-11-05)
-  3. playbook-alpine-patterns (updated 2025-11-05)
-  4. brief-auth-refactor (2025-12-01) ← references playbook
-  5. debrief-auth-refactor (2025-12-05)
-  6. playbook-alpine-patterns (updated 2025-12-05)
-  7. brief-auth-tests (2026-01-03)
-```
-
-### Pattern 6: Semantic Backlinks
-
-**Purpose:** Maintain bidirectional links automatically
-
-**How it works:**
-
-```python
-# When doc A links to doc B
-if "[[playbook-alpine-patterns]]" in doc_a.content:
-    # Update doc B's front matter automatically
-    doc_b.add_backlink(doc_a.id, similarity=0.89)
-```
-
-**Result in `playbook-alpine-patterns.md`:**
-
-```yaml
-backlinks:
-  - debrief-auth-refactor (2026-01-05, 0.89)
-  - debrief-session-management (2025-12-03, 0.82)
-  - brief-payment-refactor (2025-11-15, 0.78)
-  - debrief-login-flow (2025-11-08, 0.76)
-```
-
-**Human never maintains backlinks manually.**
-
-**Broken link detection:**
-
-```bash
-# If playbook-alpine-patterns is deleted/renamed
-$ amalfa check-links
-
-⚠️  Found 4 broken links:
-  - debrief-auth-refactor.md → [[playbook-alpine-patterns]] (deleted)
-  - debrief-session-management.md → [[playbook-alpine-patterns]] (deleted)
-  
-Suggested replacements:
-  - [[playbook-state-patterns]] (0.91 similar)
-  - [[playbook-reactive-patterns]] (0.85 similar)
-  
-Apply suggestions? (Y/n/individual)
-```
-
-### Pattern 7: Confidence-Based Tag Weighting
-
-**Purpose:** Express uncertainty in metadata
-
-**All tags have confidence scores:**
-
-```yaml
-tags:
-  explicit: 
-    - alpine.js (1.0)           # human-added, certain
-    - architecture-decision (1.0)
-  
-  extracted:
-    - state-management (0.87)   # mentioned 5 times, high confidence
-    - localStorage (0.78)       # mentioned 3 times
-    - token-refresh (0.45)      # mentioned once, low confidence
-  
-  latent:
-    - auth-state-patterns (0.91)  # strong cluster membership
-    - ui-reactivity (0.78)        # secondary cluster
-    - browser-persistence (0.65)  # weak membership
-```
-
-**Query by confidence:**
-
-```bash
-# Only high-confidence tags
-$ amalfa search --tags state-management --min-confidence 0.8
-
-# Find potentially mis-tagged docs
-$ amalfa search --tags --max-confidence 0.6
-```
-
-**Learning from human edits:**
-
-```python
-# Human removes tag "token-refresh" (was confidence 0.45)
-# System learns: tags below 0.5 confidence are often incorrect
-# Adjust threshold for future auto-tagging
-new_threshold = learn_from_removal(removed_tag, confidence=0.45)
-# → new_threshold = 0.55
-```
-
----
-
-## The Daemon's Role
-
-### Continuous File Watching
-
-```bash
-$ amalfa daemon start
-
-Daemon started. Watching:
-  - /path/to/repo/docs/**/*.md
-  - /path/to/repo/briefs/**/*.md
-  - /path/to/repo/debriefs/**/*.md
-  - /path/to/repo/playbooks/**/*.md
-
-[2026-01-06 14:45:00] File changed: debriefs/2026-01-05-auth-refactor.md
-  ✓ Re-generated embedding
-  ✓ Updated cluster assignment (moved to cluster 1)
-  ✓ Re-computed similarity neighbors
-  ✓ Updated backlinks (3 docs reference this)
-  ✓ Committed changes
-
-[2026-01-06 14:46:30] File deleted: playbooks/old-pattern.md
-  ✓ Removed from graph database
-  ✓ Found 5 broken links
-  ✓ Suggested replacements
-  ✓ Updated front matter in referring docs
-  ✓ Committed changes
-
-[2026-01-06 14:48:15] Manual edit detected: debriefs/2026-01-05-auth-refactor.md
-  (User removed tag: token-refresh)
-  ✓ Re-indexed without removed tag
-  ✓ Updated confidence threshold (0.45 → 0.55)
-  ✓ No commit needed (human already committed)
-```
-
-### Git Integration
-
-**Daemon creates atomic commits:**
-
-```bash
-$ git log --oneline --grep="Amalfa:"
-
-a7f3d2e Amalfa: auto-tagged debrief-auth-refactor (added 4 tags, 3 links)
-8b2e4f1 Amalfa: re-clustered corpus (15 new docs, 12 clusters)
-c3d5a9f Amalfa: updated backlinks for playbook-alpine-patterns (2 new references)
-d4e6b2c Amalfa: detected broken links, suggested replacements (5 links fixed)
-```
-
-**Each commit is a unit of work that can be reviewed/reverted.**
-
----
-
-## Human Audit Workflow
-
-### Weekly Review
-
-```bash
-# See what agent did this week
-$ git log --since="1 week ago" --grep="Amalfa:" --oneline
-
-a7f3d2e Amalfa: auto-tagged debrief-auth-refactor
-8b2e4f1 Amalfa: re-clustered corpus
-c3d5a9f Amalfa: updated backlinks
-d4e6b2c Amalfa: fixed broken links
-
-# Review specific commit
-$ git show a7f3d2e
-
-diff --git a/debriefs/2026-01-05-auth-refactor.md
-+tags:
-+  explicit: [alpine.js, state-management, localStorage, token-refresh]
-+  latent:
-+    - auth-state-patterns (0.91)
-
-# Looks good, move on
-```
-
-### Correction Workflow
-
-**If something is wrong:**
-
-```bash
-# Edit the document directly
-$ vim debriefs/2026-01-05-auth-refactor.md
-
-# Remove incorrect tag
-tags:
-  explicit: [alpine.js, state-management, localStorage]  # removed token-refresh
-  
-# Commit change
-$ git commit -m "Remove incorrect tag from auth-refactor debrief"
-
-# Daemon picks up change automatically
-[Amalfa daemon: detected manual edit]
-  ✓ Re-indexed without token-refresh tag
-  ✓ Updated graph query results
-  ✓ Learned: tags mentioning "refresh" once = low confidence
-```
-
-**No special commands needed.** Just edit markdown, commit, done.
-
-### Batch Corrections
-
-**If agent made systematic error:**
-
-```bash
-# Find all docs with questionable tag
-$ amalfa search --tags token-refresh --confidence "<0.6"
-
-Found 7 documents with low-confidence "token-refresh" tag:
-  - debrief-auth-refactor.md (0.45)
-  - debrief-session-management.md (0.52)
-  - debrief-login-flow.md (0.48)
-  ...
-
-# Remove tag from all
-$ amalfa untag --tag token-refresh --max-confidence 0.6
-
-Removing "token-refresh" from 7 documents...
-  ✓ Updated 7 files
-  ✓ Re-indexed 7 documents
-  ✓ Committed changes
-
-$ git show HEAD
-Amalfa: batch removed low-confidence tag "token-refresh" (7 docs)
-```
-
----
-
-## Example: Full Lifecycle
-
-### 1. Agent Writes Document
-
-```markdown
-# Debrief: Auth Refactor
-
-## What Worked
-- Alpine's x-data pattern eliminated manual state tracking
-- Token refresh using $watch is reactive
-
-## What Failed
-- Storing token in Alpine state broke on reload
-
-## Lessons Learned
-- Alpine for UI state, localStorage for persistence
-```
-
-**No metadata yet. Just content.**
-
-### 2. Agent Saves → Auto-Augmentation
-
-```bash
-[pre-commit hook or daemon watch triggers]
-
-$ amalfa auto-augment debrief-auth-refactor.md
-
-Processing...
-  ✓ Entity extraction (found: Alpine, x-data, localStorage)
-  ✓ Auto-linking (3 links inserted)
-  ✓ Clustering (assigned to: auth-state-patterns, 0.91)
-  ✓ Similarity search (found 5 neighbors)
-  ✓ Tag extraction (6 tags)
-  ✓ Metadata generation (embedding, vector_id)
-  
-Commit? (Y/n) y
-
-[Amalfa: auto-tagged debrief-auth-refactor]
-```
-
-### 3. Result Document
-
-```markdown
----
-type: debrief
-brief_id: brief-auth-refactor
-date: 2026-01-05
-author: claude-3.5
-
-# Auto-generated by Amalfa (edit freely)
-tags:
-  explicit: [alpine.js, state-management, localStorage, token-refresh]
-  latent:
-    - auth-state-patterns (0.91)
-    - ui-reactivity (0.78)
-  topics:
-    - authentication (0.45)
-    - state-patterns (0.38)
-
-links:
-  - playbook-alpine-patterns (uses-pattern, 0.89)
-  - playbook-state-persistence (extends, 0.81)
-  - debrief-token-refresh (similar-problem, 0.76)
-
-suggested_reading:
-  - debrief-session-management (0.87)
-  - playbook-reactive-patterns (0.82)
-
-semantic_neighbors:
-  - debrief-session-management (0.87)
-  - debrief-login-flow (0.83)
-
-vector_id: vec_a7f3d2e
-last_indexed: 2026-01-05T14:45:00Z
----
-
-# Debrief: Auth Refactor
-
-## What Worked
-- [[playbook-alpine-patterns|Alpine's x-data pattern]] eliminated manual state tracking
-- Token refresh using [[debrief-token-refresh|$watch]] is reactive
-
-## What Failed
-- Storing token in Alpine state broke on reload
-
-## Lessons Learned
-- Alpine for UI state, [[playbook-state-persistence|localStorage]] for persistence
-```
-
-### 4. Human Reviews (Days Later)
-
-```bash
-$ git log --since="1 week ago" --oneline --grep="Amalfa:"
-
-a7f3d2e Amalfa: auto-tagged debrief-auth-refactor
-
-$ git show a7f3d2e
-
-# Human notices: "token-refresh" tag is wrong (not the focus of this doc)
-```
-
-### 5. Human Corrects
-
-```bash
-$ vim debriefs/2026-01-05-auth-refactor.md
-
-# Remove incorrect tag
-tags:
-  explicit: [alpine.js, state-management, localStorage]  # removed token-refresh
-
-$ git commit -m "Remove irrelevant token-refresh tag"
-```
-
-### 6. Daemon Syncs
-
-```bash
-[Amalfa daemon watches git commits]
-
-Detected manual edit: debrief-auth-refactor.md
-  ✓ Re-indexed without token-refresh tag
-  ✓ Updated search results
-  ✓ Learned: increase confidence threshold (0.45 → 0.55)
-  
-No commit needed (human already committed).
-```
-
-**System adapted to correction.**
-
----
-
-## Configuration
-
-### `.amalfa.yaml`
-
-```yaml
-# Auto-augmentation settings
-auto_augment:
-  enabled: true
-  on_save: true           # Run on every save (vs manual trigger)
-  commit_changes: true    # Auto-commit augmentations
-  
-  # What to augment
-  features:
-    entity_linking: true
-    clustering: true
-    topic_modeling: true
-    similarity_search: true
-    tag_extraction: true
-    backlinks: true
-  
-  # Thresholds
-  thresholds:
-    tag_confidence: 0.55      # Learned from human corrections
-    link_similarity: 0.85     # Minimum similarity for auto-linking
-    cluster_confidence: 0.70  # Minimum for cluster assignment
-    
-  # Re-clustering
-  reclustering:
-    auto: true
-    trigger: 15               # Re-cluster after N new docs
-    min_cluster_size: 3
-
-# Daemon settings
-daemon:
-  watch_paths:
-    - docs/**/*.md
-    - briefs/**/*.md
-    - debriefs/**/*.md
-    - playbooks/**/*.md
-  
-  git_integration:
-    auto_commit: true
-    commit_prefix: "Amalfa:"
-    
-# Human audit
-audit:
-  weekly_digest: true         # Email summary of agent changes
-  confidence_alerts: true     # Alert on low-confidence tags
-  broken_link_fix: auto       # Auto-fix broken links
-```
-
----
-
-## Implementation Phases
-
-### Phase 1: Basic Auto-Augmentation
-
-**Scope:** Tag extraction, basic linking
-
-**Deliverables:**
-- Entity extraction from content
-- Auto-insert wiki links (high similarity)
-- Extract explicit tags from content
-- Generate embeddings
-- Commit changes to git
-
-**Result:** Agent writes document → tags + links added automatically
-
-### Phase 2: Latent Space Tagging
-
-**Scope:** Clustering, topic modeling
-
-**Deliverables:**
-- Cluster documents in embedding space
-- Generate cluster labels automatically
-- Assign latent tags with confidence scores
-- Topic modeling (LDA or BERTopic)
-- Re-clustering when corpus grows
-
-**Result:** Documents auto-organize without predefined taxonomy
-
-### Phase 3: Semantic Relationships
-
-**Scope:** Similarity search, suggested reading
-
-**Deliverables:**
-- K-nearest neighbor search
-- Suggested reading lists
-- Semantic neighbor detection
-- Temporal sequence tracking
-- Backlink maintenance
-
-**Result:** Agents get context quickly when starting new sessions
-
-### Phase 4: Learning from Corrections
-
-**Scope:** Adapt to human edits
-
-**Deliverables:**
-- Track human removals (tags, links)
-- Adjust confidence thresholds
-- Improve entity extraction
-- Learn project-specific patterns
-- Weekly digest of changes
-
-**Result:** System gets better over time based on human feedback
-
----
-
-## Success Metrics
-
-### Agent Productivity
-
-**Before auto-augmentation:**
-- Agent writes document: 15 minutes
-- Agent manually tags: 5 minutes
-- Agent manually links: 5 minutes
-- **Total: 25 minutes**
-
-**After auto-augmentation:**
-- Agent writes document: 15 minutes
-- Auto-augmentation runs: 2 seconds
-- **Total: 15 minutes (40% faster)**
-
-### Human Audit Overhead
-
-**Target: O(log N) effort**
-
-- 10 documents: 5 minutes weekly review
-- 100 documents: 15 minutes weekly review
-- 1000 documents: 30 minutes weekly review
-
-**Actual corrections: <5% of auto-augmentations need human fixes**
-
-### Knowledge Discovery
-
-**Measure: Time to find relevant context**
-
-**Before (manual search):**
-- Query: "authentication patterns"
-- Scan titles/filenames: 10 minutes
-- Read 5-10 docs to find relevant ones: 30 minutes
-- **Total: 40 minutes**
-
-**After (semantic search):**
-- Query: "authentication patterns"
-- Amalfa returns 5 most relevant: 5 seconds
-- Agent reads top 3: 10 minutes
-- **Total: 10 minutes (75% faster)**
-
----
-
-## Advantages Over Manual Metadata
-
-### 1. Scales Automatically
-
-**Manual:**
-- N documents = N × human_tagging_time
-- Bottleneck
-
-**Auto-augmentation:**
-- N documents = N × 2 seconds + O(log N) human audit
-- No bottleneck
-
-### 2. Consistent
-
-**Manual:**
-- Different agents tag differently
-- Tags drift over time
-- Inconsistent naming
-
-**Auto-augmentation:**
-- Same algorithm tags all docs
-- Embeddings are comparable
-- Re-clustering normalizes tags
-
-### 3. Adaptive
-
-**Manual:**
-- Static taxonomy
-- Hard to reorganize
-- Tags become obsolete
-
-**Auto-augmentation:**
-- Latent space clusters adapt
-- Re-clustering as corpus grows
-- Old docs get new tags automatically
-
-### 4. Low Friction
-
-**Manual:**
-- Agent must remember to tag
-- Separate step after writing
-- Cognitive overhead
-
-**Auto-augmentation:**
-- Happens automatically on save
-- No extra effort
-- Agent just writes
-
----
-
-## Conclusion
-
-**Agent-first metadata with git-based auditing enables:**
-
-1. **Agent autonomy** - No approval bottleneck
-2. **Human oversight** - Audit via git diff, O(log N) effort
-3. **Automatic organization** - Latent space clusters emerge
-4. **Fast context retrieval** - Agents get up to speed quickly
-5. **System learning** - Adapts to human corrections
-
-**The paradigm shift:** Metadata is **optimistically generated, occasionally corrected** rather than **pessimistically approved upfront**.
-
-**Result:** Knowledge base that scales with minimal human intervention while maintaining quality through periodic audits.
-
----
-
-## References
-
-- **Latent Space Clustering:** HDBSCAN, K-means, Gaussian Mixture Models
-- **Topic Modeling:** LDA (Latent Dirichlet Allocation), BERTopic
-- **Entity Extraction:** spaCy, BERT NER, GPT-4 prompting
-- **Semantic Search:** Vector embeddings, FAISS, cosine similarity
-- **Git Integration:** Pre-commit hooks, file watchers (watchdog)
-
----
-
-**Status:** Design document  
-**Next Steps:** Implement Phase 1 (basic auto-augmentation)  
-**Feedback:** Iterate based on PolyVis migration experience
-
----
-
-_This document describes the agent-first metadata system for Amalfa. The goal: let agents do what they're good at (pattern recognition, semantic analysis) while humans do what they're good at (judgment, correction, strategic direction)._