amalfa 1.0.0 → 1.0.2

package/docs/SESSION-2026-01-06-METADATA-PATTERNS.md

@@ -0,0 +1,346 @@
+# Session Summary: Agent-First Metadata & Latent Space Tagging
+
+**Date:** 2026-01-06  
+**Duration:** ~1 hour  
+**Topic:** Agent autonomy in metadata generation
+
+---
+
+## Key Insights
+
+### 1. The Inversion: Agent Does, Human Audits
+
+**Traditional model:**
+- Human approves every tag, link, decision
+- Bottleneck: O(N) human effort
+
+**New model:**
+- Agent auto-augments everything
+- Human audits via git diff (O(log N) effort)
+- Human corrects errors when found
+- System learns from corrections
+
+**Result:** Scales without human bottleneck.
+
+### 2. Latent Space Tagging
+
+**Innovation:** Tags emerge from vector clustering, not predefined taxonomy.
+
+**How it works:**
+```python
+# Cluster documents in embedding space
+clusters = cluster_embeddings(all_docs, min_size=3)
+
+# Generate labels from cluster content
+for cluster in clusters:
+    label = generate_label(cluster.documents)
+    # e.g., "auth-state-patterns"
+    
+    # Tag all docs with confidence scores
+    for doc in cluster.documents:
+        doc.add_tag(f"latent:{label}", confidence)
+```
+
+**Advantages:**
+- No taxonomy to maintain
+- Adapts as corpus grows
+- Multi-cluster membership
+- Confidence scores expose uncertainty
+
+### 3. Git as Safety Net
+
+**All agent augmentations are git commits:**
+
+```bash
+# Agent augments
+[Amalfa: auto-tagged debrief-auth-refactor]
+
+# Human reviews
+$ git diff
+
+# Human corrects if needed
+$ vim debrief.md  # remove incorrect tag
+$ git commit -m "Remove incorrect tag"
+
+# Daemon syncs automatically
+[Amalfa: re-indexed, learned from correction]
+```
+
+**Benefits:**
+- Atomic (one commit per augmentation)
+- Auditable (see what changed)
+- Reversible (revert anytime)
+- Non-destructive (history preserved)
+
+### 4. Seven Auto-Augmentation Patterns
+
+1. **Latent Space Tagging** - Cluster-based organization
+2. **Entity Extraction & Auto-Linking** - Wiki links inserted automatically
+3. **Topic Modeling** - High-level themes (LDA/BERTopic)
+4. **Similarity-Based Suggested Reading** - Context for new sessions
+5. **Temporal Sequences** - Track work evolution
+6. **Semantic Backlinks** - Bidirectional links maintained automatically
+7. **Confidence-Based Tag Weighting** - Express uncertainty, learn from removals
+
+---
+
+## Documents Created
+
+### 1. VISION-AGENT-LEARNING.md
+
+**Purpose:** Vision document for agent-generated knowledge
+
+**Key sections:**
+- The brief-debrief-playbook flywheel
+- Spec-driven vs. learning-driven development
+- Human as reader, not writer
+- Implications for Amalfa design
+- Evolution path (manual → emergent)
+
+**Size:** ~1200 lines
+
+**Main insight:** Documentation is cognition, not just artifact. Writing forces reflection.
+
+### 2. AGENT-METADATA-PATTERNS.md
+
+**Purpose:** Design document for auto-augmentation system
+
+**Key sections:**
+- Core principles (agent autonomy, git as truth, optimistic metadata)
+- Pattern library (7 patterns with implementations)
+- Daemon's role (continuous file watching)
+- Human audit workflow
+- Configuration & implementation phases
+
+**Size:** ~1000 lines
+
+**Main insight:** Metadata is optimistically generated, occasionally corrected (not pessimistically approved upfront).
+
+---
+
+## Folder Structure
+
+Amalfa already has the brief-debrief-playbook pattern from PolyVis:
+
+```
+amalfa/
+├── briefs/
+│   ├── pending/      # Active work
+│   ├── holding/      # Deferred work
+│   ├── archive/      # Completed work
+│   └── README.md     # Workflow guide
+│
+├── debriefs/         # ~100+ dated debriefs
+│   ├── 2025-01-03-embedder-refactor.md
+│   ├── 2025-01-02-...md
+│   └── README.md     # Debrief guide
+│
+├── playbooks/        # ~30+ domain playbooks
+│   ├── alpinejs-playbook.md
+│   ├── embeddings-and-fafcas-protocol-playbook.md
+│   ├── local-first-vector-db-playbook.md
+│   ├── problem-solving-playbook.md
+│   └── README.md     # Playbook index
+│
+└── docs/
+    ├── VISION-AGENT-LEARNING.md
+    ├── AGENT-METADATA-PATTERNS.md
+    └── SETUP.md
+```
+
+**Status:** Infrastructure already in place, ready for Amalfa enhancement.
+
+---
+
+## The Auto-Augmentation Workflow
+
+### Phase 1: Agent Writes
+
+```markdown
+# Debrief: Auth Refactor
+
+## What Worked
+- Alpine's x-data pattern eliminated manual state tracking
+
+## Lessons Learned
+- Alpine for UI state, localStorage for persistence
+```
+
+### Phase 2: Agent Saves → Auto-Augmentation
+
+```bash
+$ amalfa auto-augment debrief-auth-refactor.md
+
+Processing...
+  ✓ Entity extraction (found: Alpine, x-data, localStorage)
+  ✓ Auto-linking (3 links inserted)
+  ✓ Clustering (auth-state-patterns, 0.91)
+  ✓ Similarity search (5 neighbors)
+  ✓ Tag extraction (6 tags)
+  ✓ Metadata generation
+  
+[Amalfa: auto-tagged debrief-auth-refactor]
+```
+
+### Phase 3: Result
+
+```markdown
+---
+type: debrief
+brief_id: brief-auth-refactor
+
+# Auto-generated by Amalfa (edit freely)
+tags:
+  explicit: [alpine.js, state-management, localStorage]
+  latent:
+    - auth-state-patterns (0.91)
+    - ui-reactivity (0.78)
+
+links:
+  - playbook-alpine-patterns (uses-pattern, 0.89)
+  - debrief-session-management (similar-problem, 0.87)
+
+suggested_reading:
+  - debrief-session-management (0.87)
+  - playbook-state-patterns (0.82)
+---
+
+# Debrief: Auth Refactor
+
+## What Worked
+- [[playbook-alpine-patterns|Alpine's x-data pattern]] eliminated state tracking
+...
+```
+
+### Phase 4: Human Audits (Weekly)
+
+```bash
+$ git log --since="1 week ago" --grep="Amalfa:" --oneline
+
+a7f3d2e Amalfa: auto-tagged debrief-auth-refactor
+8b2e4f1 Amalfa: re-clustered corpus (15 new docs)
+
+# Review, correct if needed
+$ vim debrief-auth-refactor.md  # remove incorrect tag
+$ git commit -m "Remove incorrect tag"
+
+# Daemon learns from correction
+```
+
+---
+
+## Implementation Phases
+
+### Phase 1: Basic Auto-Augmentation (MVP)
+- Entity extraction
+- Auto-linking (high similarity)
+- Tag extraction
+- Embedding generation
+- Git commits
+
+**Result:** Agent writes → tags + links added automatically
+
+### Phase 2: Latent Space Tagging
+- Document clustering
+- Auto-generated cluster labels
+- Confidence scores
+- Topic modeling
+- Re-clustering trigger
+
+**Result:** Documents self-organize without predefined taxonomy
+
+### Phase 3: Semantic Relationships
+- K-nearest neighbors
+- Suggested reading
+- Temporal sequences
+- Backlink maintenance
+
+**Result:** Agents get context quickly on new sessions
+
+### Phase 4: Learning from Corrections
+- Track human edits
+- Adjust confidence thresholds
+- Improve extraction
+- Weekly digest
+
+**Result:** System improves over time
+
+---
+
+## Success Metrics
+
+### Agent Productivity
+
+**Before:** 25 minutes (write 15m + tag 5m + link 5m)  
+**After:** 15 minutes (write 15m + auto-augment 2s)  
+**Gain:** 40% faster
+
+### Human Audit Overhead
+
+**Target:** O(log N)
+
+- 10 docs: 5 min weekly
+- 100 docs: 15 min weekly
+- 1000 docs: 30 min weekly
+
+**Corrections needed:** <5% of augmentations
+
+### Knowledge Discovery
+
+**Before:** 40 minutes (scan titles + read 5-10 docs)  
+**After:** 10 minutes (semantic search 5s + read top 3)  
+**Gain:** 75% faster
+
+---
+
+## Key Quotes
+
+> "The main benefit from the tags and links is for the agent coming in to a new session and getting up to speed easily and quickly - so the more the agent can do the better."
+
+> "If the human finds anything they object to they can remove or modify it - the daemon will pick up the changes and the ingestion will handle it as usual."
+
+> "What you should do is provide a set of patterns the agent can use, EG a latent space tag system."
+
+---
+
+## Next Steps
+
+1. **Review existing briefs/debriefs/playbooks** - Understand current patterns
+2. **Design Amalfa schema** - Support brief-debrief-playbook types
+3. **Implement Phase 1** - Basic auto-augmentation (MVP)
+4. **Test on Amalfa corpus** - Use existing debriefs as test data
+5. **Iterate based on experience** - Learn from PolyVis migration
+
+---
+
+## Related Documents
+
+- `VISION-AGENT-LEARNING.md` - Why agent-generated knowledge works
+- `AGENT-METADATA-PATTERNS.md` - How to implement auto-augmentation
+- `SETUP.md` - NPM/GitHub setup for publishing Amalfa
+- `briefs/README.md` - Brief workflow guide
+- `debriefs/README.md` - Debrief workflow guide
+- `playbooks/README.md` - Playbook evolution guide
+
+---
+
+## Conversation Context
+
+This session built on:
+- Previous discussion: Agents using Amalfa for cross-session continuity
+- PolyVis experience: Brief-debrief-playbook pattern proved effective
+- Meta-insight: Agents spontaneously maintained docs without prompting
+
+**The leap:** If agents naturally maintain docs when given structure, and Amalfa provides semantic infrastructure, then agents can achieve true cross-session continuity with minimal human overhead.
+
+**The mechanism:** Latent space tagging + git-based auditing = agent autonomy at scale.
+
+---
+
+**Status:** Vision and design documents complete  
+**Next:** Implementation planning and MVP development  
+**Timeline:** Phases 1-2 achievable in weeks, not months
+
+---
+
+_Documentation is not an artifact of work - it's a cognitive tool. Auto-augmentation scales this tool to agent speed._