amalfa 1.0.2 → 1.0.4

package/docs/REPOSITORY_CLEANUP_SUMMARY.md

@@ -1,261 +0,0 @@
-# Repository Cleanup Summary
-
-**Date:** 2026-01-05  
-**Branch:** alpine-refactor  
-**Issue:** Repository was tracking 966 files with .git size of 193 MB
-
----
-
-## Problem Identified
-
-The polyvis repository was accumulating artifacts that should not be versioned:
-
-### Artifacts Found (40+ MB total)
-
-1. **Database Files** (~20 MB)
-   - `_misc/bento_ledger.sqlite`
-   - `bento_ledger.sqlite`, `.sqlite-wal`, `.sqlite-shm`
-   - `canary-persistence.db`
-   - `test-graph-integrity.db-wal`
-   - `public/resonance.db.pre-hollow-node` (5.9 MB)
-
-2. **Database Backups** (~9 MB)
-   - `backups/db/resonance.20251214140633.db` (5.9 MB)
-   - `backups/db/benchmarks/resonance.db.pre-benchmark-20251217-184046` (8.5 MB)
-   - `backups/db/benchmarks/resonance.db.corrupted-20251217-201947`
-
-3. **Large PDFs** (~11 MB)
-   - `experiments/enlightenment/representational-engineering.pdf` (10 MB)
-   - `docs/2310.08560v2.pdf` (648 KB)
-
-4. **Built Bundles**
-   - `experiments/data-star-dashboard/dist/datastar.bundle.js` (80 KB)
-
----
-
-## Actions Taken
-
-### 1. Updated .gitignore
-
-Added comprehensive patterns to prevent future commits:
-
-```gitignore
-# Database Files (Generated Artifacts - Never Commit)
-*.db
-*.db-wal
-*.db-shm
-*.sqlite
-*.sqlite-wal
-*.sqlite-shm
-
-# Database Backups
-backups/db/
-
-# Built/Bundled JavaScript
-**/dist/*.bundle.js
-**/dist/*.min.js
-experiments/**/dist/
-
-# Test Artifacts
-test-*.db
-test-*.db-wal
-canary-*.db
-
-# Large Research Papers
-*.pdf
-!docs/architecture-diagrams/*.pdf
-```
-
-**Commit:** `0c3015e`
-
-### 2. Created Documentation
-
-**docs/COMMIT_GUIDELINES.md**
-- Comprehensive guide on what to/not to commit
-- Quick reference checklist
-- Edge cases and troubleshooting
-- Philosophy: "Repository should contain minimum necessary to build and understand"
-
-**Key principles:**
-- ✅ Source code, configs, documentation, small assets
-- ❌ Generated artifacts, large binaries, backups, secrets
-
-**Commit:** `0c3015e`
-
-### 3. Created Cleanup Script
-
-**scripts/cleanup-repo-artifacts.sh**
-- Interactive script to remove artifacts from git history
-- Safety checks (prevents running on main branch)
-- Creates backup branch before cleanup
-- Uses git-filter-repo (preferred) or filter-branch (fallback)
-- Aggressive garbage collection
-
-**Usage:**
-```bash
-./scripts/cleanup-repo-artifacts.sh
-```
-
-**Commit:** `0c3015e`
-
-### 4. Removed Artifacts from Index
-
-Removed 14 files from git tracking (not history):
-
-```bash
-git rm --cached -r backups/db/
-git rm --cached _misc/bento_ledger.sqlite
-git rm --cached bento_ledger.sqlite*
-git rm --cached canary-persistence.db
-git rm --cached test-graph-integrity.db-wal
-git rm --cached public/resonance.db.pre-hollow-node
-git rm --cached experiments/enlightenment/representational-engineering.pdf
-git rm --cached docs/2310.08560v2.pdf
-git rm --cached experiments/data-star-dashboard/dist/datastar.bundle.js
-```
-
-**Result:** 966 → 954 tracked files
-
-**Commit:** `0c3015e`
-
----
-
-## Current State
-
-### Metrics (Post-Cleanup)
-
-- **Files tracked:** 954 (down from 966)
-- **Repository size:** 193 MB (unchanged - files remain in history)
-- **Untracked files:** Database files now properly ignored
-
-### Why .git Size Unchanged?
-
-The removed files are still in git history. To fully reclaim space, you need to:
-
-1. Run the cleanup script: `./scripts/cleanup-repo-artifacts.sh`
-2. Force push to rewrite remote history
-3. Coordinate with team (they'll need to re-clone or reset)
-
-**⚠️ Important:** History rewriting is disruptive. Only do this if:
-- Working on a feature branch (✅ we're on alpine-refactor)
-- Team is coordinated
-- No open PRs depend on current history
-
----
-
-## Benefits Achieved
-
-### Immediate Benefits
-
-1. **Prevention:** `.gitignore` now prevents committing artifacts
-2. **Documentation:** Clear guidelines on what to commit
-3. **Tools:** Script ready for full history cleanup
-4. **Current commits:** New work won't add artifacts
-
-### Potential Future Benefits (After History Cleanup)
-
-1. **Faster operations:** Clone, fetch, push will be quicker
-2. **Smaller repo:** ~40-50 MB reduction estimated
-3. **Cleaner history:** Only source code and docs versioned
-
----
-
-## Next Steps (Optional)
-
-### Full History Cleanup
-
-If you want to reclaim the 40+ MB from history:
-
-```bash
-# 1. Ensure you're on alpine-refactor
-git checkout alpine-refactor
-
-# 2. Run the cleanup script
-./scripts/cleanup-repo-artifacts.sh
-
-# 3. Force push (after verification)
-git push --force origin alpine-refactor
-
-# 4. Notify team members to reset their branches
-```
-
-**Team coordination required!**
-
-### Maintenance
-
-**Going forward:**
-
-1. Review `.gitignore` patterns regularly
-2. Check commit size before pushing (see COMMIT_GUIDELINES.md)
-3. Run `git ls-files | grep -E '\.(db|sqlite|pdf)$'` periodically
-4. Educate contributors about artifact policies
-
----
-
-## Related Files
-
-- **Guidelines:** `docs/COMMIT_GUIDELINES.md`
-- **Cleanup script:** `scripts/cleanup-repo-artifacts.sh`
-- **Gitignore:** `.gitignore`
-- **Beads playbooks:** `playbooks/beads-{agent,human}-playbook.md`
-
----
-
-## Philosophy
-
-**Core principle:** *Git is for source code, not generated artifacts.*
-
-**Rationale:**
-- Database files are generated from JSON (the source of truth)
-- Built bundles are generated from TypeScript source
-- Research papers should be linked, not embedded
-- Backups belong in backup systems, not version control
-
-**Goal:** Keep the repository lean, fast, and comprehensible.
-
----
-
-## Before/After Comparison
-
-### Before
-
-```
-Files tracked:    966
-.git size:        193 MB
-Issues:           Databases, PDFs, backups committed
-Prevention:       Weak .gitignore patterns
-Documentation:    None
-```
-
-### After (Current)
-
-```
-Files tracked:    954
-.git size:        193 MB (history unchanged)
-Issues:           Future commits prevented
-Prevention:       Comprehensive .gitignore
-Documentation:    COMMIT_GUIDELINES.md
-Tools:            cleanup-repo-artifacts.sh
-```
-
-### After (If History Cleaned)
-
-```
-Files tracked:    954
-.git size:        ~140 MB (estimated)
-Issues:           Resolved
-Prevention:       Comprehensive .gitignore
-Documentation:    COMMIT_GUIDELINES.md
-Tools:            cleanup-repo-artifacts.sh
-```
-
----
-
-## Commits
-
-- `0c3015e` - Remove artifacts and add commit guidelines
-- `aee1d2a` - Add Beads playbooks (includes initial .resonance/cache cleanup)
-
----
-
-**Conclusion:** Immediate improvements achieved. Full history cleanup optional but recommended for long-term repository health.

package/docs/SESSION-2026-01-06-METADATA-PATTERNS.md

@@ -1,346 +0,0 @@
-# Session Summary: Agent-First Metadata & Latent Space Tagging
-
-**Date:** 2026-01-06  
-**Duration:** ~1 hour  
-**Topic:** Agent autonomy in metadata generation
-
----
-
-## Key Insights
-
-### 1. The Inversion: Agent Does, Human Audits
-
-**Traditional model:**
-- Human approves every tag, link, decision
-- Bottleneck: O(N) human effort
-
-**New model:**
-- Agent auto-augments everything
-- Human audits via git diff (O(log N) effort)
-- Human corrects errors when found
-- System learns from corrections
-
-**Result:** Scales without human bottleneck.
-
-### 2. Latent Space Tagging
-
-**Innovation:** Tags emerge from vector clustering, not predefined taxonomy.
-
-**How it works:**
-```python
-# Cluster documents in embedding space
-clusters = cluster_embeddings(all_docs, min_size=3)
-
-# Generate labels from cluster content
-for cluster in clusters:
-    label = generate_label(cluster.documents)
-    # e.g., "auth-state-patterns"
-    
-    # Tag all docs with confidence scores
-    for doc in cluster.documents:
-        doc.add_tag(f"latent:{label}", confidence)
-```
-
-**Advantages:**
-- No taxonomy to maintain
-- Adapts as corpus grows
-- Multi-cluster membership
-- Confidence scores expose uncertainty
-
-### 3. Git as Safety Net
-
-**All agent augmentations are git commits:**
-
-```bash
-# Agent augments
-[Amalfa: auto-tagged debrief-auth-refactor]
-
-# Human reviews
-$ git diff
-
-# Human corrects if needed
-$ vim debrief.md  # remove incorrect tag
-$ git commit -m "Remove incorrect tag"
-
-# Daemon syncs automatically
-[Amalfa: re-indexed, learned from correction]
-```
-
-**Benefits:**
-- Atomic (one commit per augmentation)
-- Auditable (see what changed)
-- Reversible (revert anytime)
-- Non-destructive (history preserved)
-
-### 4. Seven Auto-Augmentation Patterns
-
-1. **Latent Space Tagging** - Cluster-based organization
-2. **Entity Extraction & Auto-Linking** - Wiki links inserted automatically
-3. **Topic Modeling** - High-level themes (LDA/BERTopic)
-4. **Similarity-Based Suggested Reading** - Context for new sessions
-5. **Temporal Sequences** - Track work evolution
-6. **Semantic Backlinks** - Bidirectional links maintained automatically
-7. **Confidence-Based Tag Weighting** - Express uncertainty, learn from removals
-
----
-
-## Documents Created
-
-### 1. VISION-AGENT-LEARNING.md
-
-**Purpose:** Vision document for agent-generated knowledge
-
-**Key sections:**
-- The brief-debrief-playbook flywheel
-- Spec-driven vs. learning-driven development
-- Human as reader, not writer
-- Implications for Amalfa design
-- Evolution path (manual → emergent)
-
-**Size:** ~1200 lines
-
-**Main insight:** Documentation is cognition, not just artifact. Writing forces reflection.
-
-### 2. AGENT-METADATA-PATTERNS.md
-
-**Purpose:** Design document for auto-augmentation system
-
-**Key sections:**
-- Core principles (agent autonomy, git as truth, optimistic metadata)
-- Pattern library (7 patterns with implementations)
-- Daemon's role (continuous file watching)
-- Human audit workflow
-- Configuration & implementation phases
-
-**Size:** ~1000 lines
-
-**Main insight:** Metadata is optimistically generated, occasionally corrected (not pessimistically approved upfront).
-
----
-
-## Folder Structure
-
-Amalfa already has the brief-debrief-playbook pattern from PolyVis:
-
-```
-amalfa/
-├── briefs/
-│   ├── pending/      # Active work
-│   ├── holding/      # Deferred work
-│   ├── archive/      # Completed work
-│   └── README.md     # Workflow guide
-│
-├── debriefs/         # ~100+ dated debriefs
-│   ├── 2025-01-03-embedder-refactor.md
-│   ├── 2025-01-02-...md
-│   └── README.md     # Debrief guide
-│
-├── playbooks/        # ~30+ domain playbooks
-│   ├── alpinejs-playbook.md
-│   ├── embeddings-and-fafcas-protocol-playbook.md
-│   ├── local-first-vector-db-playbook.md
-│   ├── problem-solving-playbook.md
-│   └── README.md     # Playbook index
-│
-└── docs/
-    ├── VISION-AGENT-LEARNING.md
-    ├── AGENT-METADATA-PATTERNS.md
-    └── SETUP.md
-```
-
-**Status:** Infrastructure already in place, ready for Amalfa enhancement.
-
----
-
-## The Auto-Augmentation Workflow
-
-### Phase 1: Agent Writes
-
-```markdown
-# Debrief: Auth Refactor
-
-## What Worked
-- Alpine's x-data pattern eliminated manual state tracking
-
-## Lessons Learned
-- Alpine for UI state, localStorage for persistence
-```
-
-### Phase 2: Agent Saves → Auto-Augmentation
-
-```bash
-$ amalfa auto-augment debrief-auth-refactor.md
-
-Processing...
-  ✓ Entity extraction (found: Alpine, x-data, localStorage)
-  ✓ Auto-linking (3 links inserted)
-  ✓ Clustering (auth-state-patterns, 0.91)
-  ✓ Similarity search (5 neighbors)
-  ✓ Tag extraction (6 tags)
-  ✓ Metadata generation
-  
-[Amalfa: auto-tagged debrief-auth-refactor]
-```
-
-### Phase 3: Result
-
-```markdown
----
-type: debrief
-brief_id: brief-auth-refactor
-
-# Auto-generated by Amalfa (edit freely)
-tags:
-  explicit: [alpine.js, state-management, localStorage]
-  latent:
-    - auth-state-patterns (0.91)
-    - ui-reactivity (0.78)
-
-links:
-  - playbook-alpine-patterns (uses-pattern, 0.89)
-  - debrief-session-management (similar-problem, 0.87)
-
-suggested_reading:
-  - debrief-session-management (0.87)
-  - playbook-state-patterns (0.82)
----
-
-# Debrief: Auth Refactor
-
-## What Worked
-- [[playbook-alpine-patterns|Alpine's x-data pattern]] eliminated state tracking
-...
-```
-
-### Phase 4: Human Audits (Weekly)
-
-```bash
-$ git log --since="1 week ago" --grep="Amalfa:" --oneline
-
-a7f3d2e Amalfa: auto-tagged debrief-auth-refactor
-8b2e4f1 Amalfa: re-clustered corpus (15 new docs)
-
-# Review, correct if needed
-$ vim debrief-auth-refactor.md  # remove incorrect tag
-$ git commit -m "Remove incorrect tag"
-
-# Daemon learns from correction
-```
-
----
-
-## Implementation Phases
-
-### Phase 1: Basic Auto-Augmentation (MVP)
-- Entity extraction
-- Auto-linking (high similarity)
-- Tag extraction
-- Embedding generation
-- Git commits
-
-**Result:** Agent writes → tags + links added automatically
-
-### Phase 2: Latent Space Tagging
-- Document clustering
-- Auto-generated cluster labels
-- Confidence scores
-- Topic modeling
-- Re-clustering trigger
-
-**Result:** Documents self-organize without predefined taxonomy
-
-### Phase 3: Semantic Relationships
-- K-nearest neighbors
-- Suggested reading
-- Temporal sequences
-- Backlink maintenance
-
-**Result:** Agents get context quickly on new sessions
-
-### Phase 4: Learning from Corrections
-- Track human edits
-- Adjust confidence thresholds
-- Improve extraction
-- Weekly digest
-
-**Result:** System improves over time
-
----
-
-## Success Metrics
-
-### Agent Productivity
-
-**Before:** 25 minutes (write 15m + tag 5m + link 5m)  
-**After:** 15 minutes (write 15m + auto-augment 2s)  
-**Gain:** 40% faster
-
-### Human Audit Overhead
-
-**Target:** O(log N)
-
-- 10 docs: 5 min weekly
-- 100 docs: 15 min weekly
-- 1000 docs: 30 min weekly
-
-**Corrections needed:** <5% of augmentations
-
-### Knowledge Discovery
-
-**Before:** 40 minutes (scan titles + read 5-10 docs)  
-**After:** 10 minutes (semantic search 5s + read top 3)  
-**Gain:** 75% faster
-
----
-
-## Key Quotes
-
-> "The main benefit from the tags and links is for the agent coming in to a new session and getting up to speed easily and quickly - so the more the agent can do the better."
-
-> "If the human finds anything they object to they can remove or modify it - the daemon will pick up the changes and the ingestion will handle it as usual."
-
-> "What you should do is provide a set of patterns the agent can use, EG a latent space tag system."
-
----
-
-## Next Steps
-
-1. **Review existing briefs/debriefs/playbooks** - Understand current patterns
-2. **Design Amalfa schema** - Support brief-debrief-playbook types
-3. **Implement Phase 1** - Basic auto-augmentation (MVP)
-4. **Test on Amalfa corpus** - Use existing debriefs as test data
-5. **Iterate based on experience** - Learn from PolyVis migration
-
----
-
-## Related Documents
-
-- `VISION-AGENT-LEARNING.md` - Why agent-generated knowledge works
-- `AGENT-METADATA-PATTERNS.md` - How to implement auto-augmentation
-- `SETUP.md` - NPM/GitHub setup for publishing Amalfa
-- `briefs/README.md` - Brief workflow guide
-- `debriefs/README.md` - Debrief workflow guide
-- `playbooks/README.md` - Playbook evolution guide
-
----
-
-## Conversation Context
-
-This session built on:
-- Previous discussion: Agents using Amalfa for cross-session continuity
-- PolyVis experience: Brief-debrief-playbook pattern proved effective
-- Meta-insight: Agents spontaneously maintained docs without prompting
-
-**The leap:** If agents naturally maintain docs when given structure, and Amalfa provides semantic infrastructure, then agents can achieve true cross-session continuity with minimal human overhead.
-
-**The mechanism:** Latent space tagging + git-based auditing = agent autonomy at scale.
-
----
-
-**Status:** Vision and design documents complete  
-**Next:** Implementation planning and MVP development  
-**Timeline:** Phases 1-2 achievable in weeks, not months
-
----
-
-_Documentation is not an artifact of work - it's a cognitive tool. Auto-augmentation scales this tool to agent speed._