amalfa 1.0.2 → 1.0.4

package/docs/AGENT_PROTOCOLS.md

@@ -1,28 +0,0 @@
-# Agent Protocols: The "Graph First" Directive
-
-**Target Audience:** AI Agents (Coding Assistants, MCP Clients)
-**Status:** Active Protocol
-
-## 1. The Prime Directive: Verify, Then Act
-You are operating within the **PolyVis Resonance System**, a knowledge graph that tracks not just code, but the *intent* and *history* behind it.
-
-**Rule:** Before you modify any complex system (Ingestion, Vector Engine, Zombie Defense), you must **Ground Yourself** in the existing context.
-
-## 2. The Trigger: When to use the MCP Server?
-You possess a tool called `search_documents` (or similar). You must use it when:
-1.  **Ambiguity:** "I don't know why this code is here." -> Query: *"Why does ZombieDefense exist?"*
-2.  **Architecture:** "Where does the data flow?" -> Query: *"Explain the ingestion pipeline data flow."*
-3.  **Standards:** "How should I log errors?" -> Query: *"What is the Harden and Flense protocol?"*
-
-## 3. The "Graph First" Workflow
-Do not rely on your training data or random file searches. The codebase evolves faster than your context window.
-1.  **Ask the Graph:** `mcp.search_documents("concept")`
-2.  **Read the Referenced Docs:** (The search results will give you file paths).
-3.  **Implement:** Only then do you write code.
-
-## 4. Known Pitfalls (Do Not Do This)
-*   **Do not hallucinate file paths.** Use `list_directory_structure` or `search_documents` to find where things live.
-*   **Do not ignore `stderr`.** If a tool fails, read the error.
-*   **Do not create "Zombie" processes.** Respect the `ServiceLifecycle`.
-
-*Trust the Graph. It remembers what you forget.*

package/docs/ARCHITECTURAL_OVERVIEW.md

@@ -1,123 +0,0 @@
-# PolyVis: Architectural Overview & Executive Summary
-
-**Version:** 1.0 (Hollow Node / Logging Verified)  
-**Date:** 2025-12-29
-
-## 1. Executive Summary
-
-PolyVis is a high-performance, local-first **Knowledge Graph & Agentic Substrate**. Unlike traditional web applications that prioritize cosmetic UI rendering, PolyVis prioritizes **data sovereignty, raw speed, and machine interpretability**.
-
-It replaces the "glue code" of modern stacks (React, Redux, REST) with a direct-to-metal approach using **Bun**, **SQLite**, and **Canvas-based visualization**. This architecture, dubbed **"Hollow Node"**, allows it to visualize and reason about knowledge graphs orders of magnitude larger than typical DOM-based tools, while interacting seamlessly with AI Agents via the **Model Context Protocol (MCP)**.
-
----
-
-## 2. Technology Stack
-
-PolyVis utilizes a bi-modal stack designed for zero-latency interaction.
-
-### Client (The Visor)
-- **Runtime:** Vanilla JavaScript (ES Modules).
-- **Rendering:** `sigma.js` (Canvas/WebGL) for graph rendering; direct DOM for simple UI panels.
-- **Interactivity:** `alpine.js` for lightweight reactivity (no Virtual DOM overhead).
-- **Styling:** `basecoat-css` + Vanilla CSS Layers; **No Tailwind** (except for utility generation), preserving pure CSS maintainability.
-- **Transport:** Standard `fetch` / `WebSocket` (if needed) - no complex client-side routers.
-
-### Server (The Substrate)
-- **Runtime:** **Bun** (Zig-based JS runtime) – providing 3x-10x startup speeds vs Node.js.
-- **Database:** `bun:sqlite` (FFI) – Direct in-process database access.
-    - **Performance:** Reads at C-speed, bypassing network serialization.
-- **ORM:** Drizzle ORM (for schema definitions and migrations only; raw SQL used for hot loops).
-- **Vector Engine:** `fastembed` (running locally via ONNX) for semantic search.
-- **Agent Interface:** `@modelcontextprotocol/sdk` (MCP) – Exposes the graph as tools (`search_documents`, `read_node`) to LLMs.
-
----
-
-## 3. Products & Services
-
-PolyVis is composed of four distinct, interoperable subsystems:
-
-### A. Resonance (The Backend)
-The beating heart of the system.
-- **ResonanceDB:** A wrapper around SQLite that handles graph topology (Nodes/Edges) and Vector embeddings in a single file (`resonance.db`).
-- **Hollow Node Pattern:** Nodes store light metadata; heavy content is read JIT from the filesystem (`read_node_content`). Result: ~60% db size reduction.
-
-### B. The Pipeline (Ingestion)
-Automatically converts raw files into the Knowledge Graph.
-- **Ingestor:** Watches filesystem (`watch`), parses Markdown/Frontmatter, embeds chunks, and upserts to DB.
-- **Semantic Harvester:** A Python bridge (`src/pipeline/SemanticHarvester.ts`) that runs complex NLP (Sieve+Net) to extract semantic triples (`Entity -> Relation -> Entity`).
-- **Gardeners:** Autonomous background agents (e.g., `AutoTagger`) that refine the graph (maintenance).
-
-### C. The Visor (The UI)
-A minimal, heavy-duty visualization tool.
-- **Sigma Explorer:** Interactive graph exploration.
-- **Quick Look:** Markdown rendering of selected nodes.
-
-### D. MCP Server (The API)
-The interface for AI Agents (Cursor, Claude, etc.).
-- **Tools:** `search_documents`, `read_node_content`, `explore_links`.
-- **Protocol Safety:** strict `stdout` (JSON-RPC) vs `stderr` (Logs) separation.
-
----
-
-## 4. Potential Uses
-
-| Use Case | Description |
-| :--- | :--- |
-| **Agentic RAG** | Providing LLMs with a structured, navigable map of a codebase or knowledge base, reducing hallucination via graph traversal. |
-| **Codebase Cartography** | Visualizing complex dependencies in legacy software projects to aid refactoring. |
-| **Personal Knowledge Graph** | A local-first "Second Brain" that connects obsidian-style markdown files semantically. |
-| **Forensic Analysis** | Ingesting logs or timeline data to visualize causal chains in incident responses. |
-
----
-
-## 5. Stats & Benchmarks
-
-### "Hollow Node" Efficiency
-By completely removing the Full-Text Search (FTS) engine and huge content blobs from the DB, PolyVis achieves extreme efficiency:
-- **Database Size:** Reduced from **5.9MB** to **~2.3MB** (61% reduction).
-- **Search Speed:** < 20ms for Vector Similarity search.
-- **Ingestion Speed:** ~450+ files processed and woven in seconds.
-
-### Ingestion Throughput
-- **Bun:** Zero-startup time allows the Daemon to restart instantly.
-- **Vectorization:** Local `fastembed` avoids API latency (OpenAI/Azure), enabling offline ingestion of thousands of chunks.
-
----
-
-## 6. Strategic Radar Analysis
-
-Comparing **PolyVis** against a standard **"Modern Enterprise Stack"** (Next.js / Python Backend / Neo4j / Cloud Vector DB).
-
-![architectural_overview](architectural_overview.png)
-
-### Key Factors for Success (Axes)
-1.  **Velocity:** Speed of runtime execution and development iteration.
-2.  **Scalability (Local):** Ability to handle node count on a single machine without lag.
-3.  **Simplicity:** Absence of "black box" frameworks; ease of audit.
-4.  **Local-First:** functionality without internet/cloud dependencies.
-5.  **Agentic Readiness:** Native support for tool-calling/MCP.
-6.  **Payload Efficiency:** Small memory/disk footprint.
-7.  **Visual Density:** Ability to render thousands of data points at once.
-
-### The Radar Data
-
-| Metric (0-10) | **PolyVis** (Bun/SQLite/Sigma) | **React / Next.js Stack** | **Enterprise Graph (Neo4j)** |
-| :--- | :---: | :---: | :---: |
-| **Velocity** | **10** (Zig/C++) | 6 (Node/V8) | 5 (Java JVM) |
-| **Scalability (Local)** | **9** (Canvas/WebGL) | 3 (DOM limits) | 8 (Backend-only) |
-| **Simplicity** | **9** (Raw SQL/JS) | 4 (Hydration/SSR complexities) | 3 (Admin overhead) |
-| **Local-First** | **10** (Bun:sqlite) | 5 (API dependent) | 6 (Server dependent) |
-| **Agentic Readiness** | **10** (Native MCP) | 6 (Requires integration) | 5 (JDBC/Bolt drivers) |
-| **Payload Efficiency** | **9** (Hollow Node) | 4 (Large bundles) | 4 (JVM Overhead) |
-| **Visual Density** | **9** (WebGL) | 2 (HTML Elements) | N/A (Backend) |
-
-### Interpretation
-*   **The "DOM Wall":** React/Next.js stacks fail at *Visual Density* and *Local Scalability* because the DOM cannot handle >3,000 nodes efficiently. PolyVis (Canvas) handles 50,000+.
-*   **The "Cloud Tax":** Enterprise stacks score low on *Velocity* and *Simplicity* due to setup overhead and cloud latency. PolyVis scores high by keeping everything in-process or IPC.
-*   **The "Agent Gap":** Most apps are built for Humans (HTML). PolyVis is built for Agents (MCP) first, Humans second, ensuring high *Agentic Readiness*.
-
----
-
-## 7. Conclusion
-
-PolyVis is not just a graph visualizer; it is a **reference architecture for the Agentic Age**. By rejecting the bloat of the Browser Wars (React/Virtual DOM) and embracing the speed of modern runtimes (Bun) and established protocols (MCP), it delivers a tool that is simultaneously lighter, faster, and smarter than its enterprise counterparts.

package/docs/BENTO_BOXING_DEPRECATION.md

@@ -1,281 +0,0 @@
-# Bento Boxing Deprecation
-
-**Date:** January 5, 2026  
-**Status:** ❌ Deprecated and Removed  
-**Replaced By:** Whole-document vector embeddings
-
----
-
-## What Was Bento Boxing?
-
-**Bento Boxing** was a markdown chunking system designed to fragment large documents into smaller, semantically meaningful pieces ("bentos") for better vector search precision.
-
-### Components
-
-**Code (Removed):**
-- `src/core/BentoBoxer.ts` - Chunking logic (split by H1-H4 headings)
-- `src/data/LocusLedger.ts` - Content deduplication (hash → UUID mapping)
-- `src/index.ts` - CLI tool for processing markdown files
-- `tests/bento_ast.test.ts` - Unit tests
-
-**Database (Removed):**
-- `bento_ledger.sqlite` - Deduplication ledger (343 entries)
-
-**Playbooks/Briefs (Removed):**
-- `briefs/archive/1-brief-polyvis-bento-implementation.md`
-- `briefs/archive/2-bento-box-core-logic.md`
-- `playbooks/bento-box-playbook-2.md`
-
----
-
-## Why It Was Deprecated
-
-### 1. Never Integrated with Vector Search
-
-**Critical issue:** Bento Boxing was an orphaned CLI tool, not integrated into the main ingestion pipeline.
-
-- ✅ Code existed and worked
-- ❌ Never used by `src/pipeline/Ingestor.ts`
-- ❌ Never used by `src/resonance/db.ts`
-- ❌ Not connected to `public/resonance.db`
-
-**Result:** Documents were ingested whole, not chunked. Vector search operated on complete documents.
-
----
-
-### 2. Whole-Document Embeddings Work Excellently
-
-**Testing revealed chunking was unnecessary:**
-
-| Metric | Value | Assessment |
-|--------|-------|------------|
-| Average best match | 85.2% | Excellent |
-| Average spread | 21.1% | Good differentiation |
-| Corpus size | 489 docs | Manageable |
-| Average doc size | 2.7 KB (~550 words) | Already chunk-sized |
-
-**Key insight:** 80% of documents are <5KB. They're already "chunk-sized" for embedding models.
-
----
-
-### 3. Document Size Distribution
-
-**Analysis of 489 documents:**
-
-```
-Size Range     | Percentage | Chunking Benefit
----------------|------------|------------------
-< 5KB          | ~80%       | None (already small)
-5-20KB         | ~15%       | Minimal
-> 20KB         | ~5%        | Potential (but not critical)
-```
-
-**Largest document:** 47KB (~9,500 words)
-- Still within LLM context windows (100K+ tokens)
-- Embedding captures main themes well
-- Can use grep for exact phrase search
-
----
-
-### 4. Complexity vs Benefit
-
-**Costs of chunking:**
-- ❌ Chunk logic (where to split?)
-- ❌ Chunk→document mapping
-- ❌ Context loss (chunks lose surrounding context)
-- ❌ Storage overhead (10x nodes for chunked docs)
-- ❌ Search complexity (multiple chunks from same doc in results)
-- ❌ UI complexity (show chunk vs full doc?)
-
-**Benefits in this corpus:**
-- ⚠️ Slightly better precision for 5% of large docs
-- ⚠️ Granular retrieval (already achievable with grep)
-
-**Verdict:** Costs > Benefits
-
----
-
-## Search Architecture (Post-Deprecation)
-
-### Two-Tier Search System
-
-**1. Vector Search (Primary)**
-- Purpose: Semantic similarity, concept discovery
-- Accuracy: 85.2% average best match
-- Speed: <10ms per query
-- Handles: "Find documents about CSS patterns"
-
-**2. Grep/Ripgrep (Secondary)**  
-- Purpose: Exact phrase matches
-- Accuracy: 100% (literal text)
-- Speed: <1ms
-- Handles: "Find exact phrase 'function fooBar'"
-
-**No chunking needed:** This two-tier approach covers all search use cases.
-
----
-
-## Decision Criteria
-
-### When Chunking IS NOT Needed
-
-✅ **Keep whole-document embeddings if:**
-- Average doc size <5KB (most docs already chunk-sized)
-- Vector search accuracy >70% (yours is 85%)
-- Documents are well-structured (markdown with headers)
-- Search is semantic (not keyword BM25)
-- Source files are easily searchable with grep
-
-**Polyvis meets ALL these criteria.**
-
----
-
-### When Chunking WOULD Be Needed
-
-Consider adding chunking if/when:
-
-**1. External large documents**
-- Research papers (30-50 pages)
-- Books, manuals (100+ pages)
-- API documentation (needs endpoint-level chunks)
-
-**2. Accuracy degradation**
-- Vector search drops below 70%
-- Users report irrelevant results
-- Long documents dominate search results
-
-**3. Specific requirements**
-- RAG system needs paragraph-level context
-- Need to cite specific sections, not whole docs
-- Document structure doesn't match search granularity
-
----
-
-## Migration Notes
-
-### What Changed
-
-**Removed:**
-- All Bento Boxing source code
-- bento_ledger.sqlite database
-- CLI tool (`bun run src/index.ts box`)
-- Related briefs and playbooks
-
-**Unchanged:**
-- Vector search pipeline (always used whole docs)
-- Ingestion pipeline
-- Database schema
-- Search accuracy (still 85%)
-
-**No migration required:** Bento Boxing was never in production.
-
----
-
-## Historical Context
-
-### Development Timeline
-
-**December 2025:**
-- Bento Boxing designed and implemented
-- CLI tool created for markdown chunking
-- Deduplication ledger built (343 entries)
-- Playbooks and briefs written
-
-**January 2026:**
-- Vector search testing revealed 85% accuracy without chunking
-- Discovered Bento Boxing never integrated with main pipeline
-- Analysis showed 80% of docs are already chunk-sized
-- Decision: Deprecate and remove
-
-**Lesson:** Test effectiveness before building infrastructure.
-
----
-
-## Future Considerations
-
-### Recommended Approach: File Splitting (Not Runtime Chunking)
-
-**If large documents (>15-20KB) become problematic, use simple file splitting:**
-
-**Strategy:**
-1. Parse document structure with `ast-grep` or `marked`
-2. Split at natural boundaries (H1/H2 headers)
-3. Create multiple markdown files (e.g., `agents-part-1.md`, `agents-part-2.md`)
-4. Add metadata: `<!-- Part 1 of 3 -->`
-5. Optional: Keep parent file as TOC with links to parts
-6. Commit split files to version control
-
-**Advantages:**
-- ✅ **Simple:** No infrastructure, just split files once
-- ✅ **Git-native:** Diffs are meaningful, history is granular
-- ✅ **Transparent:** Files are the chunks (source of truth)
-- ✅ **Reversible:** Reconstruct with `cat part-*.md > full.md`
-- ✅ **Lazy:** Only split the 5% of docs that need it
-
-**When to Split:**
-
-| Document Size | Action |
-|---------------|--------|
-| <10KB | Leave as-is |
-| 10-20KB | Consider if natural boundaries exist |
-| >20KB | Strong candidate for splitting |
-
-**Example: Splitting AGENTS.md (47KB)**
-```bash
-# Parse and split at H1 boundaries
-AGENTS.md → agents-tier1.md (protocols 1-6)
-          → agents-tier2.md (protocols 7-18)
-          → agents-tier3.md (playbooks index)
-
-# Keep parent as TOC
-AGENTS.md → "# Agent Protocols\n\nSee:\n- [Tier 1](agents-tier1.md)..."
-```
-
-**Anti-Patterns to Avoid:**
-- ❌ Premature splitting ("what if it grows?")
-- ❌ Runtime chunking infrastructure
-- ❌ Artificial boundaries (mid-paragraph splits)
-- ❌ Complex deduplication/mapping systems
-
-**Why This Works:**
-- Documents remain markdown files in git
-- Vector search ingests each part as separate node
-- Search results link to specific part files
-- Humans edit parts independently
-- Reconstruction is trivial when needed
-
----
-
-## References
-
-### Effectiveness Testing
-
-See `scripts/test-embeddings.ts` for validation:
-- 85.2% average best match
-- 21.1% spread
-- Tested across 5 query types (CSS, database, graph, debugging, tooling)
-
-### Related Documentation
-
-- `src/resonance/README.md` - Search Architecture section
-- `.legacy-databases-README.md` - bento_ledger.sqlite removal
-- `playbooks/README.md` - Updated to remove Bento Boxing references
-
----
-
-## Summary
-
-**Bento Boxing was well-designed but unnecessary:**
-- Never integrated into production pipeline
-- Whole-document embeddings achieve excellent results (85%)
-- Most documents are already chunk-sized (<5KB)
-- Two-tier search (vector + grep) covers all use cases
-
-**Decision:** Remove to simplify codebase. Revisit chunking only if:
-- Adding large external documents (books, long PDFs)
-- Vector search accuracy drops significantly
-- Specific use case emerges that requires granular retrieval
-
----
-
-**Last updated:** 2026-01-05