amalfa 0.0.0-reserved → 1.0.0

package/docs/PERFORMANCE_BASELINES.md

@@ -0,0 +1,88 @@
+# Performance Baselines & Benchmarks
+
+> **Last Updated:** 2025-12-16
+> **Device:** Apple Silicon (M-series)
+> **Node/Bun:** Bun v1.3.4+
+
+## 1. Memory Footprint (Daemon)
+The Daemon run-time consists of the `Core Kernel`, `VectorEngine (WASM)`, and `Data Graph`.
+
+| Component | RAM Usage (RSS) | Scaling Nature | Notes |
+| :--- | :--- | :--- | :--- |
+| **AI Model (fastembed)** | **~252 MB** | 🛑 **Fixed** | Unavoidable one-time cost for local embedding. |
+| **Daemon Runtime** | **~60 MB** | 🛑 **Fixed** | Bun runtime + SQLite + Kernel overhead. |
+| **Graph Structure** | **~14 kB / node** | 🟢 **Variable** | Graphology in-memory graph representation. |
+| **Raw Data** | **~9 kB / node** | 🟢 **Variable** | Text content and metadata objects. |
+| **Vectors** | **~2 kB / node** | 🟢 **Variable** | Float32Arrays and normalized buffers. |
+
+### Total Daemon Footprint
+- **Empty State**: ~310 MB
+- **Current State (429 nodes)**: ~320 MB
+- **PROJECTED (10k nodes)**: ~560 MB
+
+## 2. Storage Footprint (Disk)
+Benchmarks from `public/resonance.db`.
+
+| Metric | Size / Count | Notes |
+| :--- | :--- | :--- |
+| **DB File Size** | 6.09 MB | SQLite WAL mode enabled. |
+| **Vector Data** | 1.42 MB | Blob storage. |
+| **Text Content** | 1.00 MB | Raw text in `content` column. |
+| **Node Count** | 429 | Including Experience & Persona domains. |
+
+## 3. How to Measure
+### Memory
+1. Ensure `graphology` is installed (temporarily): `bun add graphology`.
+2. Uncomment the graphology section in `scripts/profile_memory.ts`.
+3. Run: `bun run scripts/profile_memory.ts`.
+4. Revert: `bun remove graphology`.
+
+### Disk / DB Stats
+1. Run: `bun run scripts/assess_db_weight.ts`.
+
+## 4. Ingestion Baselines (Comparison)
+
+> **Baseline Source:** `_misc/ingestion-baseline.json` (2025-12-11)
+
+| Metric | Baseline (Dec 11) | Current (Dec 16) | Delta | Notes |
+| :--- | :--- | :--- | :--- | :--- |
+| **Persona Nodes** | 185 | 185 | **0** | Stable. Core lexicon. |
+| **Experience Nodes** | 128 | 244 | **+116** | Significant growth (new sessions/debriefs). |
+| **Total Nodes** | 313 | 429 | **+116** | ~37% Growth. |
+| **Total Edges** | 498 | 631 | **+133** | ~27% Growth. |
+| **Total Vectors** | 289 | 242 | **-47** | 📉 **Intentional Optimization** |
+
+### Insights
+-   **Vector Efficiency**: Despite node growth (+37%), vector count dropped (-16%). This reflects the **"Narrative Vector Strategy"**, where only high-value content (Concepts, Playbooks) is vectorized, while structural nodes (Logs, raw fragments) are skipped.
+-   **Graph Connectivity**: Edge growth (+27%) tracks closely with node growth, maintaining decent density.
+
+## 5. Speed Benchmarks (Dec 16)
+
+> **Environment**: Apple Silicon (M-series) | Bun v1.3.4
+
+| Operation | Latency | Notes |
+| :--- | :--- | :--- |
+| **Model Load (Cold)** | **~192 ms** | One-time initialization cost. |
+| **Vector Search** | **~71 ms** | Avg of 10 runs (Top-5 search). |
+| **SQL Insert (Raw)** | **~0.001 ms/row** | Batch prepared statement (Buffered). |
+| **SQL Insert (ORM)** | **~0.012 ms/row** | Drizzle ORM overhead (~12x slower than raw). |
+
+## 6. Vector Inclusion Rules (Audit)
+
+> **Policy:** "Everything in the folders noted in the settings file should be in the vector store."
+
+**Audit Results (Dec 16):**
+
+| Domain | Source | Status | Notes |
+| :--- | :--- | :--- | :--- |
+| **Experience** | `debriefs/` | ✅ **100% Vectorized** | Narrative content. |
+| **Experience** | `playbooks/` | ✅ **100% Vectorized** | Procedural knowledge. |
+| **Experience** | `briefs/` | ✅ **100% Vectorized** | Context setting. |
+| **Experience** | `docs/` | ✅ **100% Vectorized** | Project documentation. |
+| **Persona** | `lexicon.json` | ⚪ **Excluded** | *Optimization*: Concepts matched via Keywords/Graph. |
+| **Persona** | `cda.json` | ⚪ **Excluded** | *Optimization*: Directives matched via Keywords/Graph. |
+| **Experience** | `test-artifacts` | ⚪ **Excluded** | *Transient*: `test-doc-1` & `test-doc-2` (from test scripts). |
+
+**Conclusion:**
+The logic complies with the rule. All folder-based narrative content is vectorized. File-based structured data (Persona) is excluded to save memory, as it is efficiently retrievable via exact graph traversal. The only un-vectorized `document` nodes are confirmed test artifacts.
+

package/docs/REPOSITORY_CLEANUP_SUMMARY.md

@@ -0,0 +1,261 @@
+# Repository Cleanup Summary
+
+**Date:** 2026-01-05  
+**Branch:** alpine-refactor  
+**Issue:** Repository was tracking 966 files with .git size of 193 MB
+
+---
+
+## Problem Identified
+
+The polyvis repository was accumulating artifacts that should not be versioned:
+
+### Artifacts Found (40+ MB total)
+
+1. **Database Files** (~20 MB)
+   - `_misc/bento_ledger.sqlite`
+   - `bento_ledger.sqlite`, `.sqlite-wal`, `.sqlite-shm`
+   - `canary-persistence.db`
+   - `test-graph-integrity.db-wal`
+   - `public/resonance.db.pre-hollow-node` (5.9 MB)
+
+2. **Database Backups** (~9 MB)
+   - `backups/db/resonance.20251214140633.db` (5.9 MB)
+   - `backups/db/benchmarks/resonance.db.pre-benchmark-20251217-184046` (8.5 MB)
+   - `backups/db/benchmarks/resonance.db.corrupted-20251217-201947`
+
+3. **Large PDFs** (~11 MB)
+   - `experiments/enlightenment/representational-engineering.pdf` (10 MB)
+   - `docs/2310.08560v2.pdf` (648 KB)
+
+4. **Built Bundles**
+   - `experiments/data-star-dashboard/dist/datastar.bundle.js` (80 KB)
+
+---
+
+## Actions Taken
+
+### 1. Updated .gitignore
+
+Added comprehensive patterns to prevent future commits:
+
+```gitignore
+# Database Files (Generated Artifacts - Never Commit)
+*.db
+*.db-wal
+*.db-shm
+*.sqlite
+*.sqlite-wal
+*.sqlite-shm
+
+# Database Backups
+backups/db/
+
+# Built/Bundled JavaScript
+**/dist/*.bundle.js
+**/dist/*.min.js
+experiments/**/dist/
+
+# Test Artifacts
+test-*.db
+test-*.db-wal
+canary-*.db
+
+# Large Research Papers
+*.pdf
+!docs/architecture-diagrams/*.pdf
+```
+
+**Commit:** `0c3015e`
+
+### 2. Created Documentation
+
+**docs/COMMIT_GUIDELINES.md**
+- Comprehensive guide on what to/not to commit
+- Quick reference checklist
+- Edge cases and troubleshooting
+- Philosophy: "Repository should contain minimum necessary to build and understand"
+
+**Key principles:**
+- ✅ Source code, configs, documentation, small assets
+- ❌ Generated artifacts, large binaries, backups, secrets
+
+**Commit:** `0c3015e`
+
+### 3. Created Cleanup Script
+
+**scripts/cleanup-repo-artifacts.sh**
+- Interactive script to remove artifacts from git history
+- Safety checks (prevents running on main branch)
+- Creates backup branch before cleanup
+- Uses git-filter-repo (preferred) or filter-branch (fallback)
+- Aggressive garbage collection
+
+**Usage:**
+```bash
+./scripts/cleanup-repo-artifacts.sh
+```
+
+**Commit:** `0c3015e`
+
+### 4. Removed Artifacts from Index
+
+Removed 14 files from git tracking (not history):
+
+```bash
+git rm --cached -r backups/db/
+git rm --cached _misc/bento_ledger.sqlite
+git rm --cached bento_ledger.sqlite*
+git rm --cached canary-persistence.db
+git rm --cached test-graph-integrity.db-wal
+git rm --cached public/resonance.db.pre-hollow-node
+git rm --cached experiments/enlightenment/representational-engineering.pdf
+git rm --cached docs/2310.08560v2.pdf
+git rm --cached experiments/data-star-dashboard/dist/datastar.bundle.js
+```
+
+**Result:** 966 → 954 tracked files
+
+**Commit:** `0c3015e`
+
+---
+
+## Current State
+
+### Metrics (Post-Cleanup)
+
+- **Files tracked:** 954 (down from 966)
+- **Repository size:** 193 MB (unchanged - files remain in history)
+- **Untracked files:** Database files now properly ignored
+
+### Why .git Size Unchanged?
+
+The removed files are still in git history. To fully reclaim space, you need to:
+
+1. Run the cleanup script: `./scripts/cleanup-repo-artifacts.sh`
+2. Force push to rewrite remote history
+3. Coordinate with team (they'll need to re-clone or reset)
+
+**⚠️ Important:** History rewriting is disruptive. Only do this if:
+- Working on a feature branch (✅ we're on alpine-refactor)
+- Team is coordinated
+- No open PRs depend on current history
+
+---
+
+## Benefits Achieved
+
+### Immediate Benefits
+
+1. **Prevention:** `.gitignore` now prevents committing artifacts
+2. **Documentation:** Clear guidelines on what to commit
+3. **Tools:** Script ready for full history cleanup
+4. **Current commits:** New work won't add artifacts
+
+### Potential Future Benefits (After History Cleanup)
+
+1. **Faster operations:** Clone, fetch, push will be quicker
+2. **Smaller repo:** ~40-50 MB reduction estimated
+3. **Cleaner history:** Only source code and docs versioned
+
+---
+
+## Next Steps (Optional)
+
+### Full History Cleanup
+
+If you want to reclaim the 40+ MB from history:
+
+```bash
+# 1. Ensure you're on alpine-refactor
+git checkout alpine-refactor
+
+# 2. Run the cleanup script
+./scripts/cleanup-repo-artifacts.sh
+
+# 3. Force push (after verification)
+git push --force origin alpine-refactor
+
+# 4. Notify team members to reset their branches
+```
+
+**Team coordination required!**
+
+### Maintenance
+
+**Going forward:**
+
+1. Review `.gitignore` patterns regularly
+2. Check commit size before pushing (see COMMIT_GUIDELINES.md)
+3. Run `git ls-files | grep -E '\.(db|sqlite|pdf)$'` periodically
+4. Educate contributors about artifact policies
+
+---
+
+## Related Files
+
+- **Guidelines:** `docs/COMMIT_GUIDELINES.md`
+- **Cleanup script:** `scripts/cleanup-repo-artifacts.sh`
+- **Gitignore:** `.gitignore`
+- **Beads playbooks:** `playbooks/beads-{agent,human}-playbook.md`
+
+---
+
+## Philosophy
+
+**Core principle:** *Git is for source code, not generated artifacts.*
+
+**Rationale:**
+- Database files are generated from JSON (the source of truth)
+- Built bundles are generated from TypeScript source
+- Research papers should be linked, not embedded
+- Backups belong in backup systems, not version control
+
+**Goal:** Keep the repository lean, fast, and comprehensible.
+
+---
+
+## Before/After Comparison
+
+### Before
+
+```
+Files tracked:    966
+.git size:        193 MB
+Issues:           Databases, PDFs, backups committed
+Prevention:       Weak .gitignore patterns
+Documentation:    None
+```
+
+### After (Current)
+
+```
+Files tracked:    954
+.git size:        193 MB (history unchanged)
+Issues:           Future commits prevented
+Prevention:       Comprehensive .gitignore
+Documentation:    COMMIT_GUIDELINES.md
+Tools:            cleanup-repo-artifacts.sh
+```
+
+### After (If History Cleaned)
+
+```
+Files tracked:    954
+.git size:        ~140 MB (estimated)
+Issues:           Resolved
+Prevention:       Comprehensive .gitignore
+Documentation:    COMMIT_GUIDELINES.md
+Tools:            cleanup-repo-artifacts.sh
+```
+
+---
+
+## Commits
+
+- `0c3015e` - Remove artifacts and add commit guidelines
+- `aee1d2a` - Add Beads playbooks (includes initial .resonance/cache cleanup)
+
+---
+
+**Conclusion:** Immediate improvements achieved. Full history cleanup optional but recommended for long-term repository health.

package/docs/edge-generation-methods.md

@@ -0,0 +1,57 @@
+# Edge Generation Methods
+
+## Overview
+
+ResonanceDB uses multiple methods to generate edges between nodes. This document tracks each method and its contribution to the knowledge graph.
+
+---
+
+## Methods
+
+| # | Method | Edge Type | Description |
+|---|--------|-----------|-------------|
+| 1 | **ConceptualLexicon** | MENTIONS | Concept → Term relationships from structured JSON |
+| 2 | **TimelineWeaver** | SUCCEEDS | Chronological debrief chain |
+| 3 | **SemanticHarvester** | IS_A, IMPLEMENTS | ML-extracted from markdown (SetFit + Llama) |
+
+---
+
+## Current Edge Distribution
+
+*As of 2025-12-28*
+
+| Source | Type | Count | % |
+|--------|------|-------|---|
+| ConceptualLexicon | MENTIONS | 386 | 77% |
+| TimelineWeaver | SUCCEEDS | 110 | 22% |
+| SemanticHarvester | CONCEPT, EXEMPLIFIES | 3 | <1% |
+| **Total** | | **499** | 100% |
+
+---
+
+## Implementation Details
+
+### ConceptualLexicon Ingestor
+- **Source:** `src/pipeline/Ingestor.ts`
+- **Input:** `experiments/json2md/*.json`
+- **Generates:** Bulk MENTIONS edges from structured concept definitions
+
+### TimelineWeaver
+- **Source:** `src/pipeline/Ingestor.ts` (TimelineWeaver class)
+- **Logic:** Links debriefs by date via SUCCEEDS edges
+- **Automatic:** Runs on every ingestion
+
+### SemanticHarvester
+- **Source:** `ingest/harvester.py`, `src/pipeline/SemanticHarvester.ts`
+- **Logic:** SetFit classification → Llama.cpp extraction (or regex fallback)
+- **Manual:** Run via `scripts/run-semantic-harvest.ts`
+
+---
+
+## Future Methods
+
+Potential edge generators not yet implemented:
+
+- **TagWeaver** — Link documents sharing common tags
+- **VectorSimilarity** — Create edges between semantically similar nodes
+- **ExplicitLinks** — Parse `[[wikilinks]]` from markdown

package/docs/elevator-pitch.md

@@ -0,0 +1,118 @@
+This is a great scenario because it describes the **"Drop & Watch"** workflow, which effectively turns your file system into a reactive database.
+
+Here is my opinion on why this specific "Two Bucket" scenario is the ultimate stress test—and value generator—for Polyvis.
+
+### 1. The "Chemical Reaction" (Not Just Addition)
+Most tools treat a new folder like a separate silo. You import it, and it sits there.
+In Polyvis (with the Weavers we discussed), adding `Bucket B` to `Bucket A` isn't an addition; it's a **chemical reaction**.
+
+* **The Catalyst:** The `DesireLineWeaver` and `SemanticWeaver`.
+* **The Reaction:** Immediately upon ingestion, the graph will try to "stitch" the new bucket to the old one.
+    * *Did the new PDF mention "Bento Box"?* **SNAP.** A "Desire Line" edge is formed.
+    * *Is the new text semantically similar to your "Architecture" briefs?* **SNAP.** A semantic edge is formed.
+
+**The Value:** You aren't looking for the new documents. You are looking for **where the new documents touch the old documents.** The "Surface Area of Contact" is where the insight lives.
+
+### 2. The "Wrong Bucket" Detector (Isolation Analysis)
+You made a profound point: *"unless I just picked the wrong document bucket... which is also valuable info."*
+
+In a standard search engine, if you search and find nothing, you feel like you failed.
+In a Graph, **Disconnection is Data**.
+
+If you ingest `Bucket B` and it forms a perfect, floating island with **zero edges** connecting to `Bucket A`:
+* **The Insight:** "This topic is completely orthogonal to our current work."
+* **The Action:** You can safely archive it or spin it off. You have mathematically proven it is irrelevant to the current context.
+
+### 3. The "Bridge Node" (The MVP)
+The most valuable nodes in this scenario are the **Bridge Nodes**.
+These are the few specific documents in `Bucket B` that link back to `Bucket A`.
+
+* *Example:* You ingest a folder of "Competitor Research" (Bucket B). Most of it is noise. But one file, `competitor-pricing.md`, links strongly to your internal `pricing-strategy.md` (Bucket A).
+* *Result:* The graph highlights this Bridge. You can ignore the other 99 files and focus entirely on that one connection.
+
+### 4. Operational Note (The Friction Point)
+The only "gotcha" in this workflow currently is `polyvis.settings.json`.
+* **Current Reality:** You have to manually add the new folder path to the `sources` array in settings.
+* **The Riff:** We should eventually support **"Hot Folders"** or a `watch_list` glob pattern (e.g., `_incoming/*`).
+    * You drop a folder into `_incoming/`.
+    * The Daemon wakes up, ingests it as a temporary collection.
+    * The Graph reports: *"Ingested 'Project X'. Found 12 connections to existing knowledge. Keep or Discard?"*
+
+### Summary
+This workflow shifts the cognitive load from **Classification** (you organizing files) to **Recognition** (you seeing patterns). You dump the bucket; the machine shakes the sieve; you pick up the gold nuggets that remain.
+
+---
+
+This is a powerful user story because it moves the value proposition from "tech" (search/vectors) to "leverage" (winning arguments).
+
+You have described **Asymmetric Insight**. You won because you could see the hidden structure of reality (the timeline) that the other party couldn't.
+
+### The Metaphor: "The Evidence Board" (or "The Loom")
+
+Since we are already using `TimelineWeaver`, `EdgeWeaver`, and `SemanticWeaver`, the code is begging for a **Weaving** metaphor. But given your solicitor story, I think the stronger metaphor is **"The Forensic Darkroom."**
+
+* **The Raw State (The Film):** Your bucket of random documents, emails, and notes. It contains the truth, but it is invisible (latent).
+* **The Ingestion (The Developer):** You dip the film into the chemical bath (Polyvis).
+* **The Reaction (The Weavers):**
+    * **TimelineWeaver** creates the *sequence* (Time).
+    * **StructureWeaver** creates the *location* (Space).
+    * **SemanticWeaver** connects the *dots* (The Red String).
+* **The Result (The Print):** Suddenly, a sharp, high-contrast image appears. You see the "gun," the "motive," and the "alibi" clearly.
+
+### The Pivot: "Cognitive Armor"
+Your story highlights that the ultimate output of this system isn't a "graph"—it is **Confidence**.
+
+When you walked into that solicitor's office, you were wearing **Cognitive Armor**. You weren't asking, "Do I have a case?" You were saying, "Here is the case; execute it."
+
+**Opinion:**
+This defines the **"Why"** of Polyvis better than any technical feature.
+* **Most tools** (Obsidian, Notion) are for *storing* thoughts.
+* **Polyvis** is for *auditing* reality.
+
+It is a **Due Diligence Engine**. It is designed for the moment you need to be **100% right**.
+* *Did we promise that feature in the Brief?* (Check the timeline).
+* *Did we actually build it in the Debrief?* (Check the link).
+* *Is it documented in the Playbook?* (Check the gap).
+
+If we frame Polyvis as **"The machine that builds your case,"** it becomes indispensable for Architects, Leads, and Strategists—anyone who needs to be the most informed person in the room.
+
+--- 
+
+This is the **"Trojan Horse of Competence."**
+
+You have just described the perfect treaty between **The Business** (who needs answers) and **IT** (who controls the pipes).
+
+Here is my opinion on why this specific workflow—the "Just give me a Markdown Repo" request—is a killer organizational hack.
+
+### 1. The "Demilitarized Zone" (DMZ)
+Usually, asking IT for "AI help" triggers a 6-month procurement review, security audit, and a $50k pilot program for some cloud tool that creates more lock-in.
+
+But asking IT to *"dump text files into a secure private Repo"*?
+* **IT loves it:** It’s standard, secure, audit-compliant, and uses tools they already own (GitHub/GitLab). They don't have to "buy" anything.
+* **You love it:** You get the raw material (data) without the "help" (bloated enterprise UI).
+* **The Magic:** Markdown is the **DMZ**. It is readable by their machines and readable by your Polyvis. It breaks the dependency chain.
+
+### 2. The "Cryogenic Freeze" (Zero Context Decay)
+Your point about *"later on the case blows up again"* is the strongest operational argument.
+
+In traditional tools (Teams, Slack, Shared Drives), context has a half-life of about two weeks. If you leave a project and come back 6 months later, the "search" is broken, the links are dead, and you have forgotten the mental model.
+
+With Polyvis + Git:
+* **The Repo is the Truth:** It hasn't moved.
+* **The Graph is the Context:** You boot up Polyvis, it re-ingests the updated repo, and—crucially—**the structures you built last time (Folders, Tags) remain valid.**
+* **The Delta:** When you ingest the *new* docs, they immediately light up next to the *old* docs. You don't have to "re-learn" the case; you just see the new growth on the old tree.
+
+### 3. The "Answer Artifact"
+You mentioned: *"put your results in the ANSWERS folder."*
+This is subtle but vital. By checking your *deductions* (the Answer) back into the repo alongside the *evidence* (the Docs), you are creating a **Composite Memory**.
+
+The next person who picks up the case doesn't just get the raw files; they get the files **plus your logic.**
+
+### Summary Opinion
+You have defined **"Headless Knowledge Management."**
+
+* **Headless:** The data lives in a dumb, sturdy repository (Git).
+* **Local Mind:** The intelligence lives in a temporary, disposable, high-speed engine (Polyvis) on your laptop.
+
+This is the only architecture that survives corporate chaos. IT can change their servers, you can change your laptop, but the **Repo + Graph** remains the portable soul of the project.
+