groove-dev 0.27.108 → 0.27.110

package/DYNAMIC_LEAF_ARCH.md

@@ -0,0 +1,488 @@
+# Dynamic Leaf Architecture
+
+**The tree grows itself.**
+
+---
+
+## 1. Executive Summary
+
+The current Hummingbird design uses hardcoded domains. Ten skill leaves (Python, React, PostgreSQL, DevOps, Rust, TypeScript, Data Science, Security, Mobile, System Design) and six reasoning leaves (research, strategy, analysis, debate, synthesis, teaching). The PoC validates the core primitive: a lightweight semantic router selects the right leaf with 93.8% accuracy in under 1ms. The architecture works.
+
+But it does not scale.
+
+Ten skill leaves means ten domains. Domain eleven requires manual intervention: someone must define the domain, write a description, generate a centroid, collect training data, and trigger training. This is fine for a proof of concept. It is fatal for a network that needs to serve molecular biology, legal contract analysis, Kubernetes networking, lightning physics, audio synthesis, and a thousand domains nobody has imagined yet.
+
+This document introduces **Dynamic Leaf Architecture**: leaves emerge organically from user behavior through embedding-space clustering. No human defines domains. The tree discovers them from usage patterns. The tree grows, prunes, splits, and merges itself.
+
+This is the difference between building a model and growing an intelligence.
+
+| Aspect | Fixed Domains (current) | Dynamic Leaves (this doc) |
+|--------|------------------------|--------------------------|
+| Domain count | 10 skill + 6 reasoning | Unlimited, emergent |
+| Adding a domain | Manual definition, data curation, training | Automatic from usage clusters |
+| Centroid generation | Human writes description, encodes it | Cluster centroid computed from real sessions |
+| Training data curation | Manual tagging and filtering | Cluster membership defines the dataset |
+| Tree depth | Flat (one level) | Hierarchical (2-3 levels, data-driven) |
+| Lifecycle management | None | Automated: seed, sprout, mature, prune, split, merge |
+
+---
+
+## 2. The Problem with Fixed Domains
+
+Ten hardcoded skill leaves means ten domains. Domain eleven requires someone to sit down and decide: what is the domain called, what is its description, what prompts belong to it, where does its training data come from, and when should it be trained. Then they must write a centroid description, encode it through all-MiniLM-L6-v2, add it to the router, collect tagged sessions, run the five-step training pipeline, evaluate, and deploy.
+
+This process has three failure modes:
+
+**Boundary ambiguity.** Does "Kubernetes networking" belong in `devops_docker` or a new `networking` leaf? Does "Python data science" belong in `python_expert` or `data_science_ml`? The PoC already shows this: "Write a Python script that trains a neural network" routed to `python` instead of `data_science_ml`. Fixed taxonomies create artificial boundaries that the router must navigate with imperfect cosine similarity.
+
+**Coverage gaps.** A user working on bioinformatics pipelines gets routed to whichever existing leaf is least wrong. A user debugging Erlang concurrency gets routed to `rust` because ownership semantics are vaguely similar. The system cannot serve what it does not know exists.
+
+**Flywheel stall.** The Hummingbird network effect depends on the tree growing with usage. If tree growth requires manual curation, the flywheel stalls. The vision document promises "federated neuroevolution" — algorithmic pruning of dormant leaves and propagation of successful ones. That requires an automated leaf lifecycle, not a hardcoded list maintained by a human.
+
+Fixed domains are training wheels. They proved the router works. Now the training wheels come off.
+
+---
+
+## 3. Core Principle: The Embedding Is the Domain
+
+The key insight is that you do not need domain labels. The embedding vector IS the domain definition.
+
+A session about "debugging asyncio race conditions" and a session about "writing async HTTP clients in Python" naturally embed close to each other in the MiniLM vector space. You do not need a human to label both as "python_async." The math does it.
+
+This eliminates domain taxonomy maintenance, edge case arguments, and granularity debates. The router already works this way — cosine similarity against centroids. We just stop hardcoding the centroids and let them emerge from data.
+
+**How embedding-based domain discovery works:**
+
+1. Every session's task prompt gets encoded by the router's encoder (all-MiniLM-L6-v2) at collection time. The PoC benchmarks this at 11ms average — negligible overhead.
+
+2. The 384-dimensional embedding vector is stored alongside the session telemetry.
+
+3. As sessions accumulate, their embeddings form a point cloud in 384-dimensional vector space.
+
+4. Natural clusters in this space ARE domains. They are groups of sessions that are semantically similar — not because someone labeled them, but because the math of language similarity grouped them.
+
+5. The cluster centroid IS the leaf's routing centroid. No manual centroid generation needed.
+
+6. The cluster's member sessions ARE the leaf's training data. No manual data curation needed.
+
+The domain taxonomy is not designed. It is discovered. And it is discovered from exactly the same embedding space the router already uses, which means routing accuracy improves automatically — the centroids are computed from real user prompts rather than human-authored domain descriptions.
+
+---
+
+## 4. Clustering Algorithm
+
+### Why HDBSCAN
+
+The clustering algorithm must discover domains without being told how many exist. This eliminates K-means (requires pre-specifying K) and makes DBSCAN fragile (requires a fixed epsilon distance threshold that is hard to tune across diverse domains with different densities).
+
+**HDBSCAN** (Hierarchical Density-Based Spatial Clustering of Applications with Noise) is the right algorithm:
+
+- Automatically discovers the number of clusters from data density
+- Handles variable-density clusters — some domains will have thousands of sessions, others hundreds
+- Naturally handles noise — sessions that do not belong to any clear domain are labeled as outliers rather than forced into a wrong cluster
+- Produces a hierarchy of clusters at different granularities
+
+**Parameters:**
+
+```python
+import hdbscan
+
+clusterer = hdbscan.HDBSCAN(
+    min_cluster_size=200,    # minimum sessions to form a domain
+    min_samples=50,          # density threshold — core point requirement
+    metric='cosine',         # match the router's similarity metric
+    cluster_selection_method='eom',  # Excess of Mass for natural hierarchy
+    prediction_data=True     # enable approximate prediction for new points
+)
+
+labels = clusterer.fit_predict(session_embeddings)
+```
+
+### Hierarchical Discovery
+
+The tree depth emerges from data density. Popular areas get deeper trees, niche areas stay shallow.
+
+**Level 1** — broad branches. Run HDBSCAN with `min_cluster_size=1000`. Discovers top-level categories: programming, science, operations, research, design, etc.
+
+**Level 2** — domains. Within each Level 1 cluster, run HDBSCAN with `min_cluster_size=200`. Discovers specific domains: Python, React, PostgreSQL within the programming branch.
+
+**Level 3** — specializations. Within large Level 2 clusters (1000+ sessions), optional run with `min_cluster_size=100`. Discovers sub-specializations: async Python, Python testing, Python data pipelines within the Python domain.
+
+### Hierarchical Routing
+
+The router traverses the hierarchy from most specific to most general:
+
+1. Prompt embeds → check Level 3 centroids first (most specific match)
+2. If best Level 3 match confidence < 0.25 → fall back to Level 2
+3. If best Level 2 match confidence < 0.20 → fall back to Level 1
+4. If nothing matches well → route to the chassis with no adapter (general-purpose fallback)
+
+This graceful degradation means the system always has an answer. Novel domains that have not yet clustered still get a reasonable response from the base chassis or nearest parent branch. The PoC already proved the chassis produces useful output without any leaf loaded.
+
+---
+
+## 5. The Leaf Lifecycle
+
+Leaves are not static artifacts. They are born, grow, mature, evolve, and sometimes die. Each leaf occupies exactly one lifecycle stage at any time.
+
+### SEED — cluster detected, not enough data
+
+A new cluster emerges from HDBSCAN with 200+ sessions. This is a signal: users are doing something coherent in this region of embedding space, but there is not yet enough data to train a specialized leaf.
+
+- A centroid is computed and registered in the router
+- No leaf is trained yet — sessions route to the nearest existing leaf or the parent branch
+- Sessions continue accumulating in the seed cluster
+- The router tracks that this is an imprecise match via confidence scores
+
+**Metadata:** `{ "stage": "seed", "centroid": [...], "session_count": 247, "created": "2026-06-15", "growth_rate": "12/day", "origin": "data_driven" }`
+
+### SPROUT — first training
+
+The cluster hits the training threshold: 500 sessions for skill leaves, 300 for reasoning leaves.
+
+- First LoRA leaf is trained using the SFT pipeline (Stage 1 only — not enough preference pairs for DPO yet)
+- Router activates the leaf with a confidence gate: if routing confidence < 0.3, fall back to parent branch
+- Evaluation runs against parent leaf — if the sprout does not outperform the parent on domain-specific prompts, it stays in sprout phase and collects more data
+- The sprout must prove itself before earning full routing trust
+
+**Metadata added:** `{ "stage": "sprout", "first_trained": "2026-07-01", "eval_vs_parent": "+8% keyword density", "confidence_gate": 0.3, "training_method": "sft_only" }`
+
+### MATURE — production quality
+
+The leaf has been trained on 500+ sessions AND retrained at least once with fresh data. Evaluation confirms it outperforms the parent leaf on its domain.
+
+- DPO stage applied — enough TIER_A vs TIER_C pairs now exist for preference optimization
+- Confidence gate removed — full routing trust
+- Leaf enters the gossip propagation pool — other nodes in the P2P mesh can request it
+- Retraining is triggered periodically as new sessions accumulate (every 500 new sessions or monthly, whichever comes first)
+
+**Metadata added:** `{ "stage": "mature", "maturity_date": "2026-08-15", "retrain_count": 3, "training_method": "sft+dpo", "gossip_propagation_count": 47 }`
+
+### PRUNE — dormant or obsolete
+
+A leaf stops earning its place. Triggers:
+
+- Not routed to in 30+ consecutive days
+- New sessions for this cluster dropped below 10/month
+- Evaluation on retrain shows regression vs previous version
+
+The leaf is **archived**, not deleted. Its centroid is removed from the active router. Its accumulated sessions get absorbed back into the parent cluster's training pool. If the domain revives — new sessions start clustering in that region again — the archived leaf can be restored and retrained rather than starting from scratch.
+
+### SPLIT — specialization
+
+A mature leaf's session embeddings, when re-clustered internally, reveal 2+ distinct sub-clusters with `min_cluster_size=100` each.
+
+Example: the "Python" leaf splits into "Python backend/API," "Python data processing," and "Python testing." Each child gets its own training run on its sub-cluster's sessions. The parent leaf remains as a fallback. Child leaf centroids are checked first during routing; if confidence is low, the router falls back to the parent.
+
+### MERGE — consolidation
+
+Two adjacent leaves have centroid cosine similarity > 0.92, AND neither has enough differentiated training data to justify separate existence (combined sessions < 800).
+
+Example: "Express.js middleware" and "Node.js API development" merge into a unified "Node.js backend" leaf. Sessions from both clusters combine into one training set. Both old centroids are replaced by the merged centroid.
+
+---
+
+## 6. The Lifecycle Engine
+
+A periodic process — daily by default, or triggered when new session count exceeds a threshold — that manages the entire tree.
+
+```
+┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
+│  EMBED   │───▶│ CLUSTER  │───▶│  DETECT  │───▶│  TRAIN   │───▶│ EVALUATE │───▶│  DEPLOY  │
+│          │    │          │    │          │    │          │    │          │    │          │
+│ Encode   │    │ HDBSCAN  │    │ Compare  │    │ SFT/DPO  │    │ Bench vs │    │ Update   │
+│ new      │    │ on full  │    │ clusters │    │ for new/ │    │ parent & │    │ router,  │
+│ sessions │    │ embedding│    │ against  │    │ changed  │    │ previous │    │ push S3, │
+│          │    │ space    │    │ known    │    │ leaves   │    │ version  │    │ notify   │
+│          │    │          │    │ leaves   │    │          │    │          │    │ mesh     │
+└──────────┘    └──────────┘    └──────────┘    └──────────┘    └──────────┘    └──────────┘
+```
+
+**Step 1 — EMBED:** Encode any new sessions that do not have embeddings yet. The same all-MiniLM-L6-v2 encoder the router uses, producing 384-dim vectors. At 11ms per embedding, 1000 new sessions take ~11 seconds.
+
+**Step 2 — CLUSTER:** Run HDBSCAN on the full embedding space (or incremental update at scale). Output: cluster labels, centroids, hierarchy, noise points.
+
+**Step 3 — DETECT:** Compare current clusters against known leaves.
+
+| Condition | Action |
+|-----------|--------|
+| New cluster, no matching leaf | Create SEED |
+| Existing SEED hit training threshold | Promote to SPROUT, trigger training |
+| Existing SPROUT retrained + outperforms parent | Promote to MATURE |
+| Existing MATURE not routed to in 30 days | PRUNE |
+| Existing MATURE showing internal sub-clusters | Evaluate SPLIT |
+| Two leaves with cosine similarity > 0.92 | Evaluate MERGE |
+
+**Step 4 — TRAIN:** Run SFT (and DPO if applicable) for any leaves that need training or retraining. Uses the same five-step pipeline from the AWS training plan, but triggered automatically rather than manually.
+
+**Step 5 — EVALUATE:** Benchmark new/retrained leaves against the parent leaf and the previous version. A leaf that regresses stays at its current stage or gets pruned.
+
+**Step 6 — DEPLOY:** Update router centroids, push new leaf bundles to the S3 registry, notify the P2P mesh via gossip.
+
+This is the "gossip state" from the vision document: "During low-utilization periods (overnight), the network enters a gossip state. Nodes share localized error rates and successful tool-call trajectories. The network algorithmically prunes dormant leaves and propagates highly successful LoRA weights across the mesh."
+
+The lifecycle engine IS the gossip protocol's brain.
+
+---
+
+## 7. Telemetry Changes
+
+The current telemetry pipeline captures envelope records with trajectory logs and SESSION_CLOSE outcomes. Dynamic Leaf Architecture requires two additions.
+
+### Add: Session Embedding
+
+Computed at collection time. 11ms average, negligible overhead.
+
+```json
+"session_embedding": {
+  "model": "all-MiniLM-L6-v2",
+  "vector": [0.0234, -0.0891, 0.1247, 0.0562, "... 384 dimensions"],
+  "source_text": "Write a Python decorator that caches function results with TTL expiry"
+}
+```
+
+The embedding is computed from the first 512 characters of the task prompt. The clustering layer derives domains from embeddings — the collection layer does not need to classify anything.
+
+### Add: Routing Feedback
+
+Added to SESSION_CLOSE records. Creates the feedback loop: we know which leaf handled each session and can measure per-leaf quality.
+
+```json
+"routing": {
+  "leaf_id": "python_backend_v3",
+  "leaf_lifecycle_stage": "mature",
+  "routing_confidence": 0.42,
+  "fallback_used": false,
+  "parent_leaf_id": "python_v2"
+}
+```
+
+### Remove: Fixed Domain Tags
+
+The AWS training plan references domain tagging with fixed classifications. This is replaced by embedding-based clustering. The telemetry pipeline no longer needs to assign `domain_tags` at collection time. Domains are discovered downstream by the lifecycle engine.
+
+---
+
+## 8. Cold Start Strategy
+
+Day one, there is no data to cluster. The tree needs starter leaves.
+
+**Option A — Synthetic seeding:** Use an LLM to generate 100 diverse prompts per broad area. Embed them to create initial cluster seeds. Train lightweight starter leaves on synthetic data. As real data flows in, starters get retrained and eventually replaced.
+
+**Option B — Pre-computed centroids (current PoC approach):** Use the 10 domain descriptions from the PoC as initial centroids. No trained leaves yet — route to chassis + system prompt. As data accumulates per centroid region, leaves spawn naturally.
+
+**Option C — Hybrid (recommended):** Start with Option B's pre-computed centroids for immediate routing. The system works from day one using the PoC's proven centroids. Run HDBSCAN after 1000 total sessions. Replace pre-computed centroids with data-driven clusters. From that point forward, the tree is fully emergent.
+
+The starter centroids are scaffolding. They are explicitly temporary. The real tree grows from real data.
+
+```json
+{
+  "leaf_id": "python_starter",
+  "origin": "synthetic",
+  "stage": "sprout",
+  "note": "Pre-computed centroid from PoC. Will be replaced by data-driven cluster."
+}
+```
+
+vs.
+
+```json
+{
+  "leaf_id": "python_backend_cluster_47",
+  "origin": "data_driven",
+  "stage": "mature",
+  "session_count": 4200,
+  "cluster_id": 47
+}
+```
+
+The `origin` field lets the system know which leaves are scaffolding and which grew from real usage. Once enough data-driven leaves exist, all synthetic-origin leaves are deprecated.
+
+---
+
+## 9. Scaling Considerations
+
+| Session Count | Expected Clusters | HDBSCAN Time | Recommended Approach |
+|---------------|-------------------|--------------|---------------------|
+| 1,000 | 5-10 | < 1 second | Full re-cluster on every engine run |
+| 10,000 | 15-30 | 2-5 seconds | Full re-cluster daily |
+| 100,000 | 50-100 | 1-3 minutes | Full re-cluster weekly, incremental daily |
+| 1,000,000 | 200-500 | 10-30 minutes | Full re-cluster monthly, FAISS approximate nearest neighbor for daily assignment |
+
+At 10,000 sessions, HDBSCAN on 384-dimensional vectors takes seconds. The lifecycle engine runs comfortably as a daily cron job on the existing g4dn.xlarge training instance.
+
+At 100,000 sessions, full HDBSCAN still completes in minutes. The engine switches to an incremental model: new sessions are assigned to the nearest existing centroid daily (simple cosine similarity, sub-second), and the full HDBSCAN re-cluster runs weekly to detect new clusters, splits, and merges.
+
+At 1,000,000 sessions — the scale implied by the vision document's "1,000 node" network — full HDBSCAN becomes expensive. The system switches to FAISS (Facebook AI Similarity Search) for approximate nearest neighbor assignment on daily runs, with full HDBSCAN monthly to maintain cluster accuracy. The tree at this scale has 200-500 leaves across 3 levels of hierarchy. This is a self-organizing intelligence network.
+
+---
+
+## 10. What This Means for the Training Pipeline
+
+The AWS training plan describes a manual pipeline: human defines domains, human curates training data, human triggers training, human evaluates and deploys. Dynamic Leaf Architecture automates this.
+
+| Step | Before (fixed domains) | After (dynamic leaves) |
+|------|----------------------|----------------------|
+| Domain definition | Human writes description | Lifecycle engine discovers from embedding clusters |
+| Training data curation | Human tags and filters sessions | Cluster membership defines the dataset automatically |
+| Training trigger | Human runs scripts | Automated when cluster hits threshold |
+| Evaluation | Human reviews benchmark output | Automated: benchmark against parent leaf |
+| Deployment | Human uploads to S3 | Automated: push to S3, update router, notify mesh |
+| Human role | Operate everything | Review lifecycle dashboard, approve splits/merges, set global thresholds |
+
+**New infrastructure required:**
+
+- **Embedding store:** An indexed structure (S3 with manifest or a lightweight vector database like ChromaDB) for session embeddings, queryable by cluster assignment
+- **Clustering service:** Periodic HDBSCAN job — runs on the training GPU instance alongside training jobs, or as a CPU-only job since clustering does not require a GPU
+- **Lifecycle engine:** Orchestrator that manages seed/sprout/mature/prune/split/merge state transitions and triggers the training pipeline
+- **Dashboard:** Visibility into the tree — how many leaves at each stage, which are growing, which are being pruned, routing confidence distributions, cluster evolution over time
+
+The five training scripts (`01_setup_and_preprocess.py` through `05_package_leaf.py`) remain unchanged. What changes is how they get invoked: the lifecycle engine calls them instead of a human. The preprocessing step pulls from cluster-assigned sessions instead of domain-tagged sessions.
+
+---
+
+## 11. The Tree Visualization
+
+### Early Stage — 1,000 sessions, ~2 months in
+
+```
+Hummingbird Tree
+├── programming (starter centroid)
+│   ├── python_general [SPROUT, 280 sessions, confidence gate: 0.3]
+│   └── javascript_web [SEED, 150 sessions, waiting for 500]
+├── operations (starter centroid)
+│   └── docker_k8s [SEED, 90 sessions]
+├── research (starter centroid, no children yet)
+└── 340 unassigned sessions (noise — waiting for clusters to form)
+```
+
+Most sessions are still routing to starter centroids. The first data-driven leaves are appearing as seeds and sprouts. The tree is mostly scaffolding.
+
+### Growth Stage — 50,000 sessions, ~6 months in
+
+```
+Hummingbird Tree
+├── Backend Programming
+│   ├── Python Backend [MATURE, 4,200 sessions]
+│   │   ├── Python Async/API [SPROUT, 800 sessions]
+│   │   └── Python Data Processing [SPROUT, 650 sessions]
+│   ├── Rust Systems [MATURE, 1,800 sessions]
+│   └── Go Services [SEED, 280 sessions]
+├── Frontend Development
+│   ├── React/Next.js [MATURE, 3,500 sessions]
+│   ├── Vue/Nuxt [SPROUT, 420 sessions]
+│   └── CSS/Design Systems [SEED, 190 sessions]
+├── Data & ML
+│   ├── ML Training Pipelines [MATURE, 2,100 sessions]
+│   ├── Data Engineering [SPROUT, 680 sessions]
+│   └── LLM/Prompt Engineering [MATURE, 1,900 sessions]
+├── Infrastructure
+│   ├── Kubernetes [MATURE, 2,800 sessions]
+│   ├── AWS/Cloud [MATURE, 1,500 sessions]
+│   └── CI/CD Pipelines [SPROUT, 550 sessions]
+├── Research & Analysis
+│   ├── Technical Research [MATURE, 1,200 sessions]
+│   ├── Strategy/Architecture [SPROUT, 400 sessions]
+│   └── Scientific Research [SEED, 220 sessions]
+├── Specialized Domains
+│   ├── Bioinformatics [SEED, 180 sessions]
+│   ├── Legal/Compliance [SEED, 150 sessions]
+│   └── Finance/Trading [SEED, 210 sessions]
+└── 8,200 unassigned/noise sessions
+```
+
+This tree was not designed. It grew from what 50,000 real users actually did. The "Specialized Domains" branch appeared because enough bioinformatics researchers, legal analysts, and finance developers used the network to form distinct embedding clusters. Nobody decided those should exist. The data decided.
+
+The starter centroids are gone. Every leaf is data-driven. The tree has three levels of hierarchy where data density supports it (Python splitting into async and data processing), and stays flat where it does not (Go Services still a seed).
+
+### Scale Stage — 500,000 sessions, ~18 months in
+
+The tree has 200+ leaves across 3 levels. Mature leaves are being propagated across the P2P mesh via gossip. Dormant leaves (old framework-specific leaves that lost users) are being pruned. Popular domains are splitting into ever-finer specializations. The tree is a living map of what the network's users actually do.
+
+---
+
+## 12. Build Plan — Phased Implementation
+
+### Phase 1: Embedding Collection (Week 1-2)
+
+**Goal:** Start collecting the raw material for clustering without changing any existing behavior.
+
+- Add `session_embedding` field to the telemetry pipeline
+- Embed every session at collection time using all-MiniLM-L6-v2 (11ms overhead)
+- Store embeddings alongside JSONL telemetry in S3
+- Add `routing` feedback field to SESSION_CLOSE records
+- Keep existing fixed centroids for routing — fully backward compatible
+
+**Dependencies:** None. This is purely additive.
+**Effort:** ~3 days of telemetry pipeline work.
+
+### Phase 2: Clustering Pipeline (Week 3-4)
+
+**Goal:** Prove that meaningful clusters emerge from real session embeddings.
+
+- Install HDBSCAN on the training instance (`pip install hdbscan`)
+- Build a clustering script that loads embeddings from S3, runs HDBSCAN, and outputs cluster definitions (centroid, member session IDs, hierarchy level)
+- Build a visualization script using UMAP dimensionality reduction for 2D cluster plots
+- Run on accumulated embeddings and validate that clusters align with known domains
+- Store cluster assignments and centroids in a structured manifest
+
+**Dependencies:** Phase 1 embeddings must be accumulating.
+**Effort:** ~5 days. Can start as soon as Phase 1 has 1000+ embedded sessions.
+
+### Phase 3: Lifecycle Engine (Week 5-8)
+
+**Goal:** Automate the seed/sprout/mature/prune/split/merge state machine.
+
+- Build the lifecycle engine as a Python orchestrator script
+- Implement stage detection logic (new cluster? threshold hit? dormant? sub-clusters?)
+- Integrate with training pipeline — auto-trigger `01_setup_and_preprocess.py` through `05_package_leaf.py` when a sprout threshold is reached
+- Integrate with router — auto-update centroids when leaves change
+- Integrate with S3 leaf registry — auto-deploy trained leaves
+- Build a lightweight CLI dashboard showing tree state, leaf stages, session counts, and growth rates
+
+**Dependencies:** Phase 2 clustering pipeline.
+**Effort:** ~15 days. This is the core engineering work.
+
+### Phase 4: Transition from Fixed to Dynamic (Week 8-10)
+
+**Goal:** Replace the hardcoded domain taxonomy with data-driven clusters.
+
+- Run clustering on all accumulated data
+- Compare data-driven clusters against fixed domains — validate coverage
+- Replace fixed centroids with data-driven centroids in the router
+- Retrain leaves on cluster-derived training sets
+- Deprecate fixed domain taxonomy
+- Monitor routing accuracy during transition — rollback capability if accuracy drops
+
+**Dependencies:** Phase 3 lifecycle engine. Sufficient accumulated data (target: 5000+ embedded sessions).
+**Effort:** ~8 days. Includes careful validation and rollback planning.
+**Can overlap with:** Phase 3 final integration testing.
+
+### Phase 5: Gossip Integration (Week 10-12)
+
+**Goal:** Connect the lifecycle engine to the P2P mesh.
+
+- Lifecycle engine publishes leaf state changes to the gossip protocol
+- Mature leaves propagate to nodes that request them
+- Pruned leaves are removed from node caches
+- Split/merge events trigger mesh-wide centroid updates
+- Router centroid sync becomes automatic — nodes pull updated centroid manifests periodically
+
+**Dependencies:** Phase 4 must be stable. P2P mesh gossip protocol must be operational.
+**Effort:** ~8 days.
+
+### Parallel Tracks
+
+Phases 1-2 are sequential. Phases 3-4 overlap. Phase 5 can begin development in parallel with Phase 4 validation. Total timeline: approximately 12 weeks from start to full gossip integration.
+
+---
+
+## 13. Summary
+
+Fixed domains are training wheels. Dynamic Leaf Architecture is the real system.
+
+The embedding is the domain. HDBSCAN discovers structure in the embedding space. The lifecycle engine manages leaf evolution through seed, sprout, mature, prune, split, and merge stages. The tree grows from usage. More users produce more data. More data reveals more clusters. More clusters spawn more leaves. More leaves improve routing accuracy. Better routing produces better output. Better output attracts more users.
+
+This is the self-improving intelligence loop that makes Hummingbird fundamentally different from static models. The PoC proved the router works. The AWS plan proved the training pipeline works. This document closes the gap between a manually managed set of 16 fixed leaves and a self-organizing tree of hundreds.
+
+The tree is not designed. It is grown. Every session makes it smarter, whether or not a human is watching.