prometheusnn 0.1.0__tar.gz

prometheusnn-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,18 @@
+name: CI
+on: [push, pull_request]
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --extra dev
+      - run: uv run ruff check src/ tests/
+      - run: uv run ruff format --check src/ tests/
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --extra dev
+      - run: uv run python -m pytest tests/ -v --tb=short

prometheusnn-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,23 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+permissions:
+  id-token: write  # required for PyPI Trusted Publishers (OIDC)
+
+jobs:
+  build-and-publish:
+    name: Build and publish
+    runs-on: ubuntu-latest
+    environment: pypi
+
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - name: Build distribution
+        run: uv build
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

prometheusnn-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+.DS_Store
+/data/
+__pycache__/
+*.pyc
+.venv/
+.pytest_cache/
+*.egg-info/
+dist/
+build/
+.ruff_cache/

prometheusnn-0.1.0/CHANGELOG.md

@@ -0,0 +1,39 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-04-30
+
+First public release on PyPI. PrometheusNN is an MCP server that guides users
+from a CSV file to a trained hierarchical text classifier through a conversational
+interface — the LLM mediates, the user never writes Python.
+
+### Added
+- **Entry-point tools** for LLM-mediated onboarding:
+  - `start_here(file_path)` — profiles a CSV/JSON, detects problem type,
+    surfaces concerns *before* training, returns pre-filled `next_steps`
+    so the LLM can fire the right tool without guessing column names.
+  - `scope_problem(goal_description)` — assesses whether Prometheus fits
+    a goal that doesn't have data yet; politely redirects non-fits
+    (regression, clustering, NER) to appropriate alternatives.
+- **20 MCP tools** covering the full workflow: intake, training, inference,
+  model management, evaluation, threshold tuning, feedback.
+- **Hierarchical cascade classifier** with parent-noise injection, beam search
+  with adaptive widening, fork detection, temperature-calibrated confidence,
+  and dual-signal novelty detection.
+- **Three-tier routing** (AUTO_ACCEPT / HUMAN_REVIEW / OUT_OF_TAXONOMY)
+  with per-level threshold tuning.
+- Typed configuration objects via Pydantic v2 throughout — `RouterConfig`,
+  `TrainingConfig`, `TaxonomyInfo`, `ColumnProfile`, `ProblemAssessment`, etc.
+- LRU-bounded predictor and embedding caches with mtime-based invalidation
+  so stale models don't return stale predictions.
+
+### Notes
+- Requires Python 3.11+.
+- Heavy ML dependencies (`torch`, `sentence-transformers`) are required —
+  the package only does meaningful work with them.
+- Models are saved to `~/.prometheus/models/` as JSON manifests + `.pt`
+  state dicts. No pickle.

prometheusnn-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Fahad Baig
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

prometheusnn-0.1.0/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.4
+Name: prometheusnn
+Version: 0.1.0
+Summary: Frictionless hierarchical text classification — an MCP server that guides you from CSV to trained classifier
+Project-URL: Homepage, https://github.com/mfbaig35r/prometheusnn
+Project-URL: Repository, https://github.com/mfbaig35r/prometheusnn
+Author-email: Fahad Baig <mfbaig35r@gmail.com>
+License: MIT
+License-File: LICENSE
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: General
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: mcp[cli]>=1.2.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: sentence-transformers>=2.2
+Requires-Dist: torch>=2.0
+Provides-Extra: dev
+Requires-Dist: pytest>=9.0; extra == 'dev'
+Requires-Dist: ruff>=0.9; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# PrometheusNN
+
+Build a hierarchical text classifier without writing code. Point Claude (or any MCP client) at your data and it walks you from "I have a CSV" to a trained model — profiling your columns, detecting whether the problem fits, surfacing concerns before training, and recommending the right setup.
+
+PrometheusNN is the ML expert sitting next to you. You bring the data and the goal; it handles the rest.
+
+## What this looks like
+
+```
+You:    I have a CSV of product descriptions and I want to categorize them.
+        Can you help?
+
+Claude: [calls start_here on the file]
+
+        I looked at your data — 11,400 product descriptions across three
+        nested category levels (Chapter → Heading → Subheading). One thing
+        worth flagging before we train: the deepest level has ~2 examples
+        per class on average, which usually means poor accuracy at that
+        depth. Want to do all three levels, or simplify to the top two?
+
+You:    Top two.
+
+Claude: [fires analyze_training_data → train_model with the simpler setup]
+
+        Done. Chapter accuracy 92%, Heading accuracy 78%. Want to try
+        classifying something?
+```
+
+No column-name guessing, no parameter cargo-culting, no premature training runs.
+
+## Install
+
+```bash
+pip install prometheusnn
+```
+
+Then add to your MCP client config (Claude Desktop, Cursor, Zed, etc.):
+
+```json
+{
+  "mcpServers": {
+    "prometheus": {
+      "command": "prometheusnn"
+    }
+  }
+}
+```
+
+That's it. Open your MCP client and start a conversation about classifying something.
+
+## How it works under the hood
+
+Two entry-point tools handle every new conversation:
+
+- **`start_here(file_path)`** — profiles your CSV, detects whether it's a hierarchical, flat, or multi-label classification problem, surfaces concerns before training, and returns pre-filled next steps so the LLM never has to guess column names.
+
+- **`scope_problem(goal_description)`** — for when you don't have data yet. Tells you whether Prometheus fits your goal, what data you'll need, and what to watch for. Politely redirects non-fits (regression, clustering, NER) to the right tools.
+
+From there, the LLM routes through the existing tool surface (training, evaluation, threshold tuning, prediction) based on what `start_here` recommended.
+
+## What's actually being built
+
+Under the conversational surface, PrometheusNN trains a cascade of neural network classifiers — one per level of your taxonomy — with parent-noise injection so deeper levels stay robust when upper levels are uncertain. Beam search explores alternative paths when confidence drops. Temperature calibration produces probabilities you can trust for routing. Dual-signal novelty detection catches items that don't belong.
+
+```
+Text descriptions + taxonomy labels
+        │
+        ▼
+  ┌─────────────┐
+  │  Embedding  │  sentence-transformers (384d or 768d)
+  └──────┬──────┘
+         ▼
+  ┌─────────────┐
+  │   Cascade   │  N classifiers, one per level
+  │  Classifier │  parent noise injection for robustness
+  └──────┬──────┘
+         ▼
+  ┌─────────────┐
+  │ Beam Search │  adaptive widening when confidence drops
+  └──────┬──────┘
+         ▼
+  ┌─────────────┐
+  │   Router    │  calibrated thresholds → accept / review / reject
+  │  + Novelty  │  centroid z-score + kNN distance
+  └─────────────┘
+```
+
+You don't need to know any of this to use it — but it's there if you want to dig in.
+
+## Tool reference
+
+20 MCP tools total. The two you'll be told about; the rest the LLM picks for you.
+
+**Entry points** (start here)
+- `start_here` — assess a file, detect problem type, return pre-filled next steps
+- `scope_problem` — assess a goal description (no data yet), check fit and data requirements
+
+**Training & data**
+- `analyze_training_data`, `train_model`, `resume_training`
+
+**Inference**
+- `predict`, `predict_batch`, `classify_with_context`
+
+**Model management**
+- `list_models`, `describe_model`, `delete_model`, `export_model`
+
+**Evaluation & tuning**
+- `evaluate_model`, `explain_prediction`, `get_confusion_matrix`
+- `get_threshold_report`, `set_thresholds`
+
+**Other**
+- `submit_feedback`, `list_embedding_models`, `build_code_mapping`
+
+## Environment variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `PROMETHEUS_HOME` | `~/.prometheus` | Base directory for models, logs, feedback |
+| `PROMETHEUS_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Default sentence-transformer model |
+
+## Development
+
+```bash
+git clone https://github.com/mfbaig35r/prometheusnn
+cd prometheusnn
+uv sync --extra dev
+uv run python -m pytest tests/ -v --tb=short    # 179 tests
+uv run ruff check src/ tests/                   # lint
+uv run ruff format --check src/ tests/          # format
+```
+
+## License
+
+MIT

prometheusnn-0.1.0/README.md

@@ -0,0 +1,134 @@
+# PrometheusNN
+
+Build a hierarchical text classifier without writing code. Point Claude (or any MCP client) at your data and it walks you from "I have a CSV" to a trained model — profiling your columns, detecting whether the problem fits, surfacing concerns before training, and recommending the right setup.
+
+PrometheusNN is the ML expert sitting next to you. You bring the data and the goal; it handles the rest.
+
+## What this looks like
+
+```
+You:    I have a CSV of product descriptions and I want to categorize them.
+        Can you help?
+
+Claude: [calls start_here on the file]
+
+        I looked at your data — 11,400 product descriptions across three
+        nested category levels (Chapter → Heading → Subheading). One thing
+        worth flagging before we train: the deepest level has ~2 examples
+        per class on average, which usually means poor accuracy at that
+        depth. Want to do all three levels, or simplify to the top two?
+
+You:    Top two.
+
+Claude: [fires analyze_training_data → train_model with the simpler setup]
+
+        Done. Chapter accuracy 92%, Heading accuracy 78%. Want to try
+        classifying something?
+```
+
+No column-name guessing, no parameter cargo-culting, no premature training runs.
+
+## Install
+
+```bash
+pip install prometheusnn
+```
+
+Then add to your MCP client config (Claude Desktop, Cursor, Zed, etc.):
+
+```json
+{
+  "mcpServers": {
+    "prometheus": {
+      "command": "prometheusnn"
+    }
+  }
+}
+```
+
+That's it. Open your MCP client and start a conversation about classifying something.
+
+## How it works under the hood
+
+Two entry-point tools handle every new conversation:
+
+- **`start_here(file_path)`** — profiles your CSV, detects whether it's a hierarchical, flat, or multi-label classification problem, surfaces concerns before training, and returns pre-filled next steps so the LLM never has to guess column names.
+
+- **`scope_problem(goal_description)`** — for when you don't have data yet. Tells you whether Prometheus fits your goal, what data you'll need, and what to watch for. Politely redirects non-fits (regression, clustering, NER) to the right tools.
+
+From there, the LLM routes through the existing tool surface (training, evaluation, threshold tuning, prediction) based on what `start_here` recommended.
+
+## What's actually being built
+
+Under the conversational surface, PrometheusNN trains a cascade of neural network classifiers — one per level of your taxonomy — with parent-noise injection so deeper levels stay robust when upper levels are uncertain. Beam search explores alternative paths when confidence drops. Temperature calibration produces probabilities you can trust for routing. Dual-signal novelty detection catches items that don't belong.
+
+```
+Text descriptions + taxonomy labels
+        │
+        ▼
+  ┌─────────────┐
+  │  Embedding  │  sentence-transformers (384d or 768d)
+  └──────┬──────┘
+         ▼
+  ┌─────────────┐
+  │   Cascade   │  N classifiers, one per level
+  │  Classifier │  parent noise injection for robustness
+  └──────┬──────┘
+         ▼
+  ┌─────────────┐
+  │ Beam Search │  adaptive widening when confidence drops
+  └──────┬──────┘
+         ▼
+  ┌─────────────┐
+  │   Router    │  calibrated thresholds → accept / review / reject
+  │  + Novelty  │  centroid z-score + kNN distance
+  └─────────────┘
+```
+
+You don't need to know any of this to use it — but it's there if you want to dig in.
+
+## Tool reference
+
+20 MCP tools total. The two you'll be told about; the rest the LLM picks for you.
+
+**Entry points** (start here)
+- `start_here` — assess a file, detect problem type, return pre-filled next steps
+- `scope_problem` — assess a goal description (no data yet), check fit and data requirements
+
+**Training & data**
+- `analyze_training_data`, `train_model`, `resume_training`
+
+**Inference**
+- `predict`, `predict_batch`, `classify_with_context`
+
+**Model management**
+- `list_models`, `describe_model`, `delete_model`, `export_model`
+
+**Evaluation & tuning**
+- `evaluate_model`, `explain_prediction`, `get_confusion_matrix`
+- `get_threshold_report`, `set_thresholds`
+
+**Other**
+- `submit_feedback`, `list_embedding_models`, `build_code_mapping`
+
+## Environment variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `PROMETHEUS_HOME` | `~/.prometheus` | Base directory for models, logs, feedback |
+| `PROMETHEUS_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Default sentence-transformer model |
+
+## Development
+
+```bash
+git clone https://github.com/mfbaig35r/prometheusnn
+cd prometheusnn
+uv sync --extra dev
+uv run python -m pytest tests/ -v --tb=short    # 179 tests
+uv run ruff check src/ tests/                   # lint
+uv run ruff format --check src/ tests/          # format
+```
+
+## License
+
+MIT

prometheusnn-0.1.0/docs/beam-search-strategy.md

@@ -0,0 +1,230 @@
+# Beam Search Strategy
+
+This document describes the beam search inference pipeline in Prometheus, including adaptive beam widening, diversity-aware pruning, fork detection, and self-interpreting insights.
+
+## File Reference
+
+- Beam search: `src/prometheus/inference/beam_search.py`
+- Router & forks: `src/prometheus/inference/router.py`
+- Predictor: `src/prometheus/inference/predictor.py`
+- Insights: `src/prometheus/server.py` (`_analyze_forks`)
+- Tests: `tests/test_beam_search.py`
+
+## Overview
+
+Prometheus uses an N-level hierarchical cascade classifier. At inference time, beam search explores multiple classification paths through the taxonomy tree and returns the highest-scoring complete path along with alternatives.
+
+Two classifiers are available:
+- **GreedyClassifier** — picks the single best class at each level. Fast, no alternatives.
+- **BeamSearchClassifier** — maintains `beam_width` candidate paths. Slower, but surfaces competing hypotheses.
+
+## Standard Beam Search
+
+At each taxonomy level (depth 0 through N-1):
+
+1. For each existing beam, run the level classifier to get class probabilities.
+2. Expand each beam with the top-k children (k = `beam_width`).
+3. Score candidates by cumulative log-probability: `log_score = sum(log(p_i))` across all levels.
+4. Prune to `beam_width` beams by global score.
+
+This works well when the model is confident at every level. The problem arises when it isn't.
+
+## Problem: Score-Based Pruning Kills Minority Branches
+
+Consider classifying "plumbing and heating contractor" in a 4-level NAICS taxonomy:
+
+| Level | Top candidate | Confidence | Runner-up | Confidence |
+|-------|--------------|------------|-----------|------------|
+| Sector | Construction | 0.99 | — | — |
+| Subsector | Specialty Trade | 0.99 | — | — |
+| IndustryGroup | Building Finishing | 0.56 | Building Equipment | 0.19 |
+| NAICSIndustry | (varies) | ~0.17 | — | — |
+
+At IndustryGroup (depth 2), the model is genuinely uncertain — 0.56 vs 0.19. Building Equipment Contractors is the correct answer for plumbing/heating, but its log-score deficit from depth 2 compounds at depth 3. With pure score-based pruning, all final beams collapse into the Building Finishing branch. The correct path never appears in the output.
+
+## Solution: Adaptive Widening + Diversity-Aware Pruning
+
+Two mechanisms work together to fix this.
+
+### 1. Adaptive Beam Widening
+
+When `confidence_thresholds` are configured, the beam width dynamically increases at uncertain levels.
+
+```
+effective_width = beam_width
+if max_prob < confidence_thresholds[depth]:
+    effective_width = beam_width * adaptive_factor  # default: 2x
+```
+
+This happens in both `_init_beams()` (depth 0) and `_expand_beams()` (depth 1+). Widening creates more candidates at the uncertain level, giving minority branches a chance to enter the beam. But widening alone isn't enough — at the next level, the expanded candidates get pruned back down, and pure score ranking still kills the minority branches.
+
+### 2. Diversity-Aware Pruning
+
+When widening triggers, the system records the **first uncertain level** as `_diversity_depth`. This depth becomes a diversity anchor: during pruning at all subsequent levels, at least one representative from each unique class at the diversity depth is guaranteed to survive.
+
+#### Tracking
+
+```python
+# Reset per prediction
+self._diversity_depth = None
+
+# Set on first widening (in _init_beams or _expand_beams)
+if self._diversity_depth is None:
+    self._diversity_depth = depth
+```
+
+The `is None` guard ensures only the first uncertain level becomes the anchor. If multiple levels are uncertain, deeper diversity is handled implicitly — diverse branches at the anchor level naturally produce diverse candidates downstream.
+
+#### Pruning Algorithm (`_prune_with_diversity`)
+
+Input: candidate beams (pre-sorted by log_score descending), target width.
+
+**Phase 1 — Guarantee representation:**
+- Group beams by their class index at `_diversity_depth`.
+- Take the highest-scored beam from each group (one representative per branch).
+
+**Phase 2 — Fill by score:**
+- If representatives don't fill the width, add remaining beams in score order.
+- Re-sort the final list by log_score.
+
+```
+Beams at depth 3 (6 candidates, width=6):
+  BF→OtherFinishing    score=-2.32  ← best of BF group
+  BF→FinishCarpentry   score=-2.50
+  BF→Flooring          score=-2.51
+  BE→Electrical        score=-2.72  ← best of BE group (guaranteed slot)
+  BE→Plumbing          score=-3.10
+  BF→Painting          score=-3.20
+
+After diversity pruning (width=6): all 6 survive (no pruning needed)
+After diversity pruning (width=3): BF→OtherFinishing, BF→FinishCarpentry, BE→Electrical
+```
+
+Without diversity pruning at width=3, the result would be the top 3 BF paths. With it, the BE branch survives.
+
+#### Fallback
+
+When `_diversity_depth` is `None` (no thresholds configured, or all levels confident), `_prune_with_diversity` returns `beams[:width]` — identical to the previous pure-score behavior. This means:
+- Models without `confidence_thresholds` are completely unaffected.
+- Predictions where all levels are confident use plain score ranking.
+
+## Configuration
+
+Both mechanisms are controlled via `BeamSearchClassifier` constructor parameters:
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `beam_width` | 3 | Base number of paths to maintain |
+| `confidence_thresholds` | None | Dict mapping depth → minimum confidence. Levels below threshold trigger widening. |
+| `adaptive_factor` | 2 | Multiplier for beam width when widening triggers |
+
+Thresholds are auto-generated during training as linear decay from 0.90 (root) to 0.75 (leaf). They can be overridden via `set_thresholds`.
+
+## Edge Cases
+
+**Single-level taxonomies:** No beam expansion occurs. Diversity pruning is never triggered.
+
+**Out-of-taxonomy items:** Novelty detection runs independently. Diversity pruning may activate if thresholds are configured, but the routing decision is governed by the novelty detector, not the beam structure.
+
+**Greedy mode (beam_width=1):** Uses `GreedyClassifier`, which has no beam expansion, widening, or diversity logic.
+
+**Many competing classes at diversity depth:** If more unique classes exist at the anchor depth than the effective beam width, representatives are capped at `width` (taken in score order). This means the lowest-scored branches may still be dropped. In practice this is rare — uncertain levels typically have 2-4 competing classes, well within a width of 3-6.
+
+## Fork Detection
+
+After beam search and diversity pruning produce the final set of beams, the router (`ThresholdRouter.route_with_forks`) groups them into `forked_completions` — alternative paths presented to the caller.
+
+### How forks are detected
+
+The router identifies a **fork depth** — the level where the beams diverge:
+
+- **`confident_depth` exists** (e.g., NAICS: L1 and L2 are confident): `fork_depth = confident_depth + 1`. Forks are alternative completions below the confident prefix. All forks share the same upper levels (e.g., Construction → Specialty Trade) and diverge at the uncertain level (IndustryGroup).
+
+- **`confident_depth` is None** (e.g., MeSH: L1 itself is uncertain): `fork_depth = 0`. The entire path is uncertain, so forks are grouped by L1 class — one representative per top-level branch, each showing the best full path through that branch.
+
+### Why both modes matter
+
+NAICS "plumbing and heating contractor": L1 (Construction, 0.99) and L2 (Specialty Trade, 0.99) are confident. The question is which IndustryGroup, so forks show competing L3→L4 suffixes below the shared prefix.
+
+MeSH "rheumatoid arthritis": L1 itself is split (Musculoskeletal 0.43 vs Connective Tissue 0.14). There's no confident prefix. Forks show competing full paths — one through Musculoskeletal, one through Connective Tissue — because both are legitimate MeSH placements.
+
+### Format
+
+Each `ForkedCompletion` contains:
+- `suffix` — labels from `fork_depth` onward
+- `suffix_indices` — class indices from `fork_depth` onward
+- `suffix_score` — cumulative log-prob of the suffix portion
+- `per_level_probs` — probability at each suffix level
+
+Forks are sorted by `suffix_score` descending (best first).
+
+## Self-Interpreting Insights
+
+The `insights` field in the predict response is generated by `_analyze_forks()` in `server.py`. It runs three checks against the already-computed `forked_completions` — no additional model calls, no ML, just pattern matching on beam search output.
+
+### Insight 1: Minority branch with higher leaf confidence
+
+```
+"Alternative path via 'Skin and Connective Tissue Diseases' has higher leaf confidence
+ for 'Arthritis, Rheumatoid' (95% vs 61%)"
+```
+
+**What it detects:** A forked completion whose leaf-level probability exceeds the best path's leaf probability by more than 5 percentage points.
+
+**Why it matters:** In polyhierarchical taxonomies, the best path (highest cumulative score) often has a weak leaf prediction because its strength comes from a confident upper level. A minority branch with low L1 confidence can produce a much more confident leaf — this insight flags exactly that.
+
+**What a caller should do:** Present both the best path and the flagged alternative. The alternative's leaf prediction may be more reliable even though its overall path score is lower.
+
+### Insight 2: Branch convergence
+
+```
+"Multiple branches ('Musculoskeletal Diseases' and 'Skin and Connective Tissue Diseases')
+ converge on 'Arthritis, Rheumatoid'"
+```
+
+**What it detects:** Two or more forked completions that arrive at the same leaf label through different upper-level branches.
+
+**Why it matters:** Independent paths reaching the same conclusion is strong corroboration. If the Musculoskeletal branch and the Connective Tissue branch both end at "Arthritis, Rheumatoid", the model is confident in the leaf even though it's uncertain about which branch to use. This is the hallmark of polyhierarchical data — the item genuinely belongs in multiple places.
+
+**What a caller should do:** Treat the converged leaf as high-confidence. Optionally present both branches as valid placements.
+
+### Insight 3: Low best-path leaf confidence
+
+```
+"Best path leaf confidence is low (22%), review forked_completions for alternatives"
+```
+
+**What it detects:** The best path's leaf probability is below 30%, and neither of the above insights fired.
+
+**Why it matters:** Safety net. The model picked a path but isn't confident about where it ends. The forked completions may contain a better answer or the item may need human review.
+
+**What a caller should do:** Don't present the best path as authoritative. Show the top 2-3 forked completions and let the user decide.
+
+### When insights are absent
+
+No `insights` field means: the best path is the clear winner and no fork has a stronger leaf prediction. The caller can trust the best path.
+
+## Design Philosophy
+
+Each layer does one simple thing:
+
+| Layer | What it does | Complexity |
+|-------|-------------|-----------|
+| Beam search | Explore multiple paths | Expansion + log-score |
+| Diversity pruning | Don't let score kill minority branches | One dict lookup per beam |
+| Fork detection | Group paths by where they diverge | Prefix matching |
+| Insights | Compare leaf confidences across groups | Arithmetic |
+
+No layer "understands" polyhierarchy. Stacked together, they surface it — because they don't throw away information prematurely.
+
+## Tested On
+
+**NAICS** (4 levels, 20 sectors, clean hierarchy, 1,012 samples):
+- Diversity pruning keeps Building Equipment Contractors alive for "plumbing and heating contractor"
+- Forks show competing IndustryGroup completions below the confident Sector/Subsector prefix
+
+**MeSH Diseases** (4 levels, 22 L1 categories, 2,459 leaf classes, polyhierarchical, 10,984 samples):
+- Diversity pruning keeps minority L1 branches alive (Connective Tissue at 14% L1 confidence finds RA at 95% leaf confidence)
+- Fork detection at depth 0 surfaces all competing top-level branches
+- Convergence insight detects when two branches agree on the same leaf
+- Composite text ("Name: scope note") improved L4 accuracy by 2.3 points