probat-mcp 0.1.0__tar.gz

probat_mcp-0.1.0/.gitignore

@@ -0,0 +1,4 @@
+.env
+venv/
+**/__pycache__/
+chroma_data/

probat_mcp-0.1.0/DESIGN.md

@@ -0,0 +1,304 @@
+# Orpheus — Architecture
+
+A/B UI Variant Recommender. MCP server. Given a component + context, returns 3 visually distinct design variant bundles, each backed by corpus evidence and a testable hypothesis. Learns from selections, edits, and A/B results over time.
+
+---
+
+## End-to-End Flow
+
+```
+User: recommend_variants("hero section", "landing", "B2B SaaS")
+                │
+                ▼
+        ┌───────────────┐
+        │  1. EMBED     │  pipeline._embed(query)
+        │               │  → 1 OpenAI call, shared across all domain searches
+        └───────┬───────┘
+                │ embedding vector
+                ▼
+        ┌───────────────┐
+        │  2. RETRIEVE  │  query_with_embedding(embedding, domain_filter=X)
+        │  per domain   │  × 6 domains = 6 ChromaDB + BM25 searches, 0 extra API calls
+        │               │
+        │  style        │  top 8 candidates
+        │  color        │  top 8 candidates
+        │  typography   │  top 8 candidates
+        │  landing      │  top 8 candidates
+        │  ux           │  top 8 candidates
+        │  ui_reasoning │  top 1 candidate  ← rule lookup
+        └───────┬───────┘
+                │ 41 candidate docs total (with stored embeddings)
+                ▼
+        ┌───────────────────────────────────────────┐
+        │  3. COMPOSE — 3 bundles                   │
+        │                                           │
+        │  VARIANT 1 — ui_reasoning anchor          │
+        │    Parse rule doc → Style Priority,       │
+        │    Color Mood, Typography Mood             │
+        │    _select_best_match(candidates, kw)     │
+        │    +10 metadata match / +3 metadata /     │
+        │    +1 prose text                          │
+        │                                           │
+        │  VARIANT 2 — max embedding distance       │
+        │    per domain: argmax cosine_distance(    │
+        │      candidate, v1_doc)                   │
+        │    where doc_id not in used_doc_ids        │
+        │                                           │
+        │  VARIANT 3 — max distance from V1 + V2   │
+        │    per domain: argmax min(               │
+        │      dist(c, v1), dist(c, v2))            │
+        │    where doc_id not in used_doc_ids        │
+        │                                           │
+        │  used_doc_ids enforces: no doc appears    │
+        │  in more than one variant                 │
+        └───────┬───────────────────────────────────┘
+                │ 3 × {style, color, typography, landing, ux}
+                ▼
+        ┌───────────────┐
+        │  4. LABEL     │  assign_strategy_label(bundle)
+        │  (post-hoc)   │  cosine_sim(bundle_emb, strategy_profile_emb)
+        │               │  → "minimal" | "bold" | "layered"
+        │               │  → attach pre-written hypothesis
+        └───────┬───────┘
+                │
+                ▼
+        ┌───────────────┐
+        │  5. LTR RANK  │  LambdaRank scores each bundle
+        │  (post-MVP)   │  features: doc win rates, combo win rate,
+        │               │  context similarity
+        │               │  → re-orders V1/V2/V3 by predicted win prob
+        └───────┬───────┘
+                │
+                ▼
+        3 ComposedVariants returned to client
+        each with: strategy label, hypothesis,
+        5 domain docs, full attribution
+```
+
+---
+
+## Retrieval Pipeline Detail
+
+Each domain search runs the same 3-stage pipeline:
+
+```
+query_text + embedding
+    │
+    ├── BM25 (in-memory, rank_bm25)
+    │     tokenize → get_scores → top 20
+    │     filter to domain post-hoc (metadata check)
+    │
+    └── Dense (ChromaDB cosine similarity)
+          pre-computed embedding stored at ingest time
+          domain_filter passed as ChromaDB where clause
+          → top 20
+    │
+    └── RRF Fusion
+          score = 1/(60 + bm25_rank) + 1/(60 + dense_rank)
+          → sorted by RRF score
+    │
+    └── top N returned with stored embeddings attached
+          (batch ChromaDB .get() call — no extra API call)
+```
+
+**Why hybrid:** BM25 catches exact technical terms (`backdrop-filter`, `glassmorphism`). Dense catches semantic equivalents ("frosted glass" ↔ "glassmorphism"). They fail in opposite directions; RRF combines their rank lists without score normalization.
+
+**Why stored embeddings on results:** The composer needs them for cosine distance scoring. Fetched in one batch ChromaDB call after RRF, not per-doc.
+
+---
+
+## Corpus
+
+3,624 prose docs across 12 domains, embedded with `text-embedding-3-large` (3072 dims).
+
+| Domain | Docs | What it covers |
+|--------|------|----------------|
+| `style` | 86 | Visual styles: Glassmorphism, Minimalism, Brutalism, etc. |
+| `color` | 161 | Color palettes per product type |
+| `typography` | ~200 | Font pairings with mood/context metadata |
+| `landing` | 39 | Page layout patterns (Hero + CTA, etc.) |
+| `ux` | 99 | UX guidelines: navigation, forms, CTAs |
+| `ui_reasoning` | 161 | Expert rules: best style/color/typo per product type |
+| `app_interface` | ~200 | Web/app interface patterns |
+| `react_perf` | ~100 | React performance guidelines |
+| `charts` | ~50 | Data visualization patterns |
+| `icons` | ~200 | Icon usage guidelines |
+| `google_fonts` | ~900 | Font metadata |
+| `stack_*` | ~800 | Framework-specific (Next.js, Tailwind, SwiftUI, etc.) |
+
+Prose format (not raw CSV) because embedding models produce higher quality vectors on coherent language. Each doc is one CSV row serialized as natural prose.
+
+---
+
+## Composition Logic
+
+### Variant 1 — ui_reasoning anchor
+
+`ui_reasoning` domain contains 161 docs, one per product/page category. Finding the right rule is a retrieval problem — `query_with_embedding(domain_filter="ui_reasoning", n_results=1)` returns the semantically closest rule for the query.
+
+The rule doc's text is parsed with regex into fields:
+```
+{"Style Priority": "Glassmorphism + Flat Design",
+ "Color Mood": "Trust blue + Accent contrast",
+ "Typography Mood": "Professional + Hierarchy", ...}
+```
+
+Each domain maps to specific rule fields:
+- `style` → `Style Priority`
+- `color` → `Color Mood`
+- `typography` → `Typography Mood`
+- `landing` → `Recommended Pattern` + `Style Priority`
+- `ux` → `Key Effects` + `Style Priority`
+
+`_select_best_match` scores candidates against the extracted keywords:
+- +10 if keyword matches `style_category` metadata field
+- +3 if keyword matches any other metadata value
+- +1 if keyword appears anywhere in prose text
+
+Highest scorer per domain = Variant 1's doc for that domain.
+
+### Variant 2 — max embedding distance from V1
+
+For each domain, iterate remaining candidates (not in `used_doc_ids`), compute `1 - cosine_similarity(candidate.embedding, v1_doc.embedding)`, pick the max. Add to `used_doc_ids`.
+
+### Variant 3 — max distance from V1 and V2
+
+Same loop, but score = `min(dist(c, v1), dist(c, v2))`. Picks the candidate furthest from **both** prior choices — maximizes spread across the 3 variants.
+
+### Why this guarantees diversity
+
+`used_doc_ids` is a flat set shared across all domains and all variants. Once a doc is assigned to any variant in any domain, it cannot be reused. Combined with the embedding distance maximization, the three bundles are structurally prevented from converging.
+
+---
+
+## Feedback Loop
+
+Three signal types, in order of granularity and build priority:
+
+### 1. Selection events (bundle-level)
+User picks Variant 2 → pairwise labels: `V2 > V1`, `V2 > V3`, weight 0.3.
+Written to `selection_events` table. Used in nightly LTR retraining.
+
+### 2. Edit events (domain-level) ← most granular
+User edits a specific domain within a chosen variant.
+
+| Edit type | What happens | Signal weight |
+|-----------|-------------|---------------|
+| `swap_doc` | replaces corpus doc for one domain | negative on original, positive on replacement |
+| `freeform` | types own value, not in corpus | negative on original, corpus gap flagged |
+| `partial` | keeps doc, tweaks a parameter | soft negative at 0.3× |
+
+Freeform edits aggregate nightly: ≥3 identical edits across different users → corpus gap candidate for human review.
+
+### 3. A/B results (CVR-level)
+Experiment completes → winning bundle's doc_ids get positive labels weighted by `cvr_lift × test_confidence`. Staged: day 0 (60%), day 7 (25%), day 30 (15%).
+
+### Nightly update job
+Reads selection + edit + A/B signals, updates `global_win_rate` and `global_appearance_count` in ChromaDB metadata per doc. This makes retrieval adaptive before LTR is built — docs that appear in winning bundles float up through LTR scoring features.
+
+---
+
+## LTR (Learning-to-Rank)
+
+Built after ≥30 labeled examples. LambdaRank (LightGBM) on bundle-level features:
+
+```python
+{
+    "style_doc_win_rate":         float,   # from ChromaDB metadata, updated nightly
+    "color_doc_win_rate":         float,
+    "typo_doc_win_rate":          float,
+    "style_color_combo_win_rate": float,   # how often this style+color pairing wins
+    "strategy_label_win_rate":    float,   # global win rate for this label
+    "context_similarity":         float,   # cosine sim of query to past winning queries
+    "avg_doc_win_rate":           float,
+}
+```
+
+Per-customer model kicks in at ≥100 examples. Global model used until then.
+
+---
+
+## Data Model
+
+```python
+@dataclass
+class DesignVariant:
+    """Single retrieved doc from the corpus. Output of RetrievalPipeline."""
+    doc_id:     str
+    text:       str               # full prose document
+    metadata:   dict              # domain, style_category, global_win_rate, etc.
+    bm25_rank:  int | None
+    dense_rank: int | None
+    rrf_score:  float
+    mmr_score:  float
+    embedding:  list[float] | None  # populated by query_with_embedding
+
+@dataclass
+class ComposedVariant:
+    """Fully assembled design bundle. Output of compose_variants()."""
+    label:       str              # "minimal" | "bold" | "layered" — post-hoc
+    hypothesis:  str              # human-readable testable claim
+    docs:        dict[str, DesignVariant]  # domain → doc
+    source_doc_ids:        list[str]
+    selection_method:      str    # "ui_reasoning_anchor" | "max_distance_v2" | "max_distance_v3"
+    reasoning_rule_used:   dict   # parsed rule fields from ui_reasoning doc
+    retrieval_scores:      dict
+    ltr_feature_importance: dict
+    corpus_basis:          list[str]   # human-readable attribution lines
+    confidence:            float
+    brand_applied:         bool
+    auto_applied_attrs:    list[str]
+```
+
+---
+
+## File Map
+
+```
+server/
+├── retrieval.py                # RetrievalPipeline: BM25 + dense + RRF + MMR
+│                               # _embed(), query_with_embedding() ← Module 1
+├── retrieval/
+│   ├── __init__.py
+│   └── composer.py             # Module 2a: select_variant_1()
+│                               #   _parse_rule_doc(), _extract_keywords_from_rule()
+│                               #   _select_best_match()
+│                               # Module 2b: select_by_max_distance()
+│                               #   select_by_max_distance_from_multiple()
+│                               # Module 2c (next): compose_variants()
+├── mcp_server.py               # MCP tool: recommend_variants → calls compose_variants
+└── config.py                   # API keys, model names, tuning constants
+
+corpus/
+├── csv/                        # Raw CSVs (source of truth)
+├── prose/                      # Serialized prose docs (3,624 .txt files)
+└── ground_truth.csv            # Validation queries
+
+scripts/
+├── serialize_corpus.py         # CSV → prose
+├── ingest.py                   # Embed + load into ChromaDB, build BM25 index
+└── evaluate.py                 # Recall@5, MRR
+
+tests/
+├── test_composer_2a.py         # _parse_rule_doc, _select_best_match, select_variant_1
+└── test_composer_2b.py         # _cosine_distance, select_by_max_distance
+```
+
+---
+
+## Key Design Decisions
+
+**Retrieval over string matching for rule lookup**
+`ui_reasoning` docs are embedded like every other domain. Finding the right rule for "enterprise software checkout" uses BM25 + dense — not fragile substring matching on "UI_Category".
+
+**Post-hoc strategy labels, not retrieval constraints**
+Labels ("minimal", "bold", "layered") describe what was found. They don't constrain what gets retrieved. This prevents pigeonholing (e.g. "bold" being forced for fintech where it's wrong).
+
+**Shared embedding across all domain searches**
+One `_embed()` call per request. All 6 domain searches (5 content domains + ui_reasoning) reuse that embedding. Cost: ~$0.001 per request.
+
+**Uniqueness enforced by `used_doc_ids`**
+A flat set shared across domains and variants. The same doc cannot appear twice anywhere in the response, regardless of how similar its embedding is to multiple queries.
+
+**No LLM in composition**
+Design decisions (which style, which color, which layout) are deterministic. LLM is used downstream only to generate code/markup from the chosen docs. This keeps composition fast (<200ms), free, deterministic, and fully attributable.