glyphsteer 0.3.0__tar.gz

glyphsteer-0.3.0/PKG-INFO

@@ -0,0 +1,18 @@
+Metadata-Version: 2.4
+Name: glyphsteer
+Version: 0.3.0
+Summary: Dual-regime retrieval steering via compiled in-band glyph annotations
+Requires-Python: >=3.11
+Requires-Dist: click
+Requires-Dist: numpy
+Provides-Extra: dense
+Requires-Dist: fastembed; extra == "dense"
+Provides-Extra: st
+Requires-Dist: sentence-transformers; extra == "st"
+Provides-Extra: serve
+Requires-Dist: fastapi; extra == "serve"
+Requires-Dist: uvicorn; extra == "serve"
+Requires-Dist: fastembed; extra == "serve"
+Provides-Extra: grammar
+Requires-Dist: rulecatcher==0.3.0; extra == "grammar"
+Requires-Dist: chaincompiler==0.3.0; extra == "grammar"

glyphsteer-0.3.0/README.md

@@ -0,0 +1,130 @@
+# GlyphSteer
+
+**Dual-regime retrieval steering via compiled in-band glyph annotations.**
+
+An LLM annotation pass writes a *controlled glyph vocabulary* onto a corpus. Those
+markers **steer retrieval in two regimes at once** — lexically (a rare glyph-tag is a
+maximal-IDF exact-match *facet*) and densely (an emoji carries a *learned direction* that
+nudges the chunk's embedding) — and are then **stripped from the returned text**, so the
+reader/generator never sees them.
+
+> One artifact (the annotated chunk). Two indexes (BM25 + dense). Invisible at output.
+
+This is **HoneyC pointed at the index instead of Prolog** (a glyph code is a HoneyC
+chain), and the cousin of Anthropic's *Contextual Retrieval* — but with a **compiled,
+queryable, faceted controlled vocabulary** in place of free-text context.
+
+## The sidecar split (the keystone)
+
+```
+indexed-form  = clean_text ⊕ glyph_code     → embedded / FTS5-indexed  (MATCHING)
+returned-form = clean_text                   → handed to the reader     (READING)
+```
+
+The glyph lives only in the matching representation. `check_hidden` / `assert_hidden`
+make the hide a hard invariant.
+
+## One axis, two renderings (a verified finding)
+
+FTS5's `unicode61` tokenizer **drops emoji** (verified — it matches `0`). So each axis
+carries two markers:
+
+| | marker | regime | why |
+|---|---|---|---|
+| **glyph** | 😡 | dense | emoji has a learned direction, near-orthogonal to prose |
+| **tag** | `gsxnegative` | lexical | rare ASCII single-token ⇒ maximal IDF, exact facet |
+
+## What the vocabulary guarantees
+
+- **Matching.** A vocabulary may hold a glyph that is part of another — an emoji and its variation-selector or
+  skin-tone form (`⚠` and `⚠️`). Every reader — `parse`, `glyphs_in`, `code_tags`, `strip`, the hide check,
+  `to_honeyc` and the grammar's order check — reads left to right, at each place the longest glyph that starts there,
+  and never two that overlap: `⚠️` is one glyph, never `⚠` with a stray selector.
+- **The hide.** `strip` removes each glyph whole and the separator with its space, and goes on until a pass removes
+  nothing — removing one glyph can join its neighbours into another — so the returned form holds no glyph.
+- **Order.** `to_honeyc` writes the glyphs in the code's own order — a chain is order-significant.
+- **The gate.** `GlyphGrammar.lint` refuses (`syntax_break`) whatever in a code is neither a vocabulary glyph nor the
+  separator, and names it.
+- **Legends.** `merge` keeps the last author when two entries clash on a name, a glyph or the tag their names derive
+  (`risk` and `Risk!` both derive `gsxrisk`), and keeps every entry that clashes with nothing — a glyph a name gave
+  up, or an evicted entry held, is free again.
+- **Search.** A query with no word in it finds nothing.
+
+`tests/test_glyphsteer_regressions.py` pins each one.
+
+## Install & use
+
+```bash
+pip install -e .                # lexical regime, zero heavy deps
+pip install -e '.[dense]'       # adds fastembed (local ONNX, no torch) for the dense regime; '.[st]' adds sentence-transformers
+```
+
+```python
+from glyphsteer import SENTIMENT, Chunk, RuleAnnotator, annotate_chunk, build_index, search
+
+ann = RuleAnnotator.keyword(SENTIMENT, {
+    SENTIMENT.by_name("negative").glyph: ["failed", "broke", "terrible"],
+    SENTIMENT.by_name("positive").glyph: ["great", "excellent", "succeeded"],
+})
+chunks = [annotate_chunk(Chunk(c.id, c.text), ann, SENTIMENT) for c in corpus]
+con = build_index(chunks)
+hits = search(con, "deployment", facet=SENTIMENT.by_name("negative").glyph)
+# hits[i]["text"] is CLEAN — the glyph never leaks into the output
+```
+
+CLI: `glyphsteer vocab` · `glyphsteer annotate corpus.jsonl -k 😡=failed,broke` ·
+`glyphsteer query corpus.jsonl "deployment" -f 😡`
+
+## Flagship use: grade a corpus, search shows grades
+
+The vocabulary is general (`SENTIMENT` is just the demo). The built-in `GRADE` vocabulary
+(🏆 excellent / ✅ good / ⚠️ fair / ❌ poor) lets an LLM grade "what's good and what isn't";
+search then shows the grade and can facet to it. **Grading rides the lexical regime** — a
+grade is a label you filter/display by, not a semantic direction (`experiments/grade_demo.py`):
+
+```
+search 'database' — results WITH grades (best→worst):
+  [🏆 excellent ] Use parameterized queries to prevent SQL injection (OWASP A03).
+  [✅ good      ] Connection pooling improves DB throughput; benchmark before tuning.
+  [⚠️ fair     ] Indexes might help query speed, not totally sure when.
+  [❌ poor      ] If the database is slow just restart the server.
+facet = poor only (the audit view):
+  [❌ poor      ] You can concatenate user input into SQL strings, usually fine.
+```
+
+The grade glyph rides back in each hit's `code` for display; the body text stays clean.
+
+## The dense sidecar (containerized, no torch)
+
+The heavy neural runtime lives in a Docker image (`serve/`): **fastembed → onnxruntime**,
+model weights pre-cached, ~888MB, **no torch**. The host calls it over HTTP and stays
+dependency-light (the remote client is stdlib urllib). Swapping the embedding model is one
+env var — which is exactly what made the emoji-collapse finding below discoverable.
+
+```bash
+cd serve && docker compose up --build -d           # serves :8088
+export GLYPHSTEER_EMBED_URL=http://localhost:8088   # host now uses the sidecar
+python experiments/emoji_collapse_check.py          # gate: is the model emoji-blind?
+python experiments/magnitude_probe.py               # Q1–Q3
+```
+
+## Results (measured via the sidecar, 2026-06-16)
+
+- **Lexical Q4 (retrieval lift):** MRR **0.417 → 1.000** on the toy sentiment-faceted
+  corpus (`experiments/retrieval_lift.py`). Zero deps.
+- **THE dense finding — emoji-direction is model-dependent:**
+
+  | model | distinct-emoji cosine | Q1 separation | verdict |
+  |---|---|---|---|
+  | `bge-small-en-v1.5` (English) | **1.000** (collapse) | −0.01…+0.08 (noise) | emoji → `[UNK]`, **blind** |
+  | `paraphrase-multilingual-MiniLM-L12-v2` (XLM-R) | 0.77–0.97 | **+0.28…+0.58** | carries direction ✓ |
+
+- **With the byte-aware model:** Q1 (alignment) **passes strongly**; Q2 (per-token nudge)
+  is **modest** (+0.02…+0.09 Δcos); Q3 (anchor reshape) is a **controllable gain** (×2 at
+  weight 1) — so steering strength is a tuning knob, not a yes/no. See `implementation_plan.md`
+  §3.1 and `experiments/results/magnitude_probe.json`.
+
+## Layout
+See [`implementation_plan.md`](implementation_plan.md) (canonical) and
+[`.claude/rules/`](.claude/rules) for the component/flow diagrams and the discovered
+development workflows.

glyphsteer-0.3.0/pyproject.toml

@@ -0,0 +1,27 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "glyphsteer"
+version = "0.3.0"
+description = "Dual-regime retrieval steering via compiled in-band glyph annotations"
+requires-python = ">=3.11"
+dependencies = ["click", "numpy"]
+
+[project.optional-dependencies]
+# the light local ONNX backend (no torch); the container uses the same runtime
+dense = ["fastembed"]
+# the heavy local backend (pulls torch) — only if you want sentence-transformers
+st = ["sentence-transformers"]
+# build the embedding sidecar locally (normally you just `docker compose up`)
+serve = ["fastapi", "uvicorn", "fastembed"]
+# syntax gating: the grammar imports ChainCompiler's lint/gate engine (`chaincompiler.bridge`) and rulecatcher's
+# store inside its functions, so the extra brings both
+grammar = ["rulecatcher==0.3.0", "chaincompiler==0.3.0"]
+
+[project.scripts]
+glyphsteer = "glyphsteer.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

glyphsteer-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

glyphsteer-0.3.0/src/glyphsteer/__init__.py

@@ -0,0 +1,36 @@
+"""GlyphSteer — dual-regime retrieval steering via compiled in-band glyph annotations.
+
+A compiled controlled vocabulary of glyph markers, written onto a corpus by an LLM
+annotation pass, steers retrieval in BOTH regimes at once — lexical (a rare glyph is a
+maximal-IDF exact-match facet) and dense (a glyph carries a learned direction that
+nudges the chunk vector) — while being STRIPPED from the returned text (the sidecar
+split: indexed-form ≠ returned-form). HoneyC pointed at the index instead of Prolog.
+
+The dense layer (`dense`, `eval.magnitude_probe`) needs the optional `dense` extra;
+everything else (vocab, encode, the lexical index, the steer math) is dependency-light.
+"""
+from __future__ import annotations
+
+__version__ = "0.3.0"
+
+from .annotate import Annotation, Annotator, LLMAnnotator, RuleAnnotator
+from .encode import Chunk, annotate_chunk, check_hidden, indexed_form, returned_form
+from .index import assert_hidden, build_index, facet_only, search
+from .legend import (author, from_legend, load_legend, merge, render_legend,
+                     save_legend, to_legend)
+from .grammar import GlyphGrammar, GlyphLint
+from .steer import cosine, rerank, sim_eff, unit
+from .vocab import (GRADE, GRADE_RANK, SENTIMENT, SEP, Axis, Vocabulary,
+                    grade_label, is_lexically_clean)
+
+__all__ = [
+    "Axis", "Vocabulary", "SENTIMENT", "GRADE", "GRADE_RANK", "grade_label",
+    "SEP", "is_lexically_clean",
+    "Annotation", "Annotator", "RuleAnnotator", "LLMAnnotator",
+    "Chunk", "indexed_form", "returned_form", "annotate_chunk", "check_hidden",
+    "build_index", "search", "facet_only", "assert_hidden",
+    "to_legend", "from_legend", "author", "merge", "save_legend", "load_legend",
+    "render_legend", "GlyphGrammar", "GlyphLint",
+    "sim_eff", "cosine", "rerank", "unit",
+    "__version__",
+]

glyphsteer-0.3.0/src/glyphsteer/annotate.py

@@ -0,0 +1,73 @@
+"""The annotation pass — the LLM/judge half that writes the glyph sidecar.
+
+An `Annotator` reads a chunk's clean text and returns the axes that apply. Two
+implementations:
+
+  RuleAnnotator  — deterministic predicates (no LLM). Reproducible; the CI/test pass.
+  LLMAnnotator   — an injectable judge callable. The real research pass; the judge
+                   returns a glyph-code string which is parsed back THROUGH the
+                   vocabulary (unknown glyphs dropped — the vocab is the gate).
+
+The annotation pass over a whole corpus is a ChainCompiler skillchain (SC); these
+classes are the per-chunk step it runs.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Callable, Protocol
+
+from .vocab import Vocabulary
+
+
+@dataclass(frozen=True)
+class Annotation:
+    """One applied axis: which axis fired and the glyph it contributes."""
+    axis: str
+    glyph: str
+
+
+class Annotator(Protocol):
+    def annotate(self, text: str) -> list[Annotation]: ...
+
+
+class RuleAnnotator:
+    """Deterministic annotation by predicate. `rules: {glyph: text -> bool}`."""
+
+    def __init__(self, vocab: Vocabulary, rules: dict[str, Callable[[str], bool]]):
+        self.vocab = vocab
+        unknown = [g for g in rules if vocab.by_glyph(g) is None]
+        if unknown:
+            raise ValueError(f"rules reference glyphs not in the vocabulary: {unknown}")
+        self.rules = rules
+
+    def annotate(self, text: str) -> list[Annotation]:
+        out: list[Annotation] = []
+        for glyph, pred in self.rules.items():
+            if pred(text):
+                axis = self.vocab.by_glyph(glyph)
+                out.append(Annotation(axis.name, glyph))
+        return out
+
+    @classmethod
+    def keyword(cls, vocab: Vocabulary, keywords: dict[str, list[str]]) -> "RuleAnnotator":
+        """Convenience: `{glyph: [words]}` → case-insensitive substring rules."""
+        def make(words: list[str]) -> Callable[[str], bool]:
+            low = [w.lower() for w in words]
+            return lambda t, low=low: any(w in t.lower() for w in low)
+        return cls(vocab, {g: make(ws) for g, ws in keywords.items()})
+
+
+class LLMAnnotator:
+    """Annotation by an injected judge: `judge(text, vocab) -> glyph_code_str`."""
+
+    def __init__(self, vocab: Vocabulary, judge: Callable[[str, Vocabulary], str]):
+        self.vocab = vocab
+        self.judge = judge
+
+    def annotate(self, text: str) -> list[Annotation]:
+        code = self.judge(text, self.vocab)
+        out: list[Annotation] = []
+        for g in self.vocab.parse(code or ""):
+            axis = self.vocab.by_glyph(g)
+            out.append(Annotation(axis.name, g))
+        return out

glyphsteer-0.3.0/src/glyphsteer/cli.py

@@ -0,0 +1,69 @@
+"""GlyphSteer CLI — annotate / index / search / probe."""
+from __future__ import annotations
+
+import json
+import sys
+
+import click
+
+from .annotate import RuleAnnotator
+from .encode import Chunk, annotate_chunk, indexed_form, returned_form
+from .index import build_index, search
+from .vocab import SENTIMENT
+
+
+def _load_chunks(path: str) -> list[Chunk]:
+    """JSONL of {id, text} → chunks."""
+    out = []
+    for line in open(path, encoding="utf-8"):
+        line = line.strip()
+        if line:
+            d = json.loads(line)
+            out.append(Chunk(d["id"], d["text"], d.get("code", ""), d.get("meta", {})))
+    return out
+
+
+@click.group()
+def main() -> None:
+    """GlyphSteer — dual-regime retrieval steering via glyph annotations."""
+
+
+@main.command()
+@click.argument("corpus")
+@click.option("--keyword", "-k", multiple=True, metavar="GLYPH=word,word",
+              help="keyword rule, e.g. -k 😊=great,love")
+def annotate(corpus: str, keyword: tuple[str, ...]) -> None:
+    """Annotate a JSONL corpus with the sentiment vocabulary (keyword rules)."""
+    kw: dict[str, list[str]] = {}
+    for spec in keyword:
+        g, _, words = spec.partition("=")
+        kw[g] = [w for w in words.split(",") if w]
+    ann = RuleAnnotator.keyword(SENTIMENT, kw) if kw else RuleAnnotator(SENTIMENT, {})
+    for c in _load_chunks(corpus):
+        ac = annotate_chunk(c, ann, SENTIMENT)
+        click.echo(json.dumps({"id": ac.id, "text": ac.text, "code": ac.code,
+                               "indexed": indexed_form(ac),
+                               "returned": returned_form(ac, SENTIMENT)},
+                              ensure_ascii=False))
+
+
+@main.command()
+@click.argument("corpus")
+@click.argument("query")
+@click.option("--facet", "-f", default=None, help="glyph to facet on")
+def query(corpus: str, query: str, facet: str | None) -> None:
+    """Index an (already-annotated) JSONL corpus and search it; returns CLEAN text."""
+    con = build_index(_load_chunks(corpus))
+    for h in search(con, query, facet=facet):
+        click.echo(json.dumps(h, ensure_ascii=False))
+
+
+@main.command()
+def vocab() -> None:
+    """Print the built-in sentiment vocabulary."""
+    for a in SENTIMENT.axes:
+        click.echo(f"{a.glyph}  {a.name:10s}  {a.description}")
+
+
+if __name__ == "__main__":  # pragma: no cover
+    sys.exit(main())

glyphsteer-0.3.0/src/glyphsteer/dense.py

@@ -0,0 +1,122 @@
+"""The dense layer — embeddings + anchor measurement, via a pluggable backend.
+
+The heavy neural runtime is OFF the host by default: GlyphSteer talks to the
+`glyphsteer-serve` container (ONNX/fastembed, no torch) over HTTP. The host import
+stays dependency-light (the remote backend is stdlib-only urllib). Local backends
+exist as fallbacks.
+
+Backend resolution (override with `backend=`):
+  1. "remote"    — POST to $GLYPHSTEER_EMBED_URL/embed  (default if the env var is set)
+  2. "fastembed" — local ONNX runtime (the `dense` extra: `fastembed`)
+  3. "st"        — local sentence-transformers (the `st` extra: `sentence-transformers`, heavy)
+
+`measure_anchor(glyph)` returns the glyph's learned direction — the â_k that
+`steer.sim_eff` uses to reshape the metric, and what `eval.magnitude_probe` measures.
+"""
+from __future__ import annotations
+
+import json
+import os
+import urllib.request
+from functools import lru_cache
+
+import numpy as np
+
+_DEFAULT_MODEL = os.environ.get("GLYPHSTEER_MODEL", "BAAI/bge-small-en-v1.5")
+_ENV_URL = "GLYPHSTEER_EMBED_URL"
+
+
+def _default_backend() -> str:
+    if os.environ.get(_ENV_URL):
+        return "remote"
+    try:
+        import fastembed  # noqa: F401
+        return "fastembed"
+    except ImportError:
+        return "st"
+
+
+# ---- remote (the container; stdlib only) ------------------------------------
+def _remote_embed(texts: list[str], url: str | None = None) -> np.ndarray:
+    base = (url or os.environ[_ENV_URL]).rstrip("/")
+    payload = json.dumps({"texts": list(texts)}).encode("utf-8")
+    req = urllib.request.Request(base + "/embed", data=payload,
+                                 headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=120) as r:
+        out = json.load(r)
+    return np.asarray(out["vectors"], dtype=np.float32)
+
+
+# ---- fastembed (local ONNX) -------------------------------------------------
+@lru_cache(maxsize=4)
+def _fe(model: str):
+    try:
+        from fastembed import TextEmbedding
+    except ImportError as e:  # pragma: no cover
+        raise ImportError("backend 'fastembed' needs fastembed (the 'dense' extra): pip install fastembed") from e
+    return TextEmbedding(model)
+
+
+def _fastembed_embed(texts: list[str], model: str) -> np.ndarray:
+    return np.asarray(list(_fe(model).embed(list(texts))), dtype=np.float32)
+
+
+# ---- sentence-transformers (local, heavy) -----------------------------------
+@lru_cache(maxsize=4)
+def _st(model: str):
+    try:
+        from sentence_transformers import SentenceTransformer
+    except ImportError as e:  # pragma: no cover
+        raise ImportError("backend 'st' needs sentence-transformers (the 'st' extra): "
+                          "pip install sentence-transformers") from e
+    return SentenceTransformer(model)
+
+
+def _st_embed(texts: list[str], model: str) -> np.ndarray:
+    return np.asarray(_st(model).encode(list(texts), normalize_embeddings=False),
+                      dtype=np.float32)
+
+
+# ---- public API -------------------------------------------------------------
+def embed(texts: list[str], *, backend: str | None = None,
+          model: str = _DEFAULT_MODEL, url: str | None = None) -> np.ndarray:
+    """Embed texts → (n, dim) float32 array, via the resolved backend."""
+    backend = backend or _default_backend()
+    if backend == "remote":
+        return _remote_embed(texts, url)
+    if backend == "fastembed":
+        return _fastembed_embed(texts, model)
+    if backend == "st":
+        return _st_embed(texts, model)
+    raise ValueError(f"unknown embed backend {backend!r} (remote|fastembed|st)")
+
+
+def measure_anchor(glyph: str, **kw) -> np.ndarray:
+    """The glyph's learned direction = its embedding (the â_k for steer.sim_eff)."""
+    return embed([glyph], **kw)[0]
+
+
+def measure_anchors(glyphs: list[str], **kw) -> np.ndarray:
+    """Stack of anchor directions for a list of glyphs → (k, dim)."""
+    return embed(list(glyphs), **kw)
+
+
+def health(url: str | None = None) -> dict:
+    """Ping the remote embedding service."""
+    base = (url or os.environ[_ENV_URL]).rstrip("/")
+    with urllib.request.urlopen(base + "/health", timeout=10) as r:
+        return json.load(r)
+
+
+def active_model(model: str = _DEFAULT_MODEL, url: str | None = None) -> str:
+    """The model name actually serving embeddings, for honest reporting.
+
+    With the remote backend the sidecar decides the model (the local `model` arg is
+    ignored), so we ask it; otherwise the local model name is authoritative.
+    """
+    if _default_backend() == "remote":
+        try:
+            return health(url).get("model", model)
+        except Exception:
+            return model
+    return model

glyphsteer-0.3.0/src/glyphsteer/encode.py

@@ -0,0 +1,62 @@
+"""The sidecar split — the keystone of GlyphSteer.
+
+A chunk has TWO representations that are deliberately NOT the same string:
+
+  indexed-form  = clean_text ⊕ SEP ⊕ glyph_code     → embedded / FTS5-indexed (MATCH)
+  returned-form = clean_text                          → handed to the reader (READ)
+
+The glyph code lives ONLY in the indexed-form. It steers retrieval (lexical facet +
+dense direction) and then disappears: `returned_form` is provably glyph-free. This
+module owns that contract and the invariant that proves it.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from .vocab import SEP, Vocabulary
+
+
+@dataclass
+class Chunk:
+    """A unit of the corpus. `text` is ALWAYS clean; `code` is the glyph sidecar."""
+    id: str
+    text: str
+    code: str = ""
+    meta: dict = field(default_factory=dict)
+
+
+def indexed_form(chunk: Chunk) -> str:
+    """What gets embedded / FTS5-indexed: clean text + the glyph code."""
+    return f"{chunk.text}{SEP}{chunk.code}" if chunk.code else chunk.text
+
+
+def returned_form(chunk: Chunk, vocab: Vocabulary) -> str:
+    """What the reader/generator sees: clean text, guaranteed glyph-free."""
+    return vocab.strip(chunk.text)
+
+
+def annotate_chunk(chunk: Chunk, annotator, vocab: Vocabulary, grammar=None) -> Chunk:
+    """Run the annotation pass over a chunk and attach its glyph code.
+
+    The chunk's `text` is first stripped of any stray vocabulary glyphs so the
+    sidecar is the ONLY source of markers (keeps the hide-invariant total).
+
+    If a `grammar` (glyphsteer.grammar.GlyphGrammar) is given, the code is gated for
+    syntax consistency via rulecatcher: `syntax_break` raises; `orthogonal` is repaired
+    to canonical order. `grammar=None` ⇒ unchanged behaviour.
+    """
+    clean = vocab.strip(chunk.text)
+    glyphs = [a.glyph for a in annotator.annotate(clean)]
+    code = vocab.code(glyphs)
+    if grammar is not None:
+        lint = grammar.lint(code)
+        if lint.verdict == "syntax_break":
+            raise ValueError(f"syntax_break in annotated code {code!r}: {lint.violations}")
+        if lint.verdict == "orthogonal":
+            code = lint.fix
+    return Chunk(id=chunk.id, text=clean, code=code, meta=dict(chunk.meta))
+
+
+def check_hidden(chunk: Chunk, vocab: Vocabulary) -> bool:
+    """The hide invariant: no vocabulary glyph survives into the returned-form."""
+    return vocab.glyphs_in(returned_form(chunk, vocab)) == []

glyphsteer-0.3.0/src/glyphsteer/eval.py

@@ -0,0 +1,104 @@
+"""The evaluation harness — produces the numbers the claim lives or dies on.
+
+Q1 alignment   : cos(glyph, aligned_word) ≫ cos(glyph, unrelated_word)?
+Q2 magnitude   : injecting a glyph into a neutral chunk moves its vector by Δcos toward
+                 the axis — big enough to reorder, or a toy?
+Q3 projection  : does the steer.py anchor reshape amplify Q2?
+Q4 retrieval   : on labeled data, does annotate→steer beat the no-annotation baseline
+                 (MRR / recall@k), lexically and densely?
+
+The dense part (Q1–Q3) needs a dense backend — the sidecar, or the `dense` or `st` extra
+(`dense.py`); the lexical arm of Q4 runs with zero deps. ASPIRATIONAL: the dense arm of Q4.
+"""
+from __future__ import annotations
+
+import numpy as np
+
+from .encode import Chunk, annotate_chunk
+from .index import build_index, search
+from .steer import cosine, sim_eff
+from .vocab import Vocabulary
+
+
+# ---- retrieval metrics -------------------------------------------------------
+def mrr(ranked_ids: list, relevant: set) -> float:
+    for i, rid in enumerate(ranked_ids, 1):
+        if rid in relevant:
+            return 1.0 / i
+    return 0.0
+
+
+def recall_at_k(ranked_ids: list, relevant: set, k: int) -> float:
+    if not relevant:
+        return 0.0
+    hit = len(set(ranked_ids[:k]) & relevant)
+    return hit / len(relevant)
+
+
+# ---- Q1/Q2/Q3 : the dense probe ---------------------------------------------
+def magnitude_probe(vocab: Vocabulary, *, aligned: dict[str, str],
+                    unrelated: dict[str, str], neutral: str,
+                    model: str | None = None) -> dict:
+    """Answer Q1–Q3 for each glyph.
+
+    `aligned`/`unrelated`: {glyph: word}. `neutral`: a sentence to inject into.
+    Returns a per-glyph report with cos-alignment, the Δcos injection nudge, and the
+    projected (steered) nudge.
+    """
+    from . import dense  # lazy — needs the extra
+    # report the model ACTUALLY serving embeddings (the remote sidecar may differ from
+    # any local default), so the results JSON never carries a stale/wrong model label.
+    model = dense.active_model() if model is None else model
+    rows = []
+    nvec = dense.embed([neutral], model=model)[0]
+    for a in vocab.axes:
+        g = a.glyph
+        gvec = dense.measure_anchor(g, model=model)
+        aw = aligned.get(g)
+        uw = unrelated.get(g)
+        cos_aligned = cos_unrelated = None
+        if aw:
+            cos_aligned = float(cosine(dense.embed([aw], model=model)[0],
+                                       gvec[None, :])[0])
+        if uw:
+            cos_unrelated = float(cosine(dense.embed([uw], model=model)[0],
+                                         gvec[None, :])[0])
+        injected = dense.embed([f"{neutral} {g}"], model=model)[0]
+        # Q2: how much did injecting g move the chunk toward g's own direction?
+        delta_cos = float(cosine(gvec, injected[None, :])[0]
+                          - cosine(gvec, nvec[None, :])[0])
+        # Q3: the steered (projected) view of the same nudge
+        steered = float(sim_eff(gvec, injected[None, :], gvec[None, :], [1.0])[0]
+                        - sim_eff(gvec, nvec[None, :], gvec[None, :], [1.0])[0])
+        rows.append({"glyph": g, "axis": a.name,
+                     "cos_aligned": cos_aligned, "cos_unrelated": cos_unrelated,
+                     "Q1_separation": (None if cos_aligned is None or cos_unrelated is None
+                                       else cos_aligned - cos_unrelated),
+                     "Q2_delta_cos": delta_cos, "Q3_steered_delta": steered})
+    return {"model": model, "neutral": neutral, "glyphs": rows}
+
+
+# ---- Q4 : retrieval lift -----------------------------------------------------
+def retrieval_lift_lexical(corpus: list[Chunk], queries: list[dict],
+                           annotator, vocab: Vocabulary, *, k: int = 5) -> dict:
+    """Lexical Q4: MRR/recall with vs without the glyph sidecar.
+
+    `queries`: [{"text": str, "facet": glyph|None, "relevant": set[id]}].
+    Baseline indexes clean text; steered indexes annotated chunks and faceting on.
+    """
+    base_chunks = [Chunk(c.id, c.text) for c in corpus]
+    ann_chunks = [annotate_chunk(c, annotator, vocab) for c in corpus]
+    base_con = build_index(base_chunks, vocab)
+    ann_con = build_index(ann_chunks, vocab)
+
+    def run(con, use_facet):
+        mrrs, recs = [], []
+        for q in queries:
+            facet = q.get("facet") if use_facet else None
+            hits = search(con, q["text"], facet=facet, vocab=vocab, limit=k)
+            ids = [h["id"] for h in hits]
+            mrrs.append(mrr(ids, q["relevant"]))
+            recs.append(recall_at_k(ids, q["relevant"], k))
+        return {"mrr": float(np.mean(mrrs)), f"recall@{k}": float(np.mean(recs))}
+
+    return {"baseline": run(base_con, False), "steered": run(ann_con, True), "k": k}