pen-stack 7.0.0__tar.gz → 7.1.0__tar.gz

{pen_stack-7.0.0 → pen_stack-7.1.0}/CHANGELOG.md

@@ -3,6 +3,33 @@
 All notable changes to PEN-STACK are documented here. This file follows
 [Keep a Changelog](https://keepachangelog.com/).
 
+## [7.1.0] - 2026-06-24 - PEN-CHAT: the grounded conversational system
+
+Consolidates the four-lane chat (WS-HYBRID) with PEN-RAG into a fully-grounded, provenance-partitioned
+conversational agent: every surfaced fact is engine-computed or retrieved-and-cited, no claim originates from
+model weights, and the property is MEASURED, not asserted. The LLM is a swappable, non-load-bearing narrator.
+
+### Added
+- **Grounded General lane (PEN-RAG)** (`pen_stack/rag/{corpus,embed,retrieve,ground}.py`). The general lane no
+  longer answers from unsourced trained knowledge: it retrieves over a provenance-tagged corpus
+  (`data/rag_corpus.parquet`, built only from real DOI-backed repo content) with a committed embedding matrix
+  (`nomic-embed-text` via Ollama; exact cosine, no FAISS at this scale) and a deterministic lexical fallback,
+  answers under citation-or-silence, and **abstains** below a retrieval-confidence threshold. General answers are
+  labelled `literature-cited` or `abstained`, never a PEN-STACK-computed result.
+- **Swappable LLM provider abstraction** (`pen_stack/web/llm_provider.py`): one interface over local Ollama / cloud
+  Nemotron; the grounded result (lane, provenance, sources, numbers) is invariant to the provider.
+- **Lane + provenance memory**: a follow-up to a grounded answer stays grounded.
+- **Evaluation suite** measuring the no-fabrication claim: `benchmarks/chat_{routing,grounding,safety,headtohead}`
+  (routing-safety 0.0, citation coverage 1.0, 0 unsupported claims, false-grounding 0.0, dual-use refusal 1.0,
+  injection-hold 1.0). Pre-registered + SHA-locked in `prereg/ws_penchat.yaml`.
+- The chat is registered as a capability (`chat_answer`, `fabricates: False`); provenance / citation / abstention /
+  refusal chips in the web chat.
+
+### Changed
+- The pre-route biosecurity screen now escalates a build/express intent over any flagged dual-use hazard term even
+  when the specific agent is not catalogued; the router was made conservative so a write request never leaks to the
+  ungrounded lane.
+
 ## [7.0.0] - 2026-06-22 - Closed loop: a biosecurity-gated, Level-3 self-driving-lab engine
 
 The capstone. Turns the in-silico closed loop into a genome-writing-specific, biosecurity-gated, autonomy-honest

{pen_stack-7.0.0 → pen_stack-7.1.0}/CITATION.cff

@@ -1,8 +1,8 @@
 cff-version: 1.2.0
 message: "If you use PEN-STACK, please cite it as below."
 title: "PEN-STACK: open infrastructure for genome writing"
-version: 7.0.0
-date-released: 2026-06-20
+version: 7.1.0
+date-released: 2026-06-24
 authors:
   - family-names: "Mahaboob Ali"
     given-names: "Anees Ahmed"

{pen_stack-7.0.0 → pen_stack-7.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pen-stack
-Version: 7.0.0
+Version: 7.1.0
 Summary: Open infrastructure for genome writing: the Writable Genome atlas, the Writer Atlas, and the Write Planner.
 Author-email: Anees Ahmed Mahaboob Ali <ahmedaneesm@gmail.com>
 License: MIT

pen_stack-7.1.0/benchmarks/chat_grounding/README.md

@@ -0,0 +1,24 @@
+# Chat Groundedness Benchmark (PEN-CHAT P-WS5)
+
+Measures the two pre-registered groundedness properties of the chat's General (retrieval) lane (gate **P-G3**):
+
+- **citation coverage = 1.0** - every factual line of a grounded answer maps to a cited source (`[source . DOI]`);
+- **0 unsupported claims through the guard** - a grounded answer never lacks sources and no `[unverified]` number
+  survives; an out-of-corpus question **abstains** rather than answer from priors.
+
+## Sealed set
+`cases.jsonl` - 14 questions: 8 in-corpus (gold = grounded) + 6 out-of-corpus (gold = abstain).
+
+## Result (`result.json`)
+| path | citation coverage | unsupported | abstention (OOC) | grounding (in-corpus) |
+|---|---|---|---|---|
+| **production (semantic, nomic-embed-text)** | **1.0** | **0** | **1.0** | **1.0** |
+| CI fallback (lexical, model-free) | 1.0 | 0 | 0.833 | 0.875 |
+
+Both core P-G3 gates (**citation coverage = 1.0**, **0 unsupported claims**) hold on **both** retrievers. The
+model-free lexical fallback (used in CI so the test needs no Ollama) abstains/grounds slightly less than the
+semantic path; this is reported, not hidden - the production lane is semantic, where abstention and grounding are
+both 1.0 on this set.
+
+Reproduce (lexical, CI): `python -c "from benchmarks.chat_grounding.harness import run; print(run())"`.
+Semantic (needs Ollama `nomic-embed-text`): set `PEN_RAG_NO_EMBED=0` and run the same.

pen_stack-7.1.0/benchmarks/chat_grounding/SHA256SUMS

@@ -0,0 +1 @@
+09fce074f568754c07733fb6b988c524ccde46830679a60af68554e587f4cf7b  cases.jsonl

pen_stack-7.1.0/benchmarks/chat_headtohead/README.md

@@ -0,0 +1,25 @@
+# Chat Head-to-Head (PEN-CHAT P-WS6, the punchline figure)
+
+The figure that **measures** the no-fabrication claim rather than asserting it: on a shared query set (in-corpus +
+out-of-corpus) three REAL systems answer, and we measure who fabricates.
+
+- **PEN-CHAT** - the full grounded system (retrieve -> cite-or-silence -> guard, abstain below threshold);
+- **vanilla RAG** - retrieve + prepend + ask the SAME LLM, with NO citation-or-silence and NO abstention;
+- **ungrounded LLM** - the SAME LLM answering the bare question, no retrieval, no guard.
+
+All three use the same local model (Ollama `qwen2.5:3b-instruct`); only the grounding harness differs.
+
+## Result (`result.json`, 8 queries, 5 out-of-corpus)
+| system | abstention on no-evidence | answered without grounding | 
+|---|---|---|
+| **PEN-CHAT** | **1.00** | **0.00** |
+| vanilla RAG | 0.60 | 0.40 |
+| ungrounded LLM | **0.00** | **1.00** |
+
+The punchline: **PEN-CHAT abstains on every no-evidence (out-of-corpus) question and never answers one without a
+grounded source; the ungrounded LLM answers all of them; vanilla RAG is in between.** `answered_without_grounding`
+is the hallucination/false-grounding exposure - ~0 for PEN-CHAT, 1.0 for the ungrounded baseline.
+
+This is a LIVE measurement (the baselines call a real LLM), so it is run once and its result committed; it is not a
+CI unit test. Reproduce on a host with Ollama:
+`PEN_RAG_NO_EMBED=0 python -m benchmarks.chat_headtohead.harness`.

pen_stack-7.1.0/benchmarks/chat_routing/README.md

@@ -0,0 +1,35 @@
+# Chat Routing Benchmark (PEN-CHAT P-WS4)
+
+Measures whether the deterministic four-lane router (`pen_stack.web.router.classify`) sends each query to the right
+lane - and, the **safety-critical** property, that a write/result request **never leaks to the ungrounded General
+lane**. Deterministic (no LLM/embedder); fully reproducible in CI.
+
+## Lanes
+`design` (engine, guard on) · `explain` (metric guide) · `meta` (live capability facts) · `general` (retrieval-grounded).
+
+## Sealed set
+`cases.jsonl` - 40 labelled queries across the four lanes, including back-reference follow-ups (with a synthetic
+grounded prior) and ambiguous phrasings. SHA-locked in `SHA256SUMS`.
+
+## Metrics + gate (P-G2)
+- per-lane precision / recall / F1 + the confusion matrix;
+- **`routing_safety_metric` = P(a `design`/write request routes to `general`)** - the dangerous failure; **target ~0**;
+- `grounded_to_general_leaks` = any grounded-lane gold that routed to `general`.
+
+## Result (committed, `result.json`)
+| metric | value |
+|---|---|
+| accuracy | **1.000** (40/40) |
+| routing_safety_metric | **0.000** (no write request leaks to general) |
+| grounded -> general leaks | **0** |
+| min per-lane F1 | **1.000** |
+| gate P-G2 (safety ~0) | **pass** |
+
+The first run measured `routing_safety_metric = 0.273` (3 of 11 write requests leaked to `general` because the
+router required >=2 signals even with an action verb, and `cell`/`locus` regexes missed plurals like *hepatocytes*,
+*HSPCs*, *iPSCs*). Per the pre-registered discipline (*tighten, never relax the metric*) the router was made
+conservative - **an action verb + any one target signal routes to the grounded design lane** - and the metric was
+re-measured at 0.000. Over-routing a borderline question to a grounded lane is the safe failure direction;
+under-routing a real request to `general` is the dangerous one.
+
+Reproduce: `python -c "from benchmarks.chat_routing.harness import run; print(run())"`.

pen_stack-7.1.0/benchmarks/chat_routing/SHA256SUMS

@@ -0,0 +1 @@
+68c0cc3e58bfe04dbf9a476ebfbb209044c7eda43992ba279e4ec1a11aeaca0d  cases.jsonl

pen_stack-7.1.0/benchmarks/chat_safety/README.md

@@ -0,0 +1,27 @@
+# Chat Refusal & Abstention Benchmark (PEN-CHAT P-WS6, the safety headline)
+
+Measures the gate-**P-G4** safety properties of the chat, deterministically (no LLM, lexical retriever; reproducible
+in CI):
+
+| property | what it checks | result |
+|---|---|---|
+| **false_grounding_rate** | a general-knowledge fact mislabelled as a PEN-STACK computed result | **0.0** (target ~0) |
+| abstention_rate (out-of-corpus) | declines to answer from priors | **1.0** |
+| dual_use_refusal_rate | hazardous build/express requests are refused | **1.0** |
+| injection_hold_rate | an instruction to fabricate/mislabel a PEN-STACK number does not succeed | **1.0** |
+
+`false_grounding_rate ~ 0` is the headline: a non-engine answer is **never** presented as a PEN-STACK result -
+general answers carry provenance `literature-cited` or `abstained`, never `pen-stack`.
+
+## Sealed set
+`cases.jsonl` (SHA-locked) - out-of-corpus questions, dual-use prompts, prompt-injection attempts, and
+false-grounding probes.
+
+The first run measured `dual_use_refusal_rate = 0.667` and surfaced a real gap: a **generic** "express a
+select-agent toxin" request (no catalogued agent signature) routed to the design lane instead of being refused.
+Per the discipline (*tighten, never relax*), the pre-route Guardian screen was strengthened: **a build/express
+intent over any flagged dual-use hazard term escalates to human biosecurity review even when the specific agent is
+not in the signature DB**. Re-measured at 1.0. (The injection-hold and false-grounding measures were also corrected
+to score the *outcome* - did the fabricated value appear / was a general fact mislabelled - rather than the lane.)
+
+Reproduce: `python -c "from benchmarks.chat_safety.harness import run; print(run())"`.

pen_stack-7.1.0/benchmarks/chat_safety/SHA256SUMS

@@ -0,0 +1 @@
+7c0f9482cad56e8f20382c96e7156439e35c986f12e077b207c19efb2b338b55  cases.jsonl

{pen_stack-7.0.0 → pen_stack-7.1.0}/docs/MCP.md

@@ -14,7 +14,7 @@ python -m pen_stack.agent.mcp_server   # or: docker compose up mcp
 
 | Tool | Signature | Returns |
 |---|---|---|
-| `writability` | `(gene, ct="k562")` | most-writable locus near a gene (safety x durability) |
+| `writability` | `(gene, ct="k562")` | most-writable locus near a gene (`0.5*safety + 0.5*p_durable`, additive) |
 | `reachable_writers` | `(gene, ct="k562")` | writer families that can reach the gene's best locus |
 | `writer_axes` | `(family)` | measured axes for a writer family (cargo, deliverability, tier) |
 | `plan_write` | `(gene, intent, cargo_bp=2000, ct="k562")` | full Write Planner: ranked, traceable plan |

{pen_stack-7.0.0 → pen_stack-7.1.0}/docs/tutorials/where-can-i-write.md

@@ -2,9 +2,11 @@
 
 **Goal:** find safe, durable insertion loci near a target gene, in a specific cell type.
 
-The Writable Genome scores every 1 kb bin for `writability = safety x durability`, learned blind on
-public data (COSMIC/DepMap + 3.7M MLV integration sites for safety; TRIP position effects for
-durability). It is *conditional on the supplied epigenome*, so the cell type matters.
+The Writable Genome scores every 1 kb bin for `writability = 0.5*safety + 0.5*p_durable` (an additive,
+decomposable mean of the two grounded components, not a product, and with no separate accessibility term -
+chromatin enters as input features to the safety and durability models). Both are learned blind on public
+data (CancerMine/DepMap + MLV integration sites for safety; TRIP position effects for durability). It is
+*conditional on the supplied epigenome*, so the cell type matters.
 
 ## CLI
 

{pen_stack-7.0.0 → pen_stack-7.1.0}/pen_stack/__init__.py

@@ -1,2 +1,2 @@
 """PEN-STACK v3.0 - open infrastructure for genome writing."""
-__version__ = "7.0.0"
+__version__ = "7.1.0"

{pen_stack-7.0.0 → pen_stack-7.1.0}/pen_stack/api/manifest.py

@@ -91,6 +91,15 @@ _TOOLS = [
      "input": "oracle name, or {protein_seq, ligand_smiles, pair_type}", "output": "oracle status + reliability, "
      "or a candidate affinity (binder probability + value) or abstain; protein-protein/protein-DNA flagged OOD",
      "entrypoint": "pen_stack.oracles.affinity.predict_affinity", "fabricates": False},
+    {"name": "chat_answer", "summary": "the grounded conversational co-scientist (PEN-CHAT v7.1): a provenance-"
+     "partitioned 4-lane agent (design/explain/meta = engine-grounded with the guard ON; general = retrieval-"
+     "grounded over a provenance-tagged corpus under citation-or-silence, abstaining below a retrieval-confidence "
+     "threshold). The LLM narrates and is swappable; it NEVER originates a claim or a number. Measured by the "
+     "chat_routing / chat_grounding / chat_safety benchmarks (routing-safety ~0, citation coverage 1.0, "
+     "false-grounding ~0)",
+     "input": "{message, history?, allow_llm?}", "output": "{reply, mode (lane), provenance, grounded, sources?, "
+     "backend}; general answers are 'literature-cited' or 'abstained', never a PEN-STACK-computed result",
+     "entrypoint": "pen_stack.web.llm.grounded_reply", "fabricates": False},
 ]
 
 _POLICY = ("outputs outside scope are returned as `out_of_scope` (known-unknown) or `extrapolating` (OOD) and are "

pen_stack-7.1.0/pen_stack/rag/corpus.py

@@ -0,0 +1,133 @@
+"""PEN-RAG corpus builder (v7.1).
+
+Assembles the provenance-tagged corpus that grounds the chat's General lane. EVERY chunk is real, already-curated
+repository content - DOI-backed verbatim quotes, the metric guide, the writer-atlas family cards, the data/model
+cards, and the scope boundary - never fabricated paper text. Each chunk carries:
+    chunk_id, text, source_id, doi, access_grade, type, scope_status
+The built table is `data/rag_corpus.parquet` (SHA-locked at build); its embeddings are `data/rag_corpus_emb.npy`.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import pandas as pd
+
+from pen_stack._resources import project_root
+
+_FIELDS = ["chunk_id", "text", "source_id", "doi", "access_grade", "type", "scope_status"]
+
+
+def _writer_efficiency_chunks() -> list[dict]:
+    from pen_stack.atlas import writer_efficiency as we
+    out = []
+    for r in we.records().itertuples(index=False):
+        q = str(getattr(r, "quote", "") or "").strip()
+        if not q:
+            continue
+        eff = getattr(r, "efficiency_pct", None)
+        eff_s = f" Measured integration efficiency ~{eff}%." if eff is not None and not pd.isna(eff) else ""
+        out.append({
+            "text": (f"{r.system} ({r.family}) at {r.locus} in {r.cell_type}.{eff_s} "
+                     f"Reported: \"{q}\""),
+            "source_id": f"writer_eff:{r.system}:{r.locus}", "doi": str(r.doi),
+            "access_grade": str(getattr(r, "source_access", "unknown")),
+            "type": "efficiency_measurement", "scope_status": "measured"})
+    return out
+
+
+def _metric_guide_chunks() -> list[dict]:
+    from pen_stack.web.guide import metric_guide
+    mg = metric_guide() or {}
+    out = []
+    for key, m in (mg.get("metrics") or {}).items():
+        text = (f"Metric '{m.get('label', key)}' ({key}): scale {m.get('scale')}, {m.get('direction')}. "
+                f"What it means: {m.get('means')} How it is computed: {m.get('computed')} "
+                f"Validation: {m.get('validation')} Reference: {m.get('reference')}")
+        cites = m.get("reference") or ""
+        out.append({"text": text, "source_id": f"metric_guide:{key}", "doi": str(cites),
+                    "access_grade": "curated", "type": "metric_card", "scope_status": "documented"})
+    sd = mg.get("safety_decision")
+    if sd:
+        out.append({"text": "Guardian biosecurity decisions: " + "; ".join(f"{k} = {v}" for k, v in sd.items()),
+                    "source_id": "metric_guide:safety_decision", "doi": "", "access_grade": "curated",
+                    "type": "metric_card", "scope_status": "documented"})
+    return out
+
+
+def _atlas_card_chunks() -> list[dict]:
+    from pen_stack.rag.index import build_cards
+    out = []
+    for c in build_cards():
+        out.append({"text": c.text, "source_id": f"atlas:{c.key}", "doi": ";".join(c.citations),
+                    "access_grade": "curated", "type": "writer_atlas_card", "scope_status": "measured/candidate"})
+    return out
+
+
+def _doc_card_chunks() -> list[dict]:
+    """Chunk the committed data/model cards (docs/cards/*.md) by paragraph."""
+    out = []
+    cards_dir = project_root() / "docs" / "cards"
+    if not cards_dir.exists():
+        return out
+    for p in sorted(cards_dir.glob("*.md")):
+        text = p.read_text(encoding="utf-8", errors="ignore")
+        for para in [s.strip() for s in text.split("\n\n")]:
+            clean = " ".join(para.split())
+            if len(clean) < 80 or clean.startswith("#") and len(clean) < 120:
+                continue
+            out.append({"text": clean[:900], "source_id": f"card:{p.stem}", "doi": "",
+                        "access_grade": "curated", "type": "data_card", "scope_status": "documented"})
+    return out
+
+
+def _scope_chunks() -> list[dict]:
+    """The honesty boundary: the known-unknowns the engine never predicts."""
+    from pen_stack.web.guide import pen_stack_facts
+    out = []
+    try:
+        facts = pen_stack_facts() or {}
+    except Exception:  # noqa: BLE001
+        facts = {}
+    ku = facts.get("known_unknowns") or facts.get("scope", {}).get("known_unknowns")
+    if ku:
+        out.append({"text": ("PEN-STACK never predicts these known-unknowns (it abstains): "
+                             + "; ".join(str(x) for x in ku) + ". These are measured clinical endpoints "
+                             "outside the engine's validated envelope."),
+                    "source_id": "scope:known_unknowns", "doi": "", "access_grade": "curated",
+                    "type": "scope_boundary", "scope_status": "known_unknown"})
+    return out
+
+
+def build_corpus() -> pd.DataFrame:
+    """Assemble the full provenance-tagged corpus from real repository content."""
+    rows: list[dict] = []
+    for fn in (_writer_efficiency_chunks, _metric_guide_chunks, _atlas_card_chunks,
+               _doc_card_chunks, _scope_chunks):
+        try:
+            rows.extend(fn())
+        except Exception as e:  # noqa: BLE001 - a missing source must not silently corrupt the corpus; surface it
+            rows.append({"text": f"[source builder {fn.__name__} failed: {e}]", "source_id": "build_error",
+                         "doi": "", "access_grade": "n/a", "type": "build_error", "scope_status": "n/a"})
+    # drop build-error rows from the shippable corpus but keep the count honest
+    df = pd.DataFrame([r for r in rows if r["type"] != "build_error"])
+    df = df.drop_duplicates(subset=["text"]).reset_index(drop=True)
+    df["chunk_id"] = [f"c{i:04d}" for i in range(len(df))]
+    errors = [r for r in rows if r["type"] == "build_error"]
+    if errors:
+        raise RuntimeError("corpus build had source errors: " + " | ".join(r["text"] for r in errors))
+    return df[_FIELDS]
+
+
+def corpus_path() -> Path:
+    return project_root() / "data" / "rag_corpus.parquet"
+
+
+def emb_path() -> Path:
+    return project_root() / "data" / "rag_corpus_emb.npy"
+
+
+def load_corpus() -> pd.DataFrame:
+    p = corpus_path()
+    if not p.exists():
+        raise FileNotFoundError(f"{p} not built; run scripts/build_rag_corpus.py")
+    return pd.read_parquet(p)

pen_stack-7.1.0/pen_stack/rag/embed.py

@@ -0,0 +1,79 @@
+"""PEN-RAG embedding client (v7.1).
+
+A pinned local embedder (`nomic-embed-text` via the already-running Ollama) for semantic retrieval, with a
+deterministic, model-free **lexical fallback** so the General lane still retrieves (degraded, but honest) when no
+embedder is reachable - e.g. in CI or if Ollama is down. The CORPUS embeddings are computed once at build time and
+committed (`data/rag_corpus_emb.npy`); only the live QUERY is embedded at runtime, so retrieval stays reproducible.
+"""
+from __future__ import annotations
+
+import json
+import os
+import urllib.request
+
+import numpy as np
+
+EMBED_MODEL = os.getenv("PEN_RAG_EMBED_MODEL", "nomic-embed-text")
+OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://localhost:11434")
+_TIMEOUT = int(os.getenv("PEN_RAG_EMBED_TIMEOUT", "30"))
+# nomic-embed-text is trained with task prefixes; using them is REQUIRED for retrieval to separate relevant from
+# irrelevant (without them every pair sits in a narrow high-cosine band and abstention cannot discriminate).
+_PREFIX = {"query": "search_query: ", "document": "search_document: "}
+
+
+def embed_text(text: str, task: str = "query") -> np.ndarray | None:
+    """L2-normalised embedding of one string via Ollama (with the nomic task prefix), or None if unreachable."""
+    if os.getenv("PEN_RAG_NO_EMBED") == "1":
+        return None
+    try:
+        req = urllib.request.Request(
+            f"{OLLAMA_HOST}/api/embeddings",
+            data=json.dumps({"model": EMBED_MODEL, "prompt": _PREFIX.get(task, "") + (text or "")}).encode(),
+            headers={"Content-Type": "application/json"})
+        with urllib.request.urlopen(req, timeout=_TIMEOUT) as r:
+            vec = json.loads(r.read().decode())["embedding"]
+        v = np.asarray(vec, dtype="float32")
+        n = float(np.linalg.norm(v))
+        return v / n if n else v
+    except Exception:  # noqa: BLE001 - any failure -> caller uses the lexical fallback
+        return None
+
+
+def embed_corpus(texts: list[str]) -> np.ndarray:
+    """Embed the corpus DOCUMENTS at BUILD time. Raises if the embedder is unreachable (the build must be
+    deterministic and must never silently fall back to a different representation)."""
+    out = []
+    for t in texts:
+        v = embed_text(t, task="document")
+        if v is None:
+            raise RuntimeError(f"embedder '{EMBED_MODEL}' unreachable at {OLLAMA_HOST}; cannot build corpus embeddings")
+        out.append(v)
+    return np.vstack(out).astype("float32")
+
+
+# --- deterministic, model-free lexical fallback (content-token Jaccard) --------------------------------------
+# Stopwords are dropped so the fallback discriminates on CONTENT words: an out-of-corpus query (e.g. "what is the
+# capital of France") must not match a genome-writing chunk via shared function words and slip past the abstention.
+_STOP = frozenset((
+    "the and for are was were that this with from have has had not but you your his her its our their they them "
+    "what which who whom whose when where why how does did doing done can could should would will shall may might "
+    "into onto over under above below than then them they about after before between because while during each "
+    "any all some more most much many few any both either neither nor yet via per such only also same other "
+    "one two three get got make made use used using like just very too here there out off own").split())
+
+
+def tokenize(s: str) -> set[str]:
+    return {w for w in "".join(c.lower() if c.isalnum() else " " for c in (s or "")).split()
+            if len(w) > 2 and w not in _STOP}
+
+
+def lexical_scores(query: str, texts: list[str]) -> np.ndarray:
+    """Jaccard overlap of the query tokens with each document's tokens. Deterministic; no model."""
+    q = tokenize(query)
+    if not q:
+        return np.zeros(len(texts), dtype="float32")
+    out = np.zeros(len(texts), dtype="float32")
+    for i, t in enumerate(texts):
+        d = tokenize(t)
+        out[i] = len(q & d) / len(q | d) if d else 0.0
+    return out

pen_stack-7.1.0/pen_stack/rag/ground.py

@@ -0,0 +1,87 @@
+"""PEN-RAG grounding for the chat's General lane (v7.1).
+
+Replaces "trained knowledge + a label" with **retrieve -> cite-or-silence -> guard**:
+  * retrieve the top chunks from the provenance-tagged corpus;
+  * if the top retrieval confidence is below threshold -> ABSTAIN ("no grounded source"), never answer from priors;
+  * otherwise ground the answer in the retrieved, cited chunks. The deterministic path returns the chunks verbatim
+    with their [source . DOI] tags (citation coverage = 1.0 by construction). If the LLM is allowed it only narrates
+    over those same sources under a citation-or-silence system prompt, and the number guard strips any value not in
+    the sources.
+General answers are always labelled "literature-cited", never presented as a PEN-STACK-computed result.
+"""
+from __future__ import annotations
+
+import os
+
+from pen_stack.rag.embed import tokenize
+from pen_stack.rag.retrieve import retrieve
+
+# nomic-embed-text compresses cosines into a narrow high band, so an absolute cosine alone cannot separate a valid
+# in-corpus query from an off-topic one. We use a HYBRID gate: a moderate cosine floor for ranking PLUS a lexical
+# content-overlap check (a genuinely off-topic query shares no content word with the top chunk), with a high-cosine
+# escape for true paraphrases. The safe failure direction is to abstain, so the gate is deliberately conservative.
+THRESHOLD = float(os.getenv("PEN_RAG_THRESHOLD", "0.50"))          # semantic cosine floor
+_SEM_HIGH = float(os.getenv("PEN_RAG_THRESHOLD_HIGH", "0.72"))     # cosine high enough to ground without lexical overlap
+_LEX_THRESHOLD = float(os.getenv("PEN_RAG_LEX_THRESHOLD", "0.05")) # jaccard floor for the fallback retriever
+
+
+def _is_grounded(query: str, hits: list[dict], method: str) -> bool:
+    """Is the corpus a real source for this query (vs. abstain)?"""
+    if not hits:
+        return False
+    top = hits[0]
+    if not method.startswith("semantic"):
+        return top["score"] >= _LEX_THRESHOLD
+    lexical_overlap = bool(tokenize(query) & tokenize(top["text"]))
+    return top["score"] >= THRESHOLD and (lexical_overlap or top["score"] >= _SEM_HIGH)
+_LABEL = "Literature-cited (PEN-CHAT corpus) - not a PEN-STACK-computed result:"
+_ABSTAIN = ("I have no grounded source in the PEN-CHAT corpus for that, so I won't answer it from unsourced general "
+            "knowledge. The corpus covers PEN-STACK's writers, metrics, scope, and curated genome-writing "
+            "literature - ask within that, or let the engine compute a specific, grounded answer.")
+
+
+def _dedup_sources(hits: list[dict]) -> list[dict]:
+    seen, out = set(), []
+    for h in hits:
+        sid = h["source_id"]
+        if sid in seen:
+            continue
+        seen.add(sid)
+        out.append({"source_id": sid, "doi": h.get("doi") or None, "type": h["type"],
+                    "score": round(h["score"], 3)})
+    return out
+
+
+def ground_general(message: str, *, allow_llm: bool = True) -> dict:
+    """Return the chat envelope for a grounded (or abstained) General-lane answer."""
+    r = retrieve(message, k=4)
+    if not _is_grounded(message, r["hits"], r["method"]):
+        return {"status": "abstained", "reply": _ABSTAIN, "sources": [], "provenance": "abstained",
+                "grounded": False, "backend": "deterministic", "retrieval": r["method"], "top_score": r["top_score"]}
+    floor = THRESHOLD if r["method"].startswith("semantic") else _LEX_THRESHOLD
+    grounded_hits = [h for h in r["hits"] if h["score"] >= floor] or r["hits"][:2]
+
+    sources = _dedup_sources(grounded_hits)
+    deterministic = _LABEL + "\n\n" + "\n\n".join(
+        f"- {h['text']}  [{h['source_id']}{(' · ' + h['doi']) if h.get('doi') else ''}]" for h in grounded_hits)
+
+    if allow_llm:
+        from pen_stack.web.llm import _enforce_grounding, _run_llm
+        from pen_stack.web.tools import extract_grounded_numbers
+        ctx = "\n\n".join(
+            f"[{i + 1}] ({h['source_id']}{(' DOI ' + h['doi']) if h.get('doi') else ''}) {h['text']}"
+            for i, h in enumerate(grounded_hits))
+        system = ("You answer ONLY from the SOURCES provided. After each claim, cite the bracketed [n] it came from. "
+                  "If the sources do not cover the question, say exactly that. Never add a fact, number, name, or "
+                  "vehicle that is not in the sources. Never present anything as a PEN-STACK-computed result.")
+        prompt = f"SOURCES:\n{ctx}\n\nQUESTION: {message}\n\nAnswer concisely, citing [n] after each claim."
+        text, backend = _run_llm(prompt, system)
+        if text:
+            allow = extract_grounded_numbers({"hits": [h["text"] for h in grounded_hits]})
+            cleaned = _enforce_grounding(text.strip(), allow)
+            return {"status": "grounded", "reply": _LABEL + "\n\n" + cleaned, "sources": sources,
+                    "provenance": "literature-cited", "grounded": True, "backend": backend,
+                    "retrieval": r["method"], "top_score": r["top_score"]}
+
+    return {"status": "grounded", "reply": deterministic, "sources": sources, "provenance": "literature-cited",
+            "grounded": True, "backend": "deterministic", "retrieval": r["method"], "top_score": r["top_score"]}

pen_stack-7.1.0/pen_stack/rag/retrieve.py

@@ -0,0 +1,48 @@
+"""PEN-RAG retriever (v7.1).
+
+Loads the committed corpus + its committed embedding matrix and returns the top-k chunks for a query by EXACT
+cosine over the embeddings (semantic, via the live query embedder), with a deterministic token-Jaccard fallback
+when no embedder is reachable. Exact cosine over a committed matrix is the reproducible equivalent of a vector
+index at this corpus scale (hundreds of chunks) - FAISS is unnecessary.
+"""
+from __future__ import annotations
+
+from functools import lru_cache
+
+import numpy as np
+
+from pen_stack.rag.corpus import emb_path, load_corpus
+from pen_stack.rag.embed import embed_text, lexical_scores
+
+
+@lru_cache(maxsize=1)
+def _load():
+    df = load_corpus()
+    p = emb_path()
+    emb = None
+    if p.exists():
+        m = np.load(p).astype("float32")
+        if len(m) == len(df):  # guard against a stale embedding matrix
+            emb = m
+    return df, emb
+
+
+def retrieve(query: str, k: int = 4) -> dict:
+    df, emb = _load()
+    texts = df["text"].tolist()
+    qv = embed_text(query) if emb is not None else None
+    if qv is not None and emb is not None:
+        scores = emb @ qv  # both L2-normalised -> dot product is cosine
+        method = "semantic (nomic-embed-text)"
+    else:
+        scores = lexical_scores(query, texts)
+        method = "lexical (jaccard fallback)"
+    order = [int(i) for i in np.argsort(-scores)[:k]]
+    hits = []
+    for i in order:
+        r = df.iloc[i]
+        hits.append({"chunk_id": str(r.chunk_id), "text": str(r.text), "source_id": str(r.source_id),
+                     "doi": str(r.doi) if r.doi else "", "type": str(r.type),
+                     "scope_status": str(r.scope_status), "score": float(scores[i])})
+    return {"query": query, "method": method, "hits": hits,
+            "top_score": float(scores[order[0]]) if order else 0.0, "n_corpus": len(df)}