pen-stack 7.0.0__tar.gz → 7.1.1__tar.gz

{pen_stack-7.0.0 → pen_stack-7.1.1}/CHANGELOG.md

@@ -3,6 +3,50 @@
 All notable changes to PEN-STACK are documented here. This file follows
 [Keep a Changelog](https://keepachangelog.com/).
 
+## [7.1.1] - 2026-06-24 - PEN-CHAT: General-lane helpfulness fix + benchmark-validity correction
+
+### Fixed
+- The General lane over-abstained: it declined simple/social ("hi") and general-knowledge ("what is DNA")
+  questions because P-WS1 made corpus retrieval a GATE on the whole lane. Retrieval is now **additive**
+  (`pen_stack/rag/ground.py`): the lane ANSWERS general + social questions, clearly labelled
+  "general - not PEN-STACK-verified" (a labelled general answer is honest, not a fabrication); a corpus hit
+  upgrades to "literature-cited"; abstention is reserved for a SPECIFIC unsourceable empirical claim. The "general"
+  provenance chip is restored in the web chat.
+
+### Changed
+- Benchmark-validity correction: the head-to-head headline moves from "answers-without-grounding" (which rewarded
+  over-abstention) to the **false-grounding rate** (a non-engine fact presented as a PEN-STACK result; 0.0 by
+  construction). Abstention is re-scoped to specific unsourceable empirical claims; the routing/grounding/safety
+  sets now include positive general + social cases that require a helpful, labelled answer (a regression guard).
+  `prereg/ws_penchat.yaml` amended + re-locked accordingly.
+
+## [7.1.0] - 2026-06-24 - PEN-CHAT: the grounded conversational system
+
+Consolidates the four-lane chat (WS-HYBRID) with PEN-RAG into a fully-grounded, provenance-partitioned
+conversational agent: every surfaced fact is engine-computed or retrieved-and-cited, no claim originates from
+model weights, and the property is MEASURED, not asserted. The LLM is a swappable, non-load-bearing narrator.
+
+### Added
+- **Grounded General lane (PEN-RAG)** (`pen_stack/rag/{corpus,embed,retrieve,ground}.py`). The general lane no
+  longer answers from unsourced trained knowledge: it retrieves over a provenance-tagged corpus
+  (`data/rag_corpus.parquet`, built only from real DOI-backed repo content) with a committed embedding matrix
+  (`nomic-embed-text` via Ollama; exact cosine, no FAISS at this scale) and a deterministic lexical fallback,
+  answers under citation-or-silence, and **abstains** below a retrieval-confidence threshold. General answers are
+  labelled `literature-cited` or `abstained`, never a PEN-STACK-computed result.
+- **Swappable LLM provider abstraction** (`pen_stack/web/llm_provider.py`): one interface over local Ollama / cloud
+  Nemotron; the grounded result (lane, provenance, sources, numbers) is invariant to the provider.
+- **Lane + provenance memory**: a follow-up to a grounded answer stays grounded.
+- **Evaluation suite** measuring the no-fabrication claim: `benchmarks/chat_{routing,grounding,safety,headtohead}`
+  (routing-safety 0.0, citation coverage 1.0, 0 unsupported claims, false-grounding 0.0, dual-use refusal 1.0,
+  injection-hold 1.0). Pre-registered + SHA-locked in `prereg/ws_penchat.yaml`.
+- The chat is registered as a capability (`chat_answer`, `fabricates: False`); provenance / citation / abstention /
+  refusal chips in the web chat.
+
+### Changed
+- The pre-route biosecurity screen now escalates a build/express intent over any flagged dual-use hazard term even
+  when the specific agent is not catalogued; the router was made conservative so a write request never leaks to the
+  ungrounded lane.
+
 ## [7.0.0] - 2026-06-22 - Closed loop: a biosecurity-gated, Level-3 self-driving-lab engine
 
 The capstone. Turns the in-silico closed loop into a genome-writing-specific, biosecurity-gated, autonomy-honest

{pen_stack-7.0.0 → pen_stack-7.1.1}/CITATION.cff

@@ -1,8 +1,8 @@
 cff-version: 1.2.0
 message: "If you use PEN-STACK, please cite it as below."
 title: "PEN-STACK: open infrastructure for genome writing"
-version: 7.0.0
-date-released: 2026-06-20
+version: 7.1.1
+date-released: 2026-06-24
 authors:
   - family-names: "Mahaboob Ali"
     given-names: "Anees Ahmed"

{pen_stack-7.0.0 → pen_stack-7.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pen-stack
-Version: 7.0.0
+Version: 7.1.1
 Summary: Open infrastructure for genome writing: the Writable Genome atlas, the Writer Atlas, and the Write Planner.
 Author-email: Anees Ahmed Mahaboob Ali <ahmedaneesm@gmail.com>
 License: MIT

pen_stack-7.1.1/benchmarks/chat_grounding/README.md

@@ -0,0 +1,26 @@
+# Chat Groundedness Benchmark (PEN-CHAT P-WS5, re-scoped v7.1.1)
+
+Measures the corrected groundedness properties of the chat. The lane ANSWERS general and social questions
+(labelled), grounds genome-writing questions in the cited corpus, and abstains ONLY on a specific unsourceable
+empirical claim. The honesty metric is **false-grounding**, not answer-suppression. Deterministic (lexical
+retriever, no LLM) -> reproducible in CI.
+
+## Sealed set
+`cases.jsonl` (SHA-locked) - `cited` (in-corpus genome-writing) + `general` (general knowledge) + `social` +
+`abstain` (specific unsourceable empirical) cases.
+
+## Result (`result.json`)
+| metric | value |
+|---|---|
+| citation_coverage (on cited answers) | **1.0** |
+| unsupported_claims_through_guard | **0** |
+| **false_grounding_rate** (a general fact mislabelled as a PEN-STACK result) | **0.0** |
+| **helpful_answer_rate** (general + social ANSWERED, not abstained) | **1.0** |
+| abstention_on_specific_unsourceable | **1.0** |
+| per-gold pass | cited 6/6 · general 4/4 · social 2/2 · abstain 2/2 |
+
+The regression guard `helpful_answer_rate = 1.0` exists because v7.1.0 over-abstained (it declined "hi" and "what
+is DNA"); v7.1.1 made retrieval additive, so general + social questions are answered (labelled) while the cited
+answers keep citation coverage 1.0 and false-grounding stays 0.
+
+Reproduce: `python -c "from benchmarks.chat_grounding.harness import run; print(run())"`.

pen_stack-7.1.1/benchmarks/chat_grounding/SHA256SUMS

@@ -0,0 +1 @@
+e3e632af2b4728d03997585c946fb0b996fc2c8d5fd2f18cc154d16696004db5  cases.jsonl

pen_stack-7.1.1/benchmarks/chat_headtohead/README.md

@@ -0,0 +1,29 @@
+# Chat Head-to-Head (PEN-CHAT P-WS6, re-framed v7.1.1)
+
+The corrected punchline measures the property that actually encodes no-fabrication - **provenance labelling and
+false grounding** - NOT answer suppression. All three systems ANSWER general questions (a labelled general answer
+is honest); the differentiator is whether they tell you *what kind* of answer it is, and whether they invent a
+statistic for a made-up entity.
+
+Three REAL systems on a shared set, all using the same local model (Ollama `qwen2.5:3b-instruct`); only the
+grounding harness differs.
+
+## Result (`result.json`, 6 queries: 3 general + 3 specific-unsourceable)
+| system | provenance-labelled | fabricated a stat for a made-up entity |
+|---|---|---|
+| **PEN-CHAT** | **1.00** | **0.00** |
+| vanilla RAG | 0.00 | 0.67 |
+| ungrounded LLM | 0.00 | 0.67 |
+
+- **provenance_labelled_rate** - every PEN-CHAT answer carries its lane + provenance (general / literature-cited /
+  pen-stack / abstained), so a reader can never mistake a general-knowledge answer for a platform result. The
+  ungrounded baselines carry NO provenance - every answer looks equally authoritative.
+- **fabricated_stat_on_unsourceable_rate** - asked for a specific number about a made-up writer/locus, PEN-CHAT
+  routes to the engine or cites the corpus (a grounded number) or abstains - it never presents an *ungrounded*
+  number; the baselines invent a confident statistic 67% of the time.
+
+This is the honest claim: PEN-CHAT answers general questions like any assistant, but it never presents an unsourced
+fact as a platform result - measured - and the baselines do.
+
+Live measurement (the baselines call a real LLM); result committed. Not a CI unit test (a CI test reads the result).
+Reproduce on a host with Ollama: `PEN_RAG_NO_EMBED=0 python -m benchmarks.chat_headtohead.harness`.

pen_stack-7.1.1/benchmarks/chat_routing/README.md

@@ -0,0 +1,35 @@
+# Chat Routing Benchmark (PEN-CHAT P-WS4)
+
+Measures whether the deterministic four-lane router (`pen_stack.web.router.classify`) sends each query to the right
+lane - and, the **safety-critical** property, that a write/result request **never leaks to the ungrounded General
+lane**. Deterministic (no LLM/embedder); fully reproducible in CI.
+
+## Lanes
+`design` (engine, guard on) · `explain` (metric guide) · `meta` (live capability facts) · `general` (retrieval-grounded).
+
+## Sealed set
+`cases.jsonl` - 40 labelled queries across the four lanes, including back-reference follow-ups (with a synthetic
+grounded prior) and ambiguous phrasings. SHA-locked in `SHA256SUMS`.
+
+## Metrics + gate (P-G2)
+- per-lane precision / recall / F1 + the confusion matrix;
+- **`routing_safety_metric` = P(a `design`/write request routes to `general`)** - the dangerous failure; **target ~0**;
+- `grounded_to_general_leaks` = any grounded-lane gold that routed to `general`.
+
+## Result (committed, `result.json`)
+| metric | value |
+|---|---|
+| accuracy | **1.000** (40/40) |
+| routing_safety_metric | **0.000** (no write request leaks to general) |
+| grounded -> general leaks | **0** |
+| min per-lane F1 | **1.000** |
+| gate P-G2 (safety ~0) | **pass** |
+
+The first run measured `routing_safety_metric = 0.273` (3 of 11 write requests leaked to `general` because the
+router required >=2 signals even with an action verb, and `cell`/`locus` regexes missed plurals like *hepatocytes*,
+*HSPCs*, *iPSCs*). Per the pre-registered discipline (*tighten, never relax the metric*) the router was made
+conservative - **an action verb + any one target signal routes to the grounded design lane** - and the metric was
+re-measured at 0.000. Over-routing a borderline question to a grounded lane is the safe failure direction;
+under-routing a real request to `general` is the dangerous one.
+
+Reproduce: `python -c "from benchmarks.chat_routing.harness import run; print(run())"`.

pen_stack-7.1.1/benchmarks/chat_routing/SHA256SUMS

@@ -0,0 +1 @@
+68c0cc3e58bfe04dbf9a476ebfbb209044c7eda43992ba279e4ec1a11aeaca0d  cases.jsonl

pen_stack-7.1.1/benchmarks/chat_safety/README.md

@@ -0,0 +1,23 @@
+# Chat Refusal & Abstention Benchmark (PEN-CHAT P-WS6, re-scoped v7.1.1)
+
+Measures the gate-**P-G4** safety properties, deterministically (no LLM, lexical retriever; reproducible in CI).
+The headline is **false-grounding** - a non-engine fact presented as a PEN-STACK result - NOT answer-suppression.
+
+| property | what it checks | result |
+|---|---|---|
+| **false_grounding_rate** | a general fact mislabelled as a PEN-STACK computed result | **0.0** (target ~0) |
+| **general_answered_rate** | general-knowledge questions are ANSWERED, labelled 'general' (no over-abstention) | **1.0** |
+| specific_empirical_abstention_rate | a specific unsourceable empirical claim is declined / redirected | **1.0** |
+| dual_use_refusal_rate | hazardous build/express requests are refused | **1.0** |
+| injection_hold_rate | an instruction to fabricate/mislabel a PEN-STACK number does not succeed | **1.0** |
+
+`false_grounding_rate ~ 0` is the metric that encodes no-fabrication: a non-engine answer is **never** presented as
+a PEN-STACK result (general answers carry provenance `general`, never `pen-stack`). Note (v7.1.1): general-knowledge
+questions like "what is the capital of France" are now ANSWERED (labelled general), not abstained - abstention is
+reserved for a specific unsourceable empirical claim, which is the genuine citation-or-silence case.
+
+## Sealed set
+`cases.jsonl` (SHA-locked) - general-knowledge questions, specific-unsourceable-empirical claims, dual-use prompts,
+and prompt-injection attempts.
+
+Reproduce: `python -c "from benchmarks.chat_safety.harness import run; print(run())"`.

pen_stack-7.1.1/benchmarks/chat_safety/SHA256SUMS

@@ -0,0 +1 @@
+9b05cccd08632ff09b895ba2d39565ce788dea6b3c727fd78f9a1ab0e04cbbfc  cases.jsonl

{pen_stack-7.0.0 → pen_stack-7.1.1}/configs/expression/promoters.yaml

@@ -38,8 +38,8 @@ tissue_specific:
   lp1: { strength: 0.85, context: liver, confidence: medium, note: "HCR1 enhancer + hAAT core; clinical FIX from single AAV (2nd after TBG)", citation: "10.1182/blood-2005-10-4035" }
   haat: { strength: 0.80, context: liver, confidence: medium, note: "human α1-antitrypsin; base of ApoE-hAAT/LP1 hybrids", citation: "10.1182/blood-2005-10-4035" }
   hlp: { strength: 0.70, context: liver, confidence: medium, note: "HCR+hAAT+ApoE ~800 bp; below TBG/LP1; therapeutic at compact size", citation: "10.1016/j.gene.2012.07.009" }
-  camkiia: { strength: 0.90, context: neuron, confidence: medium, note: "excitatory neurons; greater overall expression than hSyn1", citation: "10.1016/j.neulet.2021.135889" }
-  hsyn: { strength: 0.70, context: neuron, confidence: medium, note: "pan-neuronal ~460 bp; robust but slower kinetics (CAG-level only by ~3 mo)", citation: "10.1016/j.neulet.2021.135889" }
+  camkiia: { strength: 0.90, context: neuron, confidence: low, note: "excitatory neurons; greater overall expression than hSyn1; prior DOI did not resolve, relative strength not independently sourced", citation: null }
+  hsyn: { strength: 0.70, context: neuron, confidence: low, note: "pan-neuronal ~460 bp; robust but slower kinetics; prior DOI did not resolve, relative strength not independently sourced", citation: null }
   mecp2: { strength: 0.50, context: neuron, confidence: low, note: "neuron-restricted compact; weaker, relative fold not sourced", citation: null }
   gfaabc1d: { strength: 0.80, context: astrocyte,confidence: high, note: "~681 bp; ~2× stronger than full gfa2; astrocyte-restricted", citation: "10.1002/glia.20622" }
   gfap: { strength: 0.60, context: astrocyte,confidence: medium,note: "gfa2 ~2.2 kb astrocyte-selective", citation: "10.1002/glia.20622" }

{pen_stack-7.0.0 → pen_stack-7.1.1}/docs/MCP.md

@@ -14,7 +14,7 @@ python -m pen_stack.agent.mcp_server   # or: docker compose up mcp
 
 | Tool | Signature | Returns |
 |---|---|---|
-| `writability` | `(gene, ct="k562")` | most-writable locus near a gene (safety x durability) |
+| `writability` | `(gene, ct="k562")` | most-writable locus near a gene (`0.5*safety + 0.5*p_durable`, additive) |
 | `reachable_writers` | `(gene, ct="k562")` | writer families that can reach the gene's best locus |
 | `writer_axes` | `(family)` | measured axes for a writer family (cargo, deliverability, tier) |
 | `plan_write` | `(gene, intent, cargo_bp=2000, ct="k562")` | full Write Planner: ranked, traceable plan |

{pen_stack-7.0.0 → pen_stack-7.1.1}/docs/tutorials/where-can-i-write.md

@@ -2,9 +2,11 @@
 
 **Goal:** find safe, durable insertion loci near a target gene, in a specific cell type.
 
-The Writable Genome scores every 1 kb bin for `writability = safety x durability`, learned blind on
-public data (COSMIC/DepMap + 3.7M MLV integration sites for safety; TRIP position effects for
-durability). It is *conditional on the supplied epigenome*, so the cell type matters.
+The Writable Genome scores every 1 kb bin for `writability = 0.5*safety + 0.5*p_durable` (an additive,
+decomposable mean of the two grounded components, not a product, and with no separate accessibility term -
+chromatin enters as input features to the safety and durability models). Both are learned blind on public
+data (CancerMine/DepMap + MLV integration sites for safety; TRIP position effects for durability). It is
+*conditional on the supplied epigenome*, so the cell type matters.
 
 ## CLI
 

{pen_stack-7.0.0 → pen_stack-7.1.1}/pen_stack/__init__.py

@@ -1,2 +1,2 @@
 """PEN-STACK v3.0 - open infrastructure for genome writing."""
-__version__ = "7.0.0"
+__version__ = "7.1.1"

{pen_stack-7.0.0 → pen_stack-7.1.1}/pen_stack/api/manifest.py

@@ -91,6 +91,15 @@ _TOOLS = [
      "input": "oracle name, or {protein_seq, ligand_smiles, pair_type}", "output": "oracle status + reliability, "
      "or a candidate affinity (binder probability + value) or abstain; protein-protein/protein-DNA flagged OOD",
      "entrypoint": "pen_stack.oracles.affinity.predict_affinity", "fabricates": False},
+    {"name": "chat_answer", "summary": "the grounded conversational co-scientist (PEN-CHAT v7.1): a provenance-"
+     "partitioned 4-lane agent (design/explain/meta = engine-grounded with the guard ON; general = retrieval-"
+     "grounded over a provenance-tagged corpus under citation-or-silence, abstaining below a retrieval-confidence "
+     "threshold). The LLM narrates and is swappable; it NEVER originates a claim or a number. Measured by the "
+     "chat_routing / chat_grounding / chat_safety benchmarks (routing-safety ~0, citation coverage 1.0, "
+     "false-grounding ~0)",
+     "input": "{message, history?, allow_llm?}", "output": "{reply, mode (lane), provenance, grounded, sources?, "
+     "backend}; general answers are 'literature-cited' or 'abstained', never a PEN-STACK-computed result",
+     "entrypoint": "pen_stack.web.llm.grounded_reply", "fabricates": False},
 ]
 
 _POLICY = ("outputs outside scope are returned as `out_of_scope` (known-unknown) or `extrapolating` (OOD) and are "

pen_stack-7.1.1/pen_stack/rag/corpus.py

@@ -0,0 +1,133 @@
+"""PEN-RAG corpus builder (v7.1).
+
+Assembles the provenance-tagged corpus that grounds the chat's General lane. EVERY chunk is real, already-curated
+repository content - DOI-backed verbatim quotes, the metric guide, the writer-atlas family cards, the data/model
+cards, and the scope boundary - never fabricated paper text. Each chunk carries:
+    chunk_id, text, source_id, doi, access_grade, type, scope_status
+The built table is `data/rag_corpus.parquet` (SHA-locked at build); its embeddings are `data/rag_corpus_emb.npy`.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import pandas as pd
+
+from pen_stack._resources import project_root
+
+_FIELDS = ["chunk_id", "text", "source_id", "doi", "access_grade", "type", "scope_status"]
+
+
+def _writer_efficiency_chunks() -> list[dict]:
+    from pen_stack.atlas import writer_efficiency as we
+    out = []
+    for r in we.records().itertuples(index=False):
+        q = str(getattr(r, "quote", "") or "").strip()
+        if not q:
+            continue
+        eff = getattr(r, "efficiency_pct", None)
+        eff_s = f" Measured integration efficiency ~{eff}%." if eff is not None and not pd.isna(eff) else ""
+        out.append({
+            "text": (f"{r.system} ({r.family}) at {r.locus} in {r.cell_type}.{eff_s} "
+                     f"Reported: \"{q}\""),
+            "source_id": f"writer_eff:{r.system}:{r.locus}", "doi": str(r.doi),
+            "access_grade": str(getattr(r, "source_access", "unknown")),
+            "type": "efficiency_measurement", "scope_status": "measured"})
+    return out
+
+
+def _metric_guide_chunks() -> list[dict]:
+    from pen_stack.web.guide import metric_guide
+    mg = metric_guide() or {}
+    out = []
+    for key, m in (mg.get("metrics") or {}).items():
+        text = (f"Metric '{m.get('label', key)}' ({key}): scale {m.get('scale')}, {m.get('direction')}. "
+                f"What it means: {m.get('means')} How it is computed: {m.get('computed')} "
+                f"Validation: {m.get('validation')} Reference: {m.get('reference')}")
+        cites = m.get("reference") or ""
+        out.append({"text": text, "source_id": f"metric_guide:{key}", "doi": str(cites),
+                    "access_grade": "curated", "type": "metric_card", "scope_status": "documented"})
+    sd = mg.get("safety_decision")
+    if sd:
+        out.append({"text": "Guardian biosecurity decisions: " + "; ".join(f"{k} = {v}" for k, v in sd.items()),
+                    "source_id": "metric_guide:safety_decision", "doi": "", "access_grade": "curated",
+                    "type": "metric_card", "scope_status": "documented"})
+    return out
+
+
+def _atlas_card_chunks() -> list[dict]:
+    from pen_stack.rag.index import build_cards
+    out = []
+    for c in build_cards():
+        out.append({"text": c.text, "source_id": f"atlas:{c.key}", "doi": ";".join(c.citations),
+                    "access_grade": "curated", "type": "writer_atlas_card", "scope_status": "measured/candidate"})
+    return out
+
+
+def _doc_card_chunks() -> list[dict]:
+    """Chunk the committed data/model cards (docs/cards/*.md) by paragraph."""
+    out = []
+    cards_dir = project_root() / "docs" / "cards"
+    if not cards_dir.exists():
+        return out
+    for p in sorted(cards_dir.glob("*.md")):
+        text = p.read_text(encoding="utf-8", errors="ignore")
+        for para in [s.strip() for s in text.split("\n\n")]:
+            clean = " ".join(para.split())
+            if len(clean) < 80 or clean.startswith("#") and len(clean) < 120:
+                continue
+            out.append({"text": clean[:900], "source_id": f"card:{p.stem}", "doi": "",
+                        "access_grade": "curated", "type": "data_card", "scope_status": "documented"})
+    return out
+
+
+def _scope_chunks() -> list[dict]:
+    """The honesty boundary: the known-unknowns the engine never predicts."""
+    from pen_stack.web.guide import pen_stack_facts
+    out = []
+    try:
+        facts = pen_stack_facts() or {}
+    except Exception:  # noqa: BLE001
+        facts = {}
+    ku = facts.get("known_unknowns") or facts.get("scope", {}).get("known_unknowns")
+    if ku:
+        out.append({"text": ("PEN-STACK never predicts these known-unknowns (it abstains): "
+                             + "; ".join(str(x) for x in ku) + ". These are measured clinical endpoints "
+                             "outside the engine's validated envelope."),
+                    "source_id": "scope:known_unknowns", "doi": "", "access_grade": "curated",
+                    "type": "scope_boundary", "scope_status": "known_unknown"})
+    return out
+
+
+def build_corpus() -> pd.DataFrame:
+    """Assemble the full provenance-tagged corpus from real repository content."""
+    rows: list[dict] = []
+    for fn in (_writer_efficiency_chunks, _metric_guide_chunks, _atlas_card_chunks,
+               _doc_card_chunks, _scope_chunks):
+        try:
+            rows.extend(fn())
+        except Exception as e:  # noqa: BLE001 - a missing source must not silently corrupt the corpus; surface it
+            rows.append({"text": f"[source builder {fn.__name__} failed: {e}]", "source_id": "build_error",
+                         "doi": "", "access_grade": "n/a", "type": "build_error", "scope_status": "n/a"})
+    # drop build-error rows from the shippable corpus but keep the count honest
+    df = pd.DataFrame([r for r in rows if r["type"] != "build_error"])
+    df = df.drop_duplicates(subset=["text"]).reset_index(drop=True)
+    df["chunk_id"] = [f"c{i:04d}" for i in range(len(df))]
+    errors = [r for r in rows if r["type"] == "build_error"]
+    if errors:
+        raise RuntimeError("corpus build had source errors: " + " | ".join(r["text"] for r in errors))
+    return df[_FIELDS]
+
+
+def corpus_path() -> Path:
+    return project_root() / "data" / "rag_corpus.parquet"
+
+
+def emb_path() -> Path:
+    return project_root() / "data" / "rag_corpus_emb.npy"
+
+
+def load_corpus() -> pd.DataFrame:
+    p = corpus_path()
+    if not p.exists():
+        raise FileNotFoundError(f"{p} not built; run scripts/build_rag_corpus.py")
+    return pd.read_parquet(p)

pen_stack-7.1.1/pen_stack/rag/embed.py

@@ -0,0 +1,79 @@
+"""PEN-RAG embedding client (v7.1).
+
+A pinned local embedder (`nomic-embed-text` via the already-running Ollama) for semantic retrieval, with a
+deterministic, model-free **lexical fallback** so the General lane still retrieves (degraded, but honest) when no
+embedder is reachable - e.g. in CI or if Ollama is down. The CORPUS embeddings are computed once at build time and
+committed (`data/rag_corpus_emb.npy`); only the live QUERY is embedded at runtime, so retrieval stays reproducible.
+"""
+from __future__ import annotations
+
+import json
+import os
+import urllib.request
+
+import numpy as np
+
+EMBED_MODEL = os.getenv("PEN_RAG_EMBED_MODEL", "nomic-embed-text")
+OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://localhost:11434")
+_TIMEOUT = int(os.getenv("PEN_RAG_EMBED_TIMEOUT", "30"))
+# nomic-embed-text is trained with task prefixes; using them is REQUIRED for retrieval to separate relevant from
+# irrelevant (without them every pair sits in a narrow high-cosine band and abstention cannot discriminate).
+_PREFIX = {"query": "search_query: ", "document": "search_document: "}
+
+
+def embed_text(text: str, task: str = "query") -> np.ndarray | None:
+    """L2-normalised embedding of one string via Ollama (with the nomic task prefix), or None if unreachable."""
+    if os.getenv("PEN_RAG_NO_EMBED") == "1":
+        return None
+    try:
+        req = urllib.request.Request(
+            f"{OLLAMA_HOST}/api/embeddings",
+            data=json.dumps({"model": EMBED_MODEL, "prompt": _PREFIX.get(task, "") + (text or "")}).encode(),
+            headers={"Content-Type": "application/json"})
+        with urllib.request.urlopen(req, timeout=_TIMEOUT) as r:
+            vec = json.loads(r.read().decode())["embedding"]
+        v = np.asarray(vec, dtype="float32")
+        n = float(np.linalg.norm(v))
+        return v / n if n else v
+    except Exception:  # noqa: BLE001 - any failure -> caller uses the lexical fallback
+        return None
+
+
+def embed_corpus(texts: list[str]) -> np.ndarray:
+    """Embed the corpus DOCUMENTS at BUILD time. Raises if the embedder is unreachable (the build must be
+    deterministic and must never silently fall back to a different representation)."""
+    out = []
+    for t in texts:
+        v = embed_text(t, task="document")
+        if v is None:
+            raise RuntimeError(f"embedder '{EMBED_MODEL}' unreachable at {OLLAMA_HOST}; cannot build corpus embeddings")
+        out.append(v)
+    return np.vstack(out).astype("float32")
+
+
+# --- deterministic, model-free lexical fallback (content-token Jaccard) --------------------------------------
+# Stopwords are dropped so the fallback discriminates on CONTENT words: an out-of-corpus query (e.g. "what is the
+# capital of France") must not match a genome-writing chunk via shared function words and slip past the abstention.
+_STOP = frozenset((
+    "the and for are was were that this with from have has had not but you your his her its our their they them "
+    "what which who whom whose when where why how does did doing done can could should would will shall may might "
+    "into onto over under above below than then them they about after before between because while during each "
+    "any all some more most much many few any both either neither nor yet via per such only also same other "
+    "one two three get got make made use used using like just very too here there out off own").split())
+
+
+def tokenize(s: str) -> set[str]:
+    return {w for w in "".join(c.lower() if c.isalnum() else " " for c in (s or "")).split()
+            if len(w) > 2 and w not in _STOP}
+
+
+def lexical_scores(query: str, texts: list[str]) -> np.ndarray:
+    """Jaccard overlap of the query tokens with each document's tokens. Deterministic; no model."""
+    q = tokenize(query)
+    if not q:
+        return np.zeros(len(texts), dtype="float32")
+    out = np.zeros(len(texts), dtype="float32")
+    for i, t in enumerate(texts):
+        d = tokenize(t)
+        out[i] = len(q & d) / len(q | d) if d else 0.0
+    return out

pen_stack-7.1.1/pen_stack/rag/ground.py

@@ -0,0 +1,130 @@
+"""PEN-RAG grounding for the chat's General lane (v7.1.1).
+
+Retrieval is ADDITIVE, not a gate. The General lane ANSWERS general and social questions by default (restoring the
+WS-HYBRID behaviour), clearly LABELLED as general knowledge - a labelled general answer is honest, not a
+fabrication, because it is never presented as a PEN-STACK result. Corpus retrieval, when it finds a relevant chunk,
+UPGRADES the answer to 'literature-cited' (with its sources). Abstention is the rare exception: it fires only for a
+SPECIFIC, unsourceable empirical / quantitative genome-writing claim the corpus does not cover and the engine is
+not computing - the genuine citation-or-silence case - so the model cannot dress a fabricated statistic as fact.
+
+Branches:
+  social    -> a friendly answer + a pointer to what the engine can compute (no retrieval, no abstention).
+  cited     -> a relevant corpus chunk was retrieved -> cite-or-silence answer, provenance 'literature-cited'.
+  general   -> background / textbook knowledge -> answer from the LLM, labelled (provenance 'general').
+  abstained -> a specific unsourceable empirical claim -> decline + redirect to the engine (provenance 'abstained').
+"""
+from __future__ import annotations
+
+import os
+import re
+
+from pen_stack.rag.embed import tokenize
+from pen_stack.rag.retrieve import retrieve
+
+# nomic-embed-text compresses cosines into a narrow high band, so an absolute cosine alone cannot separate a valid
+# in-corpus query from an off-topic one. The hybrid gate (cosine floor + lexical content-overlap, with a high-cosine
+# escape) decides only whether the corpus is a relevant SOURCE to cite - NOT whether to answer at all.
+THRESHOLD = float(os.getenv("PEN_RAG_THRESHOLD", "0.50"))
+_SEM_HIGH = float(os.getenv("PEN_RAG_THRESHOLD_HIGH", "0.72"))
+_LEX_THRESHOLD = float(os.getenv("PEN_RAG_LEX_THRESHOLD", "0.05"))
+
+# A social / conversational opener: answered naturally, never gated on the corpus.
+_SOCIAL = re.compile(r"^\s*(hi|hii+|hey+|hello+|yo|hola|sup|greetings|good (morning|afternoon|evening|day)|"
+                     r"thanks?|thank you|cheers|ok(ay)?|cool|nice|great|awesome|who are you|what are you|"
+                     r"how are you|what'?s up|introduce yourself|help)\b", re.I)
+
+# A SPECIFIC genome-writing QUANTITATIVE / empirical claim (not definitional / mechanistic background). When the
+# corpus has no source for it and it is in the general lane (so the engine is not computing it), we abstain +
+# redirect rather than let the model invent a statistic dressed up as fact.
+_SPECIFIC_EMPIRICAL = re.compile(
+    r"\b(efficienc\w*|integration rate|specificit\w*|on-?target rate|off-?target rate|knock-?in rate|titer|titre|"
+    r"affinit\w*|\bkd\b|ic50|fold[- ]improvement|expression level|copy number|"
+    r"percent\w* (of )?(integration|editing|knock|insertion))\b", re.I)
+
+_SOCIAL_REPLY = ("Hi - I'm PEN-STACK's genome-writing co-scientist. Ask a genome-writing question (where to insert, "
+                 "which writer, delivery, immune risk, off-targets, safety) and the engine computes a grounded, "
+                 "traceable answer. I can also answer general background questions, clearly labelled as such.")
+_GENERAL_NO_LLM = ("I can answer general background questions when the language model is enabled. Meanwhile, "
+                   "PEN-STACK's engine can compute grounded, specific answers for your case.")
+_LABEL = "Literature-cited (PEN-CHAT corpus) - not a PEN-STACK-computed result:"
+_ABSTAIN_SPECIFIC = ("I don't have a grounded source for that specific value and I won't invent one. PEN-STACK's "
+                     "engine can compute it from a concrete design - give me the gene, cell type, and cargo (or "
+                     "the writer/guide) and I'll run the site, writer-efficiency, or off-target tools.")
+
+
+def _is_grounded(query: str, hits: list[dict], method: str) -> bool:
+    """Is the corpus a relevant SOURCE to cite for this query?"""
+    if not hits:
+        return False
+    top = hits[0]
+    if not method.startswith("semantic"):
+        return top["score"] >= _LEX_THRESHOLD
+    lexical_overlap = bool(tokenize(query) & tokenize(top["text"]))
+    return top["score"] >= THRESHOLD and (lexical_overlap or top["score"] >= _SEM_HIGH)
+
+
+def _dedup_sources(hits: list[dict]) -> list[dict]:
+    seen, out = set(), []
+    for h in hits:
+        sid = h["source_id"]
+        if sid in seen:
+            continue
+        seen.add(sid)
+        out.append({"source_id": sid, "doi": h.get("doi") or None, "type": h["type"], "score": round(h["score"], 3)})
+    return out
+
+
+def _cited_answer(message: str, r: dict, allow_llm: bool) -> dict:
+    floor = THRESHOLD if r["method"].startswith("semantic") else _LEX_THRESHOLD
+    grounded_hits = [h for h in r["hits"] if h["score"] >= floor] or r["hits"][:2]
+    sources = _dedup_sources(grounded_hits)
+    deterministic = _LABEL + "\n\n" + "\n\n".join(
+        f"- {h['text']}  [{h['source_id']}{(' · ' + h['doi']) if h.get('doi') else ''}]" for h in grounded_hits)
+    base = {"status": "grounded", "sources": sources, "provenance": "literature-cited", "grounded": True,
+            "retrieval": r["method"], "top_score": r["top_score"]}
+    if allow_llm:
+        from pen_stack.web.llm import _enforce_grounding, _run_llm
+        from pen_stack.web.tools import extract_grounded_numbers
+        ctx = "\n\n".join(f"[{i + 1}] ({h['source_id']}{(' DOI ' + h['doi']) if h.get('doi') else ''}) {h['text']}"
+                          for i, h in enumerate(grounded_hits))
+        system = ("You answer ONLY from the SOURCES provided. After each claim, cite the bracketed [n] it came from. "
+                  "If the sources do not cover the question, say so. Never add a number, name, or vehicle that is "
+                  "not in the sources. Never present anything as a PEN-STACK-computed result.")
+        text, backend = _run_llm(f"SOURCES:\n{ctx}\n\nQUESTION: {message}\n\nAnswer concisely, citing [n].", system)
+        if text:
+            allow = extract_grounded_numbers({"hits": [h["text"] for h in grounded_hits]})
+            return {**base, "reply": _LABEL + "\n\n" + _enforce_grounding(text.strip(), allow), "backend": backend}
+    return {**base, "reply": deterministic, "backend": "deterministic"}
+
+
+def ground_general(message: str, *, allow_llm: bool = True) -> dict:
+    """Return the chat envelope for the General lane: answer (social / general / cited) by default; abstain only on a
+    specific unsourceable empirical claim."""
+    msg = (message or "").strip()
+    if _SOCIAL.match(msg):
+        return {"status": "social", "reply": _SOCIAL_REPLY, "sources": [], "provenance": "general",
+                "grounded": False, "backend": "deterministic", "retrieval": "none", "top_score": 0.0}
+
+    r = retrieve(msg, k=4)
+    if _is_grounded(msg, r["hits"], r["method"]):
+        return _cited_answer(msg, r, allow_llm)
+
+    # no corpus source. A specific empirical genome-writing claim -> abstain + redirect to the engine.
+    if _SPECIFIC_EMPIRICAL.search(msg):
+        return {"status": "abstained", "reply": _ABSTAIN_SPECIFIC, "sources": [], "provenance": "abstained",
+                "grounded": False, "backend": "deterministic", "retrieval": r["method"], "top_score": r["top_score"]}
+
+    # general / background knowledge -> ANSWER, clearly labelled as general (never a PEN-STACK result).
+    if allow_llm:
+        from pen_stack.web.llm import SYSTEM_GENERAL, _GENERAL_LABEL, _run_llm
+        text, backend = _run_llm(
+            f"USER: {msg}\n\nAnswer from general knowledge, clearly and at a graduate level. Do not present anything "
+            f"as a PEN-STACK result.", SYSTEM_GENERAL)
+        if text:
+            return {"status": "general", "reply": _GENERAL_LABEL + "\n\n" + text.strip(), "sources": [],
+                    "provenance": "general", "grounded": False, "backend": backend, "retrieval": r["method"],
+                    "top_score": r["top_score"]}
+    from pen_stack.web.llm import _GENERAL_LABEL
+    return {"status": "general", "reply": _GENERAL_LABEL + "\n\n" + _GENERAL_NO_LLM, "sources": [],
+            "provenance": "general", "grounded": False, "backend": "deterministic", "retrieval": r["method"],
+            "top_score": r["top_score"]}