pen-stack 6.6.0__tar.gz → 6.7.0__tar.gz

{pen_stack-6.6.0 → pen_stack-6.7.0}/CHANGELOG.md

@@ -3,6 +3,45 @@
 All notable changes to PEN-STACK are documented here. This file follows
 [Keep a Changelog](https://keepachangelog.com/) and the program's phase structure.
 
+## [6.7.0] - 2026-06-19 - PEN-EXPRESS: learned, trained-conformal Stage H + TPE-Bench
+
+**MINOR feature release.** Upgrades the digital twin's Stage H expression/outcome layer from a validation-failing
+closed-form **heuristic** to a **learned, trained-conformal, decomposable** position-effect model — and ships the
+held-out benchmark the expression capability never had. Wrap, don't rebuild: extends `twin` + `wgenome.uncertainty`
++ `wgenome.ood` + `benchmarks`. No fabrication: every metric is from a real CV run on real TRIP supervision, and
+the cross-cell-type transfer claim is **data-gated**, never faked.
+
+### Added — the learned model + trained conformal (WS-EXPRESS2: WS-D/M/U)
+- `pen_stack/twin/data/position_effect.py` — unified position-effect schema + dataset registry with **verified
+  accessions/DOIs** (TRIP live; PatchMPRA/MPIRE/lentiMPRA/Leemans registered + honestly `available=False` until
+  fetched), z-normalization within (dataset × cassette), domain-blocked + held-out-cell-type splits + leakage check.
+- `pen_stack/twin/position_effect.py` — `PositionEffectModel` (factored `f_cassette` + `g_context`, LightGBM),
+  `evaluate()` (chromosome-blocked CV vs the v3.x durability head + cassette-only, paired-bootstrap CIs,
+  separability), split-conformal calibration (`ConformalRegressor`, chromosome-Mondrian, OOD-widened),
+  `predict_stage_h()` serving seam. **Result (real TRIP):** expression ρ **0.428 → 0.469** (CI excludes 0);
+  held-out conformal coverage **0.885** vs 0.90 nominal.
+- `configs/twin/position_effect_conformal.json` — the shipped calibration (qhat + N + held-out coverage).
+- `scripts/p1_build_position_effect.py` — regenerates the model + conformal artifacts (real CV report).
+
+### Changed — Stage H integration (WS-I)
+- `pen_stack/twin/outcome.py` — when a chromatin context is supplied and the artifact is present, `predict_outcome`
+  serves the **learned trained-conformal** interval + `p_silenced` + OOD tier (`position_effect` block,
+  `stage_h_mode`); with no context/artifact it falls back to the heuristic band — **backward compatible** (the
+  v5.9 relative-scale contract is intact).
+
+### Added — TPE-Bench + controls (WS-B / WS-V)
+- `benchmarks/position_effect/` — TPE-Bench: a **sealed, SHA-locked** held-out-chromosome track + baseline
+  leaderboard (cassette-only / durability head / PEN-EXPRESS factored), submission harness. Leave-one-cell-type-out
+  transfer track scaffolded + **data-gated** (no fabricated transfer number).
+- `pen_stack/validate/{expr_controls,known_biology_expr,heldout_celltype_expr}.py` — label-shuffle→chance control,
+  H3K9me3-heterochromatin→silencing recovery, the data-gated transfer harness.
+- `tests/unit/test_ws_pe.py` — CI-safe (synthetic planted signal); the real-TRIP claim runs on a checkout, skips in CI.
+
+### Honesty
+- Public data **cannot** flip expression to ✅ (the v6.5 wall) — v6.7 ships the learned+calibrated upgrade + the
+  benchmark + the honest data-gating, not a manufactured checkmark. Cross-cell-type transfer needs the additional
+  human datasets (a data-acquisition step), reported as such. Wet-lab validation omitted by scope.
+
 ## [6.6.0] - 2026-06-16 - License-clean provenance (COSMIC → CancerMine)
 
 **MINOR — provenance refactor, no new science, no capability lost.** The shipped artifact now sources the

{pen_stack-6.6.0 → pen_stack-6.7.0}/CITATION.cff

@@ -1,8 +1,8 @@
 cff-version: 1.2.0
 message: "If you use PEN-STACK, please cite it as below."
 title: "PEN-STACK: open infrastructure for genome writing"
-version: 6.6.0
-date-released: 2026-06-12
+version: 6.7.0
+date-released: 2026-06-19
 authors:
   - family-names: "Mahaboob Ali"
     given-names: "Anees Ahmed"

{pen_stack-6.6.0 → pen_stack-6.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pen-stack
-Version: 6.6.0
+Version: 6.7.0
 Summary: Open infrastructure for genome writing: the Writable Genome atlas, the Writer Atlas, and the Write Planner.
 Author-email: Anees Ahmed Mahaboob Ali <ahmedaneesm@gmail.com>
 License: MIT
@@ -90,7 +90,7 @@ every design against rule-grounded mechanism, reports calibrated confidence, cit
 [![codecov](https://codecov.io/gh/ahmedanees-m/pen-stack/branch/main/graph/badge.svg)](https://codecov.io/gh/ahmedanees-m/pen-stack)
 [![License: MIT](https://img.shields.io/badge/License-MIT-informational.svg)](LICENSE)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
-[![Version](https://img.shields.io/badge/version-6.6.0-blue.svg)](CHANGELOG.md)
+[![Version](https://img.shields.io/badge/version-6.7.0-blue.svg)](CHANGELOG.md)
 [![Status](https://img.shields.io/badge/status-1.0%20First%20Stable-success.svg)](docs/STABILITY.md)
 [![Tests](https://img.shields.io/badge/tests-378%20passing-success.svg)](tests/)
 [![Lint: ruff](https://img.shields.io/badge/lint-ruff-purple.svg)](https://github.com/astral-sh/ruff)
@@ -148,6 +148,32 @@ demonstrated (v5.12) and the benchmark went public (v5.13).
 > unknown funnel remains — made legible (scope flags, known-unknowns, honest baselines, no fabrication), not
 > hidden.
 
+## What is new in v6.7 — PEN-EXPRESS (a learned, trained-conformal Stage H + TPE-Bench)
+
+Stage H's expression/outcome layer was the stack's weakest link: a closed-form **heuristic** that *failed*
+independent validation (ρ=0.12 vs Damdindorj 2014), with an interval the code itself labelled "**NOT a trained
+conformal interval**." v6.7 replaces it — **wrapping, not rebuilding** the digital twin:
+
+- **A learned cassette × context position-effect model** (`pen_stack/twin/position_effect.py`) — factored and
+  decomposable (`f_cassette` + `g_context`), trained on the **real TRIP supervision** (Akhtar 2013; GEO
+  GSE49806/49807). On chromosome-blocked CV it **beats the v3.x durability head** on expression (ρ **0.428 →
+  0.469**, paired-bootstrap CI excludes 0) and matches it on silencing — reported honestly, with the separability
+  result (additive `f_cassette + g_context` suffices at this N).
+- **Stage H is now trained-conformal.** The model ships a split-conformal calibration (`ConformalRegressor`,
+  chromosome-Mondrian, OOD-widened): **held-out coverage 0.885 vs 0.90 nominal** — the named gap, closed. When a
+  chromatin context is supplied, `predict_outcome` serves the calibrated interval + `p_silenced` + OOD tier; with
+  no context (or no artifact) it falls back to the heuristic — **backward compatible**.
+- **TPE-Bench** (`benchmarks/position_effect/`) — the held-out benchmark the expression capability never had: a
+  **sealed, SHA-locked** held-out-chromosome split + a baseline leaderboard (cassette-only / durability head /
+  PEN-EXPRESS). A leave-one-**cell-type**-out transfer track is scaffolded and **honestly data-gated** until the
+  additional human datasets (PatchMPRA / MPIRE / lentiMPRA / Leemans) are fetched — **no transfer number is
+  fabricated**. Controls: label-shuffle → chance; known-biology recovery (H3K9me3 ↑ → silencing ↑).
+
+The honest limit (the v6.5 wall, restated): public data alone cannot flip expression to ✅ — that needs measured
+outcome data at a power the literature does not provide. v6.7 ships the learned+calibrated upgrade + the
+benchmark; the green checkmark stays earned, not manufactured. See [docs/position_effect.md](docs/position_effect.md)
+and [docs/tpe_bench.md](docs/tpe_bench.md).
+
 ## What is new in v6.4 — Live Oracles (the foundation models actually execute)
 
 The foundation-model oracles now **run for real**, not just defer: **ViennaRNA** (in-process), **AlphaGenome**
@@ -976,18 +1002,6 @@ independently verified.
 - **Grounded services** - every quantitative answer comes from a validated tool call (never a language
   model); the living database never auto-edits the atlas; clinical directives are refused.
 
-## Papers and phases
-
-| # | Title | Phase | Status |
-|---|---|---|---|
-| 1 (flagship) | The Writable Genome: a predictive, writer-aware atlas of safe & durable insertion sites | 1 | complete |
-| 2 (platform) | PEN-STACK: unified open infrastructure for non-destructive genome writing | 2 | complete |
-| 3 (capstone) | The Write Planner: end-to-end inverse design of genomic writes | 3 | complete |
-| 4 (beachhead) | Genome-wide off-target prediction for RNA-guided bridge recombinases | 1.5 | complete |
-| M1 (v3.1) | Writable Genome hardened: strong baselines, AlphaGenome sequence + 3D structural-risk axis | v3.1 B,C,D,F | complete |
-| M2 (v3.1) | The Genome-Writing Bench + PEN-Agent: the writing-side benchmark and a grounded agent | v3.1 E | complete |
-| M3 (v3.1) | Multiplex translocation-risk + bridge-RNA guide QC | v3.1 G | complete |
-
 The v3.1 cycle (workstreams A-H) is recorded in `CHANGELOG.md`, `docs/positioning.md`, and the SHA-locked
 `prereg/ws_*.yaml`; preprint drafts are in `manuscripts/`.
 

{pen_stack-6.6.0 → pen_stack-6.7.0}/README.md

@@ -15,7 +15,7 @@ every design against rule-grounded mechanism, reports calibrated confidence, cit
 [![codecov](https://codecov.io/gh/ahmedanees-m/pen-stack/branch/main/graph/badge.svg)](https://codecov.io/gh/ahmedanees-m/pen-stack)
 [![License: MIT](https://img.shields.io/badge/License-MIT-informational.svg)](LICENSE)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
-[![Version](https://img.shields.io/badge/version-6.6.0-blue.svg)](CHANGELOG.md)
+[![Version](https://img.shields.io/badge/version-6.7.0-blue.svg)](CHANGELOG.md)
 [![Status](https://img.shields.io/badge/status-1.0%20First%20Stable-success.svg)](docs/STABILITY.md)
 [![Tests](https://img.shields.io/badge/tests-378%20passing-success.svg)](tests/)
 [![Lint: ruff](https://img.shields.io/badge/lint-ruff-purple.svg)](https://github.com/astral-sh/ruff)
@@ -73,6 +73,32 @@ demonstrated (v5.12) and the benchmark went public (v5.13).
 > unknown funnel remains — made legible (scope flags, known-unknowns, honest baselines, no fabrication), not
 > hidden.
 
+## What is new in v6.7 — PEN-EXPRESS (a learned, trained-conformal Stage H + TPE-Bench)
+
+Stage H's expression/outcome layer was the stack's weakest link: a closed-form **heuristic** that *failed*
+independent validation (ρ=0.12 vs Damdindorj 2014), with an interval the code itself labelled "**NOT a trained
+conformal interval**." v6.7 replaces it — **wrapping, not rebuilding** the digital twin:
+
+- **A learned cassette × context position-effect model** (`pen_stack/twin/position_effect.py`) — factored and
+  decomposable (`f_cassette` + `g_context`), trained on the **real TRIP supervision** (Akhtar 2013; GEO
+  GSE49806/49807). On chromosome-blocked CV it **beats the v3.x durability head** on expression (ρ **0.428 →
+  0.469**, paired-bootstrap CI excludes 0) and matches it on silencing — reported honestly, with the separability
+  result (additive `f_cassette + g_context` suffices at this N).
+- **Stage H is now trained-conformal.** The model ships a split-conformal calibration (`ConformalRegressor`,
+  chromosome-Mondrian, OOD-widened): **held-out coverage 0.885 vs 0.90 nominal** — the named gap, closed. When a
+  chromatin context is supplied, `predict_outcome` serves the calibrated interval + `p_silenced` + OOD tier; with
+  no context (or no artifact) it falls back to the heuristic — **backward compatible**.
+- **TPE-Bench** (`benchmarks/position_effect/`) — the held-out benchmark the expression capability never had: a
+  **sealed, SHA-locked** held-out-chromosome split + a baseline leaderboard (cassette-only / durability head /
+  PEN-EXPRESS). A leave-one-**cell-type**-out transfer track is scaffolded and **honestly data-gated** until the
+  additional human datasets (PatchMPRA / MPIRE / lentiMPRA / Leemans) are fetched — **no transfer number is
+  fabricated**. Controls: label-shuffle → chance; known-biology recovery (H3K9me3 ↑ → silencing ↑).
+
+The honest limit (the v6.5 wall, restated): public data alone cannot flip expression to ✅ — that needs measured
+outcome data at a power the literature does not provide. v6.7 ships the learned+calibrated upgrade + the
+benchmark; the green checkmark stays earned, not manufactured. See [docs/position_effect.md](docs/position_effect.md)
+and [docs/tpe_bench.md](docs/tpe_bench.md).
+
 ## What is new in v6.4 — Live Oracles (the foundation models actually execute)
 
 The foundation-model oracles now **run for real**, not just defer: **ViennaRNA** (in-process), **AlphaGenome**
@@ -901,18 +927,6 @@ independently verified.
 - **Grounded services** - every quantitative answer comes from a validated tool call (never a language
   model); the living database never auto-edits the atlas; clinical directives are refused.
 
-## Papers and phases
-
-| # | Title | Phase | Status |
-|---|---|---|---|
-| 1 (flagship) | The Writable Genome: a predictive, writer-aware atlas of safe & durable insertion sites | 1 | complete |
-| 2 (platform) | PEN-STACK: unified open infrastructure for non-destructive genome writing | 2 | complete |
-| 3 (capstone) | The Write Planner: end-to-end inverse design of genomic writes | 3 | complete |
-| 4 (beachhead) | Genome-wide off-target prediction for RNA-guided bridge recombinases | 1.5 | complete |
-| M1 (v3.1) | Writable Genome hardened: strong baselines, AlphaGenome sequence + 3D structural-risk axis | v3.1 B,C,D,F | complete |
-| M2 (v3.1) | The Genome-Writing Bench + PEN-Agent: the writing-side benchmark and a grounded agent | v3.1 E | complete |
-| M3 (v3.1) | Multiplex translocation-risk + bridge-RNA guide QC | v3.1 G | complete |
-
 The v3.1 cycle (workstreams A-H) is recorded in `CHANGELOG.md`, `docs/positioning.md`, and the SHA-locked
 `prereg/ws_*.yaml`; preprint drafts are in `manuscripts/`.
 

pen_stack-6.7.0/benchmarks/position_effect/README.md

@@ -0,0 +1,60 @@
+# TPE-Bench — the Position-Effect / Expression track
+
+The **expression** capability of PEN-STACK never had a held-out benchmark. TPE-Bench is the position-effect
+track for the [Genome-Writing Challenge](../genome_writing_challenge/) (v6.7 PEN-EXPRESS): given a genomic
+**chromatin context** + **cassette**, predict the **integrated-reporter expression** (and silencing) — scored on
+a **sealed held-out split** whose labels the submitter never sees.
+
+## Why
+
+Position-effect (where an integrated cassette lands → how strongly/durably it expresses) is the writing-relevant
+quantity no safe-harbour resource predicts. TRIP (Akhtar 2013) supervises it directly. TPE-Bench seals a held-out
+split and anchors a baseline leaderboard, so others can build *to* a calibrated expression predictor.
+
+## Tracks
+
+| Track | Status | What is held out | Metric |
+|---|---|---|---|
+| `chrom_holdout` | **LIVE** | whole chromosomes (`chr2, chr5, chr14, chrX`), frozen + SHA-locked in `split.json` | Spearman ρ (expression) + AUROC (silenced) |
+| `celltype_holdout` | **DATA-GATED** | leave-one-cell-type-out (the headline transfer track) | — |
+
+`celltype_holdout` is the headline cross-cell-type transfer test. With a single available position-effect cell
+type (mESC) it reports `data_gated` **honestly** and activates once PatchMPRA / MPIRE / lentiMPRA / Leemans are
+fetched — **no transfer number is fabricated** until then.
+
+## Baseline leaderboard (sealed `chrom_holdout`, n_test = 2257)
+
+| Predictor | Expression ρ | Silenced AUROC |
+|---|---|---|
+| cassette-only (f_cassette) | 0.178 | — |
+| context-only (v3.x durability head) | 0.431 | 0.660 |
+| **PEN-EXPRESS factored (f_cassette + g_context)** | **0.475** | 0.660 |
+
+The factored model's gain is on **expression** (the cassette term lifts ρ 0.431 → 0.475 on the sealed test); the
+silenced classifier matches the durability head (the silencing question is chromatin-driven — reported honestly,
+not inflated). See `../../out/position_effect_report.json` for the full CV report + bootstrap CIs.
+
+## How to submit
+
+```python
+from benchmarks.position_effect.harness import Submission, evaluate
+
+def my_predict(public_input: dict):
+    # public_input = {task_id, family, cassette, chromatin_features:{H3K27ac,...}, instructions}; label hidden
+    return {"expression": 0.0, "p_silenced": 0.5}   # return your prediction (abstain-safe)
+
+print(evaluate(Submission(name="my-model", predict_fn=my_predict)))
+```
+
+Reference baselines: `python -c "from benchmarks.position_effect.harness import baseline_leaderboard as b; print(b())"`.
+
+## Rules
+
+- **Sealed + SHA-locked.** `split.json` (held-out chromosomes) is frozen and checksummed (`SHA256SUMS`) before
+  model selection — verify with `sha256sum -c SHA256SUMS`.
+- **No circular labels.** The label is the **measured** TRIP expression, never a submitter claim.
+- **Leakage-controlled.** Held out by whole chromosome (nearby integrations share chromatin).
+- **Honest data-gating.** The transfer track abstains until ≥2 cell types exist; no fabricated number.
+- **Deterministic.** PEN-EXPRESS anchors the leaderboard.
+
+Data: TRIP (Akhtar et al., *Cell* 2013; GEO GSE49806/GSE49807; trip.nki.nl). License: see `DATA_LICENSES.md`.

pen_stack-6.7.0/benchmarks/position_effect/SHA256SUMS

@@ -0,0 +1 @@
+2fc12cbf28531e68f6e25586da1d84003eeb6a098c9015998ce03f518546bc8a  split.json

pen_stack-6.7.0/docs/cards/position_effect_data.md

@@ -0,0 +1,36 @@
+# Data card — position-effect supervision (v6.7 PEN-EXPRESS)
+
+The unified table behind Stage H (`pen_stack/twin/data/position_effect.py`). One row = one integrated reporter /
+element measurement. Schema: `dataset, organism, cell_type, chrom, pos, cassette, expression_raw, expression_z,
+silenced, <chromatin features>`. `expression_z` is z-scored within (dataset × cassette).
+
+## Dataset registry (verified accessions — 2026-06-19 verification pass)
+
+| Dataset | Citation | DOI | Accession | Cell types | Status in v6.7 |
+|---|---|---|---|---|---|
+| **TRIP** | Akhtar et al., *Cell* 2013 | 10.1016/j.cell.2013.07.018 | GEO **GSE49806** (tetO) + **GSE49807** (mPGK); trip.nki.nl | mESC | **LIVE** (n=11,433) |
+| PatchMPRA | Maricque, Chaudhari & Cohen, *Nat Biotechnol* 2019 | 10.1038/nbt.4285 | GEO (per paper) | mESC | registered, data-gated |
+| MPIRE | Hong et al., *Nat Commun* 2024 | 10.1038/s41467-024-52599-6 | GEO **GSE223403**; github.com/claricehong/MPIRE_insulators | K562 | registered, data-gated |
+| lentiMPRA | Agarwal et al., *Nature* **639**:411–420 (2025) | 10.1038/s41586-024-08430-9 | ENCODE + GEO; bioRxiv 2023.03.05.531189 | HepG2, K562, WTC11 | registered, data-gated |
+| Leemans | Leemans et al., *Cell* 2019 | 10.1016/j.cell.2019.03.009 | GEO (per paper); van Steensel lab | K562 | registered, data-gated |
+
+> The bioRxiv id `2023.03.05.531189` is **lentiMPRA's** (Agarwal → *Nature* 2025), not e2MPRA/ccMPRA — corrected
+> in the 2026-06-19 citation verification pass.
+
+## TRIP (the LIVE supervision)
+
+- **What:** thousands of identical reporters integrated in parallel across the mESC genome; each row is a genomic
+  position with normalized expression → the position effect on an integrated cassette (the writing-relevant quantity).
+- **Columns used:** `chrom, pos, promoter` (cassette: tetO 10,903 / mPGK 530), `expression` (log2, −13…+9),
+  `silenced` (low-expression tail, 25%), + 5 chromatin marks (H3K27ac, H3K4me1, H3K4me3, H3K9me3, H3K27me3).
+- **Provenance:** same `trip_with_chromatin.parquet` the v3.x durability head trains on (regenerated by
+  `pen_stack/data/ingest_trip.py` + chromatin extraction).
+- **License:** TRIP data is from a public GEO deposit (Akhtar 2013); see `DATA_LICENSES.md`. Raw data is gitignored
+  (`data/external/`), pulled from the VM for local runs; the shipped wheel carries the loaders + accessions, not the data.
+
+## Honest limits
+
+- **Mouse, single cell type.** TRIP is mESC. The model learns `chromatin → expression` (never a coordinate), so it
+  *applies* to a human epigenome — but the cross-cell-type **transfer** is unproven until ≥2 cell types are unified.
+  Until then the transfer track is **data-gated** (no fabricated number).
+- **Reporter-based** (GFP-class), relative not absolute. Titer / % of normal stay known-unknowns.

pen_stack-6.7.0/docs/position_effect.md

@@ -0,0 +1,57 @@
+# Stage H — the learned, trained-conformal position-effect model (v6.7 PEN-EXPRESS)
+
+Stage H predicts how strongly an integrated cassette expresses in its **chromatin context** (the position effect).
+Through v6.6 this was a closed-form **heuristic** with a heuristic ±0.20 band that the code itself labelled *"NOT
+a trained conformal interval"*, and which **failed** independent validation (ρ=0.12 vs Damdindorj 2014). v6.7
+replaces the model behind Stage H with a **learned, decomposable, trained-conformal** one — wrapping the digital
+twin, not rebuilding it.
+
+## The model
+
+`pen_stack/twin/position_effect.py::PositionEffectModel` is **factored and decomposable**:
+
+```
+E_raw  ≈  f_cassette(cassette)            # the cassette's intrinsic strength (per-cassette mean)
+       +  g_context(chromatin features)    # the position effect — a LightGBM on local chromatin
+       (+ h_interaction, reported)         # does the context function differ by cassette? (separability)
+```
+
+`g_context` is supervised on the residual `E_raw − f_cassette`, so it learns the *position* effect on a scale
+comparable across cassettes. A `silenced` classifier shares the chromatin features. The model is wrapped with the
+**existing** `wgenome.uncertainty.ConformalRegressor` (chromosome-Mondrian split-conformal) and `wgenome.ood.OODDetector`
+— so a prediction is a **calibrated interval that widens out of distribution**.
+
+## Results (real, on TRIP supervision — Akhtar 2013, GEO GSE49806/49807, mESC, n=11,433)
+
+Chromosome-blocked GroupKFold; paired bootstrap 95% CIs. *Every number is from a real CV run — no fabrication.*
+
+| Metric | cassette-only | context-only (v3.x durability head) | **PEN-EXPRESS factored** | Δ vs head (CI) |
+|---|---|---|---|---|
+| Expression Spearman ρ | 0.032 | 0.427 | **0.469** | +0.041 [0.036, 0.046] ✅ excludes 0 |
+| Silenced AUROC | — | 0.647 | 0.651 | +0.004 [0.001, 0.007] ✅ excludes 0 |
+
+- **Gate G-M passed:** the factored model beats the durability head (CI excludes 0) → it serves behind Stage H.
+- **Separability:** interaction adds **−0.002** R² → *additive `f_cassette + g_context` suffices at this N*
+  (reported honestly; the cassette term lifts expression, the silencing question is chromatin-driven).
+- **Trained-conformal (the named gap, closed):** split-conformal (α=0.10) → **held-out coverage 0.885 vs 0.90
+  nominal** (within tolerance), qhat=5.50 on the log2 scale. Coverage is measured on a **half-chromosome held-out**
+  split, not on the calibration set.
+
+## Stage H integration (`twin/outcome.py`)
+
+`predict_outcome(design, cell_state)` now:
+- **With a chromatin context** (`design["chromatin_features"]`) **and the model artifact present** → serves the
+  learned **trained-conformal** interval + `p_silenced` + OOD tier in a `position_effect` block; `stage_h_mode =
+  "learned_trained_conformal"`.
+- **Without a context (or artifact)** → the closed-form heuristic band, exactly as before — **backward compatible**
+  (the v5.9 relative-scale contract and all prior tests are intact).
+
+## Honest limits
+
+- **Single-context supervision.** TRIP is mESC. The cross-cell-type **transfer** claim is **data-gated** — see
+  [tpe_bench.md](tpe_bench.md); no transfer number is fabricated until PatchMPRA/MPIRE/lentiMPRA/Leemans are fetched.
+- **Public data cannot earn the ✅** (the v6.5 wall). v6.7 ships the learned+calibrated upgrade + the benchmark, not
+  a manufactured validated axis. Titer / absolute expression / phenotype stay **known-unknowns**.
+- The model artifact (`models/position_effect.pkl`) is gitignored; regenerate it with
+  `python scripts/p1_build_position_effect.py` (the shipped calibration `configs/twin/position_effect_conformal.json`
+  is committed). Without the artifact, Stage H falls back to the heuristic.

pen_stack-6.7.0/docs/tpe_bench.md

@@ -0,0 +1,48 @@
+# TPE-Bench — the position-effect / expression benchmark (v6.7)
+
+The **expression** capability never had a held-out benchmark. TPE-Bench fills that gap as a track of the
+[Genome-Writing Challenge](../benchmarks/genome_writing_challenge/): given a genomic **chromatin context** +
+**cassette**, predict the **integrated-reporter expression** (and silencing), scored on a **sealed** split.
+
+Location: `benchmarks/position_effect/` (`harness.py`, `split.json`, `SHA256SUMS`, `README.md`).
+
+## Two tracks
+
+| Track | Status | Held out | Metric |
+|---|---|---|---|
+| `chrom_holdout` | **LIVE** | whole chromosomes `chr2, chr5, chr14, chrX` (frozen + SHA-locked) | Spearman ρ + AUROC |
+| `celltype_holdout` | **DATA-GATED** | leave-one-cell-type-out (the headline transfer test) | — |
+
+`celltype_holdout` is the cross-cell-type transfer test. With one available position-effect cell type (mESC) it
+returns `data_gated` **honestly** and activates once PatchMPRA / MPIRE / lentiMPRA / Leemans are fetched — **no
+transfer number is fabricated**.
+
+## Baseline leaderboard (`chrom_holdout`, sealed, n_test = 2257)
+
+| Predictor | Expression ρ | Silenced AUROC |
+|---|---|---|
+| cassette-only | 0.178 | — |
+| context-only (v3.x durability head) | 0.431 | 0.660 |
+| **PEN-EXPRESS factored** | **0.475** | 0.660 |
+
+The factored model's gain is on **expression** (ρ 0.431 → 0.475 on the sealed held-out chromosomes); the silencing
+classifier matches the durability head (chromatin-driven) — reported honestly, not inflated.
+
+## Discipline
+
+- **Sealed + SHA-locked.** `split.json` is frozen and checksummed (`SHA256SUMS`) **before model selection**;
+  verify with `sha256sum -c benchmarks/position_effect/SHA256SUMS`.
+- **Non-circular labels.** The label is the **measured** TRIP expression, never a submitter claim.
+- **Leakage-controlled.** Held out by whole chromosome (nearby integrations share chromatin).
+- **Honest data-gating.** The transfer track abstains until ≥2 cell types exist.
+
+## Submit
+
+```python
+from benchmarks.position_effect.harness import Submission, evaluate
+def predict(pi):  # pi = {task_id, cassette, chromatin_features, instructions}; label hidden
+    return {"expression": 0.0, "p_silenced": 0.5}
+print(evaluate(Submission("my-model", predict)))
+```
+
+Reproduce the baselines: `python -c "from benchmarks.position_effect.harness import baseline_leaderboard as b; print(b())"`.

{pen_stack-6.6.0 → pen_stack-6.7.0}/pen_stack/__init__.py

@@ -1,2 +1,2 @@
 """PEN-STACK v3.0 - open infrastructure for genome writing."""
-__version__ = "6.6.0"
+__version__ = "6.7.0"

pen_stack-6.7.0/pen_stack/twin/data/__init__.py

@@ -0,0 +1,12 @@
+"""Position-effect / expression supervision data (v6.7 PEN-EXPRESS, WS-D)."""
+from pen_stack.twin.data.position_effect import (  # noqa: F401
+    DATASETS,
+    FEATURE_COLS,
+    SCHEMA,
+    available_datasets,
+    blocked_splits,
+    heldout_celltype_splits,
+    leakage_report,
+    load_position_effect,
+    normalize_within,
+)

pen_stack-6.7.0/pen_stack/twin/data/position_effect.py

@@ -0,0 +1,224 @@
+"""Position-effect / expression supervision — the unified table behind Stage H (v6.7 PEN-EXPRESS, WS-D).
+
+ONE schema over the scattered human/mouse position-effect datasets, so a single learned cassette x context
+model (`twin.position_effect.PositionEffectModel`) can be trained and a held-out-cell-type benchmark
+(`benchmarks/position_effect/`, TPE-Bench) can be sealed. Wrap, do not rebuild: TRIP supervision is the same
+table the v3.x durability head already uses; this module just unifies it with the other sources and adds the
+cassette identity + cross-dataset normalization + leakage-controlled splits.
+
+NO FABRICATION. Each dataset is registered with its verified accession/DOI and a loader. A dataset whose raw
+data is not present is reported `available=False` (an honest "not fetched"), never silently imputed. The only
+dataset wired live in v6.7 is **TRIP** (Akhtar 2013, on the VM + pulled locally); the additional human
+position-effect sources (PatchMPRA, MPIRE, lentiMPRA, Leemans) are registered with their accessions and a
+loader contract, and become available when their raw data is fetched — the cross-cell-type *transfer* claim is
+explicitly gated on that acquisition (documented, not asserted).
+
+Canonical schema (one row = one integrated reporter / element measurement):
+    dataset, organism, cell_type, chrom, pos, cassette, expression_raw, expression_z, silenced, <features...>
+`expression_z` is z-scored WITHIN (dataset x cassette) so a promoter's strength does not leak across datasets
+and the model learns the *context* effect on a comparable scale.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Callable
+
+import numpy as np
+import pandas as pd
+
+from pen_stack._resources import project_root
+
+# canonical chromatin context features (subset present per dataset; missing -> excluded, never imputed silently)
+FEATURE_COLS = ["atac", "dnase", "H3K27ac", "H3K4me1", "H3K4me3", "H3K9me3", "H3K27me3", "H3K36me3"]
+
+SCHEMA = ["dataset", "organism", "cell_type", "chrom", "pos", "cassette",
+          "expression_raw", "expression_z", "silenced"]
+
+
+# --------------------------------------------------------------------------------------------------
+# dataset registry — verified accessions/DOIs (see Final_Part_v3.0 verification pass, 2026-06-19)
+# --------------------------------------------------------------------------------------------------
+@dataclass
+class Dataset:
+    name: str
+    citation: str
+    doi: str
+    accession: str
+    organism: str
+    cell_types: tuple[str, ...]
+    role: str                       # "position_effect" (locus->expr) | "cassette_activity" (CRE/MPRA)
+    loader: Callable[[Path], pd.DataFrame] | None = None
+    note: str = ""
+    _rel: str = ""                  # repo-relative raw path the loader reads (for availability check)
+
+    def available(self, root: Path | None = None) -> bool:
+        if self.loader is None or not self._rel:
+            return False
+        root = root or project_root()
+        return (root / self._rel).exists()
+
+
+def _load_trip(root: Path) -> pd.DataFrame:
+    """TRIP (Akhtar 2013) mESC integrations + mES chromatin marks -> canonical schema.
+
+    The parquet is the same `trip_with_chromatin.parquet` the durability head trains on (chrom, pos, promoter,
+    expression [log2], silenced, + 5 histone marks). cassette = promoter; cell_type = mESC.
+    """
+    path = root / "data/external/trip/trip_with_chromatin.parquet"
+    df = pd.read_parquet(path)
+    marks = [c for c in FEATURE_COLS if c in df.columns]
+    out = pd.DataFrame({
+        "dataset": "TRIP_Akhtar2013",
+        "organism": "mouse",
+        "cell_type": "mESC",
+        "chrom": df["chrom"].astype(str),
+        "pos": df["pos"].astype("int64"),
+        "cassette": df["promoter"].astype(str),
+        "expression_raw": df["expression"].astype(float),
+        "silenced": df["silenced"].astype(bool),
+    })
+    for m in marks:
+        out[m] = df[m].astype(float)
+    return out
+
+
+def _loader_not_fetched(name: str, accession: str) -> Callable[[Path], pd.DataFrame]:
+    def _raise(_root: Path) -> pd.DataFrame:
+        raise FileNotFoundError(
+            f"{name}: raw data not fetched (accession {accession}). v6.7 wires TRIP only; this source is "
+            f"registered with its accession + loader contract and becomes available once fetched. The "
+            f"cross-cell-type transfer claim is gated on this acquisition (honest, not asserted).")
+    return _raise
+
+
+DATASETS: dict[str, Dataset] = {
+    "TRIP_Akhtar2013": Dataset(
+        name="TRIP_Akhtar2013",
+        citation="Akhtar et al., Cell 2013 (thousands of reporters integrated in parallel)",
+        doi="10.1016/j.cell.2013.07.018",
+        accession="GEO GSE49806 (tetO) + GSE49807 (mPGK); trip.nki.nl",
+        organism="mouse", cell_types=("mESC",), role="position_effect",
+        loader=_load_trip, _rel="data/external/trip/trip_with_chromatin.parquet",
+        note="position effect on an integrated cassette — the writing-relevant supervision (LIVE in v6.7)."),
+    "PatchMPRA_Maricque2019": Dataset(
+        name="PatchMPRA_Maricque2019",
+        citation="Maricque, Chaudhari & Cohen, Nat Biotechnol 2019 (genomically integrated MPRA)",
+        doi="10.1038/nbt.4285", accession="GEO (per paper)",
+        organism="mouse", cell_types=("mESC",), role="position_effect",
+        loader=_loader_not_fetched("PatchMPRA_Maricque2019", "10.1038/nbt.4285"),
+        note="cassette x context separability evidence (data-gated)."),
+    "MPIRE_Hong2024": Dataset(
+        name="MPIRE_Hong2024",
+        citation="Hong et al., Nat Commun 2024 (massively parallel insulator-activity MPRA)",
+        doi="10.1038/s41467-024-52599-6", accession="GEO GSE223403; github.com/claricehong/MPIRE_insulators",
+        organism="human", cell_types=("K562",), role="position_effect",
+        loader=_loader_not_fetched("MPIRE_Hong2024", "GSE223403"),
+        note="human K562 position/insulator context (data-gated)."),
+    "lentiMPRA_Agarwal2025": Dataset(
+        name="lentiMPRA_Agarwal2025",
+        citation="Agarwal et al., Nature 639:411-420 (2025) (~680k regulatory elements)",
+        doi="10.1038/s41586-024-08430-9", accession="ENCODE portal + GEO; bioRxiv 2023.03.05.531189",
+        organism="human", cell_types=("HepG2", "K562", "WTC11"), role="cassette_activity",
+        loader=_loader_not_fetched("lentiMPRA_Agarwal2025", "10.1038/s41586-024-08430-9"),
+        note="cassette/CRE activity supervision across 3 human cell types (data-gated)."),
+    "Leemans2019": Dataset(
+        name="Leemans2019",
+        citation="Leemans et al., Cell 2019 (promoter-intrinsic + local chromatin determine repression in LADs)",
+        doi="10.1016/j.cell.2019.03.009", accession="GEO (per paper); van Steensel lab",
+        organism="human", cell_types=("K562",), role="position_effect",
+        loader=_loader_not_fetched("Leemans2019", "10.1016/j.cell.2019.03.009"),
+        note="human LAD-repression context (data-gated)."),
+}
+
+
+def available_datasets(root: Path | None = None) -> list[str]:
+    root = root or project_root()
+    return [k for k, d in DATASETS.items() if d.available(root)]
+
+
+# --------------------------------------------------------------------------------------------------
+# normalization + loader
+# --------------------------------------------------------------------------------------------------
+def normalize_within(df: pd.DataFrame, by=("dataset", "cassette"), col: str = "expression_raw",
+                     out: str = "expression_z") -> pd.DataFrame:
+    """z-score expression within each (dataset x cassette) group, so a cassette's intrinsic strength does not
+    leak across datasets and the model is supervised on the *context* deviation on a comparable scale. A
+    singleton/zero-variance group maps to 0.0 (no spurious scale)."""
+    df = df.copy()
+    def _z(s: pd.Series) -> pd.Series:
+        sd = s.std(ddof=0)
+        return (s - s.mean()) / sd if sd and sd > 1e-12 else pd.Series(0.0, index=s.index)
+    df[out] = df.groupby(list(by))[col].transform(_z)
+    return df
+
+
+def load_position_effect(datasets: list[str] | None = None, root: Path | None = None,
+                         require: bool = False) -> pd.DataFrame:
+    """Unified position-effect table. Loads every AVAILABLE registered dataset (or the named subset),
+    concatenates to the canonical schema, and z-normalizes expression within (dataset x cassette).
+
+    `require=True` raises if a requested dataset is unavailable; default skips unavailable ones (logging the
+    skip in the returned frame's `.attrs['skipped']`) — honest about what is and is not in the table.
+    """
+    root = root or project_root()
+    names = datasets or list(DATASETS)
+    frames, skipped = [], []
+    for name in names:
+        d = DATASETS[name]
+        if not d.available(root):
+            skipped.append(name)
+            if require:
+                d.loader(root)            # raise the informative FileNotFoundError
+            continue
+        frames.append(d.loader(root))
+    if not frames:
+        raise FileNotFoundError(
+            f"no position-effect dataset available under {root}. Available registry: {list(DATASETS)}; "
+            f"skipped (not fetched): {skipped}. Pull TRIP via scratch/v67_pull_trip.py or set PEN_STACK_HOME.")
+    df = pd.concat(frames, ignore_index=True)
+    df = normalize_within(df)
+    df.attrs["skipped"] = skipped
+    df.attrs["datasets"] = [f["dataset"].iloc[0] for f in frames]
+    # reorder: schema first, then whatever feature columns are present
+    feats = [c for c in FEATURE_COLS if c in df.columns]
+    return df[[c for c in SCHEMA if c in df.columns] + feats]
+
+
+# --------------------------------------------------------------------------------------------------
+# leakage-controlled splits
+# --------------------------------------------------------------------------------------------------
+def blocked_splits(df: pd.DataFrame, n_splits: int = 5, group: str = "chrom",
+                   seed: int = 20260619) -> list[tuple[np.ndarray, np.ndarray]]:
+    """Domain-blocked CV: no `group` (default chromosome) value appears in both train and test of a fold —
+    the leakage control the position-effect task needs (nearby integrations share chromatin)."""
+    from sklearn.model_selection import GroupKFold
+    g = df[group].astype("category").cat.codes.to_numpy()
+    k = min(n_splits, len(np.unique(g)))
+    return list(GroupKFold(n_splits=k).split(df, groups=g))
+
+
+def heldout_celltype_splits(df: pd.DataFrame) -> list[tuple[str, np.ndarray, np.ndarray]]:
+    """Leave-one-cell-type-out: train on all but one cell type, test on the held-out one. This is the headline
+    transfer evaluation. With a single available cell type it returns [] and the caller reports the transfer
+    axis as data-gated (honest), never a fabricated transfer number."""
+    cts = sorted(df["cell_type"].unique())
+    if len(cts) < 2:
+        return []
+    out = []
+    idx = np.arange(len(df))
+    for ct in cts:
+        te = idx[df["cell_type"].to_numpy() == ct]
+        tr = idx[df["cell_type"].to_numpy() != ct]
+        out.append((ct, tr, te))
+    return out
+
+
+def leakage_report(df: pd.DataFrame, splits: list[tuple[np.ndarray, np.ndarray]],
+                   group: str = "chrom") -> dict:
+    """Verify no `group` value is co-located across the train/test of any fold (the split's integrity claim)."""
+    bad = 0
+    for tr, te in splits:
+        if set(df.iloc[tr][group]) & set(df.iloc[te][group]):
+            bad += 1
+    return {"n_folds": len(splits), "folds_with_leakage": bad, "clean": bad == 0, "group": group}