if-split 0.2.0__tar.gz → 0.3.0__tar.gz

if_split-0.3.0/CHANGELOG.md

@@ -0,0 +1,88 @@
+# Changelog
+
+All notable changes to IF-Split are recorded here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to
+follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+The **split is always computed from metadata + sequences only** — `build` never
+downloads structure coordinates. That invariant holds across every release below.
+
+## [0.3.0] — 2026-07-14
+
+A large release: fold-honest splitting, split-output certification, a two-corpus
+training model, a metadata-only curation overhaul, and offline re-derivability.
+
+### Added
+
+- **Fold-level structural leakage control** (opt-in `structural_clustering`:
+  `off` | `cath` | `ecod` | `scop2`). Same-fold protein chains are union-merged into
+  one leakage-safe component in addition to shared sequence clusters, so a fold cannot
+  straddle train/test — using RCSB's precomputed CATH/ECOD/SCOP2 classifications
+  (metadata only, no coordinates).
+- **Balance-aware split strategy** (`split_strategy: balanced`). Caps dominant folds
+  to train and fills val/test to their *entry* targets from the fold tail, restoring
+  ~80/10/10 by entries with thousands of held-out folds. `config/masterclass.yaml`
+  ships the fold-honest recipe (`scop2` + `balanced`).
+- **Split-output certification.** The `@2` `dataset.lock` records `split_sha256` (a
+  hash of the entry→split partition); `verify` re-derives Stages 3–6 and certifies the
+  split *output* reproduced, not just the Stage-1 candidate set.
+- **Two training corpora from one split**: all kept structures as design *backbones*,
+  plus a functional-ligand *conditioning-target* corpus (`targets.jsonl`, one row per
+  ligand keyed to entry + split + class + tier). `SplitView` exposes both views.
+- **Offline `resplit`** (`if-split resplit --candidates candidates.jsonl --config X`):
+  re-derives Stages 3–7 from a cached snapshot with no RCSB — ablate curation /
+  clustering / split settings, or tighten a filter, in seconds instead of
+  re-enumerating the PDB. The lock records `source` (`build` | `resplit`).
+- **Offline `verify`** (`verify LOCK --candidates candidates.jsonl`): integrity-check
+  a distributed dataset with no network; a corrupt candidates file is reported as an
+  integrity failure. A `resplit` lock is steered to offline verification.
+- **Per-method resolution caps** (`resolution_max_A_by_method`) and a **cryo-EM
+  map-fit floor** (`min_em_backbone_inclusion`, wiring in the previously-unused
+  `em_backbone_inclusion` metric). Resolution is now re-derived in Stage 3, so the cut
+  is auditable from `candidates.jsonl` and tightenable offline.
+- **Opt-in sequence-usability floor** (`min_modeled_residues`) and an always-on drop of
+  empty / all-`X` (poly-UNK) protein chains, which carry no learnable label.
+- RCSB **metal-binding annotations** (GO/InterPro/Pfam) captured to rescue native
+  metalloenzymes; `if-split spec` to emit a portable, self-identifying split spec.
+
+### Changed / curation
+
+- **Metal tiering**: heavy-atom / lanthanide **phasing derivatives** (Hg/Au/Pt/Pb/Tl/…)
+  demoted to `ambiguous` (reported, recoverable) rather than counted as functional
+  metal sites; inorganic **Fe-S / metal-oxo / FeMo clusters** (SF4/FES, the OEC) now
+  classed `metal`; native Ni/Co (and heavy/lanthanide) sites rescued via annotation,
+  affinity, or subject-of-investigation. The lone-Ni/Co His-tag figure was corrected
+  (~96% → ~82%).
+- **Glycans** (RCSB CCD `type` = *saccharide*) with no measured affinity are tiered
+  `glycan` (decorative / detergent), recoverable via an opt-in tier — not counted as
+  small-molecule conditioning targets.
+- **Small molecules**: a measured binding affinity now overrides the additive
+  blacklist, so a blacklisted comp that is the real measured ligand stays functional.
+- **Nucleic acids**: `is_nucleic` now recognizes the `NA-hybrid` polymer type; the
+  ligand class was renamed `nucleotide` → `nucleic_acid`.
+- The size cap keeps `< 6000` residues correctly (`> max_total_residues`, not `>=`).
+- Adding a resolution-less method (NMR/SAXS) now warns instead of silently returning
+  zero entries.
+
+### Fixed
+
+- `verify` warns (rather than fails) on a version-only lock mismatch.
+- `fetch` reads split id-lists from the manifest directory, not the current directory.
+- `identity_threshold` is validated against RCSB's precomputed cluster levels
+  (30/50/70/90/95/100) so an unsupported level can't silently disable clustering.
+- A bound halide is tiered a counterion, not a functional small molecule.
+
+## [0.2.0] — 2026
+
+- Recover non-covalently bound cofactors (FAD/NAD/FMN/NADP, inhibitors) via RCSB's
+  `is_subject_of_investigation` flag.
+- Harden Ni/Co metal curation against His-tags absent from the deposited sequence.
+- Shareable split spec (`if-split spec`) and a self-identifying config header.
+- Rename the ligand class `nucleotide` → `nucleic_acid`; PyPI/CI badges + install docs.
+
+## [0.1.0] — 2026
+
+- Initial release: a reproducible, date-pinned, ligand-aware train/val/test splitter
+  for the PDB. Enumerate → filter → tier ligands → cluster (union-find, leakage-safe)
+  → deterministic split → manifest + lock, all from RCSB Search + Data API metadata
+  (no coordinates). Optional `fetch` downloads structures for a built split.

{if_split-0.2.0 → if_split-0.3.0}/CLAUDE.md

@@ -10,7 +10,8 @@ usage. This file is the orientation for working in the repo.
 **The split is computed from metadata + sequences only — `build` never downloads
 structure coordinates.** Everything needed (resolution, method, release date,
 residue counts, per-entity sequences, ligand chem-comp + bound-component signals,
-RCSB cluster membership) comes from the RCSB Search + Data APIs. Coordinates
+RCSB sequence-cluster membership, and CATH/ECOD/SCOP2 structural classifications)
+comes from the RCSB Search + Data APIs. Coordinates
 (mmCIF) are large and only needed downstream, so `fetch` (Stage 2) is optional.
 Keep it that way: do not add coordinate access to the build path.
 
@@ -36,6 +37,8 @@ wsl -d ubuntu bash -lc 'cd ~/projects/IF-Split && export PATH="$HOME/.local/bin:
 uv run ruff check .      # lint (must pass)
 uv run ruff format .     # format
 uv run if-split build --limit 50 --out /tmp/ifs   # dev build (small, live RCSB)
+uv run if-split build --config config/masterclass.yaml --out /tmp/mc  # fold-honest split (scop2 + balanced)
+uv run if-split resplit --candidates data/out/candidates.jsonl --config X.yaml --out /tmp/rs  # re-derive Stages 3-7 offline (no RCSB)
 ```
 
 - `uv sync` sets up the env; `uv sync --extra mlops` adds pyarrow for `fetch`'s
@@ -47,17 +50,22 @@ uv run if-split build --limit 50 --out /tmp/ifs   # dev build (small, live RCSB)
 
 `enumerate.py`+`rcsb.py` (Stage 1, Search+Data API → candidates.jsonl) →
 `parse.py` (3, metadata filters) → `ligands.py` (4, confidence tiering) →
-`cluster.py` (5, union-find components) → `split.py` (6, deterministic hash) →
+`cluster.py` (5, union-find components: sequence + optional fold-level structural
+clustering) → `split.py` (6, split assignment: `hash` | `balanced`) →
 `manifest.py` (7, lock + manifest + registry, verify/stats) → `dataset.py` (8,
-loader). `download.py`+`hydrate.py` are the optional Stage 2 `fetch`.
+loader). `download.py`+`hydrate.py` are the optional Stage 2 `fetch`. `cli.py`'s
+`resplit` re-runs Stages 3-7 from a cached `candidates.jsonl` (no Stage 1) via the
+shared `_run_pipeline`; `verify --candidates` does the same for offline checking.
 
 Invariants that must not regress:
 - **Determinism:** same config → byte-identical `manifest.json` (no wall-clock
   fields). `test_manifest_is_deterministic` guards this.
 - **No cross-split leakage:** sequence clusters joined by a shared multi-chain
-  entry are union-find–merged into one component; a component maps to exactly one
-  split, so overlap is impossible by construction. `check_no_leakage` is a real
-  invariant (not a tautology) — keep it that way.
+  entry (and, with `structural_clustering` on, by a shared fold superfamily) are
+  union-find–merged into one component; a component maps to exactly one split, so
+  overlap is impossible by construction. This holds for both split strategies
+  (`hash` and `balanced`, which only chooses *which* split a whole component lands
+  in). `check_no_leakage` is a real invariant (not a tautology) — keep it that way.
 - **Growth stability:** a cluster/component's split is `hash(salt + canonical_key)`
   into cumulative fractions, keyed on the global-min member id (not RCSB's volatile
   integer id). A larger snapshot only *adds* components; `splits.registry.json`

{if_split-0.2.0 → if_split-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: if-split
-Version: 0.2.0
+Version: 0.3.0
 Summary: Reproducible, date-pinned, ligand-aware train/val/test splitter for the PDB (LigandMPNN-style).
 Author: WSobo
 License: MIT
@@ -43,9 +43,10 @@ records and sequences. Coordinates are an optional, downstream concern.
 | | |
 |---|---|
 | **Fresh** | Builds from the current PDB, not a years-old frozen copy. |
-| **Reproducible** | A `dataset.lock` pins the snapshot; `verify` re-derives it and reports any drift. |
+| **Reproducible** | A `dataset.lock` pins the snapshot **and the split output**; `verify` re-derives both and certifies they reproduced byte-for-byte (or reports exactly what drifted). |
 | **Cheap** | Metadata-only — a split is megabytes of JSON, not a terabyte of mmCIF. |
 | **Honest about quality** | Every ligand is tiered (`functional` / `ambiguous` / `artifact`) with a reason; nothing is silently dropped. |
+| **Fold-aware** | Controls *structural* leakage, not just sequence: same-fold chains can't straddle train/test — the leak that matters most for structure→sequence models. |
 
 ### Two reproducibility guarantees
 
@@ -60,6 +61,51 @@ records and sequences. Coordinates are an optional, downstream concern.
    `splits.registry.json` pins prior assignments to make this exact even across
    re-clustering.
 
+### Fold-level leakage control
+
+Sequence clustering alone is not enough for inverse folding. A model learns
+**structure → sequence**, so two chains below the 30% identity threshold that
+nonetheless share a **fold** (TIM barrels, Rossmann folds, globins…) leak
+structural information across the split — the model has effectively seen the test
+backbone during training. `structural_clustering` closes this: protein entities
+sharing a structural **(super)family** are union-merged into the same component in
+addition to shared sequence clusters, so a fold cannot straddle train/test.
+
+It uses RCSB's precomputed **CATH / ECOD / SCOP2** classifications — still metadata
+only, no coordinates — selectable per build (`off | cath | ecod | scop2`). It is
+**purely additive**: it can only merge components, never split them, and a chain
+with no classification simply contributes no structural edge. `if-split stats`
+reports how many components the structural pass folded together, so the effect is
+always measurable (`scripts/eval_structural_clustering.py` compares the methods).
+
+Coverage is partial by nature: CATH ≈ 55%, ECOD ≈ 80%, SCOP2 ≈ 52% of protein
+chains are classified (measured on the full 2026-07-14 snapshot); the rest fall
+back to sequence-only.
+
+**Why it needs a balance-aware split.** On its own, fold-merging collapses the
+dominant superfamilies (antibodies, TIM barrels) into mega-components that land
+wholesale in one split, skewing the *entry* balance to ~95/3/2 (the *component*
+split stays ~80/10/10). `split_strategy: "balanced"` fixes this: it **caps the
+dominant folds to train and fills val/test to their entry targets from the tail of
+smaller folds** — restoring ~80/10/10 by entries with thousands of distinct,
+held-out folds per split. It stays leakage-safe (whole components) and
+growth-stable (via the registry), and reports a gap if a method's tail is too thin
+(`cath`/`ecod` starve val; **`scop2` is the sweet spot**). `balanced` also fixes
+the plain sequence-only skew (88/6/6 → 80/10/10) from the antibody mega-cluster.
+
+### The masterclass split
+
+```bash
+uv run if-split build --config config/masterclass.yaml --out data/mc
+```
+
+`config/masterclass.yaml` = default **+ `structural_clustering: scop2` + `split_strategy: balanced`**: a fold-honest ~80/10/10 split whose val/test are thousands of folds held entirely out of train — the truest generalization measure the tool can produce, still from metadata alone.
+
+| split | strategy | structural | entry balance | val/test |
+|---|---|---|--:|---|
+| default | hash | off | 88 / 6 / 6 | sequence-clustered |
+| masterclass | balanced | scop2 | 80 / 10 / 10 | thousands of held-out folds |
+
 ---
 
 ## Install
@@ -94,9 +140,19 @@ uv run if-split build --limit 50 --out /tmp/ifs
 # Summarize a build: split sizes, per-class test counts, curation tiers.
 uv run if-split stats data/out/manifest.json
 
-# Reproduce-check: re-derive from a lock and report drift vs the live PDB.
+# Reproduce-check: re-derive from a lock, report candidate drift vs the live PDB,
+# and (when candidates reproduce) certify the split output matches its hash.
 uv run if-split verify data/out/dataset.lock
 
+# Offline verify: integrity-check a distributed candidates.jsonl + lock, no network.
+uv run if-split verify data/out/dataset.lock --candidates data/out/candidates.jsonl
+
+# Re-derive the split from a CACHED candidates.jsonl — no RCSB. Ablate curation,
+# clustering, split strategy, or TIGHTEN a filter (e.g. resolution) on a fixed
+# snapshot in ~seconds instead of re-enumerating the whole PDB.
+uv run if-split resplit --candidates data/out/candidates.jsonl \
+    --config config/masterclass.yaml --out data/mc
+
 # Growth-stable regeneration: pin prior cluster→split assignments.
 uv run if-split build --registry data/out/splits.registry.json --out data/out2
 
@@ -112,7 +168,7 @@ uv run if-split spec data/out/manifest.json --name my-split --out my-split.ifspl
 | File | Purpose |
 |---|---|
 | `candidates.jsonl` | The snapshot definition — one canonical JSON record per entry. Hashed into the lock. |
-| `dataset.lock` | Reproduction anchor: embedded config + candidates SHA-256 + entry list. |
+| `dataset.lock` | Reproduction anchor: embedded config + candidates SHA-256 + entry list + a `split` hash of the entry→split partition (so `verify` certifies the split output, not just the inputs). |
 | `manifest.json` | Human-facing run record: per-split entry lists, ligand classes + tiers, per-class (and ambiguous) counts, drop log, cluster/leakage stats, entry→cluster map. |
 | `splits.registry.json` | `cluster key → split`, for growth-stable regeneration. |
 
@@ -176,10 +232,10 @@ A `build` runs eight stages; none touch coordinates.
 | Stage | Module | What it does |
 |---|---|---|
 | 1 — enumerate | `enumerate.py`, `rcsb.py` | RCSB Search → entry IDs; Data API (GraphQL, batched) → sequences, ligands, residue counts, cluster membership → `candidates.jsonl`. |
-| 3 — filter | `parse.py` | Drop no-protein / no-sequence / oversized entries (assembly-1 residue count vs `max_total_residues`), plus optional wwPDB validation-report quality caps (clashscore, R-free, Ramachandran/rotamer/RSRZ) — all from metadata. Every drop is logged with its reason. |
+| 3 — filter | `parse.py` | Drop no-protein / no-usable-sequence (empty **or** all-`X` poly-UNK) / too-short (opt-in `min_modeled_residues`) / oversized entries (assembly-1 residue count `> max_total_residues`) / over-resolution (re-derived here so it is auditable; per-method caps via `resolution_max_A_by_method`), plus optional wwPDB validation-report quality caps (clashscore, R-free, Ramachandran/rotamer/RSRZ, cryo-EM map-fit floor) — all from metadata. Every drop is logged with its reason. |
 | 4 — ligands | `ligands.py` | Tier each non-protein component `functional`/`ambiguous`/`artifact`; derive class labels (metal / small-molecule / nucleic-acid). `nucleic_acid` = a protein↔DNA/RNA *complex* (verified assembly interface), **not** a bound mononucleotide. **Annotate, never drop.** |
-| 5 — cluster | `cluster.py` | Group protein entities by RCSB precomputed cluster id at `identity_threshold`; canonical key = smallest member id. |
-| 6 — split | `split.py` | Deterministic hash → train/val/test; assert no cluster spans two splits; audit residual secondary-chain overlap. |
+| 5 — cluster | `cluster.py` | Group protein entities by RCSB precomputed cluster id at `identity_threshold`; canonical key = smallest member id. Optionally union same-fold entities (CATH/ECOD/SCOP2) for structural-leakage control. |
+| 6 — split | `split.py` | Assign components → train/val/test (`hash`, or `balanced` for entry-balanced fold-tail val/test); assert no cluster spans two splits; audit residual secondary-chain overlap. |
 | 7 — manifest | `manifest.py` | Emit lock + manifest + registry (all deterministic, no wall-clock fields). |
 | 8 — loader | `dataset.py` | Read a manifest into train/val/test views with cluster-balanced sampling. |
 | 2 — fetch *(opt-in)* | `download.py`, `hydrate.py` | Download mmCIF for a built manifest into a sharded, indexed, ML-ready tree. |
@@ -201,6 +257,7 @@ metrics come straight from the deposited report:
 | `max_rotamer_outlier_pct` | % sidechain rotamer outliers | X-ray + cryo-EM |
 | `max_rfree` | R-free (DCC) | X-ray |
 | `max_rsrz_outlier_pct` | % real-space-R Z-score outliers | X-ray |
+| `min_em_backbone_inclusion` | backbone atom-in-density (a **floor** — higher is better) | cryo-EM |
 
 Two rules keep it honest: a cap fires **only when the metric is present**, so a
 cryo-EM entry is never dropped for a missing R-free; and every cap is **off by
@@ -220,8 +277,8 @@ machine-readable reason, from RCSB metadata signals:
 
 | Tier | Meaning | Example reasons |
 |---|---|---|
-| `functional` | Real ligand/site → gets a class label | bound to protein (`nonpolymer_bound_components`) or has measured binding affinity |
-| `ambiguous` | Present but uncorroborated → reported, **not** labelled | `metal_unbound`, `ligand_unbound` |
+| `functional` | Real ligand/site → gets a class label | `metal_bound`/`ligand_bound` (contacts protein), `*_affinity` (measured), `*_investigated` (RCSB SOI), `metal_annotated` (protein annotated to bind this metal) |
+| `ambiguous` | Present but uncorroborated → reported, **not** labelled | `metal_unbound`, `ligand_unbound`, `metal_site_nonnative`, `glycan`, `purification_metal_uncorroborated` |
 | `artifact` | Buffer / counterion / purification tag → excluded from labels | `additive`, `counterion`, `histag_metal` |
 
 **Holo gating (metadata-only).** Presence isn't enough. A small molecule or metal
@@ -244,13 +301,23 @@ affinity purification. A poly-His run anywhere — or a short run at a chain
 terminus (`histag_terminal_min_run`, catching 6×His tags left partial by
 unmodeled or trimmed residues) — flags the entry's Ni/Co as an `artifact`.
 
-But a real audit showed the deeper issue: **~96% of lone Ni/Co entries have no
-His-tag in the deposited sequence at all** (the tag is trimmed from the SEQRES
-record, not just unmodeled), so a sequence scan can never see it. So even with no
-detectable tag, a *lone* Ni/Co (the entry's only metal) with no measured affinity
-is demoted from `functional` to `ambiguous` — reported, not labelled. Real metals
-(Zn, Mg, Fe, …), and Ni/Co backed by an affinity or sitting alongside a genuine
-metal, are untouched. On the full PDB this re-tiers ~2.7% of the metal set.
+But an audit (reproducible via [`scripts/audit_nico_histag.py`](scripts/audit_nico_histag.py))
+showed a subtler issue: **~82% of lone Ni/Co entries carry no detectable His-tag
+in the deposited sequence** — IMAC tags are frequently absent from the SEQRES
+record, not just unmodeled, so a sequence scan can't recover them. So even with no
+detectable tag, a *lone* Ni/Co (the entry's only metal) with no corroboration is
+demoted from `functional` to `ambiguous` — reported, not labelled.
+
+To avoid over-firing on genuine bare-Ni/Co enzymes (urease, cobalt methionine
+aminopeptidase, nitrile hydratase, …), a lone Ni/Co is **rescued to `functional`
+(`metal_annotated`)** when the protein's RCSB GO/InterPro/Pfam annotation says it
+binds that metal. A protein that binds a *different* native metal (Ni/Co as an
+isomorphous substitute — e.g. Co in a Mg enzyme) is reported `metal_site_nonnative`
+so a consumer can choose to keep it; one with no metal annotation at all stays
+`purification_metal_uncorroborated`. All from RCSB's own metadata (no extra
+UniProt call). Real metals (Zn, Mg, Fe, …), and Ni/Co with affinity/SOI or beside
+a genuine metal, are untouched. Rerun [`scripts/eval_metal_tiering.py`](scripts/eval_metal_tiering.py)
+to measure the tier distribution over the whole lone-Ni/Co set.
 
 Crucially, **the structure always stays in its split** — a protein with a junk
 ion is still a good backbone; we just don't label the junk. A consumer wanting
@@ -258,11 +325,29 @@ ion is still a good backbone; we just don't label the junk. A consumer wanting
 threshold, not the build. The same per-component tier is what a downstream
 featurizer reads to decide what counts as real ligand context.
 
+> **Per-instance is a featurizer concern.** These tiers are per *component* — they
+> establish whether a structure contains a real Ni/Co site, not *which* of several
+> same-element ions is it. A deposition can hold both a catalytic Ni and a surface
+> crystallization Ni under one `NI` id, and no metadata separates them (adventitious
+> Ni binds surface His/Asp with the same geometry as a catalytic site). Deciding
+> which individual ion to featurize is left to the coordinate-level featurizer.
+
+**Glycans aren't ligand pockets.** A carbohydrate (RCSB CCD type `*saccharide*` —
+NAG/BMA/MAN/…, and sugar-detergents like LMT) is overwhelmingly decorative
+glycosylation or a purification detergent, not a site an inverse-folding model
+conditions on. So a carbohydrate is tiered `glycan` (reported, not a small-molecule
+target) unless it has a *measured binding affinity* — RCSB's `is_subject_of_investigation`
+flag is too noisy for sugars (it flags glycosylation and detergents), so it doesn't
+rescue here. A genuine lectin/glycosidase ligand is recoverable as an opt-in target
+(`include_ambiguous=True`). Real cofactors (ATP, NAD, HEM, FAD) and structural lipids
+(cardiolipin, phosphatidyl-*) are `non-polymer`, not saccharides, so they're untouched.
+
 ### Test-set representation
 
-The split is a pure deterministic hash, so the test set's ligand mix is reported
-but not forced by default: `manifest.json` carries per-split, per-class
-`functional` counts plus `ambiguous` counts, so under-representation is visible.
+The split is deterministic (a per-component hash, or the `balanced` fold-tail
+fill), so the test set's ligand mix is reported but not forced by default:
+`manifest.json` carries per-split, per-class `functional` counts plus `ambiguous`
+counts, so under-representation is visible.
 An opt-in `--enforce-minimums` top-up (recruit `functional`-only ligand clusters
 into test in deterministic order) is scoped for a future release.
 
@@ -283,6 +368,75 @@ for epoch in range(3):
     batch_ids = ds.train.sample_by_cluster(seed=epoch)
 ```
 
+### Training strategies for inverse folding
+
+An inverse-folding model consumes two different things, with opposite scale/quality
+tradeoffs. IF-Split emits **both from the same leakage-safe split**, so you pick a
+strategy without re-deriving the split:
+
+| Corpus | What | Use it for |
+|---|---|---|
+| **Backbones** — every kept structure | `ds.train.backbones` | ProteinMPNN-style, ligand-agnostic. The scale lever — a structure with only junk ions or no ligand is still a good backbone. |
+| **Conditioning targets** — the `functional`-tier ligands | `ds.train.conditioning_targets()` | LigandMPNN-style. One row per `(structure, ligand)`; junk is never a target. The quality lever. |
+
+```python
+ds = load_dataset("data/out/manifest.json")
+
+# 1. Backbone-only training (max data): every structure.
+backbones = ds.train.backbones
+
+# 2. Ligand-conditioned training: condition on the real ligands only.
+targets = ds.train.conditioning_targets()                 # metal / small_molecule / nucleic_acid
+metal_targets = ds.train.conditioning_targets(classes=["metal"])
+
+# 3. Condition on ALL of a structure's ligands at once (group by entry), or one at a time:
+for entry_id, ligs in ds.train.targets_by_entry().items():
+    ctx = [(t.ligand_class, t.comp_id) for t in ligs]     # e.g. [("small_molecule","HEM")]
+
+# 4. Only structures that actually carry a conditioning target:
+conditioned = ds.train.conditioned_entry_ids()            # subset of backbones
+
+# 5. Opt in to ambiguous targets — non-native metal pockets (Ni/Co substituting the
+#    native metal) and glycans (glycosylation / lectin ligands) — off by default:
+any_site = ds.train.conditioning_targets(include_ambiguous=True)
+```
+
+The full corpus is also written to `targets.jsonl` (one row per target: entry, split,
+cluster, class, comp_id, tier, reason) and mirrored into `index.parquet`'s
+`conditioning_targets` column after `fetch`.
+
+#### From a target to its pocket (your featurizer owns this)
+
+IF-Split stops at the **labels** — it never parses coordinates. The last mile is
+yours: join a target's `comp_id` to a fetched structure with your own parser and
+pull the ligand atoms + pocket. This is deliberate — featurization is
+model-specific (ligand atoms? SMILES? a pocket mask? all-atom context?), and every
+inverse-folding model already has its own pipeline. `comp_id` is the join key:
+
+```python
+from pathlib import Path
+
+import gemmi  # or biotite / Biopython — same pattern
+from ifsplit.dataset import load_dataset
+from ifsplit.download import rel_path_for
+
+ds = load_dataset("data/out/manifest.json")
+for entry_id, targets in ds.test.targets_by_entry().items():
+    path = Path("data/structures") / rel_path_for(entry_id, "test", assembly=True)
+    model = gemmi.read_structure(str(path))[0]
+    for t in targets:
+        # A structure may hold several copies of one ligand (plus adventitious ions).
+        # Surface ALL copies; pick the instance to condition on per your model.
+        copies = [r for ch in model for r in ch if r.name == t.comp_id]
+        # ... extract residues within config.ligand_context_radius_A of each copy ...
+```
+
+When an entry has several functional ligands (e.g. a cofactor *and* an inhibitor,
+or a catalytic metal alongside an adventitious one), that shows up as **multiple
+target rows** — condition on all (group by `entry_id`) or one per example, your
+call. A runnable, copy-and-adapt version with pocket extraction is
+[`scripts/consume_split.py`](scripts/consume_split.py).
+
 ## Sharing a split spec
 
 The config **is** the shareable recipe. Everything that affects the split lives in
@@ -307,9 +461,9 @@ spec:
   ifsplit_spec: ifsplit/config@1          # schema id — the file says what it is
   name: my-split
   author: you
-  created_with: if-split 0.1.0
+  created_with: if-split 0.3.0
   expected_config_hash: 3b63318286fd2ac4994f34d10936be05
-snapshot_date: '2026-05-31'
+snapshot_date: '2026-07-14'
 resolution_max_A: 3.5
 # ... all output-affecting settings ...
 ```
@@ -323,7 +477,7 @@ differ only in their labels produce byte-identical outputs.
 |---|---|--:|
 | `*.ifsplit.yaml` (or `config.yaml`) | *"How did you make this split?"* — the recipe | ~KB |
 | `manifest.json` | *"What's in it?"* — counts, provenance, file index | ~KB |
-| `dataset.lock` | *"Reproduce the exact bytes"* — pins entry set + candidates SHA | ~MB |
+| `dataset.lock` | *"Reproduce the exact bytes"* — pins entry set + candidates SHA + split-output hash | ~MB |
 
 ## Configuration
 
@@ -336,8 +490,10 @@ doubles as a shareable **split spec** — see [Sharing a split spec](#sharing-a-
 |---|---|---|
 | `snapshot_date` | `2026-05-30` | `release_date <= this` — the reproducibility anchor. |
 | `experimental_methods` | X-ray, EM | Allowed `exptl.method` values. |
-| `resolution_max_A` | `3.5` | Resolution cutoff. |
-| `max_total_residues` | `5999` | Size cap (LigandMPNN used `< 6000`). |
+| `resolution_max_A` | `3.5` | Resolution cutoff (re-derived in Stage 3, so it is auditable from `candidates.jsonl`). |
+| `resolution_max_A_by_method` | `{}` | Optional per-method resolution overrides, e.g. `{ELECTRON MICROSCOPY: 3.0}` — cryo-EM 3.5 Å ≠ X-ray 3.5 Å. Empty = one cap for all. |
+| `max_total_residues` | `5999` | Max residues **kept** (drop if `> this`) — LigandMPNN kept `< 6000`, i.e. `<= 5999`. |
+| `min_modeled_residues` | `0` | Opt-in floor on modeled (non-`X`) residues in a protein chain. `0` = off; only the always-on empty/all-`X` (poly-UNK) drop applies. `~20` also drops tiny/mostly-unknown chains. |
 | `excluded_het` | waters + common ions | Extra components forced to `artifact`. |
 | `use_biological_assembly` | `true` | Count residues from assembly 1, not the deposited asymmetric unit. |
 | `purification_metals` | `[NI, CO]` | Metals treated as IMAC tags; `[]` disables the heuristic. |
@@ -348,7 +504,7 @@ doubles as a shareable **split spec** — see [Sharing a split spec](#sharing-a-
 | `clustering_backend` | `precomputed` | `precomputed` (RCSB clusters) or `mmseqs2` (run your own). |
 | `split_fractions` | 0.80 / 0.10 / 0.10 | train / val / test. |
 | `split_salt` | `snapsplit-v1` | Bump to intentionally reshuffle the split. |
-| `max_clashscore`, `max_rfree`, `max_ramachandran_outlier_pct`, `max_rotamer_outlier_pct`, `max_rsrz_outlier_pct`, `require_validation_report` | off | Optional validation-report quality caps — see [Structure quality](#structure-quality-validation-report). |
+| `max_clashscore`, `max_rfree`, `max_ramachandran_outlier_pct`, `max_rotamer_outlier_pct`, `max_rsrz_outlier_pct`, `min_em_backbone_inclusion`, `require_validation_report` | off | Optional validation-report quality caps — see [Structure quality](#structure-quality-validation-report). |
 | `ligand_context_radius_A`, `max_ligand_atoms` | `8.0`, `25` | Featurization only (not part of the split). |
 
 ## Develop
@@ -379,6 +535,12 @@ data/out/                # generated manifests + lock files
 tests/
 ```
 
+## Changelog
+
+Release history is in [CHANGELOG.md](CHANGELOG.md). The current release is **0.3.0**
+(fold-honest splitting, split-output certification, the two-corpus training model, a
+metadata-only curation overhaul, and offline `resplit` / `verify`).
+
 ## License
 
 MIT — see [LICENSE](LICENSE).

{if_split-0.2.0 → if_split-0.3.0}/PLAN.md

@@ -106,8 +106,8 @@ IF-Split/
     download.py                # Stage 2: OPTIONAL on-demand mmCIF fetch (featurization)
     parse.py                   # Stage 3: metadata filters + drop log
     ligands.py                 # Stage 4: classify non-protein entities from metadata
-    cluster.py                 # Stage 5: precomputed RCSB clusters (default) | mmseqs2
-    split.py                   # Stage 6: deterministic hash assignment + stratification
+    cluster.py                 # Stage 5: seq clusters (precomputed|mmseqs2) + opt. fold-level structural union
+    split.py                   # Stage 6: split assignment (hash | balanced) + stratification
     manifest.py                # Stage 7: emit manifest + lock file
     dataset.py                 # Stage 8: loader / torch Dataset consuming a manifest
     cli.py                     # `if-split build`, `if-split verify`, `if-split stats`
@@ -151,7 +151,8 @@ guaranteed to have used identical settings.
 snapshot_date: "2026-05-30"        # release_date <= this. The reproducibility anchor.
 experimental_methods: ["X-RAY DIFFRACTION", "ELECTRON MICROSCOPY"]
 resolution_max_A: 3.5
-max_total_residues: 5999           # LigandMPNN used "< 6000"
+max_total_residues: 5999           # max residues KEPT (drop if > this); LigandMPNN kept < 6000
+min_modeled_residues: 0            # opt-in floor on modeled (non-X) residues; 0 = off
 excluded_het: ["HOH", "NA", "CL", "K", "BR"]   # waters + common crystallization ions
 use_biological_assembly: true      # biounits, as in LigandMPNN (assembly 1)
 # purification-artifact curation (Stage 4): His-tag + Ni/Co only -> not a metal site
@@ -215,10 +216,13 @@ re-featurize from the cleaned structures.
 **Stage 3 — Filter (`parse.py`)**
 
 - Operate on the metadata in `candidates.jsonl` (no coordinate parsing). Apply
-  filters: drop entries with `total_residues >= max_total_residues` (use the
-  assembly residue count when `use_biological_assembly`), drop entries whose only
-  non-protein components are in `excluded_het`, drop entries with no protein
-  polymer entity. Record drop reasons + counts.
+  filters: drop entries with `total_residues > max_total_residues` (use the
+  assembly residue count when `use_biological_assembly`), over the (per-method)
+  resolution cap — re-derived here from `resolution_A` so the cut is auditable and
+  tightenable offline — drop entries whose only non-protein components are in
+  `excluded_het`, drop entries with no protein polymer entity or no usable
+  (non-all-`X`) sequence, plus optional validation-report caps (incl. the cryo-EM
+  map-fit floor). Record drop reasons + counts.
 - Sequences come from the Data API canonical one-letter code, which already maps
   modified residues to canonical parents (e.g. MSE→MET). Coordinate-level
   re-parsing with `gemmi` belongs to the optional featurization path, not here.
@@ -311,12 +315,34 @@ entry may touch multiple clusters via different chains; assign the entry to the
 cluster of its longest protein chain (record all) so split assignment is
 unambiguous.
 
+**Fold-level structural clustering (`structural_clustering`, opt-in ← *implemented***).
+Sequence clustering misses *structural* redundancy: chains below the identity
+threshold can still share a fold, which a structure→sequence model leaks across
+splits. When set (`cath` | `ecod` | `scop2`), protein entities sharing an RCSB
+structural (super)family are union-merged into the same component **in addition
+to** shared sequence clusters — so a fold cannot straddle train/test. The
+classifications ride in the snapshot as metadata (`rcsb_polymer_instance_annotation`,
+Stage 1; **no coordinates**). CATH keys on the superfamily code (`1.10.490.10`),
+ECOD/SCOP2 on the family name. Purely additive (only merges, never splits);
+coverage is partial (CATH ~55%, ECOD ~81%, SCOP2 ~52% of chains) so unclassified
+chains fall back to sequence-only. See README "Fold-level leakage control".
+
 **Stage 6 — Split assignment (`split.py`) — the reproducibility core**
 
 - Assign each cluster (not each entry) to a split by deterministic hash:
   `bucket = int(blake2b(cluster_repr_id + split_salt)) ...` mapped to cumulative
   `split_fractions`. Same salt + same cluster IDs → same assignment, forever.
   (See the Stage 6 note in §1 on making `cluster_repr_id` input-independent.)
+- **Balance-aware strategy (`split_strategy: "balanced"` ← *implemented***).
+  Per-component hashing balances *components*, not *entries*; a dominant fold
+  (under structural clustering) or the antibody mega-cluster (even sequence-only)
+  balloons one split. `balanced` caps dominant folds (> 0.2% of entries) to train
+  and fills val/test to their *entry* targets from the tail of smaller folds in
+  hash order — leakage-safe (whole components), growth-stable via the registry,
+  and it reports a gap rather than forcing a target it can't reach. The
+  "masterclass" recipe `structural_clustering: scop2` + `split_strategy: balanced`
+  (`config/masterclass.yaml`) yields ~80/10/10 by entries with thousands of folds
+  held entirely out of train.
 - Stratify the test set by ligand class so SM/metal/nucleotide are all
   represented (LigandMPNN's test sets are deliberately ligand-containing).
   Implement as: within the test-bucketed clusters, label structures by ligand
@@ -333,13 +359,17 @@ Emit two artifacts in `data/out/`:
 - `manifest.json` — human-facing: snapshot_date, config hash, tool versions
   (mmseqs2, gemmi), per-split entry lists, per-structure metadata, ligand-class
   tags, per-class test counts, drop log.
-- `dataset.lock` — reproduction-facing: the two snapshot anchors. (1) every
-  entry ID + obsolescence status (the candidate set is reproduced from the Data
-  API by id + `release_date <= snapshot_date`); (2) the **cluster file**: its
-  SHA-256, RCSB `Last-Modified`, and a stored copy (≈17 MB, the only sizeable
-  artifact). `if-split verify dataset.lock` re-fetches metadata + cluster file
-  and confirms the cluster-file hash matches and the entry set is unchanged.
-  (mmCIF SHA-256s belong to the *optional* featurization fetch, not this lock.)
+- `dataset.lock` — reproduction-facing anchors: (1) the embedded config + the
+  **candidate set** — every entry ID and the canonical `candidates.jsonl` SHA-256
+  (the candidate set is reproduced from the Data API by id + `release_date <=
+  snapshot_date`; clustering rides along as per-entity `cluster_ids`, so there is
+  no separate cluster file); (2) a **`split` block** hashing the entry→split
+  partition (`@2` locks). `if-split verify dataset.lock` re-enumerates and reports
+  candidate drift (added/removed/hash); when the candidate set reproduced
+  byte-for-byte it recomputes Stages 3-6 and certifies the split-output hash
+  matches (registry-free builds), so a curation/split-logic change is caught even
+  with identical inputs. (mmCIF SHA-256s belong to the *optional* featurization
+  fetch, not this lock.)
 - Version the dataset as `IF-Split-<snapshot_date>` (e.g. `IF-Split-2026.05.30`).
 
 **Stage 8 — Loader (`dataset.py`)**