zmip 0.1.0__tar.gz

zmip-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 chansigit
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

zmip-0.1.0/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.4
+Name: zmip
+Version: 0.1.0
+Summary: Zoom-in pipeline: per-lineage re-embedding, foreign-lineage scoring and agent refinement of an msp annotation
+Author-email: chansigit <chansigit@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/chansigit/zmip
+Classifier: Programming Language :: Python :: 3
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: msp-sc>=0.2.0
+Requires-Dist: claude-agent-sdk
+Requires-Dist: scanpy
+Requires-Dist: anndata
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: matplotlib
+Dynamic: license-file
+
+# zmip — zoom-in pipeline
+
+The round after [msp](https://github.com/chansigit/msp): take msp's
+`annotated.h5ad`, split it into lineages, re-embed each lineage on its own,
+and let a per-lineage agent refine the annotation, clean noise and hand
+misassigned cells to the lineage they belong to. Same pattern as osp/msp —
+fixed computation, narrow agent decisions validated by the host, one
+self-contained report per lineage plus a global one.
+
+```
+msp annotated.h5ad ──▶ plan ──▶ per lineage: re-embed → foreign scores → agent ──▶ merge
+                       (agent)                  (msp.integrate_adata)      (agent)     annotated_zmip.h5ad
+```
+
+## Install
+
+```bash
+pip install msp-sc                                        # msp on PyPI (import name `msp`)
+pip install zmip                                          # needs claude-agent-sdk + Claude Code CLI credentials
+```
+
+## Usage
+
+```bash
+python -m zmip msp_out/annotated.h5ad --outdir zmip_out --model claude-sonnet-5 [--min-cells 800]
+python -m zmip.report zmip_out          # rebuild the global report only
+```
+
+Re-running resumes: the plan is reused, lineages whose contract files
+(`annotation_proposal.json`, `annotated.h5ad`, `report.html`) exist are
+skipped; `--force` redoes everything. Integration knobs (`--resolutions`,
+`--n-top-genes`, `--n-pcs`, `--n-neighbors`, `--harmony KEY=VALUE`) are the
+same as msp's and apply to every per-lineage re-embedding.
+
+## Steps
+
+### 1. plan (`zmip.plan`)
+
+Host writes the evidence: cells/samples per coarse label, a kNN
+cross-connectivity matrix between coarse labels (share of each label's
+graph edges landing on every other label), PAGA on the same graph, and the
+coarse-label UMAP. The agent **must read the UMAP** and pools coarse labels
+that form one connected island into one lineage — even across cell types
+when data quality fuses them (T/B/myeloid as one immune island) — and keeps
+separate islands separate even when related; states (proliferating,
+stressed) go with the island they sit in. Host rules: every coarse label
+assigned exactly once; zoom only for lineages with at least `--min-cells`
+(default 800 — below that leiden cannot resolve stable substates); archived
+to `zmip_plan.json`. One lineage or none above the threshold → nothing is
+zoomed and the msp labels pass through.
+
+### 2. per lineage (`zmip.foreign`, `msp.integrate_adata`, `zmip.annotate`)
+
+- Subset → `msp.integrate_adata`: HVG/PCA/harmony/leiden(0.3/1.0/2.0)/UMAP
+  recomputed on the lineage alone, with every msp artifact (QC tables,
+  cell-level outliers, standissect fragments, DEG at r1.0/r2.0,
+  `preannotation_removal.csv`) in `<lineage>/`.
+- **Foreign-lineage scores**: lineage-level markers (wilcoxon on the whole
+  dataset at the plan's lineage level, specific genes only) → `sc.tl.score_genes`
+  for every other lineage → `obs["foreign_<lineage>"]`, per-cluster summaries
+  and UMAPs. Evidence only: close lineages share programs, so the agent
+  decides between doublet, ambient, misassignment and genuine biology.
+- Agent on `msp_leiden_r2.0` of the subset, one Claude Code Task per
+  cluster, tools `cluster_context` / `check_genes` / `check_deg` /
+  `check_stability` / `subcluster` (reclustering allowed). Per cluster:
+  distinctness → identity → foreign signal → merge, and one action:
+  `keep` (coarse label within the lineage), `remove` (with reason), or
+  `reassign` to another lineage's coarse label (relabel only — the cells are
+  not re-embedded there this round). Host validation as in msp.annotate plus
+  the reassign rules. Removal is real: subset pre-annotation filtering ∪
+  agent-removed clusters.
+- Outputs: `annotation_proposal.json`, `annotation_removed.csv`,
+  `annotation_reassigned.csv`, `annotated.h5ad`, `report.html` (msp's report
+  with the lineage's Cell Type Annotation section).
+
+### 3. merge (`zmip.merge`, `zmip.report`)
+
+Fold every lineage back into the global object. `annotated_zmip.h5ad` keeps
+the survivors with `zmip_lineage`, `zmip_cluster` (`<lineage>:<id>`),
+`zmip_ann_coarse`, `zmip_ann_fine`, `zmip_reassigned_from`; `msp_ann_*`
+stay for the audit trail. No global re-embedding here (next round's job):
+the global figures use msp's UMAP. Archives `zmip_removed.csv` (every
+removed cell with lineage, cluster, sources) and `zmip_reassigned.csv`.
+`report.html`: plan · lineages (linked per-lineage reports) · final
+annotation · removed & reassigned.

zmip-0.1.0/README.md

@@ -0,0 +1,85 @@
+# zmip — zoom-in pipeline
+
+The round after [msp](https://github.com/chansigit/msp): take msp's
+`annotated.h5ad`, split it into lineages, re-embed each lineage on its own,
+and let a per-lineage agent refine the annotation, clean noise and hand
+misassigned cells to the lineage they belong to. Same pattern as osp/msp —
+fixed computation, narrow agent decisions validated by the host, one
+self-contained report per lineage plus a global one.
+
+```
+msp annotated.h5ad ──▶ plan ──▶ per lineage: re-embed → foreign scores → agent ──▶ merge
+                       (agent)                  (msp.integrate_adata)      (agent)     annotated_zmip.h5ad
+```
+
+## Install
+
+```bash
+pip install msp-sc                                        # msp on PyPI (import name `msp`)
+pip install zmip                                          # needs claude-agent-sdk + Claude Code CLI credentials
+```
+
+## Usage
+
+```bash
+python -m zmip msp_out/annotated.h5ad --outdir zmip_out --model claude-sonnet-5 [--min-cells 800]
+python -m zmip.report zmip_out          # rebuild the global report only
+```
+
+Re-running resumes: the plan is reused, lineages whose contract files
+(`annotation_proposal.json`, `annotated.h5ad`, `report.html`) exist are
+skipped; `--force` redoes everything. Integration knobs (`--resolutions`,
+`--n-top-genes`, `--n-pcs`, `--n-neighbors`, `--harmony KEY=VALUE`) are the
+same as msp's and apply to every per-lineage re-embedding.
+
+## Steps
+
+### 1. plan (`zmip.plan`)
+
+Host writes the evidence: cells/samples per coarse label, a kNN
+cross-connectivity matrix between coarse labels (share of each label's
+graph edges landing on every other label), PAGA on the same graph, and the
+coarse-label UMAP. The agent **must read the UMAP** and pools coarse labels
+that form one connected island into one lineage — even across cell types
+when data quality fuses them (T/B/myeloid as one immune island) — and keeps
+separate islands separate even when related; states (proliferating,
+stressed) go with the island they sit in. Host rules: every coarse label
+assigned exactly once; zoom only for lineages with at least `--min-cells`
+(default 800 — below that leiden cannot resolve stable substates); archived
+to `zmip_plan.json`. One lineage or none above the threshold → nothing is
+zoomed and the msp labels pass through.
+
+### 2. per lineage (`zmip.foreign`, `msp.integrate_adata`, `zmip.annotate`)
+
+- Subset → `msp.integrate_adata`: HVG/PCA/harmony/leiden(0.3/1.0/2.0)/UMAP
+  recomputed on the lineage alone, with every msp artifact (QC tables,
+  cell-level outliers, standissect fragments, DEG at r1.0/r2.0,
+  `preannotation_removal.csv`) in `<lineage>/`.
+- **Foreign-lineage scores**: lineage-level markers (wilcoxon on the whole
+  dataset at the plan's lineage level, specific genes only) → `sc.tl.score_genes`
+  for every other lineage → `obs["foreign_<lineage>"]`, per-cluster summaries
+  and UMAPs. Evidence only: close lineages share programs, so the agent
+  decides between doublet, ambient, misassignment and genuine biology.
+- Agent on `msp_leiden_r2.0` of the subset, one Claude Code Task per
+  cluster, tools `cluster_context` / `check_genes` / `check_deg` /
+  `check_stability` / `subcluster` (reclustering allowed). Per cluster:
+  distinctness → identity → foreign signal → merge, and one action:
+  `keep` (coarse label within the lineage), `remove` (with reason), or
+  `reassign` to another lineage's coarse label (relabel only — the cells are
+  not re-embedded there this round). Host validation as in msp.annotate plus
+  the reassign rules. Removal is real: subset pre-annotation filtering ∪
+  agent-removed clusters.
+- Outputs: `annotation_proposal.json`, `annotation_removed.csv`,
+  `annotation_reassigned.csv`, `annotated.h5ad`, `report.html` (msp's report
+  with the lineage's Cell Type Annotation section).
+
+### 3. merge (`zmip.merge`, `zmip.report`)
+
+Fold every lineage back into the global object. `annotated_zmip.h5ad` keeps
+the survivors with `zmip_lineage`, `zmip_cluster` (`<lineage>:<id>`),
+`zmip_ann_coarse`, `zmip_ann_fine`, `zmip_reassigned_from`; `msp_ann_*`
+stay for the audit trail. No global re-embedding here (next round's job):
+the global figures use msp's UMAP. Archives `zmip_removed.csv` (every
+removed cell with lineage, cluster, sources) and `zmip_reassigned.csv`.
+`report.html`: plan · lineages (linked per-lineage reports) · final
+annotation · removed & reassigned.

zmip-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = ["setuptools>=64"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "zmip"
+version = "0.1.0"
+description = "Zoom-in pipeline: per-lineage re-embedding, foreign-lineage scoring and agent refinement of an msp annotation"
+readme = "README.md"
+license = "MIT"
+authors = [{ name = "chansigit", email = "chansigit@gmail.com" }]
+requires-python = ">=3.10"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Intended Audience :: Science/Research",
+    "Topic :: Scientific/Engineering :: Bio-Informatics",
+]
+dependencies = [
+    "msp-sc>=0.2.0",   # PyPI name of msp (import name `msp`)
+    "claude-agent-sdk",
+    "scanpy",
+    "anndata",
+    "pandas",
+    "numpy",
+    "scipy",
+    "matplotlib",
+]
+
+[project.urls]
+Homepage = "https://github.com/chansigit/zmip"
+
+[tool.setuptools]
+packages = ["zmip"]

zmip-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

zmip-0.1.0/zmip/__init__.py

@@ -0,0 +1,23 @@
+"""zmip (zoom-in pipeline): per-lineage refinement of an msp annotation.
+
+    plan      UMAP-connected lineages from the coarse labels (agent + host rules)
+    zoom      each lineage re-embedded on its own (msp.integrate_adata), scored
+              for foreign-lineage signal, annotated by its own agent
+              (refine fine labels / remove noise / reassign / recluster)
+    merge     fold back with real removal → annotated_zmip.h5ad + report.html
+
+Command line:
+    python -m zmip annotated.h5ad --outdir zmip_out [--min-cells 800] [--model ...]
+    python -m zmip.report zmip_out
+
+Depends on msp (integration core, plots, report machinery) and needs the
+claude-agent-sdk for both agent steps.
+"""
+
+from .foreign import lineage_markers, score_foreign
+from .merge import merge_back
+from .plan import DEFAULT_MIN_CELLS, plan_lineages, validate_plan
+from .report import generate_report
+
+__all__ = ["DEFAULT_MIN_CELLS", "generate_report", "lineage_markers", "merge_back", "plan_lineages",
+           "score_foreign", "validate_plan"]

zmip-0.1.0/zmip/__main__.py

@@ -0,0 +1,151 @@
+"""python -m zmip: zoom-in pass over an msp annotated.h5ad.
+
+  plan      agent groups coarse labels into UMAP-connected lineages, picks
+            which to zoom (>= --min-cells)                 → zmip_plan.json
+  markers   lineage-level marker lists for foreign-lineage scores
+                                                           → lineage_markers.csv
+  per lineage (sequential): subset → msp.integrate_adata (re-embed) →
+            foreign scores → annotation agent (refine / remove / reassign /
+            recluster) → <lineage>/{annotation_proposal.json, annotated.h5ad,
+            report.html}
+  merge     fold back, real removal                        → annotated_zmip.h5ad,
+            zmip_removed.csv, zmip_reassigned.csv, report.html
+
+Re-running resumes: the plan is reused, lineages whose contract files exist
+are skipped; --force redoes everything. One lineage (or none above the
+threshold) → nothing is zoomed and annotated_zmip.h5ad carries the msp
+labels unchanged.
+"""
+
+import argparse
+import os
+import sys
+
+import pandas as pd
+import scanpy as sc
+
+from msp.integrate import integrate_adata
+from msp.plots import save_single_umap, slug
+
+from .annotate import PREV_SUFFIX, PREVIOUS_COLS, annotate_lineage
+from .foreign import lineage_markers, score_foreign
+from .merge import merge_back
+from .plan import DEFAULT_MIN_CELLS, plan_lineages
+from .report import generate_report
+from msp.report import write_report_context
+
+parser = argparse.ArgumentParser(prog="zmip", description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+parser.add_argument("h5ad", help="msp annotated.h5ad (survivors with msp_ann_coarse/msp_ann_fine)")
+parser.add_argument("--outdir", required=True)
+parser.add_argument("--coarse-col", default="msp_ann_coarse")
+parser.add_argument("--fine-col", default="msp_ann_fine")
+parser.add_argument("--batch-col", default=None, help="defaults to uns['msp']['batch_col']")
+parser.add_argument("--species", default=None, help="defaults to uns['msp']['species']")
+parser.add_argument("--min-cells", type=int, default=DEFAULT_MIN_CELLS,
+                    help=f"smallest lineage that gets zoomed (default {DEFAULT_MIN_CELLS})")
+parser.add_argument("--resolutions", type=float, nargs="+", default=[0.3, 1.0, 2.0])
+parser.add_argument("--n-top-genes", type=int, default=2000)
+parser.add_argument("--n-pcs", type=int, default=50)
+parser.add_argument("--n-neighbors", type=int, default=15)
+parser.add_argument("--harmony", action="append", default=[], metavar="KEY=VALUE",
+                    help="harmonypy override for the per-lineage re-embedding, repeatable")
+parser.add_argument("--language", default="English")
+parser.add_argument("--model", default=None)
+parser.add_argument("--effort", default=None, choices=["low", "medium", "high", "xhigh", "max"])
+parser.add_argument("--max-turns", type=int, default=200)
+parser.add_argument("--report-context", default=None, metavar="TEXT",
+                    help='where this run sits, for report titles (e.g. "round 2 · fu2022-meniscus")')
+parser.add_argument("--force", action="store_true")
+args = parser.parse_args()
+
+
+def _kv(items):
+    def conv(v):
+        for cast in (int, float):
+            try:
+                return cast(v)
+            except ValueError:
+                pass
+        return v
+    out = {}
+    for it in items:
+        if "=" not in it:
+            sys.exit(f"--harmony expects KEY=VALUE, got {it!r}")
+        k, v = it.split("=", 1)
+        out[k.strip()] = [conv(x) for x in v.split(",")] if "," in v else conv(v)
+    return out
+
+
+out = os.path.abspath(args.outdir)
+os.makedirs(out, exist_ok=True)
+write_report_context(out, args.report_context)
+ad = sc.read_h5ad(args.h5ad)
+meta = ad.uns.get("msp", {})
+batch_col = args.batch_col or meta.get("batch_col")
+if not batch_col:
+    sys.exit("no --batch-col and uns['msp']['batch_col'] absent")
+species = args.species or (meta.get("species") or None)
+for c in (args.coarse_col, args.fine_col):
+    if c not in ad.obs:
+        sys.exit(f"obs[{c!r}] missing — input must be msp's annotated.h5ad (or pass --coarse-col/--fine-col)")
+print(f"== {ad.n_obs} cells, batch={batch_col!r}, species={species}", flush=True)
+
+plan = plan_lineages(ad, args.coarse_col, batch_col, out, min_cells=args.min_cells, species=species,
+                     model=args.model, effort=args.effort)
+label_to_lineage = {lab: ln["name"] for ln in plan["lineages"] for lab in ln["coarse_labels"]}
+ad.obs["_zmip_lineage"] = ad.obs[args.coarse_col].astype(str).map(label_to_lineage).astype("category")
+for ln in plan["lineages"]:
+    print(f"== lineage {ln['name']}: {ln['coarse_labels']} n={ln['n_cells']} zoom={ln['zoom']}", flush=True)
+
+markers_p = os.path.join(out, "lineage_markers.csv")
+if os.path.exists(markers_p) and not args.force:
+    mk = pd.read_csv(markers_p)
+    markers = {g: mk.loc[mk["lineage"] == g, "gene"].tolist() for g in mk["lineage"].unique()}
+else:
+    print("== lineage-level markers (for foreign-lineage scores)", flush=True)
+    markers = lineage_markers(ad, "_zmip_lineage", out)
+
+all_labels = set(label_to_lineage)
+zoomed = [ln for ln in plan["lineages"] if ln["zoom"]]
+if not zoomed:
+    print("== no lineage reaches min_cells — nothing to zoom; passing msp labels through", flush=True)
+results = {}
+harmony_kwargs = _kv(args.harmony)
+keys_for_foreign = [f"msp_leiden_r{r}" for r in args.resolutions if r in (1.0, 2.0)]
+
+for ln in zoomed:
+    name, labels = ln["name"], ln["coarse_labels"]
+    d = os.path.join(out, slug(name))
+    contract = [os.path.join(d, f) for f in ("annotation_proposal.json", "annotated.h5ad", "report.html")]
+    if all(os.path.exists(p) for p in contract) and not args.force:
+        print(f"== [{name}] already done — skipping (resume)", flush=True)
+        results[name] = {"dir": d, "removed": pd.read_csv(os.path.join(d, "annotation_removed.csv")),
+                         "reassigned": pd.read_csv(os.path.join(d, "annotation_reassigned.csv"))}
+        continue
+    sub = ad[ad.obs[args.coarse_col].astype(str).isin(labels).values].copy()
+    for c in PREVIOUS_COLS:
+        src = {"msp_ann_coarse": args.coarse_col, "msp_ann_fine": args.fine_col}[c]
+        sub.obs[c + PREV_SUFFIX] = sub.obs[src].astype(str).astype("category")
+    del sub.obs["_zmip_lineage"]
+    print(f"== [{name}] re-embedding {sub.n_obs} cells", flush=True)
+    integrate_adata(sub, batch_col, d, species=species, resolutions=tuple(args.resolutions),
+                    n_top_genes=args.n_top_genes, n_pcs=args.n_pcs, n_neighbors=args.n_neighbors,
+                    harmony_kwargs=harmony_kwargs, inputs=[args.h5ad],
+                    meta_extra={"zmip_lineage": name, "zmip_coarse_labels": list(labels)})
+    figdir = os.path.join(d, "figures")
+    print(f"== [{name}] foreign-lineage scores", flush=True)
+    foreign_cols = score_foreign(sub, markers, name, keys_for_foreign, d, figdir)
+    for c in PREVIOUS_COLS:
+        col = c + PREV_SUFFIX
+        n = sub.obs[col].nunique()
+        save_single_umap(sub, col, os.path.join(figdir, f"umap_{col}.png"), repel=True,
+                         repel_fontsize=8 if n > 15 else 11, figsize=(9, 9) if n > 15 else None)
+    proposal, rm, ra = annotate_lineage(sub, d, name, labels, sorted(all_labels - set(labels)), foreign_cols,
+                                        species=species, language=args.language, model=args.model,
+                                        effort=args.effort, max_turns=args.max_turns)
+    results[name] = {"dir": d, "removed": rm, "reassigned": ra}
+    del sub
+
+merge_back(ad, plan, results, out, coarse_col=args.coarse_col, fine_col=args.fine_col)
+print(f"== report: {generate_report(out)}", flush=True)