msp-sc 0.2.0__tar.gz

msp_sc-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 chansigit
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

msp_sc-0.2.0/PKG-INFO

@@ -0,0 +1,223 @@
+Metadata-Version: 2.4
+Name: msp-sc
+Version: 0.2.0
+Summary: Multi-sample-pipeline: harmony integration of osp per-sample outputs, cluster QC inspection and cell-type annotation agents, self-contained HTML report
+Author-email: chansigit <chansigit@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/chansigit/msp
+Classifier: Programming Language :: Python :: 3
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: scanpy
+Requires-Dist: anndata
+Requires-Dist: igraph
+Requires-Dist: harmonypy==0.2.0
+Requires-Dist: torch
+Requires-Dist: standissect-lite>=0.2.0
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: scikit-learn
+Requires-Dist: matplotlib
+Requires-Dist: seaborn
+Requires-Dist: adjustText
+Provides-Extra: agent
+Requires-Dist: claude-agent-sdk; extra == "agent"
+Dynamic: license-file
+
+# msp — multi-sample-pipeline
+
+Integrates the per-sample outputs of [osp](https://github.com/chansigit/osp)
+(one `clustered.h5ad` per 10x run) into one harmony-corrected space, then
+runs two optional Claude-agent steps on the result: a per-cluster QC
+**inspection** (proposals only) and a cell-type **annotation** (coarse/fine
+labels, explicit merges, real removal). Every step writes a self-contained
+`report.html` (all figures base64-embedded) that the next step reads.
+
+```
+osp per-sample ──▶ integrate ──▶ inspect ──▶ annotate
+                   propose-only   propose-only   removes cells
+                   integrated.h5ad              annotated.h5ad
+```
+
+## Install
+
+```bash
+pip install msp-sc                     # PyPI name; `import msp`
+# with the agent steps (needs claude-agent-sdk + Claude Code CLI credentials):
+pip install "msp-sc[agent]"            # + claude-agent-sdk for inspect / annotate
+```
+
+Dependencies of note: `harmonypy>=0.2.0` (the torch-based fork — set
+`MSP_DEVICE=cpu|cuda|mps` to override device auto-detection) and
+`standissect-lite` (minor-sibling fragment detection inside clusters).
+
+## Quick usage
+
+```bash
+# integration + report
+python -m msp A/clustered.h5ad B/clustered.h5ad --batch-col project --outdir msp_out --species human
+
+# the whole chain (integration → inspection agent → annotation agent)
+python -m msp A/clustered.h5ad B/clustered.h5ad --batch-col project --outdir msp_out \
+    --species human --annotate --model claude-sonnet-5
+
+# re-integrate one merged h5ad (e.g. a previous round's survivors) instead of per-sample inputs
+python -m msp --from-h5ad prev_round/annotated_zmip.h5ad --batch-col project --outdir msp_out2 --annotate --model claude-sonnet-5
+
+# individual steps
+python -m msp.inspect  msp_out --model claude-sonnet-5   # QC verdicts
+python -m msp.annotate msp_out --model claude-sonnet-5   # identity + merges + removal (after inspect)
+python -m msp.report   msp_out                           # rebuild report.html only
+```
+
+Re-running the same `python -m msp` command resumes: a step is skipped when
+its contract files already exist (`integrated.h5ad`+`report.html`,
+`inspection_proposal.json`, `annotation_proposal.json`+`annotated.h5ad`);
+`--force` redoes everything.
+
+```python
+from msp import run_multi_sample_pipeline, generate_report
+run_multi_sample_pipeline(["A/clustered.h5ad", "B/clustered.h5ad"], batch_col="project", outdir="msp_out")
+generate_report("msp_out")
+
+from msp.inspect import inspect_clusters     # optional agent steps
+from msp.annotate import annotate_clusters
+```
+
+## The three steps
+
+### 1. integrate (`msp.integrate`, propose-only)
+
+concat (hard checks: identical gene axis, one batch value per file,
+globally unique barcodes, no cell lost) → normalize from raw
+`layers["counts"]` → per-batch HVG → scale/PCA on the merged cells → harmony
+→ neighbors on `X_pca_harmony` → leiden at each resolution (`msp_leiden_r*`,
+default 0.3/1.0/2.0) → UMAP → standissect-lite fragments on the coarsest
+resolution → QC/DEG artifacts → `integrated.h5ad` + `report.html`.
+
+Removal *candidates* are computed but never applied:
+
+- `minor_sibling_qc.csv` — standissect fragments failing QC against their parent;
+- `cell_outliers.csv` — per-cluster doublet/ambient outliers (cell flagged only
+  when it clears BOTH gates: cluster median + 3×MAD **and** an absolute floor
+  of 0.5; OR across metrics and across r1.0/r2.0);
+- osp's own per-sample `_qc_action == "drop"` cells;
+- their union is `preannotation_removal.csv` and the "Pre-annotation
+  filtering" UMAP. The precomputed DEG tables (`deg_global_*` one-vs-rest,
+  `deg_local_*` vs the 3 nearest PAGA neighbours, at r1.0 and r2.0) exclude
+  these cells; `stress_clusters.csv` flags clusters whose top genes are a
+  dissociation-stress/mitochondrial signature.
+
+### 2. inspect (`msp.inspect`, propose-only)
+
+One agent session puts every r1.0 cluster through the five-test battery
+(markers / QC axis / composition / geometry / stability) with live tools
+(`check_genes`, `check_qc_scores`, `check_stability`, `check_deg`,
+`subcluster`). Output: `inspection_proposal.json`, `inspection_notes.md`,
+`obs["_msp_action"]` (keep/flag/drop) and `obs["_msp_verdict"]` written
+into `integrated.h5ad`, the verdict UMAP. Live DEG excludes the
+pre-annotation removal set, matching the precomputed tables.
+
+### 3. annotate (`msp.annotate`, removes cells)
+
+One agent session annotates every **r2.0** cluster. Coverage is enforced
+twice: the agent keeps one Claude Code Task per cluster
+(TaskCreate/TaskUpdate/TaskList), and the host refuses `finalize_annotation`
+until every cluster has a validated `submit_cluster`. Per cluster the agent
+answers a fixed reasoning chain — (1) distinct entity or splinter of its
+r1.0 parent/siblings, (2) coarse + fine label, or noise/low-quality →
+remove, (3) merge target or keep separate — using `cluster_context`
+(parent/siblings/PAGA neighbours/sample composition/QC/inspect verdict/prior
+label compositions), `check_genes` and `check_deg`. Prior label columns
+(osp's `_ann_coarse`/`_ann_fine`, the authors' own cell-type columns) are
+detected, not assumed, and shown as reference evidence only.
+
+Merge decisions are made in one session and validated deterministically on
+the host (union-find over `merge_target`; a merged group shares one
+coarse/fine label; one fine label belongs to one coarse label; equal fine
+labels must be merged explicitly; nothing merges into a removed cluster) —
+no separate harmonization pass.
+
+Removal is real at this step: removed = `preannotation_removal.csv` ∪
+inspect drop ∪ agent-removed clusters, archived per cell with sources in
+`annotation_removed.csv`. `annotated.h5ad` keeps the survivors with
+`msp_ann_cluster` (merged id, e.g. `1+2+4`), `msp_ann_coarse`,
+`msp_ann_fine`, `msp_ann_action`; `integrated.h5ad` is left untouched.
+
+## Output directory
+
+| file | written by | what |
+|---|---|---|
+| `integrated.h5ad` | integrate (+inspect adds `_msp_*`) | all cells, harmony space, `msp_leiden_r*`, `standissect_product` |
+| `report.html` | every step | self-contained report: Sample Summary · UMAPs · Per-cluster QC (standissect) · Leiden Cluster QC · Cluster Annotations (DEG) · Cell Type Annotation |
+| `integration_summary.csv`, `per_sample_qc.csv`, `sample_decisions.csv`* | integrate | sample-level tables (*optional, written by the caller) |
+| `cluster_qc_*.csv`, `cell_outliers.csv`, `cell_outlier_summary.csv` | integrate | per-cluster / per-cell QC |
+| `fragments_*.csv`, `overlap_*.csv`, `minor_sibling_qc.csv`, `fractal_markers.csv` | integrate | standissect-lite bundle |
+| `deg_global_*.csv`, `deg_local_*.csv`, `paga_neighbors_*.csv`, `stress_clusters.csv` | integrate | DEG at r1.0/r2.0 |
+| `preannotation_removal.csv` | integrate | union of removal candidates (cell, recommend_removal) |
+| `inspection_proposal.json`, `inspection_notes.md` | inspect | five-test verdicts per cluster |
+| `annotation_proposal.json`, `annotation_notes.md` | annotate | per-cluster labels, merges, evidence, merged groups |
+| `annotation_removed.csv` | annotate | every removed cell with its sources |
+| `annotated.h5ad` | annotate | survivors with `msp_ann_*` columns |
+| `figures/*.png` | all | one signal per file, fixed UMAP geometry |
+
+## Conventions
+
+- **Propose, never remove** until `annotate`: `integrate` and `inspect` add
+  columns and CSVs, never drop cells; computation-only exclusions (DEG) are
+  documented where they happen.
+- Inherited per-sample obs columns (QC metrics, `_ann_*`, `_qc_action`,
+  doublet calls) ride along; sample-local leiden labels are prefixed with
+  the sample value (`H12inner:3`). Per-sample embeddings/uns/layers are
+  dropped — only raw counts travel; everything integrated is recomputed.
+- Doublet detection is NOT rerun: it belongs to the per-sample stage.
+- `checkpoint`-style writes: h5ad files are written to `*.tmp.h5ad` and
+  renamed, never in place.
+- All heavy computation lives in `msp.integrate`; `msp.report` only renders
+  artifacts already on disk, so `python -m msp.report` is always safe.
+
+Driven in production by `ecarsi.crosssample`, which decides which samples
+enter integration (agent decision, archived) before calling this package.
+
+## Tuning integration
+
+`python -m msp` exposes the integration knobs; the Python entry point takes
+the same names as keyword arguments (`run_multi_sample_pipeline(...,
+n_top_genes=, n_pcs=, n_neighbors=, resolutions=, harmony_kwargs={...})`).
+
+| knob | default | CLI |
+|---|---|---|
+| HVGs per batch | 2000 | `--n-top-genes` |
+| PCs | 50 | `--n-pcs` |
+| kNN neighbours (on `X_pca_harmony`) | 15 | `--n-neighbors` |
+| leiden resolutions | 0.3 1.0 2.0 | `--resolutions` (1.0 and 2.0 required by inspect/annotate) |
+| harmony | harmonypy defaults | `--harmony KEY=VALUE` (repeatable) |
+
+Harmony is called as `harmonypy.run_harmony(X_pca, obs[[batch_col]],
+batch_col, random_state=0, device=<auto or $MSP_DEVICE>, **harmony_kwargs)`;
+anything not overridden is harmonypy's default:
+
+| harmony parameter | default | meaning |
+|---|---|---|
+| `theta` | 2 (per covariate) | diversity penalty — higher = stronger mixing across batches |
+| `lamb` | 1 | ridge penalty on the correction; `-1` = auto-estimate (R behaviour, uses `alpha`=0.2) |
+| `sigma` | 0.1 | soft k-means width — larger = softer cluster assignment |
+| `nclust` | min(round(N/30), 100) | number of harmony clusters |
+| `tau` | 0 | discounting for small batches (expected cells per cluster) |
+| `block_size` | 0.05 | fraction of cells updated per block |
+| `max_iter_harmony` | 10 | outer iterations |
+| `max_iter_kmeans` | 20 | inner clustering iterations |
+| `epsilon_cluster` / `epsilon_harmony` | 1e-5 / 1e-4 | convergence tolerances |
+
+Example: gentler correction that keeps more within-batch structure and runs longer:
+
+```bash
+python -m msp ... --harmony theta=1 --harmony max_iter_harmony=20
+```
+
+The effective overrides are recorded in `uns["msp"]["harmony"]` of
+`integrated.h5ad` (empty = all defaults).

msp_sc-0.2.0/README.md

@@ -0,0 +1,193 @@
+# msp — multi-sample-pipeline
+
+Integrates the per-sample outputs of [osp](https://github.com/chansigit/osp)
+(one `clustered.h5ad` per 10x run) into one harmony-corrected space, then
+runs two optional Claude-agent steps on the result: a per-cluster QC
+**inspection** (proposals only) and a cell-type **annotation** (coarse/fine
+labels, explicit merges, real removal). Every step writes a self-contained
+`report.html` (all figures base64-embedded) that the next step reads.
+
+```
+osp per-sample ──▶ integrate ──▶ inspect ──▶ annotate
+                   propose-only   propose-only   removes cells
+                   integrated.h5ad              annotated.h5ad
+```
+
+## Install
+
+```bash
+pip install msp-sc                     # PyPI name; `import msp`
+# with the agent steps (needs claude-agent-sdk + Claude Code CLI credentials):
+pip install "msp-sc[agent]"            # + claude-agent-sdk for inspect / annotate
+```
+
+Dependencies of note: `harmonypy>=0.2.0` (the torch-based fork — set
+`MSP_DEVICE=cpu|cuda|mps` to override device auto-detection) and
+`standissect-lite` (minor-sibling fragment detection inside clusters).
+
+## Quick usage
+
+```bash
+# integration + report
+python -m msp A/clustered.h5ad B/clustered.h5ad --batch-col project --outdir msp_out --species human
+
+# the whole chain (integration → inspection agent → annotation agent)
+python -m msp A/clustered.h5ad B/clustered.h5ad --batch-col project --outdir msp_out \
+    --species human --annotate --model claude-sonnet-5
+
+# re-integrate one merged h5ad (e.g. a previous round's survivors) instead of per-sample inputs
+python -m msp --from-h5ad prev_round/annotated_zmip.h5ad --batch-col project --outdir msp_out2 --annotate --model claude-sonnet-5
+
+# individual steps
+python -m msp.inspect  msp_out --model claude-sonnet-5   # QC verdicts
+python -m msp.annotate msp_out --model claude-sonnet-5   # identity + merges + removal (after inspect)
+python -m msp.report   msp_out                           # rebuild report.html only
+```
+
+Re-running the same `python -m msp` command resumes: a step is skipped when
+its contract files already exist (`integrated.h5ad`+`report.html`,
+`inspection_proposal.json`, `annotation_proposal.json`+`annotated.h5ad`);
+`--force` redoes everything.
+
+```python
+from msp import run_multi_sample_pipeline, generate_report
+run_multi_sample_pipeline(["A/clustered.h5ad", "B/clustered.h5ad"], batch_col="project", outdir="msp_out")
+generate_report("msp_out")
+
+from msp.inspect import inspect_clusters     # optional agent steps
+from msp.annotate import annotate_clusters
+```
+
+## The three steps
+
+### 1. integrate (`msp.integrate`, propose-only)
+
+concat (hard checks: identical gene axis, one batch value per file,
+globally unique barcodes, no cell lost) → normalize from raw
+`layers["counts"]` → per-batch HVG → scale/PCA on the merged cells → harmony
+→ neighbors on `X_pca_harmony` → leiden at each resolution (`msp_leiden_r*`,
+default 0.3/1.0/2.0) → UMAP → standissect-lite fragments on the coarsest
+resolution → QC/DEG artifacts → `integrated.h5ad` + `report.html`.
+
+Removal *candidates* are computed but never applied:
+
+- `minor_sibling_qc.csv` — standissect fragments failing QC against their parent;
+- `cell_outliers.csv` — per-cluster doublet/ambient outliers (cell flagged only
+  when it clears BOTH gates: cluster median + 3×MAD **and** an absolute floor
+  of 0.5; OR across metrics and across r1.0/r2.0);
+- osp's own per-sample `_qc_action == "drop"` cells;
+- their union is `preannotation_removal.csv` and the "Pre-annotation
+  filtering" UMAP. The precomputed DEG tables (`deg_global_*` one-vs-rest,
+  `deg_local_*` vs the 3 nearest PAGA neighbours, at r1.0 and r2.0) exclude
+  these cells; `stress_clusters.csv` flags clusters whose top genes are a
+  dissociation-stress/mitochondrial signature.
+
+### 2. inspect (`msp.inspect`, propose-only)
+
+One agent session puts every r1.0 cluster through the five-test battery
+(markers / QC axis / composition / geometry / stability) with live tools
+(`check_genes`, `check_qc_scores`, `check_stability`, `check_deg`,
+`subcluster`). Output: `inspection_proposal.json`, `inspection_notes.md`,
+`obs["_msp_action"]` (keep/flag/drop) and `obs["_msp_verdict"]` written
+into `integrated.h5ad`, the verdict UMAP. Live DEG excludes the
+pre-annotation removal set, matching the precomputed tables.
+
+### 3. annotate (`msp.annotate`, removes cells)
+
+One agent session annotates every **r2.0** cluster. Coverage is enforced
+twice: the agent keeps one Claude Code Task per cluster
+(TaskCreate/TaskUpdate/TaskList), and the host refuses `finalize_annotation`
+until every cluster has a validated `submit_cluster`. Per cluster the agent
+answers a fixed reasoning chain — (1) distinct entity or splinter of its
+r1.0 parent/siblings, (2) coarse + fine label, or noise/low-quality →
+remove, (3) merge target or keep separate — using `cluster_context`
+(parent/siblings/PAGA neighbours/sample composition/QC/inspect verdict/prior
+label compositions), `check_genes` and `check_deg`. Prior label columns
+(osp's `_ann_coarse`/`_ann_fine`, the authors' own cell-type columns) are
+detected, not assumed, and shown as reference evidence only.
+
+Merge decisions are made in one session and validated deterministically on
+the host (union-find over `merge_target`; a merged group shares one
+coarse/fine label; one fine label belongs to one coarse label; equal fine
+labels must be merged explicitly; nothing merges into a removed cluster) —
+no separate harmonization pass.
+
+Removal is real at this step: removed = `preannotation_removal.csv` ∪
+inspect drop ∪ agent-removed clusters, archived per cell with sources in
+`annotation_removed.csv`. `annotated.h5ad` keeps the survivors with
+`msp_ann_cluster` (merged id, e.g. `1+2+4`), `msp_ann_coarse`,
+`msp_ann_fine`, `msp_ann_action`; `integrated.h5ad` is left untouched.
+
+## Output directory
+
+| file | written by | what |
+|---|---|---|
+| `integrated.h5ad` | integrate (+inspect adds `_msp_*`) | all cells, harmony space, `msp_leiden_r*`, `standissect_product` |
+| `report.html` | every step | self-contained report: Sample Summary · UMAPs · Per-cluster QC (standissect) · Leiden Cluster QC · Cluster Annotations (DEG) · Cell Type Annotation |
+| `integration_summary.csv`, `per_sample_qc.csv`, `sample_decisions.csv`* | integrate | sample-level tables (*optional, written by the caller) |
+| `cluster_qc_*.csv`, `cell_outliers.csv`, `cell_outlier_summary.csv` | integrate | per-cluster / per-cell QC |
+| `fragments_*.csv`, `overlap_*.csv`, `minor_sibling_qc.csv`, `fractal_markers.csv` | integrate | standissect-lite bundle |
+| `deg_global_*.csv`, `deg_local_*.csv`, `paga_neighbors_*.csv`, `stress_clusters.csv` | integrate | DEG at r1.0/r2.0 |
+| `preannotation_removal.csv` | integrate | union of removal candidates (cell, recommend_removal) |
+| `inspection_proposal.json`, `inspection_notes.md` | inspect | five-test verdicts per cluster |
+| `annotation_proposal.json`, `annotation_notes.md` | annotate | per-cluster labels, merges, evidence, merged groups |
+| `annotation_removed.csv` | annotate | every removed cell with its sources |
+| `annotated.h5ad` | annotate | survivors with `msp_ann_*` columns |
+| `figures/*.png` | all | one signal per file, fixed UMAP geometry |
+
+## Conventions
+
+- **Propose, never remove** until `annotate`: `integrate` and `inspect` add
+  columns and CSVs, never drop cells; computation-only exclusions (DEG) are
+  documented where they happen.
+- Inherited per-sample obs columns (QC metrics, `_ann_*`, `_qc_action`,
+  doublet calls) ride along; sample-local leiden labels are prefixed with
+  the sample value (`H12inner:3`). Per-sample embeddings/uns/layers are
+  dropped — only raw counts travel; everything integrated is recomputed.
+- Doublet detection is NOT rerun: it belongs to the per-sample stage.
+- `checkpoint`-style writes: h5ad files are written to `*.tmp.h5ad` and
+  renamed, never in place.
+- All heavy computation lives in `msp.integrate`; `msp.report` only renders
+  artifacts already on disk, so `python -m msp.report` is always safe.
+
+Driven in production by `ecarsi.crosssample`, which decides which samples
+enter integration (agent decision, archived) before calling this package.
+
+## Tuning integration
+
+`python -m msp` exposes the integration knobs; the Python entry point takes
+the same names as keyword arguments (`run_multi_sample_pipeline(...,
+n_top_genes=, n_pcs=, n_neighbors=, resolutions=, harmony_kwargs={...})`).
+
+| knob | default | CLI |
+|---|---|---|
+| HVGs per batch | 2000 | `--n-top-genes` |
+| PCs | 50 | `--n-pcs` |
+| kNN neighbours (on `X_pca_harmony`) | 15 | `--n-neighbors` |
+| leiden resolutions | 0.3 1.0 2.0 | `--resolutions` (1.0 and 2.0 required by inspect/annotate) |
+| harmony | harmonypy defaults | `--harmony KEY=VALUE` (repeatable) |
+
+Harmony is called as `harmonypy.run_harmony(X_pca, obs[[batch_col]],
+batch_col, random_state=0, device=<auto or $MSP_DEVICE>, **harmony_kwargs)`;
+anything not overridden is harmonypy's default:
+
+| harmony parameter | default | meaning |
+|---|---|---|
+| `theta` | 2 (per covariate) | diversity penalty — higher = stronger mixing across batches |
+| `lamb` | 1 | ridge penalty on the correction; `-1` = auto-estimate (R behaviour, uses `alpha`=0.2) |
+| `sigma` | 0.1 | soft k-means width — larger = softer cluster assignment |
+| `nclust` | min(round(N/30), 100) | number of harmony clusters |
+| `tau` | 0 | discounting for small batches (expected cells per cluster) |
+| `block_size` | 0.05 | fraction of cells updated per block |
+| `max_iter_harmony` | 10 | outer iterations |
+| `max_iter_kmeans` | 20 | inner clustering iterations |
+| `epsilon_cluster` / `epsilon_harmony` | 1e-5 / 1e-4 | convergence tolerances |
+
+Example: gentler correction that keeps more within-batch structure and runs longer:
+
+```bash
+python -m msp ... --harmony theta=1 --harmony max_iter_harmony=20
+```
+
+The effective overrides are recorded in `uns["msp"]["harmony"]` of
+`integrated.h5ad` (empty = all defaults).

msp_sc-0.2.0/msp/__init__.py

@@ -0,0 +1,35 @@
+"""msp (multi-sample-pipeline): integrate osp per-sample outputs (harmony)
+→ multi-resolution leiden + UMAP → cluster QC / DEG tables → self-contained
+HTML report, with two optional Claude-agent steps that run afterwards:
+
+    integrate  (msp.integrate)  propose-only: nothing deleted, nothing named
+    inspect    (msp.inspect)    per-cluster five-test QC verdicts, proposals only
+    annotate   (msp.annotate)   coarse/fine cell identity on msp_leiden_r2.0,
+                                explicit merges, REAL removal → annotated.h5ad
+
+Entry points:
+    from msp import run_multi_sample_pipeline, generate_report
+
+    run_multi_sample_pipeline(["A/clustered.h5ad", "B/clustered.h5ad"],
+                              batch_col="project", outdir="msp_out")
+    generate_report("msp_out")
+
+Command line:
+    python -m msp A/clustered.h5ad B/clustered.h5ad --batch-col project --outdir msp_out
+    python -m msp ... --inspect --annotate --model claude-sonnet-5   # full chain
+    python -m msp.inspect  msp_out          # QC inspection agent only
+    python -m msp.annotate msp_out          # annotation agent only (after inspect)
+    python -m msp.report   msp_out          # rebuild the report only
+
+msp.inspect / msp.annotate are intentionally not imported here — they depend
+on the optional claude-agent-sdk (`pip install "msp[agent]"`); use
+`from msp.inspect import inspect_clusters` / `from msp.annotate import
+annotate_clusters` when needed.
+"""
+
+from .integrate import integrate_adata, load_and_merge, run_multi_sample_pipeline
+from .plots import save_single_umap
+from .report import generate_report
+
+__all__ = ["integrate_adata", "load_and_merge", "run_multi_sample_pipeline", "generate_report",
+           "save_single_umap"]

msp_sc-0.2.0/msp/__main__.py

@@ -0,0 +1,122 @@
+"""python -m msp: integrate osp per-sample outputs (concat → harmony → leiden →
+UMAP → QC/DEG tables → HTML report), end to end.
+
+With --inspect, the per-cluster QC inspection agent (msp.inspect) runs
+afterwards; with --annotate, the cell-type annotation agent (msp.annotate)
+runs after that (both need the optional claude-agent-sdk). Each step is
+skipped when its contract file already exists, so re-running the same
+command resumes where it stopped; --force redoes everything.
+"""
+
+import argparse
+import os
+import sys
+
+from .integrate import integrate_adata, run_multi_sample_pipeline
+from .report import generate_report, write_report_context
+
+parser = argparse.ArgumentParser(prog="msp", description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+parser.add_argument("inputs", nargs="*", help="per-sample clustered.h5ad files (osp outputs)")
+parser.add_argument("--from-h5ad", default=None, metavar="H5AD",
+                    help="instead of per-sample inputs: one already-merged h5ad with layers['counts'] "
+                         "(e.g. a previous round's annotated_zmip.h5ad) — re-integrated from scratch via "
+                         "integrate_adata; prior obs columns ride along as annotation evidence")
+parser.add_argument("--batch-col", required=True, help="obs column naming the sample/batch")
+parser.add_argument("--outdir", required=True)
+parser.add_argument("--species", default=None, help="stored in uns['msp']; context for the agents")
+parser.add_argument("--resolutions", type=float, nargs="+", default=[0.3, 1.0, 2.0],
+                    help="leiden resolutions; 1.0 and 2.0 must be present for inspect/annotate")
+parser.add_argument("--n-top-genes", type=int, default=2000)
+parser.add_argument("--n-pcs", type=int, default=50)
+parser.add_argument("--n-neighbors", type=int, default=15)
+parser.add_argument("--harmony", action="append", default=[], metavar="KEY=VALUE",
+                    help="harmonypy.run_harmony override, repeatable: e.g. --harmony theta=1 "
+                         "--harmony lamb=-1 --harmony max_iter_harmony=20 --harmony sigma=0.2 "
+                         "(defaults: theta=2, lamb=1, sigma=0.1, nclust=min(N/30,100), "
+                         "max_iter_harmony=10, max_iter_kmeans=20)")
+parser.add_argument("--inspect", action="store_true",
+                    help="after integration, run the per-cluster QC inspection agent (msp.inspect)")
+parser.add_argument("--annotate", action="store_true",
+                    help="after inspection, run the cell-type annotation agent (msp.annotate); "
+                         "implies --inspect")
+parser.add_argument("--language", default="English", help='agent prose language (default "English")')
+parser.add_argument("--model", default=None, help='model for the agents, e.g. "claude-sonnet-5"')
+parser.add_argument("--effort", default=None, choices=["low", "medium", "high", "xhigh", "max"],
+                    help="reasoning effort for the agents (models that support it)")
+parser.add_argument("--max-turns", type=int, default=None,
+                    help="agent turn budget (defaults: inspect 100, annotate 200)")
+parser.add_argument("--report-context", default=None, metavar="TEXT",
+                    help='where this run sits, for report titles (e.g. "round 2 · fu2022-meniscus"); '
+                         "persisted in <outdir>/report_context.txt so later report refreshes keep it")
+parser.add_argument("--force", action="store_true", help="redo steps whose outputs already exist")
+args = parser.parse_args()
+
+if bool(args.inputs) == bool(args.from_h5ad):
+    sys.exit("give either per-sample inputs or --from-h5ad, not both / neither")
+if args.annotate:
+    args.inspect = True
+if args.inspect and not {1.0, 2.0} <= set(args.resolutions):
+    sys.exit("--inspect/--annotate need leiden resolutions 1.0 and 2.0 (see --resolutions)")
+
+out = args.outdir
+write_report_context(out, args.report_context)
+
+
+def _parse_kv(items):
+    """KEY=VALUE → {key: number|list|str}; comma-separated values become lists."""
+    def conv(v):
+        for cast in (int, float):
+            try:
+                return cast(v)
+            except ValueError:
+                pass
+        return v
+    out = {}
+    for it in items:
+        if "=" not in it:
+            sys.exit(f"--harmony expects KEY=VALUE, got {it!r}")
+        k, v = it.split("=", 1)
+        out[k.strip()] = [conv(x) for x in v.split(",")] if "," in v else conv(v)
+    return out
+
+
+harmony_kwargs = _parse_kv(args.harmony)
+
+
+def _done(*names):
+    return all(os.path.exists(os.path.join(out, n)) for n in names)
+
+
+if args.force or not _done("integrated.h5ad", "report.html"):
+    kw = dict(species=args.species, resolutions=tuple(args.resolutions), n_top_genes=args.n_top_genes,
+              n_pcs=args.n_pcs, n_neighbors=args.n_neighbors, harmony_kwargs=harmony_kwargs)
+    if args.from_h5ad:
+        import scanpy as sc
+
+        ad = sc.read_h5ad(args.from_h5ad)
+        _, summary = integrate_adata(ad, args.batch_col, out, inputs=[args.from_h5ad], **kw)
+    else:
+        _, summary = run_multi_sample_pipeline(args.inputs, batch_col=args.batch_col, outdir=out, **kw)
+    print(summary)
+    print(f"report: {generate_report(out)}")
+else:
+    print(f"[resume] integration already done in {out} (integrated.h5ad + report.html) — skipping")
+
+agent_kw = dict(species=args.species, language=args.language, model=args.model, effort=args.effort)
+
+if args.inspect:
+    if args.force or not _done("inspection_proposal.json"):
+        from .inspect import inspect_clusters
+
+        inspect_clusters(out, max_turns=args.max_turns or 100, **agent_kw)
+    else:
+        print(f"[resume] inspection_proposal.json exists in {out} — skipping inspect")
+
+if args.annotate:
+    if args.force or not _done("annotation_proposal.json", "annotated.h5ad"):
+        from .annotate import annotate_clusters
+
+        annotate_clusters(out, max_turns=args.max_turns or 200, **agent_kw)
+    else:
+        print(f"[resume] annotation_proposal.json + annotated.h5ad exist in {out} — skipping annotate")

msp_sc-0.2.0/msp/agent_util.py

@@ -0,0 +1,70 @@
+"""Shared agent-call helper for every claude-agent-sdk session in msp / zmip.
+
+The CLI itself retries transient API errors (429 / overloaded / 5xx) with
+backoff, but a subscription usage window that is used up ("Claude usage
+limit reached …") ends the session with an error result and the CLI does
+NOT wait for the reset. A self-driving loop must not stop for that: this
+wrapper recognises limit-type failures and re-runs the whole query after a
+wait (session state is in-memory on the host — submitted entries persist,
+the agent simply starts its investigation again), bounded by a total wait
+budget. Any other failure is raised immediately.
+
+    async for message in run_query(prompt, options, label="inspect"):
+        ...
+
+Env: AGENT_LIMIT_WAIT_MIN (minutes between retries, default 10),
+AGENT_LIMIT_WAIT_MAX_H (total hours to keep waiting, default 12).
+"""
+
+import asyncio
+import os
+import re
+import time
+
+LIMIT_PATTERN = re.compile(
+    r"usage limit|rate[ _-]?limit|limit will reset|resets at|too many requests|overloaded|"
+    r"quota|429|capacity|out of extra usage|spend limit",
+    re.IGNORECASE,
+)
+
+
+def is_limit_error(text) -> bool:
+    return bool(text) and bool(LIMIT_PATTERN.search(str(text)))
+
+
+class AgentLimitExhausted(RuntimeError):
+    pass
+
+
+async def run_query(prompt, options, label="agent"):
+    """Yield the SDK's messages exactly like query(); if the run ends in a
+    limit-type error, wait and start over (bounded)."""
+    from claude_agent_sdk import ResultMessage, query
+
+    wait_min = float(os.environ.get("AGENT_LIMIT_WAIT_MIN", "10"))
+    max_h = float(os.environ.get("AGENT_LIMIT_WAIT_MAX_H", "12"))
+    waited = 0.0
+    attempt = 0
+    while True:
+        attempt += 1
+        limit_hit = None
+        try:
+            async for message in query(prompt=prompt, options=options):
+                if isinstance(message, ResultMessage) and getattr(message, "is_error", False) \
+                        and is_limit_error(getattr(message, "result", "")):
+                    limit_hit = message.result
+                    continue  # swallow: the retry below replaces this result
+                yield message
+        except Exception as e:  # transport-level failure carrying a limit message
+            if not is_limit_error(str(e)):
+                raise
+            limit_hit = str(e)
+        if limit_hit is None:
+            return
+        if waited / 3600 >= max_h:
+            raise AgentLimitExhausted(f"[{label}] usage limit still in force after {waited / 3600:.1f} h: {limit_hit}")
+        print(f"== [{label}] usage/rate limit (attempt {attempt}): {str(limit_hit)[:160]!r} — "
+              f"waiting {wait_min:.0f} min, {max_h - waited / 3600:.1f} h of wait budget left", flush=True)
+        t0 = time.time()
+        await asyncio.sleep(wait_min * 60)
+        waited += time.time() - t0