opencode-skills-collection 4.1.7 → 4.1.8

package/bundled-skills/scarf-single-cell/references/performance-and-export.md

@@ -0,0 +1,291 @@
+# Performance and export
+
+Resource budgets, out-of-core rules, remote I/O on mounts, and exporting counts, metadata,
+markers, and pipeline-run results. Docs: <https://scarf.readthedocs.io/en/latest/concepts/memory_and_execution.html>,
+<https://scarf.readthedocs.io/en/latest/tutorials/remote_stores.html>, <https://scarf.readthedocs.io/en/latest/tutorials/import_and_export.html>,
+<https://scarf.readthedocs.io/en/latest/tutorials/custom_analyses.html>, <https://scarf.readthedocs.io/en/latest/concepts/benchmarks.html>.
+
+## When to use
+
+- Before a large run, on a shared host, or in a batch job.
+- When analysis on a mount is slow or will make many passes over counts.
+- When handing results to Scanpy, Seurat, R, a count model, or a colleague.
+- When a step may outlast the agent shell's foreground limit.
+
+## Key concepts
+
+- `mem_budget` is a planning budget (block size, write concurrency, feature batch size), not an
+  RSS cap. Graphs, marker batches, and allocator overhead come on top: in the benchmarks a 5M-cell
+  run peaked at 51.6 GiB on a 48 GiB budget, and marker search alone added about 32 GiB.
+- Resolution order: explicit `mem_budget`/`nthreads` argument, then env `SCARF_MEM_BUDGET` /
+  `SCARF_WORKERS`, then detection (cgroup and CPU-affinity aware). Specs: bytes (int), `"8G"`,
+  `"512M"` (binary units, G = 1024^3 bytes), or a fraction of detected memory such as `"0.6"`.
+  A bare `"8"` is rejected as ambiguous.
+- More workers mean more concurrent buffers and remote requests; pair them with a `mem_budget`.
+- Converters need a `mem_budget` that fits the default count layout and refuse before writing
+  when it does not (`references/data-access.md`). Prefer more memory over the smaller layout the
+  refusal names: smaller layouts slow every later gene-major read.
+- A `DataStore` is not thread-safe: call it from one thread at a time. Use separate stores or
+  separate mount targets for independent parallel analyses.
+- Opt-in parallel UMAP, tSNE, and ANN builds record the resolved worker count in provenance.
+  Pass `nthreads` explicitly when such results must stay reusable across machines.
+- Counts stream in blocks. `ds.RNA.rawData` is a lazy `ChunkedArray`; `.compute()` on a large
+  slice materializes it. `ds.to_anndata(...)` always builds `X` in memory; ordinary
+  `scarf.to_h5ad(assay, path)` and `scarf.to_mtx` stream.
+- On a mount, every pass over counts is a network read; artifacts (normalized, PCA, graph) live
+  in the local target. `local_cache` (PCA) stages normalized data only when the datastore
+  location itself is remote, so it is skipped for a local mount target.
+
+## Recipes
+
+### Set budgets and batch-friendly output
+
+```python
+import scarf
+
+scarf.configure_output(progress=False, timestamps=True)
+scarf.set_verbosity(level="INFO", filepath="scarf-run.log")   # logs go to this file, not stdout
+ds = scarf.DataStore("analysis.zarr", mem_budget="16G", nthreads=8, min_features_per_cell=-1)
+print(ds.summary().to_dict()["resources"])   # {'memory_bytes': ..., 'workers': 8, ...}
+```
+
+Environment equivalent: `SCARF_MEM_BUDGET=16G SCARF_WORKERS=8 python job.py`.
+`configure_output(level=...)` does not hide progress bars; pass `progress=False` explicitly.
+
+### Stream counts instead of materializing
+
+Slice lazily, then iterate ordered row blocks; never `.compute()` the whole matrix.
+
+```python
+import numpy as np
+
+run = ds.pipeline.open(label="baseline")
+cell_index = np.flatnonzero(run.cells.fetch_all("I"))
+hvg = np.flatnonzero(np.asarray(run.features.fetch_all("highly_variable_features"), dtype=bool))
+selected = ds.RNA.rawData[:, hvg][cell_index, :]
+detected = np.concatenate(
+    [np.count_nonzero(block, axis=1) for block in selected.stream_blocks(nthreads=4, msg="HVGs")]
+)
+print(ds.RNA.rawData[:3, :5].compute())      # tiny slices only
+```
+
+### Make repeated passes over a mount cheap
+
+Repack copies the remote counts once into a self-contained local store, keeping artifacts and
+pipeline runs. It needs disk for the full counts plus `countsT`.
+
+```bash
+python -m scarf.tools.repack_zarr work/pbmc.zarr work/pbmc_local.zarr --mem-budget 4G
+```
+
+For a mount of a few thousand cells, repack takes tens of seconds and reopening drops from 10 to
+20 s to under 1 s. The output has no `matrixSource`, and `ds.pipeline.open(label=...)` still
+works. `--data-only` drops analyses; `--nthreads`, `--profile {fast_local,cloud}` and
+`--storage-options` (JSON, e.g. `'{"skip_signature": true}'` for public S3) also exist.
+
+### Run long steps from an agent shell
+
+Some agent shells stop a foreground command after about 10 minutes, and an unbounded wait loop can
+hang for as long. Write the step as a script that logs to a file and ends with `DONE` or `FAILED`:
+
+```python
+# step_baseline.py
+import sys
+import traceback
+
+import scarf
+
+scarf.configure_output(level="WARNING", progress=False)
+try:
+    ds = scarf.DataStore("analysis.zarr", min_features_per_cell=-1)
+    run = ds.pipeline.run(label="baseline")
+    print("run", run.run_id, run.status, flush=True)
+except Exception:
+    traceback.print_exc()
+    print("FAILED", flush=True)
+    sys.exit(1)
+print("DONE", flush=True)
+```
+
+Start it detached with a timeout and keep the PID in a file (shell variables do not survive between
+agent tool calls). Make sure `timeout`'s SIGTERM reaches Python. Behind a wrapper such as `uv run`, put `timeout` inside it: Scarf then recorded
+the run as `interrupted` in 6 of 7 tests; `timeout 1800 uv run ...` left it `running` both times.
+
+```bash
+mkdir -p logs
+nohup timeout 1800 python step_baseline.py > logs/baseline.log 2>&1 &
+echo $! > logs/baseline.pid
+```
+
+Wait in bounded slices that stay under the shell limit and fail fast:
+
+```bash
+LOG=logs/baseline.log; PID=$(cat logs/baseline.pid); state=running
+for i in $(seq 1 50); do                        # 50 x 10 s, under a 10-minute limit
+  alive=1; kill -0 "$PID" 2>/dev/null || alive=0   # check first: a dead process wrote its last line
+  if grep -qx DONE "$LOG"; then state=done; break; fi
+  if grep -qx FAILED "$LOG" || grep -q Traceback "$LOG"; then state=failed; break; fi
+  if [ "$alive" = 0 ]; then state=died; break; fi
+  sleep 10
+done
+echo "state=$state"; tail -n 5 "$LOG"
+```
+
+`running`: call the loop again. `failed`: read the traceback. `died`: the process ended without a
+final line (timeout, out of memory, kill); `ds.pipeline.list_runs()` shows what was recorded. The
+variable is `state` because zsh reserves `status`.
+Check the saved PID with `kill -0`: `pgrep -f step_baseline.py` also matches the launching shell,
+so it never reports the job as gone. On macOS, `timeout` is `gtimeout` from GNU coreutils.
+
+### Export a completed run to AnnData or H5AD
+
+Run export uses the frozen cells and feature universe and holds `X` in memory. `cell_type` below
+and in later recipes is a placeholder for your own label column, not held-out author labels.
+
+```python
+adata = ds.to_anndata(run=run)               # obs: frozen fields; obsm: X_umap; X: raw counts
+live = ds.cells.to_pandas_dataframe(["ids", "cell_type", "RNA_nCounts"]).set_index("ids")
+adata.obs = adata.obs.join(live)             # live labels are not in the run: join on cell ids
+adata.obsm["X_pca"] = np.asarray(ds.load_artifact(run["pca"])["data"][:])  # run analysis cells
+graph = ds.load_graph(graph=run["connectivity_map"], symmetric=True, upper_only=False)
+assert graph.shape[0] == adata.n_obs
+adata.obsp["connectivities"] = graph
+adata.write_h5ad("baseline_run.h5ad")
+
+scarf.to_h5ad(ds.RNA, "baseline_run_plain.h5ad", run=run)   # one call, no extras
+```
+
+`adata.var_names` are feature IDs; symbols are in `adata.var["names"]`. Pass
+`snapshot_columns=["cell_type"]` to `ds.pipeline.run` to freeze live labels into future run
+exports (see `pipeline-runs-and-artifacts.md`).
+
+### Export a complete assay (streaming)
+
+```python
+scarf.to_h5ad(ds.RNA, "full_assay.h5ad")              # every cell, including I False; live metadata
+scarf.to_mtx(ds.RNA, "full_assay_mtx")                # matrix.mtx, genes.tsv, barcodes.tsv
+scarf.to_mtx(ds.RNA, "full_assay_mtx_gz", compress=True)   # Cell Ranger 3 names, .gz
+```
+
+### Export a gene panel or normalized values
+
+```python
+panel = ds.to_anndata(from_assay="RNA", feature_names=["CD3E", "MS4A1", "LYZ", "NKG7"])
+normed = ds.to_anndata(run=run, matrix="normed")      # library size scaled to 1000 per cell, no log
+```
+
+`feature_names` (exact, unique, ordered) and `feature_indexes` are mutually exclusive and are
+rejected together with `run=`. Without `run`, `cell_key` (default `"I"`) selects cells, and
+`obs_names` are the cell `ids`: a panel export is the cheapest per-cell marker table.
+
+### Export markers and metadata tables
+
+```python
+ds.export_markers_to_csv(marker=run["markers"], csv_filename="markers_wide.csv")  # names only
+ds.get_markers(marker=run["markers"]).to_csv("markers_long.csv", index=False)     # with stats
+run.cells.to_pandas_dataframe(["ids", "clusters", "doublet_score"]).to_csv("run_cells.csv", index=False)
+ds.cells.to_pandas_dataframe(["ids", "cell_type"], key="I").to_csv("cells.csv", index=False)
+```
+
+### Export pseudobulk counts
+
+Raw counts per population x sample plus a design table, for DESeq2, edgeR or another count model.
+
+```python
+bulk = ds.make_bulk(
+    "cell_type", secondary_groups="sample_id",        # one column per population x sample
+    cell_selection=run["analysis_cell_selection"],
+    aggr_type="sum", feature_label="id",              # raw counts; Ensembl ids are unique
+)
+cells = ds.cells.to_pandas_dataframe(["cell_type", "sample_id", "donor_id", "condition"])
+cells = cells[np.asarray(run.cells.fetch_all("I"), dtype=bool)]
+assert cells.groupby("sample_id")[["donor_id", "condition"]].nunique().eq(1).all().all()
+cells["column"] = cells["cell_type"].astype(str) + "_" + cells["sample_id"].astype(str)
+by_column = cells.groupby("column")
+design = by_column.first().assign(n_cells=by_column.size()).reindex(bulk.columns)  # empty -> NaN
+design["n_cells"] = design["n_cells"].fillna(0).astype(int)
+keep = design["n_cells"] >= 10                        # choose a floor and report what you drop
+assert np.allclose(bulk.to_numpy(), np.round(bulk.to_numpy()))
+bulk.loc[:, keep].round().astype("int64").to_csv("pseudobulk_counts.csv")
+design.loc[keep].to_csv("pseudobulk_design.csv")
+print(design[keep].groupby(["cell_type", "condition"])["donor_id"].nunique().unstack(fill_value=0))
+```
+
+`make_bulk` emits every group x subgroup pair (all-zero when empty), named `f"{group}_{subgroup}"`;
+a name collision raises. Sums keep the count dtype (often float32). Model one population at a time
+with donor-level terms; with repeated samples per donor, include the donor (paired) or sum its
+samples first, and avoid rank-deficient designs (integration-and-comparisons.md).
+
+### Write a self-contained subset store
+
+```python
+keep = np.flatnonzero(run.cells.fetch_all("I"))      # or cell_key="my_bool_column"
+scarf.SubsetZarr("run_cells.zarr", assays=[ds.RNA], cell_idx=keep).dump()
+sub = scarf.DataStore("run_cells.zarr")              # first open must be writable
+```
+
+The subset copies counts locally (no `matrixSource`), keeps every feature and cell column, and
+contains no artifacts or pipeline runs. Use it to hand off, or before `to_mtx` of run cells.
+
+## What a run export contains
+
+| Item | `to_anndata(run=run)` / `to_h5ad(assay, path, run=run)` | Otherwise |
+|---|---|---|
+| Cells | the run's analysis selection | `to_h5ad(ds.RNA, path)` exports all rows |
+| Features | the full feature universe; HVG flag in `var["highly_variable_features"]` | `feature_names=` without `run` |
+| `X` | raw counts; `matrix="normed"` (to_anndata only) for scaled values | |
+| `obs` | frozen fields: cell-cycle, `leiden_*`, `paris`, `clusters`, `doublet_score`, `snapshot_columns` | join `ds.cells` by `ids` |
+| `obsm` | `X_umap` | PCA: `ds.load_artifact(run["pca"])["data"]` |
+| `obsp`, markers, run report | not exported | `ds.load_graph`, `get_markers`, `run.report(format="markdown")` |
+| MTX | `to_mtx` has no `run=` | `SubsetZarr` with run cells, then `to_mtx` |
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+|---|---|---|
+| `mem_budget` / `SCARF_MEM_BUDGET` | detected system or cgroup memory | shared host or batch job; leave headroom for graphs and markers |
+| `nthreads` / `SCARF_WORKERS` | detected CPUs | shared host, remote stores, reproducible parallel UMAP/ANN |
+| `configure_output(progress=)` | `True` | `False` in logs and batch jobs |
+| `run_pca(local_cache=)` | `"auto"` | path string to keep scratch for a remote store; plan `cells x features x 4` bytes |
+| `to_anndata(matrix=)` | `"raw"` | `"normed"` for scaled values |
+| `to_h5ad(embeddings_cols=)` | `["UMAP", "tSNE"]` live column prefixes | legacy stores only; rejected with `run=` |
+| `to_mtx(compress=)` | `False` | `True` for Cell Ranger 3 file names |
+| `get_markers` / `export_markers_to_csv` `min_score`, `min_frac_exp` | `0.25`, `0.2` | widen or narrow the exported marker list |
+| `make_bulk(aggr_type=, feature_label=, pseudo_reps=)` | `"mean"`, `"index"`, `1` | `"sum"` and `"id"` for count models; never `pseudo_reps` above 1 for inference |
+| `SubsetZarr(reset_cell_filter=)` | `True` (new `I` all true) | `False` to carry `I` over |
+
+## Check before moving on
+
+- `ds.summary().to_dict()["resources"]` shows the budget actually in effect.
+- At `INFO`, lines such as `execution countsTCellBand: ... fetch=...s` show remote fetch time per
+  stage; the run report records stage timing and sampled RSS.
+- After an export: reload, compare shape to `run.cells.fetch("ids").shape` and feature count,
+  confirm `obsm["X_umap"]`, and check joined labels have no unexpected missing values.
+- Pseudobulk: integer counts, `design` aligned to `bulk.columns`, empty and tiny columns dropped,
+  enough donors per condition in each population you model.
+- A background step ended with `DONE`, and its run report says `completed`.
+
+## Pitfalls
+
+- Every `to_anndata` call (panel, run, or full) materializes `X`; size it before exporting a
+  large atlas. Ordinary `to_h5ad` streams but writes every cell, inactive ones included.
+- Run export rejects `from_assay`, `cell_key`, `feature_names`, and `feature_indexes`.
+  `to_h5ad(run=)` also rejects `embeddings_cols`, `skip_recalc_nfeats=False`, and `nthreads`.
+- Run `obs` lacks live columns (author labels, QC) unless captured with `snapshot_columns`.
+- `export_markers_to_csv` writes names only, filtered by `min_score`/`min_frac_exp`; use
+  `get_markers(marker=ref)` for statistics. Pass the exact marker ref, never a guess.
+- Remote costs for a few thousand cells on a Hugging Face mount: open 10 to 20 s, a new mount a
+  few minutes, the pipeline about a minute (cell cycle, HVG, normalization and markers each
+  re-read `countsT`), exports seconds to tens of seconds. A 4-gene panel export still reads every
+  cell's count rows.
+- `SubsetZarr` keeps all features, drops artifacts and runs, refuses an existing destination
+  unless `overwrite_existing_file=True`, and refuses a destination overlapping the source.
+- `to_mtx(compress=True)` on CELLxGENE stores writes gene biotypes as feature types; see
+  `data-access.md` before re-importing.
+
+## See also
+
+- `data-access.md`: opening stores, mounts, Cytebase, imports.
+- `pipeline-runs-and-artifacts.md`: run reports, `snapshot_columns`, artifact loading.
+- `markers-and-annotation.md`: marker statistics and thresholds.
+- `integration-and-comparisons.md`: donors, two-arm donors and tests behind a pseudobulk design.

package/bundled-skills/scarf-single-cell/references/pipeline-runs-and-artifacts.md

@@ -0,0 +1,273 @@
+# Pipeline runs and artifacts
+
+Run the standard RNA recipe, reopen and audit runs, inspect and trace immutable artifacts, branch
+from a run with exact refs, and record the analysis for handoff. Docs:
+<https://scarf.readthedocs.io/en/latest/reference/api/pipeline.html>, <https://scarf.readthedocs.io/en/latest/reference/api/artifacts.html>,
+<https://scarf.readthedocs.io/en/latest/concepts/provenance.html>, <https://scarf.readthedocs.io/en/latest/tutorials/reuse_and_tracing.html>,
+<https://scarf.readthedocs.io/en/latest/analysis_with_agents.html>.
+
+## When to use
+
+- First pass on any RNA store: one `ds.pipeline.run(label=...)` gives QC filtering, HVGs, PCA,
+  graph, UMAP, Leiden candidates, Paris, doublet scores and markers as one durable record.
+- Resuming work: open a run by label or id; inspect any result before consuming it.
+- Alternatives outside the fixed recipe: take refs from a completed run and pass them on.
+
+## Key concepts
+
+- Every result is an immutable artifact addressed by an `ArtifactRef` (frozen dataclass):
+  `scope` (`"assay"` or `"datastore"`), `kind`, `artifact_id` (64 hex chars; there is no `.id`),
+  `assay` (`None` for datastore scope). Cell selections and metadata snapshots are datastore-scoped.
+- Reuse is content-addressed: same operation, parameters and exact input refs returns the existing
+  complete artifact. Changing anything creates a new artifact; upstream ones are reused.
+  `invalidate_cache=True` on a granular method forces a fresh one.
+- `ds.pipeline.run` writes artifacts plus a run record only. It never edits live `I` or adds
+  metadata columns, and needs `zarr_mode="r+"` (the default); a read-only store raises
+  `PermissionError`, though `open`, reports and plots still work.
+- `PipelineRun` is a read-only `Mapping[str, ArtifactRef]` (`keys`, `items`, `values`, `get`,
+  `run["pca"]`) plus `run_id`, `label`, `status`, `recipe` (`"basic_rna_analysis"`), `assay`,
+  `started_at_ns`, `finished_at_ns`, `report()`, `cells`, `features`. Outputs and views work only
+  on `status == "completed"`; otherwise they raise `RuntimeError`.
+- Labels are immutable. Only a completed run acquires its label. A completed label cannot be
+  reused (`ValueError` before any work). Failed and interrupted runs reserve nothing.
+- There is no resume. A new run reuses every complete artifact from earlier attempts.
+- `run.cells` and `run.features` are frozen views of captured fields only. Live columns such as
+  author labels are absent unless listed in `snapshot_columns` on the run.
+
+## Recipes
+
+### Run the baseline
+
+Default outputs and their kinds: `input_cell_selection`, `analysis_cell_selection`
+(`cell_selection`), `feature_universe`, `highly_variable_features` (`feature_selection`),
+`cell_cycle`, `normalized`, `pca` (`reduction`), `ann_index`, `neighbors`, `connectivity_map`,
+`embedding_initialization`, `umap` (`embedding`), `leiden_0.5` ... `leiden_1.25`
+(`cluster_labels`), `paris` (`cluster_cut`), `cluster_selection`, `clusters` (the selected Leiden
+ref itself, not a copy), `doublets` (`doublet_score`), `markers` (`marker_table`). Harmony adds
+`harmony` after `pca`; disabled stages omit their keys. A fresh baseline took 22 s on local
+counts, 10 s of it one-time compilation in the graph stage; a second run in the same process took
+11 s (10x 5K PBMC docs dataset). On remote counts expect about a minute for a few thousand cells.
+
+```python
+import scarf
+
+scarf.configure_output(level="WARNING", progress=False)
+ds = scarf.DataStore("analysis.zarr", nthreads=4, min_features_per_cell=-1)
+run = ds.pipeline.run(label="baseline")
+list(run.keys())
+run["clusters"] == run["leiden_0.5"]  # True on the 10x 5K PBMC docs dataset
+```
+
+### Reopen, list and report runs
+
+`open` needs exactly one of `label` or `run_id`; an unknown label raises `KeyError`. `list_runs`
+is newest first, all statuses unless filtered, `limit=20` by default.
+
+```python
+run = ds.pipeline.open(label="baseline")
+same = ds.pipeline.open(run_id=run.run_id)
+ds.pipeline.list_runs(status="completed", limit=5)
+report = run.report(format="dict")  # keys: run, stages, summary
+[(s["stage"], s["status"], s["metrics"]["wallSeconds"]) for s in report["stages"]]
+report["run"]["config"]  # resolved settings, including filtering method and params
+len(report["summary"]["createdArtifacts"]), len(report["summary"]["reusedArtifacts"])
+print(run.report(format="markdown"))  # config, stage table, outputs, failure or interruption
+```
+
+### Read frozen fields
+
+`run.cells.columns` here: `I, ids, names, s_score, g2m_score, cell_cycle_phase, umap_1, umap_2,
+leiden_*, paris, clusters, doublet_score`, then any `snapshot_columns`; filter and Harmony columns
+are not exposed. `run.features.columns`: `I, ids, names, highly_variable_features`. `fetch` returns
+analysed rows; `fetch_all` aligns to every stored cell (3,847 and 5,025 on the 10x 5K PBMC docs
+dataset, whose live `I` holds 3,948).
+
+```python
+df = run.cells.to_pandas_dataframe(["umap_1", "umap_2", "clusters"])
+clusters = run.cells.fetch("clusters")  # int labels starting at 1
+analysed = run.cells.fetch_all("I")  # boolean mask over every stored cell
+n_hvgs = int(run.features.fetch_all("highly_variable_features").sum())
+mito = ds.cells.fetch_all("RNA_percentMito")[analysed]  # a filter column, aligned to run cells
+```
+
+### Configure a run
+
+Shortcuts and `params` sections are mutually exclusive for the same setting. Validation errors
+(unknown section, column collision, reused label, `leiden={}`) raise before any run record exists.
+`leiden.selected` replaces the silhouette choice and skips `cluster_selection`.
+
+```python
+custom = ds.pipeline.run(label="leiden_custom", leiden={"partitions": [0.4, 0.8]})
+pinned = ds.pipeline.run(
+    label="leiden_pinned",
+    params={"leiden": {"partitions": [0.5, 1.0], "selected": 1.0}},
+    umap=False, doublets=False, markers=False, cell_cycle=False, paris=False,
+)
+pinned["clusters"] == pinned["leiden_1.0"]  # True; no cluster_selection key
+labelled = ds.pipeline.run(label="with_labels", snapshot_columns=["cell_type"])  # full recompute
+"cell_type" in labelled.cells.columns  # True
+```
+
+`cell_type` stands for a label column you own; never snapshot held-out author labels.
+
+### Handle failures and interruptions
+
+A stage error raises `PipelineExecutionError` (`run_id`, `stage`, cause in `__cause__`). SIGTERM,
+SIGINT, SIGHUP or `KeyboardInterrupt` stop at the next safe checkpoint (in testing, the running
+stage finished and kept its artifact) and record `status="interrupted"`. Rerun with the same
+label; completed artifacts are reused (interrupted during UMAP: rerun 10 s, 16 reused; 10x 5K
+PBMC docs dataset). Run it as `timeout 1800 python job.py`; behind a wrapper such as `uv run`, put
+`timeout` inside it (`uv run timeout 1800 python job.py`), because `timeout 1800 uv run ...` killed the run without the handler (performance-and-export.md).
+
+```python
+from scarf import PipelineExecutionError
+
+try:
+    ds.pipeline.run(label="bad", params={"paris": {"min_cluster_size": -5}})
+except PipelineExecutionError as error:
+    failed = ds.pipeline.open(run_id=error.run_id)
+    error.stage, failed.status, failed.report(format="dict")["run"]["error"]
+interrupted = ds.pipeline.list_runs(status="interrupted")
+```
+
+A process killed mid-stage (SIGKILL, OOM, or a `timeout` outside a `uv run` wrapper) leaves a record
+stuck at `status="running"` with `summary["uncleanIncomplete"]` true; it claimed no label, so a
+rerun with the label works. Check the process, not the status. Not executed (needs a hard-killed
+finalizer): if a process died after claiming its label, the label stays blocked. After confirming
+the process is gone, call `ds.pipeline.abandon_label_claim(label=..., run_id=..., reason=...)`,
+then rerun. It refuses completed owners and has no timeout heuristic.
+
+### Inspect, load and find artifacts
+
+```python
+status = ds.inspect_artifact(run["pca"])
+status.exists, status.complete, status.path, status.operation  # ..., "run_pca"
+status.parameters  # {'dims': 21, 'feat_scaling': True}
+status.inputs  # serialized refs: normalized, feature_scaling, pca_cell_selection
+status.input_ref("normalized") == run["normalized"]  # True
+status.execution_options, status.created_at_ns, status.scarf_version
+group = ds.load_artifact(run["pca"])  # zarr group: data, loadings, center
+group["data"].shape  # (analysed cells, 21)
+
+ds.list_artifacts(complete_only=True)  # assay scope, default assay
+ds.list_artifacts(kind="cell_selection", scope="datastore")
+on_graph = ds.list_artifacts(
+    kind="cluster_labels",
+    operation="run_leiden_clustering",
+    inputs={"graph": run["connectivity_map"]},
+)
+[ds.inspect_artifact(ref).parameters["resolution"] for ref in on_graph]
+
+decision = ds.load_artifact(run["cluster_selection"])
+decision.attrs["selectedKey"], decision.attrs["candidateKeys"], decision["scores"][:]
+```
+
+### Branch from a completed run and trace lineage
+
+Pass exact run refs to granular methods. Same inputs return the run's own artifacts; new
+parameters create a branch. Explicit normalization/PCA/graph chains: features-and-graphs.md.
+Clustering choices: clustering-and-embedding.md.
+
+```python
+graph = run["connectivity_map"]
+ds.run_leiden_clustering(graph, resolution=0.5) == run["leiden_0.5"]  # reused, no recompute
+fine_clusters = ds.run_leiden_clustering(graph, resolution=2.0)  # new cluster_labels ref
+fine_markers = ds.run_marker_search(fine_clusters, features=run["feature_universe"])
+
+cells = run["analysis_cell_selection"]
+hvgs = run["highly_variable_features"]
+ds.run_normalization(cells, hvgs) == run["normalized"]  # True: reused
+ds.plots.embedding(layout=run["umap"], color_by=fine_clusters, show=False).close()
+
+lineage = ds.lineage({"selected": run["clusters"], "fine": fine_clusters})
+markdown = lineage.to_markdown()  # Mermaid graph (shared upstream once) plus details
+mermaid = lineage.to_mermaid()
+snapshot = ds.summary().to_dict()  # columns, artifact inventory, run counts, labels
+```
+
+### Record the analysis for handoff
+
+Write the prospective record (question, unit of inference, input refs, planned writes,
+alternatives, decision criteria) before the first mutating call, then extend it into the handoff:
+add metadata roles and confounding, evidence per alternative, unsupported claims, required
+validation and exported files. Refs round-trip with `to_dict()` and `ArtifactRef.from_dict()`.
+
+```python
+import json
+from pathlib import Path
+
+out = Path("handoff")
+out.mkdir(exist_ok=True)
+refs = {"cellSelection": "analysis_cell_selection", "featureSelection": "highly_variable_features",
+        "graph": "connectivity_map", "selectedClusters": "clusters"}
+record = {
+    "question": "Which broad populations are present?",
+    "unitOfInference": "cells from one sample; no replicate-level claims",
+    "pipelineRun": {"runId": run.run_id, "label": run.label, "status": run.status},
+    "refs": {name: run[key].to_dict() for name, key in refs.items()}
+    | {"fineClusters": fine_clusters.to_dict()},
+    "alternatives": "Leiden 0.5 (pipeline choice) vs 2.0",
+    "decision": "provisional", "openQuestions": ["Does MAD filtering remove a real population?"],
+}
+(out / "handoff.json").write_text(json.dumps(record, indent=2))
+(out / "run_report.md").write_text(run.report(format="markdown"))
+(out / "lineage.md").write_text(lineage.to_markdown())
+saved = json.loads((out / "handoff.json").read_text())
+scarf.ArtifactRef.from_dict(saved["refs"]["fineClusters"]) == fine_clusters  # True
+```
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+| --- | --- | --- |
+| `label` | `None` | Always name runs you will reopen; pick a new name per variant. |
+| `cell_key` | `"I"` | A boolean column already defines the cohort. |
+| `filtering` | `True` (pooled MAD, 3 MADs, over `RNA_nCounts/nFeatures/percentMito/percentRibo`) | Data already filtered (`False`) or per-sample or manual bounds needed (mapping; see quality-control.md). |
+| `harmony_batch_columns` | `None` | Real batch columns exist; see integration-and-comparisons.md. |
+| `hvg_count` / `pca_dims` / `neighbors_k` | `1000` / `21` / `11` | A representation choice is under test. `pca_dims=0` skips PCA (output key `reduction`). |
+| `umap` | `True` | `False` skips `embedding_initialization` and `umap`. |
+| `leiden` | `True` = `[0.5, 0.75, 1.0, 1.25]` | Other candidates: `{"partitions": [...]}`; pin with `params["leiden"]["selected"]`. |
+| `cell_cycle`, `paris` | `True` | Skip to save time (cell cycle is the slowest stage on remote counts). |
+| `doublets`, `markers` | `True` | `False` to skip; both are required off when `leiden=False`. |
+| `snapshot_columns` | `()` | Run-mode plots or `run.cells` need live columns. Choose on the first run (see Pitfalls). |
+| `params` | `None` | Stage settings (`hvg`, `pca`, `umap`, `leiden`, `tsne`, `membership_strength`, ...). |
+| `callback` | `None` | Progress events: `event.kind`, `event.stage`, `event.error`. |
+
+## Check before moving on
+
+- `run.status == "completed"` and the expected keys are present.
+- `run.report(format="markdown")`: stage statuses, wall seconds, created versus reused counts,
+  resolved `filtering` config, and any failure or interruption section.
+- Kept cells: `run.cells.fetch_all("I").sum()` versus the store total. Audit removed cells (by
+  your own clusters or a marker panel, not held-out labels) before trusting clusters.
+- `decision["scores"]` from `cluster_selection`: silhouette is a provisional baseline, not ground
+  truth.
+- For any ref you consume: `ds.inspect_artifact(ref).complete` is `True`, and `operation`,
+  `parameters` and `inputs` match what you intend.
+
+## Pitfalls
+
+- `snapshot_columns` (and Harmony columns) enter the cell snapshot that feeds filtering, so adding
+  one changes `analysis_cell_selection` and recomputes everything downstream. Same parameters under
+  a new label reused all 26 artifacts in 2 s; adding `["cell_type"]` created 24 of 27 again
+  (10x 5K PBMC docs dataset).
+- `ds.list_artifacts(kind="cell_selection")` silently returns `[]`; pass `scope="datastore"`.
+- Compare cell selections from different routes by mask, not by ref: equal values can have
+  different ids.
+- A `PipelineRun` is bound to the `DataStore` object that opened it. After reopening the store,
+  reopen the run, or plotting raises `ValueError`.
+- `run.get(key)` returns `None` for a disabled stage, but raises `RuntimeError` on a run that did
+  not complete.
+- `run.cells.fetch("clusters")` is integer, while `get_markers(...)["group_id"]` is a string.
+- In a backgrounded non-interactive shell, SIGINT is ignored; use SIGTERM to stop a run cleanly.
+  `SIGKILL` or OOM leaves an incomplete run with no cleanup.
+- Never read private Zarr paths or mutate `ds.z`, `ds.zw` or run records; use the methods above.
+
+## See also
+
+- quality-control.md: filtering policies behind `analysis_cell_selection`.
+- features-and-graphs.md: explicit normalization, PCA, ANN, neighbours and graph chains.
+- clustering-and-embedding.md: re-clustering, UMAP/t-SNE branches, partition comparison.
+- markers-and-annotation.md: `get_markers`, marker tables and labels.
+- plotting.md: `ds.plots.*(run=run, ...)` versus explicit refs.
+- performance-and-export.md: `ds.to_anndata(run=run)` and resource settings.