opencode-skills-collection 4.1.7 → 4.1.8

package/bundled-skills/scarf-single-cell/references/integration-and-comparisons.md

@@ -0,0 +1,295 @@
+# Study design, integration, and condition comparisons
+
+Identify donors, samples, batches and conditions, correct a technical batch with Harmony and score
+it, then compare composition and expression at the donor level. Docs: <https://scarf.readthedocs.io/en/latest/tutorials/batch_correction.html>, and in the same folder
+`pseudobulk_and_differential_expression.html`, `condition_comparisons.html`,
+`dataset_merging.html`, `mapping_and_label_transfer.html`, `downsampling.html`.
+
+## When to use
+
+- The store has several donors, samples, batches or conditions (most CELLxGENE atlases), or a
+  layout separates by donor or batch and you must decide whether to correct it.
+- The question compares conditions (abundance of a population, or expression within one), or you
+  need to combine stores, map a query onto a fixed reference, or work on fewer cells.
+
+## Key concepts
+
+- **Units.** A donor is the biological replicate; a sample is one specimen from one donor; a
+  library is one capture; a batch is a processing group. Cells are never replicates: the n of a
+  condition comparison is the number of donors. Sample, library and batch columns may be absent.
+- **Nesting.** Donors nested in conditions need between-donor tests. A donor sampled at several
+  timepoints is repeated measures: pair on the donor. A donor in both arms needs the rules below.
+- **Denominators.** A share of all cells moves when any other population moves. A state's share of
+  its lineage parent (an activated state as a share of CD14 monocytes) does not depend on the
+  parent's abundance. Excluding contaminants (erythrocytes, platelets) or unresolved cells raises
+  every remaining share by `1 / (1 - excluded share)`, a factor that differs between samples.
+- **Harmony corrects coordinates only.** `run_harmony` returns a `batch_correction` ref that
+  replaces PCA for ANN, neighbours, graph, UMAP and clustering; counts are untouched. A batch
+  holding one condition loses that condition's signal; Scarf ran silently with `condition`.
+- **Metrics are evidence, not a verdict.** `metric_*` take exact neighbour or graph refs and a live
+  column and return a float. iLISI (mixing) should rise while cLISI and connectivity hold.
+- **What Scarf tests.** `run_statistical_testing` runs donor-level rank tests (a
+  `statistical_tests` artifact; several keys share one BH pass). No count GLM (DESeq2, edgeR) or
+  compositional model (scCODA, propeller): export pseudobulk counts ("Export pseudobulk counts" in
+  `performance-and-export.md`) and model one population at a time with donor-level terms.
+
+## Recipes
+
+### Audit the study design
+
+Later recipes reuse these imports and `ds`. All column names are placeholders; `cell_type` stands
+for your own annotation column, never for author labels that are held out.
+
+```python
+import re
+
+import numpy as np
+import pandas as pd
+import scarf
+from scarf.plotting import CellField
+
+ds = scarf.DataStore("analysis.zarr", min_features_per_cell=-1)  # reopen; see data-access.md
+hits = re.compile(r"donor|individual|patient|subject|sample|library|batch|"
+                  r"condition|disease|treatment|time|day|assay|sex", re.I)
+print([c for c in ds.cells.columns if hits.search(c)])
+
+meta = ds.cells.to_pandas_dataframe(["donor_id", "condition", "timepoint", "batch"], key="I")
+print(meta.isna().sum())                     # resolve missing labels first
+print(meta.groupby("donor_id").agg(
+    cells=("condition", "size"), conditions=("condition", "nunique"),
+    timepoints=("timepoint", "nunique"), batches=("batch", "nunique"),
+))
+print(meta.drop_duplicates(["donor_id", "condition"])["condition"].value_counts())  # donors per arm
+units = meta.drop_duplicates(["donor_id", "batch", "condition"])
+print(pd.crosstab(units["batch"], units["condition"]))  # donors per batch x condition
+```
+
+`conditions == 1` for every donor means nested; `timepoints > 1` means repeated measures;
+`conditions > 1` puts a donor in two arms. A batch row with donors in one condition only is
+confounded with it. No sample column: join donor and timepoint strings (every row, none missing)
+and `ds.cells.insert("sample_id", values, overwrite=True)`; `insert` writes to the store at once.
+
+### Harmony through the pipeline, with a matched baseline
+
+```python
+design_cols = ("batch", "donor_id", "sample_id", "condition")   # Harmony columns first
+native = ds.pipeline.run(label="native", snapshot_columns=design_cols)
+corrected = ds.pipeline.run(
+    label="harmony_batch", harmony_batch_columns=["batch"], snapshot_columns=design_cols,
+)
+assert native["pca"] == corrected["pca"]   # same frozen cells and PCA; only the graph differs
+print(corrected["harmony"])                # ArtifactRef(kind='batch_correction')
+```
+
+The run's input snapshot is `names`, filter columns, Harmony columns, then `snapshot_columns`, in
+order; another list or order gives a new analysis selection and recomputes everything (24
+artifacts created, then 16 reused by the Harmony run; 10x 5K PBMC docs dataset). `run.cells`
+exposes only `snapshot_columns`: read a filter or Harmony column with
+`ds.cells.fetch_all(col)[run.cells.fetch_all("I")]`. Doublet scores still use plain PCA.
+
+As a branch of a run, `harmony = ds.run_harmony(run["pca"], ["batch"])` returns a ref to pass to
+`build_ann_index` in place of PCA (chain in `features-and-graphs.md`; keep the run's `k`). Score it
+below with the run's `neighbors` and `connectivity_map` versus the new pair.
+
+### Score native versus corrected
+
+A shuffled batch column gives the iLISI of perfect mixing for the same graph and `k`. Label
+concordance (ARI) needs two clustering refs on the same frozen cell selection.
+
+```python
+rng = np.random.default_rng(0)
+ds.cells.insert("batch_shuffled", rng.permutation(ds.cells.fetch_all("batch")), overwrite=True)
+
+def scores(neighbors, graph):
+    return {
+        "iLISI batch": ds.metric_ilisi("batch", neighbors),
+        "iLISI ceiling": ds.metric_ilisi("batch_shuffled", neighbors),
+        "batch mixing": ds.metric_proportional_batch_mixing("batch", neighbors),
+        "cLISI cell_type": ds.metric_clisi("cell_type", neighbors),
+        "connectivity cell_type": ds.metric_graph_connectivity("cell_type", graph),
+    }
+
+print(pd.DataFrame({
+    "native": scores(native["neighbors"], native["connectivity_map"]),
+    "harmony": scores(corrected["neighbors"], corrected["connectivity_map"]),
+}).T.round(3))
+print(ds.metric_label_concordance(native["clusters"], corrected["clusters"], metric="ari"))
+for r in (native, corrected):   # a condition-only cluster that vanishes signals overcorrection
+    print(pd.crosstab(r.cells.fetch("clusters"), r.cells.fetch("condition")))
+```
+
+### A donor sampled in two arms
+
+Example: a donor's first sample is in arm A and a later one in arm B (ventilation status changed
+between timepoints). Decide per comparison and write the rule down before testing:
+
+- **Pool to one value per donor** when all of its samples fall in one group of this comparison
+  (patients versus healthy donors). Pool donors with several samples in one arm the same way.
+- **Drop it from the between-group test** of the arms it spans (A versus B). Its samples share
+  genotype and history: counting it twice makes the groups dependent and invents a donor.
+- **Describe it within the donor only.** Its B minus A change is one pair (n = 1): report the two
+  values, never a p-value. An exact Wilcoxon needs 6 such pairs to reach 0.03125.
+- If dropping leaves 3 versus 3 donors (minimum exact two-sided p 0.1), report n, medians and
+  effect sizes without a test. Add a sensitivity run that excludes the donor everywhere.
+- `sample_by="donor_id"` makes Scarf refuse such a donor ("Each sample must belong to exactly one
+  group"); `sample_by="sample_id"` silently counts it in both arms.
+
+### Composition per donor with exact tests
+
+Per-unit proportions (a donor's samples pooled), two-sided Mann-Whitney between independent donor
+groups, BH across populations, and the n = 1 rows of two-arm donors.
+
+```python
+from math import comb
+
+from scipy.stats import PermutationMethod, false_discovery_control, mannwhitneyu
+
+run = ds.pipeline.open(label="baseline")
+in_run = np.asarray(run.cells.fetch_all("I"), dtype=bool)
+cells = ds.cells.to_pandas_dataframe(["donor_id", "sample_id", "condition", "cell_type",
+                                      "cell_state"])[in_run]
+excluded = ["Erythrocyte", "Platelet", "Unresolved"]      # your contaminant and unresolved labels
+drop = cells["cell_type"].isin(excluded)
+print(drop.groupby(cells["sample_id"]).mean().round(3))   # report the removed share per sample
+kept = cells[~drop]
+arms = kept.groupby("donor_id")["condition"].nunique()
+two_arm = arms.index[arms > 1].tolist()                   # donors sampled in both arms
+
+per_sample = pd.crosstab(kept["sample_id"], kept["cell_type"], normalize="index")
+independent = kept[~kept["donor_id"].isin(two_arm)]       # two-arm donors leave the tests
+per_donor = pd.crosstab(independent["donor_id"], independent["cell_type"], normalize="index")
+arm = independent.groupby("donor_id")["condition"].first()
+
+def exact_tests(props, arm, g1, g2):
+    rows = []
+    for pop in props.columns:
+        x = props[pop].reindex(arm.index[arm == g1]).dropna()   # units without a value drop out
+        y = props[pop].reindex(arm.index[arm == g2]).dropna()
+        row = {"population": pop, "n_1": len(x), "n_2": len(y), "median_1": x.median(),
+               "median_2": y.median(), "auc": np.nan, "p_exact": np.nan}
+        if min(len(x), len(y)) >= 2:   # as in Scarf: fewer than 2 units in a group is not tested
+            exact = comb(len(x) + len(y), len(x)) <= 100_000    # Scarf's switch to asymptotic
+            method = PermutationMethod(n_resamples=np.inf) if exact else "asymptotic"
+            u = mannwhitneyu(x, y, method=method)               # the exact null counts ties
+            row.update(auc=u.statistic / (len(x) * len(y)), p_exact=u.pvalue)
+        rows.append(row)
+    table = pd.DataFrame(rows)
+    tested = table["p_exact"].notna()   # untested rows keep NaN and stay out of the correction
+    table["p_bh"] = np.nan
+    table.loc[tested, "p_bh"] = false_discovery_control(table.loc[tested, "p_exact"], method="bh")
+    return table
+
+print(exact_tests(per_donor, arm, "ctrl", "case").round(4))
+parent = kept[kept["cell_type"] == "CD14 monocyte"]       # lineage parent as denominator
+state = (parent["cell_state"] == "CD14 monocyte (state 1)").groupby(parent["donor_id"]).mean()
+print(exact_tests(state.to_frame("state 1 of CD14 monocytes"), arm, "ctrl", "case").round(4))
+sample_info = kept.groupby("sample_id")[["donor_id", "condition"]].first()
+print(per_sample.join(sample_info)[lambda t: t["donor_id"].isin(two_arm)].round(3))  # n = 1
+```
+
+The same test in Scarf (a saved artifact): a 0/1 column averaged per donor is the donor's share,
+and the cell selection is the denominator (snapshot the parent's cells for a parent share).
+
+```python
+labels, donors = ds.cells.fetch_all("cell_type"), ds.cells.fetch_all("donor_id")
+keys = []
+for pop in per_donor.columns:
+    keys.append("is_" + re.sub(r"\W+", "_", pop))
+    ds.cells.insert(keys[-1], (labels == pop).astype(np.float64), overwrite=True)
+counted = in_run & ~np.isin(labels, excluded) & ~np.isin(donors, two_arm)
+ds.cells.insert("in_composition", counted, overwrite=True)
+result = ds.run_statistical_testing(
+    keys, CellField("condition"), cell_selection=ds.snapshot_cell_selection("in_composition"),
+    groups=["ctrl", "case"], test="mann_whitney", sample_by="donor_id",
+)
+print(result.p_value_method, pd.concat([result.tables[k].assign(key=k) for k in keys]))
+```
+
+Both routes gave identical exact and BH p-values on a synthetic design. Report n per group,
+medians, AUC (`U / (n_1 n_2)`; 0.5 is no shift) and BH values, then repeat with all cells as the
+denominator. `ds.plots.composition(..., kind="per_sample", study_design=...)` draws a strip of
+sample proportions per category (its intervals count samples, so a two-sample donor counts twice);
+`kind="stacked", sample_by=...` draws one stacked bar per sample.
+
+### Donor-level expression tests
+
+```python
+is_b = in_run & (labels == "B cell")
+ds.cells.insert("is_b_cell", is_b, overwrite=True)
+ds.cells.insert("one_arm_donor", ~np.isin(donors, two_arm), overwrite=True)
+b_cells = ds.snapshot_cell_selection("is_b_cell")                  # frozen cell_selection ref
+between = ds.run_statistical_testing(                              # donors nested in condition
+    ["CD79A", "MS4A1"], CellField("condition"), cell_selection=b_cells, subset_by="one_arm_donor",
+    groups=["ctrl", "case"], test="mann_whitney", sample_by="donor_id",
+)
+within = ds.run_statistical_testing(                               # same donor, two timepoints
+    ["CD79A", "MS4A1"], CellField("timepoint"), cell_selection=b_cells,
+    groups=["t0", "t1"], test="wilcoxon", sample_by="sample_id", pair_by="donor_id",
+)
+print(between.tables["CD79A"], within.tables["CD79A"], between.artifact, sep="\n")
+```
+
+Expression is averaged per sample, then ranked; `groups` fixes the order (`mean_difference =
+mean_1 - mean_2`). Mann-Whitney is exact up to 100,000 group splits (`result.p_value_method`);
+Wilcoxon keeps donors seen at both levels (`n_pairs`). Reload: `ds.get_statistical_tests(ref)`.
+
+### Merge, map, downsample
+
+`scarf.DataStoreMerge(datasets=[ds_a, ds_b], zarr_path=..., names=[...], assays=["RNA"],
+prepend_text="orig", source_column="dataset", overwrite=True).dump()` gives an uncorrected joint
+store; drop `is_primary_data == False` cells first and score iLISI before Harmony (not executed:
+`MemoryError` under a 3 GB budget). Mapping keeps a reference fixed; the query is another store.
+
+```python
+reference = ds.get_mapping_reference(ds.build_mapping_reference(native["neighbors"]))
+query = scarf.DataStore("query.zarr", default_assay="RNA")
+mapping_ref = query.run_mapping(reference, query.snapshot_cell_selection("I"), save_k=5)
+print(query.get_mapping_result(mapping_ref, reference=reference).diagnostics)
+labels = query.get_target_classes(mapping_ref, "cell_type", reference=reference,
+                                  threshold_fraction=0.6)
+query.cells.insert("transferred_cell_type", labels.to_numpy(), overwrite=True)  # "NA" = abstained
+```
+
+`ds.run_topacedo_sampler(graph, paris, max_sampling_rate=0.1)` keeps graph coverage, not
+proportions (`sampled` is in graph-row order); to balance donors, snapshot a seeded mask instead.
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+| --- | --- | --- |
+| `harmony_batch_columns` / `batch_columns` | `None` (off) / required | Technical columns only |
+| `neighbors_k` / `query_neighbors(k=)` | 11 | 21 or more when LISI metrics decide |
+| metric `perplexity` | `None` = `floor(k/3)` | Keep fixed across graphs; capped at `k/3` |
+| `sample_by` / `pair_by` | `None` | Always `sample_by`; `pair_by` only if repeated |
+| `sample_stat` | `"mean"` | `"fraction"` + `expression_cutoff` for detection |
+| `subset_by` | `None` | A boolean column, e.g. to drop two-arm donors |
+
+## Check before moving on
+
+- Every sample maps to one donor and one condition; donors per arm is the n; read confounding from
+  donors (`units`), not cells.
+- Each donor counts once per test: `n_1` and `n_2` are donors, two-arm donors are pooled or
+  dropped as decided, and the denominator and the excluded share per sample are reported.
+- After Harmony: iLISI moves toward the shuffled ceiling, cLISI and connectivity do not drop, no
+  condition-only cluster vanishes, and the ARI against native is explainable.
+
+## Pitfalls
+
+- `insert` stores `None` as `""` (Harmony silently treats it as a level; metrics raise) and `pd.NA`
+  as the literal `"<NA>"` (a real group everywhere). `to_pandas_dataframe` shows missingness.
+- Correcting on `donor_id` or `sample_id` with a donor-level condition aligns donors and can absorb
+  the condition. Prefer library or batch; otherwise recheck condition-specific clusters.
+- iLISI depends on `k`: a random synthetic batch scored 0.34, 0.67 and 0.76 at `k` 11, 21 and 31
+  (10x 5K PBMC docs dataset). CELLxGENE `cell_type` may come from the authors' own integration,
+  biasing cLISI toward it.
+- `pair_by` on donors nested in conditions raises "Duplicate (pair, group) rows"; use `sample_by`.
+- `scipy.stats.mannwhitneyu(method="exact")` ignores ties (donors at 0 tie), so it disagrees with
+  Scarf; `PermutationMethod(n_resamples=np.inf)` matches it but is exhaustive (12 versus 12: 20 s).
+- Without `cell_selection`, a `CellField` grouping covers every stored cell, `I` False included.
+- Marker search, Welch and ANOVA treat cells as observations; never use them for condition claims.
+- A Symphony-style mapping correction (`query_batches`) needs a technical batch, never a condition.
+
+## See also
+
+- `quality-control.md`, `features-and-graphs.md`, `markers-and-annotation.md`, `plotting.md`
+  (`composition`), `pipeline-runs-and-artifacts.md`, `performance-and-export.md` (pseudobulk).

package/bundled-skills/scarf-single-cell/references/markers-and-annotation.md

@@ -0,0 +1,280 @@
+# Markers and annotation
+
+Find cluster markers, turn marker evidence into cautious cell-type labels, separate progenitors
+from multiplets, score gene sets and the cell cycle, and run a final comparison with existing
+labels. Docs: <https://scarf.readthedocs.io/en/latest/tutorials/annotation.html>, <https://scarf.readthedocs.io/en/latest/tutorials/gene_set_scoring.html>,
+<https://scarf.readthedocs.io/en/latest/tutorials/cell_cycle.html>, <https://scarf.readthedocs.io/en/latest/reference/api/datastore.html>.
+
+## When to use
+
+- A partition is chosen (`clustering-and-embedding.md`) and its clusters need names.
+- You need marker tables, panel checks, multiplet checks, signature scores or cell-cycle phases.
+- Labels are final and a held-out reference exists for a post-hoc evaluation.
+
+## Key concepts
+
+- `run_marker_search(clusters, features=...)` takes an exact `cluster_labels` or `cluster_cut` ref
+  and a feature-selection ref, streams counts once (remote for mounted stores) and returns
+  `ArtifactRef(kind="marker_table")`. `run["markers"]` is the table for `run["clusters"]`.
+- Columns: `score` (Scarf specificity, 0 to 1, summing to 1 across groups for each gene; near
+  1/n_groups means ubiquitous), `mean`/`mean_rest`, `frac_exp`/`frac_exp_rest`, `fold_change`,
+  `auc`, two-sided Mann-Whitney `p_value` and `p_value_adjusted` (BH within each group). They are
+  one-versus-rest statistics over cells, not replicate-aware differential expression.
+- `get_markers` returns a DataFrame with a string `group_id`, rows sorted by `score` within each
+  group and groups in natural order (`"2"` before `"10"`). Defaults `min_score=0.25`,
+  `min_frac_exp=0.2` hide most genes; pass `-1` for both to get every gene for every group.
+- A label is a hypothesis. It needs named positive markers, low lineage-negative markers and a
+  plausible QC and doublet profile. Keep an explicit `unresolved` label instead of forcing a name.
+- Doublet scores rise with RNA content, so plasma cells and other high-RNA types score high as
+  whole clusters. Judge doublets within a lineage and per cell, not by a global cut.
+- No public API saves user-assigned labels as an artifact. Store them as a live column with
+  `ds.cells.insert` (mutable, unversioned, written into the store).
+- AUCell and WAGGR return `kind="enrichment_scores"` (activity scores, not p-values); cell-cycle
+  scoring returns `kind="cell_cycle"` (`phase`, `s_score`, `g2m_score`). Rows follow the selection.
+
+## Recipes
+
+### Get and read marker tables
+
+```python
+import numpy as np
+import pandas as pd
+import scarf
+
+scarf.configure_output(level="WARNING", progress=False)
+ds = scarf.DataStore("analysis.zarr", nthreads=4, min_features_per_cell=-1)
+run = ds.pipeline.open(label="baseline")
+clusters, markers = run["clusters"], run["markers"]
+# Another partition needs its own table (one pass over counts):
+markers_res1 = ds.run_marker_search(run["leiden_1.0"], features=run["feature_universe"])
+
+top = ds.get_markers(marker=markers)  # min_score=0.25, min_frac_exp=0.2
+print(top.groupby("group_id", sort=False).head(8)
+      .groupby("group_id", sort=False)["feature_name"].agg(", ".join))
+one = ds.get_markers(marker=markers, group_id=3)  # an int matches the label "3"
+print(one[["feature_name", "score", "frac_exp", "frac_exp_rest", "auc", "p_value_adjusted"]].head())
+everything = ds.get_markers(marker=markers, min_score=-1, min_frac_exp=-1)  # all genes x groups
+# ds.export_markers_to_csv(markers, "markers.csv") writes marker names per group, wide format
+```
+
+### Tabulate a canonical marker panel
+
+```python
+panel = ["CD3E", "IL7R", "CD8A", "NKG7", "GNLY", "MS4A1", "CD79A", "MZB1", "JCHAIN", "CD14",
+         "LYZ", "FCGR3A", "FCER1A", "LILRA4", "PPBP", "HBB", "CD34", "MKI67"]
+order = list(dict.fromkeys(everything["group_id"]))  # pivot would sort "10" before "2"
+hits = everything[everything["feature_name"].isin(panel)]
+frac = (hits.pivot(index="group_id", columns="feature_name", values="frac_exp")
+        .reindex(index=order, columns=[g for g in panel if g in set(hits["feature_name"])]))
+score = hits.pivot(index="group_id", columns="feature_name", values="score").reindex_like(frac)
+print(frac.round(2), score.round(2), sep="\n\n")  # detection versus exclusivity
+```
+
+### Flag clusters for review: QC, doublets, multiplets and progenitors
+
+Per cell, count lineages whose two specific genes are both detected, and the progenitor program
+genes detected, from one raw-count panel export. Ambient genes (LYZ, HBB, JCHAIN, Ig) are left out
+because they are detected almost everywhere.
+
+```python
+in_run = run.cells.fetch_all("I").astype(bool)
+review = run.cells.to_pandas_dataframe(["ids", "clusters", "doublet_score"]).set_index("ids")
+review = review.assign(**{c: ds.cells.fetch_all(c)[in_run]
+                          for c in ("RNA_nCounts", "RNA_percentMito")})
+lineages = {"T": ["CD3D", "CD3E"], "B": ["MS4A1", "CD79B"], "myeloid": ["CD14", "FCN1"],
+            "plasma": ["MZB1", "TNFRSF17"]}
+program = ["CD34", "PRSS57", "SPINK2", "CYTL1", "SOX4"]  # SOX4 alone is broadly detected
+wanted = list(dict.fromkeys(sum(lineages.values(), []) + program))
+present = [g for g in wanted if g in set(ds.RNA.feats.fetch_all("names"))]
+counts = ds.to_anndata(from_assay="RNA", feature_names=present)  # obs_names are cell ids
+hit = (pd.DataFrame(counts.X.toarray() > 0, index=counts.obs_names, columns=present)
+       .reindex(index=review.index, columns=wanted, fill_value=False))
+review["n_lineages"] = sum(hit[genes].all(axis=1).astype(int) for genes in lineages.values())
+review["progenitor"] = hit["CD34"] & (hit[program[1:]].sum(axis=1) >= 2)  # coherent program
+review["doublet_rank"] = review.groupby("clusters")["doublet_score"].rank(pct=True)
+summary = review.groupby("clusters").agg(
+    n=("doublet_score", "size"), doublet=("doublet_score", "median"),
+    counts=("RNA_nCounts", "median"), mito=("RNA_percentMito", "median"),
+    two_lineages=("n_lineages", lambda s: (s >= 2).mean()), progenitor=("progenitor", "mean"))
+summary["n_markers"] = (top["group_id"].value_counts()
+                        .reindex(summary.index.astype(str), fill_value=0).to_numpy())
+print(summary.round(2))
+suspects = review[(review["n_lineages"] >= 2) & (review["doublet_rank"] >= 0.9)]
+print(len(suspects), "cells co-detect two lineages and sit in their cluster's top doublet decile")
+```
+
+Read the flags together, per cluster and per cell:
+
+- Progenitor (HSPC): a coherent program in the same cells (CD34 with PRSS57, SPINK2, CYTL1; SOX4
+  supports) and few two-lineage cells. Primed progenitors add early genes of one lineage
+  (GATA2, KLF1, CPA3) that vary between cells, not mature markers of two lineages per cell.
+- Multiplet: mature markers of two lineages in the same cells (T with myeloid, B or plasma with
+  T or myeloid), counts or features above both parent lineages, few markers of its own and a
+  high doublet rank. Single multiplets inside large clusters show up only in `suspects`.
+- Doublet scores: compare a cluster with clusters of the same lineage, not the global median. A
+  whole plasma cluster scoring high is expected; a plasma subset co-detecting T genes is not.
+- Ambiguous (a program plus two-lineage cells, or high counts without co-detection): keep
+  `unresolved` and record both readings.
+
+### Assign labels and store them
+
+Write the evidence next to each label. The map below is a template with placeholder IDs: never
+copy a map from another dataset. Clusters not named stay `unresolved`.
+
+```python
+cluster_values = run.cells.fetch("clusters").astype(str)
+labels = {  # TEMPLATE: replace "<id>" and every gene with this dataset's own evidence
+    # "<id>": "CD14 monocyte",  # + CD14, LYZ, FCN1 | - CD3E, MS4A1 | confidence high
+    # "<id>": "CD4 T",          # + CD3E, IL7R, CD40LG | - CD8A, NKG7 | medium: CD4 sparse
+    # "<id>": "HSPC",           # + CD34, PRSS57, SPINK2 | - no two-lineage cells | medium
+    # "<id>": "unresolved",     # + CD3E with CD14 in the same cells, high counts | low
+}
+per_cell = pd.Series(cluster_values).map(labels).fillna("unresolved").to_numpy()
+print(pd.Series(per_cell).value_counts())
+annotation = np.full(ds.cells.N, "not analyzed", dtype=object)
+annotation[in_run] = per_cell
+ds.cells.insert("annotation_v1", annotation, overwrite=True)  # live column, full cell axis
+ds.cells.insert("annotation_v1_cells", in_run, overwrite=True)
+named = sorted(int(c) for c, name in labels.items() if name != "unresolved")
+keep = ds.select_cells(clusters, include=named) if named else run["analysis_cell_selection"]
+```
+
+`select_cells(include=[])` raises, hence the guard; `keep` can seed a graph without unresolved
+clusters. Plot evidence with `ds.plots.dotplot(features=..., group_by="annotation_v1",
+cell_key="annotation_v1_cells")` and `ds.plots.marker_heatmap(marker=markers, topn=3)`.
+
+### Score gene sets with AUCell and WAGGR
+
+```python
+net = pd.DataFrame({  # or scarf.read_gmt("sets.gmt"): one source-target row per gene
+    "source": ["T_cell"] * 4 + ["B_cell"] * 3 + ["Monocyte"] * 4 + ["NK"] * 4,
+    "target": ["CD3D", "CD3E", "TRAC", "IL7R", "MS4A1", "CD79A", "CD79B",
+               "CD14", "LYZ", "S100A8", "FCN1", "NKG7", "GNLY", "KLRD1", "PRF1"]})
+cells = run["analysis_cell_selection"]
+universe = ds.select_all_features(from_assay="RNA")
+aucell = ds.run_aucell(net, cells, features=universe, tmin=3)  # one pass over counts
+waggr = ds.run_waggr(net.assign(weight=1.0), cells, features=universe, tmin=3)
+result = ds.get_enrichment(aucell)  # lazy; sources=[...] loads a subset
+assert result.cell_selection == cells
+scores = pd.DataFrame(result.data.compute(), columns=list(result.source_names))
+print(scores.groupby(run.cells.fetch("clusters")).mean().round(2))
+```
+
+AUCell ranks raw counts within each cell (0 to 1, `n_up` defaults to 5% of the universe). WAGGR
+averages library-size-normalized expression with edge weights (unbounded, signed weights allowed).
+
+### Read or compute cell-cycle phase
+
+```python
+phase = run.cells.to_pandas_dataframe(["clusters", "cell_cycle_phase", "s_score", "g2m_score"])
+print(pd.crosstab(phase["clusters"], phase["cell_cycle_phase"], normalize="index").round(2))
+cycle = ds.run_cell_cycle_scoring(keep)  # new artifact; one pass over counts
+print(pd.Series(np.asarray(ds.load_artifact(cycle)["phase"][:]).astype(str)).value_counts())
+```
+
+### Compare with existing labels (final post-hoc evaluation only)
+
+Run once, after labels are frozen. Read the reference column here first, write the map from its
+vocabulary before any crosstab, and never tune labels or parameters on the result.
+
+```python
+import hashlib
+from sklearn.metrics import adjusted_rand_score, classification_report
+
+ds.cells.to_pandas_dataframe(["ids", "annotation_v1"]).to_csv("labels_v1_frozen.csv", index=False)
+print("sha256", hashlib.sha256(open("labels_v1_frozen.csv", "rb").read()).hexdigest())
+reference = pd.Series(ds.cells.fetch_all("author_cell_type")[in_run]).astype(str)  # held out
+print(sorted(reference.unique()))  # the vocabulary only; now write the map below
+to_reference = {  # TEMPLATE: our label -> reference label, fixed before seeing any agreement
+    # "CD4 T": "<reference label>",
+}
+ours = pd.Series(ds.cells.fetch_all("annotation_v1")[in_run]).astype(str)
+pred = ours.map(to_reference).fillna("unmatched")
+print(pd.crosstab(pred, reference))
+shared = sorted(set(to_reference.values()))
+both = (pred != "unmatched") & reference.isin(shared)  # drops unresolved and unmapped classes
+print("ARI all cells", round(adjusted_rand_score(reference, pred), 3))
+if both.any():
+    print("ARI mapped only", round(adjusted_rand_score(reference[both], pred[both]), 3))
+    print(classification_report(reference, pred, labels=shared, zero_division=0))
+```
+
+Report precision and recall per class, ARI with and without ambiguous classes, and reference
+classes with no counterpart. Disagreement is evidence: inspect large off-diagonal cells.
+
+## Canonical markers
+
+Confirm with dataset markers and use two or more genes. Human symbols; mouse symbols are title
+case (`Cd3e`, `Ptprc`, `Mki67`), and panel lookups are exact, so convert the panel.
+
+| Population | Positive markers | Notes and negatives |
+|---|---|---|
+| T cell | CD3D, CD3E, TRAC | MS4A1, CD14 low |
+| CD4 T | IL7R, CD40LG, CD4 (often poorly detected) | CD8A, CD8B low; CCR7, SELL, LEF1 mark naive T |
+| CD8 T | CD8A, CD8B | effector GZMK, GZMH, NKG7 |
+| Treg | FOXP3, IL2RA, CTLA4 | within CD4 T |
+| MAIT, gamma-delta T | SLC4A10, KLRB1 (MAIT); TRDC, TRGC1 (gamma-delta) | often within CD8 or effector T |
+| NK | NKG7, GNLY, KLRD1, KLRF1, NCAM1 | CD3E low; NKG7, GNLY also in CD8 effector T |
+| B cell | MS4A1, CD79A, CD19 | naive IGHD, TCL1A |
+| Plasma cell, plasmablast | MZB1, JCHAIN, XBP1, TNFRSF17 | MS4A1 low; plasmablasts express MKI67 |
+| CD14 monocyte | CD14, LYZ, S100A8, S100A9, FCN1 | |
+| CD16 monocyte | FCGR3A, MS4A7, CDKN1C | FCGR3A also on NK |
+| Conventional DC | FCER1A, CD1C, CLEC10A; CLEC9A (cDC1) | FCER1A also on basophils |
+| Plasmacytoid DC | LILRA4, IL3RA, CLEC4C | GZMB shared with NK; MZB1 also high |
+| Basophil, mast cell | CPA3, HDC, MS4A2; TPSAB1, KIT (mast) | |
+| Neutrophil | CSF3R, FCGR3B, CXCR2 | low counts; rare after density gradients |
+| Platelet, megakaryocyte | PPBP, PF4 | often attached to monocytes |
+| Erythroid | HBB, HBA1 | frequent ambient contamination |
+| HSPC | CD34, PRSS57, SPINK2, CYTL1, SOX4 | rare in blood; primed subsets add GATA2, KLF1 or CPA3 |
+| Cycling (a state) | MKI67, TOP2A | label as a state, not a type |
+| Immune, macrophage | PTPRC; C1QA, CD68 (macrophage) | |
+| Epithelium, gut | EPCAM, KRT8, KRT18; LGR5, OLFM4 (stem) | goblet MUC2; colonocyte CA1 |
+| Fibroblast | PDGFRA, COL1A1, DCN | subtypes vary by tissue, e.g. APOD or CCN2 (CTGF) high; `^CCN` is in the default HVG blacklist, markers still test it |
+| Endothelium | PECAM1, VWF, CDH5 | |
+| Smooth muscle, pericyte | ACTA2, MYH11, DES; RGS5 (pericyte) | ACTA2 also on myofibroblasts |
+| Keratinocyte, melanocyte | KRT5, KRT14 (basal), KRT10 (suprabasal); PMEL, MLANA (melanocyte) | |
+| Neuron | RBFOX3, SNAP25; SLC17A7 (excitatory); GAD1, GAD2 (inhibitory) | |
+| Astrocyte, oligodendrocyte | GFAP, AQP4; MBP, PLP1, MOG (oligodendrocyte) | |
+| OPC, microglia | PDGFRA, CSPG4 (OPC); CSF1R, P2RY12 (microglia) | CSF1R also on macrophages |
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+|---|---|---|
+| `min_score` (`get_markers`) | `0.25` | `-1` to see negatives and weak genes; raise for a short list |
+| `min_frac_exp` (`get_markers`) | `0.2` | `-1` with `min_score=-1` for complete panel tables |
+| `features` (`run_marker_search`) | required | `run["feature_universe"]` tests every gene; an HVG ref limits the search |
+| `tmin` (AUCell, WAGGR) | `5` | Lower for short sets; sources with fewer matched targets are dropped |
+| `n_up` (AUCell) | `None` (5% of universe) | Fix it when comparing runs; changes the score scale |
+| `s_genes` / `g2m_genes` | built-in human and mouse lists | Other species or naming schemes |
+
+## Check before moving on
+
+- Every named cluster has at least two positive markers with high `frac_exp` and `score` well above
+  1/n_groups, and lineage negatives with low `frac_exp`. The evidence is written next to the label.
+  Few markers usually means a split within one lineage: merge, or name a state only with evidence.
+- Clusters with two lineages' markers in the same cells, counts above both parents or a high
+  within-lineage doublet rank are `unresolved`. A CD34 program without co-detection is HSPC.
+- Rare types hide in big clusters; check finer resolutions or subcluster the lineage.
+- The label map, marker ref and run ID are recorded. Held-out labels were not read before this.
+
+## Pitfalls
+
+- Default `get_markers` filters hide negative markers. `group_id` is a string while Leiden labels
+  are integers; convert before joining.
+- Direct calls do not reuse pipeline outputs: `run_marker_search(run["clusters"], ...)` and
+  `run_cell_cycle_scoring(...)` write new artifacts and reread counts. Read the run's refs instead.
+- Ambient genes (HBB, LYZ, S100A8, JCHAIN, immunoglobulins) show nonzero `frac_exp` in most
+  clusters. Use `score` and `auc`, not detection alone, and leave them out of co-detection checks.
+- Small groups inflate `fold_change` and can pass `min_score` with odd genes; check group size.
+- `ds.cells.insert(..., overwrite=True)` silently replaces a column; use versioned names.
+- `select_cells(include=...)` needs integers for integer cluster artifacts (`TypeError` otherwise).
+- Scores from different feature universes, `n_up` or WAGGR `mode`/`log_transform` are not comparable.
+
+## See also
+
+- `clustering-and-embedding.md`: choosing the partition and aligning artifacts to cell IDs.
+- `features-and-graphs.md`: subclustering one lineage before naming its states.
+- `quality-control.md`: doublet scores and QC covariates behind unresolved clusters.
+- `plotting.md` (`dotplot`, `marker_heatmap`); `integration-and-comparisons.md` (label transfer,
+  condition comparisons); `pipeline-runs-and-artifacts.md` (run fields, lineage).