opencode-skills-collection 4.1.7 → 4.1.8

package/bundled-skills/scarf-single-cell/references/features-and-graphs.md

@@ -0,0 +1,279 @@
+# Features and graphs
+
+Select features, normalize, reduce with PCA and build the neighbourhood graph as an explicit
+chain of artifacts, then branch or subcluster it safely. Docs: <https://scarf.readthedocs.io/en/latest/tutorials/feature_selection.html>,
+<https://scarf.readthedocs.io/en/latest/tutorials/graph_construction.html>, <https://scarf.readthedocs.io/en/latest/tutorials/dimensionality_reduction.html> and
+<https://scarf.readthedocs.io/en/latest/reference/api/graph_construction.html>.
+
+## When to use
+
+- A pipeline default (`hvg_count=1000`, `pca_dims=21`, `neighbors_k=11`) needs to change, or two
+  settings need comparing side by side.
+- Building a graph on a custom cell selection (QC variant, subset) or feature list.
+- Subclustering one lineage of a coarse partition with its own HVGs, PCA and graph.
+- Diagnosing a graph before clustering or embedding.
+
+## Key concepts
+
+- RNA chain, each step returning an `ArtifactRef` that the next step consumes:
+  cell selection + `select_hvgs` (`feature_selection`) -> `run_normalization` (`normalized`) ->
+  `run_pca` (`reduction`) -> optional `run_harmony` -> `build_ann_index` (`ann_index`) ->
+  `query_neighbors` (`neighbors`) -> `build_connectivity_map` (`connectivity_map`).
+  `build_embedding_initialization(pca)` is a side branch that seeds UMAP and t-SNE only.
+- Identity is operation + scientific parameters + exact inputs. An identical call returns the
+  existing ref without recomputing; a changed `k` reuses normalization, PCA and the ANN index.
+  Changing cells or features invalidates every downstream stage.
+- `ds.pipeline.run(hvg_count=, pca_dims=, neighbors_k=)` forwards to `select_hvgs(top_n=)`,
+  `run_pca(dims=)` and `query_neighbors(k=)`. Explicit calls with the same arguments return the
+  very refs in `run[...]`. `pca_dims=0` skips PCA and builds the graph on normalized HVG values.
+- RNA normalization: counts scaled to 1000 over the selected features, then `log1p`. PCA
+  standardizes features. HVGs are an artifact, not a feature column; the chain writes no metadata.
+- Count passes: a new cell selection costs one pass for the feature summary (`select_hvgs`,
+  `select_detected_features`) and one per `run_normalization`; later stages read local artifacts.
+
+## Recipes
+
+### Rebuild the chain explicitly from a run
+
+Start from any frozen cell selection: `run["analysis_cell_selection"]`, a QC ref, or
+`ds.snapshot_cell_selection("I")`.
+
+```python
+import numpy as np
+import pandas as pd
+import scarf
+import scarf.plotting as splt
+
+ds = scarf.DataStore("analysis.zarr", min_features_per_cell=-1)  # reopen; see data-access.md
+run = ds.pipeline.open(label="baseline")
+cells = run["analysis_cell_selection"]
+
+hvg = ds.select_hvgs(cells, top_n=1000, show_plot=False)
+normalized = ds.run_normalization(cells, hvg)
+pca = ds.run_pca(normalized, dims=21)
+ann = ds.build_ann_index(pca)
+neighbors = ds.query_neighbors(ann, k=11)
+graph = ds.build_connectivity_map(neighbors)
+init = ds.build_embedding_initialization(pca)
+print({key: ref == run[key] for key, ref in {
+    "highly_variable_features": hvg, "normalized": normalized, "pca": pca, "ann_index": ann,
+    "neighbors": neighbors, "connectivity_map": graph, "embedding_initialization": init}.items()})
+```
+
+### Feature-selection diagnostics
+
+The HVG artifact stores `values` (selected mask) and `corrected_variance` for every feature.
+
+```python
+from scarf.features.gene_families import GENE_FAMILY_PATTERNS
+
+hvg_group = ds.load_artifact(hvg)
+selected = np.asarray(hvg_group["values"][:], dtype=bool)
+corrected = np.asarray(hvg_group["corrected_variance"][:])
+names = ds.RNA.feats.fetch_all("names").astype(str)
+print(int(selected.sum()), pd.Series(corrected[selected], index=names[selected]).nlargest(15).round(1).to_dict())
+panel = ["CD3D", "MS4A1", "LYZ", "NKG7", "PPBP", "JCHAIN"]
+print({gene: bool(selected[names == gene].any()) for gene in panel})
+print({family: len(ds.RNA.feats.grep(p)) for family, p in GENE_FAMILY_PATTERNS.items()})
+
+# Headless mean-variance plot. show_plot=True only displays, then closes, its figure.
+summary = ds.load_artifact(ds.inspect_artifact(hvg).input_ref("feature_summary"))
+n_cells = np.asarray(summary["normed_n"][:], dtype=float)
+total = np.asarray(summary["normed_tot"][:], dtype=float)
+splt.highly_variable_features(
+    mean_nonzero=np.divide(total, n_cells, out=np.zeros_like(total), where=n_cells > 0),
+    corrected_variance=corrected, n_cells=n_cells, selected=selected, show=False,
+).save("hvg_mean_variance.png")
+```
+
+(10x 5K PBMC docs dataset) The top HVGs were platelet (`PPBP`, `NRGN`), light-chain (`IGLC2`,
+`IGKC`) and `IGHM` genes, which the default blacklist keeps. What to exclude: `gene-blacklists.md`.
+
+### Alternative feature sets and a custom blacklist
+
+`blacklist` is one regex, matched (`re.match`, ignoring case) against every feature name. It
+replaces the default entirely, so pass a complete string: the default plus add-ons, or a full
+string from `gene-blacklists.md`.
+
+```python
+from scarf.features.variability import DEFAULT_HVG_BLACKLIST
+
+base = DEFAULT_HVG_BLACKLIST  # or a complete string chosen with gene-blacklists.md
+extra_patterns = []  # add-on regexes chosen with gene-blacklists.md; empty adds nothing
+blacklist = "|".join([base, *extra_patterns])
+new, old = set(ds.RNA.feats.grep(blacklist)), set(ds.RNA.feats.grep(DEFAULT_HVG_BLACKLIST))
+print("added:", sorted(new - old)[:20], "released:", sorted(old - new)[:20])  # check both
+hvg_2k = ds.select_hvgs(cells, top_n=2000, show_plot=False, blacklist=blacklist)
+detected = ds.select_detected_features(cells, min_cells=20)  # inclusive detection threshold
+everything = ds.select_all_features(from_assay="RNA")  # canonical all-true universe
+custom = ds.set_feature_selection(from_assay="RNA", mask=np.isin(names, panel))  # or feature_indexes=
+shared = np.asarray(ds.load_artifact(hvg_2k)["values"][:], dtype=bool) & selected
+print(int(shared.sum()), "of the 1000 baseline HVGs kept")
+```
+
+`blacklist=""` disables the name filter; `max_cells=np.inf` disables the ubiquitous-gene filter.
+The blacklist affects HVG selection only; marker search still tests every gene.
+
+### Choose PCA dimensions
+
+```python
+pca_30 = ds.run_pca(normalized, dims=30)  # more PCs than needed; local, no count pass
+scores = np.asarray(ds.load_artifact(pca_30)["data"])  # cells x dims; also "loadings", "center"
+share = scores.var(axis=0, ddof=1)
+share = share / share.sum()
+print(np.round(np.cumsum(share)[[9, 14, 20, 29]], 3))  # share of the 30-PC variance at 10/15/21/30
+splt.elbow(share, show=False).save("pca_elbow.png")
+```
+
+An elbow is a hint, not a rule. Too few PCs merge populations; too many restore noise.
+
+### Graph diagnostics and a k branch
+
+`load_graph` returns a SciPy CSR matrix over the graph's cells, in cell-selection order. The default
+is directed: each row holds exactly `k` weighted edges. Keep every returned ref; downstream calls
+take one graph explicitly, so branches never overwrite each other.
+
+```python
+in_graph = np.asarray(ds.load_artifact(cells)["values"][:], dtype=bool)
+G = ds.load_graph(graph)
+S = ds.load_graph(graph, symmetric=True)
+degree = np.asarray((S != 0).sum(axis=1)).ravel()
+print(G.shape, G.nnz, "degree min/median/max", degree.min(), np.median(degree), degree.max())
+qc = ds.cells.to_pandas_dataframe(["RNA_nCounts", "RNA_nFeatures"])[in_graph]
+print(qc.assign(degree=degree).corr(method="spearman").round(2)["degree"])
+splt.graph_qc(G, show=False).save("graph_qc.png")
+
+graph_21 = ds.build_connectivity_map(ds.query_neighbors(ann, k=21))  # reuses the ANN index
+leiden = {name: ds.run_leiden_clustering(g, resolution=1.0)
+          for name, g in {"k11": graph, "k21": graph_21}.items()}
+labels = {name: np.asarray(ds.load_artifact(ref)["values"][:]) for name, ref in leiden.items()}
+print(pd.crosstab(labels["k11"], labels["k21"]))
+```
+
+(10x 5K PBMC docs dataset) Degree vs `RNA_nCounts` Spearman 0.10; 16 clusters at k=11 versus 15
+at k=21 (ARI 0.81). A larger `k` usually absorbs small clusters; the crosstab shows which.
+
+### Same knobs through the pipeline
+
+`params` forwards other stage settings, including the blacklist. Giving a setting both ways (for
+example `hvg_count` and `params["hvg"]["top_n"]`) raises before anything is written. A completed
+label cannot be run again, even unchanged: reopen it with `ds.pipeline.open` or use a new label.
+
+```python
+run_alt = ds.pipeline.run(
+    label="hvg2k_pc30_k21_blacklist", hvg_count=2000, pca_dims=30, neighbors_k=21,
+    params={"hvg": {"blacklist": blacklist}, "connectivity": {"bandwidth": 1.5}},
+)
+print(run_alt.status, ds.inspect_artifact(run_alt["neighbors"]).parameters["k"])
+print(run_alt["highly_variable_features"] == hvg_2k)  # same cells and blacklist: same artifact
+```
+
+### Subcluster one lineage
+
+Fix a coarse lineage partition first (`clustering-and-embedding.md`, "Choose the partition"), then
+rebuild HVGs, normalization, PCA and the graph inside the lineage. `ds.cells.insert` writes to the
+store itself; prefix and version column names on shared stores.
+
+```python
+from itertools import combinations
+from sklearn.metrics import adjusted_rand_score
+
+def on_cell_axis(ref):
+    """Spread Leiden labels over every store cell (-1 outside the ref's selection)."""
+    mask = ds.load_artifact(ds.inspect_artifact(ref).input_ref("cell_selection"))["values"][:]
+    full = np.full(ds.cells.N, -1)
+    full[np.asarray(mask, dtype=bool)] = ds.load_artifact(ref)["values"][:]
+    return full
+
+def build_graph(cell_selection, top_n=1000, dims=21, k=11):
+    hvg = ds.select_hvgs(cell_selection, top_n=top_n, show_plot=False)  # summary: one count pass
+    pca = ds.run_pca(ds.run_normalization(cell_selection, hvg), dims=dims)  # one count pass
+    return ds.build_connectivity_map(ds.query_neighbors(ds.build_ann_index(pca), k=k))
+
+parent, parent_markers = run["clusters"], run["markers"]  # or your chosen Leiden ref and its table
+table = ds.get_markers(marker=parent_markers, min_score=-1, min_frac_exp=-1)
+cd3e = table[table["feature_name"] == "CD3E"].set_index("group_id")["frac_exp"]
+lineage_ids = sorted(int(g) for g in cd3e.index[cd3e > 0.5])  # a T gate; confirm with markers
+sub_cells = ds.select_cells(parent, include=lineage_ids)  # frozen, over every store cell
+ds.cells.insert("t_lineage_v1", np.isin(on_cell_axis(parent), lineage_ids), overwrite=True)
+alt_cells = ds.snapshot_cell_selection("t_lineage_v1")  # same cells from any boolean column
+assert np.array_equal(*(ds.load_artifact(r)["values"][:] for r in (sub_cells, alt_cells)))
+
+sub_graph, sub_leiden, rows = build_graph(sub_cells), {}, []
+for resolution in (0.3, 0.5, 0.8):
+    refs = [ds.run_leiden_clustering(sub_graph, resolution=resolution, random_seed=seed)
+            for seed in (4444, 1, 2)]
+    seeds = [np.asarray(ds.load_artifact(ref)["values"][:]) for ref in refs]
+    sub_leiden[resolution] = refs[0]
+    rows.append({"resolution": resolution, "n_clusters": len(np.unique(seeds[0])),
+                 "min_seed_ari": min(adjusted_rand_score(a, b) for a, b in combinations(seeds, 2))})
+print(pd.DataFrame(rows).round(3))
+
+# Choose as for a global partition; then check markers, covariates, HVG count, parent overlap.
+sub_resolution = 0.3  # the finest resolution that passed the seed check above
+sub = sub_leiden[sub_resolution]
+sub_full, parent_full = on_cell_axis(sub), on_cell_axis(parent)
+in_sub = sub_full >= 0
+sub_markers = ds.run_marker_search(sub, features=run["feature_universe"])  # one count pass
+print(ds.get_markers(marker=sub_markers).groupby("group_id", sort=False)["feature_name"]
+      .agg(n="size", top=lambda s: ", ".join(s.head(6))))  # groups missing here have 0 markers
+print(pd.crosstab(sub_full[in_sub], parent_full[in_sub]))
+print(pd.DataFrame({"sub": sub_full, "counts": ds.cells.fetch_all("RNA_nCounts"), "doublet":
+                    run.cells.fetch_all("doublet_score")})[in_sub].groupby("sub").median().round(2))
+sub_2k = ds.run_leiden_clustering(build_graph(sub_cells, top_n=2000), resolution=sub_resolution)
+print("ARI vs 2000 HVGs", round(ds.metric_label_concordance(sub, sub_2k), 3))
+
+combined = np.where(in_sub, np.char.add("T.", sub_full.astype(str)), parent_full.astype(str))
+combined = np.where(parent_full < 0, "not analyzed", combined).astype(object)
+ds.cells.insert("clusters_tsub_v1", combined, overwrite=True)  # new suffix for each new version
+```
+
+(10x 5K PBMC docs dataset) The T subgraph at 0.3 split naive and memory CD4, naive CD8,
+cytotoxic and FOXP3/CTLA4 groups, with ARI 0.94 at 2,000 HVGs. The parent had no FOXP3 group.
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+| --- | --- | --- |
+| `select_hvgs(top_n=)` / `hvg_count` | `1000` | Weak or rare populations (raise); noisy programs dominate (lower). |
+| `select_hvgs(blacklist=)` | `DEFAULT_HVG_BLACKLIST` | A family dominates the HVGs or is the question; what to add: `gene-blacklists.md`. |
+| `select_hvgs(min_cells=, max_cells=)` | `20`, `None` (= selected cells - 20) | Tiny subsets; `max_cells=np.inf` keeps ubiquitous genes. |
+| `run_pca(dims=)` / `pca_dims` | `21` | After a variance and stability check; `0` in the pipeline means no PCA. |
+| `run_pca(pca_cell_selection=)` | `None` | Fit on a subset (for example one batch) and project all cells. |
+| `query_neighbors(k=)` / `neighbors_k` | `11` (clamped to cells - 1) | Larger data or smoother graphs (15 to 30); smaller to resolve rare groups. |
+| `load_graph(symmetric=, use_k=)` | `False`, all `k` | `True` for degree and connectivity checks; `use_k` to thin edges. |
+
+## Check before moving on
+
+- HVGs: count equals `top_n`; expected lineage markers are selected; the top of the list is not
+  one contaminant or technical program. If canonical lineage genes are missing, or one family
+  (Ig, haemoglobin, platelet) leads the list, branch with more HVGs and an explicit blacklist
+  (`gene-blacklists.md`) and compare partitions.
+- PCA and `k`: a second setting gives a similar partition, or the differences are explained by
+  markers. Boundaries that move with `k` or `dims` are provisional.
+- Graph: symmetric degree min is at least `k`; degree barely tracks counts. A strong correlation
+  means depth is shaping the graph.
+- Subclusters: keep one only when it is seed-stable, has its own named markers, is not explained
+  by counts or doublet score, and survives the HVG change. Merge the rest into the parent label.
+  Write labels to a versioned column and keep the parent labels.
+
+## Pitfalls
+
+- Reading in-degree from the directed graph as isolation. `(G != 0).sum(axis=0)` is zero for 85
+  cells (10x 5K PBMC docs dataset), yet every cell has `k` out-edges; use `symmetric=True`.
+- Passing a feature selection to UMAP, clustering or markers on a graph: graph consumers take the
+  graph ref. `query_neighbors(coordinates=)` only checks equality; it cannot retarget an index.
+- `show_elbow_plot=True` draws only on a new PCA fit. More than 4096 features or several narrow
+  blocks switch PCA to IncrementalPCA, and `batch_size` then joins the artifact identity.
+- Subclustering on the parent graph, or carrying subcluster IDs across artifacts. Rebuild inside
+  the subset, join on the cell axis, and keep the parent labels. `select_cells(include=[])`
+  raises; integer cluster artifacts need integer `include` values.
+- `select_hvgs` is RNA only. ATAC uses `select_prevalent_peaks` then `run_lsi`.
+
+## See also
+
+- `gene-blacklists.md`: which gene families to pass as `blacklist=`, and why.
+- `quality-control.md`: building the cell selection this chain starts from.
+- `clustering-and-embedding.md`: Leiden, Paris, UMAP and t-SNE on the graph; choosing a partition.
+- `integration-and-comparisons.md`: `run_harmony` between PCA and the ANN index.
+- `pipeline-runs-and-artifacts.md`: `inspect_artifact`, lineage and reuse.

package/bundled-skills/scarf-single-cell/references/gene-blacklists.md

@@ -0,0 +1,182 @@
+# Gene blacklists for HVG selection
+
+What `select_hvgs(blacklist=...)` removes, how to check it on your store, the corrected strings to
+pass today, and when to add the clonotype or haemoglobin add-ons. Docs:
+<https://scarf.readthedocs.io/en/latest/tutorials/feature_selection.html>, <https://scarf.readthedocs.io/en/latest/reference/api/pipeline.html>.
+
+## When to use
+
+- Before the baseline run on any new store, to see what the default blacklist matches there.
+- The store uses current HGNC or MGI histone symbols (`H2AC*`, `H3C*`, `H4C*`, `H1-*`), as Cytebase
+  datasets do. The default `^HIST` then matches nothing; only older `HIST1H*` references match.
+- Fibroblast, smooth-muscle, endothelial or tumour data, where `CCN1`/`CCN2` (CYR61/CTGF) or
+  `CCND1`/`CCND2` carry biology that the default `^CCN` hides.
+- Plasma-cell-rich or B-cell-rich samples, mouse data, or haemoglobin genes among the top HVGs.
+
+## Key concepts
+
+- The blacklist is one regular expression, matched case-insensitively with `re.match` against
+  feature `names` (never `ids`). Each `|` alternative is anchored at the start of the name.
+- It only removes HVG candidates. Markers, plots, gene scores and QC still see every gene.
+- The string and the matched-gene fingerprint are part of the HVG artifact identity. Changing the
+  string creates new HVG, normalization, PCA and graph artifacts; the same string reuses them.
+- HVG selection also drops ubiquitous genes (`max_cells` = selected cells - 20), so most `MT-` and
+  cytosolic ribosomal genes never compete. On the 10x 5K PBMC docs dataset, unblocking either
+  family added no gene to the top 1000.
+- `RNA_percentMito` and `RNA_percentRibo` were fixed at first open from the `^MT-` and
+  `^RPS|^RPL|^MRPS|^MRPL` patterns; `blacklist=` never changes them. The RPS6K kinases and other
+  off-targets added under 1% of `percentRibo` counts in testing, so the column needs no correction.
+- Names that are Ensembl IDs match nothing: no blacklist effect, no error, and no
+  `percentMito`/`percentRibo` columns (only a warning at first open). Build the string from the
+  family genes' IDs instead (`^ENSG00000198888$|...`), and percentages with `run_feature_percentage`.
+- Default families and verdicts:
+
+| Family | Default | Problem | Recommended |
+| --- | --- | --- | --- |
+| mitochondrial | `^MT-` | none; mouse `mt-` matches | keep |
+| ribosomal | `^RPS\|^RPL\|^MRPS\|^MRPL` | also RPS6KA/B kinases, `RPS19BP1`, `L` paralogs, -AS1/-DT RNAs | exact protein symbols |
+| cellCycleCcn | `^CCN` | hits CCN1-6 (CTGF, CYR61, NOV, WISP), CCND1-3, CCNT/K/H/L; misses MKI67, TOP2A | drop; score the cell cycle instead |
+| hla / h2 | `^HLA-` / `^H2-` | whole MHC I and II; removal did not create donor clusters in testing | keep |
+| histone | `^HIST` | matches only pre-2020 symbols | old and current replication-dependent names |
+| sexLinked | 10 human names | mouse `Eif2s3y`, `Zfy1/2` missed | add mouse genes |
+
+## Recipes
+
+### Recommended strings
+
+```python
+import re
+
+import numpy as np
+import pandas as pd
+import scarf
+from scarf.features.gene_families import GENE_FAMILY_PATTERNS
+from scarf.features.variability import DEFAULT_HVG_BLACKLIST
+
+MITO = r"^MT-"
+RIBO = r"^RP[SL]\d+[AXY]?\d*$|^RPLP[0-2]$|^RPSA$|^MRP[SL]\d+[A-C]?$"
+HISTONE = (r"^HIST[1-4]H|^H1-[1-6]$|^H1F[1-6]$|^H2AC\d+$|^H2BC\d+$|^H3C\d+$|^H4C\d+$"
+           r"|^H2AW$|^H3-4$|^H4-16$")  # replication-dependent; replacement histones kept
+HUMAN_BLACKLIST = "|".join([MITO, RIBO, r"^HLA-", HISTONE,
+    r"^XIST$|^DDX3Y$|^USP9Y$|^EIF1AY$|^KDM5D$|^SRY$|^ZFY$|^UTY$|^TMSB4Y$|^NLGN4Y$"])
+MOUSE_BLACKLIST = "|".join([MITO, RIBO, r"^H2-", HISTONE,
+    r"^XIST$|^TSIX$|^DDX3Y$|^EIF2S3Y$|^KDM5D$|^UTY$|^ZFY[12]$|^SRY$"])
+CLONOTYPE_ADDON = r"^IG[HKL]V|^IG[HKL]J\d|^IGHD\d"  # V, J, D segments; not IGHD or IGKC/IGLC
+HAEMOGLOBIN_ADDON = r"^HB[ABDEGMQZ]\d*$|^HB[AB]-"  # spares HBEGF, HBP1, HBS1L
+```
+
+Add-ons are appended: `HUMAN_BLACKLIST + "|" + CLONOTYPE_ADDON`. Patterns are case-insensitive,
+so the human string also works on title-case names; prefer the species string anyway.
+
+### Inspect what the default removes on this store
+
+```python
+scarf.configure_output(level="WARNING", progress=False)
+ds = scarf.DataStore("analysis.zarr", min_features_per_cell=-1)  # reopen; see data-access.md
+names = ds.RNA.feats.fetch_all("names").astype(str)
+print("Ensembl-like names:", int(pd.Series(names).str.match(r"ENS[A-Z]*G\d").sum()), "of", len(names))
+for family, pattern in GENE_FAMILY_PATTERNS.items():
+    hits = ds.RNA.feats.grep(pattern)  # upper-cased names
+    print(f"{family:13s} {len(hits):4d}", hits[:6])
+print("current histone symbols the default misses:",
+      len(ds.RNA.feats.grep(r"^H2AC\d|^H2BC\d|^H3C\d|^H4C\d|^H1-[1-6]$")))
+print("default-only matches:",
+      sorted(set(ds.RNA.feats.grep(DEFAULT_HVG_BLACKLIST)) - set(ds.RNA.feats.grep(HUMAN_BLACKLIST))))
+
+run = ds.pipeline.open(label="baseline")
+cells = run["analysis_cell_selection"]
+
+
+def selected(ref):
+    return set(names[np.asarray(ds.load_artifact(ref)["values"][:], dtype=bool)])
+
+
+hvg_default = run["highly_variable_features"]  # same as select_hvgs(cells, top_n=1000)
+hvg_open = ds.select_hvgs(cells, top_n=1000, blacklist="", show_plot=False)  # no name filter
+print("removed by the default:", sorted(selected(hvg_open) - selected(hvg_default)))
+```
+
+The last line is what the default actually costs. 10x 5K PBMC docs dataset: 17 genes, namely 9
+MHC class II, 6 histones, `CCND3` and `CCNL1`.
+
+### Apply a string and check the HVGs
+
+```python
+hvg_fixed = ds.select_hvgs(cells, top_n=1000, blacklist=HUMAN_BLACKLIST, show_plot=False)
+print("entered:", sorted(selected(hvg_fixed) - selected(hvg_default)))
+print("left:", sorted(selected(hvg_default) - selected(hvg_fixed)))
+blocked = re.compile(HUMAN_BLACKLIST, re.IGNORECASE)
+assert not any(blocked.match(gene) for gene in selected(hvg_fixed))
+
+run_fixed = ds.pipeline.run(label="baseline_blacklist_fixed",
+                            params={"hvg": {"blacklist": HUMAN_BLACKLIST}})
+print(run_fixed["highly_variable_features"] == hvg_fixed,
+      ds.inspect_artifact(run_fixed["highly_variable_features"]).parameters["blacklist"]
+      == HUMAN_BLACKLIST)
+```
+
+The pipeline returns the very ref of the explicit call when its cells match. Compare the two
+runs' partitions with `clustering-and-embedding.md`. 10x 5K PBMC docs dataset: `CCND3` and `CCNL1`
+entered, and the Leiden 1.0 partitions agreed (ARI 0.82) about as well as two Leiden seeds of the
+baseline (0.85).
+
+### Decide on the add-ons with marker evidence
+
+```python
+top = ds.get_markers(marker=run_fixed["markers"]).groupby("group_id", sort=False).head(10)
+clonal = top[top["feature_name"].str.match(CLONOTYPE_ADDON, case=False)]
+print("V/J genes among HVGs:", sum(bool(re.match(CLONOTYPE_ADDON, g, re.I)) for g in selected(hvg_fixed)))
+print(clonal.groupby("group_id")["feature_name"].agg(", ".join))  # clusters led by V genes
+panel = ["JCHAIN", "MZB1", "IGKC", "IGLC2", "IGHA1", "IGHG1", "IGHM", "MKI67",
+         "HBB", "HBA1", "ALAS2", "CA1", "AHSP", "SLC4A1"]
+markers = ds.get_markers(marker=run_fixed["markers"], min_score=-1, min_frac_exp=-1)
+frac = markers[markers["feature_name"].isin(panel)].pivot_table(
+    index="group_id", columns="feature_name", values="frac_exp")
+print(frac.round(2).to_string())
+```
+
+- Clonotype add-on: use it when two or more clusters share plasma or B markers (`JCHAIN`, `MZB1`,
+  `CD79A`) and differ mainly by `IGKC` versus `IGLC*` or by V genes in their top markers. Rerun
+  with `HUMAN_BLACKLIST + "|" + CLONOTYPE_ADDON` under a new label and confirm that the plasma
+  clusters now differ by isotype (`IGHA1`, `IGHG1`, `IGHM`) or state (`MKI67`), not light chain.
+  Skip it when expanded clones are the question; use VDJ data then.
+- Haemoglobin add-on: use it when `HBB`/`HBA1` are among the HVGs and are detected at low
+  fractions across clusters that lack `ALAS2`, `CA1`, `AHSP` and `SLC4A1` (ambient RNA). Do not
+  use it when a cluster expresses those erythroid markers; remove contaminating red cells at QC.
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+| --- | --- | --- |
+| `select_hvgs(blacklist=)` | `DEFAULT_HVG_BLACKLIST` | Always pass a species string on current-symbol stores. |
+| `params={"hvg": {"blacklist": ...}}` | default regex | Same string through `ds.pipeline.run`; new label per string. |
+| `blacklist=""` | | Diagnostics only: shows what the name filter removes. |
+| `select_hvgs(max_cells=)` | selected cells - 20 | `np.inf` lets ubiquitous genes compete; the blacklist then matters more. |
+
+## Check before moving on
+
+- The store has symbol names; every family you rely on matches the genes you expect.
+- `inspect_artifact(hvg).parameters["blacklist"]` equals the string you recorded in the log.
+- No HVG matches the string; the entered and left genes are listed in the analysis log.
+- With the clonotype add-on, no cluster's top markers are V genes and plasma clusters are not
+  split by `IGKC` versus `IGLC*`.
+
+## Pitfalls
+
+- Trusting the default on current symbols: replication-dependent histones become HVGs unnoticed.
+- Writing `^IGHD` for D segments: it also removes the IgD constant gene. Use `^IGHD\d`.
+- `^TR[ABDG]D` also matches `TRADD`; TCR V genes such as `TRDV2`, `TRGV9` and `TRAV1-2` mark
+  gamma-delta and MAIT cells. T-cell clusters did not follow TRBV usage in testing, so no TCR
+  add-on is recommended; consider `^TR[AB]V` only when a cluster is led by one TRAV/TRBV pair.
+- `^HB` also removes `HBEGF`. Small clusters (under about 50 cells) move with any HVG change;
+  compare across Leiden seeds before attributing a change to the blacklist.
+- Run `groupby(...).head` on default-filtered marker tables; on a full table (`min_score=-1`) it
+  crashed pandas 3.0 with a segmentation fault in testing.
+- A blacklist never fixes QC: high `percentMito` cells or red-cell contamination need filtering.
+
+## See also
+
+- `features-and-graphs.md`: HVG diagnostics and branching the graph.
+- `quality-control.md`: `percentMito`, `percentRibo` and `run_feature_percentage`.
+- `markers-and-annotation.md`: marker tables and panels used for the add-on decisions.
+- `pipeline-runs-and-artifacts.md`: `params`, labels and artifact identity.