opencode-skills-collection 4.1.7 → 4.1.8

package/bundled-skills/scarf-single-cell/references/clustering-and-embedding.md

@@ -0,0 +1,270 @@
+# Clustering and embedding
+
+Partition a cell graph with Leiden or Paris, choose a partition with evidence, build UMAP or t-SNE
+layouts, and extract labels and coordinates as cell-aligned arrays. Docs:
+<https://scarf.readthedocs.io/en/latest/tutorials/clustering.html>, <https://scarf.readthedocs.io/en/latest/tutorials/dimensionality_reduction.html>,
+<https://scarf.readthedocs.io/en/latest/reference/api/datastore.html>, <https://scarf.readthedocs.io/en/latest/reference/api/integration.html>.
+
+## When to use
+
+- A completed run or an explicit graph exists, and you need other resolutions, a hierarchy or a
+  new layout on that same graph.
+- You must justify a partition (resolution, seed, representation) before naming clusters.
+- You need cluster labels or layout coordinates as a NumPy array or a DataFrame indexed by cell ID.
+
+## Key concepts
+
+- Every producer takes an exact graph `ArtifactRef` (`run["connectivity_map"]` or an integrated
+  graph) and returns a new immutable `ArtifactRef`. None of them writes to `ds.cells`. Repeating a
+  call reuses the stored artifact; `invalidate_cache=True` creates another one.
+- Payloads: Leiden `kind="cluster_labels"` stores `values` (int, numbered from 1); Paris
+  `kind="cluster_cut"` stores `labels`; UMAP and t-SNE `kind="embedding"` store `values` with shape
+  `(n_cells, dims)`.
+- Payload rows follow the graph's frozen cell selection in stored-axis order, not every cell in
+  the store (3,847 of 5,025 cells in the 10x 5K PBMC docs dataset baseline). Recover that
+  selection with `ds.inspect_artifact(ref).input_ref("cell_selection")`.
+- The default pipeline stores `leiden_0.5`, `leiden_0.75`, `leiden_1.0`, `leiden_1.25` and `paris`.
+  `run["clusters"]` is the Leiden candidate with the best PCA silhouette, recorded in
+  `run["cluster_selection"]`. Silhouette favours few compact clusters and can merge distinct
+  lineages or states. Treat that pick as a reproducible baseline only.
+- Leiden is seeded (`random_seed=4444`), so a single run says nothing about stability. Agreement
+  across seeds, nested resolutions, markers and representations is the evidence.
+- Leiden, Paris, layouts, membership strength and separability use only the graph or PCA and take
+  seconds. Marker search is the only step on this page that reads counts.
+
+## Recipes
+
+### Open the graph and baseline refs
+
+```python
+import numpy as np
+import pandas as pd
+import scarf
+
+scarf.configure_output(level="WARNING", progress=False)
+ds = scarf.DataStore("analysis.zarr", nthreads=4, min_features_per_cell=-1)  # writable
+run = ds.pipeline.open(label="baseline")
+graph = run["connectivity_map"]
+print([key for key in run if key.startswith(("leiden", "paris", "cluster"))])
+
+def labels_of(ref):
+    """Return a cluster payload as an array aligned to the ref's cell selection."""
+    key = "labels" if ref.kind == "cluster_cut" else "values"
+    return np.asarray(ds.load_artifact(ref)[key][:])
+```
+
+### Build the partition evidence table
+
+One row per candidate: stability over seeds, nesting in the next coarser candidate, separability,
+graph support and marker support. Each marker search streams counts once.
+
+```python
+from itertools import combinations
+from sklearn.metrics import adjusted_rand_score
+
+leiden, values, rows = {}, {}, []
+for resolution in (0.5, 1.0, 1.5):
+    refs = [ds.run_leiden_clustering(graph, resolution=resolution, random_seed=seed)
+            for seed in (4444, 1, 2)]
+    name = f"leiden_{resolution}"
+    leiden[name], values[name] = refs[0], labels_of(refs[0])  # seed 4444 = the pipeline's
+    seeds = [labels_of(ref) for ref in refs]
+    rows.append({"candidate": name, "n_clusters": len(np.unique(seeds[0])),
+                 "smallest": pd.Series(seeds[0]).value_counts().min(),
+                 "min_seed_ari": min(adjusted_rand_score(a, b) for a, b in combinations(seeds, 2))})
+evidence = pd.DataFrame(rows).set_index("candidate")
+names = list(leiden)
+evidence["nested_frac"] = [np.nan] + [  # share of cells inside one cluster of the coarser one
+    pd.crosstab(values[fine], values[coarse]).max(axis=1).sum() / len(values[fine])
+    for coarse, fine in zip(names, names[1:])]
+sep = ds.metric_cluster_separability(run["pca"], leiden)  # coordinates the graph was built on
+evidence = evidence.join(sep.clustering_scores.set_index("clustering")[
+    ["macro_f1_mean", "silhouette_score"]])
+for name, ref in leiden.items():
+    strength = np.asarray(ds.load_artifact(ds.calc_membership_strength(ref, graph))["values"][:])
+    evidence.loc[name, "weak_cells"] = (strength < 0.5).mean()
+    per_cluster_strength = pd.Series(strength).groupby(values[name]).mean()
+    evidence.loc[name, "min_cluster_strength"] = per_cluster_strength.min()
+    found = ds.get_markers(marker=ds.run_marker_search(ref, features=run["feature_universe"]))
+    n_markers = found["group_id"].value_counts().reindex(
+        [str(g) for g in np.unique(values[name])], fill_value=0)
+    evidence.loc[name, "clusters_lt3_markers"] = int((n_markers < 3).sum())
+print(evidence.round(3).to_string())
+print(sep.cluster_scores.sort_values("f1_score").head(5))  # weakest individual clusters
+nesting = ds.smart_label(leiden["leiden_1.0"], leiden["leiden_0.5"])  # "2a", "2b": splits of 2
+print(pd.Series(labels_of(nesting)).value_counts().head(10))
+selection = ds.load_artifact(run["cluster_selection"])  # the pipeline's silhouette choice
+print(dict(zip(selection.attrs["candidateKeys"], np.asarray(selection["scores"][:]).round(3))))
+print("selected:", selection.attrs["selectedKey"], "invalid:", selection.attrs["invalidReasons"])
+```
+
+Membership strength is the fraction of a cell's graph neighbours sharing its most common label.
+Pass the Harmony ref instead of `run["pca"]` when the graph was built on Harmony coordinates.
+
+### QC and doublet covariates per cluster
+
+```python
+in_run = run.cells.fetch_all("I").astype(bool)
+per_cell = pd.DataFrame({c: ds.cells.fetch_all(c)[in_run]
+                         for c in ("RNA_nCounts", "RNA_nFeatures", "RNA_percentMito")})
+per_cell["doublet_score"] = run.cells.fetch("doublet_score")
+per_cluster = per_cell.groupby(values["leiden_1.0"]).median()
+print(per_cluster.round(2), per_cell.median().round(2), sep="\n")  # each cluster vs all cells
+```
+
+A cluster that stands out only on counts, mito or doublet score is technical until markers say
+otherwise. Doublet scores run high in high-RNA types (plasma cells), so compare within a lineage.
+
+### Representation sensitivity
+
+Repeat the evidence on one branch (more HVGs, other PCs or `k`; `features-and-graphs.md`). The
+cells must match, so labels compare directly.
+
+```python
+alt = ds.pipeline.run(label="hvg2k_pc30_k21", hvg_count=2000, pca_dims=30, neighbors_k=21)
+assert alt["analysis_cell_selection"] == run["analysis_cell_selection"]
+alt_refs = [ds.run_leiden_clustering(alt["connectivity_map"], resolution=resolution)
+            for resolution in (0.5, 1.0, 1.5)]
+for name, ref in leiden.items():  # best match over branch resolutions; k shifts the scale
+    evidence.loc[name, "ari_vs_branch"] = max(ds.metric_label_concordance(ref, other)
+                                              for other in alt_refs)
+print(evidence.round(3).to_string())
+```
+
+### Choose the partition
+
+`run["clusters"]` (the silhouette pick) is a baseline candidate, never the answer by itself.
+Combine the evidence; no single score decides.
+
+1. Stability: keep candidates whose `min_seed_ari` stays near the coarsest one (about 0.9 or
+   more is a rule of thumb). Below that, the optimizer moves boundaries between seeds.
+2. Nesting: prefer a candidate that refines the coarser one (`nested_frac` near 1, `smart_label`
+   shows "2a", "2b"). A non-nested jump, where many cells change parent, is a reason to reject it.
+3. Markers: every cluster a finer candidate adds needs its own specific markers (about 3 or more
+   at default `get_markers` filters, named genes, not only ribosomal, mitochondrial or ambient).
+4. Graph support: added clusters keep high mean membership strength and few weak cells.
+5. QC and doublets: a split explained by counts, mito or doublet score is not a cell type. Keep it
+   as `unresolved` (`markers-and-annotation.md`) instead of naming it.
+6. Representation: boundaries that survive an HVG, PC or `k` branch are robust; those that move
+   are provisional and need markers to stand.
+
+Choose the finest candidate whose added splits all pass 1 to 5, and report 6. If the silhouette
+pick merges populations that markers separate, a finer candidate that passes wins. Prefer a coarser
+global partition plus targeted subclustering (`features-and-graphs.md`, "Subcluster one lineage")
+when the lineage split is stable at a low resolution but finer global resolutions lose seed
+stability, break nesting, or split only one lineage. The usual case is T and NK subsets in blood:
+global HVGs are dominated by between-lineage genes, so a within-lineage graph resolves states
+(naive, memory, regulatory, cytotoxic) that a finer global cut either misses or splits unstably.
+
+(10x 5K PBMC docs dataset) Minimum seed ARI was 0.97, 0.82 and 0.60 at 0.5, 1.0 and 1.5; at 1.0,
+four clusters had fewer than 3 markers; ARI with the branch fell from 0.86 to 0.61. The evidence
+kept 0.5. A T-cell subgraph then gave a FOXP3/CTLA4 group (FOXP3 in 0.43 of cells); no global
+cluster up to resolution 2.0 exceeded 0.22.
+
+### Build a Paris hierarchy
+
+```python
+from dataclasses import asdict
+
+paris = ds.run_paris_clustering(graph)  # n_clusters="auto"; reuses run["paris"] when identical
+result = ds.load_paris_clustering(paris)
+diag = pd.DataFrame([asdict(item) for item in result.diagnostics])
+print(result.n_clusters, diag[["label", "size", "persistence", "decision_margin", "forced"]])
+paris_8 = ds.run_paris_clustering(graph, n_clusters=8)
+print(ds.metric_label_concordance(paris, leiden["leiden_0.5"]))
+```
+
+High `persistence` means a branch survives over a wide range of the hierarchy. `forced=True` groups
+satisfy a structural constraint and are not, by themselves, biological evidence.
+
+### Build UMAP and t-SNE layouts
+
+```python
+init = run["embedding_initialization"]  # K-means initialization built on run["pca"]
+umap_tight = ds.run_umap(graph, init, min_dist=0.5)
+tsne = ds.run_tsne(graph, init, verbose=False)
+# For new coordinates, build a matching init first:
+# init = ds.build_embedding_initialization(new_pca_ref)
+print(ds.load_artifact(umap_tight)["values"].shape, ds.load_artifact(tsne)["values"].shape)
+```
+
+### Get labels and coordinates from a pipeline run
+
+```python
+cells = run.cells.to_pandas_dataframe(
+    ["ids", "clusters", "leiden_1.0", "paris", "umap_1", "umap_2", "doublet_score"])
+print(cells.head())  # one row per run cell, in stored-axis order
+full = run.cells.fetch_all("clusters")  # length ds.cells.N; -1 for cells outside the run
+print(len(full), (full == -1).sum())
+```
+
+`run.cells.columns` lists the frozen fields. `fetch(...)` returns run cells only. `fetch_all(...)`
+fills cells outside the run with -1 (integers), NaN (floats) or "" (strings).
+
+### Get labels and coordinates from explicit artifacts
+
+```python
+def cell_ids_of(ref):
+    selection = ds.inspect_artifact(ref).input_ref("cell_selection")
+    mask = np.asarray(ds.load_artifact(selection)["values"][:], dtype=bool)
+    return ds.cells.fetch_all("ids")[mask]
+
+ids = cell_ids_of(umap_tight)
+assert np.array_equal(ids, cell_ids_of(leiden["leiden_1.5"]))  # same frozen selection
+xy = np.asarray(ds.load_artifact(umap_tight)["values"][:])
+frame = pd.DataFrame({"leiden_1.5": labels_of(leiden["leiden_1.5"]),
+                      "x": xy[:, 0], "y": xy[:, 1]},
+                     index=pd.Index(ids, name="cell_id"))
+print(frame.head())
+```
+
+The same helper works for membership strength, doublet scores, cell cycle and enrichment scores.
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+|---|---|---|
+| `resolution` (Leiden) | `1.0` (pipeline: 0.5, 0.75, 1.0, 1.25) | Raise for finer splits, lower to merge; always compare several |
+| `random_seed` (Leiden) | `4444` | Rerun with 2 or more other seeds to test stability |
+| `backend` (Leiden) | `"igraph"` | `"leidenalg"` only to match an external result |
+| `n_clusters` (Paris) | `"auto"` | An integer to match another partition; it must be cuttable at one height, else `ValueError` |
+| `min_cluster_size` (Paris) | `None` (resolved automatically) | Raise to suppress tiny branches in the auto cut |
+| `min_dist` / `spread` (UMAP) | `1.0` / `2.0` | Lower `min_dist` (0.1 to 0.5) for tighter packing; appearance only |
+| `use_density_map` (UMAP) | `False` | `True` for densMAP when relative density matters |
+| `parallel` (UMAP, t-SNE) | `False` | `True` is faster but not reproducible |
+
+## Check before moving on
+
+- The evidence table exists for at least three candidates, and the chosen one is justified by
+  the six criteria above, not by silhouette alone. Record why coarser and finer were rejected.
+- Every cluster under about 20 cells, or with mean membership strength well below the rest, has
+  been reviewed: doublet group, transition, contaminant or over-split.
+- No cluster is named whose only distinction is a QC metric or doublet score.
+- Lineages that need finer structure are listed for targeted subclustering, with the parent
+  partition kept for audit.
+
+## Pitfalls
+
+- Paris stores `labels`, not `values`. `ds.load_artifact(paris)["values"]` raises `KeyError`.
+- Payload length is the selection size, not `ds.cells.N`. Align through the cell selection before
+  inserting into `ds.cells` or joining with other metadata.
+- Cluster IDs are arbitrary per artifact. Never carry annotations from one partition to another by
+  ID; join on cell IDs or use `smart_label`.
+- `metric_label_concordance` requires both artifacts to share the exact cell-selection ref. For
+  labels from different selections or from metadata, use sklearn on aligned arrays.
+- Comparing branches at the same resolution only: `k` and HVG count shift the resolution scale.
+- UMAP and t-SNE distances, empty space and island size are not biological measurements. Never
+  choose a resolution from a layout alone.
+- A read-only store (`zarr_mode="r"`) raises `PermissionError` unless an identical artifact exists.
+- `run_tsne` uses an `sgtsne` executable on `PATH` when present, otherwise the `sgtsnepi` Python
+  package. The docs state that t-SNE is unsupported on macOS and Windows.
+
+## See also
+
+- `features-and-graphs.md`: building graphs, HVG and `k` branches, subclustering one lineage.
+- `markers-and-annotation.md`: marker tables and labels for the chosen partition.
+- `quality-control.md`: doublet scores and QC covariates behind technical clusters.
+- `plotting.md`: `ds.plots.embedding(layout=umap_ref, color_by=cluster_ref)`,
+  `cluster_connectivity(graph=, groups=, layout=)` and `cluster_tree(graph=, clusters=paris_ref)`.
+- `pipeline-runs-and-artifacts.md`: run keys, `leiden=` pipeline settings, lineage.
+- `integration-and-comparisons.md`: clustering on integrated graphs and batch metrics.

package/bundled-skills/scarf-single-cell/references/data-access.md

@@ -0,0 +1,269 @@
+# Data access
+
+Open, inspect, import, and connect to Scarf DataStores, including Cytebase remote and mounted
+stores. Docs: <https://scarf.readthedocs.io/en/latest/tutorials/data_organization.html>,
+<https://scarf.readthedocs.io/en/latest/tutorials/import_and_export.html>, <https://scarf.readthedocs.io/en/latest/tutorials/cytebase.html>,
+<https://scarf.readthedocs.io/en/latest/tutorials/remote_stores.html>, <https://scarf.readthedocs.io/en/latest/reference/api/datastore.html>, `import_export.html`, `cytebase.html` (same folder).
+
+## When to use
+
+- First contact with any dataset: choose read-only or writable, then inspect before computing.
+- Converting 10x, Matrix Market, H5AD, or Seurat RDS input into a Scarf Zarr store.
+- Finding a Cytebase dataset, exploring it remotely, or mounting it for analysis that saves results.
+- Reopening a store or a mount from an earlier session.
+
+## Key concepts
+
+- A store is one Zarr directory: `cellData` (shared cell table, `ds.cells`), one group per assay
+  (`ds.RNA`; feature table `ds.RNA.feats`; counts; assay artifacts), datastore-scoped artifacts,
+  and `pipeline/runs`. Cell QC columns are assay-prefixed: `RNA_nCounts`, `RNA_nFeatures`,
+  `RNA_percentMito`, `RNA_percentRibo`. Tables are `MetaData` objects, not DataFrames.
+- `I` is the live Boolean cell key (feature tables have one too). `fetch(col)` returns rows where
+  `I` is true; `fetch_all(col)` returns every row. Analytical filtering returns a cell-selection
+  artifact and never edits `I`.
+- `zarr_mode="r+"` is the default. A writable open WRITES: first-open preparation of a freshly
+  written store (QC columns, feature `nCells`/`dropOuts`), the `min_features_per_cell` filter
+  applied to `I`, and the `defaultAssay`/`assayTypes` attributes. `zarr_mode="r"` writes nothing;
+  producers return a matching existing artifact but raise `PermissionError` instead of computing.
+- RNA assays hold cell-major `counts` plus gene-major `countsT`; other assays hold `counts` only.
+  Stores from older releases (Zarr v2, no `countsT`, an `{assay}/state` group) fail to open:
+  re-import the source.
+- A mount is a local writable target whose counts resolve from a separate source recorded in the
+  root `matrixSource` attribute (absolute path or URI). Metadata is copied once at mount time; new
+  artifacts are written only to the target.
+- Cytebase has three layers: `Catalog` (verified local DuckDB copy of the catalog, no counts),
+  `open_datastore` (read-only, everything remote), `mount_datastore` (local writable target,
+  counts remote, plus a `<target>.cytebase.json` receipt beside it).
+- `workspace` names a group inside one Zarr with its own `cellData`, assays, and artifacts (counts
+  under `matrices/`). Writers default to `None` (root layout); keep `None` unless the store was
+  written with a workspace. A mount records one source workspace and rejects another.
+
+## Recipes
+
+### Open and inspect a store
+
+Design and annotation inference starts with `scripts/inspect_store.py STORE.zarr`, which ships with
+the maintained copy of this skill in the Scarf repository (this catalogue copy does not include
+it). Read-only, it flags annotation-like columns (values hidden unless
+`--show-annotation-values`), design-like, author-derived and unflagged columns, and checks
+whether counts look raw, corrected or pre-filtered. Flags are name-based hints: review every
+column. By hand, open read-only until you need to write; `summary()` is metadata-only.
+
+```python
+import scarf
+
+scarf.configure_output(level="WARNING", progress=False)
+ds = scarf.DataStore("analysis.zarr", zarr_mode="r")
+print(ds)                                    # "N_active (N_total) cells", assays, column names
+snap = ds.summary().to_dict()                # JSON-safe
+print(snap["default_assay"], snap["total_cells"], snap["active_cells"], snap["resources"])
+print(snap["pipeline_run_counts"], snap["labeled_pipeline_runs"])
+print([(a["name"], a["assay_type"], a["total_features"], len(a["artifacts"])) for a in snap["assays"]])
+print(ds.pipeline.list_runs())                                   # newest first
+print(ds.list_artifacts(complete_only=True))                     # default-assay artifacts
+print(ds.list_artifacts(scope="datastore", complete_only=True))  # cell selections, snapshots
+ds.show_zarr_tree(start="RNA", depth=1)
+
+# Metadata: request only the columns you need
+meta = ds.cells.to_pandas_dataframe(["ids", "RNA_nCounts", "RNA_nFeatures"], key="I").set_index("ids")
+active_ids = ds.cells.fetch("ids")        # rows where I is True
+all_ids = ds.cells.fetch_all("ids")       # every row
+genes = ds.RNA.feats.to_pandas_dataframe(["ids", "names", "nCells"])
+by_upper = {str(n).upper(): str(n) for n in ds.RNA.feats.fetch_all("names")}  # case-safe lookup
+```
+
+### Convert inputs to Zarr
+
+Every writer follows reader then `*ToZarr(...).dump()`. Open each new store writable once.
+
+```python
+# 10x HDF5: assays inferred from feature types (RNA, ADT, ATAC)
+reader = scarf.CrH5Reader("filtered_feature_bc_matrix.h5")
+print(reader.assayFeats)                  # assays and feature ranges that will be written
+scarf.CrToZarr(reader, zarr_loc="pbmc.zarr", mem_budget="4G").dump()
+# A default count layout that does not fit mem_budget raises MemoryError before writing and
+# names a smaller policy=CountMatrixPolicy(...) (scarf.storage.count_matrix); raise
+# mem_budget or pass that policy.
+ds = scarf.DataStore("pbmc.zarr", default_assay="RNA")   # required when there are 2+ assays
+
+# 10x directory (matrix.mtx, genes/features.tsv, barcodes.tsv; .gz accepted)
+scarf.CrToZarr(scarf.CrDirReader("filtered_feature_bc_matrix"), zarr_loc="dir.zarr").dump()
+
+# Matrix Market: inspect, then pick one candidate explicitly
+candidates = scarf.inspect_mtx("mtx_dir")
+reader = scarf.MtxReader(candidates[0])
+print(reader.assayFeats)
+scarf.MtxToZarr(reader, zarr_loc="mtx.zarr").dump()
+
+# H5AD: inspect first; selected obsm/obs values become artifacts, not live columns
+import h5py
+
+insp = scarf.inspect_h5ad("data.h5ad")
+print(insp.matrixKey, insp.matrixCandidates, insp.integerLike, insp.layers, insp.suggestedAssays)
+with h5py.File("data.h5ad", "r") as h5:  # pass only keys the file has; a missing one is a KeyError
+    obsm = set(h5["obsm"]) if "obsm" in h5 else set()
+    obs = h5["obs"]
+    obs_cols = set(obs) if isinstance(obs, h5py.Group) else set(obs.dtype.names)  # old files
+print(sorted(obsm), sorted(obs_cols))
+roles = {k: r for k, r in {"X_umap": "umap", "X_tsne": "tsne"}.items() if k in obsm}
+kw = dict(embedding_roles=roles, cluster_keys=tuple(c for c in ("clusters",) if c in obs_cols))
+reader = scarf.H5adReader.from_inspect(insp, **kw)
+res = scarf.H5adToZarr(reader, zarr_loc="h5ad.zarr").dump()
+print(dict(res.embeddingArtifacts), dict(res.clusterArtifacts))
+
+# Seurat RDS (an on-disk .rds; .h5seurat is not read)
+si = scarf.inspect_seurat("pbmc.rds")
+reductions = [r.name for r in si.reductions if r.importable]
+with scarf.SeuratReader("pbmc.rds", assays=[si.activeAssay], reductions=reductions) as sr:
+    out = scarf.SeuratToZarr(sr, zarr_loc="seurat.zarr").dump()
+print(out.defaultAssay, dict(out.reductionArtifacts), out.activeIdentity, len(out.notices))
+
+for path in ("dir.zarr", "mtx.zarr", "h5ad.zarr", "seurat.zarr"):
+    scarf.DataStore(path)                 # writable first open prepares the store
+```
+
+`CSVReader`/`CSVtoZarr` (small dense CSV) and `SparseToZarr` (SciPy CSR plus IDs) also exist.
+
+Every converter refuses this way from 1.0.0rc18 on (1.0.0rc17 shrinks the layout to fit). The
+error is a `MemoryError`; releases after 1.0.0rc19 raise its subclass `CountLayoutMemoryError`, so
+catch `MemoryError`. Prefer a larger `mem_budget` when the host has
+the memory. Otherwise copy the policy numbers from the message exactly and record that choice:
+smaller layouts make every later gene-major read slower.
+
+```python
+from scarf.storage.count_matrix import CountMatrixPolicy
+
+policy = CountMatrixPolicy(unitBytes=3906250, chunkBytes=390625)   # copied from the message
+scarf.CrToZarr(reader, zarr_loc="pbmc.zarr", mem_budget="128M", policy=policy).dump()
+```
+
+The need follows genes per cell relative to the gene count, not the number of cells. The 1K PBMC
+CITE-seq file refuses at `512M` and imports at `1G`.
+
+### Find a Cytebase dataset
+
+```python
+from scarf import cytebase
+
+catalog = cytebase.Catalog()              # public Nygen/cytebase, no credentials
+hits = catalog.search("bladder immune", limit=5)      # every word, case-insensitive
+ids = [row["cytebase_id"] for row in hits]            # take IDs from rows, not printed tables
+blood = catalog.find_datasets(tissue="blood", organism="Homo sapiens")  # exact labels
+assay_labels = [row["label"] for row in catalog.list_terms("assay")]    # exact facet labels
+small = catalog.query(
+    "SELECT cytebase_id, cell_count FROM datasets "
+    "WHERE status = ? AND cell_count < ? ORDER BY cell_count LIMIT 3",
+    parameters=["ready", 5000],
+)
+entry = catalog.dataset(ids[0])
+print(entry.describe())                   # Markdown summary; does not open the store
+print(entry.cell_count, entry.source_embeddings())    # source obsm keys, imported or not
+```
+
+### Explore a Cytebase dataset read-only
+
+```python
+ds = catalog.open_datastore(entry.id)     # zarr_mode="r", min_features_per_cell=-1
+print(cytebase.embeddings(ds))            # {"X_umap": ArtifactRef}: keys actually imported
+umap_ref = cytebase.embedding(ds, "X_umap")
+coords = cytebase.embedding_coordinates(ds, umap_ref)  # DataFrame indexed by cell id
+meta = ds.cells.to_pandas_dataframe(["ids", "donor_id"], key="I").set_index("ids")  # a design column
+frame = coords.join(meta)                 # join on ids, never on row order
+```
+
+### Mount a Cytebase dataset for writable analysis
+
+The target must be a new local path. Mounting copies metadata and verifies the source; expect a
+few minutes for a few thousand cells.
+
+```python
+analysis = catalog.mount_datastore(entry.id, at="work/analysis.zarr")
+# Writes work/analysis.zarr (copied metadata, future artifacts) and work/analysis.zarr.cytebase.json
+print(analysis.zarr_mode, int(analysis.cells.fetch_all("I").sum()), analysis.cells.N)
+print(cytebase.embeddings(analysis))      # {}: source artifacts are not copied into a mount
+```
+
+### Reopen a mount, or mount any store that owns its counts
+
+```python
+# Verify against the current published build (needs the .cytebase.json sidecar beside it)
+analysis = catalog.mount_datastore(entry.id, at="work/analysis.zarr")
+# Or open it like any store: counts resolve from matrixSource; no catalog or sidecar needed
+analysis = scarf.DataStore("work/analysis.zarr", min_features_per_cell=-1)
+# Generic mount; for s3:// or gs:// sources also pass storage_options (not executed here)
+mounted = scarf.mount_datastore("shared/data.zarr", at="my_analysis.zarr", default_assay="RNA")
+```
+
+### Download a documentation dataset
+
+```python
+repo = scarf.cytebase.connect("scarf_docs")
+print(repo.list_datasets())
+path = repo.download_dataset("tenx_5K_pbmc_rnaseq", destination="scarf_datasets", zarr=True)
+ds = scarf.DataStore(f"{path}/data.zarr", zarr_mode="r")  # prepared; run label "docs_default"
+raw = repo.download_dataset("xin_1K_pancreas_rnaseq", destination="scarf_datasets")  # source files
+```
+
+## Parameters that matter
+
+| Parameter | Default | Change when |
+|---|---|---|
+| `zarr_mode` | `"r+"` | `"r"` to inspect, share, or protect a store; a fresh store needs one `"r+"` open |
+| `default_assay` | stored value, or the only assay | first open of a multi-assay store (otherwise `ValueError`) |
+| `min_features_per_cell` | `10` | `-1` to leave `I` untouched on writable opens |
+| `mito_pattern` / `ribo_pattern` | `None` (`^MT-`, `^RPS\|^RPL\|^MRPS\|^MRPL`) | non-human genes (`^mt-`), only on the FIRST writable open |
+| `assay_types` | inferred from assay names | custom names, e.g. `{"GEX": "RNA"}` |
+| `nthreads`, `mem_budget` | env `SCARF_WORKERS`, `SCARF_MEM_BUDGET`, else detected | see `performance-and-export.md` |
+| `workspace` | `None` | store written into a named workspace |
+| `storage_options` | `None` | object-store credentials or endpoints (read from env vars) |
+| `zarrProfile` | from location (`fast_local`/`cloud`) | affects newly written arrays only |
+| `Catalog(bucket=, token=)` | `Nygen/cytebase`, `None` (HF login if present) | private bucket; `token=False` forces anonymous |
+| `search(limit=)`, `ready_only=` | `50`, `True` | `limit=None` for all; `ready_only=False` for unfinished |
+
+## Check before moving on
+
+- `print(ds)`: active versus total cells, expected assays, QC columns present.
+- `snap["default_assay"]` is the assay you intend; `snap["labeled_pipeline_runs"]` and
+  `list_artifacts` show existing work to reuse (see `pipeline-runs-and-artifacts.md`).
+- Imports: `reader.assayFeats` lists the assays to be written; afterwards check
+  `ds.RNA.rawData.shape` and dtype. For H5AD, `insp.integerLike` should be `True` for raw counts,
+  but SCT-corrected counts are integers too: run "Check the matrix first" in `quality-control.md`.
+- Cytebase: `entry.row["status"] == "ready"`; check `cytebase.embeddings(ds)` before plotting an
+  imported layout.
+- Mounts: `<target>.cytebase.json` exists beside the target; `ds.RNA.rawData.shape` resolves.
+
+## Pitfalls
+
+- A writable open with `min_features_per_cell=k` silently removes cells with at most `k` features
+  from `I` and persists it. Reopening with a lower value does not restore cells;
+  `ds.cells.reset_key("I")` does.
+- Stores just written by a writer or `SubsetZarr` raise `Assay 'RNA' is not prepared. Rebuild ...`
+  when opened with `zarr_mode="r"`. Open once with `"r+"`; no rebuild is needed.
+- Percent patterns are fixed at first preparation; a different pattern later raises `ValueError`.
+  Use `run_feature_percentage` (see `quality-control.md`) for another gene set.
+- `open_datastore` rejects writes and any `min_features_per_cell` other than `-1`. Both mount
+  functions reject `zarr_mode="r"`; `scarf.mount_datastore` refuses an existing target, while
+  `catalog.mount_datastore` reopens one only when its matching sidecar is present.
+- Catalog reopen needs the sidecar: copying a mount without `<name>.cytebase.json` makes
+  `catalog.mount_datastore` raise `FileExistsError`; plain `scarf.DataStore` still opens it.
+- The mount source must stay at its recorded path or URI. Mounting a mount raises; repack first.
+  A changed Cytebase build requires a new mount directory.
+- Printed `CatalogResults` tables HTML-escape text (`_` as `&#95;`, `'` as `&#39;`). Read IDs and
+  facet labels from the row dicts (`row["cytebase_id"]`, `row["label"]`), never from the table.
+- A fresh mount has no imported source embeddings. Plot them from the read-only `ds` and join by id.
+- `to_pandas_dataframe(columns)` defaults to `key=None` (all rows); pass `key="I"` for active cells.
+- `to_mtx(..., compress=True)` writes the feature `feature_type` column as the 10x feature type.
+  CELLxGENE stores keep gene biotypes there, so re-import split one assay into thousands. Inspect
+  `reader.assayFeats` before `dump()` or export with `compress=False`.
+- Source column names containing `/` or `\` are stored with `_`. Reserved columns `ids`, `names`,
+  `I` from a source are skipped. H5AD multi-assay import needs `assay_split_key` on `H5adToZarr`.
+- `download_dataset(..., zarr=True)` keeps `data.zarr.tar.gz` beside `data.zarr` (double disk).
+- Public atlas matrices (CELLxGENE, Seurat submissions) can hold SCT-corrected counts in `raw.X`,
+  with genes such as rRNA removed and cells already filtered. Check before QC or count models.
+
+## See also
+
+- `performance-and-export.md`: budgets, remote I/O cost, repacking a mount, exports.
+- `pipeline-runs-and-artifacts.md`: runs, artifact inspection, lineage.
+- `quality-control.md`: filtering cells without editing `I`.
+- `plotting.md`: `ds.plots.embedding(layout=umap_ref, ...)` for imported layouts.