scatrans 0.9.8.dev1__tar.gz → 0.9.8.dev2__tar.gz

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/CHANGELOG.md

@@ -5,6 +5,50 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.8] - 2026-07-03
+
+### Changed (user-visible default behavior)
+- **`active_score()` built-in `significant` list** now uses the same default thresholds as
+  `filter_active_genes(preset="heuristic")`: `logFC > 0.35`, `unspliced_excess_residual > 1.0`,
+  `active_score >= 55`, `active_score_fdr < 0.25`, plus `p_adj` / `unspliced_excess_fdr` cutoffs.
+  Re-running analyses with defaults may return a **different** significant gene set than 0.9.2
+  (typically more genes when signal is present; aligns with the documented post-hoc workflow).
+- **`active_score(logfc_cutoff=...)` default** lowered from `0.5` to `0.35` to match the heuristic
+  preset (only affects the built-in significant mask and metadata; ranking in `all_results` unchanged).
+- **`gamma_method="empirical_bayes"` + `prior_weight`**: fixed a numeric floor so `prior_weight` in
+  the usual tuning range (0.5–5.0, including the default 5.0) now changes `count_pseudocount` and
+  shrinkage weights instead of being pinned at 1.0. EB gamma values may differ from 0.9.2 at the
+  same `prior_weight`.
+
+### Fixed
+- **`scat.pl.gseaplot`**: RES curve / hit ticks and the bottom ranked-metric bar now share the same
+  gene order by preferring `gsea_result.attrs["ranking"]` over caller-supplied row order.
+- **`scat.pl` robustness**: placeholder figures for empty or incomplete volcano/comet inputs;
+  `bias_diagnostic_plot` validates external `axes`; `active_genes_heatmap` default `show=True` no
+  longer hits a Python `UnboundLocalError` on `plt`.
+- **PyDESeq2 pseudobulk DE**: log warning + `de_df.attrs` diagnostics when genes are skipped by
+  `min_counts_per_gene` or marked NaN by DESeq2 independent filtering (neutral fill is not
+  "tested and non-significant").
+- **Memento DE**: log warning + `n_genes_not_returned_by_memento` when memento drops genes
+  internally; reindexed neutral rows are documented in diagnostics.
+- **`restore_raw_counts`**: reject `layers["counts"]` restore when `raw_gene_list` order differs
+  from current `var_names` (same guard as `adata.raw`).
+- **`active_score` pseudobulk**: `pb_x_layer="X"` now means `adata.X`, matching
+  `differential_expression()`.
+- **`_velocity._estimate_eb_prior_from_reference`**: removed unreachable trim fallback branch;
+  trim-skipped path labeled `empirical_bayes_median_mad`.
+- **`_utils._get_group_mean`**: removed redundant sparse/dense branches.
+
+### Added
+- Shared `HEURISTIC_FILTER_DEFAULTS` in `tl.py` (single source for heuristic preset + built-in
+  significant mask).
+- Tests: gseaplot ranking alignment, `prior_weight` EB sensitivity, significant vs heuristic
+  filter parity, PyDESeq2/Memento diagnostic paths, `restore_raw_counts` order guard,
+  `active_genes_heatmap` default show, `pb_x_layer="X"` sentinel.
+
+### Documentation
+- README built-in `significant` section updated to match `HEURISTIC_FILTER_DEFAULTS`.
+
 ## [0.9.2 bugfix 2026-07-02]
 
 ### Fixed

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/CITATION.cff

@@ -8,7 +8,7 @@ authors:
 repository-code: "https://github.com/scATrans/scatrans"
 url: "https://github.com/scATrans/scatrans"
 license: Apache-2.0
-version: 0.9.2
+version: 0.9.8
 keywords:
   - single-cell
   - RNA-seq

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scatrans
-Version: 0.9.8.dev1
+Version: 0.9.8.dev2
 Summary: Single-cell Active Transcription Analysis
 Author: scATrans Developers
 License: Apache-2.0
@@ -805,16 +805,25 @@ Primary result columns use **unspliced / nascent excess** terminology (not RNA v
 
 ### Built-in `significant` gene list
 
-When `use_permutation=True`, the internal mask requires **all** of:
+When `use_permutation=True`, the built-in mask uses the same default thresholds as
+`filter_active_genes(..., preset="heuristic")` (see `HEURISTIC_FILTER_DEFAULTS` in
+`tl.py`). Under default parameters it requires **all** of:
 
-- `logFC > logfc_cutoff` (default 0.5)
+- `logFC > logfc_cutoff` (default **0.35**)
 - `p_adj < pval_cutoff` (default 0.05)
-- `unspliced_excess_residual > 0`
+- `unspliced_excess_residual > 1.0` (default residual magnitude cutoff)
+- `active_score >= 55.0`
+- `active_score_fdr < 0.25` (when permutation computed composite-score FDR)
 - `unspliced_excess_fdr < unspliced_excess_fdr_cutoff` (default 0.05)
 
-Without `use_permutation=True`, the built-in `significant` list is **empty** (FDR on unspliced excess cannot be computed). Use `all_results` + `filter_active_genes` for custom thresholds.
+Without `use_permutation=True`, the built-in `significant` list is **empty** (FDR on
+unspliced excess cannot be computed). Use `all_results` + `filter_active_genes` for
+custom thresholds.
 
-On real data the built-in list often returns zero or few genes. Use the full table in `all_results`, sorted by `active_score` descending.
+On low-signal data the built-in list may still be small. Use the full table in
+`all_results`, sorted by `active_score` descending. If you need different cutoffs,
+pass explicit arguments to `filter_active_genes` rather than assuming the built-in
+list matches a custom `logfc_cutoff` override on `active_score()`.
 
 After each run inspect the diagnostics:
 

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/README.md

@@ -754,16 +754,25 @@ Primary result columns use **unspliced / nascent excess** terminology (not RNA v
 
 ### Built-in `significant` gene list
 
-When `use_permutation=True`, the internal mask requires **all** of:
+When `use_permutation=True`, the built-in mask uses the same default thresholds as
+`filter_active_genes(..., preset="heuristic")` (see `HEURISTIC_FILTER_DEFAULTS` in
+`tl.py`). Under default parameters it requires **all** of:
 
-- `logFC > logfc_cutoff` (default 0.5)
+- `logFC > logfc_cutoff` (default **0.35**)
 - `p_adj < pval_cutoff` (default 0.05)
-- `unspliced_excess_residual > 0`
+- `unspliced_excess_residual > 1.0` (default residual magnitude cutoff)
+- `active_score >= 55.0`
+- `active_score_fdr < 0.25` (when permutation computed composite-score FDR)
 - `unspliced_excess_fdr < unspliced_excess_fdr_cutoff` (default 0.05)
 
-Without `use_permutation=True`, the built-in `significant` list is **empty** (FDR on unspliced excess cannot be computed). Use `all_results` + `filter_active_genes` for custom thresholds.
+Without `use_permutation=True`, the built-in `significant` list is **empty** (FDR on
+unspliced excess cannot be computed). Use `all_results` + `filter_active_genes` for
+custom thresholds.
 
-On real data the built-in list often returns zero or few genes. Use the full table in `all_results`, sorted by `active_score` descending.
+On low-signal data the built-in list may still be small. Use the full table in
+`all_results`, sorted by `active_score` descending. If you need different cutoffs,
+pass explicit arguments to `filter_active_genes` rather than assuming the built-in
+list matches a custom `logfc_cutoff` override on `active_score()`.
 
 After each run inspect the diagnostics:
 

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/pyproject.toml

@@ -71,7 +71,7 @@ scatrans = [
 
 [tool.setuptools_scm]
 write_to = "src/scatrans/_version.py"
-fallback_version = "0.9.2"
+fallback_version = "0.9.8"
 version_scheme = "guess-next-dev"
 local_scheme = "no-local-version"
 

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/src/scatrans/__init__.py

@@ -87,7 +87,7 @@ __all__ = [
 try:
     from ._version import version as __version__
 except ImportError:
-    __version__ = "0.9.2"
+    __version__ = "0.9.8"
 
 # Optional: prevent some internal modules from appearing too prominently
 # in casual inspection while still allowing advanced users to do

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/src/scatrans/_de.py

@@ -301,7 +301,12 @@ def _run_de_wrapper(
                 )
             stat_res.summary()
 
-        res2 = stat_res.results_df.copy().reindex(ad_temp.var_names)
+        res_df = stat_res.results_df.copy()
+        n_genes_filtered_low_count = int((~gene_keep).sum())
+        n_genes_nan_from_deseq2 = (
+            int(res_df["padj"].isna().sum()) if "padj" in res_df.columns else 0
+        )
+        res2 = res_df.reindex(ad_temp.var_names)
         de_df = pd.DataFrame(index=ad_temp.var_names)
         de_df["logFC"] = res2["log2FoldChange"].reindex(ad_temp.var_names)
         de_df["p_val"] = res2.get("pvalue", pd.Series(np.nan, index=res2.index)).reindex(
@@ -314,6 +319,24 @@ def _run_de_wrapper(
         de_df["logFC"] = de_df["logFC"].fillna(0.0)
         de_df["p_val"] = de_df["p_val"].fillna(1.0)
         de_df["p_adj"] = de_df["p_adj"].fillna(1.0)
+        n_total = len(de_df)
+        de_df.attrs["n_genes_filtered_low_count"] = n_genes_filtered_low_count
+        de_df.attrs["n_genes_nan_from_deseq2"] = n_genes_nan_from_deseq2
+        de_df.attrs["pydeseq2_neutral_fill"] = True
+        if n_total > 0 and (n_genes_filtered_low_count > 0 or n_genes_nan_from_deseq2 > 0):
+            logger.warning(
+                "PyDESeq2: %d/%d genes (%.1f%%) skipped by min_counts_per_gene filter; "
+                "%d/%d genes (%.1f%%) had NaN padj from DESeq2 independent filtering/outliers. "
+                "These genes appear as neutral values (logFC=0, p_adj=1) and are NOT "
+                "'tested and non-significant'. "
+                "See de_df.attrs['n_genes_filtered_low_count'] and ['n_genes_nan_from_deseq2'].",
+                n_genes_filtered_low_count,
+                n_total,
+                100.0 * n_genes_filtered_low_count / n_total,
+                n_genes_nan_from_deseq2,
+                n_total,
+                100.0 * n_genes_nan_from_deseq2 / n_total,
+            )
         return de_df
 
     else:
@@ -429,13 +452,9 @@ def _run_mixedlm_de(
             lrt_stat = -2.0 * (m_null.llf - m_full.llf)
             lrt_p = float(chi2.sf(max(lrt_stat, 0.0), 1))
 
-            # Extract condition coef (target vs ref)
-            # The param name is typically "C(condition)[T.<target>]" or similar
-            coef_name = None
-            for pname in m_full.params.index:
-                if "condition" in str(pname) and target_group in str(pname):
-                    coef_name = pname
-                    break
+            # Extract condition coef (target vs ref) — exact statsmodels name only
+            expected_coef = f"C(condition)[T.{target_group}]"
+            coef_name = expected_coef if expected_coef in m_full.params.index else None
             if coef_name is None:
                 return idx, 0.0, 1.0, 1.0, 0.0, True
             logfc = float(m_full.params.get(coef_name, 0.0))
@@ -702,8 +721,9 @@ def _run_memento_de(
         pval_raw = result["de_pval"]
     else:
         pval_raw = pd.Series(1.0, index=res_index)
-    pvals = pd.to_numeric(pval_raw, errors="coerce").reindex(res_index).fillna(1.0)
-    de_df["p_val"] = pvals
+    pvals_for_correction = pd.to_numeric(pval_raw, errors="coerce").reindex(res_index)
+    valid_pval_mask = pvals_for_correction.notna()
+    de_df["p_val"] = pvals_for_correction.fillna(1.0)
 
     # Preserve Memento-native adjusted p if present (audit trail); package p_adj uses BH for consistency.
     for native_col in ("de_padj", "de_padjs", "padj", "p_adj", "de_fdr"):
@@ -713,8 +733,28 @@ def _run_memento_de(
             ).reindex(res_index)
             break
 
-    with _de_warning_context():
-        de_df["p_adj"] = multipletests(pvals.values, method="fdr_bh")[1]
+    de_df["p_adj"] = 1.0
+    if valid_pval_mask.sum() > 0:
+        with _de_warning_context():
+            de_df.loc[valid_pval_mask, "p_adj"] = multipletests(
+                pvals_for_correction[valid_pval_mask].values, method="fdr_bh"
+            )[1]
+
+    # Memento typically drops low-coverage genes entirely (not NaN rows). Genes absent
+    # from the memento result are reindexed below with neutral fill — same UX issue as
+    # PyDESeq2 filtered genes, but via row omission rather than NaN padj.
+    n_genes_not_returned = int(len(adata.var_names.difference(res_index)))
+    de_df.attrs["n_genes_not_returned_by_memento"] = n_genes_not_returned
+    de_df.attrs["n_genes_missing_pval"] = int((~valid_pval_mask).sum())
+    if n_genes_not_returned > 0:
+        logger.warning(
+            "Memento: %d/%d genes were not returned by binary_test_1d (e.g. internal "
+            "min_perc_group filtering) and will appear as neutral values (logFC=0, p_adj=1) "
+            "after reindexing. These were not tested. "
+            "See de_df.attrs['n_genes_not_returned_by_memento'].",
+            n_genes_not_returned,
+            len(adata.var_names),
+        )
 
     # Expose Memento's native columns for users who want mean + variability signals
     for src, dst in [

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/src/scatrans/_utils.py

@@ -537,8 +537,6 @@ def _get_group_mean(matrix: Any, mask: np.ndarray) -> np.ndarray:
     if np.sum(mask) == 0:
         raise ValueError("Cannot compute group mean for an empty group.")
     sub = matrix[mask]
-    if sparse.issparse(sub):
-        return np.asarray(sub.mean(axis=0)).ravel()
     return np.asarray(sub.mean(axis=0)).ravel()
 
 

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/src/scatrans/_velocity.py

@@ -76,25 +76,23 @@ def _estimate_eb_prior_from_reference(
     else:
         lo, hi = np.quantile(r_valid, [trim_fraction, 1.0 - trim_fraction])
         trim_mask = (r_valid >= lo) & (r_valid <= hi)
-        r_trim = r_valid[trim_mask] if trim_mask.sum() >= min_genes_for_prior else r_valid
-        if r_trim.size < min_genes_for_prior:
-            # After trimming for robustness we have too few; fall back to the same
-            # global-ratio estimator used for the "not enough genes" case. This makes
-            # fallback_triggered semantically consistent (we used the non-robust path).
-            fallback_triggered = True
-            global_ratio = (float(np.sum(U_r)) + eps) / (float(np.sum(S_r)) + eps)
-            prior_mean_log = float(np.log(global_ratio))
-            tau = max(0.25, _robust_mad_scale(r_valid) if r_valid.size else 0.5)
-            method_detail = "empirical_bayes_fallback_global_ratio"
-            n_genes_used = int(r_valid.size)
-        else:
+        n_trimmed = int(trim_mask.sum())
+        if n_trimmed >= min_genes_for_prior:
+            r_trim = r_valid[trim_mask]
             method_detail = "empirical_bayes_trimmed_median_mad"
-            prior_mean_log = float(np.median(r_trim))
-            tau = _robust_mad_scale(r_trim)
-            n_genes_used = int(r_trim.size)
-
-    # prior_weight scales count pseudocount for observation precision (backward-compat knob)
-    count_pseudo = max(count_pseudocount, prior_weight * 0.2)
+        else:
+            # Quantile trim would leave too few genes; use all valid ratios instead.
+            r_trim = r_valid
+            method_detail = "empirical_bayes_median_mad"
+        prior_mean_log = float(np.median(r_trim))
+        tau = _robust_mad_scale(r_trim)
+        n_genes_used = int(r_trim.size)
+
+    # prior_weight scales observation-precision pseudocount (not direct shrinkage like
+    # heuristic_shrink). Use a low floor so the default tunable range (0.5–5.0) is not
+    # flattened by the legacy count_pseudocount=1.0 ceiling.
+    _count_pseudo_floor = 0.05
+    count_pseudo = max(_count_pseudo_floor, prior_weight * 0.2) * float(count_pseudocount)
 
     eb_prior = {
         "prior_mean_log": prior_mean_log,

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/src/scatrans/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.9.8.dev1'
-__version_tuple__ = version_tuple = (0, 9, 8, 'dev1')
+__version__ = version = '0.9.8.dev2'
+__version_tuple__ = version_tuple = (0, 9, 8, 'dev2')
 
 __commit_id__ = commit_id = None

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/src/scatrans/pl.py

@@ -1903,6 +1903,14 @@ def gseaplot(
             fig, ax = _empty_placeholder_fig("No ranked genes")
             return fig, ax
 
+        # Prefer gseapy's post-sort ranking so RES/hits and the bottom bar share one axis.
+        if gsea_result is not None:
+            ranking_attr = getattr(gsea_result, "attrs", {}).get("ranking")
+            if ranking_attr is not None:
+                ranked = pd.Series(ranking_attr, dtype=float)
+                ranked.index = ranked.index.astype(str)
+                ranked = ranked.dropna()
+
         # Try to get precomputed data from gsea_result.attrs
         RES = None
         hits = None
@@ -2647,8 +2655,6 @@ def active_genes_heatmap(
             **kwargs,
         )
         if save_path:
-            import matplotlib.pyplot as plt
-
             plt.savefig(save_path, dpi=300, bbox_inches="tight")
             logger.info("Heatmap saved → %s", save_path)
         if show:

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/src/scatrans/tl.py

@@ -56,11 +56,26 @@ try:
 
     VERSION = _version.version
 except (ImportError, AttributeError):
-    VERSION = "0.9.2"
+    VERSION = "0.9.8"
 
 logger = logging.getLogger(__name__)
 logger.addHandler(logging.NullHandler())
 
+# Shared defaults for filter_active_genes(preset="heuristic") and the built-in
+# active_score() significant list. Keep in sync — both entry points should agree
+# under default parameters.
+HEURISTIC_FILTER_DEFAULTS: dict[str, float | None] = {
+    "active_score_cutoff": 55.0,
+    "pval_cutoff": 0.05,
+    "unspliced_excess_residual_cutoff": 1.0,
+    "logfc_cutoff": 0.35,
+    "active_score_fdr_cutoff": 0.25,
+    "unspliced_excess_fdr_cutoff": 0.05,
+    "effective_gamma_min": 0.05,
+    "effective_gamma_max": 1.0,
+    "delta_variance_min": None,
+}
+
 
 def _materialize_if_view(adata: ad.AnnData) -> ad.AnnData:
     """Return a writable AnnData when ``adata`` is an obs/var view (needed before in-place ``.obs`` writes)."""
@@ -248,7 +263,7 @@ def active_score(
     weight_unspliced: float = 1.0,
     weight_pval: float = 1.0,
     pval_cutoff: float = 0.05,
-    logfc_cutoff: float = 0.5,
+    logfc_cutoff: float = HEURISTIC_FILTER_DEFAULTS["logfc_cutoff"],  # type: ignore[arg-type]
     unspliced_excess_fdr_cutoff: float = 0.05,
     de_method: str = "t-test_overestim_var",  # freely switchable basic option, e.g. "wilcoxon"
     pseudobulk_de_backend: str = "pydeseq2",  # "pydeseq2" or "scanpy" when use_pseudobulk=True
@@ -304,7 +319,9 @@ def active_score(
     # Gamma estimation method for reference group unspliced/spliced ratio.
     # "heuristic_shrink": classic global-ratio + prior_weight shrinkage (default, prior_weight=5.0)
     # "robust_median": use median ratio from reference for better stability with small reference groups
-    # "empirical_bayes": robust log-ratio empirical Bayes shrinkage (recommended for small reference)
+    # "empirical_bayes": robust log-ratio empirical Bayes shrinkage (recommended for small reference).
+    # Note: prior_weight affects empirical_bayes via count_pseudocount (observation precision), so its
+    # impact is gentler than on heuristic_shrink/robust_median (direct additive shrinkage).
     # "raw": minimal shrinkage (use observed ratios directly)
     gamma_method: str = "heuristic_shrink",
     ranking_mode: str = "composite",
@@ -365,8 +382,10 @@ def active_score(
 
     Diagnostics (including global unspliced fraction and bias fit details) are stored
     under adata.uns["scatrans"]["diagnostics"]. The full ranked table (all_results)
-    is the main output; the built-in significant list is produced by a strict
-    conjunction of thresholds and is often small or empty.
+    is the main output; the built-in significant list uses the same default thresholds
+    as :func:`filter_active_genes` with ``preset="heuristic"`` (logFC > 0.35,
+    unspliced_excess_residual > 1.0, active_score >= 55, etc.) and may still be
+    small on low-signal data.
 
     A separate function diagnose_design is available to summarize the experimental
     design and surface relevant warnings before analysis.
@@ -679,7 +698,7 @@ def active_score(
             sample_col,
             groupby,
             layers=["spliced", "unspliced"],
-            x_layer=pb_x_layer,
+            x_layer=pb_x_layer if pb_x_layer != "X" else None,
             use_total_for_x=pb_use_total_for_x,
             min_cells=min_cells,
             min_counts=min_counts,
@@ -1004,6 +1023,24 @@ def active_score(
         if _gk in gamma_info:
             velocity_diag[_gk] = gamma_info[_gk]
 
+    pydeseq2_diag = {}
+    if is_pseudobulk and pseudobulk_de_backend == "pydeseq2" and hasattr(de_df, "attrs"):
+        pydeseq2_diag = {
+            "used": True,
+            "n_genes_filtered_low_count": int(de_df.attrs.get("n_genes_filtered_low_count", 0)),
+            "n_genes_nan_from_deseq2": int(de_df.attrs.get("n_genes_nan_from_deseq2", 0)),
+            "neutral_fill": bool(de_df.attrs.get("pydeseq2_neutral_fill", True)),
+            "note": (
+                "Genes filtered by min_counts or marked NaN by DESeq2 independent filtering "
+                "appear as logFC=0, p_adj=1 and are not 'tested and non-significant'."
+            ),
+        }
+    n_memento_not_returned = (
+        int(de_df.attrs.get("n_genes_not_returned_by_memento", 0))
+        if (use_memento_de and hasattr(de_df, "attrs"))
+        else 0
+    )
+
     diagnostics: dict[str, Any] = {
         "n_cells": int(adata.n_obs),
         "n_genes_input": int(adata.n_vars),
@@ -1032,6 +1069,17 @@ def active_score(
                 else None
             ),
         },
+        "pydeseq2": pydeseq2_diag or {"used": False},
+        "memento": {
+            "used": bool(use_memento_de),
+            "n_genes_not_returned": n_memento_not_returned,
+            "note": (
+                "Genes dropped by memento internal filters appear as logFC=0, p_adj=1 "
+                "after reindexing and were not tested."
+                if use_memento_de
+                else None
+            ),
+        },
     }
     if mode == "advanced" and moments_info:
         diagnostics["velocity"]["moments"] = {
@@ -1283,14 +1331,25 @@ def _finalize_active_score_results(
     ue_fdr_cutoff = extra_metadata.get("unspliced_excess_fdr_cutoff", 0.05)
 
     if use_permutation and UNSPLICED_EXCESS_FDR_COL in adata.var.columns:
+        sig_pval = extra_metadata.get("pval_cutoff", HEURISTIC_FILTER_DEFAULTS["pval_cutoff"])
+        sig_logfc = extra_metadata.get("logfc_cutoff", HEURISTIC_FILTER_DEFAULTS["logfc_cutoff"])
+        sig_resid = HEURISTIC_FILTER_DEFAULTS["unspliced_excess_residual_cutoff"]
+        sig_active = HEURISTIC_FILTER_DEFAULTS["active_score_cutoff"]
+        sig_active_fdr = HEURISTIC_FILTER_DEFAULTS["active_score_fdr_cutoff"]
         mask = (
-            (adata.var["p_adj"] < extra_metadata.get("pval_cutoff", 0.05))
-            & (adata.var["logFC"] > extra_metadata.get("logfc_cutoff", 0.5))
-            & (adata.var[residual_col] > 0)
+            (adata.var["p_adj"] < sig_pval)
+            & (adata.var["logFC"] > sig_logfc)
+            & (adata.var[residual_col] > sig_resid)
+            & (adata.var["active_score"] >= sig_active)
             & (adata.var["valid_expr"])
         )
         if extra_metadata.get("use_fdr_for_significance", True):
             mask = mask & (adata.var[UNSPLICED_EXCESS_FDR_COL] < ue_fdr_cutoff)
+        if "active_score_fdr" in adata.var.columns and sig_active_fdr is not None:
+            mask = mask & (
+                adata.var["active_score_fdr"].notna()
+                & (adata.var["active_score_fdr"] < sig_active_fdr)
+            )
         else:
             logger.warning(
                 "Permutation space is very small; unspliced_excess_fdr was not applied "
@@ -1367,15 +1426,16 @@ def _finalize_active_score_results(
         "use_fdr_for_significance": extra_metadata.get("use_fdr_for_significance"),
         "perm_disabled_reason": extra_metadata.get("perm_disabled_reason"),
         "significant_criteria": {
-            "logFC": f"> {extra_metadata.get('logfc_cutoff')}",
-            "p_adj": f"< {extra_metadata.get('pval_cutoff')}",
-            "unspliced_excess_residual": "> 0",
+            "logFC": f"> {extra_metadata.get('logfc_cutoff', HEURISTIC_FILTER_DEFAULTS['logfc_cutoff'])}",
+            "p_adj": f"< {extra_metadata.get('pval_cutoff', HEURISTIC_FILTER_DEFAULTS['pval_cutoff'])}",
+            "unspliced_excess_residual": f"> {HEURISTIC_FILTER_DEFAULTS['unspliced_excess_residual_cutoff']}",
+            "active_score": f">= {HEURISTIC_FILTER_DEFAULTS['active_score_cutoff']}",
+            "active_score_fdr": f"< {HEURISTIC_FILTER_DEFAULTS['active_score_fdr_cutoff']}",
             "unspliced_excess_fdr": (
                 f"< {ue_fdr_cutoff} (requires use_permutation=True)"
                 if use_permutation
                 else "not evaluated (use_permutation=False)"
             ),
-            "active_score": "ranking only (not used for significance)",
         },
         "use_delta_variance_pval": extra_metadata.get("use_delta_variance_pval"),
         "delta_var_pval_cutoff": extra_metadata.get("delta_var_pval_cutoff"),
@@ -1529,8 +1589,8 @@ def differential_expression(
     Returns
     -------
     (adata_with_results, results_df)
-        - results_df is a ranked DataFrame (by |logFC| or p_adj) containing
-          at minimum: logFC, p_val, p_adj, and (when use_memento_de) the
+        - results_df is sorted by p_adj (ascending; most significant first) and
+          contains at minimum: logFC, p_val, p_adj, and (when use_memento_de) the
           native memento_de_* / memento_dv_* columns.
         - adata.var is updated with the same columns for convenience.
         - Metadata is stored under adata.uns["scatrans"].
@@ -1731,6 +1791,22 @@ def differential_expression(
         if (use_mixed_model and hasattr(de_df, "attrs"))
         else 0.0
     )
+    pydeseq2_diag = {}
+    if use_pseudobulk and pseudobulk_de_backend == "pydeseq2" and hasattr(de_df, "attrs"):
+        pydeseq2_diag = {
+            "n_genes_filtered_low_count": int(de_df.attrs.get("n_genes_filtered_low_count", 0)),
+            "n_genes_nan_from_deseq2": int(de_df.attrs.get("n_genes_nan_from_deseq2", 0)),
+            "neutral_fill": bool(de_df.attrs.get("pydeseq2_neutral_fill", True)),
+            "note": (
+                "Genes filtered by min_counts or marked NaN by DESeq2 independent filtering "
+                "appear as logFC=0, p_adj=1 and are not 'tested and non-significant'."
+            ),
+        }
+    n_memento_not_returned = (
+        int(de_df.attrs.get("n_genes_not_returned_by_memento", 0))
+        if (use_memento_de and hasattr(de_df, "attrs"))
+        else 0
+    )
 
     # Build clean results table (no velocity columns)
     cols = ["logFC", "p_val", "p_adj"]
@@ -1807,6 +1883,8 @@ def differential_expression(
             ),
             "memento_num_boot": memento_num_boot if use_memento_de else None,
             "memento_n_cpus": memento_n_cpus if use_memento_de else None,
+            "n_genes_not_returned_by_memento": n_memento_not_returned,
+            "pydeseq2": pydeseq2_diag or None,
             "n_jobs": n_jobs,
             "gene_type_filter": gene_type_filter,
         }
@@ -2018,16 +2096,25 @@ def restore_raw_counts(adata: Any, layer: str = "counts", inplace: bool = False)
             "Use the object before gene subsetting, or call store_raw_counts() again on the current object."
         )
 
-    # Additional guard when restoring from adata.raw (same dimension but possibly different order/names)
-    if (
-        source == "adata.raw"
-        and hasattr(adata.raw, "var_names")
-        and not np.array_equal(adata.raw.var_names, adata.var_names)
-    ):
-        raise ValueError(
-            "adata.raw has the same number of genes as current adata, but gene names/order differ. "
-            "Cannot restore into .X without explicit gene reindexing."
-        )
+    # Guard against same-dimension but permuted gene order (layers and adata.raw).
+    if source == "adata.raw" and hasattr(adata.raw, "var_names"):
+        if not np.array_equal(adata.raw.var_names, adata.var_names):
+            raise ValueError(
+                "adata.raw has the same number of genes as current adata, but gene names/order differ. "
+                "Cannot restore into .X without explicit gene reindexing."
+            )
+    elif source.startswith("layers"):
+        raw_gene_list = adata.uns.get("scatrans", {}).get("raw_gene_list")
+        if (
+            raw_gene_list is not None
+            and len(raw_gene_list) == adata.n_vars
+            and not np.array_equal(np.asarray(raw_gene_list), adata.var_names.to_numpy())
+        ):
+            raise ValueError(
+                f"Stored counts in {source} match n_vars but stored raw_gene_list order differs "
+                "from current adata.var_names. Cannot restore into .X without explicit gene "
+                "reindexing. Re-run store_raw_counts() on the current object."
+            )
 
     target = adata if inplace else adata.copy()
     target.X = raw
@@ -2169,16 +2256,10 @@ def filter_active_genes(
         p = preset.lower()
         if p in ("heuristic", "single_cell", "default"):
             preset_vals = {
-                "active_score_cutoff": 55.0,
-                "pval_cutoff": 0.05,
-                "velocity_residual_cutoff": 1.0,
-                "unspliced_excess_residual_cutoff": 1.0,
-                "logfc_cutoff": 0.35,
-                "active_score_fdr_cutoff": 0.25,
-                "unspliced_excess_fdr_cutoff": 0.05,
-                "effective_gamma_min": 0.05,
-                "effective_gamma_max": 1.0,
-                "delta_variance_min": None,
+                **HEURISTIC_FILTER_DEFAULTS,
+                "velocity_residual_cutoff": HEURISTIC_FILTER_DEFAULTS[
+                    "unspliced_excess_residual_cutoff"
+                ],
             }
         elif p in ("pseudobulk", "bulk"):
             preset_vals = {

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/tests/test_basic.py

@@ -728,7 +728,8 @@ def test_significant_requires_permutation_fdr(adata_basic):
     )
     assert "unspliced_excess_fdr" in allr_perm.columns
     if len(sig_perm) > 0:
-        assert (sig_perm["unspliced_excess_residual"] > 0).all()
+        assert (sig_perm["unspliced_excess_residual"] > 1.0).all()
+        assert (sig_perm["active_score"] >= 55.0).all()
         assert (sig_perm["unspliced_excess_fdr"] < 0.05).all()
 
 

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/tests/test_de_backends.py

@@ -76,6 +76,23 @@ def test_differential_expression_mixed_model(adata_mixed_small):
     assert "delta_variance" in res.columns
 
 
+@pytest.mark.slow
+def test_active_score_pb_x_layer_sentinel(adata_pb):
+    """pb_x_layer='X' must mean adata.X, matching differential_expression()."""
+    _, _, _ = scat.active_score(
+        adata_pb,
+        groupby="condition",
+        target_group="Disease",
+        reference_group="Control",
+        use_pseudobulk=True,
+        sample_col="sample",
+        pb_use_total_for_x=False,
+        pb_x_layer="X",
+        use_permutation=False,
+        show_plot=False,
+    )
+
+
 @pytest.mark.skipif(
     importlib.util.find_spec("pydeseq2") is None,
     reason="pydeseq2 not installed",

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/tests/test_de_edge_cases.py

@@ -5,6 +5,7 @@ import numpy as np
 import pandas as pd
 import pytest
 import scanpy as sc
+from statsmodels.stats.multitest import multipletests
 
 from scatrans._de import (
     _run_de_wrapper,
@@ -214,3 +215,14 @@ def test_run_permutation_test_use_fdr_based_on_n_success(monkeypatch):
     )
     assert use_fdr is False
     assert reason == "small_permutation_space"
+
+
+def test_memento_bh_excludes_nan_pvals_from_denominator():
+    """Defensive: NaN p-values must not inflate BH denominator if they ever appear."""
+    pvals_raw = pd.Series([0.01, 0.04, np.nan, 0.2], index=["g0", "g1", "g2", "g3"])
+    valid = pvals_raw.notna()
+    p_adj_clean = pd.Series(1.0, index=pvals_raw.index)
+    if valid.sum() > 0:
+        p_adj_clean.loc[valid] = multipletests(pvals_raw[valid].values, method="fdr_bh")[1]
+    p_adj_polluted = multipletests(pvals_raw.fillna(1.0).values, method="fdr_bh")[1]
+    assert p_adj_clean.loc["g0"] < p_adj_polluted[0]

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/tests/test_pl_extended.py

@@ -8,6 +8,7 @@ import matplotlib.pyplot as plt
 import numpy as np
 import pandas as pd
 import pytest
+import scanpy as sc
 
 import scatrans as scat
 
@@ -28,7 +29,14 @@ def test_volcano_plot_label_genes(results_df):
 
 
 def test_gseaplot_with_stored_curves():
-    ranked = pd.Series(np.linspace(1, 0, 20), index=[f"G{i}" for i in range(20)])
+    # gseapy-sorted ranking (descending scores)
+    ranking = pd.Series(
+        {f"G{i}": float(20 - i) for i in range(20)},
+        index=[f"G{i}" for i in range(20)],
+    ).sort_values(ascending=False)
+    # Caller passes a deliberately shuffled series (common when ranked from all_results)
+    shuffled = ranking.sample(frac=1, random_state=0)
+    res_curve = np.linspace(0.0, 1.0, len(ranking))
     gsea_res = pd.DataFrame(
         {
             "Term": ["TERM1"],
@@ -37,15 +45,45 @@ def test_gseaplot_with_stored_curves():
             "p.adjust": [0.05],
         }
     )
-    gsea_res.attrs["gsea_info"] = {
-        "rank_metric": ranked,
-        "enrichment_score": np.linspace(0, 0.5, 20),
-        "hit_indices": [2, 5, 9],
+    gsea_res.attrs["gsea_details"] = {
+        "TERM1": {
+            "RES": res_curve.tolist(),
+            "hits": [2, 5, 9],
+            "nes": 1.5,
+            "pval": 0.01,
+            "fdr": 0.05,
+        }
     }
-    fig, ax = scat.pl.gseaplot(ranked, gsea_res, term="TERM1", show=False)
+    gsea_res.attrs["ranking"] = ranking.to_dict()
+    fig, ax = scat.pl.gseaplot(shuffled, gsea_res, term="TERM1", show=False)
+    bar_heights = [p.get_height() for p in fig.axes[2].patches]
+    assert np.allclose(bar_heights, ranking.values)
+    assert len(fig.axes[0].lines[0].get_ydata()) == len(ranking)
     plt.close(fig)
 
 
+@pytest.mark.plot
+def test_active_genes_heatmap_default_show():
+    import anndata as ad
+
+    X = np.random.poisson(3, size=(20, 5)).astype(float)
+    adata = ad.AnnData(
+        X,
+        obs=pd.DataFrame({"group": ["A"] * 10 + ["B"] * 10}),
+        var=pd.DataFrame(index=[f"g{i}" for i in range(5)]),
+    )
+    sc.pp.normalize_total(adata, target_sum=1e4)
+    sc.pp.log1p(adata)
+    result = scat.pl.active_genes_heatmap(
+        adata, genes=adata.var_names[:3].tolist(), groupby="group", show=True
+    )
+    assert result is not None
+    fig, _ax = result
+    assert fig is not None
+    if isinstance(fig, plt.Figure):
+        plt.close(fig)
+
+
 def test_active_score_rankplot_empty():
     empty = pd.DataFrame(columns=["active_score", "logFC", "p_adj"])
     fig, ax = scat.pl.active_score_rankplot(empty, show=False)

{scatrans-0.9.8.dev1 → scatrans-0.9.8.dev2}/tests/test_tl_coverage.py

@@ -90,6 +90,17 @@ def test_store_and_restore_raw_counts(adata_de_only):
     assert "log1p" not in restored.uns
 
 
+def test_restore_raw_counts_rejects_permuted_gene_order(adata_de_only):
+    ad = adata_de_only.copy()
+    scat.store_raw_counts(ad, layer="counts", save_raw=False)
+    perm = list(range(ad.n_vars))
+    perm.reverse()
+    ad_perm = ad[:, perm].copy()
+    sc.pp.normalize_total(ad_perm, target_sum=1e4)
+    with pytest.raises(ValueError, match="raw_gene_list order differs"):
+        scat.restore_raw_counts(ad_perm, layer="counts", inplace=False)
+
+
 def test_reconcile_log1p_marker_detects_stale_metadata(adata_de_only):
     from scatrans._utils import _reconcile_log1p_marker, _x_looks_log_normalized
 
@@ -411,3 +422,40 @@ def test_prepare_log_normalized_expression_gap(adata_de_only):
     # Should return roughly same scale (no additional log1p applied)
     assert X_out.max() < 15
     assert np.allclose(X_out.mean(), mat.mean(), atol=1.0)
+
+
+def test_prior_weight_empirical_bayes_gradients_below_default():
+    """prior_weight must affect empirical_bayes below the default (0.5–5.0 range)."""
+    from scatrans._velocity import _apply_empirical_bayes_gamma, _estimate_eb_prior_from_reference
+
+    rng = np.random.default_rng(0)
+    u_r = rng.exponential(2.0, 400)
+    s_r = rng.exponential(2.0, 400)
+
+    _, meta_lo = _estimate_eb_prior_from_reference(u_r, s_r, prior_weight=0.5)
+    _, meta_hi = _estimate_eb_prior_from_reference(u_r, s_r, prior_weight=5.0)
+    assert meta_lo["count_pseudocount"] == pytest.approx(0.1)
+    assert meta_hi["count_pseudocount"] == pytest.approx(1.0)
+    assert meta_lo["count_pseudocount"] != meta_hi["count_pseudocount"]
+
+    _, w_lo, _ = _apply_empirical_bayes_gamma(u_r, s_r, meta_lo["eb_prior"], n_ref=20.0)
+    _, w_hi, _ = _apply_empirical_bayes_gamma(u_r, s_r, meta_hi["eb_prior"], n_ref=20.0)
+    assert float(np.nanmean(w_lo)) != float(np.nanmean(w_hi))
+
+
+@pytest.mark.slow
+def test_significant_matches_filter_heuristic_preset(adata_basic):
+    """Built-in significant list should match filter_active_genes(preset='heuristic')."""
+    _, sig, allr = scat.active_score(
+        adata_basic,
+        groupby="condition",
+        target_group="Disease",
+        reference_group="Control",
+        use_permutation=True,
+        n_perm=30,
+        random_seed=1,
+        n_jobs=1,
+        show_plot=False,
+    )
+    filt = scat.filter_active_genes(allr, preset="heuristic")
+    assert sig.index.tolist() == filt.index.tolist()