blockperm 0.1.0__py3-none-any.whl

blockperm-0.1.0.dist-info/METADATA

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: blockperm
+Version: 0.1.0
+Summary: Block permutation test for cross-validated decoding when labels follow recording blocks
+Author: Dongjin Choi
+License-Expression: MIT
+Project-URL: Repository, https://github.com/jinchoi7651507/block-permutation-decoding
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.0
+Requires-Dist: scikit-learn>=1.4
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# blockperm: a block permutation test for cross-validated decoding
+
+`block_permutation_test_score` tests a cross-validated decoding score when the labels follow recording blocks (for example conditions presented in blocks of consecutive trials) and training and test trials may come from the same blocks. It has the call pattern of scikit-learn's `permutation_test_score`:
+
+```python
+from blockperm import block_permutation_test_score
+
+score, null_scores, p = block_permutation_test_score(
+    pipeline, X, y, blocks=block_id, cv=KFold(5, shuffle=True, random_state=0), n_permutations=1000, random_state=0)
+```
+
+## What it does
+- Each permutation permutes the labels of whole blocks between blocks, so the labels of training and test trials move together and every block keeps its trials.
+- The whole cross-validation is rerun for every permutation, including every step fitted to the labels. Put scaling, feature or neuron selection and the model in one scikit-learn `Pipeline`.
+- `strata` restricts the permutation to blocks within the same stratum (for example within a run or a session).
+- `draw` replaces the permutation with a user-given generator of block-label vectors, for example pseudosessions drawn from the task's own block generator.
+- With few blocks the function enumerates every distinct arrangement (`exact="auto"`) and warns about the smallest attainable p value. With two blocks per class there are six arrangements, but for a two-class decoder an arrangement and its class swap give the same score, so p cannot fall below about one in three.
+
+## When it is valid
+- The enumerated null is exact, and the sampled p value valid, when the observed assignment was drawn uniformly from the set the function permutes over: all arrangements of the block labels, or all arrangements within `strata` if the design randomized within strata (for example within runs). Permuting across a unit the design did not randomize over can reject too often. For a constrained randomization (counterbalancing, limits per run), pass the design's own generator as `draw`.
+- With a fixed block order, the test assumes that blocks are exchangeable under the null. Slow drift that follows the order can break this assumption, and then the test can reject or be conservative. A design with random block order, or several blocks per condition in random order, avoids the problem.
+- With `GroupKFold` and blocks of equal size, the block-to-fold map depends on the tie order of `numpy.argsort`, which differs between platforms (arm64 and x86 gave different maps for 24 blocks with the same versions). For a result that reruns identically elsewhere, pass explicit folds (a list of train/test index pairs) or a shuffled `GroupKFold` with a seed (scikit-learn 1.6 or later), and record the folds.
+
+## Why not the defaults
+In the analyses this function accompanies, three rules that toolboxes run were tested on labels with no information, assigned to recording blocks, with training and test trials sharing blocks:
+- trial-level permutation with the cross-validation rerun (scikit-learn `permutation_test_score`, MVPA-Light);
+- trial-level permutation within runs (the CoSMoMVPA tutorial rule);
+- permutation of training labels only, with test labels kept (the PyMVPA tutorial rule).
+
+All three rejected far more often than 5%. Permuting block labels and rerunning the whole cross-validation stayed near 5% in random block designs. With a fixed block order the result depends on the data. In the simulated fixed orders the test was conservative (0–2.4% with shuffled 5-fold). In resting EEG with fixed orders aligned with slow drift it rejected often (first half against second half in most recordings, a pairs-of-blocks order in 29%), and it stayed near 5% for an order orthogonal to a linear trend. There the blocks were not exchangeable (label = time; see "When it is valid"). With labels that vary trial by trial, the first two rules stayed near 5%, and the training-label rule was somewhat above it (10–12%); the large excess arises when labels follow blocks. `examples/fake_block_labels.py` shows the effect on a synthetic drifting recording.
+
+## One-line changes per tool
+- **scikit-learn:** call `block_permutation_test_score` with `blocks=`. Passing `groups=` to `permutation_test_score` restricts the shuffle to within groups; with labels constant within blocks, every permutation then returns the observed labels.
+- **CoSMoMVPA / PyMVPA:** permute chunk-level labels across chunks (one label per chunk, all trials of a chunk together) within the unit the design randomized over (within runs if blocks were randomized within runs; `strata=` here), and rerun the cross-validation.
+- **PyMVPA training-label permutation:** permute the labels of all trials, training and test, block by block.
+
+## Install and test
+Python 3.12 with numpy, scipy and scikit-learn (tested with numpy 2.5.4, scipy 1.18.1, scikit-learn 1.7.2); tested on macOS (arm64). No special hardware is needed; one CPU core is enough.
+
+```sh
+pip install blockperm           # from PyPI; or, in this folder, pip install ".[test]" (with pytest), or copy blockperm.py into a project
+python -m pytest tests          # a few minutes on one core; the warnings it prints are expected
+python examples/fake_block_labels.py
+```
+
+On a laptop (Apple arm64, 2026-11-10) the install took about 15 s with a network connection, the tests about 1.5 min and the example about 1.5 min. In this folder alone the tests end with "8 passed, 4 skipped" (the equivalence tests need the reference file named below); inside the paper's repository they end with "12 passed". The example prints:
+
+```
+labels with no information, 40 recordings, p < 0.05:
+  permutation_test_score (trial labels permuted): 14 (35%)
+  block_permutation_test_score (block labels permuted): 1 (2%)
+```
+
+The test `test_equivalence_with_reference` checks that, with `exact=False`, the same `random_state` Generator and the same folds, this function returns the same observed score, null scores and p value (except that it returns no null when the observed score is undefined) as the reference implementation used in the analyses (`analyses/step2-comparison/code/sim.py` in the paper's repository; another location can be given with `REFERENCE_SIM_DIR`), for shuffled 5-fold, re-split stratified 5-fold, GroupKFold by block and a user `draw`; it is skipped when that file is not present. With the default `exact="auto"`, designs with no more arrangements than `n_permutations` are enumerated instead (the null then has one score per arrangement).
+
+## Tutorial, MATLAB and Octave
+- `docs/TUTORIAL.md`: a step-by-step guide (folds, strata, pseudosessions, few blocks, fixed block orders, checking a pipeline with labels that carry no information).
+- `matlab/`: a MATLAB and Octave port, `block_permutation_test.m`, which takes a function returning the cross-validated score for a label vector. This Python function is the reference; the port is checked against it in `matlab/crosscheck/` (`matlab/README.md`).
+- On PyPI: `pip install blockperm` (version 0.1.0, the version used in the paper).
+
+## Not included
+No `n_jobs` and no fit parameters (unlike `permutation_test_score`); the permutations run one after another. License: MIT (`LICENSE`).

blockperm-0.1.0.dist-info/RECORD

@@ -0,0 +1,6 @@
+blockperm.py,sha256=Sl-ACeHUnVvx4wy0egjyX3RecSvMmbaQRW3RwggfSgg,11870
+blockperm-0.1.0.dist-info/licenses/LICENSE,sha256=zjTbufg5fIhUxUGFbaqKkdubSADuzu6GUXpjnZrIe5Q,1069
+blockperm-0.1.0.dist-info/METADATA,sha256=Jt6xm_6nRI--334q7TjD73MPPh7yjZWpZa7KkOYa3eQ,7762
+blockperm-0.1.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+blockperm-0.1.0.dist-info/top_level.txt,sha256=K9UXz8iJ__MAVlrXiMW63JX8UR3VCPco-8SyxkrBW6o,10
+blockperm-0.1.0.dist-info/RECORD,,

blockperm-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

blockperm-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dongjin Choi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

blockperm-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+blockperm

blockperm.py

@@ -0,0 +1,230 @@
+"""Block permutation test for cross-validated decoding when labels follow recording blocks.
+
+`block_permutation_test_score` has the call pattern of scikit-learn's `permutation_test_score`. It differs in what
+it permutes: the labels of whole blocks are permuted between blocks (training and held-out trials together), and
+the whole cross-validation, including every step fitted to the labels, is rerun for every permutation. A label
+vector can instead be drawn from a user-given generator (for example pseudosessions drawn from the task's own
+block generator).
+
+This is the block permutation rule of the accompanying analyses. The enumerated null is exact, and the sampled p
+value valid, when the observed assignment was drawn uniformly from the set the function permutes over: all
+arrangements of the block labels, or all arrangements within strata if the design randomised within strata. For a
+constrained randomisation (counterbalancing, limits per run), pass the design's own generator as `draw`. For a
+fixed block order the test rests on an exchangeability model, not on the randomisation.
+"""
+from __future__ import annotations
+
+import math
+import numbers
+import warnings
+from collections import Counter
+
+import numpy as np
+from sklearn.base import clone, is_classifier
+from sklearn.metrics import check_scoring
+from sklearn.model_selection import check_cv
+
+__all__ = ["block_permutation_test_score", "block_labels_of", "n_distinct_arrangements"]
+
+
+def block_labels_of(y, blocks):
+    """Per-block labels (in the order of np.unique(blocks)) and the position of each trial's block.
+
+    Raises ValueError if a block holds more than one label."""
+    y = np.asarray(y)
+    blocks = np.asarray(blocks)
+    if y.shape[0] != blocks.shape[0]:
+        raise ValueError("y and blocks must have the same length")
+    ub = np.unique(blocks)
+    pos = np.searchsorted(ub, blocks)
+    lab = []
+    for i in range(len(ub)):
+        v = np.unique(y[pos == i])
+        if len(v) != 1:
+            raise ValueError("labels must be constant within blocks (block %r holds %d labels)" % (ub[i], len(v)))
+        lab.append(v[0])
+    return np.array(lab), pos
+
+
+def n_distinct_arrangements(block_labels, strata=None):
+    """Number of distinct block-label vectors reachable by permuting labels between blocks (within strata).
+    Both arguments are per block (one value per block, in the order of np.unique(blocks))."""
+    block_labels = np.asarray(block_labels)
+    strata = None if strata is None else np.asarray(strata)
+    groups = [np.arange(len(block_labels))] if strata is None else [np.where(strata == s)[0] for s in np.unique(strata)]
+    n = 1
+    for g in groups:
+        c = Counter(block_labels[g].tolist())
+        k = math.factorial(len(g))
+        for v in c.values():
+            k //= math.factorial(v)
+        n *= k
+    return n
+
+
+def _cv_score(estimator, X, y, splits, scorer, error_score):
+    scores = []
+    for tr, te in splits:
+        if is_classifier(estimator) and len(np.unique(y[tr])) < 2:
+            return error_score
+        m = clone(estimator).fit(X[tr], y[tr])
+        scores.append(scorer(m, X[te], y[te]))
+    return float(np.mean(scores))
+
+
+def _splits(cv, X, y, groups):
+    with warnings.catch_warnings():            # splitters that ignore groups (KFold, StratifiedKFold) warn
+        warnings.filterwarnings("ignore", message="The groups parameter is ignored")
+        return list(cv.split(X, y, groups))
+
+
+def block_permutation_test_score(estimator, X, y, *, blocks, cv=None, n_permutations=1000, scoring=None,
+                                 random_state=None, strata=None, draw=None, groups="blocks", exact="auto",
+                                 error_score=np.nan):
+    """Evaluate the significance of a cross-validated score by permuting labels between blocks.
+
+    Parameters
+    ----------
+    estimator : scikit-learn estimator (a Pipeline with every label-fitted step: scaling, selection, model)
+    X : array (n_trials, n_features)
+    y : array (n_trials,), constant within each block
+    blocks : array (n_trials,), block identity of each trial (a recording block of consecutive trials)
+    cv : as in scikit-learn (int, splitter or iterable); default 5-fold. A splitter is re-applied to every
+        permuted label vector (as scikit-learn does), so label-stratified folds follow the permuted labels.
+    n_permutations : number of permutations (ignored when the exact enumeration is used)
+    scoring : as in scikit-learn; default the estimator's score (accuracy for classifiers). The score is the mean
+        over folds, as in `permutation_test_score`.
+    random_state : int, numpy Generator, RandomState or None (passed to numpy.random.default_rng unless a
+        Generator); the permutations use this stream only, so the same Generator and folds reproduce the same null
+    strata : optional array (n_trials,), constant within blocks: labels are permuted only between blocks of the
+        same stratum. Use the unit the design randomised over (for example within runs if blocks were randomised
+        within runs); permuting across that unit can reject too often.
+    draw : optional callable(rng) -> per-block label vector (order of np.unique(blocks)); replaces the
+        permutation, for example to draw pseudosessions from the task's generator. `strata` and `exact` are
+        ignored when `draw` is given.
+    groups : "blocks" (default) passes the block identity to `cv.split` as `groups`, so GroupKFold-type splitters
+        hold out whole blocks; None passes nothing; an array is passed as given.
+    exact : "auto" enumerates all distinct block-label arrangements when there are no more of them than
+        `n_permutations` (the null then has one score per arrangement, the observed one included, and its length
+        is the number of arrangements, not `n_permutations`); True forces the enumeration (at most 100,000
+        arrangements); False always samples with replacement.
+    error_score : number used as the score of a cross-validation with a one-class training fold (classifiers).
+        Other fit errors are raised. NaN null scores count as exceedances (conservative); an undefined observed
+        score returns p = NaN with a warning.
+
+    Returns
+    -------
+    score : float, observed cross-validated score
+    permutation_scores : array, scores under the null (empty when the observed score is undefined; the
+        reference implementation returned its full null in that case, with p NaN as here)
+    pvalue : float, (1 + #{null >= score} + #{null NaN}) / (n_null + 1) when sampling; when enumerating,
+        (1 + #{other arrangements with score >= observed or NaN}) / (number of arrangements), the observed
+        arrangement counted once with its observed score (also when the cross-validation is not deterministic).
+    """
+    X = np.asarray(X)
+    y = np.asarray(y)
+    blocks = np.asarray(blocks)
+    if X.shape[0] != y.shape[0] or blocks.shape[0] != y.shape[0]:
+        raise ValueError("X, y and blocks must have the same number of trials")
+    if not (isinstance(exact, (bool, np.bool_)) or (isinstance(exact, str) and exact == "auto")):
+        raise ValueError('exact must be "auto", True or False')
+    exact = exact if isinstance(exact, str) else bool(exact)
+    if not isinstance(n_permutations, numbers.Integral) or isinstance(n_permutations, bool) or n_permutations < 1:
+        raise ValueError("n_permutations must be an integer of at least 1")
+    if isinstance(groups, str) and groups != "blocks":
+        raise ValueError('groups must be "blocks", None or an array with one value per trial')
+    if not isinstance(error_score, numbers.Real) or isinstance(error_score, (bool, np.bool_)):
+        raise ValueError("error_score must be a number (NaN by default)")
+    lab, pos = block_labels_of(y, blocks)
+    rng = random_state if isinstance(random_state, np.random.Generator) else np.random.default_rng(random_state)
+    cv = check_cv(cv, y, classifier=is_classifier(estimator))
+    scorer = check_scoring(estimator, scoring=scoring)
+    g = blocks if isinstance(groups, str) else groups
+    if strata is not None:
+        if np.asarray(strata).shape[0] != y.shape[0]:
+            raise ValueError("strata must have one value per trial")
+        st_lab, _ = block_labels_of(np.asarray(strata), blocks)
+    else:
+        st_lab = None
+
+    score = _cv_score(estimator, X, y, _splits(cv, X, y, g), scorer, error_score)
+    if not np.isfinite(score):
+        warnings.warn("the observed score is undefined (a training fold with one class, or a scorer that returned "
+                      "NaN); p is NaN and no null was computed", stacklevel=2)
+        return score, np.array([], float), float("nan")
+
+    n_arr = n_distinct_arrangements(lab, st_lab) if draw is None else None
+    use_exact = draw is None and (exact is True or (exact == "auto" and n_arr <= n_permutations))
+    if draw is None and n_arr < 20:
+        warnings.warn("only %d distinct block-label arrangements: with exact enumeration p cannot fall below %.3g, "
+                      "and not below %.3g when swapping the two classes leaves the score unchanged (a refit two-class "
+                      "classifier with balanced blocks)%s"
+                      % (n_arr, 1.0 / n_arr, 2.0 / n_arr, "" if use_exact else "; consider exact=True"), stacklevel=2)
+
+    if use_exact:
+        if n_arr > 100_000:
+            raise ValueError("too many arrangements to enumerate (%d)" % n_arr)
+        arrangements = _enumerate(lab, st_lab)
+        obs = next(i for i, a in enumerate(arrangements) if np.array_equal(a, lab))
+        null = np.array([score if i == obs else
+                         _cv_score(estimator, X, a[pos], _splits(cv, X, a[pos], g), scorer, error_score)
+                         for i, a in enumerate(arrangements)])
+        others = np.delete(null, obs)
+        ge = 1 + np.sum(others[~np.isnan(others)] >= score - 1e-12) + np.isnan(others).sum()
+        return score, null, float(ge / len(null))
+
+    null = []
+    for _ in range(n_permutations):
+        if draw is not None:
+            lp = np.asarray(draw(rng))
+            if lp.shape != lab.shape:
+                raise ValueError("draw must return one label per block")
+        elif st_lab is None:
+            lp = rng.permutation(lab)
+        else:
+            lp = lab.copy()
+            for s in np.unique(st_lab):
+                i = np.where(st_lab == s)[0]
+                lp[i] = lab[rng.permutation(i)]
+        yp = lp[pos]
+        null.append(_cv_score(estimator, X, yp, _splits(cv, X, yp, g), scorer, error_score))
+    null = np.array(null, float)
+    p = (1 + np.sum(null[~np.isnan(null)] >= score - 1e-12) + np.isnan(null).sum()) / (len(null) + 1)
+    return score, null, float(p)
+
+
+def _multiset_permutations(items):
+    """Distinct orderings of a list with repeated values, each once (no n! intermediate list)."""
+    counts = Counter(items)
+    keys = sorted(counts)
+    n = len(items)
+    cur = []
+
+    def rec():
+        if len(cur) == n:
+            yield tuple(cur)
+            return
+        for k in keys:
+            if counts[k]:
+                counts[k] -= 1
+                cur.append(k)
+                yield from rec()
+                cur.pop()
+                counts[k] += 1
+    yield from rec()
+
+
+def _enumerate(lab, st_lab):
+    """All distinct block-label vectors reachable by permuting within strata (observed one included)."""
+    idx_groups = [np.arange(len(lab))] if st_lab is None else [np.where(st_lab == s)[0] for s in np.unique(st_lab)]
+    per_group = [list(_multiset_permutations(lab[i].tolist())) for i in idx_groups]
+    out = []
+    def rec(k, cur):
+        if k == len(idx_groups):
+            out.append(cur.copy())
+            return
+        for arr in per_group[k]:
+            cur[idx_groups[k]] = arr
+            rec(k + 1, cur)
+    rec(0, lab.copy())
+    return out