dfc-kit 1.0.0__tar.gz → 1.0.2__tar.gz

dfc_kit-1.0.2/CHANGELOG.md

@@ -0,0 +1,75 @@
+# Changelog
+
+## 1.0.2 - 2026-09-11
+
+- Added exact-feature selection for in-memory and disk-backed state models,
+  including clustering on a fixed NBS component's edges.
+- Added conditional Granger predictability with censor-bounded lagged samples
+  and VAR diagnostics. SciPy is loaded only when those diagnostics are used,
+  preserving the NumPy-only base installation.
+- FeatureStore summaries can now aggregate a named feature set within each
+  sample before computing acquisition-level mean, variance, standard deviation,
+  and extrema. The CLI exposes `--feature-keys` and `--feature-mean NAME` for
+  fixed-network summaries, including sparse NBS components.
+
+## 1.0.1
+
+- Fixed-length MI/CMI now shares one process pool across acquisition, window
+  length, and draw combinations. Bounded task submission avoids materializing
+  the complete window grid, and ordered results preserve the serial samples,
+  jitter, pair estimates, and artifact rows. `jobs=1` remains serial.
+- Reused full-window and half-window PCA fits across requested ranks, avoiding
+  repeated decompositions while preserving covariance endpoint definitions.
+- Shared paired-test, HMM decoding, and feature-building internals; removed
+  redundant storage forwarders and derived-score constructor arguments.
+- Added `static-fc-endpoints` to compute one whole-acquisition Fisher-z edge
+  vector from each XCP-D run's retained frames.
+- Added `infer-paired-nbs`, a machine-readable endpoint adapter for paired NBS
+  with component-level FWE, explicit thresholds, tails, component statistics,
+  and optional participant-level nuisance variables.
+- Added `lowrank-endpoints` for direct XCP-D-to-JSON summaries of effective
+  rank, eigenvalue concentration, held-out reconstruction, and split/adjacent
+  subspace similarity.
+- Added segment-weighted all-pair FC-pattern and low-rank subspace similarity,
+  together with adjacency excess, so configuration homogeneity can be separated
+  from order-specific similarity.
+- Added `window-pattern-endpoints` for direct XCP-D-to-JSON summaries of
+  all-pair, adjacent, and adjacency-excess whole-edge pattern similarity.
+- Added deterministic pass-level early stopping to out-of-core MiniBatchKMeans.
+- Reported the selected initialization's completed passes and all initialization
+  pass counts without changing the fitted-model artifact format.
+- Clarified that streaming `max_iter` is a maximum number of complete store
+  passes; `convergence_tol=0` retains fixed-pass fitting.
+- Added an explicit equal-run mean for paired inference with repeated
+  acquisitions. Runs remain separate through feature and state summarization,
+  and JSON/TSV results report the acquisition-count range in each condition.
+- Paired named-endpoint inference can now select an explicit endpoint subset,
+  allowing one source artifact to be tested as several scientifically declared
+  FDR families without changing each endpoint's permutation or bootstrap draw.
+- Added `infer-independent-endpoints` for two independent named-endpoint
+  cohorts, with optional numeric covariates, HC3 group effects, Hedges' g,
+  full and group-incremental R2, and declared-family FDR.
+- XCP-D discovery now enters session directories linked into an otherwise
+  ordinary BIDS subject hierarchy without recursively following arbitrary
+  directory symlinks.
+
+## 1.0.0 - 2026-08-24
+
+- Moved MI/CMI estimators and fixed-window orchestration into
+  `dfckit.information`.
+- Unified random sampling and frozen schedules through one fixed-window
+  estimation path.
+- Simplified state-count selection to subject-disjoint held-out score
+  comparison with compact JSON output.
+- Removed cryptographic data/model identities, workflow-directory loaders,
+  nested checkpointing, and legacy compatibility layers from the development
+  codebase.
+- Kept model, prediction, alignment, and information artifacts pickle-free and
+  validated by explicit shapes, parameters, feature keys, and participant
+  metadata.
+- ETS and MTD share the same segment-safe instantaneous-edge framework while
+  retaining their distinct sample generators.
+- Paired NBS, HC3 inference, information estimates, and all censor-gap-safe
+  connectivity methods retain their numerical APIs.
+- FeatureStore summaries support streaming acquisition-level mean, population
+  variance, standard deviation, minimum, and maximum endpoints.

{dfc_kit-1.0.0 → dfc_kit-1.0.2}/CITATION.cff

@@ -4,6 +4,6 @@ title: "dfc-kit: Censor-aware dynamic functional connectivity tools"
 type: software
 authors:
   - name: "dfc-kit contributors"
-version: 1.0.0
-date-released: 2026-08-24
+version: 1.0.2
+date-released: 2026-09-11
 license: BSD-3-Clause

{dfc_kit-1.0.0 → dfc_kit-1.0.2}/MANIFEST.in

@@ -1,11 +1,8 @@
 include CHANGELOG.md
 include CITATION.cff
 include CONTRIBUTING.md
-include ENGINEERING_PLAN.md
 include LICENSE
 include README.md
-include ROADMAP.md
-include SECURITY.md
 include mkdocs.yml
 recursive-include benchmarks *.md *.py
 recursive-include docs *.md

{dfc_kit-1.0.0 → dfc_kit-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dfc-kit
-Version: 1.0.0
+Version: 1.0.2
 Summary: Censor-aware tools for dynamic functional connectivity analysis
 License-Expression: BSD-3-Clause
 Project-URL: Homepage, https://github.com/yidao9518/dfc-kit
@@ -73,14 +73,15 @@ for larger datasets, with matching Python and command-line interfaces.
   loading, acquisition identity, and censor-bounded sequences. Censored time
   points retain their original frame indices, and temporal operations are
   evaluated separately within contiguous retained segments.
-- **Connectivity:** weighted sliding-window FC, instantaneous ETS/MTD edges,
-  LEiDA, low-rank covariance geometry, and fixed-length MI/CMI.
+- **Connectivity:** whole-acquisition and weighted sliding-window FC,
+  whole-edge all-pair and order-specific pattern similarity, instantaneous
+  ETS/MTD edges, LEiDA, low-rank covariance geometry, and fixed-length MI/CMI.
 - **Connectivity and state analysis:** partition-based graph metrics, CAP, KMeans,
   Gaussian HMMs, state alignment, occupancy/dwell/transition summaries, and
   selection of the number of states using held-out participants.
 - **Inference:** paired sign-flips, bootstrap intervals, HC3 models,
-  declared-family FDR, generic paired endpoint inference, paired NBS, and
-  within-subject motion matching.
+  declared-family FDR, generic paired endpoint inference, independent-group
+  endpoint models, paired NBS, and within-subject motion matching.
 - **Large-dataset workflows:** chunked, memory-mapped FeatureStores and
   batch-wise fitting for MiniBatch KMeans and Incremental PCA.
 - **Portable results:** models and held-out predictions stored as JSON and
@@ -152,16 +153,26 @@ dfc-kit --help
 dfc-kit inspect-xcpd --help
 dfc-kit build-store --help
 dfc-kit fixed-information --help
+dfc-kit lowrank-endpoints --help
+dfc-kit window-pattern-endpoints --help
+dfc-kit static-fc-endpoints --help
 dfc-kit describe-states --help
 dfc-kit infer-state-metrics --help
 dfc-kit summarize-store --help
 dfc-kit summarize-information --help
 dfc-kit infer-paired-endpoints --help
+dfc-kit infer-independent-endpoints --help
+dfc-kit infer-paired-nbs --help
 ```
 
 See [Command-line workflows](docs/cli.md) for complete examples and arguments,
 including fixed-length MI/CMI artifacts and frozen-window replay.
 
+For two independent endpoint cohorts, `infer-independent-endpoints` fits the
+declared group contrast with HC3 standard errors, optional numeric covariates,
+and family-level Benjamini-Hochberg correction. See [Independent group
+inference](docs/group_inference.md).
+
 ## Documentation
 
 - [Documentation home](docs/index.md)

{dfc_kit-1.0.0 → dfc_kit-1.0.2}/README.md

@@ -28,14 +28,15 @@ for larger datasets, with matching Python and command-line interfaces.
   loading, acquisition identity, and censor-bounded sequences. Censored time
   points retain their original frame indices, and temporal operations are
   evaluated separately within contiguous retained segments.
-- **Connectivity:** weighted sliding-window FC, instantaneous ETS/MTD edges,
-  LEiDA, low-rank covariance geometry, and fixed-length MI/CMI.
+- **Connectivity:** whole-acquisition and weighted sliding-window FC,
+  whole-edge all-pair and order-specific pattern similarity, instantaneous
+  ETS/MTD edges, LEiDA, low-rank covariance geometry, and fixed-length MI/CMI.
 - **Connectivity and state analysis:** partition-based graph metrics, CAP, KMeans,
   Gaussian HMMs, state alignment, occupancy/dwell/transition summaries, and
   selection of the number of states using held-out participants.
 - **Inference:** paired sign-flips, bootstrap intervals, HC3 models,
-  declared-family FDR, generic paired endpoint inference, paired NBS, and
-  within-subject motion matching.
+  declared-family FDR, generic paired endpoint inference, independent-group
+  endpoint models, paired NBS, and within-subject motion matching.
 - **Large-dataset workflows:** chunked, memory-mapped FeatureStores and
   batch-wise fitting for MiniBatch KMeans and Incremental PCA.
 - **Portable results:** models and held-out predictions stored as JSON and
@@ -107,16 +108,26 @@ dfc-kit --help
 dfc-kit inspect-xcpd --help
 dfc-kit build-store --help
 dfc-kit fixed-information --help
+dfc-kit lowrank-endpoints --help
+dfc-kit window-pattern-endpoints --help
+dfc-kit static-fc-endpoints --help
 dfc-kit describe-states --help
 dfc-kit infer-state-metrics --help
 dfc-kit summarize-store --help
 dfc-kit summarize-information --help
 dfc-kit infer-paired-endpoints --help
+dfc-kit infer-independent-endpoints --help
+dfc-kit infer-paired-nbs --help
 ```
 
 See [Command-line workflows](docs/cli.md) for complete examples and arguments,
 including fixed-length MI/CMI artifacts and frozen-window replay.
 
+For two independent endpoint cohorts, `infer-independent-endpoints` fits the
+declared group contrast with HC3 standard errors, optional numeric covariates,
+and family-level Benjamini-Hochberg correction. See [Independent group
+inference](docs/group_inference.md).
+
 ## Documentation
 
 - [Documentation home](docs/index.md)

{dfc_kit-1.0.0 → dfc_kit-1.0.2}/benchmarks/README.md

@@ -1,5 +1,35 @@
 # Synthetic performance benchmarks
 
+## Fixed-length MI/CMI
+
+```bash
+OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 PYTHONPATH=src \
+  python benchmarks/profile_information.py --jobs 1 2 4 8 --runs 8 --draws 25
+```
+
+This benchmark generates 36 synthetic ROI time series with a censor gap and
+estimates all 12-by-4 MI/CMI pairs at five window lengths. It reports end-to-end
+time, including process startup and array transfer, and requires exact agreement
+with serial output for every numeric array and metadata field. Worker count is
+an execution setting, so changing it must not change the scientific result.
+For short batches, process startup can outweigh the gain from parallel work.
+
+Cloud reference run on 2026-09-05 (Python 3.10.14, NumPy 1.26.4,
+SciPy 1.13.0, 8 runs, 25 draws per length, 1,000 total windows):
+
+| Processes | Elapsed time | Speedup over serial | Exact output match |
+|---:|---:|---:|---|
+| 1 | 184.122 s | 1.00 | Yes |
+| 4 | 47.186 s | 3.90 | Yes |
+| 16 | 12.207 s | 15.08 | Yes |
+| 40 | 5.230 s | 35.20 | Yes |
+
+These timings include worker startup. All arrays and metadata matched the serial
+reference; the speedup describes this synthetic workload, not a guaranteed rate
+for arbitrary ROI counts, sample lengths, or machine allocations.
+
+## Feature and state methods
+
 Run each method in a separate process so `ru_maxrss` has one clear scope:
 
 ```bash

dfc_kit-1.0.2/benchmarks/profile_information.py

@@ -0,0 +1,130 @@
+"""Compare serial and parallel MI/CMI on synthetic ROI36-sized input.
+
+Run from the project root, for example::
+
+    python benchmarks/profile_information.py --jobs 1 2 4 8
+
+The timings include process startup and data transfer. Every parallel result is
+checked against a serial reference, including window indices and all pair values.
+This script never loads participant data or writes analysis artifacts.
+"""
+
+from __future__ import annotations
+
+import argparse
+import dataclasses
+import json
+import platform
+import time
+
+import numpy as np
+
+from dfckit import TimeSeriesDataset, TimeSeriesRun, __version__
+from dfckit.information import (
+    FixedInformationArtifact,
+    InformationGroups,
+    compute_fixed_information,
+    knn_cmi,
+)
+
+
+def synthetic_dataset(runs: int, frames: int, seed: int) -> TimeSeriesDataset:
+    rng = np.random.default_rng(seed)
+    roi_names = tuple(f"roi-{index:02d}" for index in range(36))
+    acquisitions = []
+    for index in range(runs):
+        values = rng.normal(size=(frames, 36))
+        values[:, 12:16] += 0.4 * values[:, :4]
+        split = frames // 2
+        original_indices = np.r_[np.arange(split), np.arange(split + 5, frames + 5)]
+        acquisitions.append(
+            TimeSeriesRun(
+                values=values,
+                original_indices=original_indices,
+                roi_names=roi_names,
+                subject=f"synthetic-{index:03d}",
+                session="benchmark",
+                acquisition_id=f"synthetic-{index:03d}_benchmark",
+            )
+        )
+    return TimeSeriesDataset(tuple(acquisitions))
+
+
+def assert_exact(actual: FixedInformationArtifact, expected: FixedInformationArtifact) -> None:
+    for field in dataclasses.fields(expected):
+        value = getattr(expected, field.name)
+        if isinstance(value, np.ndarray):
+            np.testing.assert_array_equal(getattr(actual, field.name), value, err_msg=field.name)
+        else:
+            if getattr(actual, field.name) != value:
+                raise AssertionError(f"artifact metadata differs: {field.name}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--jobs", type=int, nargs="+", default=[1, 2, 4, 8])
+    parser.add_argument("--runs", type=int, default=4)
+    parser.add_argument("--frames", type=int, default=800)
+    parser.add_argument("--lengths", type=int, nargs="+", default=[120, 180, 240, 300, 360])
+    parser.add_argument("--draws", type=int, default=5)
+    parser.add_argument("--seed", type=int, default=20260905)
+    args = parser.parse_args()
+    if min(args.jobs) < 1 or args.runs < 1 or args.frames < 12 or args.draws < 1:
+        parser.error("jobs, runs, and draws must be positive; frames must be at least 12")
+    if not args.lengths or min(args.lengths) <= 5 or max(args.lengths) > args.frames // 2:
+        parser.error("every length must be between 6 and the retained half-run length")
+
+    dataset = synthetic_dataset(args.runs, args.frames, args.seed)
+    names = dataset.roi_names
+    groups = InformationGroups(names[:12], names[12:16], names[16:26])
+    # Import SciPy before timing the reference, while retaining process startup
+    # and child imports in the end-to-end parallel timings.
+    knn_cmi(*(dataset.runs[0].values[:16, index] for index in range(3)))
+    job_counts = tuple(dict.fromkeys((1, *args.jobs)))
+    reference = None
+    reference_seconds = None
+    timings = []
+    for jobs in job_counts:
+        start = time.perf_counter()
+        result = compute_fixed_information(
+            dataset,
+            groups,
+            lengths=args.lengths,
+            draws=args.draws,
+            sample_seed=args.seed,
+            jobs=jobs,
+        )
+        elapsed = time.perf_counter() - start
+        if reference is None:
+            reference = result
+            reference_seconds = elapsed
+        assert_exact(result, reference)
+        timings.append(
+            {
+                "jobs": jobs,
+                "elapsed_seconds": elapsed,
+                "speedup_over_serial": reference_seconds / elapsed,
+                "all_arrays_and_metadata_exact": True,
+                "analyzed_windows": result.n_draws,
+            }
+        )
+        print(json.dumps(timings[-1]), flush=True)
+    print(
+        json.dumps(
+            {
+                "dfc_kit_version": __version__,
+                "python_version": platform.python_version(),
+                "input": vars(args),
+                "left_rois": 12,
+                "right_rois": 4,
+                "conditioning_rois": 10,
+                "timing_includes_process_startup": True,
+                "timings": timings,
+            },
+            indent=2,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

{dfc_kit-1.0.0 → dfc_kit-1.0.2}/docs/api.md

@@ -24,8 +24,13 @@ from dfckit.connectivity import (
     LEiDA,
     FixedPartitionGraph,
     LowRankCovariance,
+    adjacent_window_pattern_similarity,
+    all_pair_window_pattern_similarity,
+    edge_vector_to_symmetric_matrix,
     partition_edge_summary,
     SlidingWindowFC,
+    summarize_window_pattern_dataset,
+    window_pattern_adjacency_excess,
 )
 ```
 
@@ -46,6 +51,7 @@ from dfckit.information import (
 
 ```python
 from dfckit.storage import FeatureStore
+from dfckit.storage import summarize_static_fc_dataset
 
 store = FeatureStore.create(
     "results/window-fc.store",
@@ -54,10 +60,17 @@ store = FeatureStore.create(
     sample_interval_seconds=result.sample_interval_seconds,
 )
 store.append_dataset(dataset)
+
+# Select exact edges without duplicating the disk-backed store.
+selected = store.select_features(
+    feature_keys=(("V1_L", "PUT-DP_L"), ("V1_R", "PUT-DP_R"))
+)
 ```
 
 FeatureStore chunks are memory-mappable and retain sequence/acquisition
-boundaries. They are an execution format, not a provenance database.
+boundaries. A selected feature view preserves those boundaries and is accepted
+by the store-based state fitters. FeatureStore objects are an execution format,
+not a provenance database.
 
 ## State models
 
@@ -107,12 +120,18 @@ directories or nested artifact loaders.
 ## Inference
 
 ```python
-from dfckit.inference import ols_hc3, paired_hc3, paired_nbs, paired_sign_flip
+from dfckit.inference import (
+    hc3_confidence_interval,
+    ols_hc3,
+    paired_hc3,
+    paired_nbs,
+    paired_sign_flip,
+)
 ```
 
 Inference modules provide paired sign flips, bootstrap intervals, HC3 models,
-declared-family FDR, and paired NBS. These are independent of state-count
-selection and model persistence.
+declared-family FDR, independent named-endpoint group models, and paired NBS.
+These are independent of state-count selection and model persistence.
 
 ## Advanced interfaces