iobrx 0.3.0__tar.gz

iobrx-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 iobrx contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

iobrx-0.3.0/PKG-INFO

@@ -0,0 +1,621 @@
+Metadata-Version: 2.4
+Name: iobrx
+Version: 0.3.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Rust
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Dist: iobrpy==0.2.1
+Requires-Dist: numpy==2.2.6
+Requires-Dist: pandas>=2.3,<3
+Requires-Dist: scipy==1.16.3
+Requires-Dist: scikit-learn==1.7.2
+Requires-Dist: gseapy==1.3.1
+Requires-Dist: joblib>=1.3
+Requires-Dist: pyarrow>=20,<21
+Requires-Dist: pyreadr>=0.5.6,<0.6
+Requires-Dist: pytest>=7.0 ; extra == 'test'
+Requires-Dist: matplotlib>=3.7 ; extra == 'tutorials'
+Requires-Dist: nbformat>=5.9 ; extra == 'tutorials'
+Requires-Dist: nbclient>=0.10 ; extra == 'tutorials'
+Requires-Dist: ipykernel>=6 ; extra == 'tutorials'
+Requires-Dist: jupyterlab>=4 ; extra == 'tutorials'
+Provides-Extra: test
+Provides-Extra: tutorials
+License-File: LICENSE
+License-File: rust/vendor/THIRD_PARTY_NOTICES.md
+License-File: rust/vendor/LICENSE.libsvm
+License-File: rust/vendor/LICENSE.scikit-learn
+License-File: rust/vendor/LICENSE.x86-simd-sort
+Summary: Accelerated IOBRpy workflows with a pandas API, Rust kernels and scoped numerical parity validation
+Keywords: bioinformatics,deconvolution,cibersort,ssgsea,iobr,tumor-microenvironment
+Author: iobrx contributors
+License: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Benchmarks, https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md
+Project-URL: Changelog, https://github.com/LCGaoZzz/iobrx/blob/main/CHANGELOG.md
+Project-URL: Homepage, https://github.com/LCGaoZzz/iobrx
+Project-URL: Issues, https://github.com/LCGaoZzz/iobrx/issues
+Project-URL: Repository, https://github.com/LCGaoZzz/iobrx
+
+# iobrx
+
+**Fast, faithful tumor microenvironment analysis with a pandas API.**
+
+[![CI](https://github.com/LCGaoZzz/iobrx/actions/workflows/ci.yml/badge.svg)](https://github.com/LCGaoZzz/iobrx/actions/workflows/ci.yml)
+[![version](https://img.shields.io/badge/version-0.3.0-blue)](https://github.com/LCGaoZzz/iobrx/blob/main/CHANGELOG.md)
+[![tutorials](https://img.shields.io/badge/executed_notebooks-28-teal)](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/README.md)
+[![license](https://img.shields.io/badge/code-MIT-black)](https://github.com/LCGaoZzz/iobrx/blob/main/LICENSE)
+
+English · [中文说明](https://github.com/LCGaoZzz/iobrx/blob/main/README.zh-CN.md) · [Tutorial gallery](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/README.md) · [CPU compatibility](https://github.com/LCGaoZzz/iobrx/blob/main/docs/PORTABILITY.md) · [Benchmarks](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)
+
+**iobrx is an independently maintained acceleration and integration layer built
+on the original [IOBRpy](https://github.com/IOBR/IOBRpy) toolkit.** It depends on
+IOBRpy and reuses its reference resources, gene signatures and analysis semantics.
+iobrx adds Rust kernels, parallel/vectorized execution, a pandas API, tested
+tutorials and Omicos agent interfaces. Inputs and outputs are ordinary
+DataFrames and files. Numerical parity is tested per method and fixture;
+it is not a universal bit-exact guarantee for every new workflow, parameter or
+input. See the explicit exceptions and [validation scope](docs/VALIDATION_0.3.md).
+
+[Original IOBRpy repository](https://github.com/IOBR/IOBRpy) · [Official IOBRpy documentation](https://iobr.github.io/IOBRpy/)
+
+**29 public APIs** — 27 workflow functions (plus the `deconvolute_quantiseq`
+alias and the `load_official` data helper) covering the following IOBRpy workflow families:
+immune deconvolution (CIBERSORT, BayesPrism, EPIC, quanTIseq, MCP-counter,
+ESTIMATE), signature scoring (PCA / z-score / ssGSEA / integration), TPM
+conversion and annotation, Immunophenoscore, ligand–receptor pairing, NMF and
+TME clustering, the full `tme_profile` chain, RNA-seq file merging, and the
+FASTQ→TME orchestration stages (fastp / salmon / STAR / TRUST4 / SpecHLA).
+
+**Omicos:** the harness now exposes 27 typed analysis identifiers (the four signature modes are separate identifiers), including 16 new adapters. HLA and custom-reference BayesPrism remain direct-API capabilities. [Input contracts and limits](agent-harness/omicos/skills/iobrx/references/extended-workflows.md).
+
+**Evidence scope:** R3–R6 timings below are imported campaign measurements. Some raw campaign scripts/logs are not archived here, so these numbers are not independently reproducible from this PR alone. New local validation separates numerical tests, stub-tool contracts and actual tutorial timings.
+
+**New real-data evidence:** [FASTQ/BAM/HLA recipe, repeated timings and logs](benchmarks/real_tools/README.md)
+and tutorials 24–28 use public sequencing reads and actual tools. Four-thread
+Salmon varies between repeated original runs as well as across wrappers;
+HLA extraction was slower in this small fixture. These results do not support
+a claim that every workflow is faster or universally byte-identical.
+
+### What's new in 0.3.0
+
+- **18 additional workflow APIs** (campaign rounds R3–R6): `nmf`,
+  `merge_salmon`, `merge_star_count`, `prepare_salmon`, `log2_eset`, `ips`,
+  `mouse2human`, `lr_cal`, `tme_cluster`, `bayesprism`, `tme_profile`,
+  `fastq_qc`, `batch_salmon`, `batch_star_count`, `trust4`, `runall`,
+  `spechla`, `hla_typing`, plus a 30.3× glue optimization inside
+  `calculate_sig_score`.
+- **Standalone HLA read extraction**: `extract_hla_read` extracts reads from
+  one BAM/CRAM using prepared tools, with dependency auto-installation off by default.
+- **4 new Rust kernels**: `lr_gene_valid_mask` (LR_cal gene filter),
+  `tme_kmeans` (k-means + KL index), `merge_salmon_parse` (quant.sf reader),
+  `bp_gibbs` (BayesPrism Gibbs sampler reproducing numpy's full RNG chain
+  bit-for-bit).
+- **`tme_profile` end-to-end 10.59×** vs the original CLI on the frozen STAD
+  fixture — the endpoint of a measured bottleneck-shift chain
+  1.12× → 8.44× → 10.59× ([BENCHMARKS.md Part II](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)).
+- **Campaign-reported benchmark (R6)**: 14 core candidates re-measured under one
+  unified cold-start protocol against the unmodified original CLI; every
+  parity contract reported PASS. The original submission recorded 188 default tests; current checks and limitations are tracked in [validation](docs/VALIDATION_0.3.md).
+
+![Example: CIBERSORT composition and all 22 LM22 populations](https://raw.githubusercontent.com/LCGaoZzz/iobrx/main/tutorials/figures/07_cibersort.png)
+
+## Additional executed tutorials
+
+Eleven new notebooks cover IPS, LR scores, NMF, TME clustering, conditional
+log transforms, mouse mapping, Salmon/STAR merging, Salmon preparation,
+`tme_profile` and BayesPrism. Each includes actual outputs and an embedded
+figure with two reviewed revisions. The table reports one final notebook API
+call on the local WSL/Omicos interpreter, two requested threads, excluding
+input preparation. These are small tutorial observations, not speedup claims.
+NMF's BLAS parallelism is not governed solely by the requested thread count.
+The BayesPrism example uses a short demo chain; file-merging examples are synthetic.
+
+| Tutorial | Input shape | API call time |
+| --- | --- | --- |
+| [13_ips](tutorials/13_ips.ipynb) | 48058 × 4 | 0.016 s |
+| [14_lr_cal](tutorials/14_lr_cal.ipynb) | 48058 × 4 | 0.143 s |
+| [15_nmf](tutorials/15_nmf.ipynb) | 10 × 22 | 0.465 s |
+| [16_tme_cluster](tutorials/16_tme_cluster.ipynb) | 10 × 22 | 0.044 s |
+| [17_log2_eset](tutorials/17_log2_eset.ipynb) | 48058 × 4 | 0.235 s |
+| [18_mouse2human](tutorials/18_mouse2human.ipynb) | 4 × 3 | 0.020 s |
+| [19_merge_salmon](tutorials/19_merge_salmon.ipynb) | 3 × 3 | 0.076 s |
+| [20_prepare_salmon](tutorials/20_prepare_salmon.ipynb) | 3 × 4 | 0.014 s |
+| [21_merge_star_count](tutorials/21_merge_star_count.ipynb) | 3 × 3 | 0.073 s |
+| [22_tme_profile](tutorials/22_tme_profile.ipynb) | 48058 × 2 | 4.828 s |
+| [23_bayesprism](tutorials/23_bayesprism.ipynb) | 128 × 3 | 0.351 s |
+
+## Install
+
+Validated target: **Python 3.11, Linux x86-64 / WSL2**. Use an isolated
+environment. Versioned wheels, source archives, checksums and the container
+digest are delivered through the [v0.3.0 release](https://github.com/LCGaoZzz/iobrx/releases/tag/v0.3.0).
+After downloading its CPython 3.11 wheel:
+
+```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
+python -m pip install --only-binary=:all: ./iobrx-0.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+python -c "import iobrx; print(iobrx.backend_info())"
+```
+
+This wheel path does not require Cargo or a C++ compiler. When **0.3.0 appears
+on [PyPI](https://pypi.org/project/iobrx/)**, the index-based equivalent is:
+
+```bash
+python -m pip install --only-binary=:all: iobrx==0.3.0
+# Tsinghua mirrors PyPI asynchronously; wait until the same version is listed.
+python -m pip install --only-binary=:all: -i https://pypi.tuna.tsinghua.edu.cn/simple iobrx==0.3.0
+```
+
+PyPI publication and mirror synchronization are independent of GitHub assets.
+If an index has not synchronized, install the release wheel directly.
+
+For source development, install Cargo and a C++17 compiler:
+
+```bash
+git clone https://github.com/LCGaoZzz/iobrx.git
+cd iobrx
+python -m pip install -c tests/constraints-validated.txt ".[tutorials,test]"
+python -c "import iobrx; print(iobrx.backend_info())"
+python -m jupyterlab tutorials
+```
+
+Use the same interpreter as the Jupyter kernel. In an existing Omicos
+environment, first check dependency compatibility; use an isolated environment
+when the required numerical versions conflict. `pip install` from this source
+builds the extension; it is not an installation without compilation.
+
+The core-analysis container uses `ghcr.io/lcgaozzz/iobrx:0.3.0`; for immutable
+execution use the digest in the release's `container-digest.txt`.
+Alignment and HLA binaries require the separate external-tool environment.
+See [release notes](docs/releases/0.3.0.md).
+
+**Why the dependency pins** (details and measurements in
+[BENCHMARKS.md §I.7](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)):
+
+| Pin | Reason |
+| --- | --- |
+| `iobrpy==0.2.1` | Supplies reference resources and the original-workflow fallback. Current package and parity tests use this exact PyPI version; the earlier campaign also compared with 0.2.0. |
+| `numpy==2.2.6` | Uses the validated reduction and sorting implementation. The campaign observed different rounding and CIBERSORT support sets with newer NumPy; broader version ranges are not promised. |
+| `scikit-learn==1.7.2` | Matches the vendored `svm.cpp` and the tested reference solver. Newer sklearn versions require separate numerical validation. |
+| `scipy==1.16.3`, `gseapy==1.3.1` | Keeps optimization and enrichment calculations on the validated implementations. |
+
+These are the exact requirements in [package metadata](pyproject.toml), not
+minimum versions. Remaining dependency bounds and the complete test environment
+are recorded in [the validation constraints](tests/constraints-validated.txt).
+
+**CPU compatibility and operating-system packaging are separate.** Upstream
+IOBRpy's binary distribution limits straightforward installation on other
+Python/platform combinations. Windows users can use WSL2; macOS, ARM and
+native Windows are not claimed as validated full-stack targets. See the
+[compatibility notes](https://github.com/LCGaoZzz/iobrx/blob/main/docs/PORTABILITY.md) for fallback behavior and limits.
+
+## Quick start
+
+```python
+import numpy as np
+import pandas as pd
+import iobrx
+
+iobrx.set_threads(8)
+counts = pd.read_parquet("tutorials/data/eset_stad.parquet")  # genes × samples
+tpm = iobrx.count2tpm(counts, check_data=True, remove_version=True)
+log_expression = np.log2(tpm + 1)
+
+# Deconvolution and scores (linear TPM for the RNA-seq deconvolution examples)
+cib = iobrx.cibersort(tpm, perm=100, QN=False)          # Rust NuSVR core
+epic = iobrx.epic(tpm)["cellFractions"]
+qnt = iobrx.quantiseq(tpm, tumor=True, rmgenes="default")
+scores = iobrx.calculate_sig_score(
+    log_expression, "signature_collection", method="integration"
+)
+
+# 0.3.0 ports
+lr = iobrx.lr_cal(eset="tpm_symbol.csv", output_file="lr.csv",
+                  data_type="tpm", id_type="symbol", cancer_type="pancan")
+ips = iobrx.ips(eset="tpm_symbol.csv", output_file="ips.csv")
+clusters = iobrx.tme_cluster(df=pd.read_csv("tme_transposed.csv"), id="sample")
+iobrx.bayesprism(bulk="bulk_counts.csv", out_dir="bp_out", n_threads=8)
+#   add backend="rust" for the bit-exact Rust Gibbs kernel
+
+# The whole TME profiling chain in one process (10.59× vs the original CLI)
+iobrx.tme_profile(input="TPM.csv", output="tme_out", threads=16)
+
+# FASTQ → TME orchestration (external tools must be on PATH — see below)
+iobrx.runall(mode="salmon", outdir="run_out", fastq="raw_fastq_dir",
+             threads=16, resume=True,
+             unknown=["--index", "references/salmon"])
+```
+
+`runall(resume=True)` reuses a table only after its producing step succeeded
+and its recorded output hash matches. A failed or interrupted table write is
+retried; successful upstream steps can still be reused. State schema 2 adds
+these per-step records: pre-schema-2 run directories require a new `outdir`.
+Existing schema 2 records remain compatible. Changed inputs, parameters,
+references or recorded calculation products require a new directory; unrelated
+notes and figures can be added, edited or removed without blocking resume.
+An uncatchable kill can leave unverified calculation products, which are not
+treated as completed work.
+
+Open [the complete workflow notebook](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/12_complete_workflow.ipynb)
+for input checks, interpretation, plots, and per-stage timers on the public
+example data (no download needed after installation). The four standalone
+signature tutorials use the public IMvigor210 demonstration panel:
+872 features × 348 samples.
+
+## The 29 APIs at a glance
+
+| API | One-liner |
+| --- | --- |
+| `cibersort` | CIBERSORT LM22 immune deconvolution (NuSVR) — Rust core, seeded permutations |
+| `calculate_sig_score` | Per-sample signature scores: `pca` / `zscore` / `ssgsea` / `integration` |
+| `count2tpm` | Raw count matrix → TPM (vectorized, bit-identical) |
+| `quantiseq` | quanTIseq TIL10 deconvolution with a memoized HGNC alias map |
+| `deconvolute_quantiseq` | Alias kept for the original function name |
+| `epic` | EPIC cell fractions and mRNA proportions |
+| `mcpcounter` | MCP-counter population abundance scores |
+| `estimate_score` | ESTIMATE stromal / immune scores and tumor purity |
+| `anno_eset` | Aggregate probes / Ensembl ids to gene symbols |
+| `bayesprism` | BayesPrism deconvolution — python-fast default, opt-in Rust Gibbs, determinism-fixed |
+| `tme_profile` | The whole TME profiling chain (sig scores + 6 deconvolutions + LR_cal) in one process |
+| `nmf` | NMF clustering with silhouette-based k selection |
+| `tme_cluster` | TME k-means clustering with KL-index best-k selection (Rust core) |
+| `lr_cal` | Ligand–receptor pair expression matrix, min of log2 TPM (Rust gene-filter core) |
+| `ips` | Immunophenoscore (Charoentong 2017 four-block design) |
+| `merge_salmon` | Merge Salmon `quant.sf` dirs → TPM / count matrices (Rust parse engine) |
+| `merge_star_count` | Merge STAR `ReadsPerGene.out.tab` files → one count matrix |
+| `prepare_salmon` | Salmon TPM → deduplicated symbol / ENSG / ENST matrix |
+| `log2_eset` | `log2(x+1)` on a genes × samples matrix |
+| `mouse2human` | Mouse → human gene-symbol conversion |
+| `fastq_qc` | FASTQ QC with fastp + MultiQC (external-tool stage) |
+| `batch_salmon` | Batch Salmon quantification over paired-end FASTQs |
+| `batch_star_count` | Batch STAR two-pass alignment + GeneCounts |
+| `trust4` | TRUST4 TCR/BCR reconstruction + accelerated immune post-processing |
+| `spechla` | SpecHLA full-resolution HLA typing for one sample |
+| `extract_hla_read` | Extract HLA-related FASTQs from one BAM/CRAM without running typing |
+| `hla_typing` | Batch HLA typing from a directory of BAM files |
+| `runall` | End-to-end FASTQ → TME orchestrator (salmon / star chains) |
+| `load_official` | Resolve / download the public IOBR example data |
+
+## Three routes, one choice — how every API is accelerated
+
+Each ported module was evaluated along three routes: keep calling the
+**original iobrpy CLI**, write a **pure-Python fast path**, or build a
+**Rust kernel**. The final choice is whatever passed the bit-exact contract
+at the best measured wall time. Speedups for the 19 ported APIs are the
+**R6 formal blind benchmark** (unified cold-start protocol, fresh subprocess
+per run, interleaved arms, medians — `research/bench_r6/results.json`,
+[BENCHMARKS.md §II.6](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)); orchestration-stage ratios are the
+R5 real-data runs ([§II.8](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)); pre-port modules carry their
+official-gate numbers ([Part I](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)).
+
+| Module | Original route | Pure-Python route | Rust route | Final choice | Speedup | Parity contract |
+| --- | --- | --- | --- | --- | --- | --- |
+| cibersort | iobrpy CLI | python fallback | Rust NuSVR core (vendored sklearn-1.7.2 libsvm) | **rust (auto)** | official gates: 34.1× e2e, up to **53.6×** held-out — see BENCHMARKS Part I | bit-exact except the unseeded P-value column |
+| calculate_sig_score | iobrpy CLI | python | Rust ssGSEA/PCA cores + glue optimization | **rust (auto)** | glue step **30.3×** (R5); stage gates 6.6–11.5× (Part I) | bit-exact |
+| count2tpm / epic / estimate / mcpcounter / quantiseq / anno_eset | iobrpy CLI | python | rust / vectorized | **rust-vectorized (auto)** | see BENCHMARKS Part I (count2tpm up to 48.8×, quantiseq up to 16.9×, mcpcounter 4.2–5.0×, anno_eset 3.5–3.8×, estimate 2.5–3.4×, epic 1.6–1.8×) | bit-exact |
+| nmf | iobrpy CLI | **python (chosen)** | no Rust core (both arms share sklearn's NMF) | **python** | **1.21×** (R6) | bit-exact: 3-file sha256 (clusters / top_features / pca_plot.png) |
+| merge_salmon | iobrpy CLI | python sequential parse (1.86× R6) | Rust read/parse engine | **rust (auto)** | **2.41×** (R6) | column-aligned token-exact (upstream `as_completed` column order is nondeterministic; port order is deterministic) |
+| merge_star_count | iobrpy CLI | **python (chosen)** | none | **python** | **1.75×** (R6) | column-aligned token-exact + stat-row **bug-compat** (upstream's leading-4-stat-rows defect preserved) |
+| prepare_salmon | iobrpy CLI | **python (chosen)** | none | **python** | **2.37×** (R6) | bit-exact (sha256) |
+| log2_eset | iobrpy CLI | **python (chosen)** | none | **python** | **2.29×** (R6) | bit-exact (sha256) |
+| ips | iobrpy CLI | **python (chosen)** | none | **python** | **3.25×** (R6) | bit-exact (sha256) |
+| mouse2human | iobrpy CLI | **python (chosen)** | none | **python** | **3.76×** (R6) | bit-exact (sha256) |
+| lr_cal | iobrpy CLI | python fallback | Rust gene-filter core | **rust (auto)** | **8.48×** (R6) | bit-exact (sha256) |
+| tme_cluster | iobrpy CLI | python RNG / k-means loop | Rust k-means core | **rust (auto)** | **7.17×** (R6) | bit-exact (sha256) |
+| bayesprism | iobrpy CLI | **python-fast (auto default)** | Rust Gibbs core (opt-in) | **python auto; `backend="rust"` optional** | python **2.62×** / rust **3.74×** (R6) | bit-exact: hs0 3-file sha256; the Rust kernel reproduces numpy's full RNG chain bit-for-bit |
+| tme_profile | iobrpy CLI (9 serial sub-steps) | — | reuse_fast v3 (sig glue + Rust LR_cal) | **reuse_fast (`cibersort_backend="original"`)** | **10.59×** (R6) | 9 outputs: 7 raw-byte identical + 2 identical after stripping the unseeded cibersort P-value column |
+| fastq_qc | iobrpy CLI | **python (chosen)** | none (the tool itself does the work) | **python** | ≈1.0× real data (R5; launch layer 14.9× lighter) | bit-exact except fastp-internal HTML jitter |
+| batch_salmon | iobrpy CLI | **python (chosen)** | none | **python** | ≈1.0× per-sample real data (R5; launch layer 14.8× lighter) | bit-exact except run-metadata timestamps |
+| batch_star_count | iobrpy CLI | **python (chosen)** | none | **python** | ≈1× real data (R5, under the declared 16-vs-32-thread deviation; launch layer 10.6× lighter) | BAM record stream + count tables bit-exact; header @PG/@CO carry the declared thread count |
+| trust4 | iobrpy CLI | **python (chosen, accelerated post-processing)** | none | **python** | ≈1.0× real data (R5; stub launch 3.3× lighter) | bit-exact: 12/12 files byte-identical incl. post-processing outputs |
+| runall | iobrpy CLI | **python (chosen)** | none | **python** | **1.02×** real data (R5) | bit-exact except documented tool jitter classes |
+| spechla | iobrpy CLI | **python (chosen)** | none | **python** | **1.01×** real data (R5) | bit-exact except samtools @PG random-ID jitter |
+| hla_typing | iobrpy CLI | **python (chosen)** | none | **python** | **1.05×** real data (R5) | bit-exact except samtools @PG random-ID jitter |
+| extract_hla_read | iobrpy CLI | existing extraction helpers | none | **python** | Not benchmarked | command/output contracts tested with stub scripts |
+
+## Why the speedups differ: one floor model
+
+Every number in the table above obeys **speedup ≈ min(1/(1−p), floor)** —
+`p` is the share of the original wall time that is a compressible hotspot,
+and each module class has its own incompressible floor. This was the
+campaign's measured conclusion across rounds R1–R6
+([BENCHMARKS.md Part II](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)), not a post-hoc story:
+
+1. **A Rust kernel pays off (≥5×) only when p ≥ 0.6.** `lr_cal`'s per-gene
+   pandas filter passes were p=0.81 of the original wall → **8.48×**;
+   `tme_cluster`'s pure-Python k-means dominates → **7.17×**. BayesPrism's
+   Gibbs sampler is only p≈0.24 → the Rust kernel lands at **3.74×** (1.56×
+   over the python lane), below 5× despite being bit-exact by construction.
+2. **Bottleneck shift (multi-stage Amdahl).** `tme_profile`:
+   **1.12×** (one process, original code — the un-optimized sig-score glue
+   was ~84% of the wall) → **8.44×** (the 30.3× glue fix moves the floor to
+   cibersort + LR_cal ≈76%) → **10.59×** (LR_cal→Rust, sub-step
+   4.10 s → 0.22 s ≈18.6×; the floor is now the contract-bound original
+   CIBERSORT solver at 58.6% + the sig Rust chain at 31%). The next floor is
+   ~9.3 s (~16.6×) — **≥12× is unreachable without breaking the cibersort
+   bit-exact contract.**
+3. **Write-out ceiling (~2.4×) for the io-merge modules.** `merge_salmon`'s
+   Rust parser is **10.32×** faster in isolation, but the byte-level output
+   contract (pandas `to_csv` serialization + gzip) caps end-to-end at ~2.5×:
+   measured 2.41–2.48× ≈ 96% of the same-window theoretical ceiling. Rust
+   cannot buy its way past the write floor.
+4. **Import + pandas-IO floor (~2–3.8×) for the small modules.** Every
+   original CLI call pays a ~1.3–1.4 s `iobrpy.main` import on 1.3–1.9 s
+   total walls. Removing it exposes pandas `read_csv`/`to_csv` as the new
+   floor: pure-IO modules land at 2.29–2.37× (`log2_eset`,
+   `prepare_salmon`), modules with real net compute at 3.25–3.76× (`ips`,
+   `mouse2human`), and `nmf` converges to 1.21× because both arms run the
+   same sklearn solver (shared-kernel floor).
+5. **Orchestration ceiling (speedup ≈ 1).** For `fastq_qc` / `batch_salmon`
+   / `batch_star_count` / `trust4` / `runall` / `spechla` / `hla_typing` the
+   external binaries *are* the wall. Real-data ratios: 0.99–1.25× (the 1.25×
+   STAR residual is fully attributable to a declared 32→16 thread deviation,
+   visible in STAR's own mapping-speed log). The ported value is API
+   consistency, in-process composition (no console-script/PATH requirement),
+   resume/parallel scheduling and byte-identical products — plus a Python
+   launch layer **3.3–14.9× lighter**, measurable when the tools are
+   stubbed.
+
+## Bit-exact contracts and known nondeterminism
+
+**Bit-exact** means: identical index/columns/dtypes/NaN mask with
+`max_abs_diff == 0.0` on every numeric cell, or sha256-equal output files,
+versus the ORIGINAL iobrpy 0.2.0 executed in the same environment. The
+contracts are enforced by `tests/test_parity_*.py` (188 tests pass by
+default; the official-data gates run in CI with the frozen fixtures).
+
+Every known nondeterministic item is an **upstream** property, proven by
+original-vs-original controls, and handled by an explicit contract clause:
+
+- **CIBERSORT P-value column** — the original seeds its permutations from OS
+  entropy (`SeedSequence()` unseeded), so *the original itself* is not
+  reproducible run-to-run on that column. iobrx seeds them: stable across
+  runs and thread counts, identical formula and `1/perm` granularity. The
+  column is excluded from every byte contract that contains it (cibersort
+  output; tme_profile's `cibersort_results.csv` / `deconvo_merged.csv`).
+- **BayesPrism state order** — the original's cell-state iteration order
+  follows the per-process `PYTHONHASHSEED` and can flip discrete Gibbs draws.
+  iobrx's default `state_order="sorted"` removes the dependency (theta /
+  theta_cv 100% bit-exact vs a `PYTHONHASHSEED=0` original; Z_tumor ULP-only,
+  max abs 1.42e-14, zero discrete flips); `state_order="legacy"` with
+  `PYTHONHASHSEED=0` reproduces the frozen gold shas exactly (the R6 parity
+  configuration).
+- **`as_completed` column order** — the original `merge_salmon` /
+  `merge_star_count` column order varies run-to-run (three different orders
+  observed in three R6 reps). iobrx emits a deterministic sorted order; the
+  parity contract aligns by column name and compares token-for-token and
+  frame-bit-for-bit.
+- **samtools @PG / @RG header IDs** — spechla / hla_typing
+  `<sample>.realign.sort.bam` inherit random `samtools merge` header ids
+  (e.g. `bwa-7A10F178`); an original-vs-original rerun shows the same
+  jitter. Alignment record streams and every other product are byte-compared.
+- **fastp HTML jitter** — the duplication rate's 7th significant digit in
+  `fastp.html` varies per run *inside the fastp binary* (an original CLI
+  rerun reproduced the port's digit and differed from its own frozen
+  baseline); cleaned FASTQs are byte-identical.
+- **Run metadata** — log timestamps (salmon/STAR/TRUST4 logs, multiqc
+  uuid/creation dates) are declared run-metadata; **every data artifact is
+  sha256-compared without normalization**.
+- **Bug-compat preserved** — upstream defects that are visible in output
+  bytes are kept deliberately, e.g. `merge_star_count`'s leading four global
+  stat rows that the original never purges, and `lr_cal`'s count-branch
+  behavior.
+
+Exact parity is an **observed result on specified inputs and pinned
+dependency versions** (numpy<2.3, scikit-learn<1.8 — see Install), not a
+cross-platform floating-point guarantee ([details](https://github.com/LCGaoZzz/iobrx/blob/main/docs/PORTABILITY.md)).
+
+## External tools (bring your own)
+
+iobrx orchestrates the heavy binaries but does not bundle them. Install them
+yourself and put them on `PATH` (or pass the per-stage `*_bin` overrides);
+iobrx preserves the upstream command parameters. Equality still depends on
+the tool's determinism and the declared comparison target; see the measured
+Salmon variability and scoped FASTQ/BAM/HLA comparisons in the new recipe.
+
+| Stage | External tools |
+| --- | --- |
+| `fastq_qc` | fastp, MultiQC |
+| `batch_salmon` | salmon |
+| `batch_star_count` | STAR (+ samtools) |
+| `trust4` | TRUST4 (`run-trust4`) |
+| `spechla` / `hla_typing` | SpecHLA toolchain: samtools, bwa/bowtie2, bcftools, freebayes, vcflib, blastn, bamUtil (`bam`) |
+| `extract_hla_read` | SpecHLA extraction assets, samtools and bamUtil (`bam`); sorted/indexed BAM or CRAM |
+| `runall` | everything in the chosen salmon/star chain |
+
+All pure-compute APIs (deconvolution, signature scores, TPM, annotation,
+IPS, LR_cal, NMF/TME clustering, merges) need **no external tools** — only
+iobrpy (references + fallback) and the pinned scientific stack.
+
+For extraction without typing, use the prepared environment:
+
+```python
+iobrx.extract_hla_read("sample1", "sample1.bam", "hg38", "hla_reads")
+```
+
+This new API defaults to `auto_install=False` for both backends; explicitly
+enable `auto_install=True` only when installing tools is part of the task.
+It is currently a direct Python API, not a typed harness adapter.
+
+## Omicos and agent workflows
+
+The [agent harness](https://github.com/LCGaoZzz/iobrx/tree/main/agent-harness) adds a JSON CLI, an optional
+stdio MCP server and a portable Omicos Agent/Skill pair for 27 analysis identifiers.
+It validates matrix orientation, declared scale and gene IDs, then records
+parameters, input/output metadata, versions, backend and elapsed time in a result
+manifest. Analyses use the existing iobrx API and preserve its result layouts.
+Known requests can run directly; discovery and diagnostics are optional.
+Agents can also use the public Python API for custom work. Full SHA-256
+auditing is opt-in; default result inspection checks file availability.
+
+After installing iobrx, from this checkout:
+
+```bash
+python -m pip install ./agent-harness
+iobrx-agent run --request agent-harness/examples/signature_pca.json
+```
+
+See [Omicos setup](https://github.com/LCGaoZzz/iobrx/blob/main/agent-harness/omicos/README.md) for workspace/catalog
+installation and MCP configuration, and [harness validation](https://github.com/LCGaoZzz/iobrx/blob/main/agent-harness/VALIDATION.md)
+for the actual test record. The companion harness is installed from this
+repository; it is not yet a separately published PyPI package.
+
+## How long does each analysis take?
+
+Measured on **Intel Core i9-13900KF, WSL2 Ubuntu, Python 3.11, 8 requested
+threads, no AVX-512**, using the existing Omicos environment. Each row runs
+in a fresh Python process: first call, followed by three repeat calls.
+The main time is the **median of those three repeats**. Data loading, input
+preparation and plotting are excluded; the complete workflow includes its
+own normalization and analysis stages. OS file caches may already be warm.
+
+<!-- BENCHMARK_TABLE_START -->
+| Analysis / notebook | Input: features × samples | Median | First call | Parity vs IOBRpy |
+| --- | --- | ---: | ---: | --- |
+| [Gene annotation and duplicate resolution](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/01_gene_annotation.ipynb) | 60,483 × 10 | **16.7 ms** | 36.0 ms | bit-identical |
+| [Count-to-TPM normalization](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/02_counts_to_tpm.ipynb) | 60,483 × 10 | **67.5 ms** | 111.6 ms | bit-identical |
+| [PCA signature scoring](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/03_signature_pca.ipynb) | 872 × 348 | **244.1 ms** | 600.4 ms | bit-identical |
+| [Mean-based signature scoring](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/04_signature_zscore.ipynb) | 872 × 348 | **78.6 ms** | 487.8 ms | bit-identical |
+| [ssGSEA signature enrichment](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/05_signature_ssgsea.ipynb) | 872 × 348 | **86.7 ms** | 495.9 ms | bit-identical |
+| [Integrated signature scoring](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/06_signature_integration.ipynb) | 872 × 348 | **336.0 ms** | 817.9 ms | bit-identical |
+| [CIBERSORT immune composition](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/07_cibersort.ipynb) | 48,058 × 10 | **26.37 s** | 26.99 s | bit-identical, P-value excepted |
+| [EPIC cell fractions and mRNA proportions](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/08_epic.ipynb) | 48,058 × 10 | **6.8 ms** | 301.9 ms | bit-identical |
+| [quanTIseq immune deconvolution](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/09_quantiseq.ipynb) | 48,058 × 10 | **40.9 ms** | 475.2 ms | bit-identical |
+| [MCP-counter population scores](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/10_mcpcounter.ipynb) | 48,058 × 10 | **1.1 ms** | 10.0 ms | bit-identical |
+| [ESTIMATE stromal and immune scores](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/11_estimate.ipynb) | 48,058 × 10 | **15.8 ms** | 31.0 ms | bit-identical |
+| [A complete, inspectable TME workflow](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/12_complete_workflow.ipynb) | 60,483 × 10 | **28.29 s** | 29.59 s | per stage, as rows above |
+<!-- BENCHMARK_TABLE_END -->
+
+These are local wall-clock measurements, not a promise for other hardware or
+cohorts. CIBERSORT uses **100 permutations and `QN=False`** on the full STAD
+TPM matrix. The restricted IMvigor210 signature panel is a different
+workload. The complete workflow runs integration scoring on the full STAD
+expression matrix, so its total is not the sum of the standalone rows.
+
+The **Parity** column is not a timing: it states what the official gates
+([`tests/test_parity_official.py`](https://github.com/LCGaoZzz/iobrx/blob/main/tests/test_parity_official.py)) assert for
+that analysis on the same fixtures — equal index, labels and column names, and
+exact equality of every numeric cell (`max_abs_diff == 0.0`) against the
+ORIGINAL `iobrpy` implementations executed in the same environment
+([validation record](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/results/validation.json)). The single
+exception is CIBERSORT's P-value: the original seeds its permutations from OS
+entropy and is not reproducible run-to-run even by itself, whereas iobrx's
+P-values are seeded — stable across runs and thread counts, with the
+identical formula and `1/perm` granularity. All 11 gates are re-run by CI on
+every push and pull request under the
+[validated constraints](https://github.com/LCGaoZzz/iobrx/blob/main/tests/constraints-validated.txt), and the same
+contract held on the historical 224-thread Xeon campaign
+([BENCHMARKS.md](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)). Exact parity is an observed result on those
+environments, not a cross-platform floating-point guarantee
+([details](https://github.com/LCGaoZzz/iobrx/blob/main/docs/PORTABILITY.md)).
+
+[Raw repeats, ranges and environment](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/results/benchmark.json) ·
+[Benchmark method and reproduction](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/BENCHMARKS.md) ·
+[Historical Xeon performance campaign](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)
+[Full performance record: acceleration stack + port campaign](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)
+
+The historical 224-thread Xeon speedups (Part I) and the R6 blind port
+benchmark (Part II) are retained as separate experiments; they are not used
+to advertise desktop performance.
+
+## Choose the right output
+
+| Analysis | Input used in the tutorials | Output and interpretation |
+| --- | --- | --- |
+| `anno_eset` | Ensembl expression + annotation | Gene-symbol matrix; duplicate candidates are ranked and one row retained |
+| `count2tpm` | Raw nonnegative counts | TPM matrix; filtering/deduplication can leave sums below one million |
+| `calculate_sig_score` | Suitable preprocessed expression | `pca`, `zscore`, `ssgsea`, or `integration`; method-specific signature scores |
+| `cibersort` | Linear TPM, `QN=False` | Relative LM22 immune fractions + fit statistics |
+| `epic` | Linear TPM | Cell fractions, mRNA proportions, fit diagnostics |
+| `quantiseq` | Linear TPM, tumor settings explicit | TIL10 fractions and an uncharacterized remainder |
+| `mcpcounter` | `log2(TPM + 1)` | Population abundance scores, not percentages |
+| `estimate_score` | `log2(TPM + 1)`, `platform="rnaseq"` | Stromal, immune and combined enrichment scores |
+| `bayesprism` | Pseudo-bulk counts + scRNA reference | Cell-type/state deconvolution with credible intervals |
+| `lr_cal` | TPM (symbol or Ensembl) | Ligand–receptor pair matrix (min of the two log2 TPMs) |
+| `ips` | TPM/FPKM expression set | Immunophenoscore 0–1 across four blocks (Charoentong 2017) |
+
+The upstream method named **`zscore`** averages preprocessed signature
+expression; it does not guarantee standardized output. `integration`
+concatenates three methods rather than averaging them. Only the exact
+`platform="affymetrix"` string requests IOBRpy's calibrated ESTIMATE purity
+transformation; the historical `"affy"` default does not. These RNA-seq
+tutorials therefore show scores without that purity transformation.
+
+## Compatibility and numerical fidelity
+
+```python
+iobrx.backend_info()  # native availability, sorting dispatch, bundled BLAS
+
+# Optional: explicitly use the original IOBRpy workflow.
+cib = iobrx.cibersort(tpm, perm=100, QN=False, backend="python")
+# backend="rust" requires native support and reports an error if unavailable.
+```
+
+`backend="auto"` is the default wherever an accelerated lane exists (see the
+route table for which lane each module ships). Set `IOBRX_DISABLE_RUST=1`
+**before importing** to disable native acceleration process-wide. Missing
+native support does not prevent using the public analysis API when IOBRpy
+and its dependencies are installed. Missing compatible OpenBLAS triggers
+fallback for CIBERSORT/PCA.
+
+All parity gates run in CI on every push and pull request under the
+[validated constraints](https://github.com/LCGaoZzz/iobrx/blob/main/tests/constraints-validated.txt); their assertion is
+exact equality — labels and every numeric cell, or file sha256 — against the
+ORIGINAL executed on the same fixtures, with the documented nondeterminism
+clauses above. Exact parity is an **observed result on specified inputs and
+dependency versions**, not a cross-platform floating-point guarantee. Native
+CIBERSORT uses seeded permutations; the original Python implementation uses
+unseeded permutations, so its P-value column is excluded from
+exact-equality assertions. See
+[precision and fallback details](https://github.com/LCGaoZzz/iobrx/blob/main/docs/PORTABILITY.md).
+
+## Reproduce, test, contribute
+
+```bash
+python -m pytest -q                      # 188 smoke + parity + portability tests
+IOBRX_TESTDATA=tutorials/data python -m pytest -q -m full   # official-data gates
+python scripts/execute_tutorials.py      # all 12, fresh kernels, embedded plots
+python scripts/benchmark_tutorials.py    # first call + 3 repeats per analysis
+python scripts/validate_tutorials.py     # outputs, exports and data checksums
+```
+
+Tutorials can be read directly on GitHub. To run them interactively:
+`python -m jupyterlab tutorials`. Plotting helpers are in
+[`tutorials/_common.py`](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/_common.py); every analysis call remains
+visible in its notebook. [The figure-review record](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/FIGURE_REVIEW.md)
+documents both refinements, with draft and final overview images.
+
+Contributions should preserve the parity gates and add a focused regression
+test for changed numerical behavior. Files in `bench/` are frozen historical
+artifacts. Report package versions, backend information, input shape and
+expression scale when reporting a problem.
+
+## Upstream credit, citation and license
+
+We thank the [IOBRpy developers](https://github.com/IOBR/IOBRpy), the
+[IOBR team](https://github.com/IOBR/IOBR), and the original method authors for
+the workflows and reference resources on which this package builds.
+
+When publishing results, cite IOBR/IOBRpy and the original methods actually
+used — including CIBERSORT (Newman et al.), BayesPrism, quanTIseq, EPIC,
+MCP-counter, ESTIMATE, IPS (Charoentong et al. 2017), TRUST4, SpecHLA, and
+the tool papers for fastp/salmon/STAR when you run the orchestration stages.
+See the [IOBRpy preprint](https://doi.org/10.64898/2026.07.17.739055) and the
+[upstream citation guide](https://iobr.github.io/IOBRpy/Citation.html). The
+preprint reports CIBERSORT *thread scaling* inside the original
+implementation — a different measurement axis from the port speedups here
+(see [BENCHMARKS.md §II.9](https://github.com/LCGaoZzz/iobrx/blob/main/BENCHMARKS.md)).
+
+iobrx provides acceleration and tutorials; it does not replace those methods
+or validate clinical conclusions from an unlabeled example dataset.
+
+iobrx code: [MIT](https://github.com/LCGaoZzz/iobrx/blob/main/LICENSE). Vendored sources retain their
+[third-party notices](https://github.com/LCGaoZzz/iobrx/blob/main/rust/vendor/THIRD_PARTY_NOTICES.md). Public example data
+retains its [upstream attribution and GPL-3 terms](https://github.com/LCGaoZzz/iobrx/blob/main/tutorials/data/README.md).
+