openomicsbench 2.0.0__tar.gz → 2.2.0__tar.gz

openomicsbench-2.2.0/CITATION.cff

@@ -0,0 +1,16 @@
+cff-version: 1.2.0
+message: "Please cite the archived release corresponding to the version you used."
+title: "OpenOmicsBench"
+type: software
+version: "2.2.0"
+license: Apache-2.0
+doi: "10.5281/zenodo.22551734"
+authors:
+  - family-names: "Patni"
+    given-names: "Vivaan"
+    orcid: "https://orcid.org/0009-0005-1859-5107"
+repository-code: "https://github.com/vxxqv/openomicsbench"
+identifiers:
+  - type: doi
+    value: "10.5281/zenodo.22551734"
+    description: "Zenodo archive containing all released versions"

{openomicsbench-2.0.0 → openomicsbench-2.2.0}/MANIFEST.in

@@ -1,4 +1,7 @@
 include LICENSE
+include LICENSE-METADATA
+include NOTICE
+include CITATION.cff
 include NOTICE
 include README.md
 recursive-include datasets *

{openomicsbench-2.0.0/src/openomicsbench.egg-info → openomicsbench-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: openomicsbench
-Version: 2.0.0
+Version: 2.2.0
 Summary: Traceable omics benchmarks and deterministic FASTA and FASTQ tools
 Author: Vivaan Patni
 Maintainer: Vivaan Patni
@@ -35,6 +35,8 @@ Dynamic: license-file
 
 OpenOmicsBench is a compact benchmark collection and local sequence toolkit for bioinformatics software testing, method checks and teaching. Version 2 keeps the 12 certified bulk RNA-seq objects from version 1 and adds strict FASTA and FASTQ handling, DNA, RNA and protein support, paired-read checks, preprocessing, sequence QC and five deterministic sequence benchmarks.
 
+Version 2.2 adds multi-method benchmark matrices, report-to-report regression gates and portable benchmark bundles. Suite reports can be written as JSON, Markdown, JUnit XML, CSV or a self-contained HTML page.
+
 The package runs offline after installation. Sequence files stay on the local computer. The built-in tools cover inspection and lightweight preprocessing; they do not claim to replace aligners, variant callers, taxonomic classifiers or assay-specific statistical workflows.
 
 ## Install
@@ -104,6 +106,65 @@ The FASTA validator follows the nucleotide symbol expectations described by [NCB
 
 The full command reference is in [docs/sequence-tools.md](docs/sequence-tools.md).
 
+## Benchmark suites
+
+Validate the complete installed collection and save reports that can be attached to a CI run:
+
+```sh
+omicsbench suite validate --json reports/validation.json --markdown reports/validation.md --junit reports/validation.xml
+```
+
+The selection can be narrowed by assay or by repeating `--id`:
+
+```sh
+omicsbench suite validate --assay bulk_rna_seq
+omicsbench suite validate --id rnaseq-002 --id sequence-005
+```
+
+To test a differential-expression pipeline across several biological objects, place each result in a directory under its benchmark ID. CSV and TSV are accepted, with optional gzip compression:
+
+```text
+results/
+  rnaseq-002.tsv.gz
+  rnaseq-003.csv
+```
+
+Then run:
+
+```sh
+omicsbench suite compare results --id rnaseq-002 --id rnaseq-003 --junit reports/comparison.xml
+```
+
+Compare several tools or parameter sets in one matrix:
+
+```sh
+omicsbench suite matrix \
+  --method deseq2=results/deseq2 \
+  --method edger=results/edger \
+  --json reports/matrix.json \
+  --csv reports/matrix.csv \
+  --html reports/matrix.html
+```
+
+Use a previously accepted report as a regression baseline:
+
+```sh
+omicsbench suite regress accepted.json candidate.json --absolute-tolerance 0.01 --junit reports/regression.xml
+```
+
+Omit `--id` to require results for all 12 comparable RNA-seq objects. A completed suite exits with status 0 when every check passes, status 2 when a benchmark or regression gate fails, and status 1 for invalid input. Existing report files are protected unless `--force` is supplied. The [suite reference](docs/benchmark-suites.md) describes the file convention and report fields.
+
+## Portable bundles
+
+Create a deterministic ZIP containing selected benchmarks, their manifests, declared files and the project licence material:
+
+```sh
+omicsbench bundle create rna-and-dna.zip --id rnaseq-002 --id sequence-005
+omicsbench bundle verify rna-and-dna.zip
+```
+
+Bundle verification checks the complete file inventory, byte counts, SHA-256 digests and embedded dataset manifests without extracting the archive. Recreating the same selection with the same package version produces the same archive bytes. See [docs/portable-bundles.md](docs/portable-bundles.md).
+
 ## Assay profiles
 
 The assay profiles explain what the local tools can check and where a dedicated workflow becomes necessary:

{openomicsbench-2.0.0 → openomicsbench-2.2.0}/README.md

@@ -2,6 +2,8 @@
 
 OpenOmicsBench is a compact benchmark collection and local sequence toolkit for bioinformatics software testing, method checks and teaching. Version 2 keeps the 12 certified bulk RNA-seq objects from version 1 and adds strict FASTA and FASTQ handling, DNA, RNA and protein support, paired-read checks, preprocessing, sequence QC and five deterministic sequence benchmarks.
 
+Version 2.2 adds multi-method benchmark matrices, report-to-report regression gates and portable benchmark bundles. Suite reports can be written as JSON, Markdown, JUnit XML, CSV or a self-contained HTML page.
+
 The package runs offline after installation. Sequence files stay on the local computer. The built-in tools cover inspection and lightweight preprocessing; they do not claim to replace aligners, variant callers, taxonomic classifiers or assay-specific statistical workflows.
 
 ## Install
@@ -71,6 +73,65 @@ The FASTA validator follows the nucleotide symbol expectations described by [NCB
 
 The full command reference is in [docs/sequence-tools.md](docs/sequence-tools.md).
 
+## Benchmark suites
+
+Validate the complete installed collection and save reports that can be attached to a CI run:
+
+```sh
+omicsbench suite validate --json reports/validation.json --markdown reports/validation.md --junit reports/validation.xml
+```
+
+The selection can be narrowed by assay or by repeating `--id`:
+
+```sh
+omicsbench suite validate --assay bulk_rna_seq
+omicsbench suite validate --id rnaseq-002 --id sequence-005
+```
+
+To test a differential-expression pipeline across several biological objects, place each result in a directory under its benchmark ID. CSV and TSV are accepted, with optional gzip compression:
+
+```text
+results/
+  rnaseq-002.tsv.gz
+  rnaseq-003.csv
+```
+
+Then run:
+
+```sh
+omicsbench suite compare results --id rnaseq-002 --id rnaseq-003 --junit reports/comparison.xml
+```
+
+Compare several tools or parameter sets in one matrix:
+
+```sh
+omicsbench suite matrix \
+  --method deseq2=results/deseq2 \
+  --method edger=results/edger \
+  --json reports/matrix.json \
+  --csv reports/matrix.csv \
+  --html reports/matrix.html
+```
+
+Use a previously accepted report as a regression baseline:
+
+```sh
+omicsbench suite regress accepted.json candidate.json --absolute-tolerance 0.01 --junit reports/regression.xml
+```
+
+Omit `--id` to require results for all 12 comparable RNA-seq objects. A completed suite exits with status 0 when every check passes, status 2 when a benchmark or regression gate fails, and status 1 for invalid input. Existing report files are protected unless `--force` is supplied. The [suite reference](docs/benchmark-suites.md) describes the file convention and report fields.
+
+## Portable bundles
+
+Create a deterministic ZIP containing selected benchmarks, their manifests, declared files and the project licence material:
+
+```sh
+omicsbench bundle create rna-and-dna.zip --id rnaseq-002 --id sequence-005
+omicsbench bundle verify rna-and-dna.zip
+```
+
+Bundle verification checks the complete file inventory, byte counts, SHA-256 digests and embedded dataset manifests without extracting the archive. Recreating the same selection with the same package version produces the same archive bytes. See [docs/portable-bundles.md](docs/portable-bundles.md).
+
 ## Assay profiles
 
 The assay profiles explain what the local tools can check and where a dedicated workflow becomes necessary:

{openomicsbench-2.0.0 → openomicsbench-2.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "openomicsbench"
-version = "2.0.0"
+version = "2.2.0"
 description = "Traceable omics benchmarks and deterministic FASTA and FASTQ tools"
 readme = {file = "README.md", content-type = "text/markdown"}
 license = "Apache-2.0"

{openomicsbench-2.0.0 → openomicsbench-2.2.0}/setup.py

@@ -1,5 +1,5 @@
 from pathlib import Path
-from shutil import copytree
+from shutil import copy2, copytree
 
 from setuptools import setup
 from setuptools.command.build_py import build_py as setuptools_build_py
@@ -9,8 +9,12 @@ class build_py(setuptools_build_py):
     def run(self):
         super().run()
         source = Path(__file__).parent / "datasets"
-        destination = Path(self.build_lib) / "omicsbench" / "_collection" / "datasets"
+        collection = Path(self.build_lib) / "omicsbench" / "_collection"
+        destination = collection / "datasets"
         copytree(source, destination, dirs_exist_ok=True)
+        collection.mkdir(parents=True, exist_ok=True)
+        for name in ("CITATION.cff", "LICENSE", "LICENSE-METADATA", "NOTICE"):
+            copy2(Path(__file__).parent / name, collection / name)
 
 
 setup(cmdclass={"build_py": build_py})

{openomicsbench-2.0.0 → openomicsbench-2.2.0}/src/omicsbench/__init__.py

@@ -1,2 +1,2 @@
 """Discovery, validation and local analysis of compact omics benchmarks."""
-__version__ = "2.0.0"
+__version__ = "2.2.0"

openomicsbench-2.2.0/src/omicsbench/bundle.py

@@ -0,0 +1,193 @@
+"""Create and verify portable, deterministic benchmark bundles."""
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import zipfile
+from pathlib import Path, PurePosixPath
+
+from . import __version__
+from .models import Dataset
+from .registry import registry
+
+
+BUNDLE_FORMAT = "openomicsbench-bundle-v1"
+SUPPORT_FILES = ("CITATION.cff", "LICENSE", "LICENSE-METADATA", "NOTICE")
+FIXED_TIME = (1980, 1, 1, 0, 0, 0)
+
+
+def _sha256(data: bytes) -> str:
+    return hashlib.sha256(data).hexdigest()
+
+
+def _safe_name(value: str) -> str:
+    if "\\" in value or "\x00" in value:
+        raise ValueError(f"unsafe bundle path: {value}")
+    path = PurePosixPath(value)
+    if path.is_absolute() or not path.parts or any(part in {"", ".", ".."} for part in path.parts):
+        raise ValueError(f"unsafe bundle path: {value}")
+    return path.as_posix()
+
+
+def _selected(root: Path, dataset_ids: list[str] | None, assay: str | None):
+    records = registry(root)
+    requested = list(dict.fromkeys(dataset_ids or []))
+    unknown = [dataset_id for dataset_id in requested if dataset_id not in records]
+    if unknown:
+        raise ValueError(f"unknown dataset IDs: {', '.join(unknown)}")
+    selected = [(model, folder) for model, folder in records.values() if not assay or model.assay == assay]
+    if requested:
+        wanted = set(requested)
+        selected = [(model, folder) for model, folder in selected if model.id in wanted]
+        excluded = [dataset_id for dataset_id in requested if dataset_id not in {model.id for model, _ in selected}]
+        if excluded:
+            raise ValueError(f"dataset IDs do not match assay {assay}: {', '.join(excluded)}")
+    if not selected:
+        raise ValueError("no datasets matched the bundle selection")
+    return selected
+
+
+def _zip_info(name: str) -> zipfile.ZipInfo:
+    info = zipfile.ZipInfo(_safe_name(name), FIXED_TIME)
+    info.compress_type = zipfile.ZIP_DEFLATED
+    info.create_system = 3
+    info.external_attr = 0o100644 << 16
+    return info
+
+
+def create_bundle(
+    root: Path,
+    destination: Path,
+    dataset_ids: list[str] | None = None,
+    assay: str | None = None,
+    force: bool = False,
+) -> dict:
+    root = Path(root).resolve()
+    destination = Path(destination)
+    if destination.exists() and not force:
+        raise ValueError(f"{destination}: output already exists; use --force to replace it")
+    selected = _selected(root, dataset_ids, assay)
+    files: dict[str, bytes] = {}
+    for relative in SUPPORT_FILES:
+        path = root / relative
+        if not path.is_file():
+            raise ValueError(f"{path}: required bundle support file is missing")
+        files[relative] = path.read_bytes()
+    for model, folder in selected:
+        prefix = folder.resolve().relative_to(root).as_posix()
+        manifest_path = (folder / "manifest.json").resolve()
+        included = [manifest_path, *(folder / record.path for record in model.files)]
+        for path in included:
+            resolved = path.resolve()
+            try:
+                relative = resolved.relative_to(root).as_posix()
+            except ValueError:
+                raise ValueError(f"{path}: bundle input escapes the collection root") from None
+            if not resolved.is_file():
+                raise ValueError(f"{relative}: declared bundle file is missing")
+            data = resolved.read_bytes()
+            if resolved != manifest_path:
+                record = next(item for item in model.files if (folder / item.path).resolve() == resolved)
+                if len(data) != record.bytes or _sha256(data) != record.sha256:
+                    raise ValueError(f"{relative}: declared size or SHA-256 does not match")
+            files[_safe_name(relative)] = data
+        if f"{prefix}/manifest.json" not in files:
+            raise ValueError(f"{model.id}: manifest was not added to the bundle")
+
+    file_records = [
+        {"path": name, "bytes": len(files[name]), "sha256": _sha256(files[name])}
+        for name in sorted(files)
+    ]
+    manifest = {
+        "format": BUNDLE_FORMAT,
+        "software_version": __version__,
+        "selection": {"assay": assay, "ids": [model.id for model, _ in selected] if dataset_ids else []},
+        "datasets": [model.id for model, _ in selected],
+        "files": file_records,
+    }
+    manifest_data = (json.dumps(manifest, indent=2, sort_keys=True) + "\n").encode("utf-8")
+    destination.parent.mkdir(parents=True, exist_ok=True)
+    temporary = destination.with_name(destination.name + ".tmp")
+    try:
+        with zipfile.ZipFile(temporary, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=9) as archive:
+            archive.writestr(_zip_info("bundle.json"), manifest_data, compresslevel=9)
+            for name in sorted(files):
+                archive.writestr(_zip_info(name), files[name], compresslevel=9)
+        os.replace(temporary, destination)
+    finally:
+        temporary.unlink(missing_ok=True)
+    result = verify_bundle(destination)
+    result["path"] = str(destination)
+    return result
+
+
+def verify_bundle(path: Path) -> dict:
+    path = Path(path)
+    if not path.is_file():
+        raise ValueError(f"{path}: bundle does not exist")
+    try:
+        with zipfile.ZipFile(path) as archive:
+            infos = [info for info in archive.infolist() if not info.is_dir()]
+            names = [_safe_name(info.filename) for info in infos]
+            if len(names) != len(set(names)):
+                raise ValueError(f"{path}: bundle contains duplicate paths")
+            if names.count("bundle.json") != 1:
+                raise ValueError(f"{path}: bundle.json is missing or duplicated")
+            for info in infos:
+                if ((info.external_attr >> 16) & 0o170000) == 0o120000:
+                    raise ValueError(f"{path}: symbolic links are not permitted")
+            try:
+                manifest = json.loads(archive.read("bundle.json").decode("utf-8"))
+            except (json.JSONDecodeError, UnicodeDecodeError) as error:
+                raise ValueError(f"{path}: invalid bundle.json: {error}") from None
+            if manifest.get("format") != BUNDLE_FORMAT:
+                raise ValueError(f"{path}: unsupported bundle format")
+            datasets = manifest.get("datasets")
+            records = manifest.get("files")
+            if not isinstance(datasets, list) or not datasets or len(datasets) != len(set(datasets)):
+                raise ValueError(f"{path}: dataset list is empty or duplicated")
+            if not isinstance(records, list):
+                raise ValueError(f"{path}: file inventory is missing")
+            declared = {}
+            for record in records:
+                if not isinstance(record, dict):
+                    raise ValueError(f"{path}: invalid file inventory record")
+                name = _safe_name(record.get("path", ""))
+                if name == "bundle.json" or name in declared:
+                    raise ValueError(f"{path}: duplicate or reserved inventory path {name}")
+                if not isinstance(record.get("bytes"), int) or record["bytes"] < 0:
+                    raise ValueError(f"{path}: invalid byte count for {name}")
+                sha256 = record.get("sha256", "")
+                if not isinstance(sha256, str) or len(sha256) != 64 or any(char not in "0123456789abcdef" for char in sha256):
+                    raise ValueError(f"{path}: invalid SHA-256 for {name}")
+                declared[name] = record
+            expected = set(declared) | {"bundle.json"}
+            if set(names) != expected:
+                missing = sorted(expected - set(names))
+                extra = sorted(set(names) - expected)
+                raise ValueError(f"{path}: archive inventory differs: missing={missing}, extra={extra}")
+            for name, record in declared.items():
+                data = archive.read(name)
+                if len(data) != record["bytes"] or _sha256(data) != record["sha256"]:
+                    raise ValueError(f"{path}: size or SHA-256 differs for {name}")
+            manifest_ids = []
+            for name in sorted(declared):
+                if name.startswith("datasets/") and name.endswith("/manifest.json"):
+                    model = Dataset.model_validate_json(archive.read(name))
+                    manifest_ids.append(model.id)
+            if sorted(manifest_ids) != sorted(datasets):
+                raise ValueError(f"{path}: dataset manifests differ from the dataset list")
+    except zipfile.BadZipFile as error:
+        raise ValueError(f"{path}: invalid ZIP archive: {error}") from None
+    data = path.read_bytes()
+    return {
+        "status": "pass",
+        "format": BUNDLE_FORMAT,
+        "software_version": manifest.get("software_version"),
+        "datasets": datasets,
+        "dataset_count": len(datasets),
+        "file_count": len(declared),
+        "archive_bytes": len(data),
+        "sha256": _sha256(data),
+    }

{openomicsbench-2.0.0 → openomicsbench-2.2.0}/src/omicsbench/cli.py

@@ -11,6 +11,10 @@ from .cache import get, verify_cache
 from .compare import compare
 from .sequence_cli import add_sequence_parser, run_sequence_command
 from .assays import build_plan, get_profile, list_profiles
+from .suite import compare_suite, validate_suite, write_reports
+from .matrix import compare_matrix, parse_method
+from .regression import compare_reports
+from .bundle import create_bundle, verify_bundle
 from .validate import validate
 
 def main():
@@ -43,10 +47,48 @@ def main():
     assay_plan.add_argument("input", type=Path, nargs="+")
     assay_plan.add_argument("--output-directory", type=Path, default=Path("omicsbench-output"))
     assay_plan.add_argument("--adapter", action="append", default=[])
+    suite = sub.add_parser("suite", help="Run collection-wide checks and write CI reports.")
+    suite_sub = suite.add_subparsers(dest="suite_command", required=True)
+    suite_validate = suite_sub.add_parser("validate", help="Validate several bundled benchmarks in one run.")
+    suite_validate.add_argument("--assay", choices=["bulk_rna_seq", "sequence_dna", "sequence_rna", "sequence_protein", "short_read_dna", "whole_genome_dna_seq"])
+    suite_compare = suite_sub.add_parser("compare", help="Compare a directory of differential-expression results.")
+    suite_compare.add_argument("results", type=Path, help="Directory containing files named <dataset-id>.csv or .tsv, optionally gzip-compressed.")
+    suite_compare.add_argument("--detail-limit", type=int, default=20, help="Maximum missing and unexpected gene examples per benchmark.")
+    suite_matrix = suite_sub.add_parser("matrix", help="Compare several analysis methods across the same benchmarks.")
+    suite_matrix.add_argument("--method", action="append", required=True, help="Method and result directory as NAME=PATH. Repeat for every method.")
+    suite_matrix.add_argument("--detail-limit", type=int, default=20, help="Maximum missing and unexpected gene examples per benchmark.")
+    suite_regress = suite_sub.add_parser("regress", help="Fail when a candidate suite report regresses from a baseline report.")
+    suite_regress.add_argument("baseline", type=Path, help="Previously accepted JSON suite report.")
+    suite_regress.add_argument("candidate", type=Path, help="Candidate JSON suite report to check.")
+    suite_regress.add_argument("--absolute-tolerance", type=float, default=0.0, help="Largest permitted absolute decrease in a tracked metric.")
+    suite_regress.add_argument("--allow-missing", action="store_true", help="Do not fail when a baseline case is absent from the candidate.")
+    for command in (suite_validate, suite_compare, suite_matrix):
+        command.add_argument("--id", action="append", default=[], help="Benchmark ID to include. Repeat to select several; omit to run all eligible benchmarks.")
+    for command in (suite_validate, suite_compare, suite_matrix, suite_regress):
+        command.add_argument("--json", type=Path, help="Write the complete report as JSON.")
+        command.add_argument("--markdown", type=Path, help="Write a concise Markdown report.")
+        command.add_argument("--junit", type=Path, help="Write a JUnit XML report for CI systems.")
+        command.add_argument("--csv", type=Path, help="Write a flat CSV report for analysis or plotting.")
+        command.add_argument("--html", type=Path, help="Write a self-contained HTML report.")
+        command.add_argument("--force", action="store_true", help="Replace existing report files.")
+    bundle = sub.add_parser("bundle", help="Create or verify a portable benchmark bundle.")
+    bundle_sub = bundle.add_subparsers(dest="bundle_command", required=True)
+    bundle_create = bundle_sub.add_parser("create", help="Create a deterministic ZIP containing selected benchmarks.")
+    bundle_create.add_argument("output", type=Path)
+    bundle_create.add_argument("--id", action="append", default=[], help="Benchmark ID to include. Repeat to select several.")
+    bundle_create.add_argument("--assay", choices=["bulk_rna_seq", "sequence_dna", "sequence_rna", "sequence_protein", "short_read_dna", "whole_genome_dna_seq"])
+    bundle_create.add_argument("--force", action="store_true", help="Replace an existing bundle.")
+    bundle_verify = bundle_sub.add_parser("verify", help="Verify a bundle inventory, hashes and dataset manifests.")
+    bundle_verify.add_argument("archive", type=Path)
     args = p.parse_args()
     try:
         if args.command == "seq":
             out = run_sequence_command(args)
+        elif args.command == "bundle":
+            if args.bundle_command == "create":
+                out = create_bundle(args.root, args.output, args.id, args.assay, args.force)
+            else:
+                out = verify_bundle(args.archive)
         elif args.command == "assay":
             if args.assay_command == "list":
                 out = list_profiles()
@@ -54,6 +96,24 @@ def main():
                 out = get_profile(args.profile)
             else:
                 out = build_plan(args.profile, args.input, args.output_directory, args.adapter)
+        elif args.command == "suite":
+            if args.suite_command == "validate":
+                out = validate_suite(args.root, args.assay, args.id)
+            elif args.suite_command == "compare":
+                out = compare_suite(args.root, args.results, args.id, args.detail_limit)
+            elif args.suite_command == "matrix":
+                out = compare_matrix(args.root, [parse_method(value) for value in args.method], args.id, args.detail_limit)
+            else:
+                out = compare_reports(args.baseline, args.candidate, args.absolute_tolerance, args.allow_missing)
+            out["reports"] = write_reports(
+                out,
+                json_path=args.json,
+                markdown_path=args.markdown,
+                junit_path=args.junit,
+                force=args.force,
+                csv_path=args.csv,
+                html_path=args.html,
+            )
         elif args.command == "list":
             out = [{"id":m.id,"title":m.title,"kind":m.kind,"status":m.status,"rights":m.rights.status,"tiers":sorted({f.tier for f in m.files})} for m,_ in registry(args.root).values() if not args.assay or m.assay==args.assay]
         elif args.command == "doctor":
@@ -75,6 +135,8 @@ def main():
         print(json.dumps(out,indent=2,allow_nan=False))
         if args.command == "compare" and out["status"] == "fail":
             sys.exit(2)
+        if args.command == "suite" and out["summary"]["status"] == "fail":
+            sys.exit(2)
     except (ValueError,OSError,KeyError) as exc:
         if args.debug:
             raise

openomicsbench-2.2.0/src/omicsbench/matrix.py

@@ -0,0 +1,87 @@
+"""Compare several analysis methods across the RNA-seq benchmark collection."""
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+from .suite import _summary, compare_suite
+
+
+METHOD_NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$")
+SCORED_METRICS = ("spearman_logfc", "top_k_jaccard", "sign_concordance")
+
+
+def parse_method(value: str) -> tuple[str, Path]:
+    if "=" not in value:
+        raise ValueError(f"{value}: method must use NAME=RESULTS_DIRECTORY")
+    name, directory = value.split("=", 1)
+    if not METHOD_NAME.fullmatch(name):
+        raise ValueError(f"{name}: method name must use letters, numbers, dots, underscores or hyphens")
+    if not directory:
+        raise ValueError(f"{value}: results directory is empty")
+    return name, Path(directory)
+
+
+def _method_summary(name: str, directory: Path, results: list[dict]) -> dict:
+    summary = _summary(results)
+    completed = [item for item in results if "details" in item]
+    mean_metrics = {
+        metric: (sum(item["details"]["metrics"][metric] for item in completed) / len(completed) if completed else None)
+        for metric in SCORED_METRICS
+    }
+    mean_coverage = (
+        sum(item["details"]["genes"]["coverage"] for item in completed) / len(completed)
+        if completed else None
+    )
+    return {
+        "name": name,
+        "results_directory": str(directory),
+        **summary,
+        "pass_rate": summary["passed"] / summary["total"] if summary["total"] else 0.0,
+        "mean_metrics": mean_metrics,
+        "mean_coverage": mean_coverage,
+    }
+
+
+def compare_matrix(
+    root: Path,
+    methods: list[tuple[str, Path]],
+    dataset_ids: list[str] | None = None,
+    detail_limit: int = 20,
+) -> dict:
+    if len(methods) < 2:
+        raise ValueError("a benchmark matrix requires at least two --method entries")
+    names = [name for name, _ in methods]
+    if len(names) != len(set(names)):
+        raise ValueError("method names must be unique")
+    cells = []
+    method_summaries = []
+    benchmark_ids = None
+    for name, directory in methods:
+        if not METHOD_NAME.fullmatch(name):
+            raise ValueError(f"{name}: invalid method name")
+        report = compare_suite(root, directory, dataset_ids, detail_limit)
+        ids = [item["id"] for item in report["results"]]
+        if benchmark_ids is None:
+            benchmark_ids = ids
+        elif ids != benchmark_ids:
+            raise ValueError(f"{name}: benchmark selection differs from the first method")
+        method_results = []
+        for item in report["results"]:
+            cell = {"method": name, **item}
+            cells.append(cell)
+            method_results.append(item)
+        method_summaries.append(_method_summary(name, directory, method_results))
+
+    def rank_key(item: dict):
+        spearman = item["mean_metrics"]["spearman_logfc"]
+        return (-item["passed"], item["failed"], -(spearman if spearman is not None else -1.0), item["name"])
+
+    leaderboard = sorted(method_summaries, key=rank_key)
+    return {
+        "operation": "matrix",
+        "selection": {"assay": "bulk_rna_seq", "ids": dataset_ids or []},
+        "summary": {**_summary(cells), "methods": len(methods), "benchmarks": len(benchmark_ids or [])},
+        "leaderboard": leaderboard,
+        "results": cells,
+    }