openomicsbench 2.2.0__tar.gz → 3.0.0__tar.gz

{openomicsbench-2.2.0 → openomicsbench-3.0.0}/CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 message: "Please cite the archived release corresponding to the version you used."
 title: "OpenOmicsBench"
 type: software
-version: "2.2.0"
+version: "3.0.0"
 license: Apache-2.0
 doi: "10.5281/zenodo.22551734"
 authors:

{openomicsbench-2.2.0/src/openomicsbench.egg-info → openomicsbench-3.0.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: openomicsbench
-Version: 2.2.0
-Summary: Traceable omics benchmarks and deterministic FASTA and FASTQ tools
+Version: 3.0.0
+Summary: Traceable multi-assay benchmarks, VCF evaluation and deterministic sequence tools
 Author: Vivaan Patni
 Maintainer: Vivaan Patni
 License-Expression: Apache-2.0
@@ -11,7 +11,7 @@ Project-URL: Issues, https://github.com/vxxqv/openomicsbench/issues
 Project-URL: Changelog, https://github.com/vxxqv/openomicsbench/blob/main/CHANGELOG.md
 Project-URL: Zenodo, https://doi.org/10.5281/zenodo.22551734
 Project-URL: ORCID, https://orcid.org/0009-0005-1859-5107
-Keywords: bioinformatics,computational-biology,genomics,transcriptomics,proteomics,bulk-rna-seq,dna-seq,fasta,fastq,dna,rna,protein-sequence,amino-acid-analysis,sequence-analysis,quality-control,kmer,variant-truth,benchmarking,benchmark-data,test-data,data-validation,data-provenance,data-integrity,reproducibility,scientific-workflows,research-software,expression-atlas,ensembl,deseq2
+Keywords: bioinformatics,computational-biology,genomics,transcriptomics,proteomics,bulk-rna-seq,dna-seq,fasta,fastq,vcf,dna,rna,protein-sequence,amino-acid-analysis,sequence-analysis,quality-control,kmer,variant-truth,variant-benchmarking,multi-assay,ro-crate,benchmarking,benchmark-data,test-data,data-validation,data-provenance,data-integrity,reproducibility,scientific-workflows,research-software,expression-atlas,ensembl,deseq2
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Science/Research
@@ -33,9 +33,9 @@ Dynamic: license-file
 
 # OpenOmicsBench
 
-OpenOmicsBench is a compact benchmark collection and local sequence toolkit for bioinformatics software testing, method checks and teaching. Version 2 keeps the 12 certified bulk RNA-seq objects from version 1 and adds strict FASTA and FASTQ handling, DNA, RNA and protein support, paired-read checks, preprocessing, sequence QC and five deterministic sequence benchmarks.
+OpenOmicsBench is a compact benchmark collection and local sequence toolkit for bioinformatics software testing, method checks and teaching. It contains 12 certified bulk RNA-seq objects, five deterministic DNA, RNA and protein sequence fixtures, strict FASTA and FASTQ tools, and a CI-ready evaluation system.
 
-Version 2.2 adds multi-method benchmark matrices, report-to-report regression gates and portable benchmark bundles. Suite reports can be written as JSON, Markdown, JUnit XML, CSV or a self-contained HTML page.
+Version 3 evaluates RNA differential-expression results and DNA variant calls in the same run. It adds exact-SNV VCF scoring, method and resource receipts, cross-method multi-assay matrices, richer offline dashboards, and deterministic evidence crates with RO-Crate metadata and checksums.
 
 The package runs offline after installation. Sequence files stay on the local computer. The built-in tools cover inspection and lightweight preprocessing; they do not claim to replace aligners, variant callers, taxonomic classifiers or assay-specific statistical workflows.
 
@@ -54,6 +54,7 @@ List and validate the bundled benchmarks:
 omicsbench list
 omicsbench validate rnaseq-002
 omicsbench validate sequence-004
+omicsbench variant compare sequence-005 calls.vcf.gz
 ```
 
 The wheel contains the complete collection. A repository checkout and network connection are not required.
@@ -135,6 +136,26 @@ Then run:
 omicsbench suite compare results --id rnaseq-002 --id rnaseq-003 --junit reports/comparison.xml
 ```
 
+For a mixed RNA and DNA run, add benchmark-named VCF output alongside the expression tables:
+
+```text
+results/
+  method.json
+  rnaseq-002.tsv.gz
+  sequence-005.vcf.gz
+```
+
+```sh
+omicsbench suite evaluate results \
+  --id rnaseq-002 \
+  --id sequence-005 \
+  --json reports/evaluation.json \
+  --html reports/evaluation.html \
+  --evidence reports/evaluation-evidence.zip
+```
+
+`method.json` is optional. When supplied, it records the exact method version, command, container or source revision, parameters, runtime, peak memory and thread count. Missing selected outputs fail as benchmark cases.
+
 Compare several tools or parameter sets in one matrix:
 
 ```sh
@@ -146,13 +167,32 @@ omicsbench suite matrix \
   --html reports/matrix.html
 ```
 
+Use `evaluate-matrix` when each method directory contains a mixture of RNA tables and VCF calls:
+
+```sh
+omicsbench suite evaluate-matrix \
+  --method current=results/current \
+  --method candidate=results/candidate \
+  --id rnaseq-002 \
+  --id sequence-005 \
+  --html reports/multi-assay.html
+```
+
+The leaderboard uses pass counts and keeps assay metrics separate. It does not average unlike measurements into a single score.
+
 Use a previously accepted report as a regression baseline:
 
 ```sh
 omicsbench suite regress accepted.json candidate.json --absolute-tolerance 0.01 --junit reports/regression.xml
 ```
 
-Omit `--id` to require results for all 12 comparable RNA-seq objects. A completed suite exits with status 0 when every check passes, status 2 when a benchmark or regression gate fails, and status 1 for invalid input. Existing report files are protected unless `--force` is supplied. The [suite reference](docs/benchmark-suites.md) describes the file convention and report fields.
+Omit `--id` from `suite compare` to require results for all 12 comparable RNA-seq objects. Omit it from `suite evaluate` to require every benchmark with a supported comparator. A completed suite exits with status 0 when every check passes, status 2 when a benchmark or regression gate fails, and status 1 for invalid input. Existing report files are protected unless `--force` is supplied. The [suite reference](docs/benchmark-suites.md) describes the file convention and report fields.
+
+Verify an evaluation evidence crate without extracting it:
+
+```sh
+omicsbench evidence verify reports/evaluation-evidence.zip
+```
 
 ## Portable bundles
 
@@ -189,9 +229,11 @@ Profiles are available for whole-genome, exome, targeted-panel, bulk RNA-seq, si
 
 These five objects are project-authored synthetic fixtures. Each has an exact file inventory, checksums, format and molecule declarations, expected summary metrics and a deterministic rebuild workflow. `sequence-005` also verifies that every truth-set reference allele matches the bundled reference and that every alternate allele is supported by the paired reads. The objects test software behavior and do not represent a biological cohort or sequencing instrument.
 
+The version 3 VCF comparator for `sequence-005` matches contig, one-based position, REF and ALT exactly. It reports precision, recall and F1 for A/C/G/T substitutions and verifies REF against the packaged synthetic reference. Genotypes, indels, complex representations and confident-region stratification are outside this fixture. Use a haplotype-aware benchmarking engine such as [hap.py](https://github.com/Illumina/hap.py) or vcfeval with an appropriate truth set and confident regions for real germline benchmarking; the [GA4GH benchmarking project](https://github.com/ga4gh/benchmarking-tools) documents that broader problem.
+
 ## Bulk RNA-seq benchmarks
 
-Version 2 retains the complete version 1 biological collection unchanged.
+Version 3 retains the complete version 1 biological collection unchanged.
 
 | ID | Design | Samples | Pocket genes |
 |---|---|---:|---:|

openomicsbench-2.2.0/PKG-INFO → openomicsbench-3.0.0/README.md

@@ -1,41 +1,8 @@
-Metadata-Version: 2.4
-Name: openomicsbench
-Version: 2.2.0
-Summary: Traceable omics benchmarks and deterministic FASTA and FASTQ tools
-Author: Vivaan Patni
-Maintainer: Vivaan Patni
-License-Expression: Apache-2.0
-Project-URL: Homepage, https://github.com/vxxqv/openomicsbench
-Project-URL: Repository, https://github.com/vxxqv/openomicsbench
-Project-URL: Issues, https://github.com/vxxqv/openomicsbench/issues
-Project-URL: Changelog, https://github.com/vxxqv/openomicsbench/blob/main/CHANGELOG.md
-Project-URL: Zenodo, https://doi.org/10.5281/zenodo.22551734
-Project-URL: ORCID, https://orcid.org/0009-0005-1859-5107
-Keywords: bioinformatics,computational-biology,genomics,transcriptomics,proteomics,bulk-rna-seq,dna-seq,fasta,fastq,dna,rna,protein-sequence,amino-acid-analysis,sequence-analysis,quality-control,kmer,variant-truth,benchmarking,benchmark-data,test-data,data-validation,data-provenance,data-integrity,reproducibility,scientific-workflows,research-software,expression-atlas,ensembl,deseq2
-Classifier: Development Status :: 5 - Production/Stable
-Classifier: Environment :: Console
-Classifier: Intended Audience :: Science/Research
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3.14
-Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-License-File: LICENSE-METADATA
-License-File: NOTICE
-Requires-Dist: pydantic<3,>=2.13
-Requires-Dist: numpy<3,>=2.3
-Dynamic: license-file
-
 # OpenOmicsBench
 
-OpenOmicsBench is a compact benchmark collection and local sequence toolkit for bioinformatics software testing, method checks and teaching. Version 2 keeps the 12 certified bulk RNA-seq objects from version 1 and adds strict FASTA and FASTQ handling, DNA, RNA and protein support, paired-read checks, preprocessing, sequence QC and five deterministic sequence benchmarks.
+OpenOmicsBench is a compact benchmark collection and local sequence toolkit for bioinformatics software testing, method checks and teaching. It contains 12 certified bulk RNA-seq objects, five deterministic DNA, RNA and protein sequence fixtures, strict FASTA and FASTQ tools, and a CI-ready evaluation system.
 
-Version 2.2 adds multi-method benchmark matrices, report-to-report regression gates and portable benchmark bundles. Suite reports can be written as JSON, Markdown, JUnit XML, CSV or a self-contained HTML page.
+Version 3 evaluates RNA differential-expression results and DNA variant calls in the same run. It adds exact-SNV VCF scoring, method and resource receipts, cross-method multi-assay matrices, richer offline dashboards, and deterministic evidence crates with RO-Crate metadata and checksums.
 
 The package runs offline after installation. Sequence files stay on the local computer. The built-in tools cover inspection and lightweight preprocessing; they do not claim to replace aligners, variant callers, taxonomic classifiers or assay-specific statistical workflows.
 
@@ -54,6 +21,7 @@ List and validate the bundled benchmarks:
 omicsbench list
 omicsbench validate rnaseq-002
 omicsbench validate sequence-004
+omicsbench variant compare sequence-005 calls.vcf.gz
 ```
 
 The wheel contains the complete collection. A repository checkout and network connection are not required.
@@ -135,6 +103,26 @@ Then run:
 omicsbench suite compare results --id rnaseq-002 --id rnaseq-003 --junit reports/comparison.xml
 ```
 
+For a mixed RNA and DNA run, add benchmark-named VCF output alongside the expression tables:
+
+```text
+results/
+  method.json
+  rnaseq-002.tsv.gz
+  sequence-005.vcf.gz
+```
+
+```sh
+omicsbench suite evaluate results \
+  --id rnaseq-002 \
+  --id sequence-005 \
+  --json reports/evaluation.json \
+  --html reports/evaluation.html \
+  --evidence reports/evaluation-evidence.zip
+```
+
+`method.json` is optional. When supplied, it records the exact method version, command, container or source revision, parameters, runtime, peak memory and thread count. Missing selected outputs fail as benchmark cases.
+
 Compare several tools or parameter sets in one matrix:
 
 ```sh
@@ -146,13 +134,32 @@ omicsbench suite matrix \
   --html reports/matrix.html
 ```
 
+Use `evaluate-matrix` when each method directory contains a mixture of RNA tables and VCF calls:
+
+```sh
+omicsbench suite evaluate-matrix \
+  --method current=results/current \
+  --method candidate=results/candidate \
+  --id rnaseq-002 \
+  --id sequence-005 \
+  --html reports/multi-assay.html
+```
+
+The leaderboard uses pass counts and keeps assay metrics separate. It does not average unlike measurements into a single score.
+
 Use a previously accepted report as a regression baseline:
 
 ```sh
 omicsbench suite regress accepted.json candidate.json --absolute-tolerance 0.01 --junit reports/regression.xml
 ```
 
-Omit `--id` to require results for all 12 comparable RNA-seq objects. A completed suite exits with status 0 when every check passes, status 2 when a benchmark or regression gate fails, and status 1 for invalid input. Existing report files are protected unless `--force` is supplied. The [suite reference](docs/benchmark-suites.md) describes the file convention and report fields.
+Omit `--id` from `suite compare` to require results for all 12 comparable RNA-seq objects. Omit it from `suite evaluate` to require every benchmark with a supported comparator. A completed suite exits with status 0 when every check passes, status 2 when a benchmark or regression gate fails, and status 1 for invalid input. Existing report files are protected unless `--force` is supplied. The [suite reference](docs/benchmark-suites.md) describes the file convention and report fields.
+
+Verify an evaluation evidence crate without extracting it:
+
+```sh
+omicsbench evidence verify reports/evaluation-evidence.zip
+```
 
 ## Portable bundles
 
@@ -189,9 +196,11 @@ Profiles are available for whole-genome, exome, targeted-panel, bulk RNA-seq, si
 
 These five objects are project-authored synthetic fixtures. Each has an exact file inventory, checksums, format and molecule declarations, expected summary metrics and a deterministic rebuild workflow. `sequence-005` also verifies that every truth-set reference allele matches the bundled reference and that every alternate allele is supported by the paired reads. The objects test software behavior and do not represent a biological cohort or sequencing instrument.
 
+The version 3 VCF comparator for `sequence-005` matches contig, one-based position, REF and ALT exactly. It reports precision, recall and F1 for A/C/G/T substitutions and verifies REF against the packaged synthetic reference. Genotypes, indels, complex representations and confident-region stratification are outside this fixture. Use a haplotype-aware benchmarking engine such as [hap.py](https://github.com/Illumina/hap.py) or vcfeval with an appropriate truth set and confident regions for real germline benchmarking; the [GA4GH benchmarking project](https://github.com/ga4gh/benchmarking-tools) documents that broader problem.
+
 ## Bulk RNA-seq benchmarks
 
-Version 2 retains the complete version 1 biological collection unchanged.
+Version 3 retains the complete version 1 biological collection unchanged.
 
 | ID | Design | Samples | Pocket genes |
 |---|---|---:|---:|

{openomicsbench-2.2.0 → openomicsbench-3.0.0}/datasets/sequences/sequence-005/expected/validation.json

@@ -74,5 +74,11 @@
   "reference": "nano/reference.fasta",
   "variant_truth": "expected/variants.tsv",
   "variant_count": 3,
-  "context_bases": 10
+  "context_bases": 10,
+  "variant_comparison": {
+    "mode": "exact_allele",
+    "scope": "single_nucleotide_substitutions",
+    "minimum_precision": 1.0,
+    "minimum_recall": 1.0
+  }
 }

{openomicsbench-2.2.0 → openomicsbench-3.0.0}/datasets/sequences/sequence-005/manifest.json

@@ -2,7 +2,7 @@
   "schema_version": "2.0",
   "id": "sequence-005",
   "title": "Synthetic paired DNA-seq variant benchmark",
-  "release": "2.0.0",
+  "release": "3.0.0",
   "assay": "whole_genome_dna_seq",
   "kind": "synthetic_fixture",
   "status": "validated",
@@ -93,13 +93,13 @@
       "tier": "expected",
       "role": "metrics",
       "media_type": "application/json",
-      "bytes": 1739,
-      "sha256": "9b21fb89fe782b9e7ced92f0bebba8e84f4d3e50188e7af18ad2a429ba87b9bb"
+      "bytes": 1902,
+      "sha256": "da0813bf715df2a455a7c0eb86dc804dbb8cf582c9aa09119e8d976e3548cd88"
     }
   ],
   "validation": {
     "profile": "expected/validation.json",
-    "baseline_version": "dnaseq-truth-v1",
+    "baseline_version": "dnaseq-truth-v2",
     "metrics": []
   },
   "sequence": {

{openomicsbench-2.2.0 → openomicsbench-3.0.0}/pyproject.toml

@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "openomicsbench"
-version = "2.2.0"
-description = "Traceable omics benchmarks and deterministic FASTA and FASTQ tools"
+version = "3.0.0"
+description = "Traceable multi-assay benchmarks, VCF evaluation and deterministic sequence tools"
 readme = {file = "README.md", content-type = "text/markdown"}
 license = "Apache-2.0"
 authors = [{name = "Vivaan Patni"}]
 maintainers = [{name = "Vivaan Patni"}]
-keywords = ["bioinformatics", "computational-biology", "genomics", "transcriptomics", "proteomics", "bulk-rna-seq", "dna-seq", "fasta", "fastq", "dna", "rna", "protein-sequence", "amino-acid-analysis", "sequence-analysis", "quality-control", "kmer", "variant-truth", "benchmarking", "benchmark-data", "test-data", "data-validation", "data-provenance", "data-integrity", "reproducibility", "scientific-workflows", "research-software", "expression-atlas", "ensembl", "deseq2"]
+keywords = ["bioinformatics", "computational-biology", "genomics", "transcriptomics", "proteomics", "bulk-rna-seq", "dna-seq", "fasta", "fastq", "vcf", "dna", "rna", "protein-sequence", "amino-acid-analysis", "sequence-analysis", "quality-control", "kmer", "variant-truth", "variant-benchmarking", "multi-assay", "ro-crate", "benchmarking", "benchmark-data", "test-data", "data-validation", "data-provenance", "data-integrity", "reproducibility", "scientific-workflows", "research-software", "expression-atlas", "ensembl", "deseq2"]
 requires-python = ">=3.11"
 dependencies = ["pydantic>=2.13,<3", "numpy>=2.3,<3"]
 classifiers = [

{openomicsbench-2.2.0 → openomicsbench-3.0.0}/src/omicsbench/__init__.py

@@ -1,2 +1,2 @@
 """Discovery, validation and local analysis of compact omics benchmarks."""
-__version__ = "2.2.0"
+__version__ = "3.0.0"

{openomicsbench-2.2.0 → openomicsbench-3.0.0}/src/omicsbench/cli.py

@@ -16,6 +16,9 @@ from .matrix import compare_matrix, parse_method
 from .regression import compare_reports
 from .bundle import create_bundle, verify_bundle
 from .validate import validate
+from .variants import compare_variants
+from .evaluate import evaluate_matrix, evaluate_suite
+from .evidence import create_evidence_crate, verify_evidence_crate
 
 def main():
     p = argparse.ArgumentParser(description="Find, verify and process compact omics benchmarks.",epilog="Example: omicsbench info rnaseq-002")
@@ -37,6 +40,12 @@ def main():
             command.add_argument("results",type=Path,help="CSV or TSV file with gene_id and log2_fold_change columns.")
             command.add_argument("--detail-limit",type=int,default=20,help="Maximum missing and unexpected gene examples to return.")
     add_sequence_parser(sub)
+    variant = sub.add_parser("variant", help="Compare VCF calls with a bundled small-variant truth set.")
+    variant_sub = variant.add_subparsers(dest="variant_command", required=True)
+    variant_compare = variant_sub.add_parser("compare", help="Score exact SNV alleles in a VCF or VCF.GZ file.")
+    variant_compare.add_argument("id", help="Benchmark ID with a declared variant truth set.")
+    variant_compare.add_argument("results", type=Path, help="VCF or VCF.GZ call set.")
+    variant_compare.add_argument("--detail-limit", type=int, default=20, help="Maximum false-positive and false-negative examples to return.")
     assay = sub.add_parser("assay", help="Inspect sequencing assay profiles and build local command plans.")
     assay_sub = assay.add_subparsers(dest="assay_command", required=True)
     assay_sub.add_parser("list", help="List supported assay profiles.")
@@ -54,17 +63,23 @@ def main():
     suite_compare = suite_sub.add_parser("compare", help="Compare a directory of differential-expression results.")
     suite_compare.add_argument("results", type=Path, help="Directory containing files named <dataset-id>.csv or .tsv, optionally gzip-compressed.")
     suite_compare.add_argument("--detail-limit", type=int, default=20, help="Maximum missing and unexpected gene examples per benchmark.")
+    suite_evaluate = suite_sub.add_parser("evaluate", help="Evaluate one result directory across RNA and DNA benchmarks.")
+    suite_evaluate.add_argument("results", type=Path, help="Directory containing benchmark-named RNA tables and VCF files.")
+    suite_evaluate.add_argument("--detail-limit", type=int, default=20, help="Maximum mismatch examples per benchmark.")
     suite_matrix = suite_sub.add_parser("matrix", help="Compare several analysis methods across the same benchmarks.")
     suite_matrix.add_argument("--method", action="append", required=True, help="Method and result directory as NAME=PATH. Repeat for every method.")
     suite_matrix.add_argument("--detail-limit", type=int, default=20, help="Maximum missing and unexpected gene examples per benchmark.")
+    suite_evaluate_matrix = suite_sub.add_parser("evaluate-matrix", help="Compare several methods across RNA and DNA benchmarks.")
+    suite_evaluate_matrix.add_argument("--method", action="append", required=True, help="Method and result directory as NAME=PATH. Repeat for every method.")
+    suite_evaluate_matrix.add_argument("--detail-limit", type=int, default=20, help="Maximum mismatch examples per benchmark.")
     suite_regress = suite_sub.add_parser("regress", help="Fail when a candidate suite report regresses from a baseline report.")
     suite_regress.add_argument("baseline", type=Path, help="Previously accepted JSON suite report.")
     suite_regress.add_argument("candidate", type=Path, help="Candidate JSON suite report to check.")
     suite_regress.add_argument("--absolute-tolerance", type=float, default=0.0, help="Largest permitted absolute decrease in a tracked metric.")
     suite_regress.add_argument("--allow-missing", action="store_true", help="Do not fail when a baseline case is absent from the candidate.")
-    for command in (suite_validate, suite_compare, suite_matrix):
+    for command in (suite_validate, suite_compare, suite_evaluate, suite_matrix, suite_evaluate_matrix):
         command.add_argument("--id", action="append", default=[], help="Benchmark ID to include. Repeat to select several; omit to run all eligible benchmarks.")
-    for command in (suite_validate, suite_compare, suite_matrix, suite_regress):
+    for command in (suite_validate, suite_compare, suite_evaluate, suite_matrix, suite_evaluate_matrix, suite_regress):
         command.add_argument("--json", type=Path, help="Write the complete report as JSON.")
         command.add_argument("--markdown", type=Path, help="Write a concise Markdown report.")
         command.add_argument("--junit", type=Path, help="Write a JUnit XML report for CI systems.")
@@ -80,10 +95,21 @@ def main():
     bundle_create.add_argument("--force", action="store_true", help="Replace an existing bundle.")
     bundle_verify = bundle_sub.add_parser("verify", help="Verify a bundle inventory, hashes and dataset manifests.")
     bundle_verify.add_argument("archive", type=Path)
+    evidence = sub.add_parser("evidence", help="Verify a portable evaluation evidence crate.")
+    evidence_sub = evidence.add_subparsers(dest="evidence_command", required=True)
+    evidence_verify = evidence_sub.add_parser("verify", help="Verify checksums, inventory, report and RO-Crate metadata.")
+    evidence_verify.add_argument("archive", type=Path)
+    for command in (suite_evaluate, suite_evaluate_matrix):
+        command.add_argument("--evidence", type=Path, help="Write a deterministic ZIP with the reports, submitted results and RO-Crate metadata.")
     args = p.parse_args()
     try:
         if args.command == "seq":
             out = run_sequence_command(args)
+        elif args.command == "variant":
+            model, folder = lookup(args.root, args.id)
+            out = compare_variants(model, folder, args.results, args.detail_limit)
+        elif args.command == "evidence":
+            out = verify_evidence_crate(args.archive)
         elif args.command == "bundle":
             if args.bundle_command == "create":
                 out = create_bundle(args.root, args.output, args.id, args.assay, args.force)
@@ -101,11 +127,15 @@ def main():
                 out = validate_suite(args.root, args.assay, args.id)
             elif args.suite_command == "compare":
                 out = compare_suite(args.root, args.results, args.id, args.detail_limit)
+            elif args.suite_command == "evaluate":
+                out = evaluate_suite(args.root, args.results, args.id, args.detail_limit)
             elif args.suite_command == "matrix":
                 out = compare_matrix(args.root, [parse_method(value) for value in args.method], args.id, args.detail_limit)
+            elif args.suite_command == "evaluate-matrix":
+                out = evaluate_matrix(args.root, [parse_method(value) for value in args.method], args.id, args.detail_limit)
             else:
                 out = compare_reports(args.baseline, args.candidate, args.absolute_tolerance, args.allow_missing)
-            out["reports"] = write_reports(
+            reports = write_reports(
                 out,
                 json_path=args.json,
                 markdown_path=args.markdown,
@@ -114,6 +144,9 @@ def main():
                 csv_path=args.csv,
                 html_path=args.html,
             )
+            if getattr(args, "evidence", None) is not None:
+                reports["evidence"] = create_evidence_crate(out, args.evidence, args.force)
+            out["reports"] = reports
         elif args.command == "list":
             out = [{"id":m.id,"title":m.title,"kind":m.kind,"status":m.status,"rights":m.rights.status,"tiers":sorted({f.tier for f in m.files})} for m,_ in registry(args.root).values() if not args.assay or m.assay==args.assay]
         elif args.command == "doctor":
@@ -135,6 +168,8 @@ def main():
         print(json.dumps(out,indent=2,allow_nan=False))
         if args.command == "compare" and out["status"] == "fail":
             sys.exit(2)
+        if args.command == "variant" and out["status"] == "fail":
+            sys.exit(2)
         if args.command == "suite" and out["summary"]["status"] == "fail":
             sys.exit(2)
     except (ValueError,OSError,KeyError) as exc:

openomicsbench-3.0.0/src/omicsbench/evaluate.py

@@ -0,0 +1,211 @@
+"""Unified evaluation of RNA effects and exact synthetic SNV calls."""
+from __future__ import annotations
+
+import json
+import math
+import re
+from pathlib import Path
+
+from pydantic import Field, field_validator
+
+from .compare import compare
+from .matrix import METHOD_NAME
+from .models import StrictModel
+from .registry import registry
+from .suite import RESULT_SUFFIXES, _summary
+from .variants import compare_variants
+
+
+VCF_SUFFIXES = (".vcf", ".vcf.gz")
+
+
+class MethodReceipt(StrictModel):
+    schema_version: str = Field(pattern=r"^1\.0$")
+    name: str = Field(min_length=1, max_length=128)
+    version: str = Field(min_length=1, max_length=128)
+    description: str | None = Field(default=None, max_length=500)
+    command: str | None = Field(default=None, max_length=4000)
+    container: str | None = Field(default=None, max_length=500)
+    source_revision: str | None = Field(default=None, max_length=128)
+    runtime_seconds: float | None = Field(default=None, ge=0)
+    peak_memory_mb: float | None = Field(default=None, ge=0)
+    threads: int | None = Field(default=None, gt=0, strict=True)
+    parameters: dict[str, str | int | float | bool | None] = Field(default_factory=dict)
+
+    @field_validator("runtime_seconds", "peak_memory_mb")
+    @classmethod
+    def finite_resource_value(cls, value):
+        if value is not None and not math.isfinite(value):
+            raise ValueError("resource measurements must be finite")
+        return value
+
+
+def read_method_receipt(directory: Path) -> dict | None:
+    path = Path(directory) / "method.json"
+    if not path.exists():
+        return None
+    if not path.is_file():
+        raise ValueError(f"{path}: method receipt is not a file")
+    try:
+        receipt = MethodReceipt.model_validate_json(path.read_text(encoding="utf-8"))
+    except (ValueError, UnicodeDecodeError) as error:
+        raise ValueError(f"{path}: invalid method receipt: {error}") from None
+    return {"path": str(path), **receipt.model_dump(mode="json")}
+
+
+def _candidate_files(directory: Path, dataset_id: str, suffixes: tuple[str, ...]) -> list[Path]:
+    return [directory / f"{dataset_id}{suffix}" for suffix in suffixes if (directory / f"{dataset_id}{suffix}").is_file()]
+
+
+def _result_file(directory: Path, dataset_id: str, suffixes: tuple[str, ...]) -> Path | None:
+    matches = _candidate_files(directory, dataset_id, suffixes)
+    if len(matches) > 1:
+        raise ValueError(f"{dataset_id}: multiple result files found: {', '.join(path.name for path in matches)}")
+    return matches[0] if matches else None
+
+
+def _evaluator(model) -> tuple[str, tuple[str, ...]] | None:
+    if model.assay == "bulk_rna_seq" and model.kind == "real" and model.validation is not None:
+        return "differential_expression", RESULT_SUFFIXES
+    if model.assay == "whole_genome_dna_seq" and model.validation is not None:
+        return "exact_snv", VCF_SUFFIXES
+    return None
+
+
+def _selection(root: Path, dataset_ids: list[str] | None) -> list[tuple]:
+    records = registry(Path(root))
+    requested = list(dict.fromkeys(dataset_ids or []))
+    unknown = [dataset_id for dataset_id in requested if dataset_id not in records]
+    if unknown:
+        raise ValueError(f"unknown dataset IDs: {', '.join(unknown)}")
+    if requested:
+        unsupported = [dataset_id for dataset_id in requested if _evaluator(records[dataset_id][0]) is None]
+        if unsupported:
+            raise ValueError(f"no result comparator for: {', '.join(unsupported)}")
+        return [records[dataset_id] for dataset_id in requested]
+    selected = [record for record in records.values() if _evaluator(record[0]) is not None]
+    if not selected:
+        raise ValueError("the collection has no comparable benchmarks")
+    return selected
+
+
+def _assay_summary(results: list[dict]) -> dict:
+    assays = {}
+    for assay in sorted({item["assay"] for item in results}):
+        assays[assay] = _summary([item for item in results if item["assay"] == assay])
+    return assays
+
+
+def evaluate_suite(
+    root: Path,
+    results_directory: Path,
+    dataset_ids: list[str] | None = None,
+    detail_limit: int = 20,
+) -> dict:
+    directory = Path(results_directory)
+    if not directory.is_dir():
+        raise ValueError(f"{directory}: results directory does not exist")
+    if detail_limit < 0:
+        raise ValueError("detail limit must be zero or greater")
+    receipt = read_method_receipt(directory)
+    results = []
+    for model, folder in _selection(Path(root), dataset_ids):
+        comparator, suffixes = _evaluator(model)  # type: ignore[misc]
+        try:
+            path = _result_file(directory, model.id, suffixes)
+            if path is None:
+                expected = " or ".join(f"{model.id}{suffix}" for suffix in suffixes)
+                results.append({
+                    "id": model.id,
+                    "assay": model.assay,
+                    "comparator": comparator,
+                    "status": "fail",
+                    "reason": f"missing result file; expected {expected}",
+                })
+                continue
+            details = compare(model, folder, path, detail_limit) if comparator == "differential_expression" else compare_variants(model, folder, path, detail_limit)
+            item = {
+                "id": model.id,
+                "assay": model.assay,
+                "comparator": comparator,
+                "status": details["status"],
+                "input": str(path),
+                "details": details,
+            }
+            if details["status"] == "fail":
+                item["reason"] = "; ".join(details["reasons"])
+            results.append(item)
+        except (ValueError, OSError, KeyError) as error:
+            results.append({
+                "id": model.id,
+                "assay": model.assay,
+                "comparator": comparator,
+                "status": "fail",
+                "reason": str(error),
+            })
+    summary = _summary(results)
+    summary["assays"] = _assay_summary(results)
+    return {
+        "operation": "evaluate",
+        "selection": {"ids": dataset_ids or [], "default": "all comparable benchmarks"},
+        "results_directory": str(directory),
+        "method": receipt,
+        "summary": summary,
+        "results": results,
+    }
+
+
+def _leader(name: str, directory: Path, report: dict) -> dict:
+    summary = report["summary"]
+    receipt = report.get("method")
+    return {
+        "name": name,
+        "results_directory": str(directory),
+        "receipt": receipt,
+        "status": summary["status"],
+        "total": summary["total"],
+        "passed": summary["passed"],
+        "failed": summary["failed"],
+        "skipped": summary["skipped"],
+        "pass_rate": summary["passed"] / summary["total"] if summary["total"] else 0.0,
+        "assays": summary["assays"],
+    }
+
+
+def evaluate_matrix(
+    root: Path,
+    methods: list[tuple[str, Path]],
+    dataset_ids: list[str] | None = None,
+    detail_limit: int = 20,
+) -> dict:
+    if len(methods) < 2:
+        raise ValueError("a multi-assay matrix requires at least two --method entries")
+    names = [name for name, _ in methods]
+    if len(names) != len(set(names)):
+        raise ValueError("method names must be unique")
+    for name in names:
+        if not METHOD_NAME.fullmatch(name):
+            raise ValueError(f"{name}: invalid method name")
+    cells = []
+    leaders = []
+    benchmark_ids = None
+    for name, directory in methods:
+        report = evaluate_suite(root, directory, dataset_ids, detail_limit)
+        ids = [item["id"] for item in report["results"]]
+        if benchmark_ids is None:
+            benchmark_ids = ids
+        elif ids != benchmark_ids:
+            raise ValueError(f"{name}: benchmark selection differs from the first method")
+        cells.extend({"method": name, **item} for item in report["results"])
+        leaders.append(_leader(name, directory, report))
+    leaderboard = sorted(leaders, key=lambda item: (-item["passed"], item["failed"], item["name"]))
+    summary = _summary(cells)
+    summary.update({"methods": len(methods), "benchmarks": len(benchmark_ids or []), "assays": _assay_summary(cells)})
+    return {
+        "operation": "evaluate_matrix",
+        "selection": {"ids": dataset_ids or [], "default": "all comparable benchmarks"},
+        "ranking": "passed cases descending, failed cases ascending, then method name; assay metrics are not pooled",
+        "summary": summary,
+        "leaderboard": leaderboard,
+        "results": cells,
+    }