@galaxyproject/orbit 0.0.0 → 0.9.0

package/extensions/orbit/vendor/skills/changeset-to-galaxy-test-plan/references/notes/iwc-test-data-conventions.md

@@ -0,0 +1,308 @@
+---
+type: research
+tags:
+  - target/galaxy
+status: draft
+created: 2026-04-30
+revised: 2026-05-03
+revision: 3
+related_notes:
+  - "galaxy-workflow-testability-design"
+  - "iwc-shortcuts-anti-patterns"
+  - "planemo-asserts-idioms"
+  - "implement-galaxy-workflow-test"
+  - "tests-format"
+  - "iwc-tabular-operations-survey"
+summary: "How IWC workflows organize and reference test data — Zenodo-first, SHA-1 integrity, collection shapes, CVMFS gotchas."
+---
+
+# IWC test data conventions
+
+Reference for an agent implementing or editing a `<workflow>-tests.yml` in IWC style. All evidence cited from `https://github.com/galaxyproject/iwc/blob/main/workflows/` (raw IWC clone) and `workflows/README.md`. Authoritative spec: [planemo.readthedocs.io/en/latest/test_format.html](https://planemo.readthedocs.io/en/latest/test_format.html). The companion analysis at `https://github.com/jmchilton/galaxy-brain/blob/main/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md` is the synthesized source for several normative claims here.
+
+This note owns **test YAML fixture shapes**. For workflow-structure choices that make those fixtures possible before the test file exists, use galaxy-workflow-testability-design.
+
+## 1. Where does test data live? Remote vs in-repo
+
+Two storage patterns. They mix freely inside one job.
+
+**Remote `location:` (default for any non-trivial input).** Strongly preferred for anything bigger than a toy fixture. Order of preference, observed in the corpus:
+
+- **Zenodo** — overwhelming default, persistent DOI-backed URL.
+  - `read-preprocessing/short-read-qc-trimming/short-read-quality-control-and-trimming-tests.yml:13,17` — `https://zenodo.org/records/11484215/files/paired_r1.fastq.gz` (note the modern `/records/` plural form; older entries use `/record/`).
+  - `metabolomics/lcms-preprocessing/Mass_spectrometry__LC-MS_preprocessing_with_XCMS-tests.yml:6,17,24,...` — 11+ mzML files all hosted at `zenodo.org/record/10130758/files/...`.
+  - `sars-cov-2-variant-calling/sars-cov-2-consensus-from-variation/consensus-from-variation-tests.yml:5` — `https://zenodo.org/record/4555735/files/NC_045512.2_reference.fasta?download=1`.
+  - `repeatmasking/RepeatMasking-Workflow-tests.yml:5` — `https://zenodo.org/record/8364146/files/eco.fasta?download=1`. The `?download=1` query suffix is acceptable but optional; both forms appear.
+- **EBI/ENA REST** — for canonical reference sequences.
+  - `virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml:5` — `https://www.ebi.ac.uk/ena/browser/api/fasta/AF325528.1?download=true`.
+- **SRA / ENA FTP** — for raw read accessions where re-uploading to Zenodo is wasteful.
+  - `pox-virus-half-genome-tests.yml:27,34,49,56` — `ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/076/SRR15145276/SRR15145276_1.fastq.gz`.
+
+**In-repo `path: test-data/...` (toy fixtures + small expected outputs only).** Used when:
+
+- File is small enough to commit (rule of thumb from corpus: low single-digit MB or smaller).
+- File is a hand-crafted toy or trimmed reference (e.g. a 5-element FASTA for HyPhy).
+- File is the *expected output* baseline for a `file:` exact-comparison assertion: `consensus-from-variation-tests.yml:31` — `file: test-data/masked_consensus.fa`.
+
+`workflows/README.md:67` makes the contract explicit: "try to generate a toy dataset … and then publish it to zenodo to have a permanent URL." `workflows/README.md:98`: "if some outputs are large, it is better to use assertions than storing the whole output file to the iwc repository."
+
+Subdirectory layout inside `test-data/` is free-form and used to organize related fixtures, e.g. `comparative_genomics/hyphy/test-data/unaligned_seqs/`, `test-data/codon_alignments/`, `test-data/iqtree_trees/` (referenced from `hyphy-core-tests.yml:13,17,21,...`).
+
+**Decision rule for an agent:**
+
+1. Expected-output baseline that's small — commit to `test-data/`.
+2. Toy/trimmed input < ~5 MB — commit to `test-data/`.
+3. Anything else — Zenodo (or EBI/SRA for canonical accessions). Add `hashes:` SHA-1.
+4. If the input is a reference index resolvable from CVMFS (`hg38`, `dm6`, `mm10`, …), use the plain string form. See section 5.
+
+## 2. The `class: File` block — exact YAML shapes
+
+All snippets verbatim from the IWC corpus.
+
+### 2a. Single remote file (with integrity hash)
+
+`virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml:3-9`:
+
+```yaml
+Reference FASTA:
+  class: File
+  location: "https://www.ebi.ac.uk/ena/browser/api/fasta/AF325528.1?download=true"
+  filetype: fasta
+  hashes:
+  - hash_function: SHA-1
+    hash_value: 927d0d00b7db6ad60524bb9e50d3ab41c4ac5ecf
+```
+
+Field order is conventional but not enforced. `filetype:` is the Galaxy datatype name (`fasta`, `fastqsanger.gz`, `mzml`, `bed`, `tabular`, `csv`, `gtf`, `vcf`, …). `location:` URLs may be quoted or bare; both forms occur.
+
+### 2b. Single local file
+
+`comparative_genomics/hyphy/hyphy-core-tests.yml:3-6`:
+
+```yaml
+reference cds:
+  class: File
+  path: test-data/denv1_ref_cds.fasta
+  filetype: fasta
+```
+
+`path:` is relative to the workflow directory (the directory containing the `-tests.yml`). Hashes are usually omitted on local files — the file is in-tree, so integrity is implicit. They are nonetheless added in some workflows: `consensus-from-variation-tests.yml:15-18` shows `path: test-data/aligned_reads_for_coverage.bam` paired with a SHA-1 hash. The pattern appears to be: hashes added when the `-tests.yml` was generated by `planemo workflow_test_init --from_invocation`, which mints them automatically.
+
+A dataset attribute can be set on a single file to force a `dbkey`, e.g. `epigenetics/consensus-peaks/consensus-peaks-chip-pe-tests.yml:7-10`:
+
+```yaml
+- class: File
+  identifier: rep1
+  path: test-data/rep1.bam
+  dbkey: mm10
+```
+
+`dbkey:` and `decompress: true` (`scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:30`) are documented per-file attributes that ride alongside `class: File`.
+
+### 2c. List collection
+
+`hyphy-core-tests.yml:7-30`:
+
+```yaml
+unaligned sequences:
+  class: Collection
+  collection_type: list
+  elements:
+  - identifier: AB178040.1|2002
+    class: File
+    path: test-data/unaligned_seqs/AB178040.1|2002.fasta
+    filetype: fasta
+  - identifier: AB195673.1|2003
+    class: File
+    path: test-data/unaligned_seqs/AB195673.1|2003.fasta
+    filetype: fasta
+  ...
+```
+
+Each element is a single-file dict with an `identifier:`. A larger remote-only example with hashes per element: `metabolomics/lcms-preprocessing/Mass_spectrometry__LC-MS_preprocessing_with_XCMS-tests.yml:11-22`:
+
+```yaml
+Mass-spectrometry Dataset Collection:
+  class: Collection
+  collection_type: list
+  elements:
+  - class: File
+    identifier: Blanc05.mzML
+    location: https://zenodo.org/record/10130758/files/Blanc05.mzML
+    filetype: mzml
+    hashes:
+    - hash_function: SHA-1
+      hash_value: e98d5a4cd8849a145a032bdc31c348ad97cb59c3
+```
+
+### 2d. Paired collection (single pair)
+
+A `paired` collection is a Collection whose two elements have identifiers `forward` and `reverse`. In IWC it is rarely used at top level — it is almost always wrapped as a single-element `list:paired` (see 2e). When it appears bare, the shape is the inner block of 2e without the outer `list:paired` wrapper.
+
+### 2e. `list:paired` collection
+
+`read-preprocessing/short-read-qc-trimming/short-read-quality-control-and-trimming-tests.yml:3-18`:
+
+```yaml
+Raw reads:
+  class: Collection
+  collection_type: list:paired
+  elements:
+  - class: Collection
+    type: paired
+    identifier: pair
+    elements:
+    - class: File
+      identifier: forward
+      location: https://zenodo.org/records/11484215/files/paired_r1.fastq.gz
+      filetype: fastqsanger.gz
+    - class: File
+      identifier: reverse
+      location: https://zenodo.org/records/11484215/files/paired_r2.fastq.gz
+      filetype: fastqsanger.gz
+```
+
+Note: outer `collection_type: list:paired`, inner `type: paired` (not `collection_type:`). Inner element identifiers are the literal strings `forward` and `reverse` — these are reserved.
+
+A multi-pair list:paired with full hashes: `virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml:17-38` or `scRNAseq/fastq-to-matrix-10x/scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:3-24`.
+
+### 2f. `list:list:paired` and deeper
+
+Not directly observed in sampled IWC inputs. The shape is recursive — wrap a `list:paired` element list inside another `class: Collection, collection_type: list:list:paired`, where each outer element is `class: Collection, type: list:paired` (analogous to the `list:paired` → `paired` nesting in 2e). For deeply-nested collections planemo's [`_writing_collections.rst`](https://github.com/galaxyproject/planemo/blob/master/docs/_writing_collections.rst) is the de-facto reference.
+
+Output side: nested-collection assertions DO exist in the corpus. `scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:82-98`:
+
+```yaml
+Seurat input for gene expression (filtered):
+  attributes: {collection_type: list:list}
+  element_tests:
+    subsample:
+      elements:
+        matrix:
+          asserts:
+            has_line:
+              line: "23932 3 1171"
+        barcodes:
+          asserts:
+            has_line:
+              line: "CACATGATCATAAGGA"
+```
+
+Pattern: outer `element_tests:` keyed by outer identifier; inner `elements:` (note plural, not `element_tests:` here) keyed by inner identifier. Optional `attributes: {collection_type: ...}` asserts the shape of the produced collection.
+
+### 2g. `composite_data:`
+
+**Not observed** in the sampled IWC corpus (`grep -rln "composite\|imzml\|primary_file\|extra_files" workflows/ --include="*-tests.yml"` returns nothing). Per the planemo spec the shape is a `composite_data:` mapping listing each component file path, but an agent emitting one in IWC style has no in-corpus reference and should treat it as unproven territory — fall back to the planemo docs and verify against `galaxy.tool_util.parser.yaml`.
+
+### 2h. CWL-style shorthand (list of File dicts without identifiers)
+
+Documented in planemo; not observed in sampled IWC. IWC always supplies explicit identifiers.
+
+### 2i. `tags:` on elements (Galaxy 20.09+)
+
+Documented in planemo; not surfaced in sampled IWC `-tests.yml` files (`grep "  tags:" --include="*-tests.yml"` empty across the sampled set). Practical absence — IWC tests rely on identifier matching, not tags.
+
+## 3. SHA-1 integrity hashes — when, when not, why
+
+**Format.** Always a list under `hashes:`, even for one entry. Each entry is `{hash_function: SHA-1, hash_value: <40-hex>}`. Verbatim from `pox-virus-half-genome-tests.yml:7-9`:
+
+```yaml
+hashes:
+- hash_function: SHA-1
+  hash_value: 927d0d00b7db6ad60524bb9e50d3ab41c4ac5ecf
+```
+
+Only `SHA-1` observed in the IWC corpus. The planemo spec also accepts `MD5`, `SHA-256`, `SHA-512` as `hash_function:` values; IWC does not exercise these.
+
+**When present:**
+
+- Every remote `location:`-fetched input in the sampled corpus carries a SHA-1 (Zenodo, EBI REST, SRA FTP). The hash guards against silent upstream corruption — Zenodo records are immutable but bit-rot and CDN-edge errors do happen; ENA FASTA endpoints can quietly change line-wrap.
+- Many local `path:`-loaded inputs also carry a SHA-1. This is `--from_invocation` autogeneration leaking through (planemo emits hashes for every input it captured).
+- `consensus-from-variation-tests.yml:6-8,17-18,27-28` — every input in the test, local and remote, has a hash.
+
+**When omitted:**
+
+- Hand-written local-file fixtures often skip hashes — `hyphy-core-tests.yml:3-30` has zero `hashes:` blocks across six local-file inputs.
+- Output assertions: `file:`-comparison outputs and `location:`-comparison outputs (`RepeatMasking-Workflow-tests.yml:13`) **do not** carry `hashes:`. The integrity story for outputs is handled by `compare:` / `asserts:` instead.
+
+**Why bother on local files.** Catches accidental corruption when collaborators hand-edit `test-data/` fixtures.
+
+## 4. Element identifier conventions
+
+Identifiers are arbitrary strings, used both as the YAML key (`element_tests:`) and as the produced Galaxy collection element name. Conventions surfaced from the corpus:
+
+- **Pipes are legal** and survive YAML round-trip without quoting on the input side: `hyphy-core-tests.yml:11` — `identifier: AB178040.1|2002`. On the *output* side they're quoted because they appear as YAML mapping keys: `hyphy-core-tests.yml:34` — `"NC_001477.1|capsid_protein_C|95-394_DENV1":`. Quote when the identifier contains characters that would otherwise start a YAML flow node (`{`, `[`, `&`, `*`, `!`, `|`, `>`, `'`, `"`, `%`, `@`, `` ` ``) or look like a number/bool.
+- **Dots, hyphens, underscores** — common, no quoting needed: `Blanc05.mzML`, `HU_neg_048.mzML`, `SRR11578257`, `Rep1`, `subsample`, `pair`.
+- **Reserved inside `paired` collections:** `forward` and `reverse` (lowercase). Never use these names for outer identifiers in `list:paired`.
+- **Inline comments after identifier** are legal and used for provenance: `pox-virus-half-genome-tests.yml:23` — `identifier: 20L70   # SRR15145276`.
+- **Whitespace and special chars in workflow input *labels*** survive too — they are job-mapping keys, not collection identifiers, but the same YAML quoting rules apply. Example from the analysis doc: `Manually annotate celltypes?: true` as a job key in `Preprocessing-and-Clustering-of-single-cell-RNA-seq-data-with-Scanpy-tests.yml:23`.
+
+### 4a. Workflow-interface implications
+
+Fixture shape should feed back into workflow input design. Tests supply job inputs by workflow input label, and collection fixtures must match the workflow's declared collection type.
+
+- The 10x CellPlex test uses `fastq PE collection GEX`, `fastq PE collection CMO`, and `sample name and CMO sequence collection` as job keys with `list:paired` and `list` collection fixtures; the workflow declares matching collection inputs at `$IWC_FORMAT2/scRNAseq/fastq-to-matrix-10x/scrna-seq-fastq-to-matrix-10x-cellplex.gxwf.yml:4-72`.
+- CVMFS/data-table shortcuts like `reference genome: dm6` are workflow-interface choices as much as test YAML choices: they require the workflow input to be a string/data-table-backed parameter, not a file input.
+- If a fixture needs stable sample names, decide whether those names should enter through collection element identifiers, a mapping file, or a workflow parameter before writing assertions.
+
+Rule for an agent: when a new workflow input is being designed and the test fixture is already known, check galaxy-workflow-testability-design before locking the input label/type.
+
+## 5. CVMFS / `.loc` / built-in-index references
+
+When a workflow input is a Galaxy data-table-backed reference (built-in genome index, dbsnp file, kraken DB, …), the input takes a **plain string** matching the `value` column of a `.loc` file. No `class: File`, no `location:`. Examples:
+
+- `scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:25` — `reference genome: dm6 # To test on usegalaxy.eu dm6full`
+- `epigenetics/cutandrun/cutandrun-tests.yml:27` — `reference_genome: 'hg38canon'`
+- `epigenetics/atacseq/atacseq-tests.yml:25` — `reference_genome: hg19`
+- `microbiome/host-contamination-removal/host-contamination-removal-short-reads/host-or-contamination-removal-on-short-reads-tests.yml:19` — `Host/Contaminant Reference Genome: hg38`
+- `microbiome/pathogen-identification/nanopore-pre-processing/Nanopore-Pre-Processing-tests.yml:22` — `host_reference_genome: apiMel3`
+- `transcriptomics/goseq/goseq-go-kegg-enrichment-analsis-tests.yml:24` — `Select genome to use: mm10`
+
+The string is the first column of the `.loc` entry. `workflows/README.md:169-182` shows the lookup procedure: browse `http://datacache.galaxyproject.org/indexes/location/<table>.loc`, find the `.loc` row, take the value column (which column varies by tool — check the matching `tool_data_table_conf.xml.sample`).
+
+**Portability implication:** these tests **only** run on a Galaxy with CVMFS mounted at `/cvmfs/data.galaxyproject.org`. IWC CI mounts CVMFS in-runner (`setup-cvmfs: true` in `.github/workflows/test_workflows.yml`). On a plain developer laptop running `planemo test`, CVMFS-string inputs fail unresolved. An agent generating tests should either:
+
+1. Stay URL-driven: replace the CVMFS string with a `class: File` + `location:` to a Zenodo-hosted reference (portable, slower CI, larger inputs).
+2. Use the CVMFS string and document "CI-only" — the user cannot reproduce locally without a CVMFS mount.
+
+There is no `.loc`-resolution shorthand in the YAML — it is just a bare string parameter. Agents must not invent `class: DataTable` or similar.
+
+## 6. Reviewer-feedback patterns (PR-review)
+
+Recurring asks from IWC reviewers, distilled from the contribution README and community help threads:
+
+- **Move large data to Zenodo before merge.** Anything bigger than a toy fixture committed in `test-data/` will be flagged. `workflows/README.md:67`.
+- **Use assertions, not exact-file comparison, for large outputs.** `workflows/README.md:98` — "if some outputs are large, it is better to use assertions than storing the whole output file."
+- **Generate tests with `planemo workflow_test_init --from_invocation`, don't hand-write.** `workflows/README.md:88-94`. The autogenerated test will mint correct hashes, correct labels, and a working `test-data/` snapshot.
+- **Set creator `identifier:` to a full ORCID URL** (e.g. `https://orcid.org/0000-0002-1825-0097`) — `workflows/README.md:53`. The most common lint failure; enforced in [planemo#1458](https://github.com/galaxyproject/planemo/pull/1458).
+- **Don't use `compare: diff` on outputs that embed timestamps or non-deterministic ordering.** Switch to `has_text` / `has_n_lines` (with `delta:`) or `compare: sim_size`+`delta:`.
+- **Bump `release:` in the `.ga` and add a `CHANGELOG.md` entry** in the same PR — enforced by the IWC PR template; reviewers verify via `bump_version.py`.
+- **Lower-case + `-` only** in directory names — `workflows/README.md:11,72`.
+- **Element identifiers must round-trip** — quote in YAML when they contain pipes/colons/whitespace.
+
+## 7. What is *not* covered by current convention
+
+Gaps an agent should be aware of:
+
+- **`composite_data:` is unrepresented in the IWC corpus.** No working examples to imitate for imzml+ibd or other multi-file datatypes. Treat as off-trail; cite planemo docs and verify by parser.
+- **`tags:` on elements unused.** Galaxy 20.09+ supports them; IWC does not exercise them.
+- **CWL-style shorthand File lists unused.** IWC always names elements.
+- **No negative tests.** Zero `expect_failure:` across the sampled corpus. There is no IWC convention for asserting a workflow *should* fail on bad input.
+- **No checksum-based output assertion in the wild.** `checksum: "sha1$..."` is documented and used at the input side as `hashes:`, but output-side `checksum:` was not surfaced in sampled tests. The corpus prefers `file:` (exact), `compare: sim_size`+`delta:` (size only), or `asserts:` (content probes).
+- **No data-cache layer beyond pip/planemo.** Every test re-pulls remote inputs from Zenodo / EBI / SRA on every CI run. A Zenodo outage breaks many PRs simultaneously. There is no IWC-side mirror or cache.
+- **No per-test timeout / retry convention.** A hung tool blocks the whole chunk until the GitHub Actions 6-hour limit.
+- **No portable story for CVMFS-only tests.** The same `-tests.yml` either works locally (URL-driven) or works in CI (CVMFS-string-driven), not both. There is no documented switch.
+- **Hash function is de-facto SHA-1.** The spec accepts SHA-256/SHA-512/MD5; IWC does not use them. Agents that emit a stronger hash will pass planemo but break corpus convention.
+- **Local-file `hashes:` are inconsistently present.** `--from_invocation`-generated tests have them, hand-written tests usually don't. Either is accepted; don't expect uniformity.
+- **`decompress: true` and `dbkey:` are file-level attributes** but undocumented at the planemo test_format page in the same place as `class:`/`location:`/`hashes:`. Cross-reference against the `gxformat2` Schema-Salad bindings if unsure.
+- **`?download=1` query string on Zenodo URLs** is optional. Both forms appear (`/record/4555735/files/X?download=1` vs `/records/11484215/files/Y`). The newer `/records/` plural is the modern Zenodo URL form; both work.
+
+## Cross-references
+
+- galaxy-workflow-testability-design — workflow input/output structure choices that make these fixture shapes testable.
+- `https://github.com/jmchilton/galaxy-brain/blob/main/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md` — full synthesis with assertion-vocabulary table, CI internals, and tooling rundown. Sections 2b, 3, 4, 5, 9 are the direct upstream of this note.
+- planemo test format spec: [planemo.readthedocs.io/en/latest/test_format.html](https://planemo.readthedocs.io/en/latest/test_format.html).
+- planemo best practices: [planemo.readthedocs.io/en/latest/best_practices_workflows.html](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html).
+- planemo collections reference: [github.com/galaxyproject/planemo/blob/master/docs/_writing_collections.rst](https://github.com/galaxyproject/planemo/blob/master/docs/_writing_collections.rst).
+- IWC contribution contract: `workflows/README.md` in [github.com/galaxyproject/iwc](https://github.com/galaxyproject/iwc).
+- Galaxy datacache (CVMFS `.loc` browsing): [datacache.galaxyproject.org/indexes/location/](http://datacache.galaxyproject.org/indexes/location/).

package/extensions/orbit/vendor/skills/changeset-to-galaxy-test-plan/references/notes/planemo-asserts-idioms.md

@@ -0,0 +1,246 @@
+---
+type: research
+tags:
+  - target/galaxy
+status: draft
+created: 2026-04-30
+revised: 2026-05-11
+revision: 6
+related_notes:
+  - "galaxy-workflow-testability-design"
+  - "iwc-test-data-conventions"
+  - "iwc-shortcuts-anti-patterns"
+  - "implement-galaxy-workflow-test"
+  - "tests-format"
+  - "planemo-workflow-test-architecture"
+  - "validate-tests"
+  - "iwc-tabular-operations-survey"
+  - "galaxy-discover-datasets"
+summary: "Decision and idiom guide for picking planemo workflow-test assertions: which family per output type, how to size tolerances, when to validate."
+---
+
+# Planemo asserts: idiom and decision guide
+
+Companion to iwc-test-data-conventions (input shapes), galaxy-workflow-testability-design (workflow structure before test YAML exists), and iwc-shortcuts-anti-patterns (what's accepted vs smell). This note is forward-looking: when authoring a new `<workflow>-tests.yml`, which assertion family fits which output, and what the recommended tolerances and operators are.
+
+The **vocabulary itself is not restated here** — every assertion's parameter list, types, defaults, required fields, and Python docstring is rendered from the test-format JSON Schema at tests-format. Assertion names below deep-link into that page (e.g. has_text jumps straight to that `$def`).
+
+## 1. Choose by output type
+
+The single most useful decision table. Pick the row that matches the file format the workflow emits; default to the recommended assertion family.
+
+| Output type | Default assertion family | Why | Fallback |
+|---|---|---|---|
+| **Plain text reports / logs** (FastQC summary, MultiQC text section) | has_text (substring on a known stable token) + has_n_lines with `delta:` | Substring is robust across versions; line count catches truncation | has_text_matching (regex) when token text changes per run |
+| **HTML reports** (MultiQC HTML, custom dashboards) | has_text against stable section names | HTML embeds timestamps and asset hashes; byte-diff is hopeless | has_size + large `delta` as last resort |
+| **Tabular** (TSV, CSV, BED-like) | has_n_columns + has_text for headers + has_n_lines with `delta:` | Schema-shape + content probes; tolerant to row reordering | has_line_matching for a known canonical row |
+| **VCF** | `compare: diff` with `lines_diff: 6` | The `lines_diff: 6` constant matches the typical VCF header preamble that embeds `##fileDate=` and `##source=` | has_text_matching for known variants |
+| **BAM** | has_size + has_archive_member (BAM is a gzipped block format) | Headers contain `@PG` lines with command lines and Galaxy job IDs; never byte-diff | Convert to SAM in a downstream step and assert on text |
+| **FASTA** (deterministic — assemblies, consensus) | `file:` exact comparison or has_text for known sequence | Output is byte-stable when the upstream tool is deterministic | has_n_lines if line wrapping varies |
+| **FASTA** (non-deterministic — RepeatModeler libraries) | `compare: sim_size` with large `delta:` | Family content varies run-to-run | has_n_lines with `delta:` |
+| **FASTQ** (rare as workflow output) | has_n_lines (must be multiple of 4) | Quality scores are read-id-dependent | `compare: sim_size` |
+| **JSON** (deterministic — config dumps, params) | has_json_property_with_value / has_json_property_with_text | Surgical: assert on the property you care about, ignore the rest | `compare: diff` if the JSON is fully canonical |
+| **JSON** (stochastic — HyPhy stats, MCMC results) | `has_text: text: "{"` (existence-only) | Embedded floats break any structural assertion; see iwc-shortcuts-anti-patterns §1 | has_h5_keys-equivalent for top-level structural keys |
+| **HDF5 / AnnData** | has_h5_keys + has_h5_attribute for known structure | Asserts shape, not values — values are float-noisy | has_size with `delta:` |
+| **XML** | is_valid_xml + has_element_with_path + `element_text_is`/`element_text_matches` | Surgical structural probes; byte-diff fails on whitespace/attribute-order | has_n_elements_with_path for cardinality |
+| **PNG / image plots** | has_image_width + has_image_height + has_size with `delta:` (5–10%) | Catches "rendered something with the right dimensions"; corpus norm | has_image_center_of_mass if mask/segmentation outputs are deterministic |
+| **TIFF / multipage images** | has_image_frames + has_image_channels + has_size with `delta:` | Same logic, channel/frame structure matters | — |
+| **Archives** (zip, tar.gz) | `has_archive_member: path: "regex"` with nested `asserts:` | Asserts on a specific member; archive timestamps never byte-stable | has_size with `delta:` |
+| **GFF / GTF** | has_n_lines with `delta:` + has_text for known feature | Tools reorder freely; hard to byte-diff | `has_n_columns: 9` (GFF spec) |
+| **Cool / HiC matrices** | `compare: sim_size` with multi-MB `delta:` | Binary, run-to-run variance | has_archive_member for HDF5 component |
+
+When in doubt: start with has_size + `delta_frac: 0.1`. It catches the catastrophic failure mode (empty / 10x bigger output). Then add a content probe.
+
+### 1a. If the assertion is too weak, revisit the workflow output
+
+Assertion choice sometimes reveals a workflow-design problem. If the only available output can support only a size check or image-dimension smoke test, check whether the workflow should expose a stronger checkpoint before settling for the weak assertion.
+
+- Scanpy's plot outputs use size and image-dimension assertions, but the same workflow also exposes AnnData HDF5 checkpoints and a cluster-count table (`$IWC/workflows/scRNAseq/scanpy-clustering/Preprocessing-and-Clustering-of-single-cell-RNA-seq-data-with-Scanpy-tests.yml:33-205`).
+- RNA-seq paired-end uses coarse size bands for coverage/mapped-read outputs, but expression/count tables get stronger regex and exact-line checks (`$IWC/workflows/transcriptomics/rnaseq-pe/rnaseq-pe-tests.yml:48-97`).
+
+Rule: if a translated workflow exposes only weakly assertable final reports, consult galaxy-workflow-testability-design and consider promoting a table/text/HDF5 checkpoint before writing the final assertions.
+
+## 2. The `compare:` operators
+
+Top-level on a `file:` output assertion. Vocabulary, in decreasing strictness:
+
+- **`diff`** (default). Byte-for-byte equality with optional `lines_diff:` tolerance for a fixed number of header lines. Use only when the upstream tool is deterministic on fixed inputs *and* the output has no embedded timestamps, command lines, version banners, or hash-ordered Python-dict-style keys.
+- **`re_match` / `re_match_multiline`**. Each line of the expected fixture is a regex that must match the corresponding output line. Useful when a few fields per row are timestamped but the rest is canonical. Rare in the corpus.
+- **`contains`**. The expected fixture is a substring of the output. Cheap; weak. Prefer `asserts: has_text` for new code unless you genuinely have a multi-line block to assert as a whole.
+- **`sim_size`**. Output file size matches the fixture's size within `delta:` (bytes) or `delta_frac:` (fraction). Use when the output is necessarily non-deterministic but its rough size is reproducible (RepeatModeler libraries, HiC matrices, Bayesian sampler outputs).
+
+Picking `lines_diff:`: count the mutable header lines in the output format. VCF: ~6 (`##fileformat`, `##fileDate`, `##source`, `##reference`, contig/info lines vary). SAM/text headers: count `@HD`/`@PG`/`@CO` lines. Set `lines_diff:` to the count exactly — looser values mask real diffs.
+
+## 3. Tolerance picking
+
+**`delta:`** is bytes (for has_size and `compare: sim_size`) or absolute count (for has_n_lines, has_n_columns, has_image_width, etc.). Suffix multipliers documented in the schema — `1K`, `1M`, `1G` work.
+
+**`delta_frac:`** is a fraction (0.1 = 10%). Use when expected size scales with input volume. Three IWC tests use it (`scRNAseq/baredsc/*`, `genome-assembly/polish-with-long-reads/*`); the rest use absolute `delta:`.
+
+**Picking magnitudes** (from corpus survey in iwc-shortcuts-anti-patterns §2):
+
+- Image dimensions: `delta: 25–30` pixels (5% of typical matplotlib defaults).
+- Image file size: `delta: 5K–60K` (5–10% of file size).
+- Small text reports: `delta: 1K–10K`.
+- HTML reports: `delta: 25K–100K`.
+- BAM files: `delta: 1M–10M`.
+- RepeatModeler / Bayesian sampler outputs: `delta: 30K–90M` (extreme, but justified by the underlying nondeterminism).
+
+**Heuristic for new outputs:** `delta_frac: 0.1` is a defensible default. Tighten if the output proves more deterministic than expected.
+
+## 4. Text family — has_text vs has_text_matching vs has_line vs has_line_matching vs has_n_lines
+
+All five are common; choose by what you're verifying.
+
+- **has_text** — output contains the substring `text:`. Anywhere in the output, any number of times. Add `n:` / `min:` / `max:` to constrain occurrence count. Add `delta:` to allow slack on the count.
+- **has_text_matching** — output matches the regex `expression:`. Use sparingly; prefer literal has_text when you can.
+- **has_line** — output has at least one *line* matching `line:` exactly. Use when line boundaries matter (e.g. asserting on a specific row in a table).
+- **has_line_matching** — same but with regex.
+- **has_n_lines** — assert the line count is `n:` ± `delta:`.
+
+A common combination in IWC: `has_n_lines: n: 100, delta: 5` + `has_text: text: "expected_token"` — line-count sanity-check plus a content marker. This catches both truncation and content drift in one assertion pair.
+
+`negate: true` is supported on every assertion. Used for the "this output should NOT contain X" case.
+
+## 5. Collection output assertions (`element_tests:`)
+
+For Galaxy collection outputs, the test format keys element assertions by element identifier:
+
+```yaml
+my_collection_output:
+  element_tests:
+    sample_1:
+      asserts:
+        has_text:
+          text: "expected"
+    sample_2:
+      file: test-data/expected_sample_2.txt
+```
+
+Optional `attributes:` at the collection level can assert on the produced collection shape:
+
+```yaml
+my_collection_output:
+  attributes: {collection_type: list:list}
+  element_tests:
+    ...
+```
+
+Nested collections: outer `element_tests:` keyed by outer identifier; inner uses `elements:` (note plural, no `_tests` suffix on the inner). See iwc-test-data-conventions §2f for the live example.
+
+For a list-of-files where every element should pass the same minimal check, the existence-probe pattern (`has_text: "{"` for JSON; `has_size: min: 100` for any non-empty binary) is widely used and accepted in IWC.
+
+## 6. The validate-against-workflow inner loop
+
+A `-tests.yml` file can be **structurally invalid** in two distinct ways:
+
+1. **Schema-invalid** — wrong field names, wrong nesting, wrong types. Caught by the test-format JSON Schema.
+2. **Workflow-incoherent** — schema-valid YAML, but the input/output labels don't match the actual workflow. Renaming an output in the `.ga` and forgetting to update its sibling `-tests.yml` produces this case. Planemo will surface it as an "output not found" error at test-runtime, but only after a full workflow run.
+
+The `@galaxy-tool-util/schema` npm package ships **two** validators that catch both cases statically — no Galaxy or Planemo invocation needed:
+
+- **`validateTestsFile(yaml)`** — runs the file against `tests.schema.json` (AJV). Reports schema violations with paths.
+- **`checkTestsAgainstWorkflow(workflow, tests)`** — cross-checks a `.ga` / format2 workflow against a tests file: missing input labels, missing output labels, type incompatibilities (e.g. test supplies a `File` for a parameter typed `int`).
+
+Both are pure-JS, take milliseconds, and have no Galaxy dependency. Wire them into the inner authoring loop:
+
+```
+edit -tests.yml
+  → validateTestsFile()                    # schema gate
+  → checkTestsAgainstWorkflow(.ga, tests)  # coherence gate
+  → planemo workflow_test_on_invocation    # assertion gate (no full re-run)
+  → planemo test                           # full integration (slow)
+```
+
+The first two gates short-circuit cheap mistakes before a slow planemo run. They are the static-validation equivalent of `gxwf` for tests, and the `implement-galaxy-workflow-test` mold should reference them as its primary inner-loop tooling. Source: galaxy-tool-util-ts package, `src/test-format/index.ts` exports.
+
+## 7. Authoring loop — generation, then refinement
+
+Reviewer convention is to **generate** the initial `-tests.yml` rather than hand-write it. Two planemo subcommands cover this:
+
+- **`planemo workflow_test_init --from_invocation <invocation_id>`** (planemo-workflow_test_init) — given a successful Galaxy invocation ID, emit a `-tests.yml` with a `job:` block that captures all inputs (with SHA-1 hashes) and an `outputs:` block with `file:` references to the actual outputs (downloaded into `test-data/`). Hand-tighten the assertions afterward.
+- **`planemo workflow_test_on_invocation <tests.yml> <invocation_id>`** (planemo-workflow_test_on_invocation) — re-evaluate an edited `-tests.yml` against a saved invocation without re-running the workflow. The fast inner loop for assertion iteration; complements the static gates in §6.
+
+Together these cut the assertion-iteration cost dramatically. An agent should:
+
+1. Run the workflow once on usegalaxy.* (or local) to get a known-good invocation.
+2. `--from_invocation` to bootstrap the test file.
+3. Replace the autogenerated `file:` exact-comparison assertions with assertion-family-appropriate alternatives per §1.
+4. planemo-workflow_test_on_invocation after each edit; full planemo-test at the end.
+
+## 8. What the schema gives you for free
+
+When the test-format schema lands as a Foundry-rendered note, the agent can consult any assertion's `$def` directly for: parameter types, defaults, required fields, the `that` discriminator constant, and the original Python docstring (carried through as `description`). This note does **not** restate that vocabulary — it complements it with the corpus-grounded *which-and-when*.
+
+What's still missing from the schema and worth keeping in research notes:
+
+- This decision table (§1) — output-type → assertion family.
+- Tolerance magnitudes (§3) — corpus-derived defaults.
+- The `validateTestsFile` / `checkTestsAgainstWorkflow` integration story (§6).
+- Anti-pattern flags — see iwc-shortcuts-anti-patterns.
+
+## 9. Common combinations (recipes)
+
+Six recipes worth memorizing.
+
+**Stable text report (FastQC summary, simple stats).**
+```yaml
+my_report:
+  asserts:
+    has_n_lines: { n: 12, delta: 2 }
+    has_text: { text: "Total Sequences" }
+```
+
+**MultiQC HTML report.**
+```yaml
+multiqc_report:
+  asserts:
+    has_text: { text: "Filtered Reads" }
+    has_text: { text: "FastQC" }
+```
+
+**VCF (pinned tool, fixed reference).**
+```yaml
+called_variants:
+  file: test-data/expected.vcf
+  compare: diff
+  lines_diff: 6
+```
+
+**Stochastic JSON (HyPhy-style).**
+```yaml
+hyphy_meme:
+  element_tests:
+    geneA: { asserts: { has_text: { text: "{" } } }
+    geneB: { asserts: { has_text: { text: "{" } } }
+```
+
+**Matplotlib plot.**
+```yaml
+umap_plot:
+  asserts:
+    has_size: { size: 68416, delta: 6000 }
+    has_image_width: { width: 601, delta: 30 }
+    has_image_height: { height: 429, delta: 25 }
+```
+
+**AnnData (HDF5).**
+```yaml
+clustered_anndata:
+  asserts:
+    has_h5_keys: { keys: "obs/louvain" }
+    has_h5_keys: { keys: "var/highly_variable" }
+    has_h5_keys: { keys: "uns/rank_genes_groups" }
+    has_size: { size: 12000000, delta: 1500000 }
+```
+
+## 10. Cross-references
+
+- galaxy-workflow-testability-design — decide which workflow outputs and checkpoints to expose before choosing assertions.
+- iwc-test-data-conventions — input-side conventions (job inputs, collection shapes, `hashes:`, CVMFS).
+- iwc-shortcuts-anti-patterns — accepted-vs-smell catalog and corpus prevalence; this note's mirror image.
+- Test-format schema (`@galaxy-tool-util/schema` npm package) — authoritative vocabulary; will be vendored into a Foundry-rendered schema note. See `content/meta/casting.md` for the casting story.
+- Planemo test-format spec: [planemo.readthedocs.io/en/latest/test_format.html](https://planemo.readthedocs.io/en/latest/test_format.html).
+- galaxy-xsd — Galaxy XSD assertion source of truth, vendored from upstream.
+- Tightening of the schema and Pydantic source: [galaxyproject/galaxy#22566](https://github.com/galaxyproject/galaxy/pull/22566).
+- TS schema sync into npm: [jmchilton/galaxy-tool-util-ts#75](https://github.com/jmchilton/galaxy-tool-util-ts/pull/75).