@galaxyproject/orbit 0.0.0 → 0.9.0

package/extensions/orbit/vendor/skills/compare-against-iwc-exemplar/references/notes/iwc-test-data-conventions.md

@@ -0,0 +1,308 @@
+---
+type: research
+tags:
+  - target/galaxy
+status: draft
+created: 2026-04-30
+revised: 2026-05-03
+revision: 3
+related_notes:
+  - "galaxy-workflow-testability-design"
+  - "iwc-shortcuts-anti-patterns"
+  - "planemo-asserts-idioms"
+  - "implement-galaxy-workflow-test"
+  - "tests-format"
+  - "iwc-tabular-operations-survey"
+summary: "How IWC workflows organize and reference test data — Zenodo-first, SHA-1 integrity, collection shapes, CVMFS gotchas."
+---
+
+# IWC test data conventions
+
+Reference for an agent implementing or editing a `<workflow>-tests.yml` in IWC style. All evidence cited from `https://github.com/galaxyproject/iwc/blob/main/workflows/` (raw IWC clone) and `workflows/README.md`. Authoritative spec: [planemo.readthedocs.io/en/latest/test_format.html](https://planemo.readthedocs.io/en/latest/test_format.html). The companion analysis at `https://github.com/jmchilton/galaxy-brain/blob/main/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md` is the synthesized source for several normative claims here.
+
+This note owns **test YAML fixture shapes**. For workflow-structure choices that make those fixtures possible before the test file exists, use galaxy-workflow-testability-design.
+
+## 1. Where does test data live? Remote vs in-repo
+
+Two storage patterns. They mix freely inside one job.
+
+**Remote `location:` (default for any non-trivial input).** Strongly preferred for anything bigger than a toy fixture. Order of preference, observed in the corpus:
+
+- **Zenodo** — overwhelming default, persistent DOI-backed URL.
+  - `read-preprocessing/short-read-qc-trimming/short-read-quality-control-and-trimming-tests.yml:13,17` — `https://zenodo.org/records/11484215/files/paired_r1.fastq.gz` (note the modern `/records/` plural form; older entries use `/record/`).
+  - `metabolomics/lcms-preprocessing/Mass_spectrometry__LC-MS_preprocessing_with_XCMS-tests.yml:6,17,24,...` — 11+ mzML files all hosted at `zenodo.org/record/10130758/files/...`.
+  - `sars-cov-2-variant-calling/sars-cov-2-consensus-from-variation/consensus-from-variation-tests.yml:5` — `https://zenodo.org/record/4555735/files/NC_045512.2_reference.fasta?download=1`.
+  - `repeatmasking/RepeatMasking-Workflow-tests.yml:5` — `https://zenodo.org/record/8364146/files/eco.fasta?download=1`. The `?download=1` query suffix is acceptable but optional; both forms appear.
+- **EBI/ENA REST** — for canonical reference sequences.
+  - `virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml:5` — `https://www.ebi.ac.uk/ena/browser/api/fasta/AF325528.1?download=true`.
+- **SRA / ENA FTP** — for raw read accessions where re-uploading to Zenodo is wasteful.
+  - `pox-virus-half-genome-tests.yml:27,34,49,56` — `ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/076/SRR15145276/SRR15145276_1.fastq.gz`.
+
+**In-repo `path: test-data/...` (toy fixtures + small expected outputs only).** Used when:
+
+- File is small enough to commit (rule of thumb from corpus: low single-digit MB or smaller).
+- File is a hand-crafted toy or trimmed reference (e.g. a 5-element FASTA for HyPhy).
+- File is the *expected output* baseline for a `file:` exact-comparison assertion: `consensus-from-variation-tests.yml:31` — `file: test-data/masked_consensus.fa`.
+
+`workflows/README.md:67` makes the contract explicit: "try to generate a toy dataset … and then publish it to zenodo to have a permanent URL." `workflows/README.md:98`: "if some outputs are large, it is better to use assertions than storing the whole output file to the iwc repository."
+
+Subdirectory layout inside `test-data/` is free-form and used to organize related fixtures, e.g. `comparative_genomics/hyphy/test-data/unaligned_seqs/`, `test-data/codon_alignments/`, `test-data/iqtree_trees/` (referenced from `hyphy-core-tests.yml:13,17,21,...`).
+
+**Decision rule for an agent:**
+
+1. Expected-output baseline that's small — commit to `test-data/`.
+2. Toy/trimmed input < ~5 MB — commit to `test-data/`.
+3. Anything else — Zenodo (or EBI/SRA for canonical accessions). Add `hashes:` SHA-1.
+4. If the input is a reference index resolvable from CVMFS (`hg38`, `dm6`, `mm10`, …), use the plain string form. See section 5.
+
+## 2. The `class: File` block — exact YAML shapes
+
+All snippets verbatim from the IWC corpus.
+
+### 2a. Single remote file (with integrity hash)
+
+`virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml:3-9`:
+
+```yaml
+Reference FASTA:
+  class: File
+  location: "https://www.ebi.ac.uk/ena/browser/api/fasta/AF325528.1?download=true"
+  filetype: fasta
+  hashes:
+  - hash_function: SHA-1
+    hash_value: 927d0d00b7db6ad60524bb9e50d3ab41c4ac5ecf
+```
+
+Field order is conventional but not enforced. `filetype:` is the Galaxy datatype name (`fasta`, `fastqsanger.gz`, `mzml`, `bed`, `tabular`, `csv`, `gtf`, `vcf`, …). `location:` URLs may be quoted or bare; both forms occur.
+
+### 2b. Single local file
+
+`comparative_genomics/hyphy/hyphy-core-tests.yml:3-6`:
+
+```yaml
+reference cds:
+  class: File
+  path: test-data/denv1_ref_cds.fasta
+  filetype: fasta
+```
+
+`path:` is relative to the workflow directory (the directory containing the `-tests.yml`). Hashes are usually omitted on local files — the file is in-tree, so integrity is implicit. They are nonetheless added in some workflows: `consensus-from-variation-tests.yml:15-18` shows `path: test-data/aligned_reads_for_coverage.bam` paired with a SHA-1 hash. The pattern appears to be: hashes added when the `-tests.yml` was generated by `planemo workflow_test_init --from_invocation`, which mints them automatically.
+
+A dataset attribute can be set on a single file to force a `dbkey`, e.g. `epigenetics/consensus-peaks/consensus-peaks-chip-pe-tests.yml:7-10`:
+
+```yaml
+- class: File
+  identifier: rep1
+  path: test-data/rep1.bam
+  dbkey: mm10
+```
+
+`dbkey:` and `decompress: true` (`scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:30`) are documented per-file attributes that ride alongside `class: File`.
+
+### 2c. List collection
+
+`hyphy-core-tests.yml:7-30`:
+
+```yaml
+unaligned sequences:
+  class: Collection
+  collection_type: list
+  elements:
+  - identifier: AB178040.1|2002
+    class: File
+    path: test-data/unaligned_seqs/AB178040.1|2002.fasta
+    filetype: fasta
+  - identifier: AB195673.1|2003
+    class: File
+    path: test-data/unaligned_seqs/AB195673.1|2003.fasta
+    filetype: fasta
+  ...
+```
+
+Each element is a single-file dict with an `identifier:`. A larger remote-only example with hashes per element: `metabolomics/lcms-preprocessing/Mass_spectrometry__LC-MS_preprocessing_with_XCMS-tests.yml:11-22`:
+
+```yaml
+Mass-spectrometry Dataset Collection:
+  class: Collection
+  collection_type: list
+  elements:
+  - class: File
+    identifier: Blanc05.mzML
+    location: https://zenodo.org/record/10130758/files/Blanc05.mzML
+    filetype: mzml
+    hashes:
+    - hash_function: SHA-1
+      hash_value: e98d5a4cd8849a145a032bdc31c348ad97cb59c3
+```
+
+### 2d. Paired collection (single pair)
+
+A `paired` collection is a Collection whose two elements have identifiers `forward` and `reverse`. In IWC it is rarely used at top level — it is almost always wrapped as a single-element `list:paired` (see 2e). When it appears bare, the shape is the inner block of 2e without the outer `list:paired` wrapper.
+
+### 2e. `list:paired` collection
+
+`read-preprocessing/short-read-qc-trimming/short-read-quality-control-and-trimming-tests.yml:3-18`:
+
+```yaml
+Raw reads:
+  class: Collection
+  collection_type: list:paired
+  elements:
+  - class: Collection
+    type: paired
+    identifier: pair
+    elements:
+    - class: File
+      identifier: forward
+      location: https://zenodo.org/records/11484215/files/paired_r1.fastq.gz
+      filetype: fastqsanger.gz
+    - class: File
+      identifier: reverse
+      location: https://zenodo.org/records/11484215/files/paired_r2.fastq.gz
+      filetype: fastqsanger.gz
+```
+
+Note: outer `collection_type: list:paired`, inner `type: paired` (not `collection_type:`). Inner element identifiers are the literal strings `forward` and `reverse` — these are reserved.
+
+A multi-pair list:paired with full hashes: `virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml:17-38` or `scRNAseq/fastq-to-matrix-10x/scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:3-24`.
+
+### 2f. `list:list:paired` and deeper
+
+Not directly observed in sampled IWC inputs. The shape is recursive — wrap a `list:paired` element list inside another `class: Collection, collection_type: list:list:paired`, where each outer element is `class: Collection, type: list:paired` (analogous to the `list:paired` → `paired` nesting in 2e). For deeply-nested collections planemo's [`_writing_collections.rst`](https://github.com/galaxyproject/planemo/blob/master/docs/_writing_collections.rst) is the de-facto reference.
+
+Output side: nested-collection assertions DO exist in the corpus. `scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:82-98`:
+
+```yaml
+Seurat input for gene expression (filtered):
+  attributes: {collection_type: list:list}
+  element_tests:
+    subsample:
+      elements:
+        matrix:
+          asserts:
+            has_line:
+              line: "23932 3 1171"
+        barcodes:
+          asserts:
+            has_line:
+              line: "CACATGATCATAAGGA"
+```
+
+Pattern: outer `element_tests:` keyed by outer identifier; inner `elements:` (note plural, not `element_tests:` here) keyed by inner identifier. Optional `attributes: {collection_type: ...}` asserts the shape of the produced collection.
+
+### 2g. `composite_data:`
+
+**Not observed** in the sampled IWC corpus (`grep -rln "composite\|imzml\|primary_file\|extra_files" workflows/ --include="*-tests.yml"` returns nothing). Per the planemo spec the shape is a `composite_data:` mapping listing each component file path, but an agent emitting one in IWC style has no in-corpus reference and should treat it as unproven territory — fall back to the planemo docs and verify against `galaxy.tool_util.parser.yaml`.
+
+### 2h. CWL-style shorthand (list of File dicts without identifiers)
+
+Documented in planemo; not observed in sampled IWC. IWC always supplies explicit identifiers.
+
+### 2i. `tags:` on elements (Galaxy 20.09+)
+
+Documented in planemo; not surfaced in sampled IWC `-tests.yml` files (`grep "  tags:" --include="*-tests.yml"` empty across the sampled set). Practical absence — IWC tests rely on identifier matching, not tags.
+
+## 3. SHA-1 integrity hashes — when, when not, why
+
+**Format.** Always a list under `hashes:`, even for one entry. Each entry is `{hash_function: SHA-1, hash_value: <40-hex>}`. Verbatim from `pox-virus-half-genome-tests.yml:7-9`:
+
+```yaml
+hashes:
+- hash_function: SHA-1
+  hash_value: 927d0d00b7db6ad60524bb9e50d3ab41c4ac5ecf
+```
+
+Only `SHA-1` observed in the IWC corpus. The planemo spec also accepts `MD5`, `SHA-256`, `SHA-512` as `hash_function:` values; IWC does not exercise these.
+
+**When present:**
+
+- Every remote `location:`-fetched input in the sampled corpus carries a SHA-1 (Zenodo, EBI REST, SRA FTP). The hash guards against silent upstream corruption — Zenodo records are immutable but bit-rot and CDN-edge errors do happen; ENA FASTA endpoints can quietly change line-wrap.
+- Many local `path:`-loaded inputs also carry a SHA-1. This is `--from_invocation` autogeneration leaking through (planemo emits hashes for every input it captured).
+- `consensus-from-variation-tests.yml:6-8,17-18,27-28` — every input in the test, local and remote, has a hash.
+
+**When omitted:**
+
+- Hand-written local-file fixtures often skip hashes — `hyphy-core-tests.yml:3-30` has zero `hashes:` blocks across six local-file inputs.
+- Output assertions: `file:`-comparison outputs and `location:`-comparison outputs (`RepeatMasking-Workflow-tests.yml:13`) **do not** carry `hashes:`. The integrity story for outputs is handled by `compare:` / `asserts:` instead.
+
+**Why bother on local files.** Catches accidental corruption when collaborators hand-edit `test-data/` fixtures.
+
+## 4. Element identifier conventions
+
+Identifiers are arbitrary strings, used both as the YAML key (`element_tests:`) and as the produced Galaxy collection element name. Conventions surfaced from the corpus:
+
+- **Pipes are legal** and survive YAML round-trip without quoting on the input side: `hyphy-core-tests.yml:11` — `identifier: AB178040.1|2002`. On the *output* side they're quoted because they appear as YAML mapping keys: `hyphy-core-tests.yml:34` — `"NC_001477.1|capsid_protein_C|95-394_DENV1":`. Quote when the identifier contains characters that would otherwise start a YAML flow node (`{`, `[`, `&`, `*`, `!`, `|`, `>`, `'`, `"`, `%`, `@`, `` ` ``) or look like a number/bool.
+- **Dots, hyphens, underscores** — common, no quoting needed: `Blanc05.mzML`, `HU_neg_048.mzML`, `SRR11578257`, `Rep1`, `subsample`, `pair`.
+- **Reserved inside `paired` collections:** `forward` and `reverse` (lowercase). Never use these names for outer identifiers in `list:paired`.
+- **Inline comments after identifier** are legal and used for provenance: `pox-virus-half-genome-tests.yml:23` — `identifier: 20L70   # SRR15145276`.
+- **Whitespace and special chars in workflow input *labels*** survive too — they are job-mapping keys, not collection identifiers, but the same YAML quoting rules apply. Example from the analysis doc: `Manually annotate celltypes?: true` as a job key in `Preprocessing-and-Clustering-of-single-cell-RNA-seq-data-with-Scanpy-tests.yml:23`.
+
+### 4a. Workflow-interface implications
+
+Fixture shape should feed back into workflow input design. Tests supply job inputs by workflow input label, and collection fixtures must match the workflow's declared collection type.
+
+- The 10x CellPlex test uses `fastq PE collection GEX`, `fastq PE collection CMO`, and `sample name and CMO sequence collection` as job keys with `list:paired` and `list` collection fixtures; the workflow declares matching collection inputs at `$IWC_FORMAT2/scRNAseq/fastq-to-matrix-10x/scrna-seq-fastq-to-matrix-10x-cellplex.gxwf.yml:4-72`.
+- CVMFS/data-table shortcuts like `reference genome: dm6` are workflow-interface choices as much as test YAML choices: they require the workflow input to be a string/data-table-backed parameter, not a file input.
+- If a fixture needs stable sample names, decide whether those names should enter through collection element identifiers, a mapping file, or a workflow parameter before writing assertions.
+
+Rule for an agent: when a new workflow input is being designed and the test fixture is already known, check galaxy-workflow-testability-design before locking the input label/type.
+
+## 5. CVMFS / `.loc` / built-in-index references
+
+When a workflow input is a Galaxy data-table-backed reference (built-in genome index, dbsnp file, kraken DB, …), the input takes a **plain string** matching the `value` column of a `.loc` file. No `class: File`, no `location:`. Examples:
+
+- `scrna-seq-fastq-to-matrix-10x-cellplex-tests.yml:25` — `reference genome: dm6 # To test on usegalaxy.eu dm6full`
+- `epigenetics/cutandrun/cutandrun-tests.yml:27` — `reference_genome: 'hg38canon'`
+- `epigenetics/atacseq/atacseq-tests.yml:25` — `reference_genome: hg19`
+- `microbiome/host-contamination-removal/host-contamination-removal-short-reads/host-or-contamination-removal-on-short-reads-tests.yml:19` — `Host/Contaminant Reference Genome: hg38`
+- `microbiome/pathogen-identification/nanopore-pre-processing/Nanopore-Pre-Processing-tests.yml:22` — `host_reference_genome: apiMel3`
+- `transcriptomics/goseq/goseq-go-kegg-enrichment-analsis-tests.yml:24` — `Select genome to use: mm10`
+
+The string is the first column of the `.loc` entry. `workflows/README.md:169-182` shows the lookup procedure: browse `http://datacache.galaxyproject.org/indexes/location/<table>.loc`, find the `.loc` row, take the value column (which column varies by tool — check the matching `tool_data_table_conf.xml.sample`).
+
+**Portability implication:** these tests **only** run on a Galaxy with CVMFS mounted at `/cvmfs/data.galaxyproject.org`. IWC CI mounts CVMFS in-runner (`setup-cvmfs: true` in `.github/workflows/test_workflows.yml`). On a plain developer laptop running `planemo test`, CVMFS-string inputs fail unresolved. An agent generating tests should either:
+
+1. Stay URL-driven: replace the CVMFS string with a `class: File` + `location:` to a Zenodo-hosted reference (portable, slower CI, larger inputs).
+2. Use the CVMFS string and document "CI-only" — the user cannot reproduce locally without a CVMFS mount.
+
+There is no `.loc`-resolution shorthand in the YAML — it is just a bare string parameter. Agents must not invent `class: DataTable` or similar.
+
+## 6. Reviewer-feedback patterns (PR-review)
+
+Recurring asks from IWC reviewers, distilled from the contribution README and community help threads:
+
+- **Move large data to Zenodo before merge.** Anything bigger than a toy fixture committed in `test-data/` will be flagged. `workflows/README.md:67`.
+- **Use assertions, not exact-file comparison, for large outputs.** `workflows/README.md:98` — "if some outputs are large, it is better to use assertions than storing the whole output file."
+- **Generate tests with `planemo workflow_test_init --from_invocation`, don't hand-write.** `workflows/README.md:88-94`. The autogenerated test will mint correct hashes, correct labels, and a working `test-data/` snapshot.
+- **Set creator `identifier:` to a full ORCID URL** (e.g. `https://orcid.org/0000-0002-1825-0097`) — `workflows/README.md:53`. The most common lint failure; enforced in [planemo#1458](https://github.com/galaxyproject/planemo/pull/1458).
+- **Don't use `compare: diff` on outputs that embed timestamps or non-deterministic ordering.** Switch to `has_text` / `has_n_lines` (with `delta:`) or `compare: sim_size`+`delta:`.
+- **Bump `release:` in the `.ga` and add a `CHANGELOG.md` entry** in the same PR — enforced by the IWC PR template; reviewers verify via `bump_version.py`.
+- **Lower-case + `-` only** in directory names — `workflows/README.md:11,72`.
+- **Element identifiers must round-trip** — quote in YAML when they contain pipes/colons/whitespace.
+
+## 7. What is *not* covered by current convention
+
+Gaps an agent should be aware of:
+
+- **`composite_data:` is unrepresented in the IWC corpus.** No working examples to imitate for imzml+ibd or other multi-file datatypes. Treat as off-trail; cite planemo docs and verify by parser.
+- **`tags:` on elements unused.** Galaxy 20.09+ supports them; IWC does not exercise them.
+- **CWL-style shorthand File lists unused.** IWC always names elements.
+- **No negative tests.** Zero `expect_failure:` across the sampled corpus. There is no IWC convention for asserting a workflow *should* fail on bad input.
+- **No checksum-based output assertion in the wild.** `checksum: "sha1$..."` is documented and used at the input side as `hashes:`, but output-side `checksum:` was not surfaced in sampled tests. The corpus prefers `file:` (exact), `compare: sim_size`+`delta:` (size only), or `asserts:` (content probes).
+- **No data-cache layer beyond pip/planemo.** Every test re-pulls remote inputs from Zenodo / EBI / SRA on every CI run. A Zenodo outage breaks many PRs simultaneously. There is no IWC-side mirror or cache.
+- **No per-test timeout / retry convention.** A hung tool blocks the whole chunk until the GitHub Actions 6-hour limit.
+- **No portable story for CVMFS-only tests.** The same `-tests.yml` either works locally (URL-driven) or works in CI (CVMFS-string-driven), not both. There is no documented switch.
+- **Hash function is de-facto SHA-1.** The spec accepts SHA-256/SHA-512/MD5; IWC does not use them. Agents that emit a stronger hash will pass planemo but break corpus convention.
+- **Local-file `hashes:` are inconsistently present.** `--from_invocation`-generated tests have them, hand-written tests usually don't. Either is accepted; don't expect uniformity.
+- **`decompress: true` and `dbkey:` are file-level attributes** but undocumented at the planemo test_format page in the same place as `class:`/`location:`/`hashes:`. Cross-reference against the `gxformat2` Schema-Salad bindings if unsure.
+- **`?download=1` query string on Zenodo URLs** is optional. Both forms appear (`/record/4555735/files/X?download=1` vs `/records/11484215/files/Y`). The newer `/records/` plural is the modern Zenodo URL form; both work.
+
+## Cross-references
+
+- galaxy-workflow-testability-design — workflow input/output structure choices that make these fixture shapes testable.
+- `https://github.com/jmchilton/galaxy-brain/blob/main/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md` — full synthesis with assertion-vocabulary table, CI internals, and tooling rundown. Sections 2b, 3, 4, 5, 9 are the direct upstream of this note.
+- planemo test format spec: [planemo.readthedocs.io/en/latest/test_format.html](https://planemo.readthedocs.io/en/latest/test_format.html).
+- planemo best practices: [planemo.readthedocs.io/en/latest/best_practices_workflows.html](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html).
+- planemo collections reference: [github.com/galaxyproject/planemo/blob/master/docs/_writing_collections.rst](https://github.com/galaxyproject/planemo/blob/master/docs/_writing_collections.rst).
+- IWC contribution contract: `workflows/README.md` in [github.com/galaxyproject/iwc](https://github.com/galaxyproject/iwc).
+- Galaxy datacache (CVMFS `.loc` browsing): [datacache.galaxyproject.org/indexes/location/](http://datacache.galaxyproject.org/indexes/location/).

package/extensions/orbit/vendor/skills/compare-against-iwc-exemplar/references/notes/open-requirements-ledger.md

@@ -0,0 +1,173 @@
+---
+type: research
+title: "Open-requirements ledger"
+tags:
+  - target/galaxy
+status: draft
+created: 2026-06-16
+revised: 2026-08-29
+revision: 3
+related_notes:
+  - "galaxy-workflow-draft-format"
+related_molds:
+  - "advance-galaxy-draft-step"
+  - "repair-galaxy-draft-topology"
+  - "implement-galaxy-tool-step"
+summary: "Carried unresolved-requirements artifact the source→Galaxy pipeline discharges or explicitly surrenders, autonomously."
+---
+
+# Open-requirements ledger
+
+The `open-requirements-ledger` is a single artifact threaded through the source→Galaxy pipeline that records **obligations the pipeline has taken on but not yet met** — a declared output with no producer, a parameter whose value the source never pinned, a tool with no corpus exemplar — and **source work it decided not to carry**. Each Mold that surfaces one **appends** it; each Mold whose decision closes one **marks it resolved**; the terminal path **surrenders** whatever remains open, explicitly, into the final artifact.
+
+## Framing: obligations the pipeline discharges, not questions a human answers
+
+This is deliberately *not* an "open questions for the user" list. The pipeline is autonomous — no human-in-the-loop gate is assumed. The ledger's consumers are **Molds and the loop's convergence gate**, with human readout a secondary affordance. An entry is closed by a downstream Mold doing work (wiring a producer, picking a wrapper, settling a value), or — when nothing can close it — surrendered: written into the final draft as a known, labelled gap rather than silently dropped or fabricated around.
+
+The distinction matters because a "questions for a human" framing leaks an operator's personal interaction style into a tool meant to run inside anyone's harness. The ledger must behave identically cast into a fresh harness with no ambient configuration; that holds only because every obligation is **materialized in the artifact**, never carried as operator habit.
+
+## Entry shape (v1, loose)
+
+Like the `_plan_*` family in galaxy-workflow-draft-format, ledger entries are intentionally low-ceremony for v1 — enough structure to count and close, no contract pretending to be machine-parameterizable yet. A reasonable per-entry shape:
+
+```yaml
+- id: sccmec-evidence-missing
+  status: open                # open | resolved | surrendered
+  kind: gap                   # gap | dropped — see below; omit for gap
+  blocking: true              # true only for a computability gap; the convergence gate counts these
+  raised_by: implement-galaxy-tool-step   # Mold that appended it
+  step: classify_context       # draft step the obligation attaches to (if any)
+  unmet: "SCCmec-region candidate output category"
+  missing: "no wired input carries SCCmec cassette evidence"
+  resolved_by: null            # Mold that closed it, once closed
+  supersedes: null             # id whose justification this entry refutes, once resolved
+  note: ""                     # how it was closed or why surrendered
+```
+
+`kind` separates the two things an open entry can mean, which v1 conflated. A **`gap`** is
+an obligation the pipeline cannot discharge: no producer is discoverable, Galaxy has no
+datatype for the format, the source pinned no value. A **`dropped`** entry is source work
+the pipeline *declined to carry* — a plane cut from the design, a parameter surface not
+exposed, an output the repair removed. Both are honest; they are not the same admission,
+and a reader cannot tell them apart from `unmet` / `missing` alone. Omit `kind` for a
+plain gap.
+
+A `dropped` entry carries two more fields:
+
+```yaml
+- id: annotation-planes-not-built
+  status: open
+  kind: dropped
+  raised_by: nextflow-summary-to-galaxy-data-flow
+  units: "25 of 31 designed nodes (rnaseq_short, gnomon, annot_proc, convert)"
+  because: "`gxwf tool-search asn1` returns no wrapper reading seq-submit; searched 2026-08-29"
+  unmet: "the pipeline's actual product — an annotated genome"
+  missing: "the carried spine masks an assembly; it does not annotate a genome"
+  note: ""
+```
+
+`units` names what was dropped in the source's own terms, so the drops can be totalled
+without re-reading the design.
+
+`because` says why, and **must carry its citation** — the source construct that is not
+there, the target limitation named specifically enough that someone could check it, the
+search actually run. A drop's reason is a claim about the world, and the first real run is
+why the burden exists: 25 of 31 planes were cut because "each needs an authored Galaxy
+wrapper for an undocumented binary available only inside the pipeline's container," and a
+Tool Shed search one phase later found the whole pipeline already wrapped at the exact
+pinned version. The claim was not dishonest. It was never checked, and nothing in the
+entry showed that it had not been.
+
+A reason you can cite, cite. A reason you cannot cite is still recorded — write it plainly
+and leave it uncited rather than dressing it as evidence — but an uncited drop is not a
+finding the run may reason from afterwards. It is a debt, and it should read as one.
+
+Provenance (`raised_by`, `resolved_by`) is the audit trail for *when* each obligation entered and left — the traceability #281 asked for, and the evidence the convergence gate reads.
+
+`blocking` separates the two grades of obligation the pipeline carries. A plain entry is an unmet need the chain can keep working around — an unpinned parameter, a missing exemplar. A **blocking** entry is a computability gap: a declared step output no wired input can supply, which `gxwf` validation cannot see because the connection graph knows ports connect, not what they carry. Omit `blocking` (or set it `false`) for a plain obligation. Only blocking entries drive topology-repair escalation, and only they are counted by the decreasing-blocker invariant.
+
+## How to use the ledger
+
+This section is the runtime protocol for every Mold that carries the ledger. A Mold's own page states only what is local to it — what raises an entry there, and what it does when it reads one; the mechanics are here.
+
+**Read it before you decide.** Ahead of the decisions this step owns, read the `open` entries. Ones bearing on those decisions are the ones you may be able to close. Never re-derive an obligation the chain already recorded — that re-derivation is the failure this artifact exists to end.
+
+**Start one when none is supplied.** Every carrying Mold declares the ledger as an input, but the first Mold in a run receives none. Its absence is not an error: start an empty ledger at the declared filename and proceed.
+
+**Append what you newly surface.** Each new obligation gets a stable kebab-case `id`, `status: open`, `raised_by` set to your own Mold name, and enough in `unmet` / `missing` for a later Mold to act without re-reading the source. Attach it to a draft step via `step` where one applies. Set `blocking: true` only for a computability gap.
+
+**Mark resolved what you close.** When a decision you make discharges an open entry, set `status: resolved`, `resolved_by` to your own Mold name, and `note` to how it closed. Resolving is not deleting — a resolved entry stays in the ledger as the audit trail.
+
+**Carry the rest untouched.** Entries you neither raise nor close pass through unchanged, statuses and provenance intact. Never renumber, re-word, or drop another Mold's entry.
+
+**Never fabricate around an entry.** An open obligation is the honest state. Inventing a connection, a tool id, or a value so the artifact looks complete converts a tracked gap into a silent defect — the specific failure the ledger replaces.
+
+**Record a drop as a drop.** When a decision removes source work from what the run will
+carry — narrowing a design, cutting a plane, dropping an output during repair — append a
+`kind: dropped` entry naming the `units` and a cited `because`. A cut recorded only as
+prose in a brief is invisible to every later Mold; a cut recorded as a `gap` reads as
+something the pipeline could not do rather than something it chose not to do. Recording
+the drop is not the same as discharging it, and a well-written entry is not a substitute
+for the work — an entry makes a cut *legible*, never *justified*.
+
+**A refuted justification reopens its decision.** When work you do contradicts the
+`because` another entry rests on — discovery finds the wrapper a drop assumed absent, a
+later brief supplies the evidence a narrowing assumed missing — set `supersedes` to that
+entry's `id` on your own resolved entry, and return the superseded entry to `status: open`
+with its `because` struck and the contradiction quoted in its `note`. Do not leave a
+refuted rationale standing: the terminal writes these into the final artifact, and a
+surrendered gap whose stated reason is known to be false is worse than an unexplained one.
+Reopening does not oblige *you* to do the dropped work — it obliges the run to stop
+claiming a settled reason it no longer has.
+
+**Record a surrender note; leave the status to the terminal.** When you cannot close an entry and nothing downstream can either — no producer is discoverable, the output cannot be honestly narrowed — leave it `open` and say so in `note`. `surrendered` is a terminal status, set at the end of a run: the escalation cap reached, or the final artifact emitted with the obligation still unmet. Either way the entry stays visible and is written into the final artifact as a labelled gap, never dropped.
+
+**Maintain the escalation budget** — loop Molds only. The `topology_repair` header is loop-level state, described under **Escalation budget (loop-level state)** below. On each escalation the orchestrator increments `escalations`, appends the post-repair open blocking-entry count to `open_history`, and surrenders the still-open blocking entries once `escalations` reaches `cap`.
+
+## Role in topology repair
+
+In the Galaxy per-step loop, the ledger is the substrate the topology-repair escalation rides on (see galaxy-workflow-draft-format and repair-galaxy-draft-topology):
+
+- **implement-galaxy-tool-step** detects, mid-implementation, that a declared step output cannot be computed from its wired inputs. Rather than fabricate, it appends a blocking entry and falls through to repair.
+- **repair-galaxy-draft-topology** reads the open blocking entries, re-wires the affected region (template-tier authoring — one step or a small sub-path), and marks each entry it closes `resolved`; the existing discover-or-author → implement machinery then realizes the new steps.
+- The loop's **decreasing-blocker invariant** counts open blocking entries: each repair escalation must strictly reduce that count, under a hard cap on escalations. When the cap is hit with entries still open, those entries are **surrendered** into the final draft — the clean terminal that replaces spin-or-fabricate.
+
+So the ledger is not a convenience here; it is the countable state the termination guard depends on.
+
+## Escalation budget (loop-level state)
+
+The decreasing-blocker invariant needs more than the entries themselves: "strictly reduce the open count, under a hard cap" is *loop-level* state — it spans iterations and belongs to no single entry. The ledger carries it in a small `topology_repair` header beside the `entries:` list:
+
+```yaml
+topology_repair:
+  escalations: 2           # repairs invoked this run
+  cap: 5                   # hard ceiling; at cap, still-open entries are surrendered
+  open_history: [4, 3, 2]  # open blocking-entry count after each escalation — must be strictly decreasing
+entries:
+  - id: sccmec-evidence-missing
+    status: open
+    blocking: true
+    # … per-entry shape above
+```
+
+- `escalations` — how many times advance-galaxy-draft-step has escalated to repair-galaxy-draft-topology this run. The orchestrator increments it on each escalation.
+- `cap` — the hard ceiling. When `escalations` reaches `cap`, the loop stops escalating and surrenders any still-`open` blocking entries into the final draft as labelled gaps rather than retrying forever.
+- `open_history` — the open blocking-entry count recorded after each escalation. The convergence gate requires it to be strictly decreasing: a repair that fails to lower the open count — or raises it by inserting a producer that is itself uncomputable — is non-convergent and trips the gate immediately, without waiting for the cap.
+
+This block lives in the ledger as a v1 expedient: the ledger is already the carried, cast-visible state the loop reads each iteration. Its better long-term home is the **draft itself** — a workflow-level annotation that travels with the artifact it bounds, so the budget survives even when the ledger is stripped at the terminal. Recorded here for now; migrate when the draft grows a home for it.
+
+## Threaded through the whole design tier
+
+The ledger is not loop-only. Every design-tier Mold in the source→Galaxy pipelines **consumes and re-emits** it — the interface, reference-data, data-flow, IWC-comparison, and template Molds (`*-summary-to-galaxy-interface`, `*-summary-to-galaxy-data-flow`, `*-summary-to-galaxy-reference-data`, `compare-against-iwc-exemplar`, `*-summary-to-galaxy-template`), alongside the change-set Molds on the update path. This is the direct answer to #281: the chain stops re-deriving the same unknowns because each Mold inherits them.
+
+The template Mold's computability review pass is the notable appender ahead of the loop — `*-summary-to-galaxy-template` re-reads each settled step and, where an output needs evidence no input carries, wires the producer or records the gap as a blocking entry. Source-summary Molds upstream of the design tier keep their own free-text open-questions; the design tier is where those formalize into the ledger.
+
+## Open work
+
+- Decide whether the drops should be **counted**, not just labelled. `kind: dropped` and `units` make each cut legible one entry at a time; nothing totals them, so a run that drops a plane four times over still reports four reasonable-looking entries. A conservation rule — every source unit ends the run carried, subsumed, or surrendered, and the buckets sum to the source total — would make the aggregate visible the way `open_history` makes escalation visible. Deliberately not attempted here: labelling is cheap and may prove sufficient, counting needs a definition of "source unit" per source kind.
+- Decide whether entry structure should harden further (typed `unmet` / `missing`, links back to source-summary evidence, machine-checkable `status` transitions) once two or three worked runs exercise it. Hardening it would also give the producing Molds an `output_artifacts[].schema` to cite, which none carry today.
+- Assign terminal surrender of *non-blocking* entries. advance-galaxy-draft-step surrenders open blocking entries at the escalation cap, but an unpinned-parameter entry the design tier appended and nobody closed currently rides out the run without ever being marked `surrendered`. Nothing owns that pass today.
+- Reconcile the design-tier briefs' free-text "open questions" sections with the ledger. Every design Mold still emits both, with no rule for which destination an unresolved choice belongs in.
+- Decide whether the source-summary Molds should also emit structured entries, or keep their free-text open-questions until the design tier formalizes them.
+- Specify how surrendered entries surface in the runnable gxformat2 (a workflow-level annotation, a report output, or a sidecar) when the draft is stripped of `_plan_*` and TODO sentinels.
+- Move the `topology_repair` escalation budget out of the ledger and into the draft (a workflow-level annotation) so it travels with the artifact it bounds; the ledger home is a v1 expedient.

package/extensions/orbit/vendor/skills/debug-galaxy-workflow-output/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: debug-galaxy-workflow-output
+description: "Triage failing Galaxy run outputs; classify the failure surface and capture evidence before recommending repairs."
+---
+
+# debug-galaxy-workflow-output
+
+Follow the procedure below and use the artifact/reference sections as the runtime contract.
+
+## When To Use
+
+- Triage failing Galaxy run outputs; classify the failure surface and capture evidence before recommending repairs.
+
+## Inputs
+
+- Read artifact `workflow-test-result`. Produced by `run-workflow-test`. Structured run handoff from run-workflow-test: Planemo result, invocation/job/artifact context, and the observed failure modality.
+
+## Outputs
+
+- Write artifact `workflow-debug-report` as `workflow-debug-report.md`. Format: `markdown`. Failure-surface classification with captured job/invocation/collection/assertion evidence and a recommended next step or reference-gap follow-up.
+
+## Required Tools
+
+- None declared. Procedure should not assume external CLIs are present.
+
+## Load Upfront
+
+- None declared.
+
+## Load On Demand
+
+- `references/notes/galaxy-collection-semantics.md`: Research note copied verbatim into the bundle. Diagnose collection shape, mapping, reduction, and element-identifier mismatches in failed Galaxy runs. Use when: a failing output is a collection, a mapped output, or an unexpectedly nested/flattened structure.
+- `references/notes/galaxy-collection-semantics.yml`: Companion file copied verbatim into the bundle. Sibling of `references/notes/galaxy-collection-semantics.md`; read it where that note directs.
+- `references/notes/galaxy-tool-job-failure-reference.md`: Research note copied verbatim into the bundle. Interpret Galaxy job-level failure evidence including stdio rules, exit code, job messages, and output dataset state. Use when: a failed workflow test includes errored jobs, tool stderr/stdout, non-zero exit codes, or red output datasets.
+- `references/notes/galaxy-workflow-invocation-failure-reference.md`: Research note copied verbatim into the bundle. Interpret Galaxy invocation-level failure evidence including invocation state, structured messages, and step job summaries. Use when: a failed workflow test has invocation failure, missing workflow outputs, cancelled/paused steps, subworkflow failures, or collection population errors.
+- `references/notes/iwc-shortcuts-anti-patterns.md`: Research note copied verbatim into the bundle. Decide whether a proposed debug fix aligns with accepted IWC testing shortcuts or masks a real failure. Use when: debugging suggests weakening assertions, widening deltas, switching to existence checks, or changing output labels.
+- `references/notes/nextflow-operators-to-galaxy-collection-recipes.md`: Research note copied verbatim into the bundle. Trace collection output failures back to possibly lossy operator translations. Use when: debugging wrong nesting, missing elements, branch merges, bad joins, or gather/reduction mismatches.
+- `references/notes/planemo-asserts-idioms.md`: Research note copied verbatim into the bundle. Classify whether a failure is an assertion-choice problem, tolerance problem, or real workflow-output regression. Use when: planemo reports output assertion failures or generated tests are too strict/too weak.
+- `references/notes/planemo-workflow-test-architecture.md`: Research note copied verbatim into the bundle. Locate which Planemo artifact or Galaxy API surface preserves the failure evidence. Use when: planemo output is ambiguous, structured test JSON is available, or rerunning can be avoided by inspecting an existing invocation.
+
+## Validation
+
+- None declared.
+
+## Procedure
+
+Triage a failing Galaxy workflow test. Take the structured handoff from run-workflow-test, classify the failure surface before proposing any repair, and capture the reference evidence the surface requires. When the failure cannot be classified from existing references, recommend a focused follow-up rather than converting uncertainty into a guessed fix.
+
+Classify before repairing. The same red output can be a tool/job failure, a workflow invocation failure, a collection-output mismatch, a missing workflow output, or an assertion mismatch — and each routes to a different reference surface and a different fix. Locate where the evidence lives first (planemo-workflow-test-architecture).
+
+### Sequence
+
+1. **Classify the first failure surface.** From the run's structured result, decide whether the first failure is a tool/job failure, a workflow invocation failure, a collection-output mismatch, a missing workflow output, or an assertion mismatch. Classify before proposing repairs.
+2. **Capture job-failure evidence.** When a job is in `error`/`failed`/`stopped`, record job id, tool id, exit code, job messages, the stdout/stderr distinction, and output dataset state per galaxy-tool-job-failure-reference; check whether the wrapper's failure semantics already explain it.
+3. **Capture invocation-failure evidence.** When the invocation state or messages indicate scheduling, materialization, cancellation, conditional, or output-resolution failure, record invocation state, the structured message reason, the affected step, any subworkflow path, and the jobs summary per galaxy-workflow-invocation-failure-reference; note whether Planemo surfaced or hid the relevant Galaxy API detail.
+4. **Trace collection mismatches.** When a failing output is a collection or mapped output, diagnose shape, mapping, reduction, and element-identifier mismatches with galaxy-collection-semantics; for workflows translated from Nextflow, trace wrong nesting / missing elements / bad joins back to possibly-lossy operator translations via nextflow-operators-to-galaxy-collection-recipes.
+5. **Read assertion failures honestly.** When the failure is an assertion, use planemo-asserts-idioms to decide whether it is an assertion-choice/tolerance problem or a real output regression. Before weakening an assertion, widening a delta, or switching to an existence check, confirm against iwc-shortcuts-anti-patterns that the relaxation is an accepted IWC shortcut and not masking a real failure.
+6. **Discover reference gaps.** When the failure cannot be classified confidently from the references above, recommend a focused follow-up — reference documentation, pattern capture, API verification, or eval coverage — rather than emitting a repair recipe built on a guess.
+
+## Feedback Mode
+
+- Feedback mode is off unless the caller explicitly enables `--feedback` or supplies a feedback-ledger path.
+- When enabled, read `_feedback.md` before doing the work and use its registered `foundry-feedback.ledger.yml` protocol.
+- Preserve harness-owned run and phase state. Append only concrete observations about a canonical Foundry source asset or a related project that this run showed to be at fault; do not put ordinary workflow requirements in this ledger.
+- Before reporting completion, make one explicit pass over the work you just did. Do not ask yourself whether anything was unclear — recall what happened: where you guessed at something the instructions should have settled, needed information this bundle does not carry, hit an instruction that contradicted another or contradicted the artifacts in front of you, used a packaged reference that did not cover your case, or did something the procedure never describes.
+- Append an entry for each such event that clears the protocol's bar. If none do, append nothing and report `no feedback` explicitly. Silence and a clean pass are not the same thing, and nothing downstream can tell them apart unless you say which one it was.
+- Pass the same ledger path to any subagent used for this work, and merge updates serially so one writer cannot overwrite another.
+
+## Runtime Notes
+
+- Do not read Foundry source files at runtime; use only files packaged in this skill bundle and user-supplied artifacts.
+- Preserve declared artifact filenames unless the user or harness supplies explicit paths.
+- Carry unresolved assumptions into the output artifact instead of silently inventing missing source evidence.