@galaxyproject/orbit 0.0.0 → 0.9.0

package/extensions/orbit/vendor/skills/debug-galaxy-workflow-output/references/notes/galaxy-tool-job-failure-reference.md

@@ -0,0 +1,137 @@
+---
+type: research
+title: "Galaxy tool and job failure reference"
+tags:
+  - target/galaxy
+status: draft
+created: 2026-05-02
+revised: 2026-05-02
+revision: 1
+related_notes:
+  - "galaxy-workflow-invocation-failure-reference"
+  - "planemo-workflow-test-architecture"
+  - "galaxy-collection-semantics"
+related_molds:
+  - "implement-galaxy-tool-step"
+  - "debug-galaxy-workflow-output"
+sources:
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/__init__.py"
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tool_util/parser/xml.py"
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tool_util/output_checker.py"
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs"
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/jobs.py"
+summary: "Reference for Galaxy tool stdio rules, job failure detection, job states, and job API failure surfaces."
+---
+
+# Galaxy Tool And Job Failure Reference
+
+This is reference material, not a debug recipe. Use it to understand what Galaxy can know about a failed tool job and which API surfaces preserve that evidence.
+
+## Model
+
+Galaxy tool failure handling is layered:
+
+- The tool wrapper defines expected failure semantics through `detect_errors`, `<stdio>`, exit-code checks, regex checks, and command strictness.
+- The job runner executes the command and captures exit code plus tool/job stdout and stderr streams.
+- Galaxy evaluates configured failure rules and records structured `job_messages`.
+- The job reaches a terminal state, output datasets may become `error`, and dependent jobs may pause or fail later.
+- Workflow invocation APIs summarize those jobs, but job APIs preserve the most detailed tool-level evidence.
+
+## Tool Wrapper Failure Controls
+
+Important wrapper controls:
+
+| Control | Meaning |
+|---|---|
+| `detect_errors="default"` | For non-legacy XML tools without explicit `<stdio>`, Galaxy defaults to non-zero exit-code failure detection. |
+| `detect_errors="exit_code"` | Adds fatal checks for non-zero exit codes, plus optional OOM exit-code handling. |
+| `detect_errors="aggressive"` | Adds non-zero exit-code checks plus broad stdout/stderr regexes for OOM and generic error text. |
+| `<stdio><exit_code ... /></stdio>` | Adds explicit exit-code ranges or values with levels such as `fatal`, `warning`, or `fatal_oom`. |
+| `<stdio><regex ... /></stdio>` | Adds case-insensitive stdout/stderr regex checks. |
+| `<command strict="...">` | Controls shell strictness; newer tool profiles default to `set -e` behavior. |
+
+Current Galaxy source parses these rules while loading the tool, then stores exit-code and regex rules on the tool object. XML parser behavior lives under `https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tool_util/parser/xml.py`; concrete stdio presets live under `https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tool_util/parser/stdio.py`.
+
+## Rule Ordering And Levels
+
+Galaxy evaluates explicit and preset rules in a defined order:
+
+- Exit-code rules are evaluated before regex rules.
+- Regexes search stdout and/or stderr case-insensitively.
+- A fatal rule stops later checks.
+- Warnings, log messages, and QC messages can produce `job_messages` without failing the job.
+
+Useful levels:
+
+| Level | Effect |
+|---|---|
+| `log` | Records informational message; does not fail the job. |
+| `qc` | Records QC message; does not fail the job. |
+| `warning` | Records warning; does not fail the job. |
+| `fatal` | Fails the job as a generic tool error. |
+| `fatal_oom` | Fails as out-of-memory; runner behavior can use this for OOM handling/resubmission. |
+
+Low-confidence caveat: OOM resubmission behavior is runner/destination configuration dependent. The wrapper and output checker can classify OOM, but retry policy is not solely a wrapper property.
+
+## Job And Dataset States
+
+Job states relevant to workflow tests include:
+
+- In progress or queued: `new`, `queued`, `running`, `waiting`, `upload`, `resubmitted`.
+- Success: `ok`.
+- Failure or terminal problem: `error`, `failed`, `paused`, `stopped`, `deleted`, `skipped` depending context.
+
+Dataset states relevant to downstream failures include `ok`, `error`, `paused`, `failed_metadata`, `deferred`, and `discarded`.
+
+For workflow debugging, do not collapse job state and dataset state. A job can fail, its outputs can become error datasets, and a downstream workflow step can later fail because it consumes those datasets.
+
+## Stream And Message Fields
+
+Galaxy distinguishes tool streams from job/runner streams:
+
+| Field | Meaning |
+|---|---|
+| `tool_stdout` | stdout from the executed tool command. |
+| `tool_stderr` | stderr from the executed tool command. |
+| `job_stdout` | stdout from job wrapper/runner context. |
+| `job_stderr` | stderr from job wrapper/runner context. |
+| `stdout` | Combined stdout compatibility view. |
+| `stderr` | Combined stderr compatibility view. |
+| `job_messages` | Structured failure/warning messages produced by stdio and output checking. |
+| `exit_code` | Process exit code observed by the runner. |
+
+Prefer `job_messages` plus separate streams for reference-quality failure interpretation. Combined `stdout` and `stderr` are useful for humans but lose provenance.
+
+## API Surfaces
+
+Useful job APIs from Galaxy source under `https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/jobs.py`:
+
+| API | Use |
+|---|---|
+| `GET /api/jobs` | Filter jobs by state, tool id, workflow id, invocation id, history id, etc. |
+| `GET /api/jobs/{job_id}` | Basic job detail. |
+| `GET /api/jobs/{job_id}?full=true` | Full job detail, including streams and `job_messages`. |
+| `GET /api/jobs/{job_id}/stdout` | Combined stdout as plain text. |
+| `GET /api/jobs/{job_id}/stderr` | Combined stderr as plain text. |
+| `GET /api/jobs/{job_id}/console_output` | Live or stored tool stdout/stderr, useful while a job is running. |
+| `GET /api/jobs/{job_id}/inputs` | Input datasets for the job. |
+| `GET /api/jobs/{job_id}/outputs` | Output datasets and output collections. |
+| `GET /api/jobs/{job_id}/metrics` | Job metrics, if available. |
+| `GET /api/jobs/{job_id}/common_problems` | Known simple problems such as empty or duplicate inputs. |
+
+Admin-only or admin-enriched fields can include command line, traceback, destination, runner, handler, external id, and destination parameters. Do not assume a normal workflow-testing API key can retrieve everything.
+
+## Durable Reference Use
+
+This note should inform generated skills when they need to preserve or inspect tool-level failure evidence:
+
+- A concrete workflow step should preserve tool id, version, input labels, output labels, datatype and collection shape so a later job failure can be traced back to the authoring decision.
+- A runtime failure should not be classified from stderr alone if `job_messages`, exit code, and wrapper stdio rules are available.
+- A workflow invocation failure may be caused by an upstream job, but invocation APIs are not a substitute for full job detail.
+
+## Low-Confidence Or Version-Sensitive Points
+
+- Tool profile defaults changed over time. Prefer current Galaxy source and wrapper profile over old generic rules.
+- Some docs describe failure messages as stream text; current code preserves structured `job_messages` separately.
+- OOM handling depends on job runner and destination configuration.
+- Planemo may print or summarize job detail differently from raw Galaxy APIs; see planemo-workflow-test-architecture.

package/extensions/orbit/vendor/skills/debug-galaxy-workflow-output/references/notes/galaxy-workflow-invocation-failure-reference.md

@@ -0,0 +1,138 @@
+---
+type: research
+title: "Galaxy workflow invocation failure reference"
+tags:
+  - target/galaxy
+status: draft
+created: 2026-05-02
+revised: 2026-05-02
+revision: 1
+related_notes:
+  - "galaxy-tool-job-failure-reference"
+  - "planemo-workflow-test-architecture"
+  - "galaxy-collection-semantics"
+related_molds:
+  - "run-workflow-test"
+  - "debug-galaxy-workflow-output"
+  - "validate-galaxy-workflow"
+sources:
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/schema/invocation.py"
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/workflow/run.py"
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/workflow/modules.py"
+  - "https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/workflows.py"
+summary: "Reference for Galaxy workflow invocation states, messages, failure reasons, and invocation API surfaces."
+---
+
+# Galaxy Workflow Invocation Failure Reference
+
+This note describes workflow-level failure surfaces in Galaxy. It is separate from galaxy-tool-job-failure-reference because invocation state answers whether Galaxy could schedule and drive the workflow, while job state answers whether individual tool jobs succeeded.
+
+## Invocation Versus Job Failure
+
+Important distinction:
+
+- Invocation state says whether Galaxy scheduled, cancelled, failed, or completed the workflow invocation.
+- Job state says whether jobs produced by invocation steps succeeded or failed.
+- Invocation messages explain scheduler/evaluation/cancellation problems.
+- Step states usually describe scheduling progress, not actual job success, unless a legacy serialization mode substitutes job state.
+
+A robust workflow test reference should inspect both invocation APIs and job APIs.
+
+## Invocation States
+
+Galaxy invocation states are defined in `https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/schema/invocation.py` and related model code.
+
+| State | Meaning |
+|---|---|
+| `new` | Newly created invocation. |
+| `requires_materialization` | Deferred or undeferred inputs must materialize before scheduling can continue. |
+| `ready` | Ready for another scheduler iteration. |
+| `scheduled` | Workflow has been scheduled. Older clients may treat this as successful terminal state. |
+| `cancelling` | Scheduler will cancel jobs and subworkflow invocations. |
+| `cancelled` | Cancellation terminal state. |
+| `failed` | Scheduler/evaluation/materialization failure terminal state. |
+| `completed` | Newer completion state after all jobs reach terminal states. |
+
+Version caveat: `completed` may not be uniformly available across all Galaxy/Planemo combinations. Planemo accounts for both `scheduled` and `completed` as success-like invocation states.
+
+## Invocation Message Reasons
+
+Structured invocation messages are the main durable surface for workflow-level failures.
+
+| Reason | Typical meaning |
+|---|---|
+| `dataset_failed` | Dataset input or upstream dataset was failed/unusable. |
+| `collection_failed` | Collection was failed, incompletely populated, or unusable. |
+| `job_failed` | Upstream dependent job failed. |
+| `output_not_found` | Expected step/subworkflow output could not be resolved. |
+| `expression_evaluation_failed` | Workflow expression or condition evaluation failed. |
+| `when_not_boolean` | Conditional `when` expression did not return a boolean. |
+| `unexpected_failure` | Scheduler or evaluation failure without a more specific safe reason. |
+| `workflow_parameter_invalid` | Workflow parameter validation failed. |
+| `step_input_deleted` | Step input dataset/collection was deleted. |
+| `history_deleted` | Invocation history was deleted, causing cancellation. |
+| `user_request` | User/API cancellation. |
+| `cancelled_on_review` | Pause/review step rejected continuation. |
+| `workflow_output_not_found` | Warning: declared workflow output was not found. |
+
+Message records can include affected workflow step id, dependent step id, job id, HDA/HDCA ids, output name, and nested workflow step index path. These fields matter more than free-text messages for reference-quality diagnosis.
+
+## Common Workflow-Level Failure Surfaces
+
+| Surface | Reference signal |
+|---|---|
+| Request-time validation | API returns an error before useful invocation state exists. |
+| Input materialization | Invocation enters `requires_materialization`, then may fail with `dataset_failed`. |
+| Conditional evaluation | Invocation messages include `expression_evaluation_failed` or `when_not_boolean`. |
+| Missing output graph edge | Invocation messages include `output_not_found`. |
+| Deleted/purged inputs | Invocation messages include `step_input_deleted` or `dataset_failed`. |
+| Upstream job failure | Invocation messages or summaries point to `job_failed`; job API has details. |
+| Collection population failure | Invocation messages include `collection_failed`; inspect HDCA and mapped jobs. |
+| Pause/review rejection | Cancellation reason `cancelled_on_review`. |
+| User/history cancellation | Cancellation messages `user_request` or `history_deleted`. |
+| Warning-only missing output | `workflow_output_not_found` warning; may later become Planemo assertion failure. |
+
+## API Surfaces
+
+Useful invocation APIs from `https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/workflows.py`:
+
+| API | Use |
+|---|---|
+| `GET /api/invocations/{invocation_id}` | Invocation state, messages, inputs, outputs, and steps. |
+| `GET /api/invocations/{invocation_id}?step_details=true` | Richer step detail, jobs, outputs, and output collections. |
+| `GET /api/invocations/steps/{step_id}` | Detail for one invocation step. |
+| `GET /api/invocations/{invocation_id}/steps/{step_id}` | Step detail scoped to invocation. |
+| `GET /api/invocations/{invocation_id}/jobs_summary` | Job state summary across the invocation. |
+| `GET /api/invocations/{invocation_id}/step_jobs_summary` | Job state summary per workflow invocation step. |
+| `GET /api/invocations/{invocation_id}/completion` | Completion record, if available. |
+| `DELETE /api/invocations/{invocation_id}` | Cancel invocation. |
+| `PUT /api/invocations/{invocation_id}/steps/{step_id}` | Continue or cancel pause/review step. |
+| `GET /api/invocations/{invocation_id}/report` | Invocation report surface. |
+
+Workflow aliases under `/api/workflows/{workflow_id}/invocations/...` also exist.
+
+## Planemo Caveats
+
+Planemo polls invocation and job state, but its user-facing output may not expose all structured invocation messages. Treat Planemo terminal output as a summary; use raw invocation and job APIs when the failure class matters.
+
+Likely noisy boundaries:
+
+- Planemo may show failed job details but omit structured invocation messages.
+- `step_jobs_summary` is useful in Galaxy but not necessarily surfaced directly by Planemo output.
+- Subworkflow paths can be represented in Galaxy message fields but may not be displayed by Planemo.
+- A workflow can be `completed` or `scheduled` while one or more jobs failed; job summaries must still be inspected.
+
+## Durable Reference Use
+
+Use this note when a workflow run fails before or around job scheduling, when outputs are missing without an obvious tool stderr cause, or when Planemo only reports a generic invocation failure.
+
+The goal is not to prescribe a single repair loop. The goal is to preserve which Galaxy API surface proves the failure mode so later Molds can decide whether the defect belongs to data-flow, template, concrete step implementation, workflow test assertions, or runtime environment.
+
+## Verification Gaps
+
+Actual runs should verify:
+
+- Whether Planemo surfaces invocation `messages` for each reason.
+- Whether `completed` and `/completion` are populated in the target Galaxy used by tests.
+- How mapped collections and subworkflow failures appear in `step_jobs_summary`.
+- How warning-only `workflow_output_not_found` interacts with Planemo assertions.

package/extensions/orbit/vendor/skills/debug-galaxy-workflow-output/references/notes/iwc-shortcuts-anti-patterns.md

@@ -0,0 +1,357 @@
+---
+type: research
+tags:
+  - target/galaxy
+status: draft
+created: 2026-04-30
+revised: 2026-05-03
+revision: 2
+related_notes:
+  - "galaxy-workflow-testability-design"
+  - "iwc-test-data-conventions"
+  - "planemo-asserts-idioms"
+  - "implement-galaxy-workflow-test"
+  - "tests-format"
+  - "iwc-conditionals-survey"
+  - "iwc-map-over-lifecycle-survey"
+  - "iwc-tabular-operations-survey"
+  - "iwc-transformations-survey"
+summary: "What IWC test suites cut corners on (accepted) vs what's a code smell — existence-only probes, sim_size deltas, image dim checks, label coupling."
+---
+
+# IWC test-suite shortcuts and anti-patterns
+
+## Purpose
+
+When an agent translates or authors a Galaxy workflow for IWC submission, the test suite it writes will be reviewed against IWC's *de facto* style — not against an idealized assertion ladder. That style routinely tolerates assertions that look weak in isolation. This note distinguishes the corner-cutting that is **normal and accepted** in the corpus from the patterns that an agent should treat as **smells** worth flagging.
+
+This note owns accepted-vs-smell calls. For positive workflow-structure guidance behind label stability, checkpoint promotion, and collection identifier design, use galaxy-workflow-testability-design.
+
+Grounding: 115 `*-tests.yml` files under `workflow-fixtures/iwc-src/workflows/` (mirror of `galaxyproject/iwc`), prior synthesis in `galaxy-brain/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md`. Path citations below are relative to `iwc-src/workflows/` unless absolute.
+
+## TL;DR rules of thumb
+
+1. **Default to tolerant assertions.** `compare: sim_size` + `delta:`, `has_image_*` + `delta:`, `has_text` substring, `has_h5_keys`, `has_n_lines` + `delta:` are *the IWC vocabulary*. Strict `compare: diff` or exact-`file:` is the exception, used only when the upstream tool is fully deterministic on fixed inputs.
+2. **No negative tests.** `expect_failure:` does not appear in the corpus. Don't author one.
+3. **No checksums.** `md5:` / `checksum:` do not appear on outputs in the corpus. SHA-1 hashes are used on **inputs** (integrity of remote fetch), never on output assertions.
+4. **Preserve labels.** Inputs and outputs are referenced by label. Renaming silently breaks tests; treat label changes as breaking changes that require a sibling `-tests.yml` update.
+5. **Big data goes to Zenodo.** In-repo `test-data/` is for toy fixtures and expected outputs only.
+
+---
+
+## 1. Existence-only content probes
+
+### Accepted
+
+The HyPhy test files reduce JSON output validation to "starts with `{`":
+
+- `comparative_genomics/hyphy/hyphy-core-tests.yml:32-71` — four output collections (`meme_output`, `prime_output`, `busted_output`, `fel_output`), each gene element asserts `has_text: text: "{"` and nothing else.
+- `comparative_genomics/hyphy/hyphy-compare-tests.yml`, `capheine-core-and-compare-tests.yml` — same pattern across the rest of the family.
+
+This is **accepted** because:
+- HyPhy's MEME/PRIME/BUSTED/FEL/CFEL/RELAX statistical outputs embed run-dependent floats throughout (likelihoods, AIC, posterior probabilities). Substring assertions on numeric fields would fail intermittently.
+- Selecting any specific gene name or category in the JSON would couple the test to internal HyPhy keying that the wrapper has changed across versions.
+- The assertion does verify "the tool ran, produced JSON, did not crash, and the collection structure matches expected element identifiers" — which is genuinely useful given HyPhy's history of opaque failures.
+
+A quick scan finds **298 lines** matching the existence-style `has_text:` pattern across the corpus (`grep "has_text:\s*$\|text: \"{\"$"` yields 298 hits across many files). It is widespread, not a HyPhy quirk.
+
+Other variants in the same accepted-shortcut family:
+
+- **First-line-of-header probes**: `amplicon/amplicon-mgnify/.../mgnify-amplicon-pipeline-v5-rrna-prediction-tests.yml:46-49` asserts `has_text: "# mapseq v1.2.6 (Jan 20 2023)"` — version banner only.
+- **`has_n_columns` schema probes**: same file lines 50-52, 67-69 — "the table has 15 columns" / "4 columns" with no row-content check.
+- **`has_h5_keys` structure probes**: `scRNAseq/scanpy-clustering/.../Preprocessing-...-Scanpy-tests.yml:27-32, 159-166` — confirms AnnData has `obs/louvain`, `var/highly_variable`, `uns/rank_genes_groups`, etc., but says nothing about cluster labels or values. Also `imaging/tissue-microarray-analysis/multiplex-tissue-microarray-analysis/multiplex-tma-tests.yml` for Merged anndata.
+
+### Smell
+
+Existence probes on **deterministic** outputs. If the underlying tool is deterministic on fixed inputs (alignment, simple QC stats, well-defined transformations), reducing to `has_text: "{"` is laziness — agent should at least pull a known stable substring from the expected JSON.
+
+Heuristic for an agent: **if the source workflow's tool is on the "stochastic / floating-point heavy / version-fragile" list (HyPhy, RepeatModeler, scanpy plots, MCMC samplers, ML inference), existence probes are accepted. Otherwise, prefer `has_text` against a stable token from a real output.**
+
+---
+
+## 2. Size-only comparisons (`compare: sim_size` + `delta:`)
+
+### Accepted
+
+Canonical example: `repeatmasking/RepeatMasking-Workflow-tests.yml:11-46` — every output is `compare: sim_size` with `delta: 30000` (30 KB) on small outputs and `delta: 90000000` (90 MB!) on the Stockholm seed-alignment file. RepeatModeler's discovered repeat families differ run-to-run; only output magnitude is reproducible.
+
+`grep "compare: sim_size"` returns 9 files using this pattern:
+- `repeatmasking/RepeatMasking-Workflow-tests.yml`, `repeatmasking/Repeat-masking-with-RepeatModeler-and-RepeatMasker-tests.yml` (RepeatModeler — large delta band)
+- `epigenetics/hic-hicup-cooler/chic-fastq-to-cool-hicup-cooler-tests.yml`, `epigenetics/hic-hicup-cooler/hic-fastq-to-cool-hicup-cooler-tests.yml` (HiC matrices)
+- `genome_annotation/functional-annotation/functional-annotation-of-sequences/Functional_annotation_of_sequences-tests.yml`
+- `VGP-assembly-v2/kmer-profiling-hifi-trio-VGP2/kmer-profiling-hifi-trio-VGP2-tests.yml`
+- `genome-assembly/polish-with-long-reads/Assembly-polishing-with-long-reads-tests.yml`
+- `scRNAseq/baredsc/baredSC-1d-logNorm-tests.yml`, `scRNAseq/baredsc/baredSC-2d-logNorm-tests.yml` (Bayesian sampling — uses `delta_frac:` here)
+
+### Delta-magnitude survey
+
+From `grep "delta: [0-9]+"` distribution across the corpus:
+
+| Delta band | Count | Typical use |
+|---|---|---|
+| 4–100 (tiny) | ~20 | image pixel dimensions, line counts |
+| 1K–10K | ~40 | small text/tabular outputs, plot PNGs |
+| 25K–100K | ~25 | mid-size reports, multi-page plots |
+| 200K–1M | ~10 | report HTML, BAM stats |
+| 1M–10M | ~10 | medium BAM/BCF/cool files |
+| 10M+ (up to 90M) | ~7 | RepeatModeler libraries, large alignments |
+
+The 90 MB delta on `RepeatMasking-Workflow-tests.yml:20` is at the extreme. It says "this output is somewhere between zero bytes and 180 MB" — effectively only catches the empty-output failure mode. Accepted because RepeatModeler's seed alignments are known to vary by tens of MB across runs.
+
+### `delta_frac:`
+
+Used in 3 files (`scRNAseq/baredsc/*`, `genome-assembly/polish-with-long-reads/*`). Preferred over absolute `delta:` when the expected output size scales with input. An agent translating a workflow whose output size depends on input volume should consider `delta_frac:` over `delta:`.
+
+### Smell
+
+`compare: sim_size` on outputs from a **deterministic** tool. If the tool is bwa/bowtie2/samtools-sort with fixed seeds and pinned versions, there's no excuse for size-only — `compare: diff` (with modest `lines_diff:`) or content assertions are appropriate.
+
+Also a smell: stacking size + `has_image_*` checks on a PNG without any content assertion when the workflow's claim is *about the data shown* (e.g., a clustering plot). The corpus does this routinely (Scanpy file below) — accepted, but a translated workflow that has a more deterministic plotter should do better.
+
+---
+
+## 3. Image plot assertions
+
+### Accepted
+
+`scRNAseq/scanpy-clustering/.../Preprocessing-...-Scanpy-tests.yml:33-205` is the dense example. ~15 PNG outputs each get the same triple:
+
+```yaml
+UMAP of louvain:
+  has_size:
+    size: 68416
+    delta: 6000
+  has_image_width:
+    width: 601
+    delta: 30
+  has_image_height:
+    height: 429
+    delta: 25
+```
+
+What this catches:
+- Plot was rendered (non-zero size).
+- Render dimensions are stable (matplotlib defaults didn't drift, theme didn't change).
+- Approximate file size hasn't shifted by an order of magnitude (no catastrophic content change like all-white or all-noise).
+
+What this **misses**: cluster assignments wrong, axes mislabeled, points in wrong positions, colors swapped, the wrong subset plotted, NaN handling regression. Two visually different UMAPs can have identical width/height/size-within-10%.
+
+Other observed image-assertion users (`grep "has_image"`):
+- `imaging/tissue-microarray-analysis/tissue-microarray-analysis/tissue-micro-array-analysis-tests.yml`
+- `imaging/tissue-microarray-analysis/multiplex-tissue-microarray-analysis/multiplex-tma-tests.yml`
+
+The TMA tests use a friendlier shorthand: `has_size: size: 181K, delta: 50K` (human-readable units).
+
+### Smell
+
+Asserting `has_image_width`/`has_image_height` with **zero delta** on a tool that re-encodes (PNG round-trips through matplotlib) is brittle. The corpus uses 5–10% deltas; an agent emitting `delta: 0` should be flagged unless the renderer is byte-stable.
+
+Note `has_image_channels`, `has_image_center_of_mass` are documented (galaxy XSD) but **not observed** in the sampled corpus. An agent with a deterministic mask/segmentation output could use `has_image_center_of_mass` to actually verify spatial correctness — this would be an *upgrade* over the current corpus norm, not a smell.
+
+---
+
+## 4. Happy-path-only culture (`expect_failure:`)
+
+`grep -r "expect_failure"` over all 115 tests files returns **zero hits**. The IWC corpus has no negative tests. Period.
+
+This is a structural property of IWC: the workflows are published artifacts intended to *succeed* on canonical inputs. Adversarial / error-path testing happens in tool wrappers, not in workflow tests.
+
+**Implication for an agent:** Do not author `expect_failure:` cases when translating a workflow. If the source pipeline (e.g., nf-core) had a "fail on bad reference" test, drop it — it doesn't belong in IWC. If the validation logic is important, it should be in a wrapper-level tool test, not a workflow test.
+
+---
+
+## 5. `md5:` / `checksum:` rarity
+
+`grep -r "md5:\|checksum:"` over `*-tests.yml`: **zero hits**.
+
+SHA-1 `hashes:` blocks are pervasive — but exclusively on **inputs** (`hash_function: SHA-1` paired with a remote `location:`), to guard against silent corruption of the fetched fixture. Output assertions never use them.
+
+Accepted. The reason is empirical: outputs of real bioinformatics tools (BAM with PG headers and timestamps, VCFs with command-line provenance, JSON with run dates) are almost never byte-stable across runs. A `checksum:` would fail intermittently.
+
+**Smell:** an agent emitting `checksum:` or `md5:` on a workflow output. Even for "fully deterministic" tools, embedded provenance breaks checksums. Use `compare: diff` + `lines_diff:` instead, or content assertions.
+
+---
+
+## 6. Output label coupling
+
+Test files key outputs by **workflow label**, with spaces, capitals, punctuation preserved verbatim. Examples:
+
+- `scanpy-clustering/.../Preprocessing-...-Scanpy-tests.yml:27` — `Anndata with Celltype Annotation:` (spaces, mixed case)
+- `scanpy-clustering/.../Preprocessing-...-Scanpy-tests.yml:33` — `UMAP of louvain and top ranked genes:`
+- `imaging/tissue-microarray-analysis/multiplex-tissue-microarray-analysis/multiplex-tma-tests.yml` — `Merged anndata:`, `Spatial Scatterplot Montage:`
+- `consensus-from-variation-tests.yml:30` — `multisample_consensus_fasta:` (snake-case style)
+
+**Both styles coexist.** Snake-case (older / SARS-CoV-2 family) and natural-language-with-spaces (newer / scanpy / TMA) are equally valid. Reviewers do not enforce a single convention.
+
+### Coupling consequences
+
+Renaming an output label in the `.ga` without updating the sibling `-tests.yml` is a silent breakage:
+- Test references the old label → key not present in invocation outputs → assertion mismatch surfaces as an opaque "output not found" error in planemo.
+- `planemo workflow_lint --iwc` enforces that workflow outputs are *labeled*, not that test labels match.
+
+**Discipline observed:** every output a test asserts on is a labeled workflow output. The corpus does not assert on positional / unlabeled outputs.
+
+**Smell:** a translated workflow with unlabeled outputs that later need test coverage. Agent should label every output it intends to assert on, before writing assertions.
+
+Positive design guidance now lives in galaxy-workflow-testability-design: pick stable labels before test authoring and treat renames as test-breaking API changes.
+
+---
+
+## 7. Intermediate-step output gap
+
+`-tests.yml` can only assert on workflow-level **outputs** (entries in the `.ga`'s top-level outputs). Intermediate step results are inaccessible to assertions.
+
+Observed workaround across the corpus: **promote the intermediate to a workflow output**. This is visible indirectly — many workflows expose what would naturally be intermediates as labeled outputs solely for testability:
+
+- `scanpy-clustering` exposes `Initial Anndata General Info`, `Anndata with raw attribute`, `Plot highly variable`, `Elbow plot of PCs and variance` — these are mid-pipeline checkpoints surfaced specifically to be assertable. Compare counts: 22 outputs asserted vs the 7-or-so "user-meaningful" final artifacts.
+- `MAGs-generation-tests.yml` exposes a `Full MultiQC Report` even though MultiQC is logically intermediate to MAG annotation.
+
+**Cost:** "test-only" outputs clutter the workflow's user-facing output list. Reviewers tolerate this in exchange for testability.
+
+**Accepted shortcut:** promoting an intermediate to a workflow output for test purposes. Not a smell.
+
+**Smell:** asserting on a step output via some side-channel (e.g., relying on Galaxy collection ordering, indexing into `tool_state`). The corpus does not do this and an agent should not invent it.
+
+Positive design guidance now lives in galaxy-workflow-testability-design: promote assertable checkpoints deliberately, especially when final reports or plots can only support weak smoke tests.
+
+---
+
+## 8. Remote-data fragility
+
+### Pattern
+
+Overwhelming preference for **Zenodo** as the input store. Every remote `location:` is paired with a SHA-1 `hashes:` block. Examples already cited in nearly every snippet above.
+
+Non-Zenodo remote sources observed (from `grep "ftp.sra.ebi\|ftp://\|figshare\|github.com"`):
+- `virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml:27,34,48,55` — `ftp://ftp.sra.ebi.ac.uk/...` SRA fastqs
+- `amplicon/amplicon-mgnify/.../mgnify-amplicon-pipeline-v5-rrna-prediction-tests.yml:21` — `ftp://ftp.ebi.ac.uk/...` reference DB
+- `data-fetching/parallel-accession-download/parallel-accession-download-tests.yml`, `VGP-assembly-v2/Plot-Nx-Size/...` — accession-driven
+- `variant-calling/generic-variant-calling-wgs-pe/Generic-variation-analysis-on-WGS-PE-data-tests.yml` — github raw URLs
+
+### The fragility
+
+- **Zenodo is a single point of failure across CI.** A Zenodo outage breaks every IWC PR concurrently. SHA-1 hashes guard against silent corruption but provide no mitigation for outages or HTTP 503s.
+- **EBI/SRA FTP is even less reliable** — observed flake-prone in the broader Galaxy CI history.
+- No retry / backoff configured at the test-format level; planemo-ci-action's defaults handle transient failures only via re-running the chunk job manually.
+
+### Accepted
+
+This is just life in the IWC. Don't try to "fix" it in a translated workflow by inlining large data — reviewers will push back (see §10).
+
+### Smell
+
+Inputs hosted on a contributor's personal endpoint, S3 bucket, or Dropbox. Reviewers ask for migration to Zenodo before merge.
+
+---
+
+## 9. `compare: diff` on timestamped outputs
+
+`compare: diff` usage from `grep`:
+
+- `sars-cov-2-variant-calling/sars-cov-2-ont-artic-variant-calling/ont-artic-variation-tests.yml:32` — `compare: diff, lines_diff: 6` on annotated VCF
+- `sars-cov-2-variant-calling/sars-cov-2-pe-illumina-wgs-variant-calling/pe-wgs-variation-tests.yml:35` — same
+- `imaging/fluorescence-nuclei-segmentation-and-counting/segmentation-and-counting-tests.yml:16` — `lines_diff: 0`
+- `variant-calling/variation-reporting/Generic-variation-analysis-reporting-tests.yml:21,25,29,33` — `lines_diff: 6`
+- `variant-calling/generic-variant-calling-wgs-pe/Generic-variation-analysis-on-WGS-PE-data-tests.yml:39,45,55` — `lines_diff: 6`
+
+The `lines_diff: 6` constant is suspicious — 6 lines is the typical VCF header preamble that embeds `##fileDate=...` and `##source=...`. The use is **defensible**: tolerance window matches the known mutable header lines, content lines are diffed strictly.
+
+### Smell
+
+- `compare: diff` with `lines_diff: 0` on a file that contains *any* timestamp, command-line capture, version banner, or random tie-break (hash-ordered dictionaries in Python output, etc.). The single observed `lines_diff: 0` case (`segmentation-and-counting-tests.yml:16`) appears to be on a numeric tabular output where it's defensible — verify content type before flagging.
+- `compare: diff` on a BAM. BAM headers include `@PG` lines with full command lines and Galaxy job IDs. Use `has_size` + content extracts via `has_archive_member` or `samtools view`-piped XML asserts — not byte-level diff.
+
+**Recommended replacement** when timestamps appear:
+- For VCF: `compare: diff, lines_diff: <header-line-count>` (corpus convention is 6).
+- For tabular reports: `has_text` against stable column headers + `has_n_columns`.
+- For HTML reports (MultiQC etc.): `has_text` substring on stable section names — example `short-read-qc-trimming/short-read-quality-control-and-trimming-tests.yml:25-28` asserts `"Filtered Reads"` substring on the MultiQC HTML report rather than diffing.
+
+---
+
+## 10. Reviewer-feedback recurring asks
+
+Synthesized from the COMPONENT_GALAXY_WORKFLOW_TESTING.md analysis (sections 9 and the "Common PR-review feedback" subsection) plus structural observation of what every accepted workflow has and rejected drafts apparently lack:
+
+| Reviewer ask | Where it bites | Source |
+|---|---|---|
+| **Creator `identifier:` must be a full ORCID URI** (`https://orcid.org/...`), not bare ID. | `.ga` frontmatter `creator:` block. Most common lint failure. | planemo PR #1458; `consensus-from-variation.ga:4-10` shows the conformant shape. |
+| **Move large inputs to Zenodo.** Inline `path: test-data/big.bam` for >1 MB inputs gets pushback. | `-tests.yml` job inputs. | `iwc/workflows/README.md` (per prior analysis). |
+| **Bump `release` + add CHANGELOG.md entry in the same PR.** | `.ga` `release:` and sibling `CHANGELOG.md`. Enforced by `bump_version.py`. | `iwc/workflows/README.md:217-247`. |
+| **Generate tests via `planemo workflow_test_init --from_invocation <id>`**, not by hand. Reviewers push back on hand-authored job blocks. | Any new test contribution. | help.galaxyproject.org thread 13903. |
+| **Don't use `compare: diff` on outputs that embed timestamps.** Switch to `has_text`/`has_n_lines` with `delta:`. | See §9. | Recurring review comment. |
+| **Add labeled outputs for any output you assert on.** Unlabeled outputs caught by `planemo workflow_lint --iwc`. | `.ga` outputs. | §6 above. |
+| **Hashes on every remote `location:`.** SHA-1 block paired with the URL. Reviewers spot-check. | `-tests.yml` job inputs. | Universal in the corpus; missing hashes get flagged. |
+
+### Smell to flag for an agent submission
+
+- Bare ORCID ID (`0000-0002-...`) in `creator:` instead of full URL.
+- Test job referencing >1 MB local fixture instead of a Zenodo URL.
+- PR that bumps a workflow without CHANGELOG / `release:` bump.
+- Hand-authored `-tests.yml` that reads "too clean" — reviewers know `--from_invocation` output has a recognizable fingerprint.
+
+---
+
+## Summary cheatsheet for the implement-galaxy-workflow-test mold
+
+**Use these freely (accepted shortcuts):**
+- `has_text: text: "{"` for stochastic JSON outputs.
+- `compare: sim_size, delta:` for non-deterministic file outputs; pick delta from §2 distribution by tool family.
+- `has_image_width/height/has_size` triple with 5–10% delta for matplotlib plots.
+- `has_h5_keys` for AnnData/HDF5 — assert structure not values.
+- Promoting intermediates to workflow outputs to make them assertable.
+- Labels with spaces, mixed case, punctuation as the output key.
+- SHA-1 hashes on every input `location:`.
+
+**Avoid (smells reviewers or future-you will catch):**
+- `expect_failure:` — not an IWC pattern.
+- `md5:` / `checksum:` on outputs.
+- `compare: diff, lines_diff: 0` on anything containing timestamps, BAM `@PG` lines, or Python dict ordering.
+- `has_image_*` with zero delta.
+- Existence-only `has_text: "{"` on outputs from a deterministic tool.
+- Asserting on positional/unlabeled outputs.
+- Inlining >1 MB binary fixtures in `test-data/`.
+- Bare ORCID identifier in `.ga` frontmatter.
+
+**Review-time checklist before submission:**
+1. Every output asserted on has a label in the `.ga`.
+2. Every remote `location:` has a `hashes: SHA-1` block.
+3. Inputs >1 MB live on Zenodo, not in `test-data/`.
+4. `release:` bumped and `CHANGELOG.md` updated if `.ga` changed.
+5. `creator.identifier:` is a full `https://orcid.org/...` URL.
+6. Test was generated by `planemo workflow_test_init --from_invocation`, not hand-written.
+
+---
+
+## Sources
+
+- Positive workflow-structure guidance: galaxy-workflow-testability-design.
+- Prior synthesis: `https://github.com/jmchilton/galaxy-brain/blob/main/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md` (sections 2c, 2d, 2e, 9).
+- Corpus root: `https://github.com/galaxyproject/iwc/blob/main/workflows/` (115 `*-tests.yml` files across 22 categories).
+- Specific files cited:
+  - `comparative_genomics/hyphy/hyphy-core-tests.yml`, `hyphy-compare-tests.yml`, `capheine-core-and-compare-tests.yml`
+  - `repeatmasking/RepeatMasking-Workflow-tests.yml`, `Repeat-masking-with-RepeatModeler-and-RepeatMasker-tests.yml`
+  - `scRNAseq/scanpy-clustering/Preprocessing-and-Clustering-of-single-cell-RNA-seq-data-with-Scanpy-tests.yml`
+  - `read-preprocessing/short-read-qc-trimming/short-read-quality-control-and-trimming-tests.yml`
+  - `virology/pox-virus-amplicon/pox-virus-half-genome-tests.yml`
+  - `metabolomics/lcms-preprocessing/Mass_spectrometry__LC-MS_preprocessing_with_XCMS-tests.yml`
+  - `sars-cov-2-variant-calling/sars-cov-2-consensus-from-variation/consensus-from-variation-tests.yml`
+  - `sars-cov-2-variant-calling/sars-cov-2-pe-illumina-wgs-variant-calling/pe-wgs-variation-tests.yml`
+  - `sars-cov-2-variant-calling/sars-cov-2-ont-artic-variant-calling/ont-artic-variation-tests.yml`
+  - `variant-calling/variation-reporting/Generic-variation-analysis-reporting-tests.yml`
+  - `variant-calling/generic-variant-calling-wgs-pe/Generic-variation-analysis-on-WGS-PE-data-tests.yml`
+  - `imaging/fluorescence-nuclei-segmentation-and-counting/segmentation-and-counting-tests.yml`
+  - `imaging/tissue-microarray-analysis/multiplex-tissue-microarray-analysis/multiplex-tma-tests.yml`
+  - `amplicon/amplicon-mgnify/mgnify-amplicon-pipeline-v5-rrna-prediction/mgnify-amplicon-pipeline-v5-rrna-prediction-tests.yml`
+  - `epigenetics/atacseq/atacseq-tests.yml`
+  - `epigenetics/hic-hicup-cooler/chic-fastq-to-cool-hicup-cooler-tests.yml`, `hic-fastq-to-cool-hicup-cooler-tests.yml`
+  - `microbiome/mags-building/MAGs-generation-tests.yml`
+  - `genome-assembly/polish-with-long-reads/Assembly-polishing-with-long-reads-tests.yml`
+  - `scRNAseq/baredsc/baredSC-1d-logNorm-tests.yml`, `baredSC-2d-logNorm-tests.yml`
+- Corpus-wide grep tallies:
+  - `expect_failure`: 0 hits.
+  - `md5:|checksum:` on outputs: 0 hits.
+  - `compare: sim_size`: 9 files.
+  - `compare: diff`: 5 files (variant-calling and imaging).
+  - `has_image_*`: 3 files.
+  - Existence-style `has_text:` patterns: ~298 line matches.
+  - `delta_frac:`: 3 files.