@archmax-ai/harness 0.1.0 → 0.2.0

package/skills/archmax-harness/references/cases.md

@@ -0,0 +1,41 @@
+# Cases — the `*.test.yaml` grammar
+
+Load this when writing or fixing a case. Worked examples for every shape
+(mocks, human decisions, triggered starts, fixtures):
+[`hook-and-test-scripts.md`](hook-and-test-scripts.md) Part 2.
+
+## Keys, actions, assertions
+
+One YAML document per `tests/<case>.test.yaml`; the case id is the
+tests-relative path minus the extension (the `test` filter is a substring of
+it). Host-interpreted; no sandbox runs a case. **Unknown keys are errors.**
+
+| Key | Shape |
+| --- | --- |
+| `title` | required, ≤ 60 chars, one line: the behaviour under test |
+| `description` | required, ≤ 200 chars: scenario driven + what is asserted; longer rationale in a YAML comment above |
+| `skip` | a reason; reports skipped |
+| `trigger` | `{ id }` only — the trigger every turn runs under (default `manual`); unknown id fails the case |
+| `variables` | `{ name: value }` seeded and **locked** every turn — the firing's payload; never seed what the agent must establish (assert it with `variables:` instead) |
+| `workspace` | `{ <run path>: content }` — strings verbatim, other YAML → JSON, `{ from: <tests-relative path> }` copies a file; only run-zone paths (`trigger.json`, `scratchpad/…`) |
+| `mocks` | `[{ tool, whenInput?, result }]` — intercepts agent and script calls alike (partial-input match) |
+| `steps` | one flat list; actions and assertions are peers; an assertion evaluates against the nearest action above it |
+
+Actions: `send: "<message>"` (a new turn; against a parked run, a reply-only
+message that leaves it parked), `decide: { to, comment? }`,
+`deliver: { trigger, variables? }`.
+
+Assertions (**structural** ones halt the case when they miss; later steps are
+reported `not-executed`): `succeeded: true`ˢ · `parked: true | decision | input | { channel?, state? }`ˢ ·
+`reachedState: <slug>`ˢ (a committed non-trigger trail step) ·
+`trail: { to?, kind?, reason?, count }`ˢ (`kind`: trigger/agent/human/on_error) ·
+`noTraversal: true`ˢ · `triggerArrival: <id>`ˢ ·
+`reply: { includes?, excludes? }` (token or list; `"/regex/flags"` strings are regexes; matched against the whole turn's assistant text) ·
+`calledTool` / `notCalledTool` / `blockedTool: { name, input? }` (partial input; `blockedTool` = governance refused it — the assertion that tests a guard) ·
+`ranWorkflow: { workflow, status?, count? }` · `usedNoTools: true` (whole conversation) ·
+`variables: { expect, path?, locked? }` · `grade: { closedQA, atLeast }` (LLM grades the reply plus the tool record; `atLeast` is a hard bar; needs `tests.judge`).
+
+Put the structural assertion right after the action that should satisfy it,
+before content assertions that only make sense in that state:
+
+```yaml

package/skills/archmax-harness/references/diagnostics.md

@@ -0,0 +1,51 @@
+# `archmax validate` diagnostics → fix
+
+Load this when `validate` or `test` reports something you cannot fix from the
+message alone. Migrations for removed keys in full:
+[`workflow-schema.md`](workflow-schema.md#removed-keys-you-may-meet-in-an-older-workflow).
+
+## Diagnostics
+
+`validate` output names file and field. The recurring ones:
+
+| Diagnostic says | Fix |
+| --- | --- |
+| unknown key `<k>` (anywhere) | Misspelling, or a removed key — see the table below; `metadata` is the only free slot |
+| `name:` / `description:` at root | Rename to `title:` / `instructions:` (reread the text as direction, delete narration) |
+| `policy:` or root `skills.allow:` | Move to `tools.forbid_always` / `skills.allow_always` + state lists |
+| Workflow slug / state key not kebab-case, or empty | Rename the directory or key; update every `to`, `on_error`, `reachedState` |
+| transitions to / on_error routes to undefined state | Fix the slug or add the state |
+| no start state / multiple states declare trigger | Exactly one state per trigger id declares it under `triggers:` |
+| hook must be a `{ script }` or `{ rubric }` entry / single-key tagged object | One kind per hook; the only sidecar is `max_iterations`; `specification:` and `subagent:` do not exist |
+| hook script must be inside `hooks/` / not found | `before: { script: hooks/<x>.js }`, file at `workflows/<slug>/hooks/<x>.js` |
+| `before` declares `max_iterations` | Remove it; before hooks are ok/veto |
+| Human state must declare instructions / transitions / description; self-edge; hook can never run; two transitions of one type | Add them; route out of the state; drop the hook; one target per labelled type |
+| `'*'` is not a tool name a grant may use | Move the wildcard to `forbid`/`forbid_always` |
+| allows `task` — not grantable | Delete the entry; declare a `rubric:` hook instead |
+| allows `archmax_run` on `skills/<x>/…` which the state does not enable | Add `x` to the state's `skills.allow` (or `allow_always`), or drop the entry |
+| `forbid_always` denies `archmax_advance` | Remove it |
+| not a skill slug (path/glob given) | Use the bundle directory name; narrow paths with a `tools.allow` entry |
+| `${{…}}` not a valid variable reference | `${{name}}` or `${{name.path}}`; snake_case names |
+| guards on `${{x}}` but no state requires it (warning) | Add `x` to an earlier state's `requires`, or seed it at assembly |
+| `allow` on a path in `scratchpad/` (warning) | Drop it, or govern a non-reserved run path instead |
+| `tools.allow_always` constrains essential tool (warning) | Narrow per state with `tools.allow` |
+| state both allows and forbids / forbids what nothing enables / enables what allow_always already does / `skills: { allow: [] }` reads like a deny (warnings) | Drop the redundant entry; use `forbid` to subtract |
+| no top-level `instructions` (warning) | Add the standing block |
+| state has several transitions but no `instructions` (warning) | Add routing guidance so the agent classifies the request before it advances |
+| `transitions.<i>.description` must be a non-empty string (error) | Add the condition under which that edge is taken — it is the only thing the agent knows about it. Say *when*, not *where* |
+| trigger declares unknown key (warning) | Keys read: `session`, `message`, `connection`, `requires`, `returns`; others are preserved |
+| `maxConcurrency` above 1 | Set to 1 or remove |
+| `WORKFLOW.md` has frontmatter | Strip it; the machine is `workflow.yaml` |
+| case: missing/over-long `title`/`description`; unknown key; assertion before action; `expect:` block; malformed regex; unknown trigger; `from:` missing | Fix the field named; assertions are steps; regexes are `"/p/f"` strings |
+| `agent cannot read workflows/**` (instructions/evidence/workspace path) | Move the text to a skill bundle or the path to `scratchpad/` |
+
+**Removed keys** you may meet (all load errors; details and migrations in
+`references/workflow-schema.md#removed-keys-you-may-meet-in-an-older-workflow`):
+`name:`→`title:`; `description:`→`instructions:`; `policy:`→`forbid_always`;
+root `skills.allow`→`skills.allow_always`; state `type: wait`→`archmax_wait`;
+`type: trigger`→`triggers:` on the entered state; `type: workflow`,
+`workflow:`, `params:`, `for_each:`, `result:`→an agent state allowing
+`archmax_workflow_<slug>` + `requires`; `parallel:`/`join:`→sequenced states;
+transition `on:`→nothing (the resumed state picks its edge); `subagent:` hook
+→`rubric:`; `max_corrections`→`max_iterations`; single-file `WORKFLOW.md` with
+frontmatter→`workflow.yaml` + prose.