@kirrosh/zond 0.26.1 → 0.27.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (261) hide show
  1. package/CHANGELOG.md +47 -0
  2. package/README.md +22 -21
  3. package/bin/zond.mjs +23 -0
  4. package/package.json +13 -16
  5. package/scripts/npm/postinstall.mjs +76 -0
  6. package/src/CLAUDE.md +0 -112
  7. package/src/bun-types.d.ts +0 -5
  8. package/src/cli/argv.ts +0 -122
  9. package/src/cli/commands/add-api.ts +0 -146
  10. package/src/cli/commands/api/annotate/idempotency.ts +0 -59
  11. package/src/cli/commands/api/annotate/index.ts +0 -880
  12. package/src/cli/commands/api/annotate/lifecycle.ts +0 -74
  13. package/src/cli/commands/api/annotate/overlay.ts +0 -206
  14. package/src/cli/commands/api/annotate/pagination.ts +0 -64
  15. package/src/cli/commands/api/annotate/prompts.ts +0 -220
  16. package/src/cli/commands/api/annotate/readback.ts +0 -58
  17. package/src/cli/commands/api/annotate/resources.ts +0 -91
  18. package/src/cli/commands/api/annotate/seed-bodies.ts +0 -61
  19. package/src/cli/commands/audit.ts +0 -786
  20. package/src/cli/commands/catalog.ts +0 -97
  21. package/src/cli/commands/check.ts +0 -361
  22. package/src/cli/commands/checks.ts +0 -1072
  23. package/src/cli/commands/ci-init.ts +0 -223
  24. package/src/cli/commands/clean.ts +0 -212
  25. package/src/cli/commands/cleanup.ts +0 -236
  26. package/src/cli/commands/completions.ts +0 -192
  27. package/src/cli/commands/coverage.ts +0 -872
  28. package/src/cli/commands/db.ts +0 -611
  29. package/src/cli/commands/describe.ts +0 -120
  30. package/src/cli/commands/discover.ts +0 -1356
  31. package/src/cli/commands/doctor.ts +0 -661
  32. package/src/cli/commands/fixtures.ts +0 -402
  33. package/src/cli/commands/generate.ts +0 -577
  34. package/src/cli/commands/init/agents-md.ts +0 -61
  35. package/src/cli/commands/init/bootstrap.ts +0 -111
  36. package/src/cli/commands/init/index.ts +0 -244
  37. package/src/cli/commands/init/skills.ts +0 -141
  38. package/src/cli/commands/init/templates/agents.md +0 -86
  39. package/src/cli/commands/init/templates/markdown.d.ts +0 -4
  40. package/src/cli/commands/init/templates/skills/warm-up-target.md +0 -122
  41. package/src/cli/commands/init/templates/skills/zond-checks.md +0 -621
  42. package/src/cli/commands/init/templates/skills/zond-seed.md +0 -114
  43. package/src/cli/commands/init/templates/skills/zond-triage.md +0 -272
  44. package/src/cli/commands/init/templates/skills/zond.md +0 -861
  45. package/src/cli/commands/init/templates/zond-config.yml +0 -14
  46. package/src/cli/commands/prepare-fixtures.ts +0 -97
  47. package/src/cli/commands/probe/_seed-bodies.ts +0 -52
  48. package/src/cli/commands/probe/mass-assignment.ts +0 -594
  49. package/src/cli/commands/probe/security.ts +0 -537
  50. package/src/cli/commands/probe/static.ts +0 -255
  51. package/src/cli/commands/probe/webhooks.ts +0 -163
  52. package/src/cli/commands/probe.ts +0 -535
  53. package/src/cli/commands/reference.ts +0 -87
  54. package/src/cli/commands/refresh-api.ts +0 -227
  55. package/src/cli/commands/remove-api.ts +0 -150
  56. package/src/cli/commands/report-bundle.ts +0 -310
  57. package/src/cli/commands/report.ts +0 -241
  58. package/src/cli/commands/request.ts +0 -548
  59. package/src/cli/commands/run.ts +0 -1162
  60. package/src/cli/commands/schema-from-runs.ts +0 -128
  61. package/src/cli/commands/secrets.ts +0 -133
  62. package/src/cli/commands/session.ts +0 -244
  63. package/src/cli/commands/use.ts +0 -74
  64. package/src/cli/index.ts +0 -55
  65. package/src/cli/json-envelope.ts +0 -108
  66. package/src/cli/json-schemas.ts +0 -314
  67. package/src/cli/output.ts +0 -40
  68. package/src/cli/program.ts +0 -219
  69. package/src/cli/resolve.ts +0 -105
  70. package/src/cli/runtime.ts +0 -7
  71. package/src/cli/safe-live.ts +0 -24
  72. package/src/cli/status-filter.ts +0 -114
  73. package/src/cli/util/api-context.ts +0 -85
  74. package/src/cli/version.ts +0 -8
  75. package/src/core/audit/persist.ts +0 -183
  76. package/src/core/checks/budget.ts +0 -59
  77. package/src/core/checks/checks/_crud-helpers.ts +0 -133
  78. package/src/core/checks/checks/_negative_mutator.ts +0 -133
  79. package/src/core/checks/checks/_readback-helpers.ts +0 -133
  80. package/src/core/checks/checks/content_type_conformance.ts +0 -39
  81. package/src/core/checks/checks/cross_call_references.ts +0 -147
  82. package/src/core/checks/checks/cursor_boundary_fuzzing.ts +0 -219
  83. package/src/core/checks/checks/ensure_resource_availability.ts +0 -62
  84. package/src/core/checks/checks/idempotency_replay.ts +0 -242
  85. package/src/core/checks/checks/ignored_auth.ts +0 -254
  86. package/src/core/checks/checks/index.ts +0 -68
  87. package/src/core/checks/checks/lifecycle_transitions.ts +0 -416
  88. package/src/core/checks/checks/missing_required_header.ts +0 -40
  89. package/src/core/checks/checks/negative_data_rejection.ts +0 -148
  90. package/src/core/checks/checks/not_a_server_error.ts +0 -35
  91. package/src/core/checks/checks/open_cors_on_sensitive.ts +0 -160
  92. package/src/core/checks/checks/pagination_invariants.ts +0 -419
  93. package/src/core/checks/checks/positive_data_acceptance.ts +0 -33
  94. package/src/core/checks/checks/rate_limit_headers_absent.ts +0 -77
  95. package/src/core/checks/checks/response_headers_conformance.ts +0 -74
  96. package/src/core/checks/checks/response_schema_conformance.ts +0 -30
  97. package/src/core/checks/checks/status_code_conformance.ts +0 -132
  98. package/src/core/checks/checks/unsupported_method.ts +0 -63
  99. package/src/core/checks/checks/use_after_free.ts +0 -78
  100. package/src/core/checks/index.ts +0 -30
  101. package/src/core/checks/mode.ts +0 -82
  102. package/src/core/checks/recommended-action.ts +0 -68
  103. package/src/core/checks/registry.ts +0 -78
  104. package/src/core/checks/runner.ts +0 -1461
  105. package/src/core/checks/sarif.ts +0 -230
  106. package/src/core/checks/spec-findings.ts +0 -308
  107. package/src/core/checks/stateful.ts +0 -121
  108. package/src/core/checks/types.ts +0 -305
  109. package/src/core/checks/zond-extensions.ts +0 -73
  110. package/src/core/classifier/recommended-action.ts +0 -251
  111. package/src/core/context/current.ts +0 -51
  112. package/src/core/context/session.ts +0 -78
  113. package/src/core/coverage/loader.ts +0 -216
  114. package/src/core/coverage/reasons.ts +0 -300
  115. package/src/core/diagnostics/db-analysis.ts +0 -666
  116. package/src/core/diagnostics/failure-class.ts +0 -140
  117. package/src/core/diagnostics/failure-hints.ts +0 -112
  118. package/src/core/diagnostics/spec-pointer.ts +0 -99
  119. package/src/core/diagnostics/suggested-fixes.ts +0 -155
  120. package/src/core/exporter/case-study/index.ts +0 -270
  121. package/src/core/exporter/curl.ts +0 -40
  122. package/src/core/exporter/exporter.ts +0 -48
  123. package/src/core/exporter/html-report/escape.ts +0 -24
  124. package/src/core/exporter/html-report/index.ts +0 -479
  125. package/src/core/exporter/html-report/script.ts +0 -100
  126. package/src/core/exporter/html-report/styles.ts +0 -408
  127. package/src/core/generator/catalog-builder.ts +0 -179
  128. package/src/core/generator/chunker.ts +0 -78
  129. package/src/core/generator/coverage-phase.ts +0 -0
  130. package/src/core/generator/coverage-scanner.ts +0 -87
  131. package/src/core/generator/data-factory.ts +0 -759
  132. package/src/core/generator/describe.ts +0 -302
  133. package/src/core/generator/endpoint-warnings.ts +0 -52
  134. package/src/core/generator/fixtures-builder.ts +0 -332
  135. package/src/core/generator/index.ts +0 -14
  136. package/src/core/generator/openapi-reader.ts +0 -297
  137. package/src/core/generator/path-param-disambig.ts +0 -140
  138. package/src/core/generator/resources-builder.ts +0 -898
  139. package/src/core/generator/schema-utils.ts +0 -112
  140. package/src/core/generator/serializer.ts +0 -343
  141. package/src/core/generator/suite-generator.ts +0 -1301
  142. package/src/core/generator/types.ts +0 -63
  143. package/src/core/identity/identity-file.ts +0 -0
  144. package/src/core/lint/affects.ts +0 -28
  145. package/src/core/lint/config.ts +0 -96
  146. package/src/core/lint/format.ts +0 -42
  147. package/src/core/lint/index.ts +0 -94
  148. package/src/core/lint/reporter.ts +0 -128
  149. package/src/core/lint/rules/consistency.ts +0 -158
  150. package/src/core/lint/rules/heuristics.ts +0 -97
  151. package/src/core/lint/rules/strictness.ts +0 -109
  152. package/src/core/lint/types.ts +0 -96
  153. package/src/core/lint/walker.ts +0 -248
  154. package/src/core/meta/meta-store.ts +0 -11
  155. package/src/core/output/README.md +0 -73
  156. package/src/core/output/index.ts +0 -13
  157. package/src/core/output/run.ts +0 -91
  158. package/src/core/output/types.ts +0 -122
  159. package/src/core/parser/dynamic-values.ts +0 -160
  160. package/src/core/parser/env-interpolation.ts +0 -104
  161. package/src/core/parser/filter.ts +0 -97
  162. package/src/core/parser/schema.ts +0 -359
  163. package/src/core/parser/types.ts +0 -117
  164. package/src/core/parser/variables.ts +0 -0
  165. package/src/core/parser/yaml-parser.ts +0 -181
  166. package/src/core/probe/bootstrap.ts +0 -34
  167. package/src/core/probe/dry-run-envelope.ts +0 -61
  168. package/src/core/probe/mass-assignment/classify.ts +0 -175
  169. package/src/core/probe/mass-assignment/cleanup.ts +0 -52
  170. package/src/core/probe/mass-assignment/digest.ts +0 -114
  171. package/src/core/probe/mass-assignment/orchestrator.ts +0 -459
  172. package/src/core/probe/mass-assignment/regression.ts +0 -141
  173. package/src/core/probe/mass-assignment/suspects.ts +0 -92
  174. package/src/core/probe/mass-assignment/types.ts +0 -135
  175. package/src/core/probe/mass-assignment-probe-class.ts +0 -198
  176. package/src/core/probe/mass-assignment-probe.ts +0 -27
  177. package/src/core/probe/mass-assignment-template.ts +0 -240
  178. package/src/core/probe/method-probe.ts +0 -164
  179. package/src/core/probe/method-shared.ts +0 -69
  180. package/src/core/probe/negative-probe.ts +0 -691
  181. package/src/core/probe/orphan-tracker.ts +0 -188
  182. package/src/core/probe/path-discovery.ts +0 -439
  183. package/src/core/probe/probe-harness.ts +0 -119
  184. package/src/core/probe/registry.ts +0 -89
  185. package/src/core/probe/runner.ts +0 -136
  186. package/src/core/probe/security/baseline.ts +0 -174
  187. package/src/core/probe/security/classify.ts +0 -341
  188. package/src/core/probe/security/cleanup.ts +0 -125
  189. package/src/core/probe/security/detectors.ts +0 -71
  190. package/src/core/probe/security/digest.ts +0 -104
  191. package/src/core/probe/security/orchestrator.ts +0 -398
  192. package/src/core/probe/security/regression.ts +0 -103
  193. package/src/core/probe/security/types.ts +0 -151
  194. package/src/core/probe/security-probe-class.ts +0 -207
  195. package/src/core/probe/security-probe.ts +0 -32
  196. package/src/core/probe/shared.ts +0 -531
  197. package/src/core/probe/static-probe-class.ts +0 -125
  198. package/src/core/probe/types.ts +0 -165
  199. package/src/core/probe/verdict-aggregator.ts +0 -33
  200. package/src/core/probe/webhooks-probe.ts +0 -282
  201. package/src/core/reporter/console.ts +0 -240
  202. package/src/core/reporter/index.ts +0 -22
  203. package/src/core/reporter/json.ts +0 -22
  204. package/src/core/reporter/junit.ts +0 -93
  205. package/src/core/reporter/ndjson.ts +0 -37
  206. package/src/core/reporter/types.ts +0 -15
  207. package/src/core/runner/assertions.ts +0 -383
  208. package/src/core/runner/async-pool.ts +0 -108
  209. package/src/core/runner/auth-path.ts +0 -8
  210. package/src/core/runner/ci-context.ts +0 -72
  211. package/src/core/runner/executor.ts +0 -652
  212. package/src/core/runner/expr-eval.ts +0 -41
  213. package/src/core/runner/form-encode.ts +0 -41
  214. package/src/core/runner/http-client.ts +0 -224
  215. package/src/core/runner/learn-drift.ts +0 -293
  216. package/src/core/runner/preflight-vars.ts +0 -153
  217. package/src/core/runner/progress-tracker.ts +0 -73
  218. package/src/core/runner/rate-limiter.ts +0 -185
  219. package/src/core/runner/run-kind.ts +0 -45
  220. package/src/core/runner/schema-validator.ts +0 -308
  221. package/src/core/runner/send-request.ts +0 -232
  222. package/src/core/runner/transforms.ts +0 -65
  223. package/src/core/runner/types.ts +0 -94
  224. package/src/core/secrets/registry.ts +0 -164
  225. package/src/core/secrets/secrets-file.ts +0 -115
  226. package/src/core/selectors/operation-filter.ts +0 -144
  227. package/src/core/setup-api.ts +0 -590
  228. package/src/core/severity/category.ts +0 -94
  229. package/src/core/severity/index.ts +0 -58
  230. package/src/core/spec/infer-schema.ts +0 -102
  231. package/src/core/spec/layers.ts +0 -154
  232. package/src/core/spec/merge-specs.ts +0 -156
  233. package/src/core/spec/schema-from-runs.ts +0 -117
  234. package/src/core/spec/schema-overlay.ts +0 -130
  235. package/src/core/util/ajv.ts +0 -13
  236. package/src/core/util/format-eta.ts +0 -21
  237. package/src/core/util/headers.ts +0 -9
  238. package/src/core/util/url.ts +0 -24
  239. package/src/core/utils.ts +0 -13
  240. package/src/core/workspace/config.ts +0 -129
  241. package/src/core/workspace/fixture-gap-report.ts +0 -84
  242. package/src/core/workspace/fixture-gaps.ts +0 -71
  243. package/src/core/workspace/manifest.ts +0 -283
  244. package/src/core/workspace/output-rotation.ts +0 -62
  245. package/src/core/workspace/root.ts +0 -96
  246. package/src/core/workspace/triage-path.ts +0 -87
  247. package/src/db/lint-runs.ts +0 -47
  248. package/src/db/migrate.ts +0 -128
  249. package/src/db/migrations/0001_run_kind.sql +0 -25
  250. package/src/db/migrations/0002_run_kind_request.sql +0 -59
  251. package/src/db/migrations/sql.d.ts +0 -4
  252. package/src/db/queries/collections.ts +0 -133
  253. package/src/db/queries/coverage.ts +0 -9
  254. package/src/db/queries/dashboard.ts +0 -59
  255. package/src/db/queries/results.ts +0 -216
  256. package/src/db/queries/runs.ts +0 -289
  257. package/src/db/queries/sessions.ts +0 -42
  258. package/src/db/queries/settings.ts +0 -28
  259. package/src/db/queries/types.ts +0 -172
  260. package/src/db/queries.ts +0 -75
  261. package/src/db/schema.ts +0 -298
@@ -1,861 +0,0 @@
1
- ---
2
- name: zond
3
- description: |
4
- Primary skill for any zond work. Loads on workspace touch
5
- (`apis/<name>/`, `.env.yaml`, `.api-fixtures.yaml`) AND on intent —
6
- full API audit, "find bugs", "raise coverage", "test the whole API",
7
- contract-drift check, schema-drift detection, post-deploy regression,
8
- case study, single-flow scenario authoring, fixture pack. Hand off to
9
- `zond-checks` for depth-check reference / SARIF / stateful invariants,
10
- to `zond-triage` for "what broke in the last run".
11
- allowed-tools: [Read, Write, Edit, Bash(zond *), Bash(bunx zond *), Bash(sqlite3 *)]
12
- ---
13
-
14
- # zond — Primary skill
15
-
16
- CLI-only. Run `zond --version` first; if missing:
17
- `curl -fsSL https://raw.githubusercontent.com/kirrosh/zond/master/install.sh | sh`.
18
-
19
- For depth-check reference (the catalog of conformance / security /
20
- stateful checks, SARIF output, m-20 invariants) — see `zond-checks`.
21
- For triage of a failing run — see `zond-triage`.
22
-
23
- ## Iron rules (read once, apply always)
24
-
25
- - **NEVER read raw OpenAPI** with Read/cat/grep. The workspace has
26
- pre-built artifacts (catalog/resources/fixtures) — use those. Drop
27
- into `apis/<name>/spec.json` only when probe-class authoring needs
28
- full schemas.
29
- - **NEVER `curl` or `wget`** — use `zond request <method> <url> --api <name>`
30
- for ad-hoc HTTP so it lands in the run DB and respects auth. Never
31
- shell-substitute the token by hand (`$(yq …)` is blocked by the sandbox).
32
- `--body '{…}'` always sends `application/json`. For form-encoded APIs
33
- (Stripe v1, classic webforms) pass `--form` — zond URL-encodes the
34
- JSON body. Auto-detect from spec: if `requestBody.content` in
35
- `.api-catalog.yaml` declares only `application/x-www-form-urlencoded`,
36
- use `--form`; mixed content-types → pick by `Content-Type` header you
37
- want to test.
38
- - **NEVER hardcode tokens** — put them in `apis/<name>/.secrets.yaml`
39
- (auto-gitignored), reference from `.env.yaml` as `@secret:auth_token`.
40
- Plain `${SHELL_VAR}` references also work. Tests read the resolved
41
- value as `{{auth_token}}`.
42
- - **NEVER read `.secrets.yaml` directly.** Use `zond doctor --api <name> --json`
43
- — reports `set | unset` and value length only. To WRITE / rotate a secret
44
- use `zond secrets set <key> <value> --api <name>` (writes a `.bak`, never
45
- echoes the value, auto-strips a pasted `Bearer ` prefix — pass `--literal`
46
- to keep it verbatim).
47
- - **NEVER edit `.api-*.yaml`** (catalog / resources / fixtures) by hand —
48
- regenerated by `zond refresh-api`. `.api-resources.local.yaml` IS the
49
- hand-editable / annotate-target overlay; safe to edit.
50
- `.api-schema.local.yaml` is the response-schema overlay (ARV-176) —
51
- written by `refresh-api --merge-schema`, also survives refresh; edit by
52
- re-merging a `patch.schema.json`, not by hand. `.env.yaml` values are
53
- editable (see ARV-114).
54
- - **NEVER invent endpoints.** Only use entries from `.api-catalog.yaml`.
55
- - **NEVER run destructive ops on a shared / production org without
56
- `--dry-run` first.** Probes, `prepare-fixtures --apply`, `cleanup` hit
57
- live APIs and can delete user data. `--dry-run` once → inspect → drop
58
- the flag.
59
- - **NEVER share triage artefacts** outbound without `--redact-identity`.
60
- Secret-redaction is on by default; identity values (org/member slugs,
61
- real ids) need the extra flag.
62
- - **NEVER report cleanup failure as an API bug.** POST 200 → DELETE 5xx
63
- is *probably* a fixture-isolation issue (orphan accumulation, race).
64
- Re-run with `--no-cleanup` or isolated namespace before filing.
65
- - **`recommended_action: report_backend_bug` / 5xx → STOP** in interactive
66
- mode (surface the request/response, get a decision; never `expect:`-mask).
67
- In autonomous / loop / audit-sweep mode (no user-in-the-loop), log to
68
- `api-bugs-<NN>.md`, continue the sweep — bailing on bug #1 forfeits #2..N.
69
- - **CRUD-run ≥80% 401/403 / `permission_denied` → `env_issue`, not bug.**
70
- Token scope issue. Confirm via `zond db diagnose <run-id> --json`
71
- (401/403 dominate `by_recommended_action`); do not generate
72
- case-studies, do not `expect:`-mask.
73
- - **MUST run `zond doctor --api <name> --missing-only` before generating
74
- fixtures or touching `.env.yaml`** — identifies unfilled keys early.
75
- - **`prepare-fixtures` is single-pass, never POST-creates, and never
76
- harvests values** (ARV-362) — it verifies existing fixtures and reports
77
- unfilled vars as gaps. Which record/field fills a slot is your call:
78
- fill fixtures by hand (`fixtures add` / editing `.env.yaml`).
79
-
80
- ## Workspace artifact model
81
-
82
- After `zond add api <name> --spec <path|url>`, `apis/<name>/` contains:
83
-
84
- ```
85
- spec.json ← dereferenced OpenAPI (machine source)
86
- .api-catalog.yaml ← endpoint index (method, path, params)
87
- .api-resources.yaml ← CRUD chains + FK deps (auto)
88
- .api-resources.local.yaml ← overlay: extensions + annotate output (hand-edit OK; appears after first `zond api annotate apply` — or create by hand to use as overlay)
89
- .api-fixtures.yaml ← MANIFEST: required vars + descriptions (read-only)
90
- .env.yaml ← VALUES: variable values (user / prepare-fixtures writes)
91
- .secrets.yaml ← raw secret values (auto-gitignored)
92
- .identity.yaml ← non-secret identifying values (gitignored)
93
- tests/ ← generated/handwritten suites
94
- scenarios/ ← handwritten flow suites
95
- probes/ ← probe-emitted suites
96
- ```
97
-
98
- ### Manifest vs values (the #1 source of confusion)
99
-
100
- **`.api-fixtures.yaml` = the list of vars this API needs (manifest).
101
- `.env.yaml` = their values.**
102
-
103
- | Op | Touches manifest? | Touches env? |
104
- |---|---|---|
105
- | `zond add api` / `refresh-api` | rebuilds | seeds skeleton |
106
- | `zond generate` | extends (adds new `{{vars}}` discovered) | does **not** modify |
107
- | `zond prepare-fixtures` | reads | writes values |
108
- | `zond fixtures add` / `import` | reads | writes values (manual) |
109
- | user editing | never | always |
110
-
111
- - `{{var}}` in a generated test but NOT in `.api-fixtures.yaml` is a
112
- bug in the manifest builder / generator. Don't "fix" it by adding to
113
- `.env.yaml`.
114
- - A key in `.env.yaml` but NOT in `.api-fixtures.yaml` is shadow —
115
- `prepare-fixtures` warns `not in manifest, ignored`.
116
- - "Generate should sync `.env.yaml`" is a rejected design (decision-7,
117
- m-17).
118
-
119
- ### Manual fixture-bootstrap (you fill every value)
120
-
121
- ARV-362: `prepare-fixtures` **reports** FK gaps — it never harvests a
122
- value. Every fixture is filled by hand: read the gap (empty list → create
123
- a record first; non-empty → pick a value; 404/403 → auth/drift), then
124
- supply it. Vendor-dashboard ids (Stripe `cus_*`, GitHub PR numbers,
125
- Sentry issue ids) have no list endpoint at all — same hand-off:
126
-
127
- | Input you have | Command |
128
- |---|---|
129
- | Concrete id values | `zond fixtures add <var>=<id> [--validate] --apply` |
130
- | Curl from devtools / dashboard | `pbpaste \| zond fixtures import --from-curl --apply` (URL is matched against spec paths to extract `{var}` bindings) |
131
-
132
- `--validate` GETs the spec's read-by-id endpoint and classifies the
133
- fixture as `live` / `stale` / `unknown` so dead ids don't silently
134
- break later runs (ARV-32). Both commands write to `apis/<name>/.env.yaml`
135
- with a `.env.yaml.bak` backup.
136
-
137
- ### When to read which file
138
-
139
- | To know… | Read |
140
- |---|---|
141
- | What endpoints exist | `.api-catalog.yaml` |
142
- | How resources chain (CRUD, FK, ETag) | `.api-resources.yaml` |
143
- | What vars are needed and why | `.api-fixtures.yaml` |
144
- | What var values are set | `.env.yaml` |
145
- | Annotation overlay (seed_body, lifecycle, pagination, readback_diff) | `.api-resources.local.yaml` |
146
- | Full request/response schema (only when needed) | `spec.json` |
147
-
148
- ## --json envelope (uniform across commands)
149
-
150
- ```jsonc
151
- {
152
- "ok": true|false,
153
- "command": "<name>",
154
- "data": { /* command-specific */ },
155
- "warnings": ["..."],
156
- "errors": [{ "code": "ZondErrorCode", "message": "...", "details": {} }],
157
- "exit_code": 0|1|2
158
- }
159
- ```
160
-
161
- Route on `errors[].code`, not message. Stdout discipline: `--json`
162
- emits only the envelope on stdout; progress/hints to stderr.
163
-
164
- `--json` ≠ `--report json`. `--json` wraps the **command result**;
165
- `--report json` wraps the **test-run artefact**. `zond run` doesn't
166
- accept `--json` — use `--report json`.
167
-
168
- ## Entry points — pick by intent
169
-
170
- | User asked… | Start at |
171
- |---|---|
172
- | "smoke this API", "quick first pass", "demo / CI gate" | `zond audit --api <name>` (safe-mode breadth pass — no live mutations) |
173
- | "audit this API", "test the whole API", "raise coverage", "deep pass" | walk Phase 0–9 — audit covers smoke; depth (stateful, security, learn-apply) needs phase-level decisions |
174
- | "full live audit against sandbox", "include mass-assignment/security probes" | `zond audit --api <name> --live --with-mass-assignment --with-security` (REQUIRES throwaway/sandbox account) |
175
- | "find bugs", "test for 5xx", "probe sweep" | Phase 0 → Phase 7 (Probes) |
176
- | "security only / SSRF / CRLF" | Phase 7.2 directly with `--dry-run` first |
177
- | "deep depth-checks", "SARIF", "stateful invariants" | Hand off → `zond-checks` |
178
- | "what broke in last run", "diagnose run X" | Hand off → `zond-triage` |
179
- | "write a test for X flow", "smoke this checkout" | Phase S (Scenarios) |
180
- | "what vars does this API need" | `zond doctor --api <name> --json` |
181
- | "share results", "case study" | Phase 9 (Share) |
182
- | "workspace looks messy" | `zond clean --api <name>` (dry-run → `--force`) |
183
-
184
- ## Phase 0 — Orient
185
-
186
- ```bash
187
- zond doctor --api <name> --json # gaps + freshness
188
- ```
189
-
190
- Then read the three artifacts (NOT the raw spec):
191
-
192
- ```bash
193
- cat apis/<name>/.api-catalog.yaml | head -80
194
- cat apis/<name>/.api-resources.yaml
195
- cat apis/<name>/.api-fixtures.yaml
196
- ```
197
-
198
- If doctor reports stale → `zond refresh-api <name>`.
199
-
200
- ## Phase 1 — Fixture pack
201
-
202
- `zond prepare-fixtures` is **single-pass and deterministic**: it walks
203
- `.api-resources.yaml`, hits each owner list/read endpoint, and **reports**
204
- which FK vars are unfilled (empty list → create one first; non-empty →
205
- pick a value; 404/403 → auth/drift). It does **not** POST-create resources
206
- and (ARV-362) **never harvests a value** — which record/field fills a slot
207
- is your judgment. Fill every gap by hand (`fixtures add` / `.env.yaml`).
208
-
209
- ```bash
210
- zond prepare-fixtures --api <name> # report gaps (never writes values)
211
- zond prepare-fixtures --api <name> --apply # only unsets stale ids under --refresh; never harvests
212
- zond prepare-fixtures --api <name> --refresh # = --verify --apply: drop (unset) stale 404 ids → you refill
213
- ```
214
-
215
- It also scans the generated suites and reports two named gap buckets
216
- (warnings in text, `summary.fixtureGaps` in `--json`) — report only, no
217
- values invented, no auto-seed (ARV-349/350):
218
- - **`unseededRoots`** — required manifest vars that are empty, referenced by
219
- a suite, and seeded by no step (e.g. `{{account}}` created in `crud-accounts`
220
- but referenced uncaptured in `persons-crud`). These are chain-heads: one id
221
- un-skips a whole dependent CRUD suite. Fix these first.
222
- - **`undefinedVars`** — suite `{{vars}}` no manifest entry, capture, or param
223
- produces (`{{bank_code}}`, `{{tax_id}}`). Supply by hand.
224
-
225
- To fill `unseededRoots` autonomously — author create-bodies, POST them, and
226
- capture the returned ids — hand off to **`zond-seed`** (agent reasons, zond
227
- executes via `request … --capture`). It seeds only what the API self-serves
228
- and reports external-input gaps honestly.
229
-
230
- When a var stays UNSET after `--apply`, read the per-target reason in the
231
- output, then fill the gap directly:
232
-
233
- | Input you have | Command |
234
- |---|---|
235
- | Concrete id values | `zond fixtures add <var>=<id> [--validate] --apply` |
236
- | Curl from devtools / dashboard | `pbpaste \| zond fixtures import --from-curl --apply` |
237
- | Nothing yet (empty list) | Create the resource in the product UI / via API, then harvest its id with `fixtures add` |
238
-
239
- Common reasons a var stays UNSET (all fixed outside prepare-fixtures):
240
- - empty list on the target account — no record exists to harvest;
241
- - the id lives only in a vendor dashboard (Stripe `cus_*`, GitHub PR
242
- numbers, Sentry issue ids) with no discoverable list endpoint;
243
- - the value is gated outside the API path (email verify / paid plan /
244
- SCIM / TOS limits). Document the gap; don't loop.
245
-
246
- Editing `.env.yaml` by hand is the sanctioned fallback when the CLI
247
- cannot harvest a value (write-only ingest endpoints, SDK-only ids,
248
- slugs that live in user's project config). Touch values only — never
249
- add keys not in `.api-fixtures.yaml`.
250
-
251
- ### Dynamic value functions in `.env.yaml` (ARV-190, m-21)
252
-
253
- Stale hardcoded UUIDs / dates / idempotency keys pin existing rows or
254
- expire mid-run. Use `#(funcName)` / `#(funcName(args))` references —
255
- evaluated at load time, stable within one run:
256
-
257
- | Token | Resolves to | Use for |
258
- |---|---|---|
259
- | `#(uuid)` | fresh UUID v4 (same value across all references in this run) | Idempotency-Key, request id |
260
- | `#(uuidStable(<seed>))` | deterministic UUID from seed (sha-256 shaped as v4) | reproducible tests |
261
- | `#(today)` | `YYYY-MM-DD` (UTC) | expiry-from dates |
262
- | `#(todayPlus(N))` | today + N days (N may be negative) | expires_at, valid_until |
263
- | `#(now)` | ISO 8601 timestamp | created_at hints |
264
- | `#(unix)` | seconds since epoch | epoch timestamps |
265
- | `#(alphanumeric(N))` | N random `[a-z0-9]` chars (default 8) | one-shot tokens |
266
- | `#(env:VAR)` | `process.env.VAR` (alias for `${VAR}`) | secrets passed through |
267
-
268
- ```yaml
269
- # apis/foo/.env.yaml
270
- idempotency_key: "#(uuid)" # same uuid across every step in this run
271
- trial_expires: "#(todayPlus(30))" # 30 days from now
272
- request_id: "req-#(uuid)" # embedded; shares uuid with above
273
- ```
274
-
275
- Resolution order: `${ENV}` → `#(...)` → `@secret:` → `@identity:`. Dynamic
276
- functions run BEFORE secret/identity refs so a secret value that happens to
277
- contain `#(...)` stays opaque (no double-expansion). Per-run cache means
278
- `#(uuid)` referenced twice in the same `.env.yaml` returns the same value
279
- — critical for idempotency-replay flows.
280
-
281
- ## Phase 2 — Annotate (NEW, m-20 / ARV-187)
282
-
283
- **Critical**: zond itself does NOT call any LLM. The flow is
284
- **dump → agent fills YAML → apply**. The agent (you) generates the
285
- overlay; zond validates and writes it into `.api-resources.local.yaml`.
286
-
287
- `zond api annotate` has 6 dump-subcommands (one per aspect). Pick what
288
- your downstream check needs:
289
-
290
- | Aspect | Dump produces | Feeds into |
291
- |---|---|---|
292
- | `--seed-bodies` | request-body schema per resource | all stateful checks (create-body overlay) |
293
- | `--readback` | create+read pair (write-shape vs read-shape) | `cross_call_references` ignore_fields, write_to_read_map |
294
- | `--idempotency` | header candidates | `idempotency_replay` (header, scope, body_hash_fields) |
295
- | `--pagination` | list-endpoint cursor/page detection | `pagination_invariants` (type, cursor_param) |
296
- | `--lifecycle` | action-endpoint candidates per resource | `lifecycle_transitions` (states, transitions) |
297
- | `--resources` | orphan endpoints not in resource graph | `.api-resources.local.yaml` extensions |
298
-
299
- **Workflow** (for ANY aspect):
300
-
301
- ```bash
302
- # 1. Dump — emits JSON slices on stdout. Restrict scope with --only.
303
- zond api annotate dump --api <name> --seed-bodies --only r1,r2 > /tmp/dump.json
304
-
305
- # 2. Agent reads the dump and writes a YAML response file (see format below).
306
-
307
- # 3. Apply — dry-run first (diff), then --yes to write.
308
- zond api annotate apply --api <name> --seed-bodies --input /tmp/out.yaml # dry-run + diff
309
- zond api annotate apply --api <name> --seed-bodies --input /tmp/out.yaml --yes # write
310
- ```
311
-
312
- YAML response format — a top-level list of entries, each one resource:
313
-
314
- ```yaml
315
- - resource: <name>
316
- seed_body: # (--seed-bodies)
317
- content_type: application/x-www-form-urlencoded
318
- body:
319
- amount: 1000
320
- currency: usd
321
- rationale: 'why this shape' # optional, helps future review
322
- confidence: high # high|medium|low — optional
323
- ```
324
-
325
- Same shape for `readback_diff`, `idempotency`, `pagination`,
326
- `lifecycle` blocks (see `zond-checks` for the per-aspect schemas).
327
-
328
- **When to annotate**: run dump+apply BEFORE Phase 6 (stateful checks)
329
- for any production API. Without annotation, stateful checks fall back
330
- to defaults and miss API-specific quirks (custom pagination params,
331
- non-standard lifecycle field names, write-only fields in create body).
332
-
333
- **For Stripe-style form-encoded APIs**: `--seed-bodies` is the single
334
- biggest win — the create-body overlay it produces lets stateful checks
335
- POST valid resources instead of 400-ing on schema-derived random bodies.
336
-
337
- ### Agent-in-the-loop annotate (dump → write → apply, ARV-187 / 277 / 278-282)
338
-
339
- zond does NOT infer annotations. YOU (the agent) read the spec slice and
340
- write the overlay; zond only dumps raw material and validates/merges what
341
- you produce.
342
-
343
- ```bash
344
- # 1. Dump the raw material for a resource — spec slice + expected YAML
345
- # shape + (for seed-bodies) the last N seed-POST attempts so you see
346
- # the error progression, not just the current 400 (ARV-277/278).
347
- zond api annotate dump --api <name> --seed-bodies --only <res> \
348
- --with-last-attempt --history 5
349
-
350
- # 2. YOU write the overlay: read the dumped slice + attempt history,
351
- # reason about the required create body / pagination / lifecycle /
352
- # idempotency, and produce one YAML doc per resource. Edit
353
- # .api-resources.local.yaml directly OR emit a file for apply.
354
-
355
- # 3. Apply — zond validates your YAML (zod), renders a diff, and (with
356
- # --yes) merges. With --gap-fill-only (ARV-281) your response can ONLY
357
- # add missing fields — already-set blocks are protected from overwrite.
358
- zond api annotate apply --api <name> --seed-bodies --input agent-out.yaml --gap-fill-only --yes
359
-
360
- # 4. Refresh fixtures — drop (unset) stale (404) ids so they resurface as
361
- # gaps; refill them by hand / `fixtures add` (discover never harvests).
362
- zond prepare-fixtures --api <name> --refresh
363
- ```
364
-
365
- ## Phase 3 — Generate tests
366
-
367
- ```bash
368
- zond generate apis/<name>/spec.json --output apis/<name>/tests [--tag <spec-tag>] [--uncovered-only]
369
- zond check tests apis/<name>/tests
370
- ```
371
-
372
- `generate` fills bodies with typed generators (`{{$randomString}}` etc.).
373
- Format-strict APIs reject many — that's a **test-fix**, not a backend bug.
374
- Pair with Phase 1 fixtures + Phase 2 annotation.
375
-
376
- Required FK / reference body fields (`*_id`, unresolved `*Code` like
377
- `sequenceTypeCode`) are wired to `{{fixture}}` refs (ARV-45) — canonicalised
378
- (`sequence_type_code`) and listed in `.api-fixtures.yaml` as `source: body-fk`.
379
- Fill their real values once in `.env.yaml`; no more hand-editing generated
380
- tests. Blank fixture → the var is unbound and dependent tests skip.
381
-
382
- If a CRUD chain you expected is missing, run
383
- `zond generate <spec> --explain` — diagnostic table shows every POST
384
- endpoint with verdict (no GET-by-id, non-`{id}` item param,
385
- trailing-slash mismatch, …).
386
-
387
- ## Phase 4 — Spec-lint (hygiene, separate workflow)
388
-
389
- ```bash
390
- zond lint --api <name> # 0 network requests
391
- zond lint --api <name> --json | jq '.data.stats'
392
- # Equivalent: `zond check spec --api <name>`
393
- ```
394
-
395
- ARV-255 (m-21 pivot): **spec-lint is hygiene, not security or
396
- contract.** It runs separately from `zond audit` / `zond probe` /
397
- `zond checks run` — those produce the security/reliability/contract
398
- report, lint stays out of that pile.
399
-
400
- Severity capped at **LOW / INFO** by design:
401
- - LOW: real spec violations (format mismatch in example, missing
402
- path-param format, response without schema).
403
- - INFO: style and documentation gaps (additionalProperties missing,
404
- naming, missing examples).
405
-
406
- No HIGH/MEDIUM ever — static analysis has no runtime evidence, so the
407
- severity matrix forbids escalation. `--strict` opts back into a
408
- non-zero exit when any LOW lands.
409
-
410
- ## Phase 5 — Run (sanity → smoke → CRUD)
411
-
412
- Wrap multi-run sweeps in a session so `/runs` shows one row:
413
-
414
- ```bash
415
- zond session start --label "smoke + probes"
416
- zond run apis/<name>/tests --tag sanity --report json # 5.1 sanity
417
- zond run apis/<name>/tests --safe --report json # 5.2 smoke (GET-only)
418
- zond run apis/<name>/tests --tag crud,setup --validate-schema --report json # 5.3 full CRUD
419
- zond run apis/<name>/tests --tag positive --include 'path:^/emails' # 5.4 narrow
420
- zond session end
421
- ```
422
-
423
- **Always pass `--validate-schema` for CRUD** — contract drift is
424
- invisible without it. `schema_violation` failures are real backend bugs;
425
- treat them like 5xx.
426
-
427
- **After any `zond run` with failures → delegate to `zond-triage`.** Do
428
- NOT parse `runs/run-NN.json` with `jq` by hand. The triage skill reads
429
- `zond db diagnose --json` (which already buckets by `recommended_action`
430
- in `data.by_recommended_action`, ARV-228) and routes by closed enum —
431
- re-implementing that logic with prose heuristics drifts. Only fall back
432
- to manual CLI queries when the failure is outside the example slice.
433
-
434
- Rate limit: `zond run` defaults to an adaptive limiter (no-op until
435
- `RateLimit-*` headers appear). Pass `--rate-limit <N>` for a hard cap;
436
- `--sequential` for old binaries.
437
-
438
- **Long runs**: `zond run` shows a periodic progress line on stderr when
439
- stdout/stderr is a TTY (`zond: [42s] 234/10927 steps (2%), 230 req, ~30
440
- req/s, ETA ~5m`) — overwrites itself in place. Suppressed by `--quiet`
441
- and on non-interactive output. For huge probe-suites (e.g. GitHub spec
442
- → ~10k probes under `--rate-limit 30` = 6+ min), this signals "alive,
443
- not hung". `zond probe static` itself prints a scale-warning block with
444
- ETA-at-rate-limit estimates and a `--max-per-endpoint K` sampling hint
445
- when `totalProbes ≥ 2000`.
446
-
447
- **Hard cap on requests**: pass `--max-requests N` to cap outgoing HTTP
448
- calls across the whole run. Remaining steps short-circuit to `skip` with
449
- `error: "max-requests-cap-reached"`. Each `retry_until` attempt counts as
450
- one request. Useful for sampling huge probe runs (`--max-requests 500`)
451
- and for CI time-boxing.
452
-
453
- `--all`: fold every `apis/<name>/tests/` dir into one `runs.id` per
454
- invocation (CI shape). CI context (commit_sha, branch, trigger=ci)
455
- auto-stamped from env vars.
456
-
457
- ### Spec-drift learning tail (`--learn`)
458
-
459
- After the initial CRUD run, **always** close with `--learn` /
460
- `--learn-apply`. R09 measured a 28% → 58% pass-coverage jump from this
461
- phase alone.
462
-
463
- ```bash
464
- zond run apis/<name>/tests --learn # detect (read-only)
465
- zond run apis/<name>/tests --learn-apply --learn-target test # rewrite expect.status
466
- zond run apis/<name>/tests --learn-apply --learn-target drifts # allowlist in tolerated-drifts.yaml
467
- ```
468
-
469
- **Caveat — `--learn-apply test` weakens assertions, not the code.**
470
- Diff `apis/<name>/tests/*.yaml` + `tolerated-drifts.yaml` after every
471
- apply; treat unexpected widenings as a review-blocker.
472
-
473
- ## Phase 6 — Stateful checks (m-20)
474
-
475
- After Phase 2 annotation is applied, run the cross-call invariants.
476
- See `zond-checks` for per-check semantics.
477
-
478
- ```bash
479
- zond checks run --api <name> --check stateful --report ndjson --live # CRUD create-chains need --live
480
- ```
481
-
482
- **Safe by default (ARV-299)**: without `--live`, `checks run` skips the
483
- stateful CRUD create-chains (`ensure_resource_availability`,
484
- `use_after_free`) — they'd POST real resources. Read-only stateful checks
485
- (pagination, observation-mode lifecycle, cross-call GET diff) still run.
486
- Add `--live` **only against a throwaway/sandbox account** to exercise the
487
- create-chains.
488
-
489
- `--check stateful` expands to the state-machine set (ARV-325):
490
- - `cross_call_references` — POST→GET drift (uses readback_diff overlay)
491
- - `idempotency_replay` — bit-identical replay on `Idempotency-Key`
492
- - `pagination_invariants` — `?limit=N` + `?after=last_id` non-overlap
493
- - `lifecycle_transitions` — declared state-machine validity
494
- - `use_after_free`, `ensure_resource_availability` — CRUD lifecycle
495
- - `cursor_boundary_fuzzing` — list-cursor edge values
496
- - `webhooks` (recipe-based, see `docs/recipes/webhook-receiver.md`)
497
-
498
- `ignored_auth` / `open_cors_on_sensitive` are **not** part of the alias —
499
- they're auth/security checks that roughly triple run time on wide APIs.
500
- Run them by explicit id: `--check ignored_auth,open_cors_on_sensitive`.
501
-
502
- **Iron rule**: don't run `--check stateful` without prior `api annotate`
503
- review. Defaults catch the obvious; quirks need declared config.
504
-
505
- ## Phase 7 — Proactive bug hunting (probes)
506
-
507
- ```bash
508
- zond probe static --api <name> # validation + methods (defaults; generator, always safe)
509
- zond probe mass-assignment --api <name> --live --emit-tests apis/<name>/probes/mass-assignment
510
- zond probe security ssrf,crlf,open-redirect --api <name> --live \
511
- --emit-tests apis/<name>/probes/security
512
- zond run apis/<name>/probes --report json
513
- ```
514
-
515
- **Safe by default (ARV-299)**: `probe mass-assignment` / `probe security`
516
- without `--live` plan only (no live attack payloads) — same as `--dry-run`.
517
- Add `--live` **only against a throwaway/sandbox account** to send probes.
518
- `probe static` just generates suites, so it's always safe.
519
-
520
- Flag: 5xx on null/empty/wrong-type body → missing validation. 2xx on
521
- undeclared method → contract drift. `is_admin: true` echoed in response
522
- → HIGH from mass-assignment.
523
-
524
- **Body-FK auto-discovery** — `mass-assignment` resolves required
525
- `*_id`/`*_slug`/`*_uuid`/`*_key` fields by hitting sibling list endpoints.
526
- If digest still says "unresolved body FKs:" — add to `.env.yaml` manually.
527
-
528
- **Auto-path-FK** — same logic for path params (`{domain_id}`/`{webhook_id}`).
529
- Cached per run. Pass `--no-discover` to opt out.
530
-
531
- ### 7.1 — Manual mass-assignment catch-up
532
-
533
- `mass-assignment` digest splits findings: HIGH / MED / LOW /
534
- INCONCLUSIVE (no valid body — fixture problem) / INCONCLUSIVE-5XX
535
- (baseline crashed — already in validation-probe). Sweep INCONCLUSIVE
536
- with `--emit-template`:
537
-
538
- ```bash
539
- zond probe mass-assignment --api <name> --emit-template "POST:/<resource>" \
540
- --output apis/<name>/probes/mass-assignment/<resource>.yaml
541
- ```
542
-
543
- Emits `create → verify → cleanup` chain pre-filled with classic vectors
544
- (`is_admin`, `role`, `owner_id`). If a privileged field survives the
545
- round-trip GET → **HIGH**, file via `report bundle --include case-study`.
546
-
547
- ### 7.2 — Security probes
548
-
549
- ```bash
550
- zond probe security ssrf,crlf --api <name> --dry-run # first run: which (endpoint, field) — same as default --safe
551
- zond probe security ssrf,crlf,open-redirect --api <name> --live # send, sandbox only
552
- ```
553
-
554
- **Targeting** (two filters; an endpoint must pass BOTH to be planned):
555
-
556
- 1. **Has a JSON request body** — GET-only routes are skipped with
557
- `no-body`. The probes work by mutating a body field, so read-only
558
- endpoints have nothing to attack.
559
- 2. **Has ≥1 field name matching the requested class detectors** — if
560
- no field matches, the endpoint is skipped with `no-matched-field`.
561
-
562
- Field autodetection: SSRF (`*_url`/`webhook`/`callback`/`format: uri`),
563
- CRLF (`subject`/`*_prefix`/`name`), open-redirect (`redirect`/`next`).
564
-
565
- If `--dry-run` reports `0 planned / 0 skipped / 0 total` with a narrow
566
- `--include`, the scope likely captured only GET routes — broaden the
567
- filter or drop `--include` to see the per-endpoint skip reasons. With
568
- no `--include`, the dry-run shows `no-body` / `no-matched-field` next
569
- to each skipped endpoint so the gap is obvious.
570
-
571
- Baseline-OK request first; if baseline ≠ 2xx → `INCONCLUSIVE-BASELINE`,
572
- attacks skipped (kills 5×404 noise on scope-locked endpoints).
573
-
574
- Verdict: HIGH (5xx OR payload echoed in 2xx — stored injection),
575
- LOW (2xx, no echo — verify side-effects manually), OK (4xx).
576
-
577
- **Cleanup is state-aware**: PUT/PATCH does `GET` → snapshot → attack →
578
- restore. POST falls back to `DELETE` with eventual-consistency retry.
579
- Restore failures → `## ⚠️ Cleanup failures` at top of digest.
580
-
581
- CI exit: non-zero on HIGH > 0 OR `cleanup.error > 0`.
582
-
583
- ### 7.3 — Post-probe hygiene
584
-
585
- ```bash
586
- zond prepare-fixtures --api <name> --refresh # drop stale FK ids
587
- zond cleanup --orphans # retry DELETE for ~/.zond/orphans/
588
- ```
589
-
590
- Skip only with `probe security --isolated` (isolated mode never attacks
591
- seeded-fixture endpoints).
592
-
593
- ## Phase 8 — Coverage + spec drift
594
-
595
- ```bash
596
- zond coverage --api <name> --union session # default: dual-metric (test + audit)
597
- zond coverage --api <name> --union since:1h # last hour
598
- zond coverage --api <name> --union tag:smoke # tag-filtered
599
- zond coverage --api <name> --scope audit --union session # only "did zond touch the API"
600
- zond coverage --api <name> --scope test --union session # legacy single block
601
- zond coverage --api <name> --fail-on-coverage 50 # CI gate (gates pass-coverage)
602
- ```
603
-
604
- **Dual-metric output (ARV-265).** `zond coverage` prints two blocks side-by-side by default:
605
-
606
- - **test-coverage** — runs produced by `zond run` only (`run_kind` ∈ {regular, probe}). Has two sub-metrics:
607
- - `pass-coverage`: endpoint had a passing 2xx (strict; CI gate via `--fail-on-coverage`)
608
- - `hit-coverage`: endpoint received any response (loose; breadth proxy)
609
- Three-bucket JSON: `covered2xx`, `coveredButNon2xx`, `unhit`.
610
- - **audit-coverage** — any HTTP touch from any producer (`zond run` + `checks run` + `probe` + `request` + `prepare-fixtures`). Use this to answer "did the scan reach the API at all?" — typical safe-mode runs without `zond run` still report >50% on a 1k-endpoint spec from `checks run` alone.
611
-
612
- Source breakdown (`--scope audit` text or `data.audit_coverage.by_source` JSON):
613
-
614
- ```
615
- audit-coverage: 619/1184 endpoints (52%, 1500 HTTP touches)
616
- by source:
617
- check 619 endpoints, 1500 events
618
- regular 0 endpoints, 0 events
619
- probe 0 endpoints, 0 events
620
- request 3 endpoints, 3 events
621
- ```
622
-
623
- Producers opt out via `ZOND_CHECKS_PERSIST=0`.
624
-
625
- **Quality signals to gate CI on** (not raw pass-coverage):
626
-
627
- | Signal | Command |
628
- |---|---|
629
- | Contract drift | `checks run --phase coverage` HIGH count == 0 |
630
- | Stateful invariants | `checks run --check stateful` HIGH count == 0 |
631
- | Auth / Authz / injection | `probe security` HIGH count == 0 |
632
- | Mass-assignment | `probe mass-assignment` HIGH count == 0 |
633
- | Response conformance | `run --validate-schema` `schema_violation` == 0 |
634
- | Drift allowlist review | `git diff apis/<name>/tolerated-drifts.yaml` |
635
-
636
- ## Phase 9 — Share findings
637
-
638
- ```bash
639
- zond report export <run-id> # default: triage/<api>/run-<id>/
640
- zond report bundle 135..142 -o triage/sweep/ # all artefacts (default): case-study + export + diagnose + index.md
641
- zond report bundle <run-id> --include case-study,diagnose # subset only (drop export)
642
- zond report bundle --session <id> -o triage/session/ # group by session
643
- ```
644
-
645
- Bodies > 8 KB truncated by default; `--no-body-cap` to keep full.
646
- Existing files at `--output` auto-rotate to `<stem>-vN.<ext>`;
647
- `--overwrite` to silence. **Always pass `--redact-identity` for
648
- outbound sharing** (secret-redaction is on by default; identity slugs
649
- need the extra flag).
650
-
651
- Offer this proactively after a run surfaces `definitely_bug`
652
- (5xx / schema violation / mass-assignment 2xx). Skip for `env_issue` /
653
- `quirk`.
654
-
655
- ## One-shot smoke audit — `zond audit`
656
-
657
- **`zond audit` is the breadth pass, not "full audit".** It covers the
658
- *smoke + coverage* slice — prepare-fixtures → generate → probe static
659
- → session-wrapped run on tests+probes → coverage → `audit-report.html`.
660
- Depth (stateful checks, security probes against R18 evidence-gates,
661
- m-20 annotate, `--learn-apply` iteration) is the **skill's job, not
662
- audit's**: those stages need decisions audit can't make
663
- (annotate-before-stateful, scoping after auth-cluster, evidence
664
- preconditions). Use audit for: CI smoke, demo, "first 5 minutes on a
665
- new API". For a real depth campaign, walk Phase 0–9 from this skill.
666
-
667
- ```bash
668
- zond audit --api <name> --dry-run # print stage plan
669
- zond audit --api <name> # safe mode (default) — no live probes
670
- zond audit --api <name> --live --with-mass-assignment --with-security # full live pipeline (throwaway/sandbox only)
671
- zond audit --api <name> --out reports/audit-<name>.html
672
- ```
673
-
674
- **ARV-264 safe vs live decision matrix.** Default is `--safe` (implicit;
675
- no flag needed). `--live` is the opt-in for traffic that mutates the
676
- target API.
677
-
678
- | Concern | `--safe` (default) | `--live` |
679
- |---|---|---|
680
- | `prepare-fixtures` mode | single-pass path-FK discovery (GET) | same — single-pass, never POST-creates |
681
- | `probe mass-assignment` | skipped even if `--with-mass-assignment` set, with a warning | runs when `--with-mass-assignment` set |
682
- | `probe security` (SSRF/CRLF/redirect) | skipped even if `--with-security` set, with a warning | runs when `--with-security` set |
683
- | `probe static` (validation+methods) | always runs — pure spec walk, no live traffic | same |
684
- | `run tests / probes` | runs against the live API but only generated GET-shape suites by default | same |
685
-
686
- **Pre-flight warnings (printed before any subprocess fires):** missing
687
- `auth_token` when the spec declares securitySchemes (probes will skip
688
- with 401 and the report will read misleadingly green); no
689
- securitySchemes declared in the spec (open API or incomplete spec —
690
- auth-related checks will skip).
691
-
692
- **Rule of thumb.** Unknown-scope PAT against a production API → `--safe`
693
- (default). Throwaway account / sandbox tenant → `--live` is fine. Never
694
- run `--live` against shared/production data on first contact.
695
-
696
- Each stage prints `==> Stage N/M: <name>` and a completion line
697
- `└─ OK · Ns` / `FAIL (exit K) · Ns` / `SKIPPED (reason)`. Exit 1 if
698
- any non-coverage stage failed.
699
-
700
- `generate` skipped via mtime when `tests/` is fresher than `spec.json`
701
- (pass `--force` to override).
702
-
703
- **Session reuse (ARV-65)**: if `.zond/current-session` is already set
704
- when audit starts (i.e. user ran `zond session start` first), audit
705
- reuses that session — `session-start` and `session-end` stages skip
706
- with reason `reusing active session <id>`. The user's session stays
707
- active after audit returns; runs inside audit are stamped with the
708
- outer `session_id` and roll up under the user's `coverage --union
709
- session` later.
710
-
711
- ARV-158: when any run inside the audit session has failures, the
712
- generated `audit-report.html` embeds a "Failures by run" section with
713
- collapsible `<details>` per run — `by_recommended_action` buckets +
714
- first example (method, path, status, reason) + concrete drill-down
715
- commands (`zond db diagnose --run-id N --json`, `zond report export
716
- N`). Triage starts from the rendered HTML; you only fall back to CLI
717
- queries when you need fields outside the example slice.
718
-
719
- When NOT to use audit: narrow asks ("fix run X", "why this endpoint
720
- 500s") — walk phases.
721
-
722
- ## Phase S — Single-flow scenarios
723
-
724
- When the user wants to verify one specific flow (not breadth):
725
-
726
- ```bash
727
- zond doctor --api <name> # confirm fixtures
728
- # write apis/<name>/scenarios/<flow-slug>.yaml directly (no scaffold cmd)
729
- zond session start --label "<short reason>"
730
- zond run apis/<name>/scenarios/<flow>.yaml --report json
731
- zond session end
732
- ```
733
-
734
- YAML grammar (zond runner format — NOT Postman / OpenAPI shape):
735
-
736
- ```yaml
737
- name: cancel-pending-order
738
- tags: [scenario, orders]
739
- base_url: "{{base_url}}"
740
- headers: { Authorization: "Bearer {{auth_token}}" }
741
- tests:
742
- - name: create order
743
- POST: /workspaces/{{ws_id}}/orders
744
- json: { amount: 100, currency: USD }
745
- expect:
746
- status: 201
747
- body:
748
- id: { capture: order_id }
749
- status: { equals: pending }
750
- - name: cancel
751
- POST: /orders/{{order_id}}/cancel
752
- expect: { status: 200 }
753
- - name: verify cancelled
754
- GET: /orders/{{order_id}}
755
- expect:
756
- status: 200
757
- body: { status: { equals: cancelled } }
758
- - name: cleanup
759
- DELETE: /orders/{{order_id}}
760
- always: true # runs even on prior failure
761
- expect: { status: [200, 204, 404] }
762
- ```
763
-
764
- Grammar:
765
- - HTTP verb is a top-level key on the step (`GET:`, `POST:`, …).
766
- - Body: `json: {...}` / `form: {...}` / `multipart:` / `text:`.
767
- - Captures: `expect.body.<field>: { capture: <var> }`.
768
- - Assertions: `equals`, `type`, `exists`, `matches`, `gt`/`lt`,
769
- `length*`, `each`, `contains_item`, `set_equals`, `oneOf`.
770
- - `expect.status`: number or array. `oneOf` / `anyOf` string forms NOT
771
- supported — pass an array directly.
772
- - `always: true` — runs on prior failure (cleanup).
773
- - `skip_if: "{{var}} =="` — skip when fixture var unset.
774
- - Generators: `{{$randomEmail}}`, `{{$uuid}}`, `{{$randomString}}`,
775
- `{{$randomInt}}`, `{{$randomIsoDate}}`. Full list:
776
- `zond reference random-helpers`.
777
-
778
- Hand-crafted scenarios are for **bug repro / post-deploy smoke /
779
- verifying a fix** — small focused work. For breadth, use Phase 3
780
- (generate) and the full audit chain.
781
-
782
- ## Diagnose failures
783
-
784
- ```bash
785
- zond db runs --limit 5 --json
786
- zond db diagnose <run-id> --json # grouped by root_cause
787
- zond db run <id> --status 500 --json
788
- zond db compare <idA> <idB> --json # regression diff + field-level body diff (.data.body_changes[], ARV-339)
789
- zond db run <id> --report yaml # run snapshot as YAML (ARV-338) — keep/diff as text
790
- ```
791
-
792
- > **db --json envelope shape** (consistent across siblings): the array
793
- > always lives under `data.<plural>`, not directly under `data`.
794
- > `db runs` → `.data.runs[]`, `db collections` → `.data.collections[]`,
795
- > `db run <id> --status …` → `.data.results[]`. Top-level totals on runs
796
- > are at `runs[].total/passed/failed`, not under a `.summary` wrapper.
797
-
798
- `recommended_action` enum (closed):
799
- - `report_backend_bug` — STOP, surface to user
800
- - `fix_test_logic` — edit the YAML
801
- - `fix_auth_config` / `fix_network_config` / `fix_env` — config issues
802
- - `fix_spec` — edit OpenAPI (from `check spec`)
803
- - `fix_fixture` — fill `.env.yaml` (from `prepare-fixtures` miss-* or
804
- inconclusive mass-assignment baseline)
805
- - `update_spec` — status-drift from `--learn`
806
-
807
- `cascade` failures collapse under root cause — fix upstream, not
808
- individually.
809
-
810
- ### Fixing 4xx from stub generators (`fix_test_logic`)
811
-
812
- When body rejected on format (400/422 + "expected ..."):
813
-
814
- 1. Read failure: `zond db run <id> --status 422 --json`.
815
- 2. **Fixture pack first** — if it's a FK id / verified resource /
816
- constrained enum, add to `.env.yaml` (Phase 1).
817
- 3. **Typed generator** — swap `{{$randomString}}` for matching format
818
- helper.
819
- 4. **Hardcoded literal** — when typed generators fail (regex too strict).
820
- 5. **Runtime captures** — for resources the test creates (capture from
821
- prior `create_*` step or `setup: true` suite). For pre-existing FKs,
822
- prefer step 2.
823
-
824
- For deep triage of a specific failing run → hand off to `zond-triage`.
825
-
826
- ## Auth / environments
827
-
828
- - `apis/<name>/.env.yaml` is both auth and fixture pack — any key
829
- interpolatable as `{{key}}`. Auto-gitignored.
830
- - Login-flow tokens: `setup: true` suite captures vars that propagate
831
- to later suites in the same run.
832
- - `zond run --env <name>` loads `.env.<name>.yaml`. Discovery walks up
833
- to workspace root; deeper files override shallower.
834
-
835
- ## File lifecycle
836
-
837
- Everything zond writes is tracked in `.zond/manifest.json` with
838
- sha256-at-write. Files whose hash no longer matches are **skipped**
839
- on clean (user edits stay safe). Re-running a generator overwrites
840
- the entry — manual edits to generator-owned files are lost on
841
- regenerate; promote them to a separate suite first.
842
-
843
- ```bash
844
- zond clean --api <name> # dry-run: lists what would be removed
845
- zond clean --api <name> --force # delete (preserves probes/)
846
- zond clean --api <name> --probes --force # also probe suites
847
- zond clean --all --force # nuclear
848
- ```
849
-
850
- Triage outputs default to
851
- `triage/<api>/<run-id>/<command>-<ts>.<ext>`. Don't pass `--output`
852
- unless the user has a specific destination.
853
-
854
- ## When to hand off
855
-
856
- - "deep depth-checks", "SARIF", "boundary coverage", "stateful
857
- invariants details", "annotate aspect XYZ" → `zond-checks`
858
- - "what failed in last run", "why is it red", "case study from run X"
859
- → `zond-triage`
860
-
861
- If the user is mid-workflow in one of those, don't restart from here.