@kirrosh/zond 0.23.0 → 0.26.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +176 -1
- package/README.md +8 -7
- package/package.json +2 -3
- package/src/CLAUDE.md +112 -0
- package/src/cli/commands/add-api.ts +19 -7
- package/src/cli/commands/api/annotate/index.ts +359 -4
- package/src/cli/commands/api/annotate/lifecycle.ts +1 -1
- package/src/cli/commands/api/annotate/overlay.ts +1 -1
- package/src/cli/commands/api/annotate/pagination.ts +10 -6
- package/src/cli/commands/api/annotate/prompts.ts +39 -2
- package/src/cli/commands/audit.ts +360 -54
- package/src/cli/commands/check.ts +15 -2
- package/src/cli/commands/checks.ts +352 -36
- package/src/cli/commands/cleanup.ts +4 -30
- package/src/cli/commands/coverage.ts +275 -57
- package/src/cli/commands/db.ts +311 -8
- package/src/cli/commands/discover.ts +281 -161
- package/src/cli/commands/doctor.ts +57 -3
- package/src/cli/commands/fixtures.ts +1 -1
- package/src/cli/commands/generate.ts +24 -7
- package/src/cli/commands/init/bootstrap.ts +4 -1
- package/src/cli/commands/init/index.ts +2 -2
- package/src/cli/commands/init/skills.ts +43 -0
- package/src/cli/commands/init/templates/agents.md +12 -3
- package/src/cli/commands/init/templates/skills/warm-up-target.md +122 -0
- package/src/cli/commands/init/templates/skills/zond-checks.md +268 -44
- package/src/cli/commands/init/templates/skills/zond-seed.md +114 -0
- package/src/cli/commands/init/templates/skills/zond-triage.md +88 -26
- package/src/cli/commands/init/templates/skills/zond.md +274 -64
- package/src/cli/commands/init/templates/zond-config.yml +1 -1
- package/src/cli/commands/prepare-fixtures.ts +14 -52
- package/src/cli/commands/probe/_seed-bodies.ts +52 -0
- package/src/cli/commands/probe/mass-assignment.ts +101 -10
- package/src/cli/commands/probe/security.ts +95 -12
- package/src/cli/commands/probe/webhooks.ts +2 -0
- package/src/cli/commands/probe.ts +87 -11
- package/src/cli/commands/refresh-api.ts +59 -1
- package/src/cli/commands/report-bundle.ts +3 -11
- package/src/cli/commands/request.ts +116 -0
- package/src/cli/commands/run.ts +53 -5
- package/src/cli/commands/schema-from-runs.ts +128 -0
- package/src/cli/commands/secrets.ts +133 -0
- package/src/cli/json-envelope.ts +0 -20
- package/src/cli/json-schemas.ts +51 -0
- package/src/cli/output.ts +17 -1
- package/src/cli/program.ts +5 -4
- package/src/cli/safe-live.ts +24 -0
- package/src/cli/status-filter.ts +0 -10
- package/src/core/audit/persist.ts +183 -0
- package/src/core/checks/budget.ts +59 -0
- package/src/core/checks/checks/cross_call_references.ts +17 -4
- package/src/core/checks/checks/cursor_boundary_fuzzing.ts +219 -0
- package/src/core/checks/checks/idempotency_replay.ts +1 -5
- package/src/core/checks/checks/ignored_auth.ts +44 -1
- package/src/core/checks/checks/index.ts +3 -0
- package/src/core/checks/checks/lifecycle_transitions.ts +169 -26
- package/src/core/checks/checks/negative_data_rejection.ts +119 -16
- package/src/core/checks/checks/not_a_server_error.ts +8 -0
- package/src/core/checks/checks/open_cors_on_sensitive.ts +47 -18
- package/src/core/checks/checks/pagination_invariants.ts +298 -117
- package/src/core/checks/checks/positive_data_acceptance.ts +1 -4
- package/src/core/checks/checks/status_code_conformance.ts +78 -7
- package/src/core/checks/mode.ts +3 -0
- package/src/core/checks/recommended-action.ts +5 -1
- package/src/core/checks/runner.ts +614 -27
- package/src/core/checks/spec-findings.ts +308 -0
- package/src/core/checks/types.ts +117 -1
- package/src/core/checks/zond-extensions.ts +73 -0
- package/src/core/classifier/recommended-action.ts +35 -6
- package/src/core/coverage/loader.ts +31 -0
- package/src/core/diagnostics/db-analysis.ts +200 -106
- package/src/core/diagnostics/failure-class.ts +21 -1
- package/src/core/diagnostics/failure-hints.ts +4 -208
- package/src/core/diagnostics/suggested-fixes.ts +2 -3
- package/src/core/generator/chunker.ts +1 -8
- package/src/core/generator/data-factory.ts +199 -61
- package/src/core/generator/fixtures-builder.ts +38 -31
- package/src/core/generator/index.ts +0 -2
- package/src/core/generator/openapi-reader.ts +98 -4
- package/src/core/generator/path-param-disambig.ts +30 -4
- package/src/core/generator/resources-builder.ts +276 -26
- package/src/core/generator/schema-utils.ts +22 -0
- package/src/core/generator/suite-generator.ts +168 -15
- package/src/core/generator/types.ts +6 -0
- package/src/core/identity/identity-file.ts +0 -0
- package/src/core/output/README.md +11 -29
- package/src/core/output/index.ts +1 -1
- package/src/core/output/run.ts +0 -35
- package/src/core/output/types.ts +0 -7
- package/src/core/parser/dynamic-values.ts +160 -0
- package/src/core/parser/variables.ts +0 -0
- package/src/core/probe/dry-run-envelope.ts +4 -0
- package/src/core/probe/mass-assignment/classify.ts +175 -0
- package/src/core/probe/mass-assignment/cleanup.ts +52 -0
- package/src/core/probe/mass-assignment/digest.ts +114 -0
- package/src/core/probe/mass-assignment/orchestrator.ts +459 -0
- package/src/core/probe/mass-assignment/regression.ts +141 -0
- package/src/core/probe/mass-assignment/suspects.ts +92 -0
- package/src/core/probe/mass-assignment/types.ts +135 -0
- package/src/core/probe/mass-assignment-probe.ts +23 -1118
- package/src/core/probe/mass-assignment-template.ts +32 -4
- package/src/core/probe/path-discovery.ts +3 -4
- package/src/core/probe/probe-harness.ts +21 -22
- package/src/core/probe/security/baseline.ts +174 -0
- package/src/core/probe/security/classify.ts +341 -0
- package/src/core/probe/security/cleanup.ts +125 -0
- package/src/core/probe/security/detectors.ts +71 -0
- package/src/core/probe/security/digest.ts +104 -0
- package/src/core/probe/security/orchestrator.ts +398 -0
- package/src/core/probe/security/regression.ts +103 -0
- package/src/core/probe/security/types.ts +151 -0
- package/src/core/probe/security-probe-class.ts +8 -2
- package/src/core/probe/security-probe.ts +28 -1449
- package/src/core/probe/shared.ts +26 -0
- package/src/core/probe/webhooks-probe.ts +5 -7
- package/src/core/runner/assertions.ts +1 -1
- package/src/core/runner/executor.ts +3 -18
- package/src/core/runner/form-encode.ts +8 -18
- package/src/core/runner/http-client.ts +38 -1
- package/src/core/runner/preflight-vars.ts +19 -15
- package/src/core/runner/rate-limiter.ts +11 -29
- package/src/core/runner/run-kind.ts +7 -1
- package/src/core/runner/schema-validator.ts +2 -6
- package/src/core/runner/send-request.ts +11 -6
- package/src/core/runner/types.ts +6 -0
- package/src/core/setup-api.ts +53 -15
- package/src/core/severity/index.ts +0 -63
- package/src/core/spec/infer-schema.ts +102 -0
- package/src/core/spec/merge-specs.ts +156 -0
- package/src/core/spec/schema-from-runs.ts +117 -0
- package/src/core/spec/schema-overlay.ts +130 -0
- package/src/core/util/ajv.ts +13 -0
- package/src/core/util/headers.ts +9 -0
- package/src/core/util/url.ts +24 -0
- package/src/core/workspace/fixture-gap-report.ts +84 -0
- package/src/core/workspace/fixture-gaps.ts +71 -0
- package/src/core/workspace/root.ts +13 -11
- package/src/db/migrate.ts +2 -0
- package/src/db/migrations/0002_run_kind_request.sql +59 -0
- package/src/db/queries/collections.ts +2 -2
- package/src/db/queries/results.ts +88 -0
- package/src/db/queries/runs.ts +56 -2
- package/src/db/queries.ts +3 -0
- package/src/db/schema.ts +7 -7
- package/src/cli/commands/bootstrap.ts +0 -710
- package/src/core/anti-fp/bootstrap.ts +0 -34
- package/src/core/anti-fp/index.ts +0 -33
- package/src/core/anti-fp/registry.ts +0 -44
- package/src/core/anti-fp/rules/baseline-echo.ts +0 -74
- package/src/core/anti-fp/rules/schemathesis/body_negation_becomes_valid.ts +0 -52
- package/src/core/anti-fp/rules/schemathesis/coverage_phase_boundary_positive.ts +0 -38
- package/src/core/anti-fp/rules/schemathesis/has_unverifiable_mutations.ts +0 -35
- package/src/core/anti-fp/rules/schemathesis/index.ts +0 -24
- package/src/core/anti-fp/rules/schemathesis/string_type_mutation_becomes_valid.ts +0 -53
- package/src/core/anti-fp/rules/subscription-gated/index.ts +0 -11
- package/src/core/anti-fp/rules/subscription-gated/paid-plan-403.ts +0 -75
- package/src/core/anti-fp/types.ts +0 -68
- package/src/core/generator/create-body.ts +0 -89
|
@@ -58,9 +58,23 @@ Pragmatic режим (default) — реалистичный для production AP
|
|
|
58
58
|
|
|
59
59
|
## Iron rules
|
|
60
60
|
|
|
61
|
+
- **Safe by default (ARV-299).** `checks run` and the mutating probes
|
|
62
|
+
(`probe mass-assignment`, `probe security`) default to `--safe`: no
|
|
63
|
+
destructive traffic. For `checks run` that means the stateful CRUD
|
|
64
|
+
create-chains (`ensure_resource_availability`, `use_after_free`) self-skip
|
|
65
|
+
— read-only checks (conformance, pagination, observation-mode lifecycle)
|
|
66
|
+
still run. For probes it means plan-only (no live attack payloads). Pass
|
|
67
|
+
`--live` to opt into mutating traffic — **only against a throwaway/sandbox
|
|
68
|
+
account.** Same vocabulary as `zond audit --safe/--live`. `probe static`
|
|
69
|
+
only generates suites, so it's always safe (`--live` is a no-op there).
|
|
61
70
|
- **NEVER hand-roll these checks in YAML.** The catalog encodes
|
|
62
71
|
schemathesis V4 semantics 1-to-1 — replicating them in YAML drifts
|
|
63
72
|
silently. `zond checks run` is the single source of truth.
|
|
73
|
+
- **`--check stateful` = state-machine set only (ARV-325):**
|
|
74
|
+
cross_call_references, idempotency_replay, pagination_invariants,
|
|
75
|
+
lifecycle_transitions, use_after_free, ensure_resource_availability,
|
|
76
|
+
cursor_boundary_fuzzing. `ignored_auth` / `open_cors_on_sensitive` are
|
|
77
|
+
NOT included — run them by explicit id when you want the security pair.
|
|
64
78
|
- **NEVER run `--check stateful` without prior `api annotate` review.**
|
|
65
79
|
m-20 stateful checks (cross_call_references, idempotency_replay,
|
|
66
80
|
pagination_invariants, lifecycle_transitions) read `.api-resources.local.yaml`
|
|
@@ -74,15 +88,86 @@ Pragmatic режим (default) — реалистичный для production AP
|
|
|
74
88
|
- **Auth headers are auto-derived from `--api <name>` `.env.yaml`**
|
|
75
89
|
(`auth_token` → `Authorization: Bearer …`, `api_key` → `X-API-Key`).
|
|
76
90
|
Add explicit ones with `--auth-header 'Name: value'` (repeatable).
|
|
91
|
+
- **Prewarm a fresh workspace before `zond checks run`.** Right after
|
|
92
|
+
`zond add api <name>`, `.env.yaml` only has the auth skeleton — list
|
|
93
|
+
endpoints can return 401 until a `setup:` suite captures session
|
|
94
|
+
vars. Running checks cold (e.g. `--check ignored_auth` or a full
|
|
95
|
+
run) immediately surfaces dozens of
|
|
96
|
+
spurious `ignored_auth` / `report_backend_bug` findings (cold-state
|
|
97
|
+
race, F1/F18 in feedback loop). Fix: run one smoke pass first
|
|
98
|
+
(`zond run apis/<name>/tests --safe --report json` or `zond doctor
|
|
99
|
+
--api <name> --missing-only`), let `prepare-fixtures` warm
|
|
100
|
+
`.env.yaml`, then `zond checks run`.
|
|
101
|
+
- **Treat per-check noise as fixture / generator drift, not a backend
|
|
102
|
+
bug.** Two checks are reliably noisy on production APIs and need a
|
|
103
|
+
pre-filter before triage:
|
|
104
|
+
- `positive_data_acceptance` — emits MEDIUM when a generated body is
|
|
105
|
+
rejected with 422. On APIs with **semantic validation** (Resend
|
|
106
|
+
domain ownership, Stripe currency vs country, …) this is almost
|
|
107
|
+
always `fix_generator` / `fix_fixture`, not `report_backend_bug`.
|
|
108
|
+
Inspect the 422 reason; if it mentions a business rule the
|
|
109
|
+
generator can't satisfy, file under `regenerate_suite` or annotate
|
|
110
|
+
`seed_body`.
|
|
111
|
+
- `response_schema_conformance` — HIGH on format-only mismatches
|
|
112
|
+
(`format: email` rejecting a free-form string the server returned)
|
|
113
|
+
is contract drift; HIGH on a field the server actually doesn't
|
|
114
|
+
persist is a backend bug. Use `cross_call_references` to
|
|
115
|
+
disambiguate before reporting.
|
|
116
|
+
- **No response schema declared** (Sentry-style specs where the
|
|
117
|
+
check skips 200+ endpoints): mine schemas from a real run and
|
|
118
|
+
patch them in — `zond schema-from-runs --api <name>` writes
|
|
119
|
+
`patch.schema.json` from the run's 2xx bodies, then
|
|
120
|
+
`zond refresh-api --api <name> --merge-schema patch.schema.json`
|
|
121
|
+
folds them into `.api-schema.local.yaml` (survives refresh) and
|
|
122
|
+
re-runs conformance with real schemas (ARV-175/176).
|
|
123
|
+
|
|
124
|
+
## In-spec `x-zond-*` extensions (ARV-189, m-21)
|
|
125
|
+
|
|
126
|
+
Low-friction per-endpoint policy via OpenAPI vendor extensions —
|
|
127
|
+
when a single operation needs an exception, avoid the round-trip
|
|
128
|
+
through `.api-resources.local.yaml`. Implemented today:
|
|
129
|
+
|
|
130
|
+
| Extension | Type | Effect |
|
|
131
|
+
|---|---|---|
|
|
132
|
+
| `x-zond-skip: [check_id, …]` (or single string) | string\[] | skip those checks for this op |
|
|
133
|
+
| `x-zond-public: true` | boolean | shortcut: skip auth-class checks (`ignored_auth`, `missing_required_header`) |
|
|
134
|
+
|
|
135
|
+
Place at operation level (preferred) or path-item level — operation
|
|
136
|
+
wins on key collision.
|
|
137
|
+
|
|
138
|
+
```yaml
|
|
139
|
+
paths:
|
|
140
|
+
/v1/public/status:
|
|
141
|
+
get:
|
|
142
|
+
x-zond-public: true # public health route
|
|
143
|
+
responses: { '200': { description: ok } }
|
|
144
|
+
/v1/admin/raw-debug:
|
|
145
|
+
get:
|
|
146
|
+
x-zond-skip: [status_code_conformance] # known 500-on-bad-input
|
|
147
|
+
responses: { '500': { description: 'debug 500' } }
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Skipped runs surface in `--report` `skipped_outcomes` as
|
|
151
|
+
`<check>: x-zond-skip listed "<id>" at the spec level` or
|
|
152
|
+
`<check>: x-zond-public: true (auth check suppressed …)` so the
|
|
153
|
+
operator can tell spec-level suppression apart from runtime skips.
|
|
154
|
+
|
|
155
|
+
Priority (highest first): `.api-resources.local.yaml` overlay >
|
|
156
|
+
`x-zond-*` extensions > `.api-resources.yaml` baseline > built-in
|
|
157
|
+
defaults. For one-shots use extensions; for resource-wide policy
|
|
158
|
+
(`idempotency.header`, `lifecycle.field`, …) use the overlay yaml.
|
|
159
|
+
|
|
160
|
+
Not yet implemented: `x-zond-resource` / `x-zond-idempotent` /
|
|
161
|
+
`x-zond-lifecycle-field` — these need deeper overlay-config wiring
|
|
162
|
+
and are tracked separately. Use `annotate apply` for those today.
|
|
77
163
|
|
|
78
164
|
## Phase pre-0 — Annotation (mandatory for `--check stateful`)
|
|
79
165
|
|
|
80
|
-
> **Heads-up for `pagination_invariants`:**
|
|
81
|
-
>
|
|
82
|
-
>
|
|
83
|
-
>
|
|
84
|
-
>
|
|
85
|
-
> runtime effect**, skip the dump+apply step for those resources.
|
|
166
|
+
> **Heads-up for `pagination_invariants`:** `cursor` (Stripe-style) and
|
|
167
|
+
> `page` (GitHub/GitLab/Atlassian/Notion-style `?page=N&per_page=M`)
|
|
168
|
+
> are implemented. `offset` / `token` annotations parse but the check
|
|
169
|
+
> short-circuits with "type not implemented" — skip dump+apply for
|
|
170
|
+
> those resources until later milestones.
|
|
86
171
|
|
|
87
172
|
m-20 stateful checks rely on per-resource config in
|
|
88
173
|
`.api-resources.local.yaml` (overlay that survives `refresh-api`).
|
|
@@ -104,6 +189,12 @@ zond api annotate dump --api <name> --resources > /tmp/orphans.json # optio
|
|
|
104
189
|
|
|
105
190
|
Restrict scope with `--only r1,r2,r3` on any dump.
|
|
106
191
|
|
|
192
|
+
`annotate dump` emits **JSON** (a top-level array of per-resource
|
|
193
|
+
slices). YAML is a JSON superset, so the file parses either way — when
|
|
194
|
+
piping into a YAML-aware reader, treat it as a list of objects with the
|
|
195
|
+
same shape documented below. The corresponding response file the agent
|
|
196
|
+
writes back is **YAML** (more comments-friendly).
|
|
197
|
+
|
|
107
198
|
Agent reads each dump and writes a YAML response file (top-level list
|
|
108
199
|
of entries — see per-check schemas below for the block shape).
|
|
109
200
|
Optional `rationale` and `confidence: high|medium|low` per entry help
|
|
@@ -124,7 +215,7 @@ existing by default (renders `! field: (conflict — kept existing; pass
|
|
|
124
215
|
**Recommended pre-stateful sweep on a new API:**
|
|
125
216
|
|
|
126
217
|
```bash
|
|
127
|
-
zond api annotate dump --api <name> --seed-bodies > /tmp/seed.json # for
|
|
218
|
+
zond api annotate dump --api <name> --seed-bodies > /tmp/seed.json # create-body overlay for stateful checks
|
|
128
219
|
zond api annotate dump --api <name> --readback > /tmp/readback.json # for cross_call_references
|
|
129
220
|
zond api annotate dump --api <name> --pagination > /tmp/pag.json # for pagination_invariants
|
|
130
221
|
zond api annotate dump --api <name> --lifecycle > /tmp/life.json # for lifecycle_transitions
|
|
@@ -139,6 +230,13 @@ zond api annotate apply --api <name> --idempotency --input /tmp/idem.yaml --yes
|
|
|
139
230
|
|
|
140
231
|
Each block's YAML format is documented under the per-check section below.
|
|
141
232
|
|
|
233
|
+
**Who writes the annotations:** zond does not infer them. The agent reads
|
|
234
|
+
the `dump` output (spec slice + expected YAML shape) and writes the overlay;
|
|
235
|
+
`apply` validates it against the zod contract and merges into
|
|
236
|
+
`.api-resources.local.yaml`. For large specs, dump per-resource with
|
|
237
|
+
`--only <res>` and (for seed-bodies) `--with-last-attempt --history N` to
|
|
238
|
+
pull the recent seed-POST error progression onto your screen before writing.
|
|
239
|
+
|
|
142
240
|
## Reading findings
|
|
143
241
|
|
|
144
242
|
Every finding carries a closed-enum `recommended_action` so the agent
|
|
@@ -154,16 +252,64 @@ can route without parsing free-form messages:
|
|
|
154
252
|
| `fix_network_config` | Transport-level error (timeout / DNS / refused). Verify `base_url` and reachability before re-running. |
|
|
155
253
|
| `wontfix_known_limitation` | Known accepted gap. Don't retry, don't file a bug. |
|
|
156
254
|
|
|
157
|
-
|
|
158
|
-
|
|
255
|
+
**Severity is a coarse deterministic CI-gate default, not a priority signal
|
|
256
|
+
(ARV-346).** zond stamps each check a fixed severity so the exit code
|
|
257
|
+
(`high_or_critical`) and SARIF mapping are reproducible — it is NOT a judgment
|
|
258
|
+
about which finding matters most on *your* API. You (the agent) prioritize
|
|
259
|
+
from `recommended_action` + the raw evidence, not from zond's severity number:
|
|
260
|
+
route by the closed enum first, read the response body, decide impact. The
|
|
261
|
+
severity field only answers "does this trip the CI gate?", nothing more. (This
|
|
262
|
+
is the deterministic per-check default — not the removed ARV-337 calibrator.)
|
|
263
|
+
|
|
264
|
+
HIGH/CRITICAL gates exit-code 1; LOW/MEDIUM is informational. In an
|
|
265
|
+
orchestrator/CI that must tell "found drift" from "command crashed", pass
|
|
266
|
+
`--no-fail-on-findings` (alias `--advisory`, ARV-308): exit 0 even with
|
|
267
|
+
HIGH/CRITICAL findings — the findings are still in the envelope. Mirrors
|
|
268
|
+
`zond run --no-fail-on-failures`. Exit 2 stays reserved for real failures.
|
|
269
|
+
|
|
270
|
+
### Spec-level rollup (ARV-60)
|
|
271
|
+
|
|
272
|
+
When a single spec-level gap manifests on many operations (`401 not
|
|
273
|
+
declared in spec` × 83 sites; `no JSON Schema on this response branch` ×
|
|
274
|
+
all skipped cases; `use_after_free` ran 0 cases because there's no
|
|
275
|
+
DELETE+GET pair in the spec), the runner emits a single `spec_finding`
|
|
276
|
+
instead of N per-op rows. Threshold: ≥80% of the check's applicable
|
|
277
|
+
operations sharing one root cause.
|
|
278
|
+
|
|
279
|
+
Each `spec_finding` has:
|
|
280
|
+
|
|
281
|
+
| Field | Meaning |
|
|
282
|
+
|---|---|
|
|
283
|
+
| `kind` | `status_drift` / `missing_declaration` / `no_detector` / `broken_baseline` / `other` |
|
|
284
|
+
| `reason` | One-line root cause statement |
|
|
285
|
+
| `fix_hint` | Actionable next step (spec edit / tolerate flag / annotate command) |
|
|
286
|
+
| `affected_operations` | Operations covered (empty for skip/no-detector clusters) |
|
|
287
|
+
| `count` / `applicable` | Cluster size + the population it was measured against |
|
|
288
|
+
|
|
289
|
+
`broken_baseline` (ARV-307) is special: when the positive/success probes
|
|
290
|
+
overwhelmingly failed (>90% non-2xx — e.g. a fully auth-rejected scan),
|
|
291
|
+
`status_code_conformance` / `content_type_conformance` findings are baseline
|
|
292
|
+
artifacts, so they're removed from `findings[]` and rolled into this one
|
|
293
|
+
row. Fix the baseline (auth + path-param fixtures) and re-run before trusting
|
|
294
|
+
conformance — undeclared statuses on an all-4xx scan are not real drift.
|
|
295
|
+
|
|
296
|
+
Triage spec_findings first — they collapse 1×N noise into one decision.
|
|
297
|
+
Per-op findings still live in `data.findings`; `--verbose` brings the
|
|
298
|
+
full unaggregated list back to stdout. SARIF + JSON envelope always
|
|
299
|
+
carry both layers.
|
|
300
|
+
|
|
301
|
+
```bash
|
|
302
|
+
zond checks run --api myapi --json | jq '.data.spec_findings'
|
|
303
|
+
zond checks run --api myapi --report ndjson | jq -c 'select(.type == "spec_finding")'
|
|
304
|
+
```
|
|
159
305
|
|
|
160
306
|
## Output formats
|
|
161
307
|
|
|
162
308
|
```bash
|
|
163
|
-
zond checks run --api myapi --json # one envelope: findings[] + summary
|
|
309
|
+
zond checks run --api myapi --json # one envelope: findings[] + spec_findings[] + summary
|
|
164
310
|
zond checks run --api myapi --report sarif --output zond.sarif
|
|
165
311
|
# SARIF v2.1.0 for github/codeql-action/upload-sarif@v3
|
|
166
|
-
zond checks run --api myapi --report ndjson | jq -c '.' # streaming events: check_start
|
|
312
|
+
zond checks run --api myapi --report ndjson | jq -c '.' # streaming events: check_start | check_result | finding | spec_finding | summary
|
|
167
313
|
```
|
|
168
314
|
|
|
169
315
|
NDJSON event schema is published at `docs/json-schema/ndjson-events.schema.json`
|
|
@@ -183,13 +329,40 @@ zond checks run --api myapi --mode positive # contract verification only
|
|
|
183
329
|
zond checks run --api myapi --mode negative # malicious-input probes only
|
|
184
330
|
|
|
185
331
|
# Pick a phase
|
|
186
|
-
zond checks run --api myapi --phase coverage #
|
|
187
|
-
zond checks run --api myapi --phase examples # default —
|
|
332
|
+
zond checks run --api myapi --phase coverage # ~×1.75 cases per op; real bug-hunt phase
|
|
333
|
+
zond checks run --api myapi --phase examples # default — 1 positive + 1 negative per op (smoke)
|
|
188
334
|
```
|
|
189
335
|
|
|
336
|
+
**`--phase examples` is smoke, not depth.** It runs one positive + one
|
|
337
|
+
negative example per operation, finishes ~3× faster, but routinely
|
|
338
|
+
returns zero HIGH findings on well-formed APIs because boundary
|
|
339
|
+
mutations never fire. For an actual bug-hunt sweep, pass
|
|
340
|
+
`--phase coverage` — it expands each operation to ~×1.75 cases with
|
|
341
|
+
deterministic boundary values (min/max ints, empty strings, length
|
|
342
|
+
overflows, RFC-3339 edge dates, …). Use `examples` only for fast
|
|
343
|
+
gate-keeping in CI; use `coverage` once before sign-off.
|
|
344
|
+
|
|
190
345
|
`--allow-x00` adds the NUL byte (`\x00`) to string boundaries during
|
|
191
346
|
coverage — off by default (some HTTP/JSON stacks panic on it).
|
|
192
347
|
|
|
348
|
+
## Contribution to audit-coverage (ARV-265)
|
|
349
|
+
|
|
350
|
+
Every case `checks run` dispatches is now written to `runs`/`results`
|
|
351
|
+
with `run_kind = 'check'`. This means `zond coverage` picks it up in
|
|
352
|
+
its **audit-coverage** block — answering "did the scan touch endpoint
|
|
353
|
+
X?" without requiring a separate `zond run`. The test-coverage block
|
|
354
|
+
(pass/hit) is unaffected — `checks run` does not produce passing test
|
|
355
|
+
suites, so it doesn't contribute to `pass-coverage`.
|
|
356
|
+
|
|
357
|
+
```bash
|
|
358
|
+
zond checks run --api myapi # touches every endpoint at least once
|
|
359
|
+
zond coverage --api myapi --scope audit # 100% audit-coverage on the GET surface
|
|
360
|
+
zond coverage --api myapi # default dual output: test + audit blocks
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
Opt-out: `ZOND_CHECKS_PERSIST=0 zond checks run …` skips the
|
|
364
|
+
persistence step (matches pre-ARV-265 behaviour).
|
|
365
|
+
|
|
193
366
|
## Concurrency
|
|
194
367
|
|
|
195
368
|
```bash
|
|
@@ -304,52 +477,70 @@ resources:
|
|
|
304
477
|
Anti-FP: 429/409 on the 2nd POST → skip with cleanup. No DELETE on the
|
|
305
478
|
group → finding still fires, evidence carries `cleanup_warning`.
|
|
306
479
|
|
|
307
|
-
## Pagination invariants (m-20 ARV-171)
|
|
480
|
+
## Pagination invariants (m-20 ARV-171, m-21 ARV-220)
|
|
481
|
+
|
|
482
|
+
`pagination_invariants` — fetch two consecutive pages and assert the
|
|
483
|
+
contract holds. Two styles implemented:
|
|
308
484
|
|
|
309
|
-
|
|
310
|
-
|
|
485
|
+
**Cursor style** (Stripe / Notion / Linear, default when
|
|
486
|
+
`starting_after` / `cursor` / `after` / `page_token` query param is
|
|
487
|
+
detected):
|
|
311
488
|
|
|
312
489
|
- **duplicate_items** — an item id appears on both page A and page B
|
|
313
490
|
(off-by-one / cursor stops one short). HIGH-signal.
|
|
314
491
|
- **has_more_inconsistent** — page A said `has_more=true`, but page B is
|
|
315
|
-
empty and doesn't flip to `has_more=false`.
|
|
316
|
-
signalling.
|
|
492
|
+
empty and doesn't flip to `has_more=false`. Broken end-of-list signal.
|
|
317
493
|
- **partial_page_with_has_more** — page A returns fewer items than the
|
|
318
|
-
requested limit *yet* advertises `has_more=true`.
|
|
319
|
-
|
|
494
|
+
requested limit *yet* advertises `has_more=true`. Cursor truncates.
|
|
495
|
+
|
|
496
|
+
**Page-number style** (GitHub / GitLab / Atlassian / Notion, default
|
|
497
|
+
when `page` / `page_number` query param is detected):
|
|
498
|
+
|
|
499
|
+
- **duplicate_items** — items overlap across pages N and N+1.
|
|
500
|
+
- **per_page_exceeded** — server returned more items than `per_page=N`
|
|
501
|
+
(e.g. asked for 2, got 5). `has_more` is NOT enforced — most APIs in
|
|
502
|
+
this family signal end-of-list via Link headers / `total_pages`.
|
|
320
503
|
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
short-circuits with a "type not implemented" reason so the yaml block
|
|
324
|
-
stays a stable schema.
|
|
504
|
+
`offset` / `token` declarations parse but the check short-circuits with
|
|
505
|
+
a "type not implemented" reason so the yaml block stays a stable schema.
|
|
325
506
|
|
|
326
|
-
|
|
327
|
-
`
|
|
328
|
-
|
|
329
|
-
`has_more_field=has_more`, `limit=2`).
|
|
507
|
+
Defaults: `cursor_field=id`, `items_field=data|items|results|value`,
|
|
508
|
+
`has_more_field=has_more`, `limit=2`. Page-style adds `page_param=page`,
|
|
509
|
+
`start_page=1`, `limit_param=per_page`.
|
|
330
510
|
|
|
331
511
|
Per-resource yaml override (author via Phase pre-0
|
|
332
512
|
`annotate dump --pagination` → agent → `apply`):
|
|
333
513
|
|
|
334
514
|
```yaml
|
|
335
515
|
resources:
|
|
336
|
-
- resource: customer
|
|
337
|
-
# … existing fields …
|
|
516
|
+
- resource: customer # Stripe-style cursor
|
|
338
517
|
pagination:
|
|
339
|
-
type: cursor
|
|
340
|
-
cursor_param: starting_after
|
|
341
|
-
cursor_field: id
|
|
342
|
-
has_more_field: has_more
|
|
343
|
-
limit_param: limit
|
|
344
|
-
default_limit: 2
|
|
345
|
-
items_field: data
|
|
518
|
+
type: cursor
|
|
519
|
+
cursor_param: starting_after # Stripe; "after" / "cursor" / "page_token" also work
|
|
520
|
+
cursor_field: id # field on each item that feeds the next cursor
|
|
521
|
+
has_more_field: has_more # response field that flips on end-of-list
|
|
522
|
+
limit_param: limit
|
|
523
|
+
default_limit: 2
|
|
524
|
+
items_field: data # array container (falls back to items / results / value)
|
|
525
|
+
|
|
526
|
+
- resource: issue # GitHub-style page
|
|
527
|
+
pagination:
|
|
528
|
+
type: page
|
|
529
|
+
page_param: page # query param carrying page number (default `page`)
|
|
530
|
+
start_page: 1 # 1-based on GitHub/GitLab; 0 for some APIs
|
|
531
|
+
limit_param: per_page # default `per_page` for page-style
|
|
532
|
+
default_limit: 2
|
|
533
|
+
cursor_field: id # dedupe key across pages
|
|
534
|
+
items_field: data # omit if response is a bare array
|
|
346
535
|
```
|
|
347
536
|
|
|
348
|
-
## Lifecycle transitions (m-20 ARV-172)
|
|
537
|
+
## Lifecycle transitions (m-20 ARV-172, m-21 ARV-219)
|
|
538
|
+
|
|
539
|
+
`lifecycle_transitions` runs in one of two modes based on the
|
|
540
|
+
`actions:` field of the yaml manifest:
|
|
349
541
|
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
named actions, asserting:
|
|
542
|
+
**Action-driven mode** (preferred — needs create + read endpoints):
|
|
543
|
+
The check creates a resource and walks the named actions, asserting:
|
|
353
544
|
|
|
354
545
|
- **undeclared_state** — observed state isn't in declared `states[]`.
|
|
355
546
|
- **wrong_expected_state** — action landed the resource in a state other
|
|
@@ -363,6 +554,28 @@ named actions, asserting:
|
|
|
363
554
|
- **action_rejected** — first-call non-2xx (server-side gating). Not a
|
|
364
555
|
contract bug per se, surfaced as INCONCLUSIVE-class info.
|
|
365
556
|
|
|
557
|
+
**Observation mode** (ARV-219 — read-only state machines): when
|
|
558
|
+
`actions: {}` (or omitted) AND the resource has a list endpoint, the
|
|
559
|
+
check GETs the list once and asserts every observed `field` value is
|
|
560
|
+
in `states[]`. Reports `undeclared_state` with sample item ids for
|
|
561
|
+
each unexpected value.
|
|
562
|
+
|
|
563
|
+
When to use observation mode:
|
|
564
|
+
- The API exposes a status enum the agent enumerated from the spec,
|
|
565
|
+
but the auth scope is read-only and cannot drive mutations (GitHub
|
|
566
|
+
Issues with a read PAT, public-read endpoints).
|
|
567
|
+
- You want to detect contract-doc drift (spec says `enum: [open,
|
|
568
|
+
closed]`, prod returns `archived`).
|
|
569
|
+
|
|
570
|
+
Observation mode cannot verify the `transitions` graph (no time
|
|
571
|
+
series in a single list call) — the transition arrows stay purely
|
|
572
|
+
documentation. If write scope is available, prefer action mode for
|
|
573
|
+
the full guarantee surface.
|
|
574
|
+
|
|
575
|
+
Skip reasons unique to observation mode: `list returned <code>` (broken
|
|
576
|
+
baseline), `list empty — no data to observe`, `state field "<x>"
|
|
577
|
+
missing on all N observed items — yaml mismatch or nested field`.
|
|
578
|
+
|
|
366
579
|
Manifest validation runs at yaml load (catches cycles, unreachable
|
|
367
580
|
states, missing terminal, actions referencing undeclared states)
|
|
368
581
|
before any HTTP call goes out.
|
|
@@ -370,12 +583,12 @@ before any HTTP call goes out.
|
|
|
370
583
|
Author via Phase pre-0 `annotate dump --lifecycle` → agent → `apply`.
|
|
371
584
|
The dump emits action-endpoint candidates (POST `/{resource}/{id}/cancel`,
|
|
372
585
|
PATCH `/{resource}/{id}/status` etc.); agent decides the state machine
|
|
373
|
-
graph.
|
|
586
|
+
graph. For observation-only resources, return `actions: {}` and the
|
|
587
|
+
check picks the observation path automatically.
|
|
374
588
|
|
|
375
589
|
```yaml
|
|
376
590
|
resources:
|
|
377
|
-
- resource: subscription
|
|
378
|
-
# … existing fields …
|
|
591
|
+
- resource: subscription # action-driven mode
|
|
379
592
|
lifecycle:
|
|
380
593
|
field: status
|
|
381
594
|
states: [pending, active, cancelled]
|
|
@@ -390,6 +603,17 @@ resources:
|
|
|
390
603
|
cancel:
|
|
391
604
|
endpoint: POST /v1/subscriptions/{id}/cancel
|
|
392
605
|
expected_state: cancelled
|
|
606
|
+
|
|
607
|
+
- resource: issue # observation mode
|
|
608
|
+
lifecycle:
|
|
609
|
+
field: state
|
|
610
|
+
states: [open, closed]
|
|
611
|
+
transitions:
|
|
612
|
+
- from: open
|
|
613
|
+
to: [closed]
|
|
614
|
+
- from: closed
|
|
615
|
+
to: []
|
|
616
|
+
actions: {} # no actions → check observes via GET /issues
|
|
393
617
|
```
|
|
394
618
|
|
|
395
619
|
Action endpoints accept the `{id}` placeholder (replaced with the
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: zond-seed
|
|
3
|
+
description: |
|
|
4
|
+
Agent-orchestrated auto-seed: create the fixtures a deep audit needs from
|
|
5
|
+
what the API itself offers, so CRUD depth stops being gated by hand-seeded
|
|
6
|
+
`{{account}}`/`{{customer}}`/… ids. Use when `prepare-fixtures` reports
|
|
7
|
+
`unseededRoots`, when `db diagnose` shows `cascade_skips`, or the user asks
|
|
8
|
+
to "seed fixtures", "create test data", "un-skip the CRUD chains", "raise
|
|
9
|
+
depth". YOU reason (read the API graph, author create-bodies, order them,
|
|
10
|
+
read the 4xx and fix it); zond only EXECUTES (POST + capture id into
|
|
11
|
+
`.env.yaml`). Sibling `zond` runs the full audit; hand back to it once
|
|
12
|
+
fixtures are seeded.
|
|
13
|
+
allowed-tools: [Read, Write, Edit, Bash(zond *), Bash(bunx zond *)]
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
# zond-seed — agent-orchestrated fixture creation
|
|
17
|
+
|
|
18
|
+
The m-24 endgame: an autonomous seed loop where the DECISION is yours and the
|
|
19
|
+
EXECUTION is zond's. This is **not** a revert of ARV-336 (the blind recursive
|
|
20
|
+
cascade zond removed — 1% success on Stripe). The difference is the feedback
|
|
21
|
+
loop: you read the spec + resource graph, author a create-body, POST it, and
|
|
22
|
+
when the server says 400 you read *why* and fix the body. That is exactly why
|
|
23
|
+
it works where the heuristic engine failed.
|
|
24
|
+
|
|
25
|
+
## Iron rules (read once, apply always)
|
|
26
|
+
|
|
27
|
+
- **You reason, zond executes.** You author every create-body and pick the
|
|
28
|
+
order; zond only POSTs and captures the returned id. Never ask zond to
|
|
29
|
+
"figure out" a body — that heuristic engine was deleted (ARV-336, m-24) and
|
|
30
|
+
must not grow back. There is no `zond seed` command by design.
|
|
31
|
+
- **No blind cascade — one create at a time, with a feedback loop.** POST a
|
|
32
|
+
body, read the response. On 4xx, read the error message, revise the *specific*
|
|
33
|
+
field it names, retry. Cap at **~3 attempts per resource**, then mark it
|
|
34
|
+
un-seedable and move on. Never fire a chain of creates without reading each
|
|
35
|
+
reply.
|
|
36
|
+
- **Seed only what the API can self-serve.** Fixtures needing external input —
|
|
37
|
+
KYC-verified accounts, verified bank accounts, webhook signing secrets,
|
|
38
|
+
paid-plan / SCIM / TOS-gated resources, or ids that only exist after a real
|
|
39
|
+
event (`issue_id`, `file_id`, `integration_id`) — are **reported, not
|
|
40
|
+
invented** (pairs with `prepare-fixtures` `unseededRoots` and ARV-349/350).
|
|
41
|
+
Guessing a value here just 422s and lies about coverage. Hand those to the
|
|
42
|
+
**`warm-up-target`** skill, which warms them via the target's own SDK/CLI.
|
|
43
|
+
- **Live + throwaway/sandbox + cleanup only.** Never seed against production or
|
|
44
|
+
shared data. Confirm `base_url` is a sandbox/test account first. Track every
|
|
45
|
+
id you create and DELETE it at the end (or seed inside a disposable account /
|
|
46
|
+
Stripe test-clock). If you cannot guarantee cleanup, ask before creating.
|
|
47
|
+
- **Honor prose exclusion.** Specs document mutual exclusion in `description`,
|
|
48
|
+
not machine-readable `oneOf` (Stripe: "You may only specify one of A, B").
|
|
49
|
+
If a 400 says that, drop one member and retry (folded from ARV-347).
|
|
50
|
+
|
|
51
|
+
## Inputs — what to seed
|
|
52
|
+
|
|
53
|
+
| Signal | Command | Field |
|
|
54
|
+
|---|---|---|
|
|
55
|
+
| Chain-heads gating suites | `zond prepare-fixtures --api <name> --json` | `summary.fixtureGaps.unseededRoots[]` |
|
|
56
|
+
| Empty captures from last run | `zond db diagnose --json` | `cascade_skips[].capture_var` |
|
|
57
|
+
| Dependency order | `apis/<name>/.api-resources.yaml` | each resource's `fkDependencies` |
|
|
58
|
+
| Body schema + last server reply | `zond api annotate dump --seed-bodies --with-last-attempt --api <name>` | `seed_body`, `last_attempt` |
|
|
59
|
+
|
|
60
|
+
`unseededRoots` is your worklist. `fkDependencies` gives the order: a resource
|
|
61
|
+
whose FK points at another must be created *after* its parent (parent id is
|
|
62
|
+
captured first, then referenced). Sort parent-before-child yourself; skip
|
|
63
|
+
cycles and resources whose only dep is external input.
|
|
64
|
+
|
|
65
|
+
**miss-no-list `candidates` (ARV-382):** when prepare-fixtures can't confidently
|
|
66
|
+
derive the owner list for a fixture var, the item carries a `candidates[]` of
|
|
67
|
+
plausible GET/list endpoints (ranked by structural proximity; deprecated ones
|
|
68
|
+
marked `(deprecated)`). zond does NOT pick — that's your call: `zond request`
|
|
69
|
+
the top candidate, read a record, `zond fixtures add <var>=<id>`. An empty
|
|
70
|
+
`candidates` (and no owner) is an honest dead-end: no listable source exists —
|
|
71
|
+
create the resource or obtain the id from a parent flow.
|
|
72
|
+
|
|
73
|
+
## The loop
|
|
74
|
+
|
|
75
|
+
For each root, parent-first:
|
|
76
|
+
|
|
77
|
+
1. **Author the body.** `zond api annotate dump --seed-bodies --with-last-attempt`
|
|
78
|
+
gives the schema, a rationale, and `last_attempt` (what zond last POSTed and
|
|
79
|
+
how the server replied). Fill required fields with real-looking values;
|
|
80
|
+
reference already-captured parents as `{{parent_var}}`.
|
|
81
|
+
|
|
82
|
+
2. **Create + capture** — one command, deterministic:
|
|
83
|
+
```bash
|
|
84
|
+
zond request POST /v1/accounts --api <name> \
|
|
85
|
+
--body '{"type":"custom","country":"US","capabilities":{...}}' \
|
|
86
|
+
--json-path id --capture account
|
|
87
|
+
```
|
|
88
|
+
On 2xx zond writes `account=<id>` into `apis/<name>/.env.yaml` (with a
|
|
89
|
+
`.bak` backup) and prints `captured account=<id>`. On non-2xx it writes
|
|
90
|
+
nothing and prints `--capture account skipped — … status=400 …`.
|
|
91
|
+
|
|
92
|
+
3. **Feedback on failure.** Read the response body. Common fixes:
|
|
93
|
+
- "Missing required field X" → add X.
|
|
94
|
+
- "You may only specify one of A, B" → drop one (prose exclusion).
|
|
95
|
+
- "Invalid <field>" / 422 on an id → the value is external-input; stop
|
|
96
|
+
retrying this root and mark it un-seedable.
|
|
97
|
+
Revise the `--body`, retry. Cap ~3 attempts.
|
|
98
|
+
|
|
99
|
+
4. **Cascade.** Once a parent id is captured, its children unblock — their
|
|
100
|
+
`{{parent_var}}` now resolves. Walk down the dependency chain.
|
|
101
|
+
|
|
102
|
+
5. **Verify.** Re-run `zond prepare-fixtures --api <name> --json` — the roots
|
|
103
|
+
you seeded should drop out of `unseededRoots`.
|
|
104
|
+
|
|
105
|
+
## Cleanup & report
|
|
106
|
+
|
|
107
|
+
- DELETE every id you created (reverse order — children first), or confirm the
|
|
108
|
+
seed account is disposable.
|
|
109
|
+
- Report: **seeded** (var → id), **un-seedable** (var → reason: external input
|
|
110
|
+
/ gated / no create endpoint). The un-seedable list is the honest ceiling —
|
|
111
|
+
hand it to the user; do not paper over it with invented values.
|
|
112
|
+
|
|
113
|
+
Hand back to `zond` for the full audit once depth is unblocked. Measure the
|
|
114
|
+
lift: test coverage before vs after seeding.
|