@lariocpt/crust-mongo 0.0.0-stage → 0.3.5

package/skills/crust-mock-server/SKILL.md

@@ -0,0 +1,138 @@
+---
+name: crust-mock-server
+description: Mock an API from an OpenAPI spec with crust mock-server — example-first responses, stateful CRUD (in-memory or persisted to sqlite/postgres with --state, seedable with --seed), request validation with 422s, and validation-proxy mode that checks a REAL API's spec conformance. Use for mocking backends, frontend dev without a server, contract/conformance testing, or checking whether an API matches its openapi.json.
+---
+
+# crust mock-server
+
+One builtin, four modes, all driven by an OpenAPI 3.x spec (Swagger 2.0
+auto-converts; URL or local json/yaml):
+
+```
+mock-server ./openapi.json -p4000                      # example-first mock
+mock-server ./openapi.json -p4000 --stateful           # + in-memory CRUD
+mock-server ./openapi.json -p4000 --state ./m.sqlite   # + persistent CRUD (implies --stateful)
+mock-server ./openapi.json -p4000 --validate           # + reject bad requests (422)
+mock-server ./openapi.json -p4747 --proxy http://localhost:3001   # conformance proxy
+```
+
+Builtins can't be piped — run the server in one terminal (or `procs()`), and
+pipe against it from another line.
+
+## Mock + stateful
+
+Responses prefer the spec's `example`/`examples`, then synthesize from the
+schema (enum[0], format-aware strings, nested objects). Status pick:
+200 → 201 → other 2xx → `default`. Unknown path → 404; known path, wrong
+method → 405.
+
+`--stateful` pairs `/things` with `/things/{id}` automatically: POST stores
+(id from body or a UUID), GET lists what you stored (shaped to the spec's
+envelope), GET/{id} → 404 for unknown ids, PUT/PATCH shallow-merge, DELETE
+→ 204. Untouched collections keep serving spec examples. Default is
+per-process memory; `--state` persists (below).
+
+**Entity envelopes round-trip.** If the POST's 201 example wraps the entity
+(`{"thing": {"id": …}}` — one object-valued prop, no top-level id), the mock
+detects it: responses are wrapped with the real stored entity inside, request
+bodies wrapped the same way (per the POST's request example) are unwrapped,
+and the store keeps the BARE entity. So capturing a wrapped id works against
+the mock:
+
+```crust
+{"thing": {"name": "x"}} | POST :4000/api/things | assert (r => r.status === 201) | (r => r.json()) | capture TID (t => t.thing.id)
+GET :4000/api/things/$TID | expect 200
+```
+
+Flat specs capture `t.id` exactly as before. Ambiguous shapes (two
+object-valued props, a top-level `id`) stay flat — deterministic, never
+guessed.
+
+## Persistent state (`--state`) and seeding (`--seed`)
+
+Both imply `--stateful` and exclude `--proxy`.
+
+- `--state <path|url>` — bare path or `sqlite://` file, or
+  `postgres://`/`postgresql://` URL (other schemes exit 2). Survives
+  restarts; shared cross-process (reads hit the DB every request, writes are
+  single-statement upserts, last write wins). Boot line shows
+  `(state: sqlite)` / `(state: postgres)`.
+- `--seed <file.json>` — `{ "/api/things": [ {…}, … ] }`; items without an
+  `id` get a uuid. EMPTY collections only: re-booting over a persistent
+  store never duplicates or clobbers. Unknown collection keys warn and boot
+  anyway; invalid file exits 1. Boot line shows `(seeded N)`.
+
+Table contract (created idempotently; assert on it from pipelines):
+`crust_mock_state(collection TEXT, id TEXT, doc TEXT|JSONB, updated_at, PRIMARY KEY (collection, id))`
+where `doc` is the BARE entity as JSON. With the `sql` builtin
+(`DATABASE_URL=sqlite://./m.sqlite` works — Bun.SQL speaks sqlite URLs;
+rows stream one item each):
+
+```crust
+sql "SELECT json_extract(doc, '$.name') AS name FROM crust_mock_state WHERE collection = '/api/things' AND id = '$TID'" | assert (r => r.name === "x")
+```
+
+(postgres: `doc->>'name'` instead of `json_extract`.)
+
+## Failure injection (`PUT /__crust/override`) — mock mode only
+
+Arm the mock to fail when the APP owns the request (no test-controlled
+header possible): `PUT /__crust/override` with `{method, path, status,
+body?, contentType?, times?, match?: {bodyContains}}`. Under `-j/--threads`
+ALWAYS scope with `match.bodyContains` (a unique value the fixture put in
+its own request body) + `times: 1` — unscoped overrides race sibling
+fixtures. An armed override beats --validate/--stateful; `GET` lists,
+`DELETE` clears. Never consulted in --proxy mode.
+
+## Request validation (`--validate`)
+
+Spec-violating requests get **422** (not the API's own 400 — deliberately
+distinguishable) with header `x-crust-validation: request` and body
+`{error, violations: [{pointer, rule, message, expected?, received?, location}]}`.
+Checks: JSON body (type/required/enum/format/pattern/lengths/ranges,
+anyOf/oneOf/allOf), path + query params (coerced by declared type). The
+governing rule: a schema the validator can't judge PASSES — it never
+invents a violation (notably `additionalProperties: false` is not
+enforced unless you add `--strict`). Composes with `--stateful`: an
+invalid POST creates nothing.
+
+`--strict` (implies `--validate`; composes with `--proxy`, both
+directions) enforces a LITERAL `additionalProperties: false` at plain
+object nodes only: allOf-merged objects, nodes with combinator siblings,
+and patternProperties stay exempt — so it still never invents a
+violation for spec composition idioms. Inside anyOf/oneOf branches an
+extra key just fails that branch's selection.
+
+## Validation proxy (`--proxy <upstream>`) — spec-conformance testing
+
+Forwards every request to the real upstream, returns the response
+UNTOUCHED, and records spec violations out-of-band in BOTH directions:
+request violations, undocumented response statuses, undocumented
+content-types, response-schema mismatches (only where the spec has response
+schemas), and `undocumented-operation` for paths the spec doesn't know
+(still forwarded — a browsing frontend keeps working). Upstream down → 502,
+not a violation. Mutually exclusive with `--stateful`.
+
+The findings are pipeable — this is the conformance gate:
+
+```crust
+GET :4747/__crust/violations | (r => r.json()) | (v => v.violations.filter(x => x.direction === "response")) | assert (a => a.length === 0)
+```
+
+The `(r => r.json())` stage is not optional: an HTTP stage hands you the
+response object, so reading `.violations` off it straight away is `undefined`
+and the gate dies with `crust: undefined is not an object (evaluating 'v.violations.filter')` — which looks like a broken assert rather than a missing decode.
+
+`DELETE /__crust/violations` clears between runs; `--report file.ndjson`
+appends every violation as NDJSON for CI artifacts (requires `--proxy`).
+
+Workflow: point your existing test suite's base URL at the proxy port, run
+the suite, then assert zero response-direction violations. (Request-direction
+violations are EXPECTED if your suite includes negative cases — filter by
+direction.)
+
+## Exit codes
+
+0 clean shutdown (SIGINT/SIGTERM) · 1 runtime failure (spec load, bad seed
+file) · 2 bad args (including `--proxy` + `--stateful`/`--state`/`--seed`,
+an unsupported `--state` scheme, or `--report` without `--proxy`).

package/skills/crust-pipelines/SKILL.md

@@ -0,0 +1,161 @@
+---
+name: crust-pipelines
+description: Compose crust pipelines — sources (range, glob, read, tail, GET, load, procs, sql), transforms (TS lambdas, filter, HTTP verbs, parallel), assertions (expect, assert), and request chaining with capture. Use when writing or debugging crust one-liners, `crust -c` scripts, .crust script files, or .pipes lines that mix HTTP, SQL, and processes.
+---
+
+# crust pipelines
+
+crust is a Bun-powered pipeline runner where every command is a typed stream
+(`Pipeline<T>`). Stages compose under `|`. Plain shell still works — a line
+whose stages are all ordinary commands is handed to `sh -c` untouched.
+
+## How a stage is classified (first token decides)
+
+| You type | You get |
+|---|---|
+| `range(0, 9)` | numbers 0..9 inclusive, one item each |
+| `**/*.ts` | glob source — file paths |
+| `lines **/*.log` | one item per LINE across every match (sorted; zero matches = error). Use this for log mining — `read` gives whole files. Bare `lines` mid-pipeline splits upstream items |
+| `read fixtures/*.json` | whole-file contents, one item per file (sorted; zero matches = error) — the request-body form: `read fixtures/*.json \| POST …` |
+| `tail -F app.log` | native tail source (`-n N`, `-F` follow); the `-n N` cut is a bounded backward read, safe on huge files |
+| `stdin` (alias `-`) | piped-stdin source, one item per line — `docker logs -f X \| crust -c 'stdin \| …'`. Source position only. Bare `cmd \| crust` treats stdin as a SCRIPT, so data pipes need `-c` |
+| `grep ERROR` mid-pipeline | native line-buffered grep (`-i`/`-v`/`-F`, ONE pattern; per-line even on multi-line items) — follow streams don't stall on grep's 4KB pipe buffer. Combined/unknown flags, two positionals, any `$`, or GNU/JS-divergent regex bits (`\`, `[[:`, `{,`, `(?`) = exact system grep via sh; first-stage grep = file grep via sh |
+| `load 30s 100/s` | paced load ticks (see the crust-load-testing skill) |
+| `{"name": "x"}` | JSON-literal source — ONE parsed item (the request body). Invalid JSON is a hard error |
+| `GET :3000/path` | HTTP. First stage: one `Response`. Mid-pipeline: per-item timed `{status, ms, url}` |
+| `POST/PUT/PATCH/DELETE url` | per-item request; upstream item = body (objects auto-JSON) |
+| `GET url --timeout 2s` | per-request timeout on any http stage (`ms`/`s`/`m`); timed paths yield `{status: 0, timedOut: true}`, plain verbs fail the pipeline; unknown `--flags` are errors |
+| `GET url --redirect error\|manual\|follow` | 3xx handling (fetch's `redirect` init): `error` refuses a redirect — plain verbs fail the line, timed paths yield `{status: 0, redirectRefused: true}`; `manual` yields the 3xx so `expect 301` asserts it; `follow` is the default. Fixtures carry it as `input.redirect` |
+| `(x => x * 2)` | TypeScript lambda, per item, async ok — MAPS, never drops (falsy results are emitted as-is) |
+| `filter (l => l.ok)` | keeps items whose predicate is truthy; falsy (`0`, `""`, `null`…) drop; async ok; empty result passes |
+| `assert (r => r.ok)` | falsy FAILS the pipeline; empty upstream also fails |
+| `capture NAME (r => r.id)` | write value to `process.env.NAME` for later lines |
+| `expect 201` / `expect 2xx` | status assertion, counts mismatches, fails at drain |
+| `stats [--every N] [--out f.json]` | summary: count/rps/status histogram/p50/p95/p99 + failure tallies `timedOut`/`redirectRefused` (records, not exits — gate them) |
+| `parallel N` | modifier: fan-out for the NEXT stage (http, lambda, or fn — anything else errors) |
+| `sql "SELECT …" ?-params` | registered fn — `?` is the bind placeholder; rows stream one item each (needs `$DATABASE_URL`) |
+| `range(1,1) \| sql "… WHERE id = ?"` | mid-pipeline, the upstream item BINDS as the first parameter when the line declares none (an explicit param wins); rows stream one item each in both positions |
+| anything else | plain shell via `sh -c` |
+
+`:3000/path` expands to `http://localhost:3000/path` everywhere.
+
+## Env expansion — parse time, opt-in per position
+
+`$VAR`/`${VAR}` expand from `process.env` in: URLs, whole `-H` header
+strings, JSON literals, `stats --out` paths, and registered-fn args. They do
+NOT expand inside lambda/`assert`/`capture` bodies (those are JS — use
+`process.env.VAR`) or shell stages (sh does its own). In a script, the
+**positional arguments** expand in those same positions — so `$1` inside a
+`sql "…"` string is the script's `$1`, not a SQL bind: the bound-parameter
+placeholder is `?`, and the trailing quoted args DO env-expand:
+`sql "… WHERE id = ?" "$MY_ID"`.
+
+`$(…)` command substitution lives in shell stages only: sh expands it once
+at the head of a shell stage, and a `$` anywhere in a mid-pipeline stage
+demotes it to sh, so `range(1,3) | grep $(echo 1)` still works. A `$(…)` in
+a *crust* stage (a URL, a lambda, an argument) is **refused before anything
+runs** — it used to build data from the literal text (`getaddrinfo ENOTFOUND
+$(echo`) — so the named route is capture first: `date +%s | capture TS`,
+then `GET :3000/?t=$TS`.
+
+Inside a parenthesized lambda/`assert`/`capture` body, anything goes: the
+tokenizer tracks paren depth and quotes, so literal `|`, `??`, single/double
+quotes, and backticks (template literals) are all safe. In double-quoted
+strings and JSON literals, `\"` escapes work.
+
+## Request chaining with capture
+
+Each line parses right before it runs, so a capture on one line feeds `$VAR`
+on every later line — the chaining contract for REPL, `-c` scripts, and
+`.pipes` files:
+
+```crust
+{"name": "Court"} | POST $BASE/api/buildings -H "authorization: Bearer $TOKEN" | assert (r => r.status === 201) | (r => r.json()) | capture BID (b => b.building.id)
+GET $BASE/api/buildings/$BID -H "authorization: Bearer $TOKEN" | expect 200
+```
+
+- Last item wins; omit the lambda to capture the item itself (objects are
+  JSON-stringified).
+- A nullish captured value or empty upstream fails IMMEDIATELY — a typo'd
+  accessor dies on the capture line, not three lines later as a baffling
+  `""` expansion.
+- Don't capture into `TOKEN`, `BASE`, or `PATH` unless you mean to overwrite
+  them.
+- Capture is also the named route for command output you want downstream:
+  a `$(…)` in a crust stage is refused, so `date +%s | capture TS` followed
+  by `GET :3000/?t=$TS` is the round trip (sh's trailing newline is trimmed
+  off the captured value).
+
+## Verified one-liner patterns
+
+```crust
+# log mining — `lines` yields one item per LINE, which is what filter wants
+# (`read` yields whole FILES, so a filter would test the entire file at once)
+lines **/*.log | grep ERROR | filter (l => !l.includes('healthcheck')) | wc -l
+
+# lambda + shell mixing — items cross as lines (objects as JSON)
+range(1, 5) | (n => n * n) | sort -rn
+
+# HTTP fixture in one line: body | request | status | parsed-body assert
+{"name": "x"} | POST :3000/users | assert (r => r.status === 201) | (r => r.json()) | assert (u => u.id > 0)
+
+# DB verification — rows stream, assert sees each row object
+sql "SELECT count(*)::int AS c FROM users WHERE email = ?" "a@b.c" | assert (r => r.c === 1)
+
+# fan out any per-item work
+read fixtures/*.json | parallel 8 | POST :3000/users | expect 2xx
+
+# follow a log live — native grep emits each match immediately
+tail -n 0 -F app.log | grep ERROR
+
+# any command's output as a source (run under `crust -c`, data piped in)
+stdin | (l => JSON.parse(l)) | filter (e => e.level >= 40) | (e => e.msg)
+```
+
+To ITERATE on filters over one held live stream, use the `logs` builtin
+(`logs tail -n 0 -F app.log`, `logs procs({…})`, `logs docker logs -f c`):
+every line typed at its prompt is a fragment like the ones above, run over
+a buffer of the recent past and then live; Ctrl-C once ends the view and
+flushes terminal stages (bare `stats` prints there), `exit` leaves.
+
+## Traps
+
+- A plain lambda MAPS — `(l => l.includes('ERROR'))` emits `true`/`false`
+  per item, and `(x => cond ? x : null)` prints literal `null` lines. To
+  drop items, use `filter (l => l.includes('ERROR'))`.
+- `filter` passes an empty stream through silently; `assert` fails on EMPTY
+  upstream ("no items reached") — deliberate, it closes the
+  sql-returned-zero-rows silent pass. `expect` does not fail on empty either.
+- `parallel` streams results in COMPLETION order, not input order.
+- `parallel N` before anything except an http verb, lambda, or registered
+  fn is a parse error; so is a trailing `parallel N`.
+- A glob source yields PATHS; `read <glob>` yields file CONTENTS. `POST`ing
+  a glob posts path strings.
+- Builtins (`test-pipes`, `mock-server`, `logs`, …) cannot be piped. A builtin
+  name in a shell context — `mock-server spec.json | grep ok`, `range(0,2) |
+  test-fixture a.ts` — refuses naming the tool instead of letting sh answer
+  `command not found` for a tool crust runs in-process. Put the builtin on its
+  own line; `logs` takes its filters at the `logs>` prompt.
+- No redirect on a crust stage. `<` / `>` mean something only on a SHELL
+  stage, whose text goes to `sh` verbatim: `… | cat > out.json` saves a
+  pipeline, `lines <path>` / `read <path>` feeds a file in. Anywhere else the
+  line is refused — an http stage used to swallow the operator (request ran,
+  nothing written, exit 0). A builtin carrying one refuses too, and the fix is
+  redirecting the whole call: `crust -c 'mock-server spec.json' > mock.log`.
+- Native grep patterns are JS regexes: quoted `'a|b'` ALTERNATES (ERE)
+  where BRE grep matched the literal — use `grep -F 'a|b'` for the literal.
+- `stdin` is single-shot per process: inside a bare `cmd | crust` script it
+  errors (the pipe was already consumed as the script itself).
+- Shell stages get npm-run-style PATH: ancestor `node_modules/.bin` dirs
+  are prepended, so local tool binaries work bare (`| pino-pretty`,
+  `| tsc --noEmit`) — no `node_modules/.bin/` prefix, no global install.
+- In `… | expect 200 | stats`, a failing expect throws at drain BEFORE stats
+  emits — gate status inside a stats assert instead if you need the summary.
+
+## Exit codes
+
+Any stage throw → `crust: <message>` on stderr, line exits 1. `crust -c`,
+`crust file.crust`, and piped stdin all run newline-separated lines through
+the same fail-fast loop and STOP at the first non-zero — put warmup and
+cleanup on their own lines. Blank lines and `#` comments are skipped, so a
+`#!/usr/bin/env crust` shebang works in `.crust` files.

package/skills/crust-procs/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: crust-procs
+description: Run and supervise dev processes with crust — procs() merges multiple long-lived commands into one pipeable stream with per-proc env, restart backoff, readiness probes (ready:), dependency ordering (after:), and process-group teardown; wait blocks until a port/URL is up. Use for dev runners ("start db, api, web in order"), readiness gating in CI, tailing multiple services, or mining merged process logs.
+---
+
+# crust procs & wait
+
+## procs() — the one-tail dev runner
+
+```crust
+procs({web: "bun run dev", api: "bun api.ts"}) | (l => l.proc + " | " + l.line)
+```
+
+Every stdout/stderr line becomes `{proc, stream, line}` — a PIPELINE ITEM,
+not text. Grep it, transform it, count it. Streams: `stdout`, `stderr`,
+`exit`, `ready`, `live`.
+
+Full spec form:
+
+```crust
+procs({ db: {cmd: "docker compose up pg", ready: "port:5432"}, api: {cmd: "bun api.ts", after: "db", ready: ":3001/health", live: {url: ":3001/health", failures: 3}, restart: {max: 3}}, web: {cmd: "bun run dev", after: "api", env: {PORT: "3001"}} })
+```
+
+- `env` — per-proc overlay on the inherited environment.
+- `restart: true` — respawn on unexpected exit (backoff 250ms→2s; >10s
+  uptime while READY resets it — a proc that hangs un-ready until its
+  ready-timeout kill never does, and with `live:` uptime ends when a fatal
+  probe streak BEGAN, so a proc that wedges right after ready still accrues
+  strikes). `restart: {max: N}` gives up after N consecutive restarts.
+  Ctrl-C never respawns. **Giving up is not a failure**: `giving up after N
+  restart(s)` is an `exit`-stream line and the line still exits 0 (same for a
+  child that exits nonzero with no `restart:` — its stream just ends). Gate it
+  in CI: `| (l => { if (/giving up|unhealthy/.test(l.line)) throw new Error(l.line); return null })`.
+- `ready:` — `":3001/health"` / `"http(s)://…"` (any 2xx) or `"port:5432"`
+  (TCP connect). Long form `{url?, port?, timeoutMs?, intervalMs?,
+  probeTimeoutMs?}` (defaults 30s / 250ms; each probe capped at
+  `min(intervalMs*4, 2s)` unless `probeTimeoutMs` raises it). `timeoutMs`
+  belongs INSIDE the ready object: as a sibling of `cmd` it is ignored
+  silently — `{cmd: "sleep 40", ready: "port:4199", timeoutMs: 1200}` waits
+  the full default 30s. Timeout: a restartable proc is killed and respawned
+  (readiness re-awaited each spawn); a non-restartable one FAILS the whole
+  pipeline — CI semantics.
+- `live:` — liveness for the unhealthy-but-alive case. Same target forms;
+  long form `{url?, port?, intervalMs?, probeTimeoutMs?, failures?,
+  graceMs?}` (defaults 5s / 3 failures / 0 grace). Use `url:` here: a
+  `port:` probe is a TCP connect, so it cannot see a wedged listener — a
+  server answering 503 for four seconds with `live: {port, failures: 3}`
+  logged nothing and the run exited 0. Arms once the proc is up
+  (after `ready:`, else at spawn); `live`-stream lines: `probe failed
+  (k/N)`, `recovered after k failed probe(s)`, `unhealthy after N
+  consecutive failed probe(s)`. Fatal streak: restartable → killed +
+  respawned (re-arms after next ready); non-restartable → FAILS the
+  pipeline, like a ready-timeout.
+- `after:` — name or list; the proc spawns only once its dependencies are
+  READY (no `ready:` on the dep = ready at spawn). One-shot gate; a dep
+  that dies before ever becoming ready fails the pipeline. Unknown names /
+  self-deps / cycles throw before anything spawns.
+- Teardown: children run in their own process groups; SIGTERM to all, 3s
+  grace, then SIGKILL — grandchildren from `sh -c` don't survive.
+
+## Log mining on the merged stream
+
+```crust
+procs({api: "bun api.ts", worker: "bun worker.ts"}) | (l => { try { return JSON.parse(l.line) } catch { return { level: 30, msg: l.line, proc: l.proc } } }) | filter (e => e.level >= 40) | (e => "WARN+ " + (e.msg ?? ""))
+```
+
+(`filter` drops the sub-WARN entries; a plain lambda with `: null` would
+print a literal `null` line for every one of them.)
+
+Objects printed to stdout are JSON lines — pipe the whole thing into
+`pino-pretty` via a shell stage, or count errors with `grep`/`wc`.
+
+To iterate on these filters WITHOUT restarting the dev stack, hold it in a
+`logs` session instead: `logs procs({api: "bun api.ts", worker: "bun
+worker.ts"})` keeps the group running and buffers its recent output; each
+line you type (`filter (l => l.proc === "api")`, `(l => l.line) | grep -i
+error`) runs over the buffer, then live until Ctrl-C. Items stay `{proc,
+stream, line}` objects. `exit`/Ctrl-D tears the group down (SIGTERM→SIGKILL).
+
+## wait — readiness as a one-liner (CI's friend)
+
+```crust
+wait :3001/api/health --timeout 40s --interval 2s
+```
+
+Blocks until the target answers (URL = any 2xx; `port:5432` = TCP connect),
+then emits `{target, ready: true, ms, attempts}` and exits 0. Not ready in
+time → error, exit 1. Durations: `300ms`, `30s`, `2m` (defaults 30s/500ms).
+`--probe-timeout 5s` raises the per-probe cap (default `min(interval*4, 2s)`)
+for targets that take longer than that to first byte.
+Works in the REPL, `crust -c` (replaces hand-rolled curl-sleep retry loops
+in CI), and `.pipes` files. Env-expanded args work:
+`wait "$BASE_URL/health" --timeout 40s`.
+
+## Traps
+
+- procs is source-only (first stage). Spec keys are JS object-literal keys, so
+  a duplicate collapses BEFORE crust sees it (`{a: "x", a: "y"}` runs only
+  `y`) — nothing warns you.
+- The merged stream never ends while a `restart: true` child keeps
+  respawning — consume it with `for await`/pipe, bound it with a downstream
+  condition, or stop it with Ctrl-C: at the REPL that cancels the running
+  line and tears the whole process group down (SIGTERM, then SIGKILL).
+- `FORCE_COLOR=0` is forced on children so lines stay parseable.
+- To capture the stream, END with a shell stage: `procs({…}) | cat > file`
+  writes each item as its JSON line. A bare `procs(...) > file` is refused —
+  crust has no redirect on its own stages (the `>` would be read as data by a
+  pipeline stage); the outer shell can still wrap the whole invocation.

package/skills/crust-verify-web-links/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: crust-verify-web-links
+description: Crawl a site from its sitemap and verify every link, #anchor, redirect chain, OG image and meta tag against .crust.ts fixtures. `--base-url` auto-discovers the sitemap (robots.txt → /sitemap.xml → /sitemap-index.xml), `--site-map-url` takes a local sitemap so you can check build output, `--fixtures` diffs meta. Use when link or SEO health must gate a deploy, when a built site should be checked before publishing, or when OG/meta regressions need catching.
+---
+
+# crust verify-web-links
+
+## What it verifies
+
+| Check | Fails when |
+| --- | --- |
+| internal + external links | any status that is not 2xx |
+| `#fragment` targets | the id/name is not on the fetched page |
+| redirect chains | a link 301s/302s somewhere (informational with `--no-redirect-warnings`) |
+| OG / twitter images | the image URL is unreachable |
+| meta vs `--fixtures` | a keyed value differs, or a predicate returns falsy |
+
+Exit codes: `0` all clear, `1` verification failures, `2` bad args or an
+unreachable sitemap. With `--strict`, `1` also covers a run that finished
+having left discovered links unchecked. `2` also covers `--site-map-url`
+together with `--base-url`, neither of them given, `no fixture files matched
+<glob>`, `--concurrency must be >= 1`, and `unexpected argument`.
+
+## Getting it pointed at the right thing
+
+```crust
+verify-web-links --base-url https://example.com
+verify-web-links --site-map-url https://example.com/sitemap.xml
+verify-web-links --site-map-url ./dist/sitemap.xml --no-recurse
+verify-web-links --base-url https://staging.example.com --fixtures site/*.meta.crust.ts
+```
+
+`--base-url` probes `robots.txt` for `Sitemap:` lines, then `/sitemap.xml`, then
+`/sitemap-index.xml`; when none answer it names all three. `--site-map-url`
+(aliased `--sitemap`) also takes a **local path**, which is the CI move: crawl
+the build output rather than a deployed site. With `--no-recurse` only the URLs
+in the sitemap are fetched.
+
+## Green does not mean checked
+
+Three ways to get exit 0 with nothing actually verified. Read the note lines and
+`totals` before believing a green run:
+
+| Symptom | What it means |
+| --- | --- |
+| `N page(s) had to be recognised from their body` | the server sends HTML without saying `text/html`. crust followed the links anyway, but the mislabel is the site's bug |
+| `N page(s) NOT read as HTML` | a sitemap entry or `<a>` target that is neither declared HTML nor smells like it (a JSON endpoint). Its links were **not** followed and the crawl ended there |
+| a note naming a count (`anchors skipped…`, `stopped at --max-pages…`) | that work did not happen. `--no-recurse` and `--no-anchors` produce it deliberately |
+| `"parsed": false` on a page in `--json` | no links and no ids were extracted from that page |
+
+`totals.complete` is the one field that means "nothing was left unchecked":
+false on any of the above. `--max-pages N` is a safety valve, not a fast mode —
+the crawl stops early, the unchecked count is reported, and the exit code still
+says 0 without `--strict`. Treat a `complete: false` run as inconclusive.
+
+## In CI
+
+**`--strict` is the CI flag.** It exits 1 when the run finished having left
+discovered links unchecked — a `--max-pages` cut-off, an unchecked `#fragment`,
+a page crust could not read — so a bounded crawl cannot pass as a clean one:
+
+```crust
+verify-web-links --base-url https://example.com --strict --max-pages 500
+```
+
+A builtin runs in-process and cannot sit mid-pipeline —
+`verify-web-links … | assert (…)` is refused with *"give it a line of its own"*.
+Redirect the whole invocation instead, then judge the report:
+
+```bash
+crust -c 'verify-web-links --base-url https://example.com --json --no-progress' > report.json
+```
+
+```crust
+read report.json | (s => JSON.parse(s)) | assert (r => r.totals.failures === 0)
+read report.json | (s => JSON.parse(s)) | assert (r => r.totals.complete)
+```
+
+`read` yields each file's **contents as one item**, so `JSON.parse` receives the
+whole document. The first line fails on a real failure; the second catches every
+inconclusive case at once — `complete` is false if anything was dropped, any
+`#fragment` went unchecked, or any page could not be read.
+
+## Meta fixtures
+
+```ts
+export default {
+  url: "https://example.com/about",
+  meta: {
+    title: "About Us",
+    description: (d: string) => d.length > 50 && d.length < 160,
+    "og:image": (u: string) => u.endsWith(".png") || u.endsWith(".jpg"),
+    "twitter:card": "summary_large_image",
+  },
+};
+```
+
+A value is compared for equality; a function is a predicate over the extracted
+value. A key a page does not have is **asked about**, not ignored — so asserting
+`og:image` on a page that lost it is a failure, which is the point.
+
+Fixture `url`s match **URL-normalized**: scheme+host case and default ports
+don't matter, `https://x.com` and `https://x.com/` are the same page, but a
+trailing slash on a non-root path stays significant (`/login/` ≠ `/login`). A
+fixture whose url never appears in the crawl fails as `meta-fixture-no-page` —
+a fixture for a page you deleted is a failure, not a skip.
+
+## Tuning
+
+```crust
+verify-web-links --base-url https://example.com --exclude /checkout/ --exclude /go/
+verify-web-links --base-url https://example.com --include-external --max-depth 2
+verify-web-links --base-url https://example.com --concurrency 16 --timeout 3000
+```
+
+- `--exclude <substring>` is repeatable. Use it for subtrees that redirect **by
+  design** (short-link senders, locale switches); redirect chains are failures
+  otherwise. `--no-redirect-warnings` silences them globally — only when you
+  mean it.
+- Off-origin links are not queued at all without `--include-external`, and even
+  then they are never recursed into.
+- `--max-depth` (default 5) bounds the crawl, `--concurrency` (default 4) and
+  `--timeout` (default 10000 ms) bound each fetch.
+- Anchors come from the crawl: with `--no-recurse` the internal links found on
+  sitemap pages are not fetched, so their `#fragments` are not checked — the
+  report says how many were skipped.