@lariocpt/crust-mongo 0.0.0-stage → 0.3.6

package/skills/crust-procs/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: crust-procs
+description: Run and supervise dev processes with crust — procs() merges multiple long-lived commands into one pipeable stream with per-proc env, restart backoff, readiness probes (ready:), dependency ordering (after:), and process-group teardown; wait blocks until a port/URL is up. Use for dev runners ("start db, api, web in order"), readiness gating in CI, tailing multiple services, or mining merged process logs.
+---
+
+# crust procs & wait
+
+## procs() — the one-tail dev runner
+
+```crust
+procs({web: "bun run dev", api: "bun api.ts"}) | (l => l.proc + " | " + l.line)
+```
+
+Every stdout/stderr line becomes `{proc, stream, line}` — a PIPELINE ITEM,
+not text. Grep it, transform it, count it. Streams: `stdout`, `stderr`,
+`exit`, `ready`, `live`.
+
+Full spec form:
+
+```crust
+procs({ db: {cmd: "docker compose up pg", ready: "port:5432"}, api: {cmd: "bun api.ts", after: "db", ready: ":3001/health", live: {url: ":3001/health", failures: 3}, restart: {max: 3}}, web: {cmd: "bun run dev", after: "api", env: {PORT: "3001"}} })
+```
+
+- `env` — per-proc overlay on the inherited environment.
+- `restart: true` — respawn on unexpected exit (backoff 250ms→2s; >10s
+  uptime while READY resets it — a proc that hangs un-ready until its
+  ready-timeout kill never does, and with `live:` uptime ends when a fatal
+  probe streak BEGAN, so a proc that wedges right after ready still accrues
+  strikes). `restart: {max: N}` gives up after N consecutive restarts.
+  Ctrl-C never respawns. **Giving up is not a failure**: `giving up after N
+  restart(s)` is an `exit`-stream line and the line still exits 0 (same for a
+  child that exits nonzero with no `restart:` — its stream just ends). Gate it
+  in CI: `| (l => { if (/giving up|unhealthy/.test(l.line)) throw new Error(l.line); return null })`.
+- `ready:` — `":3001/health"` / `"http(s)://…"` (any 2xx) or `"port:5432"`
+  (TCP connect). Long form `{url?, port?, timeoutMs?, intervalMs?,
+  probeTimeoutMs?}` (defaults 30s / 250ms; each probe capped at
+  `min(intervalMs*4, 2s)` unless `probeTimeoutMs` raises it). `timeoutMs`
+  belongs INSIDE the ready object: as a sibling of `cmd` it is ignored
+  silently — `{cmd: "sleep 40", ready: "port:4199", timeoutMs: 1200}` waits
+  the full default 30s. Timeout: a restartable proc is killed and respawned
+  (readiness re-awaited each spawn); a non-restartable one FAILS the whole
+  pipeline — CI semantics.
+- `live:` — liveness for the unhealthy-but-alive case. Same target forms;
+  long form `{url?, port?, intervalMs?, probeTimeoutMs?, failures?,
+  graceMs?}` (defaults 5s / 3 failures / 0 grace). Use `url:` here: a
+  `port:` probe is a TCP connect, so it cannot see a wedged listener — a
+  server answering 503 for four seconds with `live: {port, failures: 3}`
+  logged nothing and the run exited 0. Arms once the proc is up
+  (after `ready:`, else at spawn); `live`-stream lines: `probe failed
+  (k/N)`, `recovered after k failed probe(s)`, `unhealthy after N
+  consecutive failed probe(s)`. Fatal streak: restartable → killed +
+  respawned (re-arms after next ready); non-restartable → FAILS the
+  pipeline, like a ready-timeout.
+- `after:` — name or list; the proc spawns only once its dependencies are
+  READY (no `ready:` on the dep = ready at spawn). One-shot gate; a dep
+  that dies before ever becoming ready fails the pipeline. Unknown names /
+  self-deps / cycles throw before anything spawns.
+- Teardown: children run in their own process groups; SIGTERM to all, 3s
+  grace, then SIGKILL — grandchildren from `sh -c` don't survive.
+
+## Log mining on the merged stream
+
+```crust
+procs({api: "bun api.ts", worker: "bun worker.ts"}) | (l => { try { return JSON.parse(l.line) } catch { return { level: 30, msg: l.line, proc: l.proc } } }) | filter (e => e.level >= 40) | (e => "WARN+ " + (e.msg ?? ""))
+```
+
+(`filter` drops the sub-WARN entries; a plain lambda with `: null` would
+print a literal `null` line for every one of them.)
+
+Objects printed to stdout are JSON lines — pipe the whole thing into
+`pino-pretty` via a shell stage, or count errors with `grep`/`wc`.
+
+To iterate on these filters WITHOUT restarting the dev stack, hold it in a
+`logs` session instead: `logs procs({api: "bun api.ts", worker: "bun
+worker.ts"})` keeps the group running and buffers its recent output; each
+line you type (`filter (l => l.proc === "api")`, `(l => l.line) | grep -i
+error`) runs over the buffer, then live until Ctrl-C. Items stay `{proc,
+stream, line}` objects. `exit`/Ctrl-D tears the group down (SIGTERM→SIGKILL).
+
+## wait — readiness as a one-liner (CI's friend)
+
+```crust
+wait :3001/api/health --timeout 40s --interval 2s
+```
+
+Blocks until the target answers (URL = any 2xx; `port:5432` = TCP connect),
+then emits `{target, ready: true, ms, attempts}` and exits 0. Not ready in
+time → error, exit 1. Durations: `300ms`, `30s`, `2m` (defaults 30s/500ms).
+`--probe-timeout 5s` raises the per-probe cap (default `min(interval*4, 2s)`)
+for targets that take longer than that to first byte.
+Works in the REPL, `crust -c` (replaces hand-rolled curl-sleep retry loops
+in CI), and `.pipes` files. Env-expanded args work:
+`wait "$BASE_URL/health" --timeout 40s`.
+
+## Traps
+
+- procs is source-only (first stage). Spec keys are JS object-literal keys, so
+  a duplicate collapses BEFORE crust sees it (`{a: "x", a: "y"}` runs only
+  `y`) — nothing warns you.
+- The merged stream never ends while a `restart: true` child keeps
+  respawning — consume it with `for await`/pipe, bound it with a downstream
+  condition, or stop it with Ctrl-C: at the REPL that cancels the running
+  line and tears the whole process group down (SIGTERM, then SIGKILL).
+- `FORCE_COLOR=0` is forced on children so lines stay parseable.
+- To capture the stream, END with a shell stage: `procs({…}) | cat > file`
+  writes each item as its JSON line. A bare `procs(...) > file` is refused —
+  crust has no redirect on its own stages (the `>` would be read as data by a
+  pipeline stage); the outer shell can still wrap the whole invocation.

package/skills/crust-verify-web-links/SKILL.md

@@ -0,0 +1,160 @@
+---
+name: crust-verify-web-links
+description: Crawl a site from its sitemap and verify every link, #anchor, redirect chain, OG image and meta tag against .crust.ts fixtures. `--base-url` auto-discovers the sitemap (robots.txt → /sitemap.xml → /sitemap-index.xml), `--site-map-url` takes a local sitemap so you can check build output, `--fixtures` diffs meta. Every failure names its source page and element (structured `sources`/`meta`), and every failing run ends with an aligned fix-list. Use when link or SEO health must gate a deploy, when a built site should be checked before publishing, or when OG/meta regressions need catching.
+---
+
+# crust verify-web-links
+
+## What it verifies
+
+| Check | Fails when |
+| --- | --- |
+| internal + external links | any status that is not 2xx |
+| `#fragment` targets | the id/name is not on the fetched page |
+| redirect chains | a link 301s/302s somewhere (informational with `--no-redirect-warnings`) |
+| OG / twitter images | the image URL is unreachable |
+| meta vs `--fixtures` | a keyed value differs, or a predicate returns falsy |
+
+Exit codes: `0` all clear, `1` verification failures, `2` bad args or an
+unreachable sitemap. With `--strict`, `1` also covers a run that finished
+having left discovered links unchecked. `2` also covers `--site-map-url`
+together with `--base-url`, neither of them given, `no fixture files matched
+<glob>`, `--concurrency must be >= 1`, and `unexpected argument`.
+
+## Every failure names its source (new in 0.3.6)
+
+Failures are **one per failing target**, and each answers "which page did I
+find this on, and which element":
+
+- `sources[]` — every link that pointed at the failing target, as
+  `{fromUrl, kind, href}`. In text each prints as an indented
+  `from <page> (<kind>)` line under the failure; in `--json` it is the full
+  array. A broken URL linked from fifty pages is still **one** failure with
+  fifty sources — `totals.failures` and exit codes are unchanged.
+  `kind: "sitemap"` means the URL arrived as a sitemap `<loc>` and no page
+  linked it; the sitemap URL is named as `fromUrl` instead of the source being
+  omitted.
+- `meta` — structured `{key, expected, actual}` for `meta-mismatch`
+  (`meta-fixture-no-page` carries `key: "fixture"`).
+- **Fix-list**: every failing text run ends with a separate aligned section —
+  one line per (page, element, broken-thing) triple, nothing else (excerpt of
+  the real output; the full failing report is in USAGE and on the website):
+
+  ```text
+    fix list
+    page                           element        broken link / diff
+    http://localhost:47831/page-a  a              http://localhost:47831/broken
+    http://localhost:47831/page-a  a              http://localhost:47831/page-x#ghost
+    http://localhost:47831/page-b  img            http://localhost:47831/broken
+    http://localhost:47831/page-x  meta:og:title  got "Page X — Real Title" (expected "Page X — Wrong Title")
+  ```
+
+  A clean run prints no fix-list section. `--json` carries the structured
+  `sources`/`meta` fields instead — the data *is* the fix-list there.
+
+The `detail` prose is unchanged from v0.3.5: the structured fields are purely
+additive.
+
+## Getting it pointed at the right thing
+
+```crust
+verify-web-links --base-url https://example.com
+verify-web-links --site-map-url https://example.com/sitemap.xml
+verify-web-links --site-map-url ./dist/sitemap.xml --no-recurse
+verify-web-links --base-url https://staging.example.com --fixtures site/*.meta.crust.ts
+```
+
+`--base-url` probes `robots.txt` for `Sitemap:` lines, then `/sitemap.xml`, then
+`/sitemap-index.xml`; when none answer it names all three. `--site-map-url`
+(aliased `--sitemap`) also takes a **local path**, which is the CI move: crawl
+the build output rather than a deployed site. With `--no-recurse` only the URLs
+in the sitemap are fetched.
+
+## Green does not mean checked
+
+Three ways to get exit 0 with nothing actually verified. Read the note lines and
+`totals` before believing a green run:
+
+| Symptom | What it means |
+| --- | --- |
+| `N page(s) had to be recognised from their body` | the server sends HTML without saying `text/html`. crust followed the links anyway, but the mislabel is the site's bug |
+| `N page(s) NOT read as HTML` | a sitemap entry or `<a>` target that is neither declared HTML nor smells like it (a JSON endpoint). Its links were **not** followed and the crawl ended there |
+| a note naming a count (`anchors skipped…`, `stopped at --max-pages…`) | that work did not happen. `--no-recurse` and `--no-anchors` produce it deliberately |
+| `"parsed": false` on a page in `--json` | no links and no ids were extracted from that page |
+
+`totals.complete` is the one field that means "nothing was left unchecked":
+false on any of the above. `--max-pages N` is a safety valve, not a fast mode —
+the crawl stops early, the unchecked count is reported, and the exit code still
+says 0 without `--strict`. Treat a `complete: false` run as inconclusive.
+
+## In CI
+
+**`--strict` is the CI flag.** It exits 1 when the run finished having left
+discovered links unchecked — a `--max-pages` cut-off, an unchecked `#fragment`,
+a page crust could not read — so a bounded crawl cannot pass as a clean one:
+
+```crust
+verify-web-links --base-url https://example.com --strict --max-pages 500
+```
+
+A builtin runs in-process and cannot sit mid-pipeline —
+`verify-web-links … | assert (…)` is refused with *"give it a line of its own"*.
+Redirect the whole invocation instead, then judge the report:
+
+```bash
+crust -c 'verify-web-links --base-url https://example.com --json --no-progress' > report.json
+```
+
+```crust
+read report.json | (s => JSON.parse(s)) | assert (r => r.totals.failures === 0)
+read report.json | (s => JSON.parse(s)) | assert (r => r.totals.complete)
+```
+
+`read` yields each file's **contents as one item**, so `JSON.parse` receives the
+whole document. The first line fails on a real failure; the second catches every
+inconclusive case at once — `complete` is false if anything was dropped, any
+`#fragment` went unchecked, or any page could not be read.
+
+## Meta fixtures
+
+```ts
+export default {
+  url: "https://example.com/about",
+  meta: {
+    title: "About Us",
+    description: (d: string) => d.length > 50 && d.length < 160,
+    "og:image": (u: string) => u.endsWith(".png") || u.endsWith(".jpg"),
+    "twitter:card": "summary_large_image",
+  },
+};
+```
+
+A value is compared for equality; a function is a predicate over the extracted
+value. A key a page does not have is **asked about**, not ignored — so asserting
+`og:image` on a page that lost it is a failure, which is the point.
+
+Fixture `url`s match **URL-normalized**: scheme+host case and default ports
+don't matter, `https://x.com` and `https://x.com/` are the same page, but a
+trailing slash on a non-root path stays significant (`/login/` ≠ `/login`). A
+fixture whose url never appears in the crawl fails as `meta-fixture-no-page` —
+a fixture for a page you deleted is a failure, not a skip.
+
+## Tuning
+
+```crust
+verify-web-links --base-url https://example.com --exclude /checkout/ --exclude /go/
+verify-web-links --base-url https://example.com --include-external --max-depth 2
+verify-web-links --base-url https://example.com --concurrency 16 --timeout 3000
+```
+
+- `--exclude <substring>` is repeatable. Use it for subtrees that redirect **by
+  design** (short-link senders, locale switches); redirect chains are failures
+  otherwise. `--no-redirect-warnings` silences them globally — only when you
+  mean it.
+- Off-origin links are not queued at all without `--include-external`, and even
+  then they are never recursed into.
+- `--max-depth` (default 5) bounds the crawl, `--concurrency` (default 4) and
+  `--timeout` (default 10000 ms) bound each fetch.
+- Anchors come from the crawl: with `--no-recurse` the internal links found on
+  sitemap pages are not fetched, so their `#fragments` are not checked — the
+  report says how many were skipped.