@goflag/cli 0.2.9 → 0.2.11

package/README.md

@@ -1,9 +1,8 @@
-# Goflag
+# goflag
 
-> **A lean CLI that flags the site problems humans can't catch at scale.**
-> Point it at a URL and get four things: broken links, missing translation pages, a robots.txt that contradicts your pages, and missing or misconfigured SEO metadata. JSON-first.
+A CLI that audits a site for broken links, missing translation pages, a robots.txt that contradicts your pages, and missing or misconfigured SEO metadata — and a Next.js library that produces the HTML it would have nothing to say about.
 
-[![node](https://img.shields.io/badge/node-%3E%3D22-339933?logo=nodedotjs&logoColor=white)](./package.json)
+[![node](https://img.shields.io/badge/node-%3E%3D22-339933?logo=nodedotjs&logoColor=white)](https://github.com/tancredesimonin/goflag/blob/main/package.json)
 [![license: MIT](https://img.shields.io/badge/license-MIT-green)](#license)
 
 Goflag crawls a site once and judges what it found. It is deliberately small: no dashboard, no config system, no social-preview gallery. Just the findings, as a machine-readable report you can pipe, diff, or gate CI on.
@@ -13,14 +12,33 @@ search: a `hreflang` that points at a 404, a canonical that silently
 de-indexes a page, a `robots.txt` that contradicts every `<meta name="robots">`
 on the site. A human cannot check those on 400 pages. That is the whole job.
 
-## What it checks
+## Getting started
+
+```sh
+npx @goflag/cli https://example.com          # one-off, nothing installed
+pnpm add -D @goflag/cli                      # then: pnpm goflag https://example.com
+```
+
+Install it as a dev dependency once you run it more than once — in a script, in
+CI, in a git hook. `npx` is for trying it. The package is `@goflag/cli`; the
+command it installs is `goflag`. Everything under the brand lives in the
+`@goflag` scope; the bare `goflag` name on npm is a deprecated signpost pointing
+here and carries no code.
+
+- Node `>=22`.
+- Nothing else for a static audit. Headless rendering needs Chromium, which is **not** installed with goflag — `playwright` is an optional peer dependency. Add it only if you audit client-rendered pages without `--static`: `pnpm add -D playwright && pnpm exec playwright install chromium`.
+- Working in this repository instead of consuming it: `pnpm install`, then see [AGENTS.md](https://github.com/tancredesimonin/goflag/blob/main/AGENTS.md).
+
+## What it does
 
 - **Broken links** — scrapes every crawled page, dedupes targets globally (a footer link on 500 pages is probed once), and checks each with `HEAD`→`GET` fallback, redirect-chain/loop detection, soft-404 and anti-bot (403/429) triage. Broken links are mapped back to the pages that reference them.
 - **Missing translation pages** — builds a `route × locale` matrix and flags every route that exists in one locale but is missing in another, plus hreflang reciprocity gaps (`A → B` with no `B → A`), missing `x-default`, and locale tags naming a language, script or region that does not exist. Rows are keyed by pathname, unless the site declares its translation clusters — either with `xhtml:link` in the sitemap, or with reciprocal `hreflang` between two crawled pages' `<head>`. Then those declarations decide which URLs are one page, whatever their slugs, and the run reports `diagnostics.declaredClusters`. The sitemap source survives `--coverage structural`, where the two locales of a slug-translating family are rarely both sampled; the `<head>` source needs both pages in hand and is what fixes the common site that declares properly there and nothing in its sitemap.
-- **robots.txt policy** — flags a site that forbids crawling while its pages ask to be indexed. The two declarations cannot both hold, and robots.txt wins, so the meta tag is never even read.
+- **robots.txt policy** — the file is parsed to RFC 9309, not scanned: every group, every rule, every line it could not read, each with its number. goflag flags a site that forbids crawling while its pages ask to be indexed — site-wide, and now per path, which is the quiet version nothing was watching. It also reports the file that errors (a 5xx there is read as a site-wide ban), one over the 500 KiB a parser must honour, a typo that silently deletes the rule you meant to write, a redirect handing your crawl policy to another origin, and a `Sitemap:` that is not a full URL.
 - **SEO metadata** — lints each page's `<head>` for the handful of mistakes that actually hurt in search and social and are invisible in a browser: missing/oversized `<title>` and description, missing/relative canonical, missing `og:title`/`og:description`/`og:image`, missing viewport, and contradictory `robots`/`googlebot`/`X-Robots-Tag` directives.
 - **Open Graph, past its presence** — a declared `og:image` is judged as well as counted: relative URLs no crawler can resolve, an undeclared size (so the first share of a URL renders without the image), a shape far from the 1.91:1 the card is cropped to, a missing `og:image:alt`, and — the one no tag check can reach — whether the URL actually serves an image at all. On a translated page, `og:locale` and `og:locale:alternate` are checked against the hreflang cluster — two vocabularies for one fact, which nothing in a build keeps in step.
 - **Icons** — whether anything declares one at all (nothing in a spec requires it, which is why nothing complains), whether iOS has an `apple-touch-icon` to use instead of screenshotting the page, whether the Web App Manifest and the `<head>` contradict each other about the same file, whether every declared icon answers with an image, whether a `sizes` attribute describes the file it points at (a `.ico` carrying 16, 32 and 48 declared as `48x48` advertises a third of itself), and whether the origin actually serves `/favicon.ico` — a 200 of HTML from a catch-all route does not count. Two icon lists that merely differ are the normal case and are not reported.
+- **Sitemap** — whether there is one, whether it parses (an HTML error page served with a 200 reads as healthy to anything that only checks the status), whether it lists anything, and whether an index can reach its children. Each entry is judged too: a `<loc>` that is not absolute, entries on another host or another protocol, a `<lastmod>` that is not a W3C Datetime or is dated in the future, a `changefreq` or `priority` outside the values the protocol defines.
+- **Where the two files meet the crawl** — the contradictions no single artefact shows. A sitemap entry that `robots.txt` forbids fetching, one whose page declares `noindex`, one whose canonical points elsewhere, one that 404s or redirects, and the indexable pages the crawl found that the sitemap never lists. Entry probing answers from the crawl and the link audit before it fetches anything, so a well-built site costs almost no extra requests — and when the caps stop it short, the finding says how many went unchecked rather than implying the rest are fine. Nothing is broken when you look at either half alone, which is why nobody finds these by reading.
 - **Unreachable pages** — a page the crawl reached that answered non-2xx, or that never answered at all (timeout, reset, DNS failure — recorded as `status: 0`), is reported in `unreachablePages`. goflag asks a second time, from the back of the queue, before calling a page unreachable. Each one is an `error`-severity finding: it turns the verdict red, counts toward `--max-debt`, and fails the build even under `--fail-on error`. A page that silently dropped out of the audited set is what poisons a baseline, so it is reported rather than warned about.
 
 **Which pages get audited.** When goflag finds a sitemap it does not audit every URL in it. It groups URLs that share a path shape — and therefore a template — into families, audits every page that stands alone, and samples three pages per family. A cap answers "how many" and never "which": on a site of thousands of pages built from thirty templates, the first 200 pages a crawl reaches are four templates out of thirty.
@@ -41,31 +59,11 @@ fails loudly rather than passing quietly. It is also why `--static` is the right
 default in CI: it needs no browser, and it cannot mistake a broken page for a
 fine one.
 
-## Install
+## Where it runs
 
-Requires Node `>=22`.
+Two packages ship to npm from a tag, published by GitLab CI over OIDC with no stored credential: `@goflag/cli` (`v*`) and `@goflag/next` (`next-v*`). The documentation site `apps/website` runs at [goflag.tech](https://goflag.tech), with `develop.goflag.tech` alongside it, both deployed with Kamal onto the shared OVH host described in the `infrastructure` repository.
 
-```sh
-npx @goflag/cli https://example.com          # one-off, nothing installed
-pnpm add -D @goflag/cli                      # then: pnpm goflag https://example.com
-```
-
-Install it as a dev dependency once you run it more than once — in a script, in
-CI, in a git hook. `npx` is for trying it.
-
-The package is `@goflag/cli`; the command it installs is `goflag`. Everything
-under the brand lives in the `@goflag` scope; the bare `goflag` name on npm is a
-deprecated signpost pointing here and carries no code.
-
-Headless rendering needs Chromium, which is **not** installed with goflag —
-`playwright` is an optional peer dependency. Add it only if you audit
-client-rendered pages without `--static`:
-
-```sh
-pnpm add -D playwright && pnpm exec playwright install chromium
-```
-
-## Getting started
+## Your first audit
 
 Five steps, in this order. Each one is explained in full further down; this is
 the sequence.
@@ -139,15 +137,15 @@ npx @goflag/cli rules > rules.json
 ```
 
 `rules` answers a question about goflag rather than about a site: no URL, no
-crawl, no network. Every entry carries its scope, severity and summary; the
-twenty-three page rules and four prose rules also carry a rigor, the documents they
-cite and — where a remedy is a line of code — a fix snippet. Cross-page rules
-may carry them too, and one does; the three that predate the field still emit
-`rigor: null` with an empty `sources`, because choosing their citations is work
-nobody has done and inventing an authority is worse than admitting the gap.
-Write your consumer against that. The same document
-ships inside the package as `rules.json`, if you would rather read it than run
-anything.
+crawl, no network. It ships fifty-six rules — twenty-three page rules,
+twenty-eight site rules and five prose rules — and every entry carries its
+scope, severity and summary. All but one also carry a rigor, the documents they
+cite and, where a remedy is a line of code, a fix snippet. The exception is
+`hreflang.sitemap-mismatch`, which emits `rigor: null` with an empty `sources`
+because no specification says which of the two declarations is wrong; inventing
+an authority there is worse than admitting the gap. Write your consumer against
+that. The same document ships inside the package as `rules.json`, if you would
+rather read it than run anything.
 
 The flag list below is data too:
 
@@ -327,6 +325,36 @@ description on a page that has none, because `description.missing` already
 says that. They never count toward the summary, the verdict, or the exit
 code: nobody has judged them yet.
 
+### Every finding says how authoritative it is
+
+A finding carries the rule's `rigor`, the documents behind it, what the page
+actually said and what a passing page looks like — in the JSON, not only in the
+opt-in conformance view:
+
+```json
+{
+  "ruleId": "title.length",
+  "severity": "warning",
+  "rigor": "heuristic",
+  "sources": ["google-title-link", "moz-title-tag"],
+  "expected": "10–60 characters",
+  "observed": 65
+}
+```
+
+That distinction is the whole point of the rigor axis, and it is meant to be
+acted on: a `spec-required` finding and a `heuristic` one are not the same
+work, and a report read in CI — or by an agent that does not have the rule
+registry — could not tell them apart otherwise. `--summary` shows it per rule,
+where it costs one tag instead of one per finding:
+
+```
+  warn  title.length [heuristic] ×3
+```
+
+Severity says what your build should do; rigor says who says so. A profile
+changes the first and never the second.
+
 ### Gate on regressions, not on perfection
 
 A plain run fails on any finding, which is unusable on a site that is not clean
@@ -481,7 +509,7 @@ Three rules, whichever you use:
 stages: [build, deploy, audit]
 
 variables:
-  GOFLAG_VERSION: "0.2.9"
+  GOFLAG_VERSION: "0.2.11"
 
 # Before the merge: build the branch, let goflag boot it, audit that.
 seo:mr:
@@ -533,7 +561,7 @@ jobs:
       - run: corepack enable && pnpm install --frozen-lockfile
       - run: pnpm build
       - run: |
-          npx --yes @goflag/cli@0.2.9 http://localhost:3000 \
+          npx --yes @goflag/cli@0.2.11 http://localhost:3000 \
             --start "pnpm start" --static --no-external \
             --baseline .goflag/baseline.json --regressions-only --max-debt 41 \
             --report goflag-report.json
@@ -580,6 +608,7 @@ console.log(report.summary);
 ```
 packages/cli/      @goflag/cli, the CLI (installs the `goflag` command)
 packages/next/     @goflag/next, the Next.js library (see below)
+packages/og/       @goflag/og, the share cards and the favicon container
 apps/website/      goflag.tech, the landing page and the documentation
 tools/name-holder/ the bare `goflag` name on npm, a signpost to @goflag/cli
 ```
@@ -591,7 +620,7 @@ enforces it rather than trusting memory.
 ## The other half: `@goflag/next`
 
 goflag tells you what is wrong. It does not tell you how to stop writing it
-wrong. [`@goflag/next`](packages/next) is a route registry for the Next.js App
+wrong. [`@goflag/next`](https://github.com/tancredesimonin/goflag/tree/main/packages/next) is a route registry for the Next.js App
 Router that produces the HTML this auditor would have nothing to say about.
 
 ```ts
@@ -609,7 +638,32 @@ held in agreement by vigilance, and their disagreement is what goflag reports as
 `hreflang.sitemap-mismatch`. Projecting both from one registry makes that
 finding unrepresentable. Full documentation at
 [goflag.tech/docs/next](https://goflag.tech/docs/next), and the API reference in
-[`packages/next/README.md`](packages/next/README.md).
+[`packages/next/README.md`](https://github.com/tancredesimonin/goflag/blob/main/packages/next/README.md).
+
+## The picture: `@goflag/og`
+
+Two of this catalogue's rules had no remedy to point at. `og.image.missing`
+fired 24 times on one site and `og.image.alt` 46 times on another, and the fix
+for both is "either an asset or a route you have to write" — which is how a rule
+becomes permanent debt. [`@goflag/og`](packages/og) is that route, written once.
+
+```tsx
+// app/[locale]/opengraph-image.tsx
+const image = ogImage(og, async ({ params }) => {
+  const { locale } = await params;
+  return { title: t("hero.title"), alt: t("meta.ogAlt") };
+});
+
+export const generateImageMetadata = image.generateImageMetadata;
+export default image.render;
+```
+
+The core renders nothing: it returns a JSX tree that `next/og` — which already
+embeds satori — turns into a PNG at build time, so nothing installs a second
+renderer and a card can be unit-tested with no framework build. It also packs
+the `favicon.ico` **no Next convention emits**, guarded so that a generated file
+living in git is not dirtied by every commit. API reference in
+[`packages/og/README.md`](packages/og/README.md).
 
 ## Develop locally
 
@@ -635,8 +689,28 @@ eslint once at the root and then fans out to any package that defines its own
 across workspace packages with `pnpm -r`. To work on one package, filter:
 `pnpm --filter @goflag/cli test`.
 
-The engine is framework-agnostic and lives in [`packages/cli/src/lib/core/`](packages/cli/src/lib/core) (crawl, fetch/extract, link audit, i18n) and [`packages/cli/src/lib/rules/`](packages/cli/src/lib/rules) (SEO checks). The CLI shell — orchestration, the `GoflagReport` schema, and rendering — lives in [`packages/cli/src/report/`](packages/cli/src/report) and [`packages/cli/src/cli.ts`](packages/cli/src/cli.ts).
+The engine is framework-agnostic and lives in [`packages/cli/src/lib/core/`](https://github.com/tancredesimonin/goflag/tree/main/packages/cli/src/lib/core) (crawl, fetch/extract, link audit, i18n) and [`packages/cli/src/lib/rules/`](https://github.com/tancredesimonin/goflag/tree/main/packages/cli/src/lib/rules) (SEO checks). The CLI shell — orchestration, the `GoflagReport` schema, and rendering — lives in [`packages/cli/src/report/`](https://github.com/tancredesimonin/goflag/tree/main/packages/cli/src/report) and [`packages/cli/src/cli.ts`](https://github.com/tancredesimonin/goflag/blob/main/packages/cli/src/cli.ts).
+
+## Contributing
+
+GitHub is a read-only mirror; the canonical repository and the CI are on GitLab,
+so a pull request opened here cannot be merged. **Issues are the useful thing to
+open on GitHub** — a wrong finding especially, which is the class of bug handled
+first. [CONTRIBUTING.md](https://github.com/tancredesimonin/goflag/blob/main/CONTRIBUTING.md)
+has the whole story, including what makes a report reproducible and what a rule
+proposal needs. Vulnerabilities go to <hello@goflag.tech> rather than to an
+issue — see
+[SECURITY.md](https://github.com/tancredesimonin/goflag/blob/main/SECURITY.md).
 
 ## License
 
 MIT.
+
+## Learn more
+
+- Strategy and goals: [STRATEGY.md](https://github.com/tancredesimonin/goflag/blob/main/STRATEGY.md)
+- What comes next: [ROADMAP.md](https://github.com/tancredesimonin/goflag/blob/main/ROADMAP.md)
+- Working in this repository (human or agent): [AGENTS.md](https://github.com/tancredesimonin/goflag/blob/main/AGENTS.md)
+- The overall development plan: [docs/spec-and-lib-plan.md](https://github.com/tancredesimonin/goflag/blob/main/docs/spec-and-lib-plan.md)
+- The rule catalogue and its architecture: [docs/rules-catalog-plan.md](https://github.com/tancredesimonin/goflag/blob/main/docs/rules-catalog-plan.md)
+- Publishing `@goflag/next`: [docs/publishing.md](https://github.com/tancredesimonin/goflag/blob/main/docs/publishing.md)