@eightmedia/flightcheck 0.0.0-stage → 0.1.0

package/docs/check-api.md

@@ -0,0 +1,142 @@
+# Flightcheck check API
+
+Flightcheck is built as a modular check system. Each check is a standalone exported
+function that can run on its own, be exposed as an API endpoint, or be orchestrated
+together by the CLI.
+
+For a plain-language list of what is audited, see [checks.md](./checks.md).
+
+## Available checks
+
+| ID | Status | Description |
+| --- | --- | --- |
+| `crawl` | implemented | Crawl pages and collect SEO, metadata, tracking, and security data |
+| `analyze` | implemented | Turn crawl JSON into `*-report.json` + human-readable `*-report.md` |
+| `security-headers` | implemented | Verify HTTP security headers (CSP, HSTS, etc.) |
+| `dns` | implemented | Verify DNS records, SPF/DMARC, and domain ownership signals. SPF/DMARC missing-record warnings are skipped on Vercel/Netlify platform preview hostnames (`*.vercel.app`, `*.netlify.app`) |
+| `robots-txt` | implemented | Report robots.txt presence, sitemap directives, and crawl blocks |
+| `sitemap` | implemented | Verify sitemap exists and exactly matches crawled URLs (same hostname and path; follows sitemap index files; runs after crawl; seeds crawl with sitemap URLs) |
+| `agent-ready` | implemented | AI agent discoverability: Content Signals, AI bot rules, llms.txt, Markdown negotiation, Link headers, well-known protocol cards |
+| `lighthouse` | planned | Lighthouse performance, accessibility, and best-practices audits |
+| `redirect` | implemented | Always-on pipeline check (not a `--checks` registry id): probes the start URL for HTTP redirects, validates the apex ↔ www alias (skipped on Vercel/Netlify platform preview hostnames), and drives redirect-only mode (`dns`, `analyze`) |
+
+### Crawl check groups (Tier 1)
+
+When crawling, these optional groups extend per-page extraction:
+
+| Group | Status | Description |
+| --- | --- | --- |
+| `images` | implemented | Image alt text issues (missing, empty, generic); decoded filenames from `src`/`srcset`/`<picture>` and `/_next/image` |
+| `a11y-static` | implemented | Static accessibility hints (headings, labels, lang, tabindex) |
+
+Use `--only images,a11y-static` or include them in the default crawl groups.
+
+See `docs/qc-checks-plan.md` for the full tiered roadmap:
+
+- **Tier 0/1** — crawl + fetch (SEO, headers, DNS, links, robots, sitemap — no browser)
+- **Tier 2** — `a11y-axe` on 3–5 sampled URLs
+- **Tier 3** — optional `performance` (Lighthouse or PageSpeed API)
+
+## Programmatic usage
+
+Import from the package root or the checks subpath:
+
+```typescript
+import { runChecks, runCrawlCheck, runAnalyzeCheck, listChecks } from "@eightmedia/flightcheck";
+// or
+import { runCheck, planCheckWaves } from "@eightmedia/flightcheck/checks";
+```
+
+Run the default pipeline (all implemented checks):
+
+```typescript
+const { outputs, results } = await runChecks({
+  startUrl: "https://example.com",
+  crawl: { maxPages: 100 },
+  analyze: { errorsOnly: false },
+});
+```
+
+Default checks: `crawl`, `security-headers`, `dns`, `robots-txt`, `sitemap`, `agent-ready`, `analyze`.
+
+Run a single check:
+
+```typescript
+const crawl = await runCrawlCheck({ startUrl: "https://example.com" });
+const report = await runAnalyzeCheck({ records: crawl.records });
+```
+
+Run independent Tier 1 checks in parallel:
+
+```typescript
+await runChecks({
+  startUrl: "https://example.com",
+  checks: ["crawl", "security-headers", "dns", "robots-txt", "sitemap", "agent-ready", "analyze"],
+  parallel: true,
+});
+```
+
+Pure analysis helpers (no network) are also exported for tests and API use:
+
+```typescript
+import {
+  analyzeSecurityHeaders,
+  analyzeRobotsTxt,
+  analyzeDnsRecords,
+  analyzeSitemapCompare,
+  analyzeAgentReady,
+} from "@eightmedia/flightcheck/checks";
+```
+
+Wave planning:
+
+- Wave 1 (parallel-capable): `crawl`, `lighthouse`, `security-headers`, `dns`, `robots-txt`, `agent-ready`
+- Wave 2 (after crawl): `sitemap`
+- Wave 3: `analyze` (uses crawl output when available)
+
+## CLI
+
+```bash
+pnpm run flightcheck -- https://example.com
+pnpm run flightcheck -- https://example.com --parallel
+pnpm run flightcheck -- https://example.com --checks crawl,analyze --parallel
+pnpm run flightcheck -- https://example.com --checks crawl --json-only
+```
+
+By default, `flightcheck` runs all implemented checks including sitemap validation. When the
+start URL returns an HTTP redirect (for example apex → www), flightcheck automatically
+switches to **redirect-only mode**: it validates the redirect and skips crawl, sitemap,
+robots.txt, and security-headers checks. The redirect check also probes the alternate host
+(apex ↔ www): the non-canonical host must permanently redirect to the canonical host implied
+by the start URL (or its redirect target). That www-alias probe is skipped for Vercel and
+Netlify platform preview hostnames (`*.vercel.app`, `*.netlify.app`), and the DNS check
+likewise skips SPF/DMARC missing-record warnings on those hostnames. Use `--checks crawl`
+to force a full crawl anyway.
+
+Use `--checks` to run a subset. Add `--parallel` to run wave 1 checks concurrently.
+
+## Adding a new check
+
+1. Add the check ID to `CHECK_IDS` in `src/checks/types.ts`.
+2. Implement `runYourCheck()` in `src/checks/your-check.ts`.
+3. Register it in `src/checks/registry.ts` with `implemented: true`.
+4. Export it from `src/checks/index.ts`.
+5. Update `planCheckWaves()` if the check has special ordering rules.
+
+Each check should:
+
+- Accept typed input options
+- Return typed result data
+- Avoid coupling to CLI or filesystem when possible (allow in-memory results for API use)
+
+## API endpoint mapping
+
+In a consuming project, map checks to endpoints:
+
+- `POST /checks/crawl` → `runCrawlCheck`
+- `POST /checks/analyze` → `runAnalyzeCheck`
+- `POST /checks/run` → `runChecks`
+- `GET /checks` → `listChecks`
+
+This keeps one check per route and makes it straightforward to run checks in separate
+workers or threads.

package/docs/checks.md

@@ -0,0 +1,116 @@
+# What Flightcheck checks
+
+Simple overview of everything Flightcheck looks at when it audits a site.
+Written for anyone who wants to know what is covered — no technical setup required.
+
+For developers implementing or extending checks, see [check-api.md](./check-api.md).
+
+## On every page (SEO & basics)
+
+- Page title
+- Heading (h1)
+- Meta description
+- Robots meta (e.g. noindex / nofollow)
+- Template marker
+- Meta title
+- Character encoding (charset)
+- Language
+- Viewport (mobile-friendly)
+- Doctype
+- Canonical URL
+- Color scheme
+- Theme color
+- Main content landmark (`<main>` — exactly one)
+
+## Sharing & social previews
+
+- Open Graph title
+- Open Graph description
+- Open Graph image
+- Open Graph URL
+- Open Graph type
+- Open Graph site name
+- Twitter card type
+- Twitter title
+- Twitter description
+- Twitter image
+
+## Icons
+
+- Favicon
+- Other site icons
+
+## Tracking (detected if present)
+
+- Google Tag Manager
+- Google Analytics 4
+- Universal Analytics
+- Matomo
+- Facebook Pixel
+- Plausible
+
+## Images
+
+- Missing image alt text
+- Empty image alt text
+- Generic / unhelpful image alt text
+
+## Accessibility (static checks)
+
+- Multiple h1 headings on one page
+- Skipped heading levels
+- Missing language on the page
+- Form fields without a label
+- Positive tabindex (keyboard focus order)
+
+## Security (secrets on the page)
+
+- Leaked API keys / secrets in page HTML
+- Leaked API keys / secrets in same-site JavaScript
+
+## Security headers (site-wide)
+
+- HSTS (force HTTPS)
+- Content Security Policy
+- Clickjacking protection (X-Frame-Options or CSP)
+- X-Content-Type-Options (nosniff)
+- Referrer-Policy
+- Permissions-Policy
+
+## Domain & redirects
+
+- Domain redirect (e.g. apex → www)
+- Redirect is permanent (301/308)
+- Redirect target is reachable
+- www and non-www both point to the same preferred host
+- SPF (email authentication)
+- DMARC (email authentication)
+
+## Search engines (robots & sitemap)
+
+- robots.txt exists
+- robots.txt is valid
+- robots.txt does not block the whole site
+- robots.txt lists a sitemap
+- Sitemap exists and is reachable
+- Sitemap is not empty
+- Sitemap URLs match the pages found on the site
+- Pages found on the site are listed in the sitemap
+
+## Agent readiness
+
+- Content Signals in robots.txt (`ai-train` / `search` / `ai-input`)
+- Explicit AI bot User-agent rules (e.g. GPTBot, ClaudeBot)
+- llms.txt overview at `/llms.txt`
+- Markdown content negotiation (`Accept: text/markdown`)
+- Agent-useful Link response headers on the homepage
+- Well-known protocol discovery (MCP, Agent Skills, API Catalog, ARD, A2A) — informational
+
+## Technology (informational)
+
+- Platform / framework / CMS / hosting / CDN fingerprints
+
+## Note on redirect-only runs
+
+If the start URL only redirects (for example a domain move), Flightcheck skips most
+page checks and only covers redirects, DNS (SPF/DMARC), and the report.

package/docs/crawler.md

@@ -0,0 +1,158 @@
+# Flightcheck Crawler
+
+This crawler collects per-URL verification data for SEO, metadata, tracking, and icons.
+
+## Purpose
+
+- Crawl from a start URL; sitemap URLs are added as extra seeds by default so all pages from the sitemap are visited.
+- Write incremental JSON output after each processed URL.
+- Deduplicate on URL without query/hash.
+- Exclude by URL and pathname.
+- Safe throttling via concurrency, delay, and burst delay.
+
+## CLI usage
+
+```bash
+pnpm run flightcheck -- https://example.com
+```
+
+By default, flightcheck writes crawl JSON plus a structured analyze report to `./flightcheck/`,
+with filenames based on timestamp and hostname, for example:
+
+- `./flightcheck/20260618.1413-example.com-raw.json`
+- `./flightcheck/20260618.1413-example.com-report.json`
+- `./flightcheck/20260618.1413-example.com-report.md`
+
+Use `--json-only` to skip the analyze report.
+
+Key flags:
+
+- `--output <file>` default `{timestamp}-{hostname}-raw.json` in `./flightcheck/`
+- `--report <file>` default `{timestamp}-{hostname}-report.md` in `./flightcheck/`
+  (paired `*-report.json` is written next to it)
+- `--json-only` write crawl JSON only; with `--json-only`, default checks fall back to `crawl` only
+- `--errors-only` include only critical/error findings in the Markdown report
+- `--checks <ids>` comma-separated checks; default: all implemented checks (`crawl`, `security-headers`, `dns`, `robots-txt`, `sitemap`, `analyze`)
+- `--parallel` run independent checks in the same wave concurrently
+- `--max-pages <n>`
+- `--delay-ms <n>`
+- `--timeout-ms <n>`
+- `--concurrency <n>`
+- `--burst-delay-ms <n>`
+- `--sitemap-start` (legacy) limits initial seeds to sitemap URLs plus the start URL; by default sitemap URLs are already added to the crawl queue
+- `--sitemap-url <url>`
+- `--exclude <pattern1,pattern2>` (repeatable)
+- `--respect-robots-txt`
+- `--quiet` (suppress Crawlee info/statistics logs; warnings and errors only)
+- `--only <groups>` run only these check groups (comma-separated)
+- `--skip <groups>` skip these check groups (comma-separated)
+- `--reveal-secrets` show full secret values instead of masked
+
+Examples:
+
+```bash
+pnpm run flightcheck -- https://example.com --max-pages 200 --concurrency 5 --delay-ms 100
+pnpm run flightcheck -- https://example.com --sitemap-start
+pnpm run flightcheck -- https://example.com --exclude "/search,/preview"
+pnpm run flightcheck -- https://example.com --respect-robots-txt
+pnpm run flightcheck -- https://example.com --quiet
+pnpm run flightcheck -- https://example.com --only security
+pnpm run flightcheck -- https://example.com --skip tracking,social
+pnpm run flightcheck -- https://example.com --only security --reveal-secrets
+```
+
+## Check groups
+
+Extraction is split into named groups. All groups run by default; with `--only` you run one or a
+few, with `--skip` you exclude groups. `--only` and `--skip` cannot be used together.
+
+- `meta` — title, h1, description, robots, template, metaTitle, charset, language,
+  viewport, doctype, canonical, color-scheme, theme-color, main landmark count.
+- `icons` — favicon and icons.
+- `social` — Open Graph and Twitter metadata.
+- `tracking` — GTM, GA4, UA, Matomo, Facebook Pixel, Plausible.
+- `stack` — technology fingerprints via curated rules (see [stack.md](./stack.md)).
+- `security` — scan for leaked API keys and secrets (see below).
+- `images` — extracts image-alt issues to `images.issues` (missing-alt,
+  empty-alt, generic-alt).
+- `a11y-static` — static accessibility checks to `a11yStatic.issues`
+  (heading-skip, multiple-h1, missing-lang, unlabeled-input, positive-tabindex).
+
+### Sitemap parsing
+
+`sitemap.xml` can be a **sitemap index** (`<sitemapindex>`) that points to child sitemap files
+via nested `<loc>` entries, or a **urlset** with page URLs. The crawler follows sitemap
+index files recursively and treats apex and `www` hostnames as the same site (so
+`https://example.com/sitemap.xml` can discover URLs on `https://www.example.com/`).
+
+The `security` group scans both page HTML and linked same-origin JavaScript files (`<script src>`)
+for leaked API keys and secrets. Shared bundles are cached per URL and fetched only once.
+Findings get a severity:
+
+- `critical` — should never be in client code (AWS keys, Stripe secret keys, GitHub
+  tokens, private-key blocks, generic `secret`/`password` assignments).
+- `review` — normal in client code but should be domain-/referrer-restricted
+  (Google API keys, Stripe publishable keys, generic `apiKey`/`token`).
+
+The providers/patterns are: google-api-key (`google`), aws-access-key-id (`aws`),
+stripe-secret-key and stripe-publishable-key (`stripe`), github-token (`github`),
+private-key, generic-secret and generic-api-key (`generic`).
+
+Secret values are masked by default (`AIzaSy…BC0Y`). The summary then shows a hint to
+re-run with `--reveal-secrets` for full values.
+
+When a group is skipped, the corresponding fields stay empty; the output structure never changes.
+
+## Output schema
+
+Each record contains:
+
+- URL and status
+- H1/title/meta fields, including `template` (from `meta[name="template"]`) and `metaTitle`
+- `mainCount` (number of `<main>` elements, `null` when meta group is skipped)
+- canonical/favicon/icons
+- OG and Twitter metadata
+- tracking detection (GTM, GA4, UA, Matomo, Facebook Pixel, Plausible)
+- `stack.detections` — technology fingerprints per page (aggregated in analyze)
+- `security` with `secrets` (found keys/secrets) and `scannedAssets` (number of
+  scanned JS files)
+- `images.issues` — image-alt issues (only when the `images` group runs)
+- `a11yStatic.issues` — static a11y findings (only when the `a11y-static` group runs)
+- optional `note` and `error`
+
+Text fields are always filled with a string — no `undefined` or `null`.
+
+Each `secrets` finding contains `type`, `provider`, `severity` (`critical` or
+`review`), `masked` (masked or — with `--reveal-secrets` — full value) and
+`source` (`html` or the URL of the JS file). The closing summary contains
+`secretsFound` (total number of findings) and, with masked output and findings, a `hint`.
+
+## Architecture
+
+The CLI runs via the checks layer (`runChecks` in `src/checks/runner.ts`), which drives the
+crawler; `src/cli.ts` no longer calls `runCrawler` directly.
+The checks layer plans checks in waves and delivers crawl and analyze output.
+
+- Checks layer: `src/checks/runner.ts`, `src/checks/crawl.ts`, `src/checks/types.ts`
+- Crawl layer: `src/crawler/runCrawler.ts`
+- Extract layer: `src/crawler/extractors.ts`, `src/crawler/tier1Extractors.ts`,
+  `src/crawler/tracking.ts`, `src/crawler/secrets.ts` and `src/crawler/stack.ts`
+- Report layer: `src/crawler/reporter.ts`
+
+Additionally:
+
+- URL normalization and excludes: `src/crawler/urlUtils.ts`
+- Sitemap parsing: `src/crawler/sitemap.ts`
+- CLI parsing: `src/cli.ts`
+
+## Behavior
+
+- Only internal links are crawled. Links are discovered via `extractInternalLinks` and added to the queue; Crawlee's `enqueueLinks` is intentionally not used.
+- Sitemap URLs are added to the crawl queue by default so pages from `sitemap.xml` are visited even when they are not reachable via internal links.
+- `--max-pages` is the hard stop (`maxRequestsPerCrawl`). The crawl stops once this count is processed, even if unvisited links remain. Increase `--max-pages` to crawl the entire site.
+- Query parameters and hashes do not count toward URL identity.
+- Non-HTML responses are reported with `note`.
+- Links to files with a known non-HTML extension (such as `.pdf`, `.zip`, images, video, fonts) are not added to the queue, so Crawlee does not generate unnecessary retries and errors.
+- With `--respect-robots-txt`, URLs excluded by `robots.txt` are skipped.
+- After each URL, output is written atomically.
+- On `SIGINT`, the crawl stops gracefully while preserving partial output.

package/docs/index.md

@@ -0,0 +1,24 @@
+# Documentation index
+
+## Features
+
+- [What Flightcheck checks](./checks.md)
+- [Check API](./check-api.md)
+- [Agent readiness](./agent-ready.md)
+- [Analyze reports](./analyze.md)
+- [Crawler](./crawler.md)
+- [Stack detection](./stack.md)
+- [QC checks plan](./qc-checks-plan.md)
+- [Agent fixes (deprecated)](./agent-fixes.md)
+
+## Specs
+
+- [Structured report](./superpowers/specs/2026-10-27-structured-report-design.md)
+
+## Changelog
+
+- Package releases: [CHANGELOG.md](../CHANGELOG.md)
+- Branch notes: [changelog/](./changelog/)
+  - [2026-10-27 — npm-public-package](./changelog/2026-10-27--npm-public-package.md)
+  - [2026-10-27 — structured-report-json](./changelog/2026-10-27--structured-report-json.md)
+  - [2026-09-28 — agent-ready-check](./changelog/2026-09-28--agent-ready-check.md)