pagetrace 0.2.0

package/CHANGELOG.md

@@ -0,0 +1,121 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+While the version is below 1.0.0, breaking changes ship in a minor release.
+
+## [0.2.0] - 2026-09-06
+
+A correctness pass over the whole surface. No finding codes were renamed, so
+integrations keying on `canonical.removed` and friends are unaffected. Several
+finding *messages* changed, which is deliberate — see Fixed.
+
+### Removed
+
+- **Breaking.** The `pgt` bin alias. The CLI is `pagetrace` only. One name is
+  easier to remember, to document and to search for than two, and three-letter
+  bins collide freely across packages.
+
+### Added
+
+- `canonical.offsite` (error): a canonical pointing at a host other than the
+  site's own. A staging or CDN hostname leaking into canonicals removes the live
+  site from results, and `canonical.crosspath` could not see it — that rule
+  compares paths only, so `/services` canonicalising to
+  `https://staging.example.net/services` looked correct. Checked only when the
+  site's own origin is known: an origin crawl records it, and a `--dir` crawl
+  takes it from the new `siteUrl` config option. Inferring it from the canonicals
+  themselves would miss the site-wide leak, which is the case that matters.
+- `siteUrl` config option, and `site.origin` on the snapshot.
+
+### Changed
+
+- `canonical.crosspath` no longer fires on paginated archives (`/blog/page/2`,
+  `/blog/p/3`) or AMP variants (`/article/amp`, `/amp/guide`) whose canonical
+  points at the parent they are a variant of. Both are ordinary CMS output; the
+  rule flagged one per page and buried the real canonical mistakes. A paginated
+  page canonicalising somewhere unrelated is still flagged.
+- **Breaking.** `hreflang.missing` now fires only for a page that another
+  annotated page names as an alternate, instead of for every unannotated page on
+  a site that uses hreflang anywhere. The old rule produced a warning per page on
+  any partly translated site, and 48 warnings on a `--limit 50` crawl that
+  happened to include two annotated pages. Cross-page rules have to degrade
+  safely on a partial crawl; this one did not.
+- **Breaking.** A page that fails to fetch mid-crawl now aborts the run instead
+  of being skipped. Skipping it is what let a transient outage report the whole
+  site as deleted.
+- Several messages are now constant per finding, with the varying values moved to
+  `before` / `after`: `content.dropped`, `jsonld.property.removed`,
+  `og.removed`, `twitter.removed`, `hreflang.removed`, `canonical.crosspath`.
+  Reports aggregate by code *and* message, so a per-page number in the message
+  produced one row per page and defeated the template-wide rollup — a
+  400-page canonical defect read as 400 separate issues.
+- `--version` is injected from `package.json` at build time rather than
+  hardcoded in the CLI.
+
+### Fixed
+
+- `fetchText` swallowed every network error into "not found", so an unreachable
+  page was indistinguishable from a deleted one and `check` failed CI with
+  fabricated regressions during a 503 or a DNS blip. It now returns `null` only
+  for 404 and 410 and throws otherwise. The declared `signal` parameter was also
+  never passed by any caller, so no timeout ever applied and a hung origin hung
+  the CI job; requests now carry a 15s default timeout, configurable via the
+  `timeout` crawl option.
+- `--fail-on` was passed through unvalidated. An unrecognised value (`warning`,
+  `ERROR`, a typo) made every severity comparison false, so the exit gate never
+  fired and CI went green over a page of errors. Both the CLI and `shouldFail`
+  now reject an unknown severity.
+- Meta and link keywords are matched case-insensitively. CSS attribute selectors
+  are case-sensitive but the HTML keywords are not, so `<meta NAME="Description">`
+  and `<link rel="Canonical">` extracted as `null` and produced a false
+  `description.missing` and a false `canonical.missing`. Also affects
+  `name="Robots"` (a noindex regression went unreported), `name="Generator"`
+  (platform detection fell back to unknown), `application/ld+json`, and
+  `rel="alternate"` inside a multi-token `rel`.
+- Sitemap `<loc>` values are XML-entity decoded and CDATA-wrapped locations are
+  matched. XML requires `&` to be escaped, so every URL with a query string was
+  fetched at the wrong address, and a CDATA-emitting sitemap discovered zero
+  pages.
+- JSON-LD entities without an `@id` are keyed positionally rather than by type.
+  Keying on the bare type collapsed several entities of one type — three
+  `Product`s, a `FAQPage`'s `Question`s — into one, so a page dropping from three
+  to one reported no change at all.
+- `hreflang.noindex.target` no longer flags a page against itself, and duplicate
+  targets are collapsed. A noindexed page whose `en` and `x-default` shared an
+  href produced four findings, two of them the page accusing itself.
+- `check` writes its status messages to stderr. They were going to stdout ahead
+  of the report, so `--format json` did not parse and `--format github`
+  annotations were interleaved with prose.
+- Route collisions no longer silently drop a page. `snapshotFromDir` walks files
+  in sorted order and reports a collision (`blog.html` and `blog/index.html` both
+  map to `/blog`) instead of letting readdir order decide the winner; the origin
+  crawl discards sitemap URLs from another host, which `routeFromUrl` would have
+  folded onto the same route.
+- Sitemap index expansion deduplicates, caps nested fetches at 50, and stops once
+  `--limit` is satisfied. A 200-child index cost 200 serial round-trips before
+  slicing to 10 targets. Gzipped children are recognised so they are no longer
+  crawled as pages and parsed as HTML, which had emitted three false errors each;
+  their contents are still not read.
+- `formatMarkdown` escapes pipes. A title containing one — `Buy Widgets | Acme`,
+  a common CMS template — split the table row and swallowed the finding code.
+- `leadAnswer` measures the first paragraph after the `h1`, scoped to
+  `main`/`article` and ignoring `header`, `nav`, `footer` and `aside`, as its
+  docstring always claimed. It was taking the first `<p>` anywhere in the
+  document, so a cookie banner stood in for the lead — and edits to that banner
+  showed up as content changes in the lockfile.
+
+- `test/snapshot.test.ts` covers the crawl layer, which had no tests of its own:
+  404-versus-failure, an unreachable origin, a server error mid-crawl, off-host
+  filtering, sitemap index expansion and the `--limit` short-circuit.
+
+## [0.1.0] - 2026-09-06
+
+Initial release. `snapshot`, `check` and `audit` commands; filesystem and HTTP
+crawling; diff classified by transition; absolute, cross-page and hreflang audit
+rules; pretty, JSON, markdown, GitHub and HTML reporters.
+
+[0.2.0]: https://github.com/shyamexe/pagetrace/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/shyamexe/pagetrace/releases/tag/v0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shyam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT ShyamS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,176 @@
+# pagetrace
+
+A lockfile for your SEO and AEO surface. Snapshot it, diff every build, fail CI on regressions.
+
+Existing SEO and AEO tools tell you your score **right now**. They don't tell you that this deploy dropped the canonical tag from 400 pages, that a layout refactor added `noindex`, that a CMS migration stripped `Product` schema, or that someone quietly blocked `GPTBot` in `robots.txt`. Those regressions are silent for weeks until traffic moves.
+
+`pagetrace` records the search-visible surface of your site into a committed `pagetrace.lock.json`, then diffs every build against it. It classifies by *transition*, not by state: a reworded title is `info`, a removed canonical is `error`. So you can fail the build on real regressions without drowning in noise from ordinary content edits.
+
+## Install
+
+```bash
+npm install -D pagetrace
+```
+
+
+## Use
+
+Record a baseline from your build output:
+
+```bash
+npx pagetrace snapshot --dir ./out
+git add pagetrace.lock.json
+```
+
+Check every build against it:
+
+```bash
+npx pagetrace check --dir ./out
+```
+
+```
+(site-wide)
+  error robots.txt now blocks GPTBot.                        aeo.crawler.newly_blocked
+  warn  llms.txt sections removed: Locations.                aeo.llmstxt.sections.removed
+
+/
+  error Canonical was removed.                               canonical.removed
+  error Page became noindex.                                 robots.noindex.added
+  error Structured data entity LocalBusiness was removed.    jsonld.entity.removed
+  warn  Open Graph tags removed: og:title.                   og.removed
+
+5 error, 9 warning, 2 info
+```
+
+Exit code is `1` when anything at or above `--fail-on` (default `error`) is found.
+
+Accept the new state once you've reviewed it:
+
+```bash
+npx pagetrace check --dir ./out --update
+```
+
+### Auditing an existing site
+
+`snapshot` and `check` are for guarding a site you control. To assess a site as it stands — a WordPress install you have just inherited, a client site before a rebuild — use `audit`. No lockfile needed.
+
+```bash
+npx pagetrace audit --url https://example.com --limit 300
+npx pagetrace audit --url https://example.com --format html --out audit.html
+```
+
+Findings are rolled up by issue rather than by page, so one template defect reads as a single row affecting 43 pages instead of 43 separate lines. Each row carries why it matters and how to fix it, and the fix is platform-aware — `pagetrace` reads the generator tag and asset paths, so a WordPress site gets Yoast and Rank Math instructions rather than generic advice.
+
+```
+ERROR 2 pages canonicalise to https://acme.test/shop.                          (1)
+  Several pages pointing at one canonical means those pages are declaring
+  themselves duplicates and will not rank independently.
+  Fix: A common symptom of a plugin canonicalising every archive page to the parent.
+
+ERROR Page has no <h1>.                                                        (1)
+  The h1 anchors the document outline used for passage extraction.
+  Fix: Many themes render the post title as h2 inside archive templates.
+       Check single.php or the block template for this post type.
+  /tag/widgets
+```
+
+Auditing runs cross-page rules the per-page checks cannot see: duplicate titles and descriptions, several pages canonicalising to one URL, canonicals pointing away from their own path, and a full hreflang check.
+
+The hreflang rules are the ones hardest to run by hand. Google discards an entire hreflang cluster when the annotations are not reciprocal — if `/en/about` points at `/ml/about` but `/ml/about` does not point back, *every* link in that group is ignored, not just the broken one, and nothing reports it. `pagetrace` checks reciprocity across the whole crawl, plus self-references, `x-default`, malformed language codes, and alternates that point at noindexed pages. Sites with no hreflang anywhere are left alone.
+
+Counts are per issue, not per page: one template defect on 400 pages reads as a single item labelled `template-wide`, so you triage the fix once.
+
+`--format html` writes a self-contained report with no external assets and no scripts, suitable for sending to a client. `--format json` gives the same data keyed by stable finding codes. Detected platforms: WordPress, Next.js, Shopify, Webflow, Wix, Squarespace, Drupal.
+
+Route discovery follows `robots.txt` sitemap declarations, then falls back through `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`.
+
+### Against a live site
+
+```bash
+npx pagetrace snapshot --url https://example.com --limit 200
+```
+
+Routes are discovered from `robots.txt` sitemap declarations, falling back to `/sitemap.xml`. Sitemap indexes are followed one level.
+
+## What it records
+
+**Per page** — title, meta description, canonical, robots directives, Open Graph and Twitter Card tags, hreflang alternates, `h1` text, heading outline, every JSON-LD entity with its property list, word count, images missing `alt`, and the length of the first quotable paragraph.
+
+**Site-wide** — `robots.txt` crawlability per AI user agent (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and others), declared sitemaps, and `llms.txt` presence with its section headings and size.
+
+## What it catches
+
+| Code | Severity | Fires when |
+| --- | --- | --- |
+| `canonical.removed` | error | A page lost its canonical tag |
+| `robots.noindex.added` | error | A page became noindex |
+| `jsonld.entity.removed` | error | A structured data entity disappeared |
+| `jsonld.property.removed` | error | An entity lost a property it used to have |
+| `content.dropped` | error | Word count fell by more than half — usually a render failure |
+| `canonical.offsite` | error | A canonical points at a host other than your own |
+| `aeo.crawler.newly_blocked` | error | `robots.txt` started blocking an AI crawler |
+| `aeo.llmstxt.removed` | error | `/llms.txt` disappeared |
+| `page.removed` | warn | A route in the lockfile is no longer there |
+| `og.removed` / `hreflang.removed` | warn | Social or i18n tags dropped |
+| `title.changed` | info | Ordinary copy edit |
+
+Alongside the diff, `check` runs absolute rules: missing title, canonical, `h1`, or description; JSON-LD required and recommended properties for the twenty Schema.org types Google supports as rich results; thin content; images without `alt`; and AEO signals like whether the page opens with something an answer engine can quote. Disable with `--no-audit`.
+
+## Config
+
+`pagetrace.config.json`:
+
+```json
+{
+  "siteUrl": "https://example.com",
+  "ignoreRoutes": ["/preview/*", "/draft"],
+  "minWordCount": 300,
+  "severity": {
+    "title.changed": "off",
+    "content.thin": "error"
+  },
+  "aiAgents": ["GPTBot", "ClaudeBot", "MyCustomBot"]
+}
+```
+
+`siteUrl` is only needed for a `--dir` crawl, and only to detect canonicals pointing at another host — a staging hostname leaking into production canonicals. A `--url` crawl infers it.
+
+Every finding has a stable `code`. Set any code to `error`, `warn`, `info`, or `off`.
+
+## CI
+
+```yaml
+- run: npm run build
+- run: npx pagetrace check --dir ./out --format github
+```
+
+`--format` accepts `pretty`, `json`, `markdown` (sized for a PR comment), and `github` (workflow annotations).
+
+## Programmatic API
+
+```ts
+import { extractPage, diffPage, auditPage, snapshotFromDir } from 'pagetrace';
+
+const before = extractPage(oldHtml, '/pricing');
+const after = extractPage(newHtml, '/pricing');
+
+for (const finding of diffPage(before, after)) {
+  console.log(finding.severity, finding.code, finding.message);
+}
+```
+
+`extractPage`, `diffPage`, `diffSnapshots`, `auditPage`, `auditSnapshot`, `applyConfig` and the reporters are all pure functions over plain objects, so they compose into whatever pipeline you already have.
+
+## Notes
+
+Pages are fingerprinted from rendered HTML. For client-rendered apps, point `--dir` at a pre-rendered or statically exported build, or `--url` at a deployed preview — otherwise you are snapshotting an empty shell.
+
+The `llms.txt` convention and AI crawler behaviour are still moving. Treat those rules as signals worth tracking, not settled standards.
+
+## Notes on the name
+
+`pagetrace` here means a trace of a page's search-visible surface over time. It is unrelated to memory page tracing in the Linux kernel or the Go runtime, which share the name.
+
+## License
+
+MIT