pagetrace 0.9.1 → 0.10.0

package/CHANGELOG.md

@@ -6,6 +6,55 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 While the version is below 1.0.0, breaking changes ship in a minor release.
 
+## [0.10.0] - 2026-09-08
+
+### Added
+
+- Internal link integrity. `link.broken` (error) for a link pointing at no page,
+  and `link.broken.added` for one this build broke — the diff stays quiet about
+  breakage it did not introduce, so CI fails on the regression rather than on
+  the backlog. Only broken links are stored, keeping a site's navigation out of
+  the lockfile and its diff readable. A `--dir` crawl is authoritative; an
+  origin crawl confirms each candidate with a real request, because a sitemap
+  routinely omits pages that are live. External links are not checked.
+
+- Redirect drift. A crawl now records where each route actually landed, taken
+  from the response itself at no extra request. `redirect.added` and
+  `redirect.changed` (warn) and `redirect.removed` (info) report a route that
+  starts, moves or stops redirecting — a whole class of change that was
+  previously invisible, since a redirected route still answers. A redirect that
+  only adds or drops a trailing slash is server configuration, not drift, and is
+  not reported.
+- `canonical.redirects` (warn): a canonical naming a URL that redirects is one
+  the engine may ignore. Only raised when the target was actually crawled.
+- Sitemap health. `sitemap.dead` (error) for a sitemap entry answering 404, and
+  `sitemap.redirect` (warn) for one that redirects. Both read data the crawl
+  already gathered.
+- `--format sarif` on `check`, for `github/codeql-action/upload-sarif`. Findings
+  become pull request annotations and Security tab entries.
+- `pagetrace init`: writes a config file and takes the first snapshot. An
+  existing config is never overwritten.
+- Thirteen more Schema.org types in the rich-results table, taking it from 20 to
+  33: Book, Dataset, QAPage, Question, Answer, ClaimReview, Movie, ProfilePage,
+  DiscussionForumPosting, ImageObject, SpecialAnnouncement,
+  EmployerAggregateRating and PodcastSeries.
+
+### Changed
+
+- An origin crawl now honours `robots.txt` `Disallow`, with longest-match-wins
+  and Allow winning ties, so `Disallow: /` plus `Allow: /blog` crawls the blog.
+  **A staging origin that serves `Disallow: /` now yields no pages** — pass
+  `--ignore-robots`, or set `"ignoreRobots": true`, to crawl a site you own.
+- **`sitemap.dead` is an error**, so a sitemap that already lists a dead URL
+  fails `check` on the first run after upgrading. The problem predates the rule.
+
+### Fixed
+
+- A field absent from an older lockfile read as a transition rather than as
+  missing information, because the scalar diff compared `undefined` against
+  `null`. Every optional field added from here on would have reported a phantom
+  change on every page of the first run.
+
 ## [0.9.1] - 2026-09-07
 
 ### Changed
@@ -281,6 +330,7 @@ Initial release. `snapshot`, `check` and `audit` commands; filesystem and HTTP
 crawling; diff classified by transition; absolute, cross-page and hreflang audit
 rules; pretty, JSON, markdown, GitHub and HTML reporters.
 
+[0.10.0]: https://github.com/shyamexe/pagetrace/compare/v0.9.1...v0.10.0
 [0.9.1]: https://github.com/shyamexe/pagetrace/compare/v0.9.0...v0.9.1
 [0.9.0]: https://github.com/shyamexe/pagetrace/compare/v0.8.1...v0.9.0
 [0.8.1]: https://github.com/shyamexe/pagetrace/compare/v0.8.0...v0.8.1

package/README.md

@@ -21,13 +21,21 @@ Requires Node 20.19 or newer. No native modules, three small dependencies.
 
 ## Use
 
-Record a baseline from your build output:
+Set up a config file and the first baseline in one step:
+
+```bash
+npx pagetrace init --dir ./out
+```
+
+Or record the baseline on its own:
 
 ```bash
 npx pagetrace snapshot --dir ./out
 git add pagetrace.lock.json
 ```
 
+`init` never overwrites an existing config; it refreshes the lockfile and leaves your edits alone.
+
 Check every build against it:
 
 ```bash
@@ -113,6 +121,8 @@ npx pagetrace snapshot --url https://example.com --limit 200
 
 Routes are discovered from `robots.txt` sitemap declarations, falling back to `/sitemap.xml`. Sitemap indexes are followed one level, up to 50 children, and expansion stops once `--limit` is satisfied. Gzipped children are recognised but not read.
 
+Paths that `robots.txt` disallows are skipped, with the longest matching rule winning so an `Allow` exception still gets crawled. A staging origin that serves `Disallow: /` would therefore yield nothing — pass `--ignore-robots` (or set `"ignoreRobots": true` in the config) to crawl a site you own anyway.
+
 URLs pointing at another host are skipped. A page that cannot be fetched stops the run with an error rather than being dropped from the snapshot — a page silently missing from a crawl is indistinguishable from a page you deleted, and reporting a transient outage as a site-wide deletion is worse than failing.
 
 ### Version and updates
@@ -146,10 +156,20 @@ pagetrace update              # install it globally
 | `aeo.crawler.newly_blocked` | error | `robots.txt` started blocking an AI crawler |
 | `aeo.llmstxt.removed` | error | `/llms.txt` disappeared |
 | `page.removed` | warn | A route in the lockfile is no longer there |
+| `redirect.added` | warn | A route that used to answer directly now redirects |
+| `redirect.changed` | warn | A route redirects somewhere new |
+| `canonical.redirects` | warn | A canonical points at a URL that redirects |
+| `link.broken.added` | error | A page started linking to a URL that does not exist |
+| `sitemap.dead` | error | The sitemap lists a URL that answers 404 |
+| `sitemap.redirect` | warn | The sitemap lists a URL that redirects |
 | `og.removed` / `hreflang.removed` | warn | Social or i18n tags dropped |
 | `title.changed` | info | Ordinary copy edit |
 
-Alongside the diff, `check` runs absolute rules: missing title, canonical, `h1`, or description; JSON-LD required and recommended properties for the twenty Schema.org types Google supports as rich results; thin content; images without `alt`; and AEO signals like whether the page opens with something an answer engine can quote. Disable with `--no-audit`.
+Internal links are checked too. Only the broken ones are stored, so a site's navigation never lands in the lockfile: `link.broken` for a link that is already dead, `link.broken.added` for one this build broke. A `--dir` crawl is authoritative — the build directory is the whole site — while a crawl confirms each candidate with a real request first, because a sitemap routinely omits pages that are live. External links are deliberately not checked: a Cloudflare 403 and a rate limit both look like a dead page, and that is where link checkers earn their reputation for false positives.
+
+Redirects are recorded from the response itself, so they cost no extra requests. A redirect that only adds or drops a trailing slash is server configuration rather than drift and is not reported. `canonical.redirects` is only raised when the canonical's target was actually crawled, so a `--limit` run cannot invent it.
+
+Alongside the diff, `check` runs absolute rules: missing title, canonical, `h1`, or description; JSON-LD required and recommended properties for the thirty-three Schema.org types Google supports as rich results; thin content; images without `alt`; and AEO signals like whether the page opens with something an answer engine can quote. Disable with `--no-audit`.
 
 ## Config
 
@@ -200,7 +220,18 @@ Without the Action:
 - run: npx pagetrace check --dir ./out --baseline-branch origin/main --format github
 ```
 
-`--format` accepts `pretty`, `json`, `markdown` (sized for a PR comment), and `github` (workflow annotations).
+`--format` accepts `pretty`, `json`, `markdown` (sized for a PR comment), `github` (workflow annotations) and `sarif`.
+
+SARIF puts the findings in the Security tab and on the pull request itself, which survives longer than a comment:
+
+```yaml
+- run: npx pagetrace check --dir ./out --baseline-branch origin/main --format sarif > pagetrace.sarif
+- uses: github/codeql-action/upload-sarif@v3
+  with:
+    sarif_file: pagetrace.sarif
+```
+
+Needs `security-events: write`. Routes are not source files, so GitHub lists each finding without anchoring it to a line in the diff.
 
 ## Programmatic API