pagesight 0.18.0 → 0.20.0

package/docs/rendering.md

@@ -0,0 +1,102 @@
+# Verify rendered SEO evidence on demand
+
+`page.verify` compares fetched server HTML with a fresh anonymous Chromium load.
+It can also click one exact internal link in a separate fresh context and compare
+that result with the direct load. This is an observation at a specified capture
+point, not Google's renderer or proof of indexing.
+
+Install Chromium once from the Pagesight installation directory:
+
+```sh
+bunx playwright install chromium
+# Linux machines may also require: bunx playwright install --with-deps chromium
+```
+
+Other Pagesight operations do not launch a browser. Missing browser binaries produce
+`browser_unavailable` with setup guidance. CI installs Chromium for real browser tests.
+
+```sh
+pagesight render --url https://example.com/target
+pagesight render --url https://example.com/target \
+  --from-url https://example.com/source --link-selector 'a#target-link' \
+  --settle-ms 1000 --timeout-ms 20000 --out render.json
+```
+
+The selector must match exactly one anchor, with a resolved `href` equal to the
+requested target URL, no download attribute and same-tab navigation. Source and
+target must share an origin. Selectors identify links, not buttons, forms or arbitrary
+actions. Ambiguous links, navigation failures and URL mismatches never become empty
+successful comparisons. The initial version does not exercise form-driven filters,
+back/forward history, screenshots, or visual layout; use a browser workflow for those.
+
+API, authenticated local HTTP and MCP `observe` take the same operation:
+
+```json
+{
+  "operation": "page.verify",
+  "url": "https://example.com/target",
+  "navigation": { "fromUrl": "https://example.com/source", "linkSelector": "a#target-link" },
+  "settleMs": 1000,
+  "timeoutMs": 20000,
+  "viewport": { "width": 1280, "height": 800 }
+}
+```
+
+Browser paths wait for DOMContentLoaded, then a fixed settle interval; navigation
+also waits for the exact target URL. `settleMs` is 0–5000 and `timeoutMs` is
+1000–60000 per browser path, including source load and click. A late hydration
+update can occur after capture. Use a larger explicit interval to investigate a
+known delay; elapsed time never proves network or application completion.
+
+The result retains independent server, direct and optional navigation observations:
+
+- Arrays of title, description, canonical, robots, H1, JSON-LD and anchor values
+  preserve duplicates and empty values. Canonical and anchor URLs include raw and
+  resolved forms; JSON-LD syntax validity is separate from semantic validity.
+- The server HTML hash identifies the independently fetched response; browser DOM
+  hashes identify extracted evidence. Capture timestamps, browser version, viewport,
+  requested/final URLs, document response status/redirect headers and X-Robots-Tag
+  preserve provenance. A client route can have no target document response.
+- DOM-field comparisons (including base href) are `equal`, `different`, `unavailable` or `not_requested`. Differences
+  are data for investigation, not automatic defects. A capture failure makes the
+  envelope partial when other evidence survives. Truncated captures are explicitly
+  unavailable for comparison; missing values are not invented.
+
+Server HTML uses Pagesight's user agent, a 2 MB response limit and a same-origin
+redirect limit, then inert Chromium parsing. It can differ from the browser's
+response because of user agents, timing, experiments or deployments. Raw HTML and
+cookies are not returned. Keep URLs and extracted page content private when needed.
+
+Each browser path uses fresh storage, a maximum of 500 attempted requests, blocked
+service workers and WebSockets, and only GET/HEAD requests. Cross-origin top-level navigation and
+downloads are blocked. These restrictions can change application behavior and are
+reported with blocked-request counts. Subresources may use other public origins; ordinary
+page requests can reach analytics. No user browser profile or authentication state
+is loaded. Browser network response sizes are not globally capped; the server HTML
+limit is separate. Extracted fields cap at 200 items and 4000 characters per value. Selection stops
+at the first excess match, and extraction uses a Chromium isolated world so page
+scripts cannot replace the extraction primitives. Document response history caps
+at 20 entries; `documentsTruncated` makes the result partial and the target HTTP
+response unavailable.
+
+Both server HTML and browser requests use a temporary local proxy. It resolves each
+upstream connection once, rejects nonpublic addresses, and connects to the checked
+IP without a second DNS lookup. Chromium's implicit loopback proxy bypass is removed;
+QUIC and non-proxied WebRTC UDP are disabled. TLS certificate checks remain enabled.
+This is request filtering, not an operating-system sandbox for untrusted browsers.
+
+For deliberate local fixtures, request a literal IP URL such as
+`http://127.0.0.1:3000/target`. Only that exact IP and port is permitted as a nonpublic
+destination; `localhost` and other names resolving to private addresses are rejected.
+A fixture cannot request another local port. Blocked proxy destinations are counted
+in `network.blockedRequests`, separately from browser route restrictions. GET requests
+can still cause site-side effects, including on the deliberately selected fixture.
+
+Run this separately when investigating a URL or verifying a deployment. It does not
+schedule analysis, alter production pages, validate rich-result eligibility, or
+establish SEO impact.
+
+`documentComparison` separately compares server/browser status and X-Robots-Tag.
+DOM equality never implies equivalent HTTP responses; inspect both comparisons
+and recorded redirect chains. Browser integration tests run in a separate Bun
+process from provider/mock tests, under the same root verification command.

package/docs/seo-agent-workflow.md

@@ -0,0 +1,153 @@
+# Agent SEO workflow
+
+Pagesight supplies evidence for decisions. Run this loop on a small cohort before
+expanding it: establish access and measurement context, identify a question,
+inspect the relevant pages, propose one change, verify the deployment, and collect
+a comparable later window. Keep raw evidence private with dated filenames.
+
+## Establish access and meaning
+
+Run `pagesight doctor --config seo.config.json`. Verify the exact GSC property,
+GA property and production hostname. Distinguish access failures from empty
+reports. Define named meaningful events in project context, including introduction
+dates and tracking changes. A configured key event is not a validated conversion.
+Test the event against a real action and check provider data separately from the
+browser request. Realtime activity cannot identify an individual test session.
+
+GSC measures search visibility, GA measures collected activity, and edge analytics (Cloudflare)
+sample edge requests. They have different omissions, timezones and units. Never combine
+them into a conversion funnel or SEO score. Review access when a provider fails;
+do not request broader credentials unless a specific read requires them.
+
+## Find demand and decide what deserves investigation
+
+1. Save a snapshot for a finalized reporting interval. Run `opportunities` against
+   it, then `investigate` for one exact candidate URL. Inspect page-filtered queries,
+   device/country mix, Google's stored inspection and current HTML. Property query
+   totals cannot establish which queries led to a particular page.
+2. Read query intent and the page together. Separate model/reference-price lookup,
+   purchase intent, historical comparison, and unrelated queries. Low observed CTR
+   alone does not establish a poor snippet; position and intent matter. Missing
+   queries may be anonymized, and absent rows are not zero demand.
+3. For wording or market questions, compare 2–5 terms in one Google Trends chart:
+   same country, period, category, search type and term/topic type. Save the chart
+   URL, capture time, accessible table, relative averages, equal-window direction
+   and relevant related searches. Exclude the unfinished current week when comparing
+   complete weeks. Trends is relative interest, not search volume; rounded zero and
+   Breakout do not quantify demand. Check product/data coverage before acting on a
+   rising query.
+4. Inspect a bounded set of actual search results when intent or competing content
+   remains uncertain. Save query, country/language, device, date, result URLs and
+   observed features. Personalized samples are not stable rank tracking. A paid
+   SERP provider is justified only for repeatable geographic/device coverage that
+   the current question needs; record cost/coverage before subscribing.
+5. Make a short brief: exact question, source pointers and dates, existing useful
+   content, missing user answer, proposed page/cohort, verification plan, caveats.
+   Improve a relevant existing page before creating overlapping pages. Similar
+   keyword wording alone is not evidence of cannibalization; inspect query/page
+   overlap, intent and canonicals before consolidation.
+
+## Discovery, architecture and technical policy
+
+Use bounded site discovery to inspect real HTML links, redirects, canonical edges,
+robots policy, sitemap samples and link depth from selected seeds. Depth is measured
+in that sample; sitemap membership does not establish internal links or a true
+orphan. Prioritize broken paths from important hubs and useful links to relevant
+content. Verify source and destination before adding links. Avoid site-wide
+keyword anchors and mechanically linking every page to every other page.
+
+Keep indexable, filtered, legacy and account-only routes explicit in project
+context. A noindex diagnostic can be intentional. Check sitemap dates against
+actual publication and content/reference versions; a future lastmod is not proof
+that the underlying data is wrong. Validate structured data against visible facts:
+reference prices are not inventory offers; do not invent availability, reviews,
+shipping or merchant claims merely to satisfy a validator.
+
+## Rendered-template and performance verification
+
+For repeatable direct-load and internal-anchor checks, use [rendered-page verification](rendering.md). Keep the manual workflow below for forms, history navigation and visual inspection.
+
+Select one known URL per important template, plus an intentional noindex or
+functional-filter example. Record browser, viewport, capture time, URL and navigation
+path (direct load versus client navigation). Run `pagesight page --url URL` to save
+server-HTML evidence. In an available browser tool, inspect the rendered page and
+record the same metadata and headings with this read-only DOM expression:
+
+```js
+({
+  url: location.href,
+  title: document.title,
+  canonical: [...document.querySelectorAll('link[rel="canonical"]')].map((e) => e.getAttribute("href")),
+  robots: [...document.querySelectorAll('meta[name="robots"]')].map((e) => e.getAttribute("content")),
+  description: document.querySelector('meta[name="description"]')?.getAttribute("content"),
+  h1: [...document.querySelectorAll("h1")].map((e) => e.textContent),
+  jsonLd: [...document.querySelectorAll('script[type="application/ld+json"]')].map((e) => e.textContent),
+  linkCount: document.querySelectorAll("a[href]").length,
+});
+```
+
+HTTP status, final URL, redirects and X-Robots-Tag come from the Pagesight page
+response, not this DOM expression.
+
+Retain exact values; distinguish duplicate tags, absent directives and legitimate
+URL variants. Compare with the independently fetched HTML, noting timing/version
+differences. Exercise one internal navigation and relevant filter state: confirm
+URL, heading, content and metadata update together. Verify actual links, price or
+other primary content, and a representative mobile viewport. Inspect a screenshot
+and document overflow; a string search can miss nonbreaking spaces and does not
+prove visibility. Check console errors without treating unrelated third-party
+warnings as demonstrated indexing failures. Restore temporary viewport overrides.
+
+Save raw browser observations/screenshots where the tool supports export. If export
+is unavailable, retain the tool transcript and label manually transcribed artifacts;
+do not claim they are original screenshots. Browser findings may be imported using
+`evidence.import` with provider `other`, exact sampled URLs, capture time, coverage
+and source label. Imports remain unverified by Pagesight. Avoid cookies, client IDs,
+full tracking payloads and private browser data in shared artifacts.
+
+Run `pagesight speed psi --url URL --strategy mobile` for lab evidence and
+`pagesight speed crux --url URL` (or `--origin`) for field evidence. Save requests, collection windows, device
+and response failures. PSI variability calls for repeated comparable lab conditions
+when diagnosing a specific issue. CrUX no-record means unavailable evidence for
+that scope. A visible fast load is not a Core Web Vitals measurement. Neither
+successful rendering nor crawler permission proves Google indexed the page.
+
+## Authority and trustworthy content
+
+Start with first-party provenance, clear methodology, reference/publication dates,
+limitations, ownership/contact and useful primary data. Cite original sources and
+explain what the product adds. Inspect GSC's Links UI/export or Bing link reads when
+available, retaining their coverage and dates. GSC Links has no supported Pagesight API endpoint; import its UI/export findings
+with `evidence.import`, preserving dates and coverage. Existing API access does not imply
+access to every UI-only report. If a competitor backlink question needs a third-party
+index, specify the domains, export limits and cost first; different indexes are not
+an exhaustive census and their authority scores are not Google ranking metrics.
+
+Turn unique useful data or research into a concrete resource relevant publishers
+can cite. Verify real mentions and source pages before proposing outreach. Drafting
+an outreach idea does not authorize sending it. Avoid paid link schemes, bulk
+unrelated guest posts and fabricated expertise or reviews.
+
+## Cadence and decisions
+
+- Daily: access/collection failures and technical regressions in a fixed small
+  cohort. Keep errors and missed-run gaps; do not overwrite the last good evidence.
+- Weekly: finalized GSC/GA opportunities, page/query intent checks and one small
+  prioritized improvement with evidence and effort described in plain terms.
+- Monthly or on reference-data publication: verify date/provenance accuracy,
+  sitemap policy, template consistency, internal-link coverage and useful content.
+- After deployments: run rendered/technical checks immediately; record deployment,
+  measurement and overlapping changes; compare later equal windows with proper
+  timezone and instrumentation gates. Leave outcomes pending or inconclusive when
+  evidence is not available. No unsupported uplift claims.
+
+Use an external scheduler for recurring collection, with bounded runs and private
+credentials/artifacts. Record successful run times and failures separately from
+traffic. Alerts should cite exact observations and changes; missing capped rows
+cannot establish a disappearance. Provider retention can make a missed collection
+unrecoverable. A scheduler on a sleeping/offline laptop is not an always-on service.
+
+Further guidance: [Google SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide),
+[helpful content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content),
+[structured data policies](https://developers.google.com/search/docs/appearance/structured-data/sd-policies),
+and [Google Trends data](https://support.google.com/trends/answer/4365533).

package/docs/snapshots.md

@@ -121,3 +121,10 @@ object keys sorted lexically and array order retained. These identify comparison
 inputs, not raw file bytes; whitespace changes in saved JSON do not change them. Keep source snapshots for their full requests,
 responses and metadata. Changes are descriptive and do not establish that an SEO
 edit caused traffic changes; Pagesight does not apply SEO edits automatically.
+
+## Read the evidence
+
+Run `pagesight assess --snapshot saved.json --format text` for a deterministic
+summary of report rows, measurement findings and unknowns. It retains references
+to the saved observations and makes no new provider calls. See
+[measurement and verification](measurement.md) for scope and examples.

package/docs/usage.md

@@ -26,23 +26,27 @@ GA evidence identifies the credential source variable, credential type, and serv
 account email when present. It never includes the token or private key. Aggregate
 results include a concise per-observation `summary` alongside full observations.
 
-| Operation                                    | Required inputs                                          |
-| -------------------------------------------- | -------------------------------------------------------- |
-| `discover`                                   | `url`; optional `providers` (`["gsc", "ga"]` by default) |
-| `bing.sites`                                 | None                                                     |
-| `bing.queries`, `bing.pages`, `bing.traffic` | `site`                                                   |
-| `gsc.sites`                                  | None                                                     |
-| `gsc.sitemaps`                               | `site`                                                   |
-| `gsc.inspect`                                | `site`, `url`                                            |
-| `gsc.report`                                 | `site`, `request`; optional `maxPages`                   |
-| `ga.accounts`                                | None                                                     |
-| `ga.property`, `ga.key-events`               | `property`                                               |
-| `ga.report`                                  | `property`, `request`; optional `maxPages`               |
-| `page`                                       | `url`                                                    |
-| `speed.psi`                                  | `url`; optional `strategy` (`mobile` or `desktop`)       |
-| `speed.crux`, `speed.history`                | `url`; optional `origin: true`, `formFactor`             |
-| `doctor`                                     | `config`                                                 |
-| `snapshot`                                   | `config`, `startDate`, `endDate`; optional `maxPages`    |
+| Operation                                    | Required inputs                                                            |
+| -------------------------------------------- | -------------------------------------------------------------------------- |
+| `discover`                                   | `url`; optional `providers` (`["gsc", "ga"]` by default)                   |
+| `bing.sites`                                 | None                                                                       |
+| `bing.queries`, `bing.pages`, `bing.traffic` | `site`                                                                     |
+| `gsc.sites`                                  | None                                                                       |
+| `gsc.sitemaps`                               | `site`                                                                     |
+| `gsc.inspect`                                | `site`, `url`                                                              |
+| `gsc.report`                                 | `site`, `request`; optional `maxPages`                                     |
+| `ga.accounts`                                | None                                                                       |
+| `ga.property`, `ga.key-events`               | `property`                                                                 |
+| `ga.realtime`                                | `property`, `request`; moving window, no offset                            |
+| `ga.report`                                  | `property`, `request`; optional `maxPages`                                 |
+| `page`                                       | `url`                                                                      |
+| `speed.psi`                                  | `url`; optional `strategy` (`mobile` or `desktop`)                         |
+| `speed.crux`, `speed.history`                | `url`; optional `origin: true`, `formFactor`                               |
+| `doctor`                                     | `config`                                                                   |
+| `crawl`                                      | `config`; optional crawl bounds and `inspectLimit` (see [crawl](crawl.md)) |
+| `investigate`                                | `config`, `url`, `startDate`, `endDate`; optional `maxPages`, `maxRows`    |
+| `assess`                                     | `snapshot`; optional `maxRows` (default 10, max 100)                       |
+| `snapshot`                                   | `config`, `startDate`, `endDate`; optional `maxPages`                      |
 
 `operationSchema` and `configSchema` are exported for typed validation. The MCP
 `observe` input uses the same schema. `page` observes fetched HTML, status,
@@ -50,6 +54,10 @@ redirects, canonical, robots directives, JSON-LD and a content hash. It does not
 execute browser JavaScript. The original MCP page tool retains its additional
 link, social-meta and contrast checks.
 
+See [measurement and verification](measurement.md) for saved-snapshot assessments,
+Realtime requests, freshness limits and repeatable browser checks.
+See [investigating one URL](investigation.md) for fresh exact-page search, organic and technical evidence.
+
 ## CLI
 
 ```sh

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pagesight",
-  "version": "0.18.0",
+  "version": "0.20.0",
   "description": "See your site the way search engines and AI see it.",
   "keywords": [
     "ai-crawlers",
@@ -38,10 +38,13 @@
   "scripts": {
     "start": "bun run src/index.ts",
     "typecheck": "tsc -p tsconfig.json",
-    "test": "bun test __tests__"
+    "test": "bun test __tests__ && bun test browser-tests"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.12.1",
+    "entities": "^8.1.0",
+    "ipaddr.js": "^2.5.0",
+    "playwright": "1.63.0",
     "saxes": "^6.0.0",
     "zod": "^3.24.4"
   },