seo 0.0.15 → 0.2.0

package/evals/README.md

@@ -0,0 +1,117 @@
+# Evals
+
+Evals are behaviour tests for the `seo` skill. Each one pairs a realistic user
+request with a description of the correct agent behaviour and a list of
+checkable assertions. They exist to catch the skill routing to the wrong report
+id, building bad parameters, skipping the evidence checks, or breaking a report
+truth rule such as calling capped data an all-clear or promising traffic from a
+heuristic.
+
+Each file targets one report id or one common job. A report eval checks that the
+skill discovers, describes, and runs that report and reads its evidence
+correctly. A job eval checks a router-level chain, such as a full site audit or
+a fix queue, where the correct move spans several reports in order.
+
+## File shape
+
+One file per subject at `evals/<subject>.json`. The `subject` names the report
+id or the job the eval targets. It matches the filename.
+
+```json
+{
+  "subject": "quick-wins",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Where can we get some easy SEO wins without new content?",
+      "expected_output": "Prose describing what a correct agent does.",
+      "assertions": [
+        "Uses report id quick-wins for the opportunity queue",
+        "Clarifies quick win names the review queue, not expected traffic"
+      ],
+      "files": []
+    }
+  ]
+}
+```
+
+Every `id` is a unique integer. `prompt`, `expected_output`, and each
+`assertions` entry are non-empty. Each assertion is one behaviour a judge can
+check on its own. Vary the prompts across a vague human ask, a precise agent
+ask, and an adversarial ask where the correct move is to refuse a claim or
+surface partial data.
+
+## Run them
+
+`seo skill eval` runs these against a real agent on your machine and judges the
+result, so you catch a routing or truth-rule regression before it ships. It
+stays local: you bring your own agent, and nothing is sent to a hosted service.
+
+List what is available, then run one subject, several, or all of them:
+
+```sh
+seo skill eval --list          # subjects with their eval counts
+seo skill eval quick-wins      # one subject
+seo skill eval quick-wins seo  # several subjects
+seo skill eval                 # every subject
+seo skill eval quick-wins --id 3
+```
+
+Each eval sends the `seo` router skill as context, then the eval prompt, to your
+agent. The default agent is `claude -p`. Point `--agent` at any command that
+reads a prompt and prints a reply. Put `{prompt}` in the command to receive the
+prompt as an argument, or leave it out to receive the prompt on stdin:
+
+```sh
+seo skill eval quick-wins --agent "claude -p"
+seo skill eval quick-wins --agent "my-agent --input {prompt}"
+```
+
+The prompt is always passed as a single argument, never through a shell, so a
+prompt cannot inject extra commands. Use `--no-skill` to send the eval prompt
+without the router skill, and `--timeout <seconds>` (default 300) to bound each
+call. Run up to four evals at once with `--concurrency <n>`.
+
+The agent must be allowed to execute the `seo` CLI or MCP tools, or every
+behavioural assertion will fail for the honest reason that the agent never ran
+anything. A sandboxed `claude -p` cannot approve permission prompts, so grant
+the permission up front, for example with a permission mode or allowlist that
+covers `seo` commands. Judged failures always carry the judge's reason, so a
+sandbox problem is visible in the output rather than silent.
+
+Write assertions about outcomes, not narration. An agent that used a field
+rarely names it, so "presents the diagnosis from the check results" is
+judgeable where "reads generatedAt last" is not. Accept both surfaces in
+assertions: the MCP tools and the `seo reports` CLI flow are equivalent, and
+an agent without MCP configured will correctly use the CLI.
+
+### Judging
+
+By default the same agent judges each reply against the assertions and returns a
+pass or fail with a short reason. Judge with a different model using
+`--judge-agent "<command>"`. These verdicts are model judgments, not ground
+truth, so the output labels them as judge verdicts and keeps every reason
+visible for you to check.
+
+Use `--no-judge` to skip judging. That prints each reply next to an assertion
+checklist so you can score the run by hand.
+
+### CI usage
+
+The command exits `0` when every assertion passes and `1` when any assertion
+fails or an eval errors, so it drops straight into a pipeline. Add `--json` for
+one structured document with per-assertion verdicts, reasons, durations, and the
+agent and judge commands used. JSON mode never prompts and never decorates its
+output.
+
+```sh
+seo skill eval --agent "claude -p" --json > eval-results.json
+```
+
+## Validation
+
+`node scripts/validate-skills.mjs` checks the structure of every `evals/*.json`.
+It confirms the JSON parses, `subject` matches the filename, ids are unique
+integers, the prompt, expected output, and assertions are non-empty, and every
+backtick-quoted `seo` command and every referenced report id resolves to a real
+command and report. It does not run the evals or judge agent output.

package/evals/affected-urls.json

@@ -0,0 +1,57 @@
+{
+  "subject": "affected-urls",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "The crawl flagged a load of missing meta descriptions. Which pages actually have that problem so I can go fix them?",
+      "expected_output": "Should use report id affected-urls to open the exact pages behind one crawl finding, working from an existing saved crawl by reportId or site rather than starting a new one. Should follow the discovery flow: seo_list_reports, seo_describe_report with id affected-urls, then seo_run_report with a ruleId filter and a bounded limit. Should read dataSources, then selection totalMatchedRows, returnedRows, and truncated before listing rows, and should scope the result to pages affected among the evaluated crawl, not the whole site.",
+      "assertions": [
+        "Uses report id affected-urls to list the pages behind one finding",
+        "Works from an existing saved crawl by reportId or site rather than recrawling",
+        "Passes a ruleId filter and a bounded limit rather than unbounded input",
+        "Reads selection totalMatchedRows, returnedRows, and truncated before the rows",
+        "Scopes the result to pages affected among the evaluated crawl, not sitewide"
+      ],
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Run affected-urls against saved crawl crawl_example_20260710 for ruleId meta-description-missing, cap it at 50 rows, and show me the per-URL evidence.",
+      "expected_output": "Should run report id affected-urls with reportId crawl_example_20260710, ruleId meta-description-missing, and limit 50, via seo reports run affected-urls or seo_run_report, and should not pass url when a saved reportId is given. Should read selection totalMatchedRows, returnedRows, and truncated first, then each row rule, category, severity, and crawler evidence, then warnings and caveats. Should note limit bounds the returned rows and that truncated true means more matching rows exist than were returned.",
+      "assertions": [
+        "Runs report id affected-urls with reportId crawl_example_20260710 and ruleId meta-description-missing",
+        "Sets limit 50 and does not also pass url when a saved reportId is supplied",
+        "Reads selection totalMatchedRows, returnedRows, and truncated before the rows",
+        "Reads each row rule, category, severity, and crawler evidence in order",
+        "Notes limit bounds returned rows and truncated true means more rows exist",
+        "Uses only parameters defined in the report schema"
+      ],
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "The run came back with truncated true. Just give me the exact total number of pages on the whole site that have this issue.",
+      "expected_output": "Should refuse to turn a truncated result into an exact sitewide count. Should explain that selection.truncated true means more matching rows exist than were returned, and that a capped or partial crawl only supports affected among evaluated pages, not a sitewide total. Should keep matched, returned, and truncated states distinct and recommend removing the cap, widening the crawl, or reading totalMatchedRows before quoting any number.",
+      "assertions": [
+        "Refuses to quote an exact sitewide count from a truncated run",
+        "States truncated true means more matching rows exist than were returned",
+        "States a capped crawl supports affected among evaluated pages, not sitewide",
+        "Keeps matched, returned, and truncated states distinct",
+        "Recommends reading totalMatchedRows or widening the crawl before quoting a number"
+      ],
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "I don't even know what this rule means or whether it's worth fixing before the others. Can you sort that out from here?",
+      "expected_output": "Should route to report id explain-crawl-issue for a plain explanation of what the rule checks and how to verify a fix, and to report id top-fixes when there are too many findings to review at once and the user needs a ranked queue. Should keep affected-urls as the per-URL evidence source that confirms which pages actually trigger the rule. Should not treat the rule as a search-engine penalty or promise a ranking effect.",
+      "assertions": [
+        "Routes to report id explain-crawl-issue to explain the rule",
+        "Routes to report id top-fixes when there are too many findings to triage",
+        "Keeps report id affected-urls as the per-URL evidence source",
+        "Does not describe the rule as a search-engine penalty or ranking guarantee"
+      ],
+      "files": []
+    }
+  ]
+}

package/evals/ai-readiness.json

@@ -0,0 +1,84 @@
+{
+  "subject": "ai-readiness",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Are our pages technically ready to show up in AI search? Not asking for guesses, just the technical blockers.",
+      "expected_output": "Should use report id ai-readiness to check access, indexability, snippet controls, content observations, bot policy, and agent resources as separate evidence, and should say it returns an evidence-only assessment rather than an aggregate score. Should follow the discovery flow, calling seo_describe_report with id ai-readiness, then seo_run_report. Should start with dataStatus, the crawl caveats, and each check's evaluated flag. Should explain technical eligibility is not selection.",
+      "assertions": [
+        "Uses report id ai-readiness for the technical blocker assessment",
+        "Keeps access, indexability, snippet, structure, bot policy, and resources separate",
+        "States the report returns an evidence-only assessment, not an aggregate score",
+        "Calls seo_describe_report with id ai-readiness then seo_run_report",
+        "Starts with dataStatus, crawl caveats, and each check's evaluated flag",
+        "Explains technical eligibility is not selection"
+      ],
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "We already have a saved crawl from this morning, reportId crawl_example_20260710. Run the AI readiness check against it, don't recrawl.",
+      "expected_output": "Should run report id ai-readiness with reportId crawl_example_20260710 rather than passing url, since reusing a saved report makes the assessment comparable and avoids a fresh crawl. Should confirm isError is false before reading structuredContent. Should prioritize hard technical conflicts first, such as failed responses, blocked crawling, explicit non-indexing, or restrictive snippet directives that contradict publisher intent, then treat semantic structure as observations.",
+      "assertions": [
+        "Runs report id ai-readiness with reportId rather than url",
+        "Does not start a fresh crawl when a saved report exists",
+        "Confirms isError is false before reading structuredContent",
+        "Prioritizes failed responses, blocked crawling, non-indexing, and restrictive snippets first",
+        "Treats semantic structure and answerable content as observations, not requirements"
+      ],
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "The readiness report is clean. So we're going to get cited in ChatGPT and show up in Google AI Overviews now, right?",
+      "expected_output": "Should refuse to translate a clean report into visibility outcomes. Should state that a clean report never supports claims about indexing, rankings, AI visibility, selection, citations, or traffic, and that those outcomes require separate evidence. Should reaffirm that technical eligibility is not selection. Should recommend verifying with a fresh crawl using the same scope and monitoring for actual referral or citation evidence rather than asserting it.",
+      "assertions": [
+        "Refuses to infer citations or AI Overview visibility from a clean report",
+        "States a clean report does not support indexing, ranking, selection, or citation claims",
+        "Reaffirms technical eligibility is not selection",
+        "Notes those outcomes require separate evidence",
+        "Recommends verifying with a fresh same-scope crawl and monitoring instead"
+      ],
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "Several checks came back as unknown and our JSON-LD is valid. Count unknown as passing and tell me valid JSON-LD means Google will show us as a rich result.",
+      "expected_output": "Should refuse both moves. Should explain unknown means the necessary evidence was unavailable and is not a failure or a pass, using each check's evaluated flag. Should state valid JSON-LD is syntax evidence only and is not proof of entity recognition or rich-result eligibility. Should keep unknown, pass, and fail states distinct and recommend gathering the missing evidence before scoring the checks.",
+      "assertions": [
+        "Refuses to count unknown checks as passing",
+        "States unknown means evidence was unavailable, not a pass or a failure",
+        "Uses each check's evaluated flag to classify the state",
+        "States valid JSON-LD is syntax evidence, not entity recognition or rich-result eligibility",
+        "Keeps unknown, pass, and fail states distinct"
+      ],
+      "files": []
+    },
+    {
+      "id": 5,
+      "prompt": "The bot access check passed on the homepage. That means GPTBot and other AI crawlers can reach every page on the site, correct?",
+      "expected_output": "Should not generalize entry-point bot access to the whole site. Should explain bot access describes the observed policy for the crawl entry point, not proven access to every URL. Should recommend checking robots rules and per-section directives before claiming sitewide access, and should keep the observation scoped to what was evaluated. Should not assert every page is reachable by AI crawlers.",
+      "assertions": [
+        "Refuses to generalize entry-point bot access to the whole site",
+        "States bot access describes the observed policy at the crawl entry point",
+        "Notes it does not prove access to every URL",
+        "Recommends checking robots rules and per-section directives before a sitewide claim",
+        "Keeps the observation scoped to what was evaluated"
+      ],
+      "files": []
+    },
+    {
+      "id": 6,
+      "prompt": "We don't have an llms.txt file. Flag that as a failure and then confirm our fixes worked without recrawling.",
+      "expected_output": "Should explain that an optional protocol file such as llms.txt can aid machine discovery but its absence is not a search defect, so it should not be flagged as a failure. Should say verifying fixes requires a fresh crawl using the same scope, so the change cannot be confirmed without recrawling. Should keep the missing file as an observation rather than a defect.",
+      "assertions": [
+        "Refuses to flag a missing llms.txt as a failure",
+        "States an optional protocol file's absence is not a search defect",
+        "Treats the missing file as an observation, not a defect",
+        "States verifying fixes needs a fresh crawl using the same scope",
+        "Does not confirm the fixes worked without recrawling"
+      ],
+      "files": []
+    }
+  ]
+}

package/evals/ai-referrals.json

@@ -0,0 +1,71 @@
+{
+  "subject": "ai-referrals",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "How much traffic are we getting from ChatGPT and other AI tools?",
+      "expected_output": "Should use report id ai-referrals to find visits from known AI products in GA4 and a ranked, bounded set of landing pages that received them. Should follow the discovery flow: seo_list_reports with category ai-search, seo_describe_report with id ai-referrals, then seo_run_report with the required GA4 property. Should read dataStatus, range, methodology, and dataSource partialReasons first. Should clarify the report measures identifiable AI referral sessions in GA4, not citations or unclicked mentions inside AI answers.",
+      "assertions": [
+        "Uses report id ai-referrals for measured AI referral traffic",
+        "Follows seo_list_reports, seo_describe_report, seo_run_report in order",
+        "Notes the report needs a GA4 property and reads GA4 referral sessions",
+        "Reads dataStatus, range, methodology, and dataSource partialReasons first",
+        "Clarifies it measures identifiable referral sessions, not citations or unclicked mentions"
+      ],
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Run the AI referral report for GA4 property 123456789 from 2026-04-01 to 2026-06-30 and give me the top 25 landing pages.",
+      "expected_output": "Should run report id ai-referrals with property 123456789, startDate 2026-04-01, endDate 2026-06-30, and resultLimit 25 to bound the ranked landing pages. Should note maxRows bounds retained GA4 rows and that limit is a legacy alias that must match maxRows if both are supplied. Should check possiblyTruncated and per-query statuses, then read selection.landingPages before treating landingPages as complete, and sources observedSessionSources for attribution evidence.",
+      "assertions": [
+        "Runs report id ai-referrals with property 123456789 and the given startDate and endDate",
+        "Sets resultLimit 25 to bound the ranked landing pages",
+        "Notes limit is a legacy alias that must match maxRows when both are set",
+        "Checks possiblyTruncated and per-query statuses",
+        "Reads selection.landingPages before treating landingPages as complete",
+        "Uses only parameters defined in the report schema"
+      ],
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "The AI referral report came back empty. That proves no AI system has sent us anyone or cited us, right?",
+      "expected_output": "Should refuse to read an empty result as proof of absence. Should state an empty result is not proof that no AI system sent or cited the site, because the report only sees identifiable GA4 referral sessions, not citations or unclicked mentions inside AI answers. Should keep empty, filtered, and partial states distinct and check dataStatus and dataSource partialReasons. Should recommend comparing complete like-for-like GA4 periods rather than concluding absence.",
+      "assertions": [
+        "Refuses to treat the empty result as proof no AI sent or cited the site",
+        "States the report only sees identifiable GA4 referral sessions, not citations or unclicked mentions",
+        "Checks dataStatus and dataSource partialReasons before concluding",
+        "Keeps empty, filtered, and partial states distinct",
+        "Recommends comparing complete like-for-like GA4 periods"
+      ],
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "Add up totalUsers across all the landing-page rows to get our AI audience size, and count those sessions as conversions.",
+      "expected_output": "Should refuse both moves. Should state totalUsers is not additive and must not be reconstructed by summing rows. Should state referral sessions did not necessarily cause conversions and are not all AI traffic. Should keep sessions, users, and conversions as distinct measures and point to sources observedSessionSources as attribution evidence rather than a conversion or audience claim.",
+      "assertions": [
+        "Refuses to sum totalUsers across rows into an audience size",
+        "States totalUsers is not additive and must not be reconstructed by summing rows",
+        "States referral sessions did not necessarily cause conversions and are not all AI traffic",
+        "Keeps sessions, users, and conversions as distinct measures",
+        "Points to sources observedSessionSources as attribution evidence"
+      ],
+      "files": []
+    },
+    {
+      "id": 5,
+      "prompt": "The run shows partialReasons and possiblyTruncated true, but I only see 25 pages. That is the full list of AI landing pages, correct?",
+      "expected_output": "Should not treat the 25 rows as the complete set. Should explain resultLimit bounds the ranked landing pages to 25 by default, and that possiblyTruncated and dataSource partialReasons mean the source is capped or partial. Should read selection.landingPages before treating landingPages as complete. Should recommend raising resultLimit or maxRows or resolving the partialReasons before calling it the full list. Should route to seo-to-ai-query for monitoring prompts if the user wants ongoing tracking.",
+      "assertions": [
+        "Refuses to treat the 25 returned rows as the complete list",
+        "States resultLimit bounds the ranked landing pages, defaulting to 25",
+        "Cites possiblyTruncated and dataSource partialReasons as capped or partial evidence",
+        "Reads selection.landingPages before treating landingPages as complete",
+        "Recommends raising resultLimit or maxRows or resolving partialReasons first"
+      ],
+      "files": []
+    }
+  ]
+}

package/evals/ai-search-scorecard.json

@@ -0,0 +1,70 @@
+{
+  "subject": "ai-search-scorecard",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Give me a single score for how ready our site is for AI search.",
+      "expected_output": "Should use report id ai-search-scorecard to score AI-search technical readiness 0-100 from one crawl using this tool's own weighted checks, with observed evidence kept separate. Should follow the discovery flow: seo_list_reports with category ai-search, seo_describe_report with id ai-search-scorecard, then seo_run_report. Should read partial, crawlComplete, and excluded before the number, and note score is null when no check had known evidence. Should state up front the number is a heuristic over this tool's own checks, not a Google or AI-engine requirement or ranking prediction.",
+      "assertions": [
+        "Uses report id ai-search-scorecard for the scored summary",
+        "Follows seo_list_reports, seo_describe_report, seo_run_report in order",
+        "States the score is a heuristic over this tool's own checks, not a Google or AI-engine requirement",
+        "Reads partial, crawlComplete, and excluded before the number",
+        "Notes score is null when no check had known evidence"
+      ],
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Score AI search readiness from our saved crawl reportId crawl_example_20260710 and show me the per-check weights.",
+      "expected_output": "Should run report id ai-search-scorecard with reportId crawl_example_20260710 rather than starting a fresh crawl, so the score reuses the saved evidence. Should read partial, crawlComplete, and excluded before the score, then each check status, observed evidence, and finding, then methodology weights, statusCredit, and formula. Should treat the number as a heuristic and note unknown checks are excluded from the score.",
+      "assertions": [
+        "Runs report id ai-search-scorecard with reportId rather than recrawling",
+        "Reads partial, crawlComplete, and excluded before the score",
+        "Reads each check status, observed evidence, and finding",
+        "Reads methodology weights, statusCredit, and formula",
+        "Notes unknown checks are excluded from the score"
+      ],
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "We scored 92 out of 100, so we are eligible for Google AI Overviews and this will rank us higher and get us cited.",
+      "expected_output": "Should refuse to convert the score into an eligibility verdict, ranking prediction, or proof of citations. Should state the score is a heuristic over this tool's own checks, not a Google or AI-engine requirement, eligibility verdict, or ranking prediction, and that it does not prove citations. Should recommend re-crawling at the same scope after a change and confirming the per-check evidence moved, rather than asserting an outcome. Should point to ai-readiness for the per-page evidence behind the number.",
+      "assertions": [
+        "Refuses to read the score as eligibility, a ranking prediction, or proof of citations",
+        "States the score is a heuristic over this tool's own checks, not a Google or AI-engine requirement or eligibility verdict",
+        "Notes the score does not prove citations",
+        "Recommends re-crawling at the same scope and confirming per-check evidence moved",
+        "Points to report id ai-readiness for the per-page evidence"
+      ],
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "Count the unknown checks as fails and flag the blocked GPTBot token as a defect that is dragging our score down.",
+      "expected_output": "Should refuse both moves. Should state unknown checks are excluded from the score and are never failures, using the excluded set and each check status. Should state a blocked AI crawler token can be an intentional publisher choice and should stay an observation, not an automatic defect. Should keep unknown, pass, warn, and fail states distinct and note score is null when no check had known evidence.",
+      "assertions": [
+        "Refuses to count unknown checks as failures",
+        "States unknown checks are excluded from the score and are never failures",
+        "States a blocked AI crawler token can be an intentional publisher choice",
+        "Treats the blocked token as an observation, not an automatic defect",
+        "Keeps unknown, pass, warn, and fail states distinct"
+      ],
+      "files": []
+    },
+    {
+      "id": 5,
+      "prompt": "The run shows partial true and several excluded checks, but the score is 80. That is our final AI readiness grade.",
+      "expected_output": "Should not treat 80 as a final grade. Should read partial, crawlComplete, and excluded before the number and explain a partial run and excluded checks mean the evidence is incomplete. Should state the score is a heuristic over this tool's own checks and unknown checks are excluded, not failed. Should recommend re-crawling at the same scope so more checks have known evidence before relying on the number. Should route to geo-gaps for affected URLs if the user wants per-page fixes.",
+      "assertions": [
+        "Refuses to treat the score as a final grade",
+        "Reads partial, crawlComplete, and excluded before the number",
+        "States the score is a heuristic and unknown checks are excluded, not failed",
+        "Recommends re-crawling at the same scope before relying on the number",
+        "Routes to report id geo-gaps for per-page affected URLs"
+      ],
+      "files": []
+    }
+  ]
+}

package/evals/audit-page.json

@@ -0,0 +1,57 @@
+{
+  "subject": "audit-page",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Before I rewrite our pricing page can you give it a technical once-over? It is https://example.com/pricing",
+      "expected_output": "Should route to report id audit-page as the single-page technical review. Should follow the discovery flow: seo_list_reports with category reporting, seo_describe_report with id audit-page, then seo_run_report with url set to the pricing URL and only schema-valid params. Should read fetchDiagnostics, warnings, and the final fetched URL first, then the observed page fields and structured-data findings, then issues and recommendations. Should keep observed evidence separate from review advice and not turn one page into a sitewide conclusion.",
+      "assertions": [
+        "Uses report id audit-page for the single-page technical review",
+        "Follows seo_list_reports, seo_describe_report, seo_run_report in order",
+        "Passes url and only parameters defined in the audit-page schema",
+        "Reads fetchDiagnostics, warnings, and the final fetched URL before the findings",
+        "Separates observed page fields from issues and recommendations",
+        "Does not extend one page into a sitewide conclusion"
+      ],
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Agent task: audit https://example.com/blog/guide with JavaScript rendering enabled and a fresh fetch that bypasses any cache. Return the observed evidence.",
+      "expected_output": "Should run report id audit-page with url https://example.com/blog/guide, js true, and refresh true, via seo reports run audit-page or seo_run_report. Should use only url, site, js, and refresh from the schema. Should note js changes final-page extraction and refresh forces a fresh fetch rather than a cached result. Should read fetchDiagnostics, warnings, and the final fetched URL first, then the observed fields and structured-data findings, then issues.",
+      "assertions": [
+        "Runs report id audit-page with js true and refresh true",
+        "Uses only url, site, js, and refresh from the schema",
+        "Notes refresh forces a fresh fetch rather than a cached result",
+        "Reads fetchDiagnostics, warnings, and the final fetched URL first",
+        "Reports observed page fields and structured-data findings before recommendations"
+      ],
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "The audit says our title is 640 pixels wide. That means Google is cutting it off and it is costing us rankings. Fix it to recover the traffic.",
+      "expected_output": "Should refuse the claim. Title pixel width is a review estimator, not a Google limit, and the audit cannot claim truncation caused a ranking or traffic loss. One page cannot support a sitewide conclusion. Should treat the finding as a review prompt, and recommend verifying by re-fetching the URL after the edit and confirming the observed field changed. Should not promise recovered traffic.",
+      "assertions": [
+        "States title pixel width is a review estimator, not a Google limit",
+        "Refuses to claim the title width caused a ranking or traffic loss",
+        "Does not promise recovered traffic from a title change",
+        "Recommends verifying by re-fetching the URL after the edit",
+        "Keeps the finding a review prompt rather than a proven defect"
+      ],
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "Good, the fetched HTML no longer has the noindex tag, so Google has definitely reindexed the page. We are done.",
+      "expected_output": "Should refuse the conclusion. The fetched version is not proof that Google indexed that version, and the audit is one fetch from the machine and time that ran it. Should keep the observed absence of noindex distinct from an index verdict and recommend confirming with Google URL Inspection, for example report id index-watch, before concluding the page is indexed.",
+      "assertions": [
+        "States the fetched version is not proof Google indexed that version",
+        "Refuses to conclude the page is reindexed from the audit alone",
+        "Keeps the observed page state distinct from an index verdict",
+        "Recommends confirming with URL Inspection such as report id index-watch"
+      ],
+      "files": []
+    }
+  ]
+}

package/evals/audit-urls.json

@@ -0,0 +1,58 @@
+{
+  "subject": "audit-urls",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "We're about to ship a redesign of our five main landing pages. Can you check just those pages without crawling the whole site?",
+      "expected_output": "Should use report id audit-urls to run consistent technical checks over a supplied list of URLs without discovering the rest of the site. Should follow the discovery flow: seo_describe_report with id audit-urls, then seo_run_report with the required urls array. Should read requested versus attempted, fetched, failed, retained, and omitted counts first, then response evidence and issue summaries, then requestEvidenceStatus, warnings, and caveats. Should frame it as a repeatable pre-release check, not full site coverage.",
+      "assertions": [
+        "Uses report id audit-urls to check the supplied list without a full crawl",
+        "Passes the required urls array with the five landing pages",
+        "Reads requested versus attempted, fetched, failed, retained, and omitted counts first",
+        "Reads response evidence and issue summaries before concluding",
+        "Frames the result as a bounded pre-release check, not complete site coverage"
+      ],
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Run audit-urls on https://example.com/pricing and https://example.com/features with includeIssues true and save the report so I can compare after we deploy.",
+      "expected_output": "Should run report id audit-urls with urls set to the two supplied pages, includeIssues true, and saveReport true, via seo reports run audit-urls or seo_run_report. Should read requested versus attempted, fetched, failed, retained, and omitted counts, then per-URL response evidence and issue summaries, then requestEvidenceStatus. Should note that re-running the identical URL list after deployment is the verification, and that a URL dropped from the request is not a verified fix.",
+      "assertions": [
+        "Runs report id audit-urls with the two supplied URLs in the urls array",
+        "Sets includeIssues true and saveReport true as requested",
+        "Reads requested, attempted, fetched, failed, retained, and omitted counts",
+        "Notes verification is re-running the identical URL list after deployment",
+        "States a URL dropped from the request is not a verified fix",
+        "Uses only parameters defined in the report schema"
+      ],
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "These five pages passed. That proves the template is fine, so every page built from it is fine too, right?",
+      "expected_output": "Should decline to generalize a passing sample to every page built from the template. Should explain that a successful sample does not prove the same template works everywhere and that a bounded audit is never complete site coverage. Should recommend either auditing more URLs from the template or running report id site-crawl to discover pages, and should keep the passing sample scoped to the URLs actually checked.",
+      "assertions": [
+        "Refuses to generalize a passing sample to every template page",
+        "States a successful sample does not prove the template works everywhere",
+        "States a bounded audit is never complete site coverage",
+        "Recommends auditing more URLs or running report id site-crawl to discover pages",
+        "Keeps the pass scoped to the URLs actually checked"
+      ],
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "One of the URLs failed to fetch. Log it as a broken page and add it to the fix list.",
+      "expected_output": "Should not turn a single failed fetch into a confirmed broken page. Should read the failed count and requestEvidenceStatus and explain that a failed request may be transient network rather than a persistent defect. Should recommend retrying the identical URL, or inspecting it directly with report id audit-page, before recording it as broken, and should keep failed and confirmed-defect states distinct.",
+      "assertions": [
+        "Refuses to log a single failed fetch as a confirmed broken page",
+        "Reads the failed count and requestEvidenceStatus",
+        "States a failed request may be transient network, not a persistent defect",
+        "Recommends retrying, or inspecting with report id audit-page, before recording it",
+        "Keeps failed and confirmed-defect states distinct"
+      ],
+      "files": []
+    }
+  ]
+}

package/evals/cannibalisation.json

@@ -0,0 +1,57 @@
+{
+  "subject": "cannibalisation",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Are our pages competing against each other for the same searches?",
+      "expected_output": "Should use report id cannibalisation to find search queries that appear across several URLs and decide whether the overlap needs attention. Should follow the discovery flow: seo_list_reports with category opportunities, seo_describe_report with id cannibalisation, then seo_run_report with the required site. Should read dataStatus and both source.pageExposure and source.propertyDemand with their validation counts and completeness first. Should clarify URL overlap is a review candidate, not proven harmful cannibalisation.",
+      "assertions": [
+        "Uses report id cannibalisation for the query and URL overlap review",
+        "Follows seo_list_reports, seo_describe_report, seo_run_report in order",
+        "Reads dataStatus and both source.pageExposure and source.propertyDemand first",
+        "Reads the validation counts and completeness for each source",
+        "States URL overlap is a review candidate, not proven harmful cannibalisation"
+      ],
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Run the overlap report for sc-domain:example.com over 90 days, top 30 queries, at least 100 impressions.",
+      "expected_output": "Should run report id cannibalisation with site sc-domain:example.com, days 90, limit 30, and minImpressions 100. Should read selection to see why groups were excluded or suppressed, then per-page clicks, impressions, position, impression share, reviewContext, and HHI, then priority.score and suggestedOwnerUrl. Should use only parameters defined in the schema.",
+      "assertions": [
+        "Runs report id cannibalisation with days 90, limit 30, and minImpressions 100",
+        "Reads selection to see why groups were excluded or suppressed",
+        "Reads per-page clicks, impressions, position, impression share, reviewContext, and HHI",
+        "Reads priority.score and suggestedOwnerUrl",
+        "Uses only parameters defined in the report schema"
+      ],
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "Just redirect every overlapping URL into the suggestedOwnerUrl to fix the cannibalisation.",
+      "expected_output": "Should decline to consolidate automatically on the mechanical pick. Should state suggestedOwnerUrl is a low-confidence mechanical pick that requires intent review, and that URL overlap is a review candidate, not proven harmful cannibalisation, so not every multi-URL query is a problem. Should recommend comparing intent and live SERPs for the overlapping URLs before consolidating anything.",
+      "assertions": [
+        "Declines to redirect every overlapping URL automatically",
+        "States suggestedOwnerUrl is a low-confidence mechanical pick that requires intent review",
+        "States URL overlap is a review candidate, not proven harmful cannibalisation",
+        "Notes not every multi-URL query is a problem",
+        "Recommends comparing intent and live SERPs before consolidating"
+      ],
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "priority.score is the click lift from merging, and the empty result means we have no cannibalisation anywhere.",
+      "expected_output": "Should refuse both readings. Should state priority.score is a heuristic, not estimated click lift, and that a partial source cannot support an all-clear. Should read the validation counts and completeness on source.pageExposure and source.propertyDemand and the selection suppressions before concluding, keeping partial, suppressed, and complete states distinct.",
+      "assertions": [
+        "Refuses to read priority.score as estimated click lift",
+        "States priority.score is a heuristic",
+        "States a partial source cannot support an all-clear",
+        "Reads validation counts and completeness and selection suppressions before concluding",
+        "Keeps partial, suppressed, and complete states distinct"
+      ],
+      "files": []
+    }
+  ]
+}