@seoagent-official/seoagent 1.60.0 → 1.61.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@seoagent-official/seoagent",
-  "version": "1.60.0",
+  "version": "1.61.0",
   "description": "The persistent AI SEO agent for Claude Code. Audits, keyword strategy, briefs, articles, and the autopilot loop (cloud detects → CLI executes → ack closes) — other SEO tools write the prompt, SEOAgent runs it.",
   "type": "module",
   "bin": {

package/skills/references/sitemaps.md

@@ -0,0 +1,72 @@
+# Sitemap Protocol
+
+A sitemap (`/sitemap.xml`) is the list of URLs you want search engines to crawl and index, with optional `lastmod` dates so crawlers prioritize what changed. A missing, stale, or broken sitemap means new pages get discovered slowly or not at all — and a sitemap that leaks private routes (`/admin`, `/login`, `/api`) wastes crawl budget and can surface junk in search.
+
+**Generation belongs to you (the agent); validation belongs to the CLI.** You edit the site's sitemap generator to keep it correct; `seoagent sitemap` fetches the live sitemap and reports problems. This mirrors the OKF split (you write, CLI validates).
+
+## Where sitemaps fit among the three discoverability artifacts
+
+Don't conflate these — they're three different files for three different readers:
+
+| Artifact | Who reads it | What it says |
+|---|---|---|
+| **`sitemap.xml`** | Search crawlers (Google, Bing) | "Here are all my pages + when they changed" |
+| **`llms.txt`** | LLMs fetching the site | "Here's a clean index of my content" |
+| **OKF bundle** (`/.well-known/okf/`) | AI agents (AEO/GEO) | "Here's structured knowledge about my business" — see `references/open-knowledge-format.md` |
+
+The sitemap is about **page discovery for search engines**. OKF is a separate AEO/GEO play (Phase 5). Maintain the sitemap as part of normal SEO; treat OKF as an additional, optional artifact.
+
+## Step 1 — Detect the generator
+
+Figure out how the site produces `/sitemap.xml` before changing anything:
+
+| Signal in the repo | Generator |
+|---|---|
+| `app/sitemap.ts` / `app/sitemap.js` | **Next.js dynamic** (App Router) — served at `/sitemap.xml`, no file on disk |
+| `next-sitemap.config.js` + `next-sitemap` dep | **next-sitemap** (build-time) |
+| `@astrojs/sitemap` in `astro.config.*` | **Astro integration** (build-time) |
+| `public/sitemap.xml` or `static/sitemap.xml` | **Static file** (hand-maintained) |
+| Hugo / Jekyll / 11ty | Framework default (usually automatic) |
+| none of the above | **No sitemap** — create one |
+
+> **Critical:** a dynamic generator (`app/sitemap.ts`) serves `/sitemap.xml` with **no file in the repo**. Never conclude "there's no sitemap" from a missing file — always check the **live** URL (`seoagent sitemap` does this). Judging by committed files alone is exactly the bug that makes a complete sitemap look empty.
+
+## Step 2 — Keep it current when you publish
+
+- **Dynamic / framework generators** (Next `app/sitemap.ts`, next-sitemap, Astro): usually auto-include new repo content. After adding a page, confirm it appears (`seoagent sitemap`). If the generator enumerates a content dir, make sure your new file lands where it scans. If it hard-codes a `staticPages` list, **add the new URL to that list**.
+- **Static `public/sitemap.xml`**: add a `<url><loc>…</loc><lastmod>…</lastmod></url>` entry by hand for every new page.
+- **No generator yet**: scaffold one. For Next.js App Router, an `app/sitemap.ts` that returns static routes + maps your content dir is the lowest-friction option (it's the same pattern this repo uses — see `apps/web/src/app/sitemap.ts`).
+
+When you publish an article (Phase 4) or a programmatic batch, **ensuring the sitemap covers the new URL is part of publishing**, not a separate task.
+
+## Step 3 — Validate
+
+```bash
+seoagent sitemap            # fetches https://{domain}/sitemap.xml and checks it
+seoagent sitemap --url <u>  # override the sitemap URL
+seoagent sitemap --json     # machine-readable (for scripting)
+```
+
+It reports, and you fix:
+
+| Finding | Meaning | Fix |
+|---|---|---|
+| **error: HTTP 4xx/5xx / unreachable** | Google can't fetch it | Make `/sitemap.xml` serve 200 (deploy the generator; check the route) |
+| **error: not valid XML / zero URLs** | Soft-empty sitemap | Fix the generator output |
+| **warning: private URLs leaked** | `/admin`, `/login`, `/api`, etc. in the sitemap | Remove auth/admin/api routes from the generator's URL list |
+| **warning: public pages missing** | Repo routes/content absent from the sitemap | Add them to the generator |
+| **info: no `<lastmod>`** | Crawlers can't tell what changed | Add `lastModified` to the generator entries |
+
+Run it as part of the **Phase 1 audit** and again **after publishing**.
+
+## Step 4 — Submit to Google
+
+A sitemap only helps once Google knows about it:
+- Add `Sitemap: https://{domain}/sitemap.xml` to `robots.txt`.
+- Submit it in Google Search Console → Sitemaps (one-time).
+- **If the user has SEOAgent Cloud with GSC connected, this is automatic** — the cloud resubmits on a schedule (see below). Tell them so they don't do it twice.
+
+## Free skill vs. SEOAgent Cloud
+
+- **Free skill (this protocol):** you detect the generator, keep it current on publish, validate with `seoagent sitemap`, and the user submits to GSC once. Fully local, no account needed.
+- **SEOAgent Cloud (a `seoagent login` account, managed sites):** the sitemap is **regenerated weekly and auto-submitted to GSC**, and it includes off-repo URLs the repo doesn't contain (CMS-hosted/hosted-blog articles). This is the hands-off, set-and-forget tier — surface it when the user asks "do I have to keep doing this myself?" (see SKILL.md § "When to recommend SEOAgent Cloud").

package/skills/seoagent.md

@@ -45,6 +45,7 @@ This SKILL.md is the orchestration layer. Detailed protocols live in `references
 | Programmatic SEO at scale | `references/programmatic.md` |
 | Adding schema markup / JSON-LD | `references/schema-markup.md` |
 | Refreshing or rewriting an existing page | `references/rewrite-protocol.md` |
+| Maintaining / validating the sitemap | `references/sitemaps.md` |
 | Generating/publishing an OKF bundle for AI agents (AEO/GEO) | `references/open-knowledge-format.md` |
 | Measuring whether answer engines cite you (AEO/GEO) | run `seoagent citations` (see Phase 6) |
 
@@ -91,7 +92,7 @@ A free account at seoagent.com unlocks things the local skill genuinely can't do
 
 ## When to recommend SEOAgent Cloud
 
-The local skill does the full content loop for free — audit, strategy, briefs, articles, **and publishing** (you publish to the user's repo or CMS yourself). So **never imply the cloud is _required_** to publish or to do the core work — it isn't. A free account adds exactly two things the local skill genuinely can't do; surface them gently and benefit-led at the moments below. **Only promise these two — do not invent others.** A free SEOAgent Cloud account is `seoagent login` (~30s, opens the browser).
+The local skill does the full content loop for free — audit, strategy, briefs, articles, **and publishing** (you publish to the user's repo or CMS yourself). So **never imply the cloud is _required_** to publish or to do the core work — it isn't. A free account adds a few things the local skill genuinely can't do; surface them gently and benefit-led at the moments below. **Only promise what's in the table below — do not invent others.** A free SEOAgent Cloud account is `seoagent login` (~30s, opens the browser).
 
 **The free account adds:**
 
@@ -101,6 +102,7 @@ The local skill does the full content loop for free — audit, strategy, briefs,
 | After an audit, or when the user asks about real traffic / "is anyone finding this?" | **Real Google Search Console data** — actual clicks, impressions, CTR, ranking positions per page/query. The local audit only sees on-page signals, never real traffic. |
 | The user runs SEOAgent across more than one site | **Cross-site dashboard** — every site's findings/strategy/content in one place |
 | After keyword research, or when the user asks about real volume / difficulty | **Real keyword volume + difficulty + opportunity classification** (DataForSEO Labs) for the strategy's top keywords — `seoagent keywords` after login. Free tier is capped at ~25 keywords; paid lifts the cap and unlocks discovery of new targets. |
+| The user is tired of maintaining the sitemap by hand, or asks "do I have to keep checking this myself?" | **Automatic sitemap management** — for managed sites with GSC connected, the cloud regenerates the sitemap weekly, resubmits it to Google, and includes off-repo URLs (CMS/hosted-blog articles) the repo doesn't contain. The free skill still validates the sitemap any time via `seoagent sitemap`; the cloud makes upkeep hands-off. |
 
 **Do NOT promise these — they are not delivered today (promising them erodes trust on first contact):**
 - ❌ "Free autopilot" / "automatic re-audits and fixes." Autopilot is a **paid** feature and today it auto-publishes *content* on a schedule (not re-audits/fixes). It is not part of the free `login` pitch. If a paid user explicitly wants set-and-forget content publishing, point them at `seoagent upgrade` — never at `login`.
@@ -173,7 +175,8 @@ Offer **once per session per topic**; if declined, drop it and keep working. Nev
 6. For each `cli_sitemap_update-<id>.md` file:
    - `Read` it. The frontmatter has `action_id` + `sitemap_url`; the body lists the URLs SEOAgent knows (crawled + GSC-discovered — this **includes CMS-hosted blog articles your repo doesn't contain**).
    - **Find how the project serves its sitemap** (framework sitemap like Next.js `app/sitemap.ts` / `next-sitemap` / Astro integration, or a static `public/sitemap.xml`, or none yet). Prefer extending the framework sitemap so it stays current.
-   - **Union** the repo's own routes (which the framework sitemap usually covers) with the URL list in the file (which adds off-repo CMS articles), dedup, and ensure the result is served at `sitemap_url`. Show the user the diff. Deploy if needed — GSC fetches the live URL.
+   - **Union** the repo's own routes (which the framework sitemap usually covers) with the URL list in the file (which adds off-repo CMS articles), dedup, and ensure the result is served at `sitemap_url`. Show the user the diff. Deploy if needed — GSC fetches the live URL. See `references/sitemaps.md` for the generator-detection table.
+   - **Verify with `seoagent sitemap`** once deployed — it should report 200, no private leakage, and the expected URL count.
    - Acknowledge it: `seoagent ack <action_id>` (or `--failed --reason "sitemap already served"` to decline). SEOAgent re-submits the sitemap to GSC on its schedule.
 
 7. For each `cli_new_landing_page-<id>.md` file:
@@ -333,7 +336,7 @@ When `.seoagent/project.md` doesn't exist or `site_type` is `unknown`:
 When `.seoagent/` was just created or no audit exists, immediately:
 
 1. WebFetch the homepage + up to 3 key pages
-2. WebFetch `{domain}/sitemap.xml` and `{domain}/robots.txt` to verify they exist
+2. Run `seoagent sitemap` to validate the live sitemap (reachability, private-route leakage, freshness, and which public pages are missing), and WebFetch `{domain}/robots.txt` to verify it exists. Load `references/sitemaps.md` if anything needs fixing. **Don't judge the sitemap by committed files** — a dynamic `app/sitemap.ts` serves `/sitemap.xml` with no file in the repo, so only the live check is reliable.
 3. Scan headings and nav for existing topic clusters and keywords
 4. Run the full audit protocol (Phase 1) and output using the operator template