@bettercms-ai/mcp 0.68.1 → 0.69.0

package/dist/skills/bettercms/references/seo.md

@@ -1,6 +1,6 @@
 ---
 title: "SEO and AEO"
-description: "Native page and entry meta, Markdown twins, route audit, structured data, sitemap and robots, AEO, validators and a generated OG image route."
+description: "Native page and entry meta, carrying over a source site's meta, redirects, Markdown twins, route audit, structured data by content type, sitemap, robots and AI crawlers, AEO, validators, a generated OG image route and the SEO/AEO gate."
 ---
 
 # SEO and AEO
@@ -12,16 +12,27 @@ no one can guarantee.
 Doctrine: SEO is native and never modelled (§5b of `bettercms://playbook/schema`); flow:
 `seo_optimize` prompt.
 
+## Contents
+
+- 1. Where each value lives (carry over the source's meta, Markdown twins)
+- 2. Redirects and URLs
+- 3. Route-level audit
+- 4. Structured data (content to schema.org type, rules)
+- 5. Sitemap, robots, llms.txt (AI crawlers)
+- 6. AEO and content quality
+- 7. Publish and verify (validators, generated OG image)
+- SEO and AEO gate
+
 ## 1. Where each value lives
 
 | Value | Writer | Notes |
 |---|---|---|
 | Page title, description, `schemaType`, `schema`, noindex, canonical, OG, Twitter | `update_page` | Merges into what is stored: an absent key keeps its value, `null` or `""` clears it |
-| Entry meta (title, description, noindex, canonical, OG, Twitter, schema) | `update_content_entry` with `meta` | Same merge rule |
+| Entry meta (title, description, noindex, canonical, OG, Twitter, `schemaType`, `schema`) | `update_content_entry` with `meta` | Same merge rule |
 | Site defaults (`seoDefaults`: title, description, OG image, Twitter handle, `og`, `twitter`, `collectionSchemaTypes`), site JSON-LD (`siteSchema`), `robotsConfig`, `sitemapConfig`, `rssConfig`, `indexnowConfig`, `searchConfig` | `update_seo`, read with `get_seo` | Each key REPLACES its whole object, so read and send it complete. `collectionSchemaTypes` maps a collection's model id to its entries' schema.org type; a key the plan lacks answers 403 `FEATURE_LOCKED` |
 | Suggestions | `generate_seo_meta` | Returns a suggestion only; apply it with the writers above |
-| Audit | `list_seo_issues` | Computed live from published pages; nothing is stored |
-| Redirects | `list_redirects`, `create_redirect`, `update_redirect`, `delete_redirect` | 301, 302, 307 or 308; 301 when omitted |
+| Audit | `list_seo_issues` | Computed live from published pages; nothing is stored. Also reports accessibility issues as `a11y.` codes (image alt text, heading order, generic link text, video transcripts), each naming the page and its block ids |
+| Redirects | `list_redirects`, `create_redirect`, `import_redirects`, `update_redirect`, `delete_redirect` | 301, 302, 307 or 308; 301 when omitted. 410 takes no destination: the path answers Gone and leaves the sitemap. For a list of rules use `import_redirects` (rows, CSV, a Netlify `_redirects` file or a vercel.json) with `dryRun: true` first; each row comes back created, existing or skipped with its reason |
 
 Never create `seoTitle`-style fields on a model, page or component: the renderer, the sitemap,
 `llms.txt` and `list_seo_issues` read the native values only. Self-hosted sites own their
@@ -31,6 +42,30 @@ There is no hreflang field: translations are native rows (`locale`, `translation
 sitemap emits the hreflang alternates from them; a frontend emits the `<head>` links from each
 entry's delivered `translations` ([localization.md](localization.md)).
 
+### Carry over the source's meta
+
+On an import or conversion, read every source page's `<head>` before writing. Copy published
+values as they are; rewriting copy is a separate, reported recommendation.
+
+| Source | Native writer and key | Rule |
+|---|---|---|
+| `<title>` | `metaTitle` (`update_page`; `meta.metaTitle` on an entry) | Copy verbatim. Report one over 60 characters; `list_seo_issues` flags missing, over 60 and duplicates |
+| `<meta name="description">` | `metaDescription` | Copy verbatim. Aim for 120–160 characters; the audit flags only a missing one and one over 160. When the source has none, take a `generate_seo_meta` suggestion and list it for review |
+| `og:title`, `og:description`, `og:image`, `og:type` | `og.title`, `og.description`, `og.image`, `og.type` | Write a key only when it differs from the meta title, the description or the site default. Upload the image first and store its media URL |
+| `twitter:card`, `twitter:title`, `twitter:description`, `twitter:image` | `twitter.card`, `twitter.title`, `twitter.description`, `twitter.image` | Same rule as OG |
+| `twitter:site` | `update_seo` `seoDefaults.twitterHandle` | Once per site |
+| A site-wide share image | `update_seo` `seoDefaults.ogImage` | Then pages inherit it and `og.image` stays empty |
+| Canonical pointing at the page itself | nothing | A hosted page gets its own canonical |
+| Canonical pointing at another URL | `canonical` | Keep it: someone consolidated those URLs on purpose |
+| `<meta name="robots" content="noindex">` | `noindex: true` | Also drops the route from the sitemap, `llms.txt`, its Markdown twin and site search |
+| `nofollow`, `noarchive`, `max-snippet` and other robots directives | nothing | No native key stores them. Report each page that used one |
+| `<link rel="alternate" hreflang>` | nothing on the page; one row per language with `locale` and `translationOf` | Pair the languages by URL and structure, never by matching text ([localization.md](localization.md)) |
+| JSON-LD on a page | nothing | Read it as evidence and compare it with what BetterCMS emits (§4) |
+| Organization JSON-LD on the home page | `update_seo` `siteSchema` | Keep name, URL, logo and `sameAs`; drop any value the site does not show |
+| robots.txt | `update_seo` `robotsConfig` | Stay on `auto` unless the source has rules worth keeping, then `custom` with that text (§5) |
+| sitemap.xml | `update_seo` `sitemapConfig` | Stay on `auto`. URLs the CMS does not own (a docs path, a static app) go in `extraUrls` |
+| RSS or Atom feed | `update_seo` `rssConfig` `{ enabled: true }` | Item links come from the collection's `urlPattern` |
+
 ### Markdown twins
 
 Every published page and entry is also served as Markdown for answer engines, at its own URL plus
@@ -44,7 +79,33 @@ Every published page and entry is also served as Markdown for answer engines, at
 - `llms.txt`, `llms-full.txt` and which pages serve a twin are switched there too. Those writes
   need the SEO permission, and no MCP tool writes them.
 
-## 2. Route-level audit
+## 2. Redirects and URLs
+
+Keep every URL that ranks or is linked to. Change a path only when BetterCMS cannot express it.
+
+1. Build a URL map first: each source URL (from the sitemap and a crawl) beside its new path.
+2. Every row whose path changes gets one `create_redirect` (301) aimed at the FINAL URL, never at
+   an intermediate one. Chains are collapsed for you; a loop answers 422 and a source path that
+   already has a rule answers 409.
+3. Import the source's own rules as well: a `_redirects` file, `vercel.json`, `netlify.toml`,
+   `.htaccess` or a redirect plugin's export.
+4. Never send a batch of removed pages to the home page. Redirect each to its closest equivalent,
+   or retire it with a 410.
+5. 302 or 307 is for a temporary move, 308 when the request method must survive.
+6. A page gone for good with no equivalent takes `redirectType: "410"` and no destination. List
+   those in the report.
+
+| Case | Do |
+|---|---|
+| Slug shape | Lowercase letters, digits and hyphens. Strip `.html`, a trailing slash and capitals, each with a 301. Keep the source's words: a slug is not copy to improve |
+| Collection URLs | Set `urlPattern` (`/blog/:slug`): a fixed prefix and `:slug` exactly once |
+| Nested (`/:category/:slug`), dated (`/2024/05/post`) or query-string (`?p=123`) URLs | The pattern cannot say these. Map each to the flat pattern and add one 301 per old URL |
+| Two sources want one slug | The one with more links or traffic keeps it. Give the other a suffix from its own words (`-guide`, the city), add its 301 and report it. A taken model slug answers 409 (code slug_taken) |
+| A page or entry slug changed in BetterCMS | The 301 is written by the same save (an entry needs its collection's `urlPattern`). Do not add a second one |
+| A URL folder renamed or moved | 301s are written for every page under it. Do not add them again |
+| A closed job, a past event, a sold-out product | Keep the URL and mark the item closed in its content. Do not delete it |
+
+## 3. Route-level audit
 
 Read the route inventory, page and entry records, site defaults, the rendered `<head>`, robots,
 sitemap, structured data and redirects. For each indexable route check:
@@ -58,18 +119,81 @@ sitemap, structured data and redirects. For each indexable route check:
 
 A CMS field holding the right title does not prove the public route emits it. Fetch the HTML.
 
-## 3. Structured data
+## 4. Structured data
+
+`list_seo_issues` reports a pasted `schema` that lacks its type's required properties as
+`schema.invalid`, naming the type and what is missing.
+
+Choose in this order, and stop at the first that expresses the page:
+
+1. **Automatic.** With no `schemaType` and no `schema`, JSON-LD is built from the content at
+   render time and follows every edit: WebSite on the home page (plus Organization when the site
+   has a name and a logo, or `siteSchema` when one is stored), Blog on a list of posts,
+   CollectionPage on any other list, BlogPosting on a post entry, Product on an entry of a
+   collection with a price or SKU, WebPage everywhere else. FAQPage is never automatic. A hosted
+   page or entry also gets a BreadcrumbList beside its main type: Home, each ancestor path that
+   is itself a published page (a folder with no page is skipped), then the document.
+2. **`schemaType`.** One of nineteen: WebPage, AboutPage, ContactPage, CollectionPage, Blog,
+   BlogPosting, Article, Product, FAQPage, Event, Organization, LocalBusiness, JobPosting, Course,
+   Recipe, HowTo, Service, Person, VideoObject. Its properties are filled from the document's own
+   fields by key (a job's `company`, a recipe's `ingredients`), and a property the content lacks
+   is left out. A type missing what it cannot do without (a JobPosting with no company or posting
+   date, a Recipe with no image, a HowTo with no steps) is not emitted: the document keeps its
+   automatic type until the field is filled. Clear it with `"auto"`
+   on `update_page` or `null` in an entry's `meta`. For a whole collection, set
+   `seoDefaults.collectionSchemaTypes` once instead of typing every entry.
+3. **Pasted `schema`.** Only when neither of the above can express the type AND every value in it
+   is visible on the page. It wins over both and does not follow content edits, so it goes stale.
+
+JSON-LD found on the source site is evidence, not content. Use it to fill fields (event dates,
+opening hours, prices) and to check that the emitted type and key values match. Do not paste it
+back. Never add a second copy through custom head code.
+
+### Content to schema.org type
+
+| Source content | Type | How in BetterCMS |
+|---|---|---|
+| The whole site | Organization, WebSite | Automatic on the home page; `siteSchema` to add `sameAs` or replace it |
+| Blog or journal post | BlogPosting, or Article | Automatic for a collection with a body that is named for posts or has a date and an author; else `collectionSchemaTypes` |
+| List of posts, any other list | Blog, CollectionPage | Automatic |
+| Product | Product with an Offer | Automatic when the model has a price or SKU; else `schemaType` |
+| FAQ: visible questions with answers | FAQPage | `schemaType: "FAQPage"`. It reads the first repeater whose rows hold a question and an answer |
+| Event | Event | `schemaType` (name, start, end, location read from the fields) |
+| Company, store, restaurant, office | Organization, LocalBusiness | `schemaType`; a subtype such as Restaurant, a `PostalAddress`, geo or opening hours need pasted `schema` |
+| About or contact page | AboutPage, ContactPage | `schemaType` |
+| Breadcrumb trail | BreadcrumbList | Automatic on hosted pages (ancestors that have a page). Paste `schema` only to match a trail the page shows differently |
+| News, recipe, job, video, course, podcast episode, how-to, author profile | NewsArticle, Recipe, JobPosting, VideoObject, Course, PodcastEpisode, HowTo, ProfilePage | Pasted `schema`, visible values only |
+| Product with on-page reviews | AggregateRating, Review | Pasted `schema`, with the average and count the page displays |
+
+### Rules
+
+- Every value must be on the page. No invented rating, review count, price, availability, salary,
+  author or date. What the page does not show is left out.
+- One FAQPage per URL and one main entity per page. Two FAQ sections on one page still make one
+  FAQPage. An accordion of features is not a FAQ.
+- `AggregateRating` needs a product or another reviewable subject. Never on the home page about
+  the business itself, and never typed by hand.
+- Awards go in `award`; they are not certifications.
+- Dates are ISO 8601 with a time zone offset.
+- Cart, checkout, 404 and legal pages stay on Automatic (plain WebPage). There is no off switch;
+  never give them a richer type.
+- Pasted JSON-LD is stored as given and is not checked for required properties. Check them
+  yourself before saving:
+
+| Type | Required |
+|---|---|
+| Article, BlogPosting | `headline`, `image` |
+| Product | `name` plus one of `offers`, `review`, `aggregateRating` |
+| Recipe | `name`, `image` |
+| FAQPage | `mainEntity`, each with `name` and `acceptedAnswer.text` |
+| BreadcrumbList | `itemListElement`, each with `position` and `name` |
+| Organization | `name`, `url` |
+| LocalBusiness | `name`, `address` |
+| Event | `name`, `startDate`, `location` |
 
-- With no `schemaType` or `schema`, a BetterCMS page gets automatic JSON-LD from its content
-  (WebSite on the home page, Blog or CollectionPage on a collection list, WebPage otherwise).
-  `schemaType` picks an explicit schema.org type; pasted `schema` wins over both.
-- Prefer a small, accurate graph. Add Article, Product, BreadcrumbList, FAQ or Service only when
-  the page visibly supports it. Keep valid imported JSON-LD, and avoid duplicate nodes from
-  native metadata plus custom head code.
-- Never fabricate ratings, reviews, authors, availability, prices or dates. Valid markup does not
-  guarantee a rich result: [Google structured-data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies).
+Valid markup does not guarantee a rich result: [Google structured-data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies).
 
-## 4. Sitemap, robots, llms.txt
+## 5. Sitemap, robots, llms.txt
 
 - The hosted sitemap leaves out noindex pages, and the generated robots.txt points at the
   sitemap. Confirm drafts, private pages and noindex pages are absent from the live file.
@@ -77,7 +201,7 @@ A CMS field holding the right title does not prove the public route emits it. Fe
   sibling (its `locale`, else the project's `localeConfig.defaultLanguage`) plus `x-default` for
   the source row. A noindex or unpublished sibling is left out of the links as well.
 - `sitemapConfig` sets it `auto` or `custom`, the default changefreq and priority and `extraUrls`;
-  `robotsConfig` is `auto` or `custom` text; `indexnowConfig` { enabled, excludePatterns } pings
+  `robotsConfig` is `auto` (with `disallow`: AI crawler names such as `GPTBot` or `ClaudeBot` to block) or `custom` text, which wins when set; `indexnowConfig` { enabled, excludePatterns } pings
   IndexNow on publish; `searchConfig` { enabled, placeholder, aiEnabled } is site search, which
   `search_content` queries (published content only).
 - `refresh_search_index` costs nothing: `action: 'indexnow'` submits `urls` (this site's own, up
@@ -86,9 +210,52 @@ A CMS field holding the right title does not prove the public route emits it. Fe
 - `llms.txt` lists one link per published item. Maintain it for its intended readers, but do not
   sell it as a ranking requirement: Google's [guidance for AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
   says ordinary SEO fundamentals are the base.
+- `set_site_file` stores a hand-written `llms.txt` or `llms-full.txt` (plain text, up to 5 MB).
+  Give `llms.txt` a one-line site summary taken from the home page's description or About copy.
 - Check each redirect's real status and target, and detect loops.
 
-## 5. AEO and content quality
+### AI crawlers
+
+`robotsConfig` is `{ mode: "auto" }` or `{ mode: "custom", custom: "<robots.txt text>" }` (up to
+10,000 characters). `robotsConfig.disallow` is the per-bot switch: a list of names from GPTBot,
+OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Google-Extended,
+CCBot, Bytespider, Applebot-Extended, Amazonbot and Meta-ExternalAgent. Each named bot is
+disallowed from the whole site in auto mode. In custom mode the custom text is served as written
+and `disallow` is not applied; any other bot or a path rule is a line of custom text.
+
+- Carry the source's AI-bot rules over exactly. Changing the owner's crawler policy is not part of
+  a conversion. No rules in the source: stay on `auto` and say so in the report.
+- Custom text replaces the generated file, so include the `Sitemap:` line yourself.
+- Blocking `Google-Extended` or `Applebot-Extended` leaves Google Search and Siri untouched.
+  Blocking an answer or search bot (`OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`,
+  `DuckAssistBot`) removes the site from that product's answers. Say so before the owner asks
+  for it.
+- Other agents to look for in a source file: `GPTBot`, `ClaudeBot`, `Claude-Web`, `anthropic-ai`,
+  `Meta-ExternalAgent`, `FacebookBot`, `Amazonbot`, `cohere-ai`, `CCBot`, `Bytespider`.
+
+```text
+User-agent: *
+Allow: /
+
+User-agent: CCBot
+Disallow: /
+
+Sitemap: https://caldera.com/sitemap.xml
+```
+
+## 6. AEO and content quality
+
+Set these up while converting, without asking:
+
+| Do | How |
+|---|---|
+| Facts as text | Specs, prices, menus, opening hours and chart data become fields or a real table, never an image of them |
+| Answer first | A real question on the page becomes a FAQ row: the question as asked, the answer in the first one or two sentences, detail after |
+| One name per thing | A product, plan, person or brand is one entry that other content references, so a rename reaches every page and feed |
+| Dated facts | A statistic or comparison keeps its date and source in fields beside the number |
+| Authors as entries | An author collection with name, bio and photo, referenced from each post |
+| Transcripts | Audio and video carry one, because crawlers do not watch video. It lives on the asset (`update_media` `transcript`); never invent one ([accessibility.md](accessibility.md)) |
+| Twins and `llms.txt` | Leave them on unless the owner turned them off |
 
 Improve the page for people first: clear questions and answers where the topic warrants them,
 descriptive headings, first-hand evidence, dates and named authors where real, text alongside
@@ -98,7 +265,7 @@ snippets or unverifiable claims.
 Report concrete findings ("canonical points to a 404", "OG image missing", "FAQ schema does not
 match the visible answers"), never an untraceable score.
 
-## 6. Publish and verify
+## 7. Publish and verify
 
 After an authorized change ([publishing.md](publishing.md)): inspect the rendered `<head>` and JSON-LD,
 route status, sitemap and robots output, and redirects. Use Search Console or similar only to
@@ -145,12 +312,45 @@ for a page), or from `generateMetadata`. Rules: 1200×630, an absolute URL, a 40
 missing slug, and strip stega from any preview string first ([bindings.md](bindings.md) §5).
 A hosted site gets its OG image from the native value; this route is for self-hosted Next only.
 
-## Acceptance
+## SEO and AEO gate
+
+Run before publishing or reporting done, then run `list_seo_issues` and `get_next_steps` and fix
+what they list. The accessibility half of the gate is the check list in
+[accessibility.md](accessibility.md).
+
+**SEO**
+
+- [ ] No model, page or component carries a meta, OG, canonical, noindex, hreflang or schema field.
+- [ ] Every indexable page and entry has a `metaTitle` of at most 60 characters and a
+      `metaDescription` (aim 120–160, never over 160). No two titles match.
+- [ ] Source canonicals that point elsewhere are kept; self canonicals are not stored.
+- [ ] Source noindex pages are `noindex`, and so are search, cart, checkout and thank-you pages.
+- [ ] Each language is its own row with `locale` and `translationOf`.
+- [ ] Every changed URL has a 301 to its final URL; the source's redirects are imported; no chain,
+      no loop, no batch pointed at the home page.
+- [ ] `seoDefaults`, `siteSchema` and `collectionSchemaTypes` are set.
+- [ ] Sitemap and robots are `auto`, or `custom` carrying the source's rules.
+- [ ] Native values and the fetched HTML agree on representative static and dynamic routes.
+
+**Structured data**
+
+- [ ] Each page is on Automatic unless a type is missing; then `schemaType`; pasted `schema` last.
+- [ ] The emitted JSON-LD matches the source's for type and key values, and validates.
+- [ ] Every value is visible on the page.
+- [ ] One FAQPage per URL. No self-serving `AggregateRating`.
+- [ ] Pasted JSON-LD parses and has its type's required properties.
+
+**AEO**
+
+- [ ] Specs, prices, hours, menus and chart data are text in fields or tables.
+- [ ] Names are consistent and come from referenced entries.
+- [ ] `llms.txt` has a site summary and Markdown twins are on.
+- [ ] Videos and podcasts have a transcript, or the gap is reported.
+- [ ] The source's AI-crawler policy is carried over unchanged.
+
+**Report**
 
-- Native values and public HTML agree on representative static and dynamic routes.
-- Canonical, indexable routes are in the sitemap with no conflicting directives.
-- Structured data validates and matches visible content.
-- OG and Twitter previews use valid media and route-specific copy.
-- Internal links and redirects resolve; staging and private pages stay out of the index.
-- The report separates fixed technical issues, content recommendations, and unverified search
-  outcomes.
+- [ ] OG and Twitter previews use valid media and route-specific copy.
+- [ ] Internal links and redirects resolve; staging and private pages stay out of the index.
+- [ ] Fixed technical issues, content recommendations and unverified search outcomes are listed
+      separately, with every unsupported directive, missing transcript and page that should be gone.