@sonordev/contracts 2.2.0 → 2.4.0

package/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## 2.4.0
+
+A minor release: one new entry, `@sonordev/contracts/schema-placeholders`, and the llms.txt sanitizers stop cutting mid-word. Every other entry is unchanged.
+
+- **New entry `@sonordev/contracts/schema-placeholders`**: whether stored JSON-LD carries a template's unfilled slots, so the API that serves it and the site that renders it drop the same nodes.
+  - `withoutSchemaPlaceholders(value)` returns `{ value, dropped, notes }`: the JSON-LD without its placeholder nodes (the same reference when nothing was found, `null` when nothing real is left), every node it removed with the reason and the evidence, and the AI note keys it stripped from nodes it kept.
+  - `findSchemaPlaceholders(value)` returns just the nodes, and `describeSchemaPlaceholder(node)` turns one into a line a person can act on.
+  - It finds reserved example domains (`example.com`, `example.net`, `example.org`, `.example`) in URLs, hosts and emails, whole-value template names ("Example", "Your Business Name"), placeholder phones, bracketed and JSX slots (`[Resident Name]`, `{plan.name}`, `{{business_name}}`), `REPLACE_WITH_` markers, and objects that are only a note. A URI template's `{search_term_string}`, a real name that contains "Example" and a 555 number outside the fiction range are left alone.
+- **`llms`**: `clipAtWordBoundary(text, max)` cuts prose to a length without splitting a word. It ends at a sentence when one closes in the last 40% of the window, otherwise at the last whole word with an ellipsis, and never returns more than `max` characters. `sanitizeLlmsPublicSummary` and `sanitizeLlmsDisclaimerLine` now cut with it, so a long summary ends at a sentence or a whole word instead of mid-word. The limits are unchanged (400 and 280).
+
+## 2.3.0
+
+A minor release: one new entry, `@sonordev/contracts/error-page`. Every other entry is unchanged.
+
+- **New entry `@sonordev/contracts/error-page`**: telling a rendered error page from a page's content, and copy that calls a working page missing from copy that describes it, so every writer and reader of page text answers the same way.
+  - `looksLikeErrorPage(text, wordCount?)` is true when stored text is an error page: a 404 in its common wordings, a framework or server error, a status page, a "something went wrong" or "temporarily unavailable" screen or a maintenance notice. It reads how the text opens, decodes entities, doesn't depend on word boundaries (scraped text runs elements together, as in "404This page could not be found."), and leaves phone numbers, addresses and pages that only describe an error alone.
+  - `usablePageText(text, wordCount?)` returns the text, or `null` for an error page.
+  - `findErrorPageClaim(copy, pageText?)` and `describesErrorPage(copy, pageText?)` find generated copy that says the page itself is missing or unavailable ("Page Not Found", "this page is currently unavailable", "returns a 404", "a page-not-found message"). A phrase the page's own text really says is allowed, and copy about something unavailable on a working page isn't a claim about the page.
+  - `findPageCopyClaim(fields, pageText?)` runs that check over every string in a set of fields, JSON-LD included, and returns the first `{ field, phrase }`.
+  - `ERROR_PAGE_MAX_WORDS` and `ERROR_PAGE_CONTRACT_VERSION` are exported alongside.
+
 ## 2.2.0
 
 A minor release: one new entry, `@sonordev/contracts/proposal-sitemap`. Every other entry is unchanged.

package/README.md

@@ -14,7 +14,7 @@ pnpm add @sonordev/contracts
 | `@sonordev/contracts/sites` | The canonical shape of a site host (the multi-site `site` dimension) | every ingest and read that filters by site |
 | `@sonordev/contracts/seo-pages` | Which `seo_pages` row a path resolves to on a multi-site project | the Sonor API, Signal, site-kit's sitemap sync |
 | `@sonordev/contracts/seo-meta` | Whether a title or description is fit to ship | site-kit, Signal's SEO gate |
-| `@sonordev/contracts/llms` | llms.txt sanitizers and contract version | site-kit, the Sonor API, Signal's schema validation |
+| `@sonordev/contracts/llms` | llms.txt sanitizers, the word-safe cut they use (`clipAtWordBoundary`), and the contract version | site-kit, the Sonor API, Signal's schema validation |
 | `@sonordev/contracts/forms` | The honeypot policy | site-kit forms, the Sonor API's form intake |
 | `@sonordev/contracts/fleet` | The fleet heartbeat's wire shape | site-kit, the Sonor API |
 | `@sonordev/contracts/slots` | Slot payload signing, contract v2 (HMAC; Node only). Signs and verifies v1 for older site-kit releases | the Sonor API, site-kit on the server |
@@ -24,6 +24,42 @@ pnpm add @sonordev/contracts
 | `@sonordev/contracts/portfolio` | Where a portfolio number comes from and which may headline; the live-frame message | the Sonor API, Signal, agency-site-kit |
 | `@sonordev/contracts/proposal-sitemap` | How a proposal's site plan counts its pages, when its before-and-after comparison may show, and whether the page counts stated elsewhere in the proposal add up | the Sonor API, Signal's proposal writer, the dashboard's proposal pages |
 | `@sonordev/contracts/voice` | The house voice's contraction rule. `findUncontracted` and `contractText` find and apply full forms that should be contractions; `findContracted` finds the contractions a question-form heading should expand | anything that checks or rewrites copy: Sonor's APIs and dashboard |
+| `@sonordev/contracts/error-page` | Whether stored text is a rendered error page rather than a page's content, and whether generated copy says the page itself is missing | anything that stores page text or writes page copy from it: the Sonor API's page-view and sitemap writers, Signal's metadata generators |
+| `@sonordev/contracts/schema-placeholders` | Whether stored JSON-LD carries a template's unfilled slots (`example.com` URLs, an `Organization` named "Example", a placeholder phone, `[Resident Name]`, `{plan.name}`), and the value without them | site-kit's schema output, the Sonor API's schema reads and writes |
+
+## Schema placeholders
+
+`schema-placeholders` finds the parts of a JSON-LD value that a template left
+unfilled, so a site never publishes them as the business's identity. Schema
+extracted from a site's source, where the base URL was a variable, comes back
+as `https://example.com`; a template filled in by hand or by an AI can leave
+"Example", `+1-000-000-0000`, `[Resident Name]`, `{plan.name}`, or an object
+that's only a note about what goes there.
+
+```ts
+import { withoutSchemaPlaceholders, describeSchemaPlaceholder } from '@sonordev/contracts/schema-placeholders'
+
+const { value, dropped } = withoutSchemaPlaceholders(storedJsonLd)
+// value: the JSON-LD without its placeholder nodes (the same reference when
+// nothing was found, null when nothing real is left)
+dropped.map(describeSchemaPlaceholder)
+// ['Organization at $: name "Example" is a placeholder name']
+```
+
+- **The unit is the node.** The innermost object with an `@type` or `@id` (or
+  a top-level or `@graph` member) whose own values hold the placeholder is
+  dropped whole. Its real siblings, and a real parent it hangs off, stay. A
+  node left with nothing but JSON-LD keywords once its placeholder children are
+  gone goes too.
+- **Conservative on purpose.** A false positive removes real schema from a live
+  site, so names match whole values only ("Example Plumbing Co" is a real
+  name), domains are only the reserved example domains, phones are only
+  all-zero, sequential, `XXX` and the 555-0100 to 555-0199 fiction range, and a
+  URI template's `{search_term_string}` is how schema.org spells a
+  `SearchAction`, never a finding.
+- **Notes.** An AI's `note` key on a real node isn't schema.org vocabulary: it's
+  stripped and the node stays. An object that's only a note is dropped.
+- **Pure.** The input is never mutated.
 
 ## The voice contract
 
@@ -185,6 +221,72 @@ pages") isn't read, and a count of pages that aren't in the plan (say, a
 competitor's) is reported like any other; the issues are for a person to look
 at, not edits to apply.
 
+## The error-page contract
+
+A page's stored text is whatever a browser or crawler rendered. Visit a URL
+while it's briefly broken and the error page's words become that page's
+content, and any title, description or summary written from that content
+describes a working page as broken. `error-page` is how writers refuse both
+halves of that.
+
+- **`looksLikeErrorPage(text, wordCount?)`** is true when the text is a
+  rendered error page: a 404 in its common wordings ("404", "This page could
+  not be found", "Page not found"), a framework error ("a client-side
+  exception has occurred", "This page couldn't load", "Internal Server
+  Error", "Cannot GET /x"), a status page ("502 Bad Gateway", "HTTP Status
+  404"), a "something went wrong" or "temporarily unavailable" screen, or a
+  maintenance notice. It reads how the text *opens* (the first 300
+  characters), decodes HTML entities, and doesn't depend on word boundaries,
+  because scraped text runs elements together ("404This page could not be
+  found."). Text up to 120 words is checked for every wording, though the
+  loosest ones (a 404 glued to its copy, "refresh the page and try again", a
+  route's "<name> not found") only read the first 60 or 40 words; up to 400
+  words only the check for how it opens after a meta description and skip link
+  applies; beyond that it's prose. A phone number, room number or street
+  address that starts with 404 isn't an error page, and neither is a help page
+  that only *describes* an error ("how to fix a 404 error", "if something went
+  wrong"). Pass the word count of the text itself, never a whole page's count
+  (a browser's includes the navigation and footer and can hide a short error
+  state): a supplied count can only raise the count the check uses.
+- **`usablePageText(text, wordCount?)`** returns the text, or `null` when it's
+  an error page. It's what a model should be shown as "the page's content".
+- **`findErrorPageClaim(copy, pageText?)`** returns the phrase in generated
+  copy that says the page itself is missing or unavailable ("Page Not Found",
+  "the pricing page is currently unavailable", "returns a 404", "a
+  page-not-found message"), or `null`. Copy about something unavailable on a
+  working page ("the patio is closed on Mondays", "tours are paused for
+  winter") isn't a claim about the page. Pass the page's own text and a phrase
+  the page really says is allowed, so a guide to fixing "page not found"
+  errors can be titled that way; the text of an error page is never accepted
+  as that exemption. `describesErrorPage` is the boolean form.
+- **`findPageCopyClaim(fields, pageText?)`** runs `findErrorPageClaim` over
+  every string in a set of fields (a title, a description, keywords, a
+  JSON-LD object; any object or class instance) and returns the first
+  `{ field, phrase }`, or `null`. It's how a writer or a reader judges a whole
+  draft in one call. It reads at most 2,000 strings, 8 levels deep and 100,000
+  characters in all, and each string's first 5,000 characters, so a hostile
+  record can't make it slow.
+
+```ts
+import { describesErrorPage, findPageCopyClaim, usablePageText } from '@sonordev/contracts/error-page'
+
+usablePageText('404This page could not be found.') // null: don't generate from it
+usablePageText('Weekly mowing for property managers.') // the text
+
+describesErrorPage('Commercial Mowing Page Not Found | Example Co') // true: don't ship it
+describesErrorPage('Seasonal tours are paused until April.') // false
+
+findPageCopyClaim({
+  title: 'Commercial Mowing | Example Co',
+  schema: { '@type': 'WebPage', description: 'This URL currently returns a 404.' },
+})
+// { field: 'schema', phrase: 'returns a 404' }
+```
+
+A text-only check can't prove a page failed. Where you observed the HTTP
+status, require 400 or more as well and use this as the fallback; the header of
+`src/contracts/error-page.ts` lists the other known limits.
+
 ## Why it exists
 
 A rule that lives in several copies is how two of them end up disagreeing.

package/dist/color.d.ts

@@ -4,7 +4,7 @@
  * Single source of truth for turning a CSS colour string into something the
  * kit can reason about. Used at build time by the brand-profile extractor
  * (`src/brand/extract-brand-profile.ts`, theme detection) and in the browser
- * by Echo (`src/engage/brand-color.ts`, brand text contrast). Pure, no
+ * by Echo (`src/chat/brand-color.ts`, brand text contrast). Pure, no
  * imports, safe in both.
  */
 export type Rgb = {

package/dist/error-page.d.ts

@@ -0,0 +1,183 @@
+/**
+ * @sonordev/contracts/error-page: telling an error page from a page (v1)
+ *
+ * Three questions every writer of page content and page copy has to answer the
+ * same way, so they live here once:
+ *
+ *   1. Is this stored text a rendered ERROR PAGE rather than the page's own
+ *      content? (`looksLikeErrorPage`, `usablePageText`)
+ *   2. Does this generated COPY say the page itself is missing or unavailable?
+ *      (`findErrorPageClaim`, `describesErrorPage`)
+ *   3. Does any string in a page's managed copy (a title, a keyword list, a
+ *      JSON-LD block) say so? (`findPageCopyClaim`: question 2 over every
+ *      string of a record, naming the field that holds it)
+ *
+ * Why it exists: a page's stored text is the text a visitor's browser (or a
+ * build's crawler) rendered. Visit a URL while it's briefly broken and the
+ * error page's words are stored as that page's content. Everything generated
+ * from that content then describes a working page as broken, and the result
+ * is published as the page's title, meta description, social tags, JSON-LD
+ * and AI-crawler notes. The outage ends in a deploy; the poisoned copy
+ * doesn't, because nothing downstream asks whether its input was ever the
+ * page.
+ *
+ * How the text check works
+ *
+ *   - It reads the START of the text (300 characters), lowercased, with
+ *     whitespace collapsed, HTML entities decoded and curly quotes made
+ *     straight. Error copy leads; articles bury it.
+ *   - The signatures are generic on purpose. Every site words its 404
+ *     differently and frameworks ship their own ("This page could not be
+ *     found.", "Application error: a client-side exception has occurred"), so
+ *     the shape is what's matched, not any one site's copy. They're grouped by
+ *     family below: not found, server and client errors, unavailable and
+ *     maintenance, walls.
+ *   - Stored text is often the page's elements run together with no separator
+ *     ("404This page could not be found."), so no signature depends on a word
+ *     boundary around the text it's matching.
+ *   - A signature that follows a word that DESCRIBES errors rather than
+ *     showing one ("if something went wrong", "how to fix a 404 error", a
+ *     quote mark) doesn't count: a help page that names an error isn't one.
+ *     Navigation is Title Case and prose isn't, so a describing word counts
+ *     only as it's written in lower case, with only its first letter
+ *     capitalized ("How to fix..."), or opening the text or a sentence;
+ *     "Home Blog How To Something went wrong" is still an error page, and the
+ *     words a menu uses too ("About", "Monitoring", "Alerts for") describe an
+ *     error only in lower case. A describing word inside the matched phrase
+ *     counts too, and after a described match the search resumes one character
+ *     on, so a real error later in the same stretch is still found.
+ *   - A not-found phrase that sits in a question ("Page not found? Use our site
+ *     map", "Product not found in your size? Ask us") is a help line, and so is
+ *     most copy that merely uses the words ("Rare parts can't be found
+ *     elsewhere": the subject has to be a page, URL, link, file, site or app).
+ *   - Up to ERROR_PAGE_MAX_WORDS words every check applies, but the loosest
+ *     ones apply only to the shortest texts: a 404 glued to its copy and
+ *     "refresh the page and try again" up to 60 words, a dynamic route's own
+ *     "<name> not found" state up to 40. From there up to 400 words only the
+ *     prefix-aware check applies: a text written as "<meta description> —
+ *     <every string on the page>" carries its header and footer strings along,
+ *     which pushes a rendered error state past the short-text ceiling, so the
+ *     check looks at how the text opens after the description and a skip link,
+ *     and an opener followed by "explained" or "how to" is an article. Beyond
+ *     400 words the text is prose.
+ *
+ * `wordCount` semantics: pass the word count of `text` itself. Pass the whole
+ * document's count only when `text` is a truncated excerpt of it. A supplied
+ * count can only RAISE the count the check uses (an excerpt of a long document
+ * is prose however short it is), never lower it: text that is longer than the
+ * count you gave is counted by what it holds. A browser's page-wide count
+ * (navigation and footer included) is therefore the wrong number for a short
+ * error state it scraped: pass the count of the text you pass.
+ *
+ * How the copy check works
+ *
+ *   `findErrorPageClaim` looks for the copy saying THE PAGE is missing or
+ *   unavailable ("Page Not Found | Acme", "This URL currently returns a 404",
+ *   "This listing has been removed", "The page failed to load"). The subject
+ *   has to be the page, its URL, link or listing, so copy about something
+ *   unavailable on a working page ("sunset cruises are not available at this
+ *   time") is left alone, and so is a qualified one ("unavailable in winter").
+ *   Whitespace is collapsed and each string is read up to 5,000 characters, so
+ *   hostile whitespace costs nothing. The page's own text exempts a phrase the
+ *   page really says; a page that is itself an error page exempts nothing.
+ *
+ *   `findPageCopyClaim(fields, pageText?)` is the same check over a record:
+ *   each value of `fields` is checked if it's a string, and walked for every
+ *   string it holds if it's an array or an object (8 levels deep, 2,000
+ *   leaves at most; numbers, booleans and null are ignored). It returns the
+ *   first claim as `{ field, phrase }`, `field` being the TOP-LEVEL key (the
+ *   column or property that holds the copy), or null. Use it for a page row's
+ *   managed title, description, keywords and JSON-LD in one call.
+ *
+ * Known limits (a text-only check can't close these)
+ *
+ *   - Any error phrase in the first 300 characters of a text of 120 words or
+ *     fewer reads as an error page, wherever in those 300 characters it sits, unless
+ *     something describes it (see above): a short page ABOUT errors ("Page not
+ *     found: how to design a helpful 404 page. Five examples.", or a web
+ *     agency's line about its "custom 404 error page") can be misread. Texts of
+ *     121 to 400 words are only read for how they open, and longer ones are
+ *     prose. The loosest wordings (a 404 glued to its copy, "refresh the page
+ *     and try again", "<name> not found") apply only up to 60, 60 and 40 words.
+ *   - A business whose name is an error phrase ("404 Brewing Co.", "Page Not
+ *     Found Records") reads as an error page when its name leads the text.
+ *   - Chrome in front hides the error state from the wordings anchored to the
+ *     start of the text ("Not Found", "Be right back.", "Error: ...", "Oops!
+ *     This page...", a permission wall, a dynamic route's "<name> not found").
+ *     The wordings that hold anywhere in the first 300 characters (a status with
+ *     its reason, "could not be found", "something went wrong") are what's
+ *     left. Prefer the page's main content.
+ *   - The text alone never proves a page failed. Where a call site observed
+ *     the HTTP status (a browser reports it), require a status of 400 or more
+ *     as well, and treat this check as the fallback for the sites and the
+ *     paths that report none.
+ *   - It reads English. A 404 in another language isn't matched.
+ *   - Walls and pages served by the host in front of the site (a CDN block, a
+ *     bot challenge, a parked domain, a default server page, a browser
+ *     interstitial) never reach the writers this guards, so they aren't
+ *     matched here.
+ *   - Copy that claims the page is missing in words outside the closed list
+ *     below isn't caught; the real protection is that error text never reaches
+ *     the generator (`usablePageText`).
+ *
+ * Pure string logic: no I/O, no Node built-ins, no lookbehind (this ships to
+ * browsers, and Safari before 16.4 throws on a lookbehind at parse time).
+ */
+/** Bump on breaking changes to what counts as an error page or an error claim. */
+export declare const ERROR_PAGE_CONTRACT_VERSION: 1;
+/**
+ * Above this many words, text is real prose even if it mentions a 404: only
+ * the prefix-aware check (up to 400 words, see the header) still looks at it.
+ *
+ * The gate is what makes the check safe: an article about broken links ("How
+ * to fix 404 errors on your site") legitimately holds every phrase below and
+ * must not be discarded. A rendered error page is short, and this ceiling
+ * still separates the two by an order of magnitude.
+ */
+export declare const ERROR_PAGE_MAX_WORDS = 120;
+/**
+ * True when this text is a rendered error page rather than page content.
+ *
+ * @param text      the scraped or extracted body text
+ * @param wordCount the word count of `text` itself, or the whole document's
+ *                  when `text` is a truncated excerpt of it. It can only raise
+ *                  the count the check uses, never lower it; derived from
+ *                  `text` when absent.
+ */
+export declare function looksLikeErrorPage(text?: string | null, wordCount?: number | null): boolean;
+/**
+ * The page's stored text, or null when it's an error page. What a model may
+ * be shown as "the page's content".
+ */
+export declare function usablePageText(text?: string | null, wordCount?: number | null): string | null;
+/**
+ * The phrase in `copy` that says the page itself is missing or unavailable, or
+ * null when the copy is fine.
+ *
+ * @param pageText the page's own text. A phrase the page really says (an
+ *                 article about fixing "page not found" errors) is allowed;
+ *                 only copy that claims what the page doesn't is a finding.
+ *                 A pageText that is itself an error page allows nothing: its
+ *                 words are the poison, so the copy that repeats them would
+ *                 otherwise vouch for itself.
+ */
+export declare function findErrorPageClaim(copy?: string | null, pageText?: string | null): string | null;
+/** True when `copy` says the page itself is missing or unavailable (see `findErrorPageClaim`). */
+export declare function describesErrorPage(copy?: string | null, pageText?: string | null): boolean;
+/**
+ * The first string anywhere in a record of managed copy that says the page
+ * itself is missing or unavailable, or null. `findErrorPageClaim` over every
+ * string of `fields`, and the record's own field name with the phrase.
+ *
+ * Each value is checked if it's a string, and walked for the strings it holds
+ * if it's an array or an object (a JSON-LD block, a keyword list), up to 8
+ * levels deep, 2,000 strings and 100,000 characters in all; numbers, booleans and
+ * null are ignored (and don't count toward those limits).
+ * `field` is the TOP-LEVEL key, the column or property that holds the copy.
+ * `pageText` exempts what the page really says, as in `findErrorPageClaim`; the
+ * page's own text is read once for the whole record.
+ */
+export declare function findPageCopyClaim(fields: object, pageText?: string | null): {
+    field: string;
+    phrase: string;
+} | null;