arcane-cli 0.34.0 → 0.34.2

package/dist/assets/.arcane/governance/decision-documentation-standard.md

@@ -72,7 +72,7 @@ Arcane separates framework-level decisions from org-specific ones using two pref
 2. Confirms no collision with other entities
 3. Identifies any docs, scripts, or prompts that reference the old name
 
-This check was added after [[governance/rcas/RCA-001-naming-collision|RCA-001]] discovered that ADR-061 named a product without checking whether the ops repo already used that name. **Enforcement: explicitly advisory prose (ARC-023) — no script or spell gate verifies that an ADR introducing a new name actually contains a "Naming impact" section or performed its three checks; `spell-scry` can run the grep-and-collision check manually, but nothing requires it before such an ADR is written.**
+This check was added after [[docs/rcas/RCA-001-naming-collision|RCA-001]] discovered that ADR-061 named a product without checking whether the ops repo already used that name. **Enforcement: explicitly advisory prose (ARC-023) — no script or spell gate verifies that an ADR introducing a new name actually contains a "Naming impact" section or performed its three checks; `spell-scry` can run the grep-and-collision check manually, but nothing requires it before such an ADR is written.**
 
 ### Tier 2: Companion Detail Docs
 

package/dist/assets/.arcane/governance/rca-process-standard.md

@@ -38,7 +38,7 @@ An RCA is **recommended** (but optional) for:
 
 ## RCA Template
 
-**Every RCA follows this structure. Artifacts are stored in `governance/rcas/` with the filename `RCA-NNN-short-slug.md` (sequential numbering, matching the related ADR when applicable). Enforcement: explicitly advisory prose (ARC-023) — no script validates RCA filenames, sequential numbering, or template structure.**
+**Every RCA follows this structure. Artifacts are stored in `docs/rcas/` with the filename `RCA-NNN-short-slug.md` (sequential numbering, matching the related ADR when applicable). Enforcement: explicitly advisory prose (ARC-023) — no script validates RCA filenames, sequential numbering, or template structure.**
 
 ```markdown
 ---
@@ -104,7 +104,7 @@ Bullet points — what the team now knows that it didn't before.
 ### 1. Initiate
 
 When a trigger condition is met:
-- Create the RCA artifact file in `governance/rcas/`
+- Create the RCA artifact file in `docs/rcas/`
 - Fill in the Incident Summary and Timeline sections
 - Record the ADR that establishes the RCA (if applicable)
 
@@ -147,13 +147,13 @@ RCAs do not have formal severity levels. The trigger conditions above determine
 ## Artifact Location
 
 ```
-governance/
+docs/
   rcas/
     RCA-001-naming-collision.md
     RCA-002-...
 ```
 
-**Why `governance/rcas/`?** RCAs are governance artifacts (process improvement), not journals (session logs) or decisions (ADRs). They complement ADRs by explaining *how a decision failed* rather than *what was decided*.
+**Why `docs/rcas/`, not `governance/rcas/`?** RCAs are project/domain documents (a record of what happened in *this* repository), not a framework-managed standard — [[portable-bootstrap#where-documents-live|portable-bootstrap.md's "Where Documents Live"]] reserves `.arcane/governance/` for framework-managed standards and explicitly forbids a duplicate root `governance/` tree; `docs/` is the same explicit-descriptive-path convention it names for project/domain documents. RCAs remain governance artifacts in *purpose* (process improvement, not a journal or a decision) — they complement ADRs by explaining *how a decision failed* rather than *what was decided* — just not in *filesystem location*. Enforcement: explicitly advisory prose (ARC-023) — no script checks that an RCA file actually lives under `docs/rcas/` rather than elsewhere.
 
 ## Maintenance
 

package/dist/assets/.arcane/governance/web-discoverability-standards.md

@@ -18,6 +18,7 @@ How a web property becomes retrievable by search-engine crawlers and by the AI a
 - Any value that legitimately differs per environment — a canonical hostname, an "is this production" flag — must be resolved at runtime from the running environment, never baked into a shared build artifact (WD-04, WD-05).
 - Crawler traffic must never pass through endpoints carrying human-traffic side effects, and any crawler-facing content listing must reuse the application's own visibility rule rather than a hand-maintained copy of it (WD-06, WD-07).
 - Crawl control (robots exclusion) and index control (a response header) close different gaps and are both required on non-production origins; search-index presence, not a static convention file alone, is the real lever for AI-assistant retrieval (WD-09, WD-12).
+- AI assistants and answer engines reach a site through *retrieval* user-agents that are distinct from *training* crawlers; disallowing a retrieval agent removes the property from AI answers while a training opt-out does not — decide the two separately, and never let one edit silently cover both (WD-16). Measure the result through a dedicated AI-referrer channel, and keep sitemap `lastmod` tied to real content dates so freshness is a signal rather than a constant (WD-17, WD-18).
 
 ---
 
@@ -40,6 +41,9 @@ How a web property becomes retrievable by search-engine crawlers and by the AI a
 | WD-13 | User-supplied metadata text needs HTML-attribute escaping AND, for inline JSON-LD, "<"-character escaping. |
 | WD-14 | Share images follow the ~1.91:1 convention, keep a safe-content margin, and stay modest in file size. |
 | WD-15 | AI image generation can't reproduce exact type/logos — composite the real assets over generated artwork instead. |
+| WD-16 | Never disallow a *retrieval*-class AI user-agent when AI-assistant visibility is a goal; a *training*-crawler opt-out is a separate, explicit decision, declared as named per-agent groups. |
+| WD-17 | Measure AI-assistant referrals through a dedicated referrer channel, an access-log watch on retrieval-class user-agents, and each engine's own AI-visibility report. |
+| WD-18 | Sitemap `lastmod` values derive from each URL's real content change date — never one build-time constant shared by every URL. |
 
 ---
 
@@ -140,6 +144,8 @@ Disallow: /
 
 **Why:** see "Why two noindex mechanisms, not one" below.
 
+**Non-canonical production hostnames are not non-production origins.** A hosting platform commonly serves the production artifact on an auto-assigned default hostname (and often a `www` alias) in addition to the canonical origin, and a static host may be unable to vary response headers by hostname. Those origins serve the *real* artifact, so WD-09's dual suppression must never be applied to the artifact itself — it would noindex production. The mitigation is the one WD-01 already requires: a canonical tag on every route pointing at the canonical origin, plus registering only the canonical origin with each search console. Inventory such hostnames explicitly (spell-make-discoverable Phase 1) so their state is observed rather than assumed; a canonical tag is a hint an engine may decline, so an unlisted default hostname that has already been indexed is a finding, not noise.
+
 ---
 
 ## Why two noindex mechanisms, not one
@@ -166,7 +172,35 @@ Using only the robots-exclusion rule leaves the bare-URL-listing gap open. Using
 
 ### WD-12 — Search-index presence outweighs a static AI-readable file
 
-**Rule/observation: For the specific goal of being retrievable *by AI assistants* — as distinct from being found by a human searching directly — being present and current in a major search engine's index is a substantially stronger lever than any single static "AI-readable" convention file placed at a well-known path (for example, a plain-text site-description file such as the emerging `llms.txt` convention). Several AI assistants answer by drawing on a search engine's index rather than crawling the web independently, so an assistant's ability to surface a page is frequently bottlenecked on that page's search-index presence, not on whether a convention file exists at all. A convention file is still worth publishing — it costs little and helps the assistants that do read it directly — but it should not be mistaken for the primary lever. The everything-else in this document (WD-01 through WD-11) is the primary lever. Enforcement: structured spell gate (ARC-023) — spell-make-discoverable's Phase 2 Audit table (Search-engine registration, WD-12) reads the webmaster console's own coverage status verbatim before Phase 3's approval gate allows any fix to proceed.**
+**Rule/observation: For the specific goal of being retrievable *by AI assistants* — as distinct from being found by a human searching directly — being present and current in a major search engine's index is a substantially stronger lever than any single static "AI-readable" convention file placed at a well-known path (for example, a plain-text site-description file such as the emerging `llms.txt` convention). Several AI assistants answer by drawing on a search engine's index rather than crawling the web independently, so an assistant's ability to surface a page is frequently bottlenecked on that page's search-index presence, not on whether a convention file exists at all. A convention file is still worth publishing — it costs little and helps the assistants that do read it directly — but it should not be mistaken for the primary lever. The everything-else in this document (WD-01 through WD-11, and WD-16 through WD-18) is the primary lever. Enforcement: structured spell gate (ARC-023) — spell-make-discoverable's Phase 2 Audit table (Search-engine registration, WD-12) reads each relevant webmaster console's own coverage status verbatim before Phase 3's approval gate allows any fix to proceed.**
+
+Three refinements, each from measurement rather than from the conventions' own claims:
+
+- **The index that matters is often not the largest one.** Several AI answer engines and assistant search features build on a *second* engine's index rather than the one with the largest human query share. Coverage therefore has to be confirmed in the console of each engine whose index an AI answer engine draws on — and those consoles are also where the only citation-level AI-visibility reports live. Registering with the largest-share engine alone leaves the AI-retrieval path unverified.
+- **The measured audience for a convention file is coding agents, not answer engines.** Large-sample crawl logs show answer engines essentially never request `llms.txt`; the agents that do are IDE assistants and coding agents fetching documentation, which also commonly ask for a Markdown representation of a page. Publish the file for that audience, and where the property serves developer documentation, serve a Markdown representation of each documentation route alongside it — that is what those agents actually read.
+- **Structured data behaves the same way.** Inline JSON-LD earns rich results and gives entity-aware consumers a machine-readable anchor, but controlled measurement shows no independent effect on AI-assistant citation. Publish it for the former reasons; do not count it as an AI lever.
+
+---
+
+## AI retrieval, freshness, and measurement
+
+### WD-16 — Retrieval agents are never disallowed; training opt-out is a separate decision
+
+**Rule: A robots-exclusion file must never disallow a *retrieval*-class AI user-agent — the agents an AI assistant or answer engine fetches a page with at answer time, or builds its own search index with — when being retrievable by AI assistants is a goal of the property. Opting out of *training* crawlers (the agents that collect data for model training and have no effect on answer-time retrieval) is a legitimate, separate decision: record it explicitly, and declare any policy as named per-agent groups so the two classes are visibly distinct in the file. Classify each agent against its vendor's current published crawler documentation, never from memory — the names, and which class each belongs to, change. Enforcement: structured spell gate (ARC-023) — spell-make-discoverable's Phase 2 Audit table (Robots and sitemap policy, WD-16) parses the live `robots.txt` group by group and requires that no retrieval-class user-agent is matched by a `Disallow:` covering an intended-public route before Phase 3's approval gate allows any fix to proceed.**
+
+**Why:** Vendors publish separate user-agents for the two jobs, and the common failure is a well-intentioned "block the AI scrapers" edit that lists every AI agent name it can find. Human search rankings look untouched afterward — the ordinary search crawler was never in the list — while the property silently disappears from AI answers, because the retrieval agent was. Nothing in a search console reports that loss. A broad wildcard `Allow` is the correct posture for visibility, but it is fragile precisely because it is implicit: naming the retrieval agents as their own allow-groups makes the intent legible to the next editor and gives the audit something to assert against. Note that some user-initiated fetch agents disregard robots exclusion entirely; a `Disallow` aimed at them is a policy statement, not a control, and edge-level blocking is the actual mechanism if one is wanted.
+
+### WD-17 — Measure AI referrals, or the rest of this document is unverifiable
+
+**Rule: Give AI-assistant traffic its own measurement path: a dedicated referrer channel in the property's analytics matching the assistant hostnames that send clicks, an access-log watch on retrieval-class user-agents (so fetches that never become clicks are still visible), and each search console's AI-visibility report where one exists. Treat the resulting counts as a floor, never a total. Enforcement: explicitly advisory (ARC-023) — which analytics stack, which assistant hostnames, and which console reports exist is deployment-specific; spell-make-discoverable carries it as a Phase 6 acceptance-checklist line the operator verifies, not as a Phase 2 gate.**
+
+**Why:** Most AI-assistant exposure never produces a referred click — the answer is consumed inside the assistant, and much of what does click through arrives with no referrer and lands as direct traffic. Without a dedicated channel and a log-level view of the retrieval agents themselves, the effect of WD-01 through WD-16 is invisible: a property can be cited daily and show nothing in its analytics, or be silently dropped and show nothing either. The measurement is cheap and the alternative is guessing.
+
+### WD-18 — Sitemap `lastmod` tracks real content dates
+
+**Rule: Each sitemap `lastmod` value must derive from that URL's actual content change date — a content file's own date, the newest entry in a changelog the page renders, a commit timestamp — never a single build-time constant shared by every URL. Enforcement: structured spell gate (ARC-023) — spell-make-discoverable's Phase 2 Audit table (Robots and sitemap policy, WD-18) parses the live sitemap and fails the check when every `lastmod` carries one shared value, or when a sampled URL's `lastmod` predates content that demonstrably changed later, before Phase 3's approval gate allows any fix to proceed.**
+
+**Why:** Freshness is a measured citation factor for AI answer engines, and `lastmod` is the one freshness signal a crawler can read without fetching the page. A constant shared across every URL says either "nothing has changed since this date" or "everything changed at once" — both false — so an engine learns to ignore the field, and a page that genuinely changed yesterday looks as stale as one untouched for a year. The constant also tends to be set once at launch and never moved, which is the quiet failure mode: it passes every structural check while actively misreporting the property's freshness.
 
 ---
 

package/dist/assets/.github/prompts/spell-make-discoverable.prompt.md

@@ -94,8 +94,24 @@ for each of `{NONPROD_ORIGINS}`:
   link, Open Graph / social-card tags, and any structured-data block —
   fetched with a plain HTTP client that does not execute JavaScript.
 - **`robots.txt`** — full raw contents, and whether it exists at all.
+- **AI user-agent policy in `robots.txt`** — every `User-agent:` group
+  present, and which vendor-published *retrieval* agents (the ones an AI
+  assistant or answer engine fetches with at answer time or indexes with)
+  versus *training* crawlers each group's rules match — checked against each
+  vendor's current crawler documentation, not memory, because the names
+  change.
+- **AI-readable convention files** — whether `/llms.txt` exists, its raw
+  contents, and whether documentation routes also serve a Markdown
+  representation.
+- **Non-canonical production hostnames** — every hostname the platform
+  serves the production artifact on besides `{SITE_ORIGIN}` (a
+  platform-assigned default hostname, a `www` alias), with the canonical tag
+  and robots meta of one sampled route on each. These are not
+  `{NONPROD_ORIGINS}` — they serve the real artifact — so record them
+  separately.
 - **`sitemap.xml`** (or wherever `robots.txt` points) — full raw contents,
-  and whether it parses as well-formed.
+  whether it parses as well-formed, and the set of distinct `lastmod` values
+  it carries.
 - **Non-production noindex signals** — for each `{NONPROD_ORIGINS}` host,
   the response headers and parsed `<head>` of one sampled route.
 - **DNS state at `{DNS_ZONE}`** relevant to search-engine registration —
@@ -133,6 +149,8 @@ rule ID for it.
 | *(supplementary)* | Does `robots.txt` contain a `Sitemap:` directive, and does fetching that URL return a well-formed sitemap? | The `Sitemap:` line and the fetched sitemap's parse result. |
 | `WD-07` | Does every URL listed in the sitemap satisfy `{PUBLIC_VISIBILITY_PREDICATE}`? | Any sitemap entries that fail the predicate. |
 | `WD-08` | Where `{PUBLIC_ASSET_PREFIX}` (or another specific public path referenced by page metadata) falls under a broader disallowed prefix in `robots.txt`, is an explicit `Allow:` rule declared for that specific path? | The relevant `Allow:`/`Disallow:` lines from `robots.txt`, and confirmation the specific path is fetchable by a crawler despite the broader disallow. |
+| `WD-16` | Parse `{SITE_ORIGIN}/robots.txt` group by group. Is any *retrieval*-class AI user-agent (per each vendor's current crawler documentation) matched by a `Disallow:` that covers an intended-public route? A training-crawler `Disallow:` is not a failure — record it as the operator's declared policy. | Every `User-agent:` group with its rules, each agent name classified as retrieval or training with the vendor page that says so, and the resolved allow/disallow verdict per retrieval agent for one sampled public route. |
+| `WD-18` | Parse the sitemap's `lastmod` values. Do they differ across URLs and track each URL's actual content change date — or does every entry carry one shared constant? | The distinct `lastmod` values with their counts, and for one sampled URL, the content's own change date (file date, changelog entry, or commit) next to its `lastmod`. |
 | *(supplementary)* | For a sampled route under `{RESTRICTED_PREFIX}`: is it absent from the sitemap, does it carry a noindex signal, and is it unreachable via any link from a route that satisfies `{PUBLIC_VISIBILITY_PREDICATE}`? | Sitemap membership (should be none), noindex signal, and an inbound-link check from the public route set. |
 
 **Non-production noindexing**
@@ -148,7 +166,7 @@ rule ID for it.
 | --- | --- | --- |
 | `WD-10` | Query `{DNS_ZONE}` directly (independent of any console) for the domain-ownership verification record this property's registration depends on. Is it present, and does it match the value the registration actually expects? | Raw DNS query result — record type, name, and full current value — captured before any write, per `EV-05`. |
 | `WD-11` | Is a crawl/recrawl notification path (e.g. IndexNow) wired for URLs that change, and has `{SITE_ORIGIN}` been submitted through the webmaster console for each target search engine? | Notification endpoint configuration (if any), and the submission status shown in each webmaster console. |
-| `WD-12` | In the webmaster console for the search engine with the largest share of query volume in the target market, does `{SITE_ORIGIN}` show confirmed index coverage with no outstanding manual-action or coverage blocker? | The console's own coverage/status readout, copied verbatim — this is what downstream AI-assistant retrieval actually depends on, not the site's own claim of being indexable. |
+| `WD-12` | In the webmaster console of the search engine with the largest share of query volume in the target market, **and** in the console of each engine whose index an AI answer engine draws on (these are frequently different engines), does `{SITE_ORIGIN}` show confirmed index coverage with no outstanding manual-action or coverage blocker? | Each console's own coverage/status readout, copied verbatim, plus its AI-visibility report (citations, grounding queries, or generative-AI impressions) where the console offers one — this is what downstream AI-assistant retrieval actually depends on, not the site's own claim of being indexable. |
 
 ## Phase 3 — Propose
 
@@ -181,6 +199,11 @@ Only with `--apply` **and** after Phase 3 approval. Typical fixes:
   `{RESTRICTED_PREFIX}` from the sitemap.
 - Add the missing noindex header and meta tag for `{NONPROD_ORIGINS}`,
   gated by `{ENV_SIGNAL}` so production never picks it up by accident.
+- Declare named per-agent `robots.txt` groups so retrieval and training
+  agents are visibly separate and a later edit cannot silently block both
+  (`WD-16`); derive sitemap `lastmod` from content dates (`WD-18`); publish
+  `/llms.txt` and, for documentation routes, a Markdown representation
+  (`WD-12`).
 - Fix the escaping gap found under `WD-13`.
 
 Each fix lands as its own commit, so any one of them can be reverted on its
@@ -236,6 +259,12 @@ user-verifiable acceptance checklist:
       sitemap contains no `{RESTRICTED_PREFIX}` or `{NONPROD_ORIGINS}` URLs.
 - [ ] Each `{NONPROD_ORIGINS}` host serves both a noindex header and a
       noindex meta tag on a sampled route.
+- [ ] `{SITE_ORIGIN}/robots.txt` disallows no retrieval-class AI user-agent
+      (`WD-16`), and the sitemap's `lastmod` values are not one shared
+      constant (`WD-18`).
+- [ ] The property's analytics has a dedicated AI-referrer channel and its
+      access log is watched for retrieval-class user-agents (`WD-17`) — so
+      the effect of everything above is measurable rather than assumed.
 - [ ] The DNS verification record at `{DNS_ZONE}`, re-queried
       independently, matches what the registration expects.
 - [ ] The webmaster console for each target search engine shows the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "arcane-cli",
-  "version": "0.34.0",
+  "version": "0.34.2",
   "description": "Arcane framework CLI — scaffold and manage governance files in consuming repositories",
   "keywords": [
     "arcane",