arcane-cli 0.21.1 → 0.22.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -30,7 +30,7 @@ npm install -g arcane-cli # then: spell init
30
30
 
31
31
  <div align="center">
32
32
 
33
- **🔁 opinionated lifecycle** &nbsp;·&nbsp; **📜 34 spells** &nbsp;·&nbsp; **🤖 12 agents** &nbsp;·&nbsp; **⚖️ 19 governance standards** &nbsp;·&nbsp; **📝 markdown-native** &nbsp;·&nbsp; **🔌 any AI client / tracker**
33
+ **🔁 opinionated lifecycle** &nbsp;·&nbsp; **📜 36 spells** &nbsp;·&nbsp; **🤖 12 agents** &nbsp;·&nbsp; **⚖️ 23 governance standards** &nbsp;·&nbsp; **📝 markdown-native** &nbsp;·&nbsp; **🔌 any AI client / tracker**
34
34
 
35
35
  </div>
36
36
 
@@ -131,9 +131,9 @@ Arcane isn't a prompt snippet — it's a full framework. Everything installs int
131
131
  </details>
132
132
 
133
133
  <details>
134
- <summary><b>⚖️ The governance standards (19)</b></summary>
134
+ <summary><b>⚖️ The governance standards (23)</b></summary>
135
135
 
136
- `universal-agent-rules` · `development-methodology` · `git-conventions` · `testing-standards` · `cicd-standards` · `decision-documentation-standard` · `naming-conventions` · `agent-policies` · `agent-approved-paths` · `agent-work-queue-model` · `threat-model` · `hardening-checklist` · `authentication-strategy` · `product-excellence-standards` · `rca-process-standard` · `poc-management-pattern` · `spell-authoring-standards` · `new-business-setup` · `portable-bootstrap`
136
+ `universal-agent-rules` · `development-methodology` · `git-conventions` · `testing-standards` · `cicd-standards` · `decision-documentation-standard` · `naming-conventions` · `agent-policies` · `agent-approved-paths` · `agent-work-queue-model` · `threat-model` · `hardening-checklist` · `authentication-strategy` · `product-excellence-standards` · `rca-process-standard` · `poc-management-pattern` · `spell-authoring-standards` · `new-business-setup` · `portable-bootstrap` · `framework-decisions` · `records-conventions` · `external-verification-standards` · `web-discoverability-standards`
137
137
 
138
138
  </details>
139
139
 
@@ -0,0 +1,128 @@
1
+ ---
2
+ title: External Verification Standards
3
+ audience: both
4
+ last_updated: YYYY-MM-DD
5
+ status: active
6
+ distributable: true
7
+ tags: [verification, external-systems, deployment, reliability]
8
+ ---
9
+
10
+ # External Verification Standards
11
+
12
+ Rules for confirming that a write actually took effect in a system you do not control — a deployment console, a CI/CD pipeline, a DNS zone, or any third-party API or UI. It is vendor-neutral by design: the failure mode it addresses (an interface's report of success diverging from the system's actual resulting state) is a property of remote, opaque systems in general, not a quirk of any one product.
13
+
14
+ ## Executive Summary
15
+
16
+ - An interface's own success signal — a toast, a green exit code, a 200 response — is a claim about a write, not evidence of the resulting state; the only valid proof is re-reading the persisted or served state through an independent path and comparing it against intent, whether that state lives behind a save button or behind a deploy pipeline.
17
+ - When a system's own status readout disagrees with what you intended to set, trust the readout over the action's apparent success, stop retrying the same automated input, and hand the step to a human operator instead.
18
+ - Any value that will later be compared by exact string equality must never carry explanatory text appended or interleaved into it, because a near-identical string usually fails silently into a fallback path rather than raising a visible error.
19
+ - Before writing to externally-shared state you did not create and do not fully control, enumerate its current full value first and prefer an additive operation over a create-or-replace one, since a replace can silently destroy values you don't know are there.
20
+ - A "not found," "could not fetch," or "pending" status seen immediately after a write is often the external system's own reporting lag rather than a real failure — rule out the real, checkable causes with an independent verification path before attributing it to lag.
21
+
22
+ ---
23
+
24
+ ## Rule index
25
+
26
+ | ID | Rule (one line) |
27
+ |----|------|
28
+ | EV-01 | Persisted state is proved only by re-reading it through an independent path — never by the interface that accepted the write. |
29
+ | EV-02 | A green pipeline/deploy exit status proves the pipeline ran, not that the live system serves the change — assert on the served artifact itself, allowing a short propagation window. |
30
+ | EV-03 | When a system's own readout disagrees with intent, trust the readout over the apparent success, stop retrying blindly, and escalate to a human operator. |
31
+ | EV-04 | Values compared elsewhere by exact string equality must never carry appended or interleaved commentary. |
32
+ | EV-05 | Enumerate current state before writing to shared external state you don't fully control; prefer additive operations over create-or-replace. |
33
+ | EV-06 | An immediate "not found" / "pending" status is often propagation lag, not failure — rule out real causes independently before attributing it to lag. |
34
+
35
+ ---
36
+
37
+ ## Proving a write took effect
38
+
39
+ ### EV-01: Re-read the persisted state — never trust the interface that accepted it
40
+
41
+ Consoles, admin panels, and forms commonly drop programmatic or automated input silently: a field reports as set immediately after submission, but is actually empty when the page is freshly reloaded; a save action renders a success state — a checkmark, a toast, a disabled button flipping back to enabled — with no write request having actually been sent over the wire. The interface's success rendering and the underlying write are two different code paths, and nothing guarantees they stay in sync when the input arrived by script rather than by a human clicking through the UI at human speed.
42
+
43
+ This is not a rare edge case specific to any one product; it is a structural property of UI layers built to feel responsive to a human. Optimistic rendering, client-side validation that never reaches the server, debounced or dropped keystrokes from fast programmatic typing, and silent no-ops on unsupported input shapes all produce the same symptom: the screen says success, the backing store disagrees.
44
+
45
+ The only valid evidence of a successful write is: perform the write, then reload or re-fetch through a path independent of the one that performed the write (a fresh page load, a separate read API call, a different client), then compare the observed value against intent field by field. A screenshot taken without reloading, or a success message read from the same response that carried the write, proves only that the interface accepted input — never that the input was stored.
46
+
47
+ **Rule (EV-01):** Persisted state is proved by re-reading it, never by the interface that accepted it.
48
+
49
+ ### EV-02: A green pipeline does not mean the live system reflects the change
50
+
51
+ A build, deploy, or pipeline step reporting success (exit code 0, a green checkmark, a "deployment succeeded" notification) proves that the pipeline's own steps ran to completion — it does not prove that the target system now serves the change. The pipeline's job ends at "I sent the artifact" or "I issued the deploy command"; whether the deploy target actually picked it up, finished rolling it out, and is now serving it to a real request is a separate fact, observed at a separate layer.
52
+
53
+ The only assertion that counts as evidence is against the actual served artifact: fetch the live endpoint, read the bytes or headers back, and check them against the expected version, hash, build identifier, or timestamp — not against the pipeline's own log output. A pipeline that reports success while the target is still serving a stale build (a rollout still in progress, a cache still holding the old response, a load balancer still routing to old instances) is not a contradiction; it is the pipeline correctly reporting on its own steps while remaining silent on convergence.
54
+
55
+ This also cuts the other way: a live check performed in the same breath as a *confirmed-successful* deploy may legitimately still show the old state for a short window — propagation, cache invalidation, and instance rollout all take real time. A mismatch observed immediately after a successful deploy is not by itself proof the deploy failed; it is grounds to wait a short, bounded interval and re-check the served artifact again before escalating. What distinguishes this from ignoring a real failure is that the re-check still has to happen and still has to pass — the propagation window is a reason to retry the observation, never a reason to skip it.
56
+
57
+ **Rule (EV-02):** A green exit status from a deploy or pipeline step is evidence that the pipeline ran, not that the live system reflects the change — verify against the served artifact itself, allowing a short, bounded propagation window before treating a fresh mismatch as failure.
58
+
59
+ ---
60
+
61
+ ## When automated input silently fails
62
+
63
+ ### EV-03: Trust the system's own readout, and stop retrying blind
64
+
65
+ Some external tools expose their own summary of current state alongside the action you just took — a record counter, a status field, a "last updated" line, a settings summary. When that readout disagrees with what you just tried to set, and the action itself reported success, the readout wins. An action's success report is a claim made at submission time, before the target system has necessarily done anything with the input; the system's own current-state readout is a claim made by reading the system now. When the two disagree, only one of them is describing the present.
66
+
67
+ Retrying the identical automated input against a control that just silently dropped it is not a fix — it is a repeat of the same conditions that produced the drop, and it will typically produce the same silent drop again. If the input path is the problem (a field the automation can't reach correctly, a save handler that doesn't fire for scripted events, a control that requires a human-timed interaction the automation collapses), no number of retries changes that. What changes it is a different path: a human operator performing the same step by hand, since a human interacting normally exercises the code path the interface was actually built for.
68
+
69
+ **Rule (EV-03):** When a console or tool's own state readout disagrees with what you intended to set, trust the readout over the action's apparent success, and hand the step to a human operator rather than retrying the same automated input blindly.
70
+
71
+ ---
72
+
73
+ ## Exact-match values and shared external state
74
+
75
+ ### EV-04: Keep comparison-critical values free of embedded commentary
76
+
77
+ Some values exist purely to be compared: a marker string checked for equality, a delimiter a parser splits on, a magic constant a conditional branches against. When such a value picks up appended or interleaved explanatory text — a clarifying note, a unit, a parenthetical — the comparison it was meant to satisfy now runs against a slightly different string.
78
+
79
+ The dangerous part is what "slightly different" does at the comparison site. Equality checks against a near-miss string do not usually raise an error; they simply evaluate false and fall through to whatever default or fallback branch handles the "no match" case. The system keeps running, often with no log line and no visible symptom beyond behavior that's subtly wrong — a feature silently not activating, a record silently not routed, a check silently always taking its fallback path. Because nothing throws, the defect is invisible except by directly inspecting the actual string the comparison receives at runtime, not by inspecting the code that produced it or the documentation that describes it.
80
+
81
+ The fix is scope discipline: a value that will be compared verbatim carries only the characters the comparison expects, full stop. Explanation belongs in a comment beside the value's declaration, in a separate field, or in documentation — never folded into the value itself.
82
+
83
+ **Rule (EV-04):** A value that will be compared elsewhere by exact string equality must never carry explanatory commentary appended or interleaved into the same string.
84
+
85
+ ### EV-05: Enumerate before you write to shared external state
86
+
87
+ Some external state is not owned outright — a DNS record set that other records or services also depend on, a shared configuration blob other consumers also read, a permissions or access list other principals already appear on. Writing to state like this without first reading its current full value treats it as if it were exclusively yours, when the entire reason it lives in a shared external system is that it usually isn't.
88
+
89
+ Two operations look interchangeable at the call site but are not: an additive operation (append, add, merge) only ever adds to what's there, while a create-or-replace operation overwrites the entire value with whatever you supply — including silently deleting anything present that you didn't know about and didn't include. When the target system offers both, prefer the additive form specifically because it degrades safely: worst case, it adds something redundant or slightly wrong, which is visible and correctable. A replace's worst case is destructive and often invisible at the moment it happens — nothing errors, because as far as the write path is concerned the operation succeeded exactly as requested.
90
+
91
+ The failure this produces is also delayed and displaced: whatever else depended on the value you overwrote breaks only when that other thing is next exercised, which may be long after and far away from the change that actually caused it — making the eventual failure much harder to trace back to its cause than if the destructive write had failed loudly at the time. Enumerating the current value first, before deciding what to write, is what makes the additive-vs-replace choice an informed one rather than a guess.
92
+
93
+ **Rule (EV-05):** Before writing to a piece of externally-shared state you did not create and do not fully control, first enumerate its current full value, and prefer an additive operation over a create-or-replace operation whenever the system offers both.
94
+
95
+ ---
96
+
97
+ ## Propagation lag versus real failure
98
+
99
+ ### EV-06: Rule out real causes before blaming lag
100
+
101
+ Many external systems separate the act of accepting a write from the act of reflecting it in their own status reporting. Registering, submitting, or requesting something can succeed immediately while the system's status page, dashboard, or lookup API continues to show "not found," "could not fetch," or "pending" for a period afterward — the write landed, but the system hasn't caught its own bookkeeping up to it yet. Treating every such status as a real failure produces a steady stream of false alarms and wasted remediation on systems that were, in fact, working correctly and simply had not finished telling you so.
102
+
103
+ The trap runs the other direction too, though: propagation lag is a real phenomenon, not a universal excuse, and it is easy to reach for as an explanation precisely because it requires no further investigation. Before attributing a "not found" to lag, verify the underlying resource independently, through a check that does not depend on the same status system that's reporting the problem — fetch the resource directly with a plain client, resolve a record against an independent resolver, query the actual target rather than a summary of it. If the independent check also fails, there is a real problem to fix, and lag was never the explanation. Only once the real, checkable causes are ruled out does "still propagating" become a legitimate conclusion rather than a guess dressed up as one.
104
+
105
+ **Rule (EV-06):** A status of "not found" / "could not fetch" / "pending" observed immediately after submitting or registering something with an external system is often propagation lag in that system's own status reporting rather than a real failure — but before concluding it is lag, independently verify the underlying resource through a different, direct check, and only attribute the status to lag once the real, checkable causes have been ruled out.
106
+
107
+ ---
108
+
109
+ ## Verification pattern
110
+
111
+ The rules above share one shape. Apply it as the default sequence for any write to a system you don't control:
112
+
113
+ ```text
114
+ 1. Perform the write (submit the form, run the deploy, register the record,
115
+ update the shared value).
116
+ 2. Re-fetch the resulting state through a path independent of the one that
117
+ performed the write — a fresh page load, a separate read call, a plain
118
+ HTTP client, an independent resolver.
119
+ 3. Compare the observed value against intent, field by field.
120
+ 4. Only step 3 succeeding counts as verification. The write's own success
121
+ response, and step 1 by itself, are not evidence.
122
+ ```
123
+
124
+ If step 2 disagrees with intent: check whether the disagreement is explained by a short, bounded propagation window (EV-02, EV-06) before treating it as failure. If it is not, and the system's own readout confirms the disagreement, stop retrying and escalate to a human operator (EV-03) rather than repeating the same automated write.
125
+
126
+ ## Related
127
+
128
+ - [[.arcane/governance/web-discoverability-standards|Web Discoverability Standards]] — one consumer of these rules
@@ -0,0 +1,210 @@
1
+ ---
2
+ title: Web Discoverability Standards
3
+ audience: both
4
+ last_updated: YYYY-MM-DD
5
+ status: active
6
+ distributable: true
7
+ tags: [discoverability, metadata, crawlers, search-indexing, structured-data]
8
+ ---
9
+
10
+ # Web Discoverability Standards
11
+
12
+ How a web property becomes retrievable by search-engine crawlers and by the AI assistants that answer questions from a search index — vendor-neutral and framework-agnostic, applicable regardless of stack or hosting provider.
13
+
14
+ ## Executive Summary
15
+
16
+ - Per-route metadata must exist in the raw HTML response before any client-side JavaScript runs — link-unfurl scrapers fetch HTML once and never execute JavaScript (WD-01, WD-02).
17
+ - A route with no real content must return a genuine not-found HTTP status, not an always-200 shell that looks fine to a human and misleads a crawler (WD-03).
18
+ - Any value that legitimately differs per environment — a canonical hostname, an "is this production" flag — must be resolved at runtime from the running environment, never baked into a shared build artifact (WD-04, WD-05).
19
+ - Crawler traffic must never pass through endpoints carrying human-traffic side effects, and any crawler-facing content listing must reuse the application's own visibility rule rather than a hand-maintained copy of it (WD-06, WD-07).
20
+ - Crawl control (robots exclusion) and index control (a response header) close different gaps and are both required on non-production origins; search-index presence, not a static convention file alone, is the real lever for AI-assistant retrieval (WD-09, WD-12).
21
+
22
+ ---
23
+
24
+ ## Rule index
25
+
26
+ | ID | Rule (one line) |
27
+ |----|------------------|
28
+ | WD-01 | Per-route metadata must exist in the raw HTML response before client-side JavaScript runs. |
29
+ | WD-02 | Inject metadata at the layer already serving the built app to browsers, not a separate rendering pipeline. |
30
+ | WD-03 | Non-existent or non-public routes return a real not-found HTTP status, not an always-200 catch-all. |
31
+ | WD-04 | Values that differ per environment (e.g. canonical origin) resolve at runtime, never bake in at build time. |
32
+ | WD-05 | Confirm an "is this production" flag actually varies per environment by reading it back — don't trust its configured default. |
33
+ | WD-06 | Never route crawler traffic through endpoints with view-count, rate-limit, or other user-traffic side effects. |
34
+ | WD-07 | Crawler-facing content listings (e.g. a sitemap) reuse the application's own visibility rule — never a second copy of it. |
35
+ | WD-08 | A specific robots.txt allow-rule must be declared to override a broader disallow-prefix that would otherwise swallow it. |
36
+ | WD-09 | A non-production origin needs both a robots.txt disallow AND an index-suppression response header — not just one. |
37
+ | WD-10 | DNS apex TXT records for search-console verification are appended, never replace the existing record set (see EV-05). |
38
+ | WD-11 | Cover instant-notification protocols where supported, and keep a submitted sitemap current for engines that lack them. |
39
+ | WD-12 | For AI-assistant retrieval, search-index presence is a stronger lever than any static AI-readable convention file. |
40
+ | WD-13 | User-supplied metadata text needs HTML-attribute escaping AND, for inline JSON-LD, "<"-character escaping. |
41
+ | WD-14 | Share images follow the ~1.91:1 convention, keep a safe-content margin, and stay modest in file size. |
42
+ | WD-15 | AI image generation can't reproduce exact type/logos — composite the real assets over generated artwork instead. |
43
+
44
+ ---
45
+
46
+ ## Where metadata is produced
47
+
48
+ ### WD-01 — Metadata renders before JavaScript runs
49
+
50
+ **Rule:** Per-route share/SEO metadata — title, description, canonical URL, and structured preview tags (Open Graph, a card-style social preview format, JSON-LD) — must be present in the raw HTML response *before* any client-side JavaScript executes.
51
+
52
+ **Why:** Link-unfurl scrapers used by chat apps, messaging platforms, and social platforms fetch the HTML document once over HTTP and do not execute JavaScript. A client-side head-management approach — a library that sets `document.title` and injects meta tags after the app mounts — is invisible to that entire audience, even though it looks completely correct to a human inspecting the live DOM in a browser. The browser ran the JavaScript; the scraper didn't. "Works when I look at it" and "works for the audience that matters here" are different claims, and only one of them is being tested by opening dev tools.
53
+
54
+ ### WD-02 — Inject at the existing serving layer, not a new rendering pipeline
55
+
56
+ **Rule:** The correct injection point for per-route metadata is the layer *already* serving the built application to browsers — intercept and augment the response there (template substitution, response rewriting, an edge/proxy rule) — not a separate server-side rendering pipeline built to solve this one problem.
57
+
58
+ **Why:** That layer already sees every request and already returns the HTML document; per-route string-level substitution at that point gets the metadata into the raw response without introducing a second rendering runtime, a hydration story, or a new build target. See "Why per-route injection beats both alternatives" below for the fuller comparison.
59
+
60
+ ### WD-03 — Dead or non-public routes return a real not-found status
61
+
62
+ **Rule:** A route representing content that no longer exists, or that was never public, must return a genuine "not found" HTTP status (404, or 410 if the removal is intentional and permanent) while still rendering the application's own not-found UI in the response body.
63
+
64
+ **Why:** An always-200 catch-all — common with client-side routing, where every path resolves to the same shell and the app decides what to render after the fact — is invisible to a human browsing normally; they see a "not found" page and move on. A search engine reads the status code, not the rendered pixels. A 200 on a dead URL tells the crawler the page is valid and indexable — a "soft 404" — and the URL gets indexed, sometimes ranked, and eventually shown to a searcher as a dead end.
65
+
66
+ ---
67
+
68
+ ## Why per-route injection beats both alternatives
69
+
70
+ Two alternatives get rejected here, for different reasons.
71
+
72
+ **A full separate server-rendering pipeline (SSR/SSG)** overcorrects. It does make metadata available before any client-side JavaScript runs — but only as a side effect of rendering the *entire* page server-side. Adopting it takes on a second build/runtime target, hydration-mismatch failure modes, and an architecture decision that has nothing to do with the actual problem, which is that a handful of `<head>` tags need to exist before JavaScript runs. Teams that adopt a full rendering pipeline solely to fix metadata end up carrying its ongoing cost for the life of the application, for a problem that didn't require it.
73
+
74
+ **A client-side head/meta-tag management library** fails to solve it at all. It looks correct — inspect the live DOM in any browser and the tags are right there — but a scraper that never executes JavaScript never sees them. The library isn't buggy; it simply depends on JavaScript running to do its one job, and the audience this problem is about is defined by *not* running JavaScript.
75
+
76
+ Between an approach that solves a narrower problem than it claims to and one that solves a different, larger problem than the one that exists, neither is the right size. Per-route injection at the layer already serving the app — WD-02 — is the one that matches the actual shape of the problem.
77
+
78
+ ---
79
+
80
+ ## Values that differ by environment
81
+
82
+ ### WD-04 — Environment-specific values resolve at runtime
83
+
84
+ **Rule:** Any value that legitimately differs between deployed environments — most commonly the canonical public origin or hostname embedded in canonical URLs, structured data, or preview metadata — must be resolved at runtime from the actual running environment, never baked in at build time.
85
+
86
+ **Why:** When a single build artifact is deployed unchanged to multiple environments (build once, promote the same artifact — standard practice, and good practice for other reasons), any value fixed at build time is physically identical across every environment that artifact runs in. It cannot carry an environment-specific answer no matter how carefully it was set at build time, because build time happens once, upstream of every environment the artifact will later run in. There is no way to make a single frozen value simultaneously correct for staging and production; the fix is to stop trying and read the value from the environment that's actually running.
87
+
88
+ ### WD-05 — Confirm environment flags actually vary, don't trust their default
89
+
90
+ **Rule:** A generic "is this production" flag commonly defaults to the same value across *all* deployed environments, not only production. Branching discoverability logic on it (for example, suppressing indexing everywhere that isn't production) silently gives every environment the production answer unless that default has been checked. Before relying on any such signal, confirm it actually differs per environment by reading it back from each running environment — not by trusting how it was configured or assumed to behave.
91
+
92
+ **Why:** The mistake is invisible precisely because production looks correct — it's the one environment where the flag's default happens to match reality. The bug only shows up in every *other* environment, which is exactly where it's least likely to get noticed during a review that (reasonably) focuses attention on production. Prefer a signal already confirmed to vary — a runtime-resolved hostname, an explicitly-set per-environment value — over a flag whose default was never actually read back from a non-production environment.
93
+
94
+ ---
95
+
96
+ ## Crawler-facing paths and data access
97
+
98
+ ### WD-06 — Never route crawlers through user-traffic side effects
99
+
100
+ **Rule:** Never route crawler or bot traffic through an endpoint that has view-counting, rate-limiting-by-request, or other side effects meant for real user traffic. Serve crawler-facing responses — sitemaps, crawler-visible pages, structured-data feeds — by reading the underlying data directly, bypassing those side-effecting code paths.
101
+
102
+ **Why:** A well-behaved crawler generates a high volume of requests across a large route set in a short window, and that volume is not representative of user behavior even when the crawler is behaving exactly as intended. Run that volume through a view counter and the counter becomes meaningless noise on top of real signal. Run it through a rate limiter designed to catch abusive human traffic and it trips the limiter, throttling or blocking the crawler — and the content behind it — for doing nothing wrong.
103
+
104
+ ### WD-07 — Crawler listings reuse the application's own visibility rule
105
+
106
+ **Rule:** Any listing of publicly discoverable content generated for a crawler — most commonly a sitemap — must be derived from the *exact same* visibility rule the application's own public listing or feed uses for that content, never a second, independently written copy of that rule.
107
+
108
+ **Why:** A hand-maintained second copy of a visibility rule is correct on the day it's written and drifts silently after that. The real rule gains a condition over time — a moderation flag, a scheduled-publish date, a soft-delete check — and the sitemap-generation copy doesn't get the same update, because nothing forces the two to change together. The result is a sitemap that lists things a visitor can't actually reach, or omits things that are genuinely public. Call or import the same predicate the listing endpoint already uses; don't restate its logic anywhere else.
109
+
110
+ ---
111
+
112
+ ## Crawl control and index control
113
+
114
+ ### WD-08 — A specific allow-rule must be declared to escape a broader disallow
115
+
116
+ **Rule:** In a robots-exclusion file, when both an explicit allow-rule for a specific path (for example, a public media/asset path that page metadata references) and a broader disallow-rule for a prefix that would otherwise contain it (for example, a general API path prefix) are both relevant, the specific allow-rule *must be declared* so that it actually takes effect.
117
+
118
+ **Why:** Crawlers that respect a robots-exclusion file resolve overlapping rules by specificity — the most specific matching path generally wins over a broader one, not "whichever rule appears first in the file" and not "disallow always wins." Get the specificity wrong, or simply omit the specific allow-rule and assume the broader disallow won't apply to it, and a media-asset path nested under a disallowed API prefix becomes entirely invisible to image crawling — even though the page's own metadata correctly points at that image. A correct metadata reference doesn't help if the crawler was never permitted to fetch what it points to.
119
+
120
+ ```txt
121
+ User-agent: *
122
+ Allow: /api/public-media/
123
+ Disallow: /api/
124
+ ```
125
+
126
+ The allow-rule for the more specific path has to exist in the file; it isn't inferred from the disallow-rule simply being broader.
127
+
128
+ ### WD-09 — Non-production origins need crawl control AND index control
129
+
130
+ **Rule:** A non-production or staging origin needs *both* a crawl-blocking mechanism (a robots-exclusion rule disallowing the origin) *and* an index-suppression response header (for example, a `noindex` directive delivered via response header, or an equivalent tag) on every response.
131
+
132
+ ```http
133
+ X-Robots-Tag: noindex, nofollow
134
+ ```
135
+
136
+ ```txt
137
+ User-agent: *
138
+ Disallow: /
139
+ ```
140
+
141
+ **Why:** see "Why two noindex mechanisms, not one" below.
142
+
143
+ ---
144
+
145
+ ## Why two noindex mechanisms, not one
146
+
147
+ The two mechanisms close different gaps, and neither one covers the other's gap.
148
+
149
+ Crawl-blocking (the robots-exclusion rule) stops a well-behaved crawler from *fetching* content at all. It does not stop a search engine from listing the bare URL if that URL is linked to from somewhere else the crawler *does* fetch — a URL-only listing with no title or snippet, because the crawler was never allowed in to get one, but a listing all the same.
150
+
151
+ Index-suppression (the response header) stops indexing outright — but only for a request the crawler actually makes. A crawler that obeyed the crawl-block never fetches the response, and so never sees the header telling it not to index. The header can't suppress what the crawler was never allowed to retrieve.
152
+
153
+ Using only the robots-exclusion rule leaves the bare-URL-listing gap open. Using only the header leaves the origin fully crawlable by anything that doesn't respect (or bothers to check) the crawl-block. Only running both closes both gaps, which is why WD-09 requires both, not either.
154
+
155
+ ---
156
+
157
+ ## Registration, DNS, and notification
158
+
159
+ ### WD-10 — Apex DNS verification records are appended, not replaced
160
+
161
+ **Rule:** Registering a property's ownership with a search console commonly requires adding a DNS TXT record at the domain apex. Follow EV-05 (see [[.arcane/governance/external-verification-standards|External Verification Standards]]) when writing it, rather than re-deriving the append-vs-replace reasoning here — the apex TXT record set is very often already carrying other unrelated values (a mail-sender authorization policy, other verification strings), and a careless write destroys them.
162
+
163
+ ### WD-11 — Cover both instant notification and sitemap freshness
164
+
165
+ **Rule:** Instant crawl/recrawl notification protocols exist and are adopted by some search engines and services — IndexNow is one — but not by the search engine with the largest share of global query volume, which instead relies on periodically re-fetching a submitted sitemap. Covering "notify search engines the moment content changes" therefore requires *both*: an instant-notification call where the protocol is supported, *and* ensuring the sitemap itself is submitted through the relevant search console and kept current for the engine(s) that don't support instant notification.
166
+
167
+ ### WD-12 — Search-index presence outweighs a static AI-readable file
168
+
169
+ **Rule/observation:** For the specific goal of being retrievable *by AI assistants* — as distinct from being found by a human searching directly — being present and current in a major search engine's index is a substantially stronger lever than any single static "AI-readable" convention file placed at a well-known path (for example, a plain-text site-description file such as the emerging `llms.txt` convention). Several AI assistants answer by drawing on a search engine's index rather than crawling the web independently, so an assistant's ability to surface a page is frequently bottlenecked on that page's search-index presence, not on whether a convention file exists at all. A convention file is still worth publishing — it costs little and helps the assistants that do read it directly — but it should not be mistaken for the primary lever. The everything-else in this document (WD-01 through WD-11) is the primary lever.
170
+
171
+ ---
172
+
173
+ ## Untrusted content in metadata
174
+
175
+ ### WD-13 — Two escaping treatments, not one
176
+
177
+ **Rule:** Any user-supplied text embedded into page metadata needs two independent escaping treatments:
178
+
179
+ 1. Standard HTML-attribute escaping for the visible meta tags (title, description, preview-card values) — the ordinary defense against a value breaking out of an attribute or tag context.
180
+ 2. Separately, for structured data embedded as inline JSON inside a `<script type="application/ld+json">` tag: escape the `<` character wherever it appears inside JSON string values (for example, to its Unicode-escape form), in addition to standard JSON string escaping.
181
+
182
+ ```json
183
+ {
184
+ "@type": "Review",
185
+ "reviewBody": "quoted text containing </script><script>..."
186
+ }
187
+ ```
188
+
189
+ **Why:** A literal closing-script-tag sequence smuggled inside a JSON string value can prematurely terminate the surrounding script block the moment the HTML parser reads it — regardless of whether the JSON itself is well-formed and correctly escaped by a standard JSON serializer. A JSON serializer's job is producing valid JSON; it has no reason to know its output is about to be embedded inside an HTML `<script>` tag, so it has no reason to escape a byte sequence that is perfectly legal JSON but breaks the HTML around it. The two treatments defend two different parsing contexts reading the same untrusted input — the HTML attribute parser for the visible meta tags, and the HTML parser's script-tag boundary detection for the JSON-LD block — and one does not substitute for the other.
190
+
191
+ ---
192
+
193
+ ## Share images
194
+
195
+ ### WD-14 — Aspect ratio, safe margin, file size
196
+
197
+ **Rule:** Link-preview (share) images should follow the aspect ratio broadly adopted across major platforms — close to 1.91:1 — with a safe-content margin around the edges so platform-specific cropping doesn't cut off text or a focal subject, and should stay modest in file size.
198
+
199
+ **Why:** Scrapers fetch preview images synchronously as part of unfurling a link, and some platforms silently refuse to fetch, or silently fail to display, an image above their own size ceiling — there is usually no visible error; the preview simply renders without an image. A safe-content margin matters because different platforms crop the same source image to slightly different final frames; content placed right at the edge survives on some platforms and gets cut on others.
200
+
201
+ ### WD-15 — Composite real assets; don't ask a generator for exact type or logos
202
+
203
+ **Rule:** Current AI image-generation tools cannot reliably reproduce an exact existing typeface or an exact existing vector logo/mark — both come out as a visible approximation, and no amount of prompt refinement reliably fixes this. It's a property of how these generators work (they produce plausible-looking glyphs and marks probabilistically rather than reproducing exact vector paths or font outlines), not a quality issue specific to any one generation attempt. When brand-exact typography or an exact logo mark matters for a share image, composite the real font files and the real vector logo asset over separately generated artwork or background — don't ask the generator to produce the whole image including the exact type and mark.
204
+
205
+ ---
206
+
207
+ ## Related
208
+
209
+ - [[.arcane/governance/external-verification-standards|External Verification Standards]] — the verification steps WD-10 (and any other rule here that touches DNS records or an external console) depends on.
210
+ - A consuming spell should cite these rule IDs (WD-01 … WD-15) rather than restate them.
@@ -0,0 +1,9 @@
1
+ # Make Discoverable
2
+
3
+ Invoke the Arcane `spell-make-discoverable` spell workflow.
4
+
5
+ See the full prompt at `.github/prompts/spell-make-discoverable.prompt.md` for the complete workflow definition.
6
+
7
+ ---
8
+
9
+ @.github/prompts/spell-make-discoverable.prompt.md