@relvehq/agent-ready 0.0.0-stage → 0.2.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Relve
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,338 @@
-# Temporary Holding Version
-
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+# @relvehq/agent-ready
+
+Make your site agent-ready. When an AI agent (ChatGPT, Claude, Perplexity,
+Copilot…) asks for one of your pages as Markdown, it gets a clean, small
+Markdown version instead of 50,000+ tokens of HTML. Browsers get exactly what
+they got before.
+
+- **Serve**: Markdown on `Accept: text/markdown`, on `<page>.md` and on
+  `?format=md`, plus a generated [`/llms.txt`](https://llmstxt.org). No AI
+  runs per request: it is plain code, in milliseconds.
+- **Refine** (optional, with your own prompt files): an AI model writes a
+  better Markdown file for each page ahead of time, a script checks every
+  number and URL against the page,
+  and unchanged pages are skipped on the next run.
+- **Proof**: a free checker that scores any site out of 100, and a dashboard
+  of which AI agents read which pages.
+
+On a real tool page: raw HTML 97,944 tokens, a generic converter 4,618, a
+refined file 1,695.
+
+**Start here:** the onboarding guide at <https://relvehq.com/agent-ready>
+(every stack, the WordPress plugin download, and how to test it).
+
+## Requirements
+
+- Node.js 20 or newer
+- For the Next.js adapter: Next.js 13.4+ with the App Router
+- Anything else: the core is plain `Request` / `Response` (see "Other frameworks")
+
+## Quick start: one command
+
+In your project's root:
+
+```
+npx @relvehq/agent-ready init
+```
+
+It detects Next.js, Node servers and WordPress, and sets up the one it finds:
+
+| It finds | It does |
+|---|---|
+| `app/` or `src/app/` and `next` in `package.json` | writes the Next.js files (below) |
+| `wp-config.php` or `wp-content/` | copies the bundled plugin to `wp-content/plugins/relve-agent-ready/`, then tells you to activate it and tick Enabled on Settings > Agent Ready (an existing copy is kept unless you add `--force`) |
+| `express`, `connect`, `polka`, `fastify`, `@nestjs/core`, `h3` or `nitropack` in `package.json` | writes nothing; prints the exact lines for that framework and the env vars |
+| none of these | prints what it looked for and the guide's URL, and exits 1 |
+
+`--stack next|node|wordpress` skips detection; `--dry-run` shows the plan and
+writes nothing. Step-by-step for every stack: [SETUP.md](./SETUP.md).
+
+### Next.js
+
+```
+npm install @relvehq/agent-ready
+npx agent-ready init
+```
+
+`init` looks at your project (`app/` or `src/app/`, TypeScript or JavaScript)
+and writes:
+
+| File | What it holds |
+|---|---|
+| `agent-ready.config.ts` | `export default createAgentReady()`, smart defaults |
+| `middleware.ts` | `export default withAgentReady(agentReady)` and the matcher (only when you have no middleware yet) |
+| `app/%5F%5Fmd/[[...path]]/route.ts` | the internal route that builds the Markdown (`%5F` is `_`) |
+| `.env.local` | `AGENT_READY_ENABLED=1` and `NEXT_PUBLIC_SITE_URL=http://localhost:3000` |
+
+It never edits a file you already have. If you have a middleware, it prints
+the two lines to add:
+
+```ts
+import { withAgentReady } from "@relvehq/agent-ready/next";
+import agentReady from "./agent-ready.config";
+export default withAgentReady(agentReady, yourMiddleware);
+// and add to your matcher: "/((?!_next/).+\\.md)", "/llms.txt"
+```
+
+Running `init` again changes nothing; `--dry-run` shows the plan.
+
+Check it with the dev server running:
+
+```
+curl -i -H "Accept: text/markdown" http://localhost:3000/
+curl -i http://localhost:3000/index.md
+curl -i http://localhost:3000/llms.txt
+```
+
+In production set `AGENT_READY_ENABLED=1` and `NEXT_PUBLIC_SITE_URL=https://your-site.com`
+on your host. Without a site URL the package stays off (it never fails your build).
+
+### Node server and WordPress
+
+For a Node server, `init` prints the lines for your framework (Express,
+Connect, Polka, Nest: `app.use(...)`; Fastify through `@fastify/middie`;
+Nitro and h3 through `fromNodeMiddleware`), and the two env vars:
+`AGENT_READY_ENABLED=1` and `SITE_URL=https://your-site.com`. See "Other
+frameworks" below. For WordPress, the plugin ships inside this package
+(without its tests), so `init` needs no download.
+
+### What `createAgentReady()` works out on its own
+
+| Setting | Default |
+|---|---|
+| `site` | the first set of `NEXT_PUBLIC_SITE_URL`, `SITE_URL`, `NEXT_PUBLIC_APP_URL`, `URL` (Netlify), `VERCEL_PROJECT_PRODUCTION_URL`, `VERCEL_URL` |
+| `enabled` | `AGENT_READY_ENABLED` is `1` or `true` |
+| `exclude` | `/api`, `/_next`, auth and admin paths, plus common private pages (account, checkout, cart, dashboard, settings, billing…) and yours; `excludeCommon: false` drops the common list |
+| `llms` | `/llms.txt` built from your `sitemap.xml`; `llms: false` for none |
+| `onHit` | sends each visit to a dashboard when `AGENT_READY_KEY` and `AGENT_READY_ENDPOINT` are set |
+
+Anything you pass wins:
+
+```ts
+export default createAgentReady({
+  exclude: ["/members"],
+  templates: { "/blog/:slug": async ({ params }) => renderPostMarkdown(params.slug) },
+});
+```
+
+## How an agent asks
+
+| Ask | Example |
+|---|---|
+| `Accept` header | `Accept: text/markdown`, preferred over (or tied with) `text/html` |
+| `.md` URL | `/pricing.md`, `/index.md` for `/` |
+| query | `/pricing?format=md` |
+
+A browser's `Accept`, a bare `*/*`, or `text/markdown` with a lower q-value
+than `text/html` is not a Markdown request. The user agent is never used to
+choose what to serve: that would be cloaking.
+
+## How it works
+
+1. **Detect** (middleware, Edge): is this a Markdown request? If not, nothing changes.
+2. **Rewrite** to the internal route `/__md/<path>`; the public URL stays the same.
+3. **Resolve** (Node route), first hit wins:
+   1. a **template** for the route (`"/blog/:slug"`), written from your data
+   2. a **stored** file (`store.get(path)`), e.g. one written by `refine`
+   3. **convert**: fetch the page's own HTML (public, no cookies) and turn its
+      main content into Markdown (Readability + turndown, GFM tables). Under 200
+      characters of real text is discarded.
+   4. nothing usable: an `Accept` request gets the normal HTML page, a `.md`
+      request gets a 404
+4. **Enrich**: YAML front matter (title, canonical, source), facts from JSON-LD, token count.
+5. **Respond**:
+
+```
+Content-Type: text/markdown; charset=utf-8
+Vary: Accept
+Link: <https://your-site.com/pricing>; rel="canonical"
+X-Robots-Tag: noindex            (search engines keep indexing the HTML page)
+x-markdown-tokens: 314
+ETag: W/"…"                      (If-None-Match → 304)
+Cache-Control: see "Caching"
+x-agent-ready-source: native | stored | convert
+```
+
+6. **Measure**: one `HitEvent` per request to `onHit`, fire-and-forget.
+
+## Caching behind a CDN
+
+Most CDNs (CloudFront included) don't vary their cache on `Accept`, so:
+
+- Responses negotiated by `Accept` use `negotiatedCacheControl`, default
+  `private, max-age=300`. A shared cache never stores them, so a browser is
+  never served Markdown.
+- `.md`, `?format=md` and `llms.txt` are their own URLs and use `cacheControl`,
+  default `public, s-maxage=300, stale-while-revalidate=86400`.
+- If the CDN already has the HTML page cached, an `Accept` request may get that
+  HTML. Safe, just not Markdown. `.md` URLs always work; reliable negotiation
+  needs `Accept` in the CDN cache key.
+
+## Refine: better Markdown, written ahead of time
+
+```
+npx agent-ready refine --site https://your-site.com --prompts ./prompts --sitemap --limit 20 --dry-run
+npx agent-ready refine --site https://your-site.com --prompts ./prompts --site-name "Your Name" --sitemap --limit 20
+```
+
+For each page: fetch it, extract its text, links, JSON-LD and a fingerprint;
+skip it if the fingerprint matches the saved file; otherwise fill the prompt,
+call the model once, check the result, retry once with the problems listed,
+and save `<out>/<path>.md` plus `<out>/<path>.json` (default out:
+`.agent-ready/md`). `--dry-run` only reads pages: no model call, nothing written.
+
+| Setting | |
+|---|---|
+| `OPENAI_API_KEY` | OpenAI, default model `gpt-4o` |
+| `ANTHROPIC_API_KEY` | Claude, default model `claude-opus-5-5` |
+| `AGENT_READY_MODEL` | any other model id, e.g. `gpt-4o-mini` |
+
+Keys are read from `.env.local` and `.env`. Serve the files from the route
+file (`fileStore` is Node only, so keep it out of `agent-ready.config.ts`,
+which the Edge middleware imports):
+
+```ts
+// app/%5F%5Fmd/[[...path]]/route.ts
+import { createMarkdownRoute } from "@relvehq/agent-ready/next/route";
+import { fileStore } from "@relvehq/agent-ready/refine/file-store";
+import agentReady from "../../../agent-ready.config";
+export const { GET, HEAD } = createMarkdownRoute({ ...agentReady, store: fileStore(".agent-ready/md") });
+export const runtime = "nodejs";
+export const dynamic = "force-dynamic";
+```
+
+On serverless hosts (Vercel, Amplify) the folder must be bundled with the
+route (`outputFileTracingIncludes`); see [SETUP.md](./SETUP.md).
+
+**The check** rejects a file when a number or URL in it is not on the page, a
+required section is missing, it has more or fewer than one H1, HTML tags, a
+wrong fingerprint, or it is over the token budget. It cannot catch a reworded
+claim, so read a sample before you switch on.
+
+**The prompt.** The package ships none. Point `refine` at a folder of prompt
+files with `--prompts <dir>` (several: `a,b`) or the `AGENT_READY_PROMPTS`
+variable (`.env.local`, `.env` or the shell); without one it stops before
+fetching anything. A prompt file is Markdown with `## SYSTEM`, `## USER`
+(`{{PAGE_URL}}`, `{{PAGE_TEXT}}`, `{{PAGE_LINKS}}`, `{{JSON_LD}}`,
+`{{SOURCE_HASH}}`) and a `## Settings` table (`page_type`, `version`,
+`required_sections`, `token_budget`). A `page_type` of `page` takes every
+path no other prompt claims; for it the code guesses the page's kind from its
+URL and JSON-LD (home, product, pricing, article, docs, listing, event, about,
+contact, legal, other) and passes the guess as `{{PAGE_KIND_HINT}}`.
+
+## Proof
+
+### Free checker
+
+```
+npx agent-ready check https://any-site.com/pricing
+npx agent-ready check any-site.com --json
+```
+
+Read-only. Scores the site out of 100 (A to F): Markdown for `Accept` (25),
+a `.md` URL (15), `/llms.txt` (15), robots.txt letting AI answer bots in (15),
+a sitemap (10), JSON-LD (10), `Vary: Accept` (5), a canonical (5). Each failing
+check says what to do, and it compares the tokens an agent reads today with
+what agent-ready would serve. In code: `auditSite(url)`.
+
+### Dashboard
+
+```ts
+// agent-ready.config.ts
+import { createAgentReady, memoryHitStore } from "@relvehq/agent-ready";
+export const hits = memoryHitStore();
+export default createAgentReady({ hits });
+
+// app/agent-ready-dashboard/route.ts
+import { createDashboardRoute } from "@relvehq/agent-ready/next/dashboard";
+import { hits } from "../../agent-ready.config";
+export const { GET } = createDashboardRoute({ store: hits, site: "your-site.com" });
+export const dynamic = "force-dynamic";
+```
+
+Open `/agent-ready-dashboard?key=<AGENT_READY_DASHBOARD_KEY>` (add
+`&format=json` for the data, `&days=7` for a shorter window). Without a key
+configured the route answers 404. It shows visits per day, who is reading (AI
+assistants, AI search, training bots, search engines), top pages, how agents
+asked, where the Markdown came from, tokens served and response time.
+
+A `HitStore` is two methods, `add` and `list`. `memoryHitStore()` suits a
+long-running server; `jsonlHitStore(file)` (`@relvehq/agent-ready/proof/jsonl`)
+writes a log that `npx agent-ready stats --log <file>` summarises. On
+serverless hosts (Vercel, Amplify) memory and local files don't last between
+requests: implement `HitStore` on your database.
+
+## Other frameworks
+
+Express, Connect, Polka, Fastify (via `@fastify/middie`) or a plain `http`
+server: one middleware. Mount it before your static files and routes:
+
+```ts
+import express from "express";
+import { createAgentReady } from "@relvehq/agent-ready";
+import { agentReadyNode } from "@relvehq/agent-ready/node";
+
+const app = express();
+app.use(agentReadyNode(createAgentReady({ site: "https://your-site.com" })));
+// app.use(express.static("public")); your routes...
+```
+
+It answers Markdown requests and calls `next()` for everything else (and for
+an `Accept` request when no Markdown is available, so the page's HTML is
+served). Plain `http`: `http.createServer((req, res) => md(req, res, () => app(req, res)))`.
+
+On other runtimes (Bun, Deno, Workers, Hono…) the core uses only `Request` and
+`Response`. Detect, enrich and respond run anywhere; `resolveMarkdown` (its
+convert step) needs Node, so on Edge-only runtimes use templates and a store
+with `fallback: "none"`:
+
+```ts
+import { defineAgentReady, wantsMarkdown, enrich, markdownResponse } from "@relvehq/agent-ready";
+import { resolveMarkdown } from "@relvehq/agent-ready/core/resolve"; // Node only
+
+const cfg = defineAgentReady({ site: "https://your-site.com", enabled: true });
+const d = wantsMarkdown(request);
+if (d.markdown) {
+  const r = await resolveMarkdown(cfg, { path: d.path, search: d.search, request });
+  if (r.doc) {
+    const { body, tokens } = enrich(r.doc, { canonical: cfg.site + d.path, sourceUrl: r.sourceUrl, source: r.source! });
+    return markdownResponse(body, { request, canonical: cfg.site + d.path, tokens, cacheControl: cfg.cacheControl });
+  }
+}
+```
+
+`@relvehq/agent-ready` and `@relvehq/agent-ready/next` are Edge-safe.
+`/node`, `/next/route`, `/core/resolve`, `/core/convert` and `/refine/*` need Node.
+
+## Config reference (`defineAgentReady` / `createAgentReady`)
+
+| Key | Default | Notes |
+|---|---|---|
+| `site` | required (`createAgentReady`: from env) | public origin, for canonicals and absolute links |
+| `enabled` | `false` | boolean or `(request) => boolean` |
+| `exclude` | `[]` | path prefixes or RegExps, added to the always-excluded list |
+| `templates` | `{}` | `{ "/blog/:slug": fn }`, first matching pattern wins |
+| `store` | `null` | `{ get(path) }` |
+| `fallback` | `"convert"` | `"none"` serves only templates and stored files |
+| `cacheControl` | `public, s-maxage=300, stale-while-revalidate=86400` | `.md`, `?format=md`, llms.txt |
+| `negotiatedCacheControl` | `private, max-age=300` | `Accept` requests |
+| `llms` | `null` (`createAgentReady`: from the sitemap) | `{ path, title, summary, details, entries }` |
+| `onHit` | `null` | `(HitEvent) => void` |
+| `internalPrefix` | `/__md` | rewrite target |
+| `fetchTimeoutMs` | `8000` | page fetch for convert |
+| `minChars` | `200` | quality gate for convert |
+
+## Commands
+
+```
+agent-ready init   [--stack next|node|wordpress] [--dry-run] [--force] [--cwd <dir>]
+agent-ready check  <url> [--json]
+agent-ready refine --site <origin> (--paths /a,/b | --sitemap) --prompts <dir> [--site-name <name>]
+                   [--out <dir>] [--limit <n>] [--force] [--concurrency <n>] [--dry-run]
+agent-ready stats  --log <hits.jsonl> [--days <n>] [--json]
+```
+
+## License
+
+MIT © [Relve](https://relvehq.com). Built by the team behind Relve, the AI
+tools and market intelligence platform.