@relvehq/agent-ready 0.0.0-stage → 0.3.0

package/SETUP.md

@@ -0,0 +1,354 @@
+# Setting up @relvehq/agent-ready
+
+A step-by-step guide for the developer who adds this package to a site. The onboarding page with the same steps, the WordPress plugin download and a test block: https://relvehq.com/agent-ready
+
+## The one command
+
+In your project's root:
+
+```
+npx @relvehq/agent-ready init
+```
+
+It detects Next.js, Node servers and WordPress, and sets up the one it finds:
+
+| It finds | It does |
+|---|---|
+| `app/` or `src/app/` and `next` in `package.json` | Writes the Next.js files (Path 1), or prints the two lines to add to your middleware (Path 2). |
+| `wp-config.php` or `wp-content/` | Copies the Relve Agent Ready plugin, bundled in the package, to `wp-content/plugins/relve-agent-ready/`. Then activate it and tick Enabled (Path 5). A copy that is already there is kept unless you add `--force`. |
+| `express`, `connect`, `polka`, `fastify`, `@nestjs/core`, `h3` or `nitropack` in `package.json` | Writes nothing. Prints the exact lines for that framework and the env vars (Path 3). |
+| None of these | Prints what it looked for and the onboarding URL, and exits with code 1. |
+
+`--stack next`, `--stack node` or `--stack wordpress` skips detection. `--dry-run` prints the plan and writes nothing. The paths below are the same steps by hand, in detail.
+
+## What this does
+
+- When an AI agent (ChatGPT, Claude, Perplexity and others) asks for a page as Markdown, your site answers with a small, clean Markdown version of that page.
+- It also serves `/llms.txt`, a list of your pages built from your `sitemap.xml`.
+- No AI runs on your server. It is plain code that answers in milliseconds.
+
+What it does not change:
+
+- Browsers get exactly the same HTML as before.
+- Google and other search engines keep indexing your HTML pages. Markdown responses carry `X-Robots-Tag: noindex`.
+- Your URLs, sitemap and robots.txt stay as they are.
+
+## Before you start
+
+Paths 1 to 4 need Node.js 20 or newer. Path 5 (WordPress) needs no Node at all (Node only makes the install one command). Find your stack:
+
+| Your site | What you do |
+|---|---|
+| Next.js 13.4+ with the App Router | Path 1 or Path 2. One command sets it up. |
+| Any Node server (Express, Connect, Polka, Fastify, plain `http`) | Path 3. One middleware line. |
+| Another runtime that uses `Request` / `Response` | Path 4. Use the core. |
+| WordPress (self-hosted) | Path 5. Supported through the Relve Agent Ready plugin. No code. |
+| WordPress.com | Path 5, only on plans that allow uploading plugins. |
+| Webflow, Wix, Squarespace, Framer or another no-code builder | Not supported. These hosts do not let you run your own server code. |
+
+## Score your site first
+
+Run the free checker. It only reads your site; it changes nothing.
+
+```
+npx @relvehq/agent-ready check https://your-site.com
+```
+
+It scores the site out of 100 and says what to fix. Save the result so you can compare after setup. Once the package is installed in your project, the short form works too:
+
+```
+npx agent-ready check https://your-site.com
+```
+
+(Do not run `npx agent-ready` before installing: that would look for a different, unscoped package.)
+
+## Path 1: Next.js, no middleware yet
+
+```
+npm install @relvehq/agent-ready
+npx agent-ready init
+```
+
+`init` writes these files. It never edits a file you already have. Use `npx agent-ready init --dry-run` to see the plan first.
+
+| File | What it holds |
+|---|---|
+| `agent-ready.config.ts` | `export default createAgentReady({})` with the defaults |
+| `middleware.ts` | sends Markdown requests to the route below; everything else carries on |
+| `app/%5F%5Fmd/[[...path]]/route.ts` | the internal route that builds the Markdown (`%5F` is `_`) |
+| `.env.local` | `AGENT_READY_ENABLED=1` and `NEXT_PUBLIC_SITE_URL=http://localhost:3000` |
+
+With a `src/` folder, the files go under `src/`. Without `tsconfig.json`, they are `.js`. Commit all of them except `.env.local`.
+
+## Path 2: Next.js with an existing middleware
+
+Run the same two commands as Path 1. `init` sees your middleware, leaves it alone, and prints what to add:
+
+```ts
+// 1. Wrap your middleware:
+import { withAgentReady } from "@relvehq/agent-ready/next";
+import agentReady from "./agent-ready.config";
+export default withAgentReady(agentReady, yourMiddleware);   // instead of exporting yourMiddleware
+
+// 2. Add these two entries to your matcher:
+//    "/((?!_next/).+\\.md)", "/llms.txt"
+```
+
+Notes:
+
+- `yourMiddleware` is your current middleware function. Remove its old `export default` (or `export function middleware`).
+- Markdown requests never reach your middleware. All other requests reach it unchanged.
+- If your matcher skips paths that contain a dot, the `.md` entry is what lets `/pricing.md` through. Keep both new entries as plain strings in `export const config = { matcher: [...] }`. Next.js reads the matcher at build time, so it cannot be imported from a variable.
+
+## Path 3: Node server (Express, Connect, Polka, plain http)
+
+`npx @relvehq/agent-ready init` prints these lines for your framework (it writes no file). By hand:
+
+```
+npm install @relvehq/agent-ready
+```
+
+```ts
+import express from "express";
+import { createAgentReady } from "@relvehq/agent-ready";
+import { agentReadyNode } from "@relvehq/agent-ready/node";
+
+const app = express();
+
+// Mount it first: before static files, routes and any HTML rendering.
+app.use(agentReadyNode(createAgentReady({ site: "https://your-site.com" })));
+
+app.use(express.static("public"));
+// ...your routes
+```
+
+- A request that does not want Markdown goes straight to `next()`. Your app answers it as before.
+- `createAgentReady` still reads `AGENT_READY_ENABLED`. Set it to `1` (see "Go live") or pass `enabled: true`.
+- Connect and Polka: the same `app.use(...)` line.
+- Plain `http`: call the middleware first in your handler, with your own handler as `next`.
+- Fastify: register `@fastify/middie`, then the same line:
+
+```ts
+await app.register(import("@fastify/middie"));
+app.use(agentReadyNode(createAgentReady({ site: "https://your-site.com" })));
+```
+
+- NestJS: the same `app.use(...)` line in `main.ts`, after `NestFactory.create(...)` and before `app.listen(...)`.
+- Nitro and h3: wrap it with `fromNodeMiddleware` from `h3`. In Nitro, as `server/middleware/agent-ready.ts`:
+
+```ts
+import { fromNodeMiddleware } from "h3";
+import { createAgentReady } from "@relvehq/agent-ready";
+import { agentReadyNode } from "@relvehq/agent-ready/node";
+
+export default fromNodeMiddleware(agentReadyNode(createAgentReady({ site: process.env.SITE_URL })));
+```
+
+## Path 4: anything else
+
+The core works with Web `Request` and `Response`, so it fits any framework built on them (Hono, Bun, Deno and similar). The step that builds Markdown from your HTML needs Node APIs. See "Other frameworks" in the README for the short snippet that wires it.
+
+## Path 5: WordPress
+
+WordPress runs PHP, so the site does not run this npm package. It runs a plugin instead, **Relve Agent Ready**, which answers the same requests with the same headers, and builds the Markdown from WordPress's own data (title, author, dates, categories, tags, the post content). No code to write.
+
+Get the plugin one of two ways:
+
+- **With Node, on the server or in your local copy of the site:** run `npx @relvehq/agent-ready init` in the WordPress root (the folder with `wp-config.php`). It copies the plugin to `wp-content/plugins/relve-agent-ready/`; then skip to step 2.
+- **Without Node:** download the zip from https://relvehq.com/agent-ready (or the one Relve sends you, `relve-agent-ready-<version>.zip`) and upload it in step 1.
+
+The plugin's own tests are never part of what is installed.
+
+1. **Install.** In WordPress admin: Plugins, Add New, Upload Plugin. Choose the zip, then Install Now.
+2. **Activate** it. It is installed switched off: nothing changes on your site yet.
+3. **Switch it on.** Go to Settings > Agent Ready:
+   - Tick **Enabled**.
+   - **Post types**: tick the ones agents should read (posts and pages are ticked by default). The front page and category, tag and author archives are served when Posts is ticked.
+   - **Excluded paths**: one path prefix per line, for anything that must never be served as Markdown.
+   - **llms.txt**: `auto` (the default), `on` or `off`. `auto` serves `/llms.txt` unless Yoast SEO, or a real `llms.txt` file on the server, already serves one; if you use Yoast's llms.txt, leave it on `auto` or set `off`. `on` serves ours even when Yoast has one.
+   - Save. The status panel at the top of the page shows whether it is on and the test commands for your site.
+
+### Test it
+
+Swap in a real post URL from your site:
+
+```
+curl -i -H "Accept: text/markdown" https://your-site.com/blog/your-post/
+curl -i https://your-site.com/blog/your-post.md
+curl -i "https://your-site.com/blog/your-post/?format=md"
+curl -i https://your-site.com/blog/llms.txt
+```
+
+`/llms.txt` lives under the site's home path: on a blog at `your-site.com/blog`, it is `/blog/llms.txt`; on a site at the domain root, `/llms.txt`.
+
+What you should see:
+
+| Command | Expected |
+|---|---|
+| `Accept: text/markdown` | `200`, `Content-Type: text/markdown; charset=utf-8`, `x-agent-ready-source: native` (or `stored`), `Vary: Accept` |
+| `.md` URL and `?format=md` | the same Markdown body |
+| `llms.txt` | `200`, `Content-Type: text/plain; charset=utf-8`, links to your posts and pages |
+
+Then open the post in a browser. It must look exactly as before.
+
+### If a Cloudflare Worker or a reverse proxy serves the blog on your-site.com/blog
+
+Many sites serve WordPress at `your-site.com/blog` through a Cloudflare Worker, Nginx or another proxy in front of the WordPress server. The plugin only sees what the proxy passes on. The proxy must:
+
+- forward the `Accept` header unchanged
+- forward paths that end in `.md` (some proxies only forward known page paths or drop paths with a dot)
+- forward the query string, so `?format=md` arrives
+- route `/blog/llms.txt` to WordPress like any other blog path
+- not cache by URL alone for requests with `Accept: text/markdown`: either pass them through uncached, or add `Accept` to the cache key. Otherwise an agent may get a cached HTML page, or a browser may get Markdown.
+
+`?format=md` is the most robust of the three: it works through any proxy that forwards the query string.
+
+### Page caches (WP Rocket and similar)
+
+The plugin tells WP Rocket not to cache `.md` and `?format=md` responses, and marks every Markdown response as not cacheable (`DONOTCACHEPAGE`). A page cache can still answer an `Accept: text/markdown` request with the HTML it already holds, because it runs before WordPress and does not look at `Accept`. That is safe, just not Markdown. `.md` URLs and `?format=md` always work.
+
+### Refined files on WordPress
+
+With your own OpenAI key, the plugin can write a better Markdown file for each post itself: see the plugin's settings page (Settings > Agent Ready) and its readme. You can also upload a zip of `.md` files you made with `npx agent-ready refine` (zip the `.agent-ready/md` folder): under the refined files section, choose it and click "Upload zip" (up to 20 MB, 2 MB per file). Each upload replaces the whole stored set; "Clear refined files" removes them all. See "Add refined files" below for how to check it.
+
+## Test it locally
+
+Start your dev server, then run these three commands (change the port if yours is not 3000):
+
+```
+curl -i -H "Accept: text/markdown" http://localhost:3000/
+curl -i http://localhost:3000/index.md
+curl -i http://localhost:3000/llms.txt
+```
+
+What you should see:
+
+| Command | Expected |
+|---|---|
+| `Accept: text/markdown` on `/` | `200`, `Content-Type: text/markdown; charset=utf-8`, `x-agent-ready-source: convert` (or `native` / `stored`), `Vary: Accept` |
+| `/index.md` | the same Markdown body |
+| `/llms.txt` | `200`, `Content-Type: text/plain; charset=utf-8`, a list of your pages |
+
+Then open `http://localhost:3000/` in a browser. It must look exactly as before.
+
+Score the local site too:
+
+```
+npx agent-ready check http://localhost:3000
+```
+
+## Go live
+
+Set these environment variables on your host:
+
+| Variable | Value |
+|---|---|
+| `AGENT_READY_ENABLED` | `1` (anything else, or unset, means off) |
+| `NEXT_PUBLIC_SITE_URL` | your public origin, e.g. `https://your-site.com` |
+
+The site URL can come from any of these, first one set wins: `NEXT_PUBLIC_SITE_URL`, `SITE_URL`, `NEXT_PUBLIC_APP_URL`, `URL` (set by Netlify), `VERCEL_PROJECT_PRODUCTION_URL`, `VERCEL_URL` (set by Vercel). Passing `site` to `createAgentReady` wins over all of them. With no site URL the package stays off and logs one warning; it never fails your build.
+
+Where to set them:
+
+- Vercel: Project Settings, Environment Variables, Production.
+- AWS Amplify: App settings, Environment variables.
+- Netlify: Site configuration, Environment variables.
+- Anything else: wherever your host or container sets env vars.
+
+Then redeploy (env vars are read at build and start time) and score the live site again:
+
+```
+npx @relvehq/agent-ready check https://your-site.com
+```
+
+### Behind a CDN (CloudFront, Cloudflare, Fastly)
+
+Most CDNs do not put `Accept` in the cache key. If the CDN already holds the HTML for a page, an `Accept: text/markdown` request may get that HTML. That is safe, just not Markdown. `.md` URLs and `/llms.txt` always work, because they are their own URLs. For reliable `Accept` negotiation, add the `Accept` header to the CDN cache key.
+
+## Add refined files: better Markdown with your own OpenAI key (optional)
+
+One command writes a better Markdown file for each page ahead of time, with Relve's prompt and your own OpenAI key (BYOK).
+
+1. Put your key in `.env.local`, `.env` or the shell as `OPENAI_API_KEY`, then run from your project root:
+
+```
+npx agent-ready refine --site https://your-site.com --sitemap --dry-run
+npx agent-ready refine --site https://your-site.com --sitemap
+```
+
+`--dry-run` lists the pages and calls nothing. The real run fetches each page's HTML and sends it to the Relve API on relvehq.com with your key. Relve's prompt runs on Relve's server, calls OpenAI with your key, checks the result (every number and URL must be on the page) and returns the Markdown. The files land in `.agent-ready/md`: `index.md`, `pricing.md`, each with a `.json` next to it (fingerprint, model, tokens, date).
+
+- What it costs: your own OpenAI usage, one model call per changed page. Relve charges nothing.
+- Your key: it passes through Relve's server for that call only. Relve never stores or logs it, and it is never written to the output files.
+- Options: `--paths /a,/b` instead of `--sitemap`, `--model gpt-4o` (also `gpt-4.1`, `gpt-4.1-mini`; default `gpt-4o-mini`, the fastest), `--site-name "Your Brand"`, `--limit 20`, `--out <dir>`, `--force`.
+- The run ends with a summary (written, unchanged, rejected, errors, tokens). Rejected or failed pages are listed with a code (for example `too_little_text`) and nothing is saved for them; they keep the plain `convert` Markdown.
+- Re-run it when your content changes. Unchanged pages are skipped with no model call, so they are free. You can add the command to CI or your deploy (before the build, with `OPENAI_API_KEY` as a secret).
+
+Advanced: your own prompts. `--prompts <dir>` (or `AGENT_READY_PROMPTS`) runs your own prompt files on your machine instead of calling the Relve API. See the package README.
+
+2. Commit the folder at `.agent-ready/md` in your project root (or generate it in your deploy).
+3. Tell the package to read it. The store reads files with Node, so put it where only Node code runs.
+
+Next.js: in the route file (`app/%5F%5Fmd/[[...path]]/route.ts`), not in `agent-ready.config.ts`. The config file is also loaded by the middleware, which runs on the Edge runtime and has no file system.
+
+```ts
+import { createMarkdownRoute } from "@relvehq/agent-ready/next/route";
+import { fileStore } from "@relvehq/agent-ready/refine/file-store";
+import agentReady from "../../../agent-ready.config";
+
+export const { GET, HEAD } = createMarkdownRoute({ ...agentReady, store: fileStore(".agent-ready/md") });
+export const runtime = "nodejs";
+export const dynamic = "force-dynamic";
+```
+
+On Vercel, Amplify or Netlify, make sure the folder ships with the server code. In `next.config.js` (Next.js 15+; on 13.4 and 14 put it under `experimental`):
+
+```js
+outputFileTracingIncludes: { "/**": ["./.agent-ready/md/**/*"] },
+```
+
+Node server (Path 3): pass it straight to `createAgentReady`:
+
+```ts
+import { fileStore } from "@relvehq/agent-ready/refine/file-store";
+app.use(agentReadyNode(createAgentReady({ site: "https://your-site.com", store: fileStore(".agent-ready/md") })));
+```
+
+4. Redeploy. Check a refined page:
+
+```
+curl -i https://your-site.com/pricing.md
+```
+
+The response should carry `x-agent-ready-source: stored`. Pages without a refined file keep working as before (`convert`).
+
+WordPress (Path 5): no folder to commit. Use the plugin's OpenAI key setting or upload a zip (see "Refined files on WordPress"), then run the same `curl` on a refined post's `.md` URL. Pages without a refined file keep `x-agent-ready-source: native`.
+
+## Turn it off
+
+Set `AGENT_READY_ENABLED=0`, or remove the variable, and redeploy. Every request then goes through as if the package were not there. Removing it fully: delete the files `init` created and the two lines in your middleware.
+
+WordPress: untick **Enabled** on Settings > Agent Ready, or deactivate the plugin. Deleting the plugin also removes its settings and the uploaded refined files.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `Accept: text/markdown` returns HTML | The CDN served its cached HTML | Test the `.md` URL; add `Accept` to the CDN cache key |
+| | The package is off | Set `AGENT_READY_ENABLED=1` and redeploy |
+| | No site URL, so it stayed off | Set `NEXT_PUBLIC_SITE_URL` (look for the "no site URL" warning in your logs) |
+| | The page is too short to convert (under 200 characters of text) | An `Accept` request then gets the normal HTML on purpose (`x-agent-ready-source: html-fallback`) |
+| `.md` URL returns 404 | The path is excluded (`/api`, `/admin`, `/login`, `/account`, `/checkout`, `/dashboard` and similar, or your `exclude`) | Expected for private pages; otherwise check your `exclude` list |
+| | The page has under 200 characters of real text | Add content, or add a template or a refined file for it |
+| `.md` URL returns your site's 404 page (Next.js) | Your matcher does not include the `.md` entry | Add `"/((?!_next/).+\\.md)"` to the matcher (Path 2) |
+| `/llms.txt` is empty or lists no pages | No `/sitemap.xml`, or it lists URLs on another host | Publish a sitemap at `/sitemap.xml` with your public URLs |
+| Existing middleware stopped running, or Markdown never appears | The middleware is not wrapped | Export `withAgentReady(agentReady, yourMiddleware)` as the default (Path 2) |
+| Build passes, nothing happens | No site URL or `AGENT_READY_ENABLED` not set on the host | Set both and redeploy; env vars added after a build need a new build |
+| Node server: Markdown never appears | The middleware is mounted after static files or routes | Move `app.use(agentReadyNode(...))` above them |
+| WordPress: `Accept` request returns HTML, `?format=md` works | A page cache or the proxy in front answered first | Expected with page caches; for the proxy, see "If a Cloudflare Worker or a reverse proxy serves the blog" |
+| WordPress: `.md` URL returns your theme's 404 page | The proxy in front does not forward `.md` paths, or the plugin is not enabled | Test on the WordPress server's own address; fix the proxy rules; tick Enabled |
+| `x-agent-ready-source` is `convert`, not `stored` | The refined folder is missing on the server, or the store is not configured | Check the folder is committed and shipped, and the route file has `store: fileStore(...)` |
+
+## Getting help
+
+Contact the Relve team. Include the output of `npx @relvehq/agent-ready check https://your-site.com --json` and the response headers from one of the curl commands above.

package/dist/bin/agent-ready.d.ts

@@ -0,0 +1,23 @@
+#!/usr/bin/env node
+import { type InitFs } from "../cli/init";
+export declare function nodeFs(root: string): InitFs;
+/**
+ * The WordPress plugin folder: the copy built into the package
+ * (dist/wp/relve-agent-ready, next to dist/bin and dist/cli), else the source
+ * folder when running from the repo with tsx. null when neither is there.
+ */
+export declare function bundledPluginDir(here?: string): string | null;
+/** Every file under the plugin folder, relative with forward slashes; no tests/, no dotfiles. */
+export declare function listPluginFiles(dir: string): string[];
+export interface RunDeps {
+    log?: (s: string) => void;
+    env?: Record<string, string | undefined>;
+    fetch?: typeof fetch;
+}
+export declare function run(argv: string[], deps?: RunDeps | ((s: string) => void)): Promise<number>;
+/**
+ * Read `.env.local` then `.env` from the working directory, like Next.js, so
+ * the keys a project already has (OPENAI_API_KEY, ANTHROPIC_API_KEY) just
+ * work. A variable already set in the shell wins; values are never printed.
+ */
+export declare function loadEnvFiles(cwd?: string): string[];