@vercel/agent-readability 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vercel, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,227 @@
+# @vercel/agent-readability
+
+Detect AI agents. Serve them markdown. Audit your site against the
+[Agent Readability Spec](https://vercel.com/kb/guide/agent-readability-spec).
+
+## Install
+
+```bash
+npm install @vercel/agent-readability
+```
+
+Or run the audit CLI directly without installing:
+
+```bash
+npx @vercel/agent-readability audit https://vercel.com/docs
+```
+
+## Quick Start
+
+Add AI agent detection to your Next.js middleware in three lines:
+
+```ts
+// middleware.ts
+import { withAgentReadability } from '@vercel/agent-readability/next'
+
+export default withAgentReadability({
+  rewrite: (pathname) => `/api/docs-md${pathname}`,
+})
+
+export const config = {
+  matcher: ['/docs/:path*'],
+}
+```
+
+AI agents hitting `/docs/*` now receive markdown instead of HTML.
+
+## What It Does
+
+Three-layer detection identifies AI agents from HTTP request headers:
+
+1. **Known UA patterns.** 40+ agents including ClaudeBot, GPTBot, Cursor, and Perplexity.
+2. **Signature-Agent header.** Catches ChatGPT agent (RFC 9421).
+3. **sec-fetch-mode heuristic.** Catches unknown bots that lack browser fingerprints.
+
+The detection optimizes for recall over precision. Serving markdown to a non-AI bot
+is low cost. Missing an AI agent means a worse experience.
+
+## Core API
+
+### `isAIAgent(request)`
+
+Detect AI agents from request headers.
+
+```ts
+import { isAIAgent } from '@vercel/agent-readability'
+
+const result = isAIAgent(request)
+// { detected: true, method: 'ua-match' }
+```
+
+Accepts any object with a `headers.get()` method: `Request`, `NextRequest`,
+or a custom wrapper.
+
+Returns a discriminated union:
+- `{ detected: true, method: 'ua-match' | 'signature-agent' | 'heuristic' }`
+- `{ detected: false, method: null }`
+
+When `detected` is `true`, `method` is always non-null. TypeScript narrows automatically.
+
+### `acceptsMarkdown(request)`
+
+Check if the request prefers markdown via the `Accept` header.
+
+```ts
+import { acceptsMarkdown } from '@vercel/agent-readability'
+
+if (acceptsMarkdown(request)) {
+  return new Response(markdown, {
+    headers: { 'Content-Type': 'text/markdown', 'Vary': 'Accept' },
+  })
+}
+```
+
+### `shouldServeMarkdown(request)`
+
+Combines agent detection and content negotiation into one call.
+
+```ts
+import { shouldServeMarkdown } from '@vercel/agent-readability'
+
+const { serve, reason } = shouldServeMarkdown(request)
+// serve: true, reason: 'agent' | 'accept-header'
+```
+
+### `generateNotFoundMarkdown(path, options?)`
+
+Generates a markdown body for missing pages. Return this with a 200 status
+(not 404) because agents discard 404 response bodies.
+
+```ts
+import { generateNotFoundMarkdown } from '@vercel/agent-readability'
+
+const md = generateNotFoundMarkdown('/docs/missing', {
+  baseUrl: 'https://example.com',
+})
+// Return as 200 so agents read the body
+return new Response(md, {
+  headers: { 'Content-Type': 'text/markdown' },
+})
+```
+
+### Pattern Exports
+
+`AI_AGENT_UA_PATTERNS`, `TRADITIONAL_BOT_PATTERNS`, `SIGNATURE_AGENT_DOMAINS`,
+and `BOT_LIKE_REGEX` are exported for consumers who need to extend or inspect them.
+
+## Next.js Adapter
+
+### `withAgentReadability(options, handler?)`
+
+Middleware wrapper that detects AI agents and rewrites matching requests
+to markdown routes. Compatible with Next.js 14 and 15 (Pages and App Router).
+
+```ts
+import { withAgentReadability } from '@vercel/agent-readability/next'
+
+export default withAgentReadability({
+  docsPrefix: '/docs',
+  rewrite: (pathname) => `/en/llms.mdx/${pathname.replace('/docs/', '')}`,
+  onDetection: async ({ path, method }) => {
+    await trackMdRequest({ path, detectionMethod: method })
+  },
+})
+```
+
+The `onDetection` callback runs via `event.waitUntil()` and does not block
+the response.
+
+#### Composing with other middleware
+
+Pass your existing middleware as the second argument:
+
+```ts
+export default withAgentReadability(
+  {
+    rewrite: (p) => `/md${p}`,
+  },
+  (req, event) => i18nMiddleware(req, event),
+)
+```
+
+#### Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `docsPrefix` | `string` | `'/docs'` | URL prefix to intercept |
+| `rewrite` | `(pathname: string) => string` | required | Maps request path to markdown route |
+| `onDetection` | `(info) => void \| Promise<void>` | - | Analytics callback (runs in `waitUntil`) |
+
+#### `agentReadabilityMatcher`
+
+A pre-built matcher that excludes Next.js internals and static files.
+Use this for site-wide agent detection instead of scoping to a prefix:
+
+```ts
+import { withAgentReadability, agentReadabilityMatcher } from '@vercel/agent-readability/next'
+
+export default withAgentReadability({
+  docsPrefix: '/',
+  rewrite: (pathname) => `/md${pathname}`,
+})
+
+export const config = {
+  matcher: agentReadabilityMatcher,
+}
+```
+
+## Audit CLI
+
+Check your site against the Agent Readability Spec.
+
+```bash
+npx @vercel/agent-readability audit https://sdk.vercel.ai
+```
+
+Runs 16 weighted checks across three categories. Returns a score from 0 to 100.
+Failed checks include fix suggestions that can be copy-pasted into your coding agent.
+
+### CI Integration
+
+```yaml
+- name: Audit agent readability
+  run: npx @vercel/agent-readability audit ${{ env.SITE_URL }} --min-score 70 --json
+```
+
+Exits with code 1 if the score is below the threshold.
+
+### CLI Options
+
+| Flag | Description |
+|------|-------------|
+| `--json` | Output as JSON |
+| `--min-score <n>` | Exit with error if score < n |
+
+## Caching
+
+When the same URL serves HTML to browsers and markdown to AI agents,
+CDN caching must include the `Accept` header in the cache key.
+
+Set `Vary: Accept` on all markdown responses. Without it, CDNs may serve
+cached HTML to agents or cached markdown to browsers.
+
+The `withAgentReadability` middleware handles detection and rewriting.
+Your markdown route handler is responsible for setting response headers
+including `Vary: Accept` and `Content-Type: text/markdown`.
+
+## Edge Runtime
+
+The core library and Next.js adapter use only Web APIs. They work in
+Vercel Edge Runtime, Cloudflare Workers, and any standard `Request`/`Response`
+environment.
+
+The CLI runs on Node.js 20+.
+
+## License
+
+MIT