@waniwani/relay 0.3.0 → 0.4.0

package/README.md

@@ -17,14 +17,17 @@ The [setup screen](https://app.waniwani.ai/agent-traffic/setup) in Waniwani give
 ### `isAgent(request)`
 
 ```ts
-function isAgent(request: Request | CloudFrontRequest): boolean;
+function isAgent(
+  request: Request | CloudFrontRequest,
+  options?: Pick<RelayOptions, "region" | "apiUrl" | "waitUntil" | "rules">,
+): boolean;
 ```
 
-Tells whether the request comes from an AI agent. It reads the headers and the client IP only. It makes no network call, keeps no state between requests and never throws. It returns `true` when any of these hold:
+Tells whether the request comes from an AI agent. It reads the headers and the client IP only, never waits on the network and never throws. It returns `true` when any of these hold:
 
-- The `User-Agent` header contains a token from the AI bot list, for example `ChatGPT-User`, `Claude-User`, `GPTBot` or `PerplexityBot`. The list comes from the AI bots in the Waniwani bot catalog: crawlers that collect training data, AI search bots and fetchers that act for a user. Classic search indexers such as `Googlebot` and `Bingbot`, browsers, `curl` and uptime monitors are not on it.
+- The `User-Agent` header contains a token from the AI bot list, for example `ChatGPT-User`, `Claude-User`, `GPTBot` or `PerplexityBot`. The list holds the AI bots in the Waniwani bot catalog and the AI bots on [bots.fyi](https://bots.fyi): crawlers that collect training data, AI search bots, fetchers that act for a user and browsing agents. Classic search indexers such as `Googlebot` and `Bingbot`, browsers, `curl` and uptime monitors are not on it.
 - The request has a `Signature-Agent` header ([Web Bot Auth](https://datatracker.ietf.org/wg/webbotauth/about/)).
-- The client IP sits inside an IP range that an AI vendor publishes for its bots. Today the ranges come from OpenAI, Perplexity, Mistral, Common Crawl and DuckDuckGo (DuckAssistBot).
+- The client IP sits inside an IP range that an AI vendor publishes for its bots, for example the lists of OpenAI, Anthropic, Perplexity, Mistral, Apple, Common Crawl and DuckDuckGo (DuckAssistBot).
 - The request has a `DPoP` header, or its `Authorization` header uses the `DPoP` scheme.
 
 ### `recordAgentVisit(request, options?)`
@@ -46,6 +49,7 @@ Posts one event for the request to Waniwani after the response ([`AgentVisitReco
 | `apiUrl` | The host of `region` | Another Waniwani host, for local development |
 | `key` | `process.env.WANIWANI_AGENT_TRAFFIC_KEY` | The key from the setup screen, `wwt_<token>.<org key>` |
 | `waitUntil` | Found by itself on Vercel, none on other platforms | Keeps the post alive after the response |
+| `rules` | `"remote"` | `"remote"` loads the newest bot list from Waniwani every 12 hours, `"bundled"` uses only the list that ships in the package |
 | `omit` | `[]` | Header names, and the fields `matchKey`, `userAgent` and `referer`, to leave out of every event. An event without `matchKey` still counts, but Waniwani then joins it with the chat embed's events on site, user agent and path only, which is less exact. |
 
 #### `AgentVisitRecord`
@@ -84,6 +88,46 @@ The `headers` field can hold only these request headers:
 | AWS CloudFront (Lambda@Edge) | Yes, before the handler returns. Lambda freezes the function when the handler returns, so a post event request that is not awaited is lost. | 1 s, no retry |
 | Other | No, when you pass `waitUntil` | 2 s, one retry after 500 ms |
 
+### `serveAgentPage(request, options?)`
+
+```ts
+function serveAgentPage(
+  request: Request,
+  options?: Pick<RelayOptions, "region" | "apiUrl" | "key">,
+): Promise<Response | null>;
+```
+
+Serves the agent page of a request: a text-first copy of the page that Waniwani renders and stores. It returns that page when all of these hold, and `null` otherwise, so your middleware passes the request on to your site:
+
+- A key is set.
+- The method is `GET` or `HEAD`.
+- Agent pages are on in Waniwani for the hostname of the request, and its path is in the list of agent page paths. An entry is an exact path such as `/pricing`, or a folder such as `/products/*`, which covers `/products` and every path under it. `*` covers every path.
+- The `User-Agent` belongs to an AI agent that acts for a user, for example `ChatGPT-User` or `Claude-User`. Crawlers that collect training data get your site as usual.
+- Waniwani sends the stored page in time. The rules and the deadlines come from your Waniwani settings.
+
+The response carries the stored content type, `x-waniwani-agent-page: hit`, `cache-control: private, no-store` and `vary: user-agent, accept`. A `HEAD` request gets the same headers with no body. The settings load on the first call, with a 300 ms limit, and stay in memory for 60 seconds. After that the relay keeps serving the cached settings while it loads new ones in the background. The function never throws, and the promise never rejects. It takes a `Request` only, so it works on Vercel and Cloudflare Workers.
+
+On Vercel, in `middleware.ts`:
+
+```ts
+import { isAgent, recordAgentVisit, serveAgentPage } from "@waniwani/relay";
+import { NextResponse, type NextRequest } from "next/server";
+
+export async function middleware(request: NextRequest) {
+  if (isAgent(request)) {
+    recordAgentVisit(request);
+    return (await serveAgentPage(request)) ?? NextResponse.next();
+  }
+  return NextResponse.next();
+}
+```
+
+In a Cloudflare Worker, pass the key and fall back to your origin:
+
+```ts
+return (await serveAgentPage(request, { key: env.WANIWANI_AGENT_TRAFFIC_KEY })) ?? fetch(request);
+```
+
 ## Dependencies
 
 The package has no runtime dependencies. The IP range check uses `BlockList` from `node:net`.
@@ -94,10 +138,10 @@ The package has no runtime dependencies. The IP range check uses `BlockList` fro
 bun install
 bun run build   # typecheck, lint and dist/index.js
 bun run test    # unit tests (run the build first, one test imports dist)
-bun run rules   # refreshes src/logic/ip-ranges.generated.json from the vendors' published lists
+bun run rules   # writes the two src/logic/*.generated.json files from the app rules file
 ```
 
-`src/logic/agents.json` is copied by hand from the Waniwani bot catalog (`src/lib/agent-traffic/bot-catalog.ts` in the Waniwani app): every entry of kind `training`, `search` or `fetcher`, using its `token` when it has one and its lowercased name otherwise.
+`bun run rules` reads `https://app.waniwani.ai/api/v2/agent-traffic/rules`, or the URL in `RULES_URL`, and writes `src/logic/agents.generated.json` and `src/logic/ip-ranges.generated.json`, the fallback list that ships in the package. Never edit them by hand. `npm version` runs it before every release and commits the two files with the new version. When the rules route does not answer, the release stops before the version changes.
 
 ## License
 

package/dist/index.d.ts

@@ -20,6 +20,11 @@ declare const settings: {
     requestTimeoutMs: number;
     cloudFrontTimeoutMs: number;
     retryDelayMs: number;
+    rulesRefreshMs: number;
+    rulesRetryMs: number;
+    siteRulesTimeoutMs: number;
+    siteRulesRefreshMs: number;
+    siteRulesRetryMs: number;
 };
 
 type Region = keyof typeof settings.apiUrls;
@@ -34,7 +39,11 @@ type RelayOptions = {
     waitUntil?: (promise: Promise<unknown>) => void;
     /** Header names, and the fields `matchKey`, `userAgent` or `referer`, to leave out of every event. */
     omit?: Array<string>;
+    /** `"remote"` loads the newest AI bot list from WaniWani every 12 hours. `"bundled"` keeps the list shipped in this version. Defaults to `"remote"`. */
+    rules?: "remote" | "bundled";
 };
+type IsAgentOptions = Pick<RelayOptions, "region" | "apiUrl" | "waitUntil" | "rules">;
+type ServeAgentPageOptions = Pick<RelayOptions, "region" | "apiUrl" | "key">;
 type AgentVisitRecord = {
     occurredAt: string;
     method: string;
@@ -48,14 +57,22 @@ type AgentVisitRecord = {
     relayVersion: string;
     platform?: Platform;
 };
-/** Tells whether a request, or the `cf.request` of a Lambda@Edge event, comes from an AI agent. Reads headers only, makes no network call. */
-declare function isAgent(request: Request | CloudFrontRequest): boolean;
+/**
+ * Tells whether a request, or the `cf.request` of a Lambda@Edge event, comes from an AI agent. Reads headers only and never waits on the network.
+ * Every 12 hours it starts one background fetch of the newest AI bot list, unless `rules` is `"bundled"`.
+ */
+declare function isAgent(request: Request | CloudFrontRequest, options?: IsAgentOptions): boolean;
 /**
  * Posts one event for this request to WaniWani after the response.
  * Sends nothing when no key is set. Never throws, never rejects and never touches the request.
  * The promise settles when the post is done, so Lambda@Edge, which has no `waitUntil`, awaits it.
  */
 declare function recordAgentVisit(request: Request | CloudFrontRequest, options?: RelayOptions): Promise<void>;
+/**
+ * The stored agent page for this request, when its site lists the path and the user agent acts for a user. Otherwise null, and the caller serves the origin.
+ * Never throws and never rejects.
+ */
+declare function serveAgentPage(request: Request, options?: ServeAgentPageOptions): Promise<Response | null>;
 
-export { isAgent, recordAgentVisit };
-export type { AgentVisitRecord, CloudFrontRequest, Region, RelayOptions };
+export { isAgent, recordAgentVisit, serveAgentPage };
+export type { AgentVisitRecord, CloudFrontRequest, IsAgentOptions, Region, RelayOptions, ServeAgentPageOptions };