@traceten/ai-crawl 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/).
+
+## [0.1.0] - Unreleased
+
+Initial public release.
+
+### Added
+
+- Local pre-filter and crawler-token matcher, zero runtime dependencies.
+- Adapters: Next.js, Cloudflare Workers, Cloudflare Pages Functions, Express, Hono.
+- Crawler IP resolution and reporting to `POST /v1/ai-crawls`.
+- Optional `onError` callback, reporting non-2xx and network failures without breaking the never-throws contract.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Traceten
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,260 @@
+# @traceten/ai-crawl
+
+[![CI](https://github.com/traceten/ai-crawl/actions/workflows/ci.yml/badge.svg)](https://github.com/traceten/ai-crawl/actions/workflows/ci.yml)
+[![npm version](https://img.shields.io/npm/v/@traceten/ai-crawl.svg)](https://www.npmjs.com/package/@traceten/ai-crawl)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
+[![Node](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)
+
+Server-side AI crawler tracking for Traceten. See the GPTBot, ClaudeBot and PerplexityBot crawls that browser JavaScript can never see.
+
+AI crawlers do not execute JavaScript. They issue one HTTP GET for the HTML and leave, so the Traceten snippet is structurally blind to them. The only possible observation point is your server, edge function, or worker. This package is that observation point.
+
+## What it does
+
+1. Pre-filters locally: only GET/HEAD, skips assets and API routes, skips browser subresource fetches. Non-crawler traffic costs one string scan and no network call.
+2. Matches the user agent against a local list of known AI crawler tokens, anchored at token boundaries (a UA containing `not-really-GPTBot` does not match).
+3. Reports plausible AI crawls to `POST https://ingest.traceten.com/v1/ai-crawls`, authenticated with your crawl token. Delivery is scheduled after the response (via `waitUntil` where the runtime has one) and never delays or breaks a request.
+
+Provider, category (answer fetch, search index, training, other), verification and confidence are decided by Traceten's servers against the full crawler registry, so new crawlers are covered without upgrading this package.
+
+## Contract
+
+- Never blocks the response. Never throws per-request. Failure is silent.
+- Zero runtime dependencies.
+- 1500 ms delivery timeout, `keepalive: true`.
+- Requires Node 18+ or an edge runtime with `fetch`.
+- Each adapter bundle is under 8 KB gzipped (CI-enforced).
+
+## Install
+
+```bash
+npm install @traceten/ai-crawl
+```
+
+## Configuration
+
+```ts
+import { defineAiCrawlConfig } from "@traceten/ai-crawl";
+
+const config = defineAiCrawlConfig({
+  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12", // your site key (the ttid_… data-site value on the install page; case-sensitive)
+  authToken: process.env.TRACETEN_CRAWL_TOKEN!, // tt_bot_..., created in the dashboard. REQUIRED.
+});
+```
+
+`defineAiCrawlConfig` throws on invalid input. Call it at module scope or server boot so a missing token fails your deploy, not your visitors' requests. The endpoint rejects unauthenticated reports, so a missing token would otherwise mean every report silently returns 401.
+
+The `authToken` is a server-side secret. Do not put it in browser code, and do not reuse the public snippet key in its place.
+
+### Options
+
+| Option                    | Default                                    | Meaning                                                                                                                                                                                                                                                   |
+| ------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `endpoint`                | `https://ingest.traceten.com/v1/ai-crawls` | Ingestion endpoint                                                                                                                                                                                                                                        |
+| `allowedMethods`          | `["GET", "HEAD"]`                          | Methods considered at all                                                                                                                                                                                                                                 |
+| `extraDenyPathPrefixes`   | `[]`                                       | Added to the built-in deny-list (`/api`, `/_next`, `/static`, ...). Extends, never replaces. Prefixes match whole path segments: `/api` denies `/api` and `/api/users` but not `/apidocs`                                                                 |
+| `extraDenyExtensions`     | `[]`                                       | Added to the built-in static-extension deny-list. Extends, never replaces                                                                                                                                                                                 |
+| `disableAnswerFetch`      | `false`                                    | Skip user-triggered fetchers (ChatGPT-User, Claude-User, ...)                                                                                                                                                                                             |
+| `disableSearchCrawlers`   | `false`                                    | Skip AI search index crawlers (OAI-SearchBot, PerplexityBot, ...)                                                                                                                                                                                         |
+| `disableTrainingCrawlers` | `false`                                    | Skip training crawlers (GPTBot, ClaudeBot, CCBot, ...)                                                                                                                                                                                                    |
+| `disableOtherCrawlers`    | `false`                                    | Skip uncategorised AI bots and unrecognised agents from known vendors                                                                                                                                                                                     |
+| `trustProxy`              | `false`                                    | Trust `x-forwarded-for` for IP resolution. Off by default because the header is spoofable                                                                                                                                                                 |
+| `proxyDepth`              | `1`                                        | Number of trusted proxies appending to `x-forwarded-for` (used only with `trustProxy`)                                                                                                                                                                    |
+| `trustCfConnectingIp`     | `false`                                    | Trust the `cf-connecting-ip` header. The Cloudflare Workers and Pages adapters enable this automatically; set it manually only when your origin sits behind Cloudflare. Off by default because any client can forge the header on a non-Cloudflare origin |
+| `publicOrigin`            | none                                       | Rebuild reported URLs on this origin, for containers behind reverse proxies that see internal hosts                                                                                                                                                       |
+| `onError`                 | none                                       | Called when a report does not land: a non-2xx, or a network/timeout failure. Off by default; delivery stays silent either way                                                                                                                             |
+
+### Knowing when reports are not landing
+
+Delivery is silent by design: a broken analytics call must never surface on your
+site, so `sendReport` never throws and never retries. The cost is that a
+rejected report looks exactly like a delivered one. A wrong `authToken`, or
+egress your platform blocks, both read as "no AI crawlers visited" indefinitely.
+
+`onError` closes that gap without changing the safety posture:
+
+```ts
+export const aiCrawl = defineAiCrawlConfig({
+  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
+  authToken: process.env.TRACETEN_BOT_TOKEN!,
+  onError: (e) => {
+    // e.kind is "http" (with e.status) or "network" (with e.cause)
+    console.warn("[traceten] crawl report failed", e.kind, e.status ?? e.cause);
+  },
+});
+```
+
+Your callback runs inside a `try/catch`, so throwing from it cannot break the
+host response. A `401` or `403` means the token or the origin; repeated
+`network` failures mean egress. Worth wiring in staging at minimum: it is the
+difference between "no crawlers came" and "nothing we sent was accepted".
+
+Crawler-facing paths (`/robots.txt`, `/llms.txt`, `/llms-full.txt`, `*sitemap*.xml`) are always reported even though `.txt` and `.xml` are otherwise denied. A GPTBot hit on `/llms.txt` is one of the highest-signal events this product can capture.
+
+## Adapters
+
+### Next.js (proxy / middleware)
+
+```ts
+// proxy.ts (Next 15.5+; middleware.ts on earlier versions)
+import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
+import { defineAiCrawlConfig } from "@traceten/ai-crawl";
+import { trackAICrawlerRequest } from "@traceten/ai-crawl/next";
+
+const config = defineAiCrawlConfig({
+  siteId: process.env.TRACETEN_SITE_ID!,
+  authToken: process.env.TRACETEN_CRAWL_TOKEN!,
+});
+
+export function proxy(request: NextRequest, event: NextFetchEvent) {
+  trackAICrawlerRequest(request, event, config);
+  return NextResponse.next();
+}
+```
+
+### Cloudflare Workers
+
+For a site already behind Cloudflare this needs no application deploy at all.
+
+```ts
+import { defineAiCrawlConfig } from "@traceten/ai-crawl";
+import { withAICrawlerTracking } from "@traceten/ai-crawl/cloudflare-workers";
+
+const config = defineAiCrawlConfig({
+  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
+  authToken: "tt_bot_...", // prefer an env binding in production
+});
+
+export default {
+  fetch: withAICrawlerTracking(async (request, env, ctx) => {
+    return fetch(request); // your existing origin logic
+  }, config),
+};
+```
+
+The wrapper captures the real response status and reports through `ctx.waitUntil`. Errors from your handler propagate untouched.
+
+### Cloudflare Pages Functions
+
+```ts
+// functions/_middleware.ts
+import { createAICrawlerPagesMiddleware } from "@traceten/ai-crawl/cloudflare-pages";
+
+export const onRequest = createAICrawlerPagesMiddleware({
+  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
+  authToken: "tt_bot_...",
+});
+```
+
+### Express
+
+```ts
+import express from "express";
+import { createTracetenAICrawlerMiddleware } from "@traceten/ai-crawl/express";
+
+const app = express();
+app.use(
+  createTracetenAICrawlerMiddleware({
+    siteId: process.env.TRACETEN_SITE_ID!,
+    authToken: process.env.TRACETEN_CRAWL_TOKEN!,
+  }),
+);
+```
+
+Calls `next()` immediately and reports on the response `finish` event, so the real status code is captured and no request is ever delayed.
+
+Set `publicOrigin` here. Without it the reported URL is built from the request's `Host` header (which a client can set to anything) with an assumed `https` scheme. With `publicOrigin` the reported URL is always on your real origin:
+
+```ts
+createTracetenAICrawlerMiddleware({
+  siteId: process.env.TRACETEN_SITE_ID!,
+  authToken: process.env.TRACETEN_CRAWL_TOKEN!,
+  publicOrigin: "https://www.example.com",
+});
+```
+
+### Hono / generic Request+Response
+
+```ts
+import { Hono } from "hono";
+import { aiCrawlerTracking } from "@traceten/ai-crawl/hono";
+
+const app = new Hono();
+app.use("*", aiCrawlerTracking({ siteId: "...", authToken: "tt_bot_..." }));
+```
+
+Any other framework that exposes fetch-API objects can use `trackAICrawlerFetch(request, response, config, waitUntil?)` from the same entry point.
+
+## What is sent, including the crawler IP
+
+For each matched crawl the package sends exactly:
+
+```json
+{
+  "site_id": "ttid_7Rb4TrC1dTbnD8w3s1TS12",
+  "url": "https://example.com/docs/pricing",
+  "method": "GET",
+  "status": 200,
+  "user_agent": "Mozilla/5.0 ... compatible; GPTBot/1.1; +https://openai.com/gptbot",
+  "ip": "20.15.240.64",
+  "ts": 1765432100000
+}
+```
+
+The `ip` field is the crawler's IP address, resolved from `cf-connecting-ip` (only where trusted, see `trustCfConnectingIp`), then `x-forwarded-for` (only when `trustProxy` is set), then the socket address. The field is omitted when no trustworthy value can be derived.
+
+Sending it is deliberate and required. The crawler's TCP connection terminates at your server, so its IP is observable only there. Traceten uses it to verify the crawl against network evidence (for example OpenAI publishes GPTBot's IP ranges), which is what separates a real GPTBot crawl from a spoofed user agent. Verification runs at Traceten's edge while the value is in scope, and the reported IP is always treated as a claim to be verified, never as proof by itself.
+
+Retention depends on the verdict. Traceten keeps the raw address in exactly two cases: it fell inside a range the vendor publishes for that crawler, or its reverse DNS forward-confirmed to the vendor's own domain. In every other case, including an address that only matches the provider's wider network, the address is discarded at the edge and only an HMAC-SHA-256 hash under a per-site derived key is stored. Those rows are hash-only: Traceten does not resolve or store the network operator (ASN) or the country for a crawler IP, and writes neither field. Verification that needs a reverse-DNS lookup sends the address to Cloudflare's public DNS resolver before anything about it is known. This package never sends visitor analytics, cookies, or request bodies.
+
+No batching in v1: each matched crawl is one report. Crawl volume on typical sites is orders of magnitude below pageview volume because of the pre-filter.
+
+## Privacy and GDPR
+
+- Traceten processes crawl reports on your behalf, as your processor under the Traceten Data Processing Agreement.
+- Your privacy notice should reflect this transfer: crawler request metadata (URL, user agent, IP address) is shared with Traceten for bot and crawler detection.
+- The typical legal basis is legitimate interest in detecting and attributing automated crawler traffic. Confirm this with your own counsel for your jurisdiction.
+- Hashed unverified IPs are pseudonymized personal data. Crawl records are records of automated software requests rather than of a visitor, and they carry no visitor or session identifier, so Traceten's per-visitor access and deletion API cannot reach them — it has no key to look one up by. That is a limit of the API, not a claim that the records can never be matched: the hash is deterministic per site, so where a data subject supplies the address itself, Traceten can find and erase their rows. Those requests are handled manually — email privacy@traceten.com with the address, the site and a date range. (This is the GDPR Article 11(2) situation: no obligation to acquire extra data to identify someone, full rights where the data subject supplies what makes identification possible.) What otherwise bounds these records: the 90-day limit below, hash-only storage for anything unverified, and whole-account erasure, which removes every crawl record and rollup for every site on the account. See [the crawl privacy notes](https://docs.traceten.com/privacy/data-collected#ai-crawler-ip-addresses).
+- Individual crawl records are deleted after 90 days on every plan. The daily rollups derived from them are counts grouped by crawler and by page path, with no address, no hash and no personal identifier. Page paths therefore persist in aggregate after the records containing them expire.
+
+## Development
+
+```bash
+npm install
+npm test        # vitest
+npm run typecheck
+npm run build    # tsc + size-gate bundles + 8KB check
+```
+
+The local token list lives in `src/crawlers.ts`. It's a deliberately-loose cost filter behind Traceten's server-side crawler registry, which is the actual source of truth for provider, category, verification and confidence — this list only decides "worth reporting locally." `Google-Extended` and `Applebot-Extended` are excluded on purpose: they are robots.txt control tokens and never appear as live user agents.
+
+## Versioning
+
+This package follows [Semantic Versioning](https://semver.org/). Before `1.0.0`,
+minor versions may include breaking changes — pin an exact version in production
+until then. See [CHANGELOG.md](./CHANGELOG.md) for release history.
+
+## Contributing
+
+Issues and pull requests are welcome. For anything beyond a small fix, please
+open an issue first to discuss the change. Run the checks below before
+submitting a PR — CI enforces the same steps on every pull request:
+
+```bash
+npm install
+npm run typecheck
+npm run build
+npm test
+```
+
+## License
+
+MIT © Traceten — see [LICENSE](./LICENSE).
+
+## Links
+
+- [Documentation](https://docs.traceten.com/install/ai-crawler-tracking)
+- [Traceten](https://traceten.com) — AI traffic attribution for the AI search era
+- [Issues](https://github.com/traceten/ai-crawl/issues)
+- [Changelog](./CHANGELOG.md)
+- [Node SDK](https://github.com/traceten/sdk-node) · [Python SDK](https://github.com/traceten/sdk-python) · [Go SDK](https://github.com/traceten/traceten-go)

package/dist/adapters/cloudflare-pages.d.ts

@@ -0,0 +1,42 @@
+/**
+ * Cloudflare Pages Functions adapter — drop into `functions/_middleware.ts`.
+ *
+ * ```ts
+ * // functions/_middleware.ts
+ * import { defineAiCrawlConfig } from "@traceten/ai-crawl";
+ * import { createAICrawlerPagesMiddleware } from "@traceten/ai-crawl/cloudflare-pages";
+ *
+ * export const onRequest = createAICrawlerPagesMiddleware(
+ *   defineAiCrawlConfig({
+ *     siteId: "…",           // or read from context.env inside your own wrapper
+ *     authToken: "tt_bot_…",
+ *   }),
+ * );
+ * ```
+ *
+ * Construction throws on invalid config (deploy-time failure); the returned
+ * middleware never throws and never delays the response — delivery goes
+ * through `context.waitUntil` and observes the REAL response status.
+ */
+import { type FetchLikeRequest } from "../track.js";
+import type { AiCrawlConfig, ResolvedAiCrawlConfig } from "../types.js";
+/** Structural subset of Pages Functions' `EventContext`. */
+export interface PagesContext {
+    request: FetchLikeRequest;
+    next(): Promise<{
+        status: number;
+    }> | {
+        status: number;
+    };
+    waitUntil(promise: Promise<unknown>): void;
+}
+type PagesMiddleware = (context: PagesContext) => Promise<{
+    status: number;
+}>;
+/**
+ * Build a Pages Functions middleware. THROWS at construction when the config
+ * is invalid — that is deploy time, exactly where a bad token should fail.
+ */
+export declare function createAICrawlerPagesMiddleware(config: AiCrawlConfig | ResolvedAiCrawlConfig): PagesMiddleware;
+export {};
+//# sourceMappingURL=cloudflare-pages.d.ts.map

package/dist/adapters/cloudflare-pages.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cloudflare-pages.d.ts","sourceRoot":"","sources":["../../src/adapters/cloudflare-pages.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAGH,OAAO,EAAqC,KAAK,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACvF,OAAO,KAAK,EAAE,aAAa,EAAE,qBAAqB,EAAE,MAAM,aAAa,CAAC;AAExE,4DAA4D;AAC5D,MAAM,WAAW,YAAY;IAC3B,OAAO,EAAE,gBAAgB,CAAC;IAC1B,IAAI,IAAI,OAAO,CAAC;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,GAAG;QAAE,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IACzD,SAAS,CAAC,OAAO,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,IAAI,CAAC;CAC5C;AAED,KAAK,eAAe,GAAG,CAAC,OAAO,EAAE,YAAY,KAAK,OAAO,CAAC;IAAE,MAAM,EAAE,MAAM,CAAA;CAAE,CAAC,CAAC;AAE9E;;;GAGG;AACH,wBAAgB,8BAA8B,CAC5C,MAAM,EAAE,aAAa,GAAG,qBAAqB,GAC5C,eAAe,CAuBjB"}

package/dist/adapters/cloudflare-pages.js

@@ -0,0 +1,46 @@
+/**
+ * Cloudflare Pages Functions adapter — drop into `functions/_middleware.ts`.
+ *
+ * ```ts
+ * // functions/_middleware.ts
+ * import { defineAiCrawlConfig } from "@traceten/ai-crawl";
+ * import { createAICrawlerPagesMiddleware } from "@traceten/ai-crawl/cloudflare-pages";
+ *
+ * export const onRequest = createAICrawlerPagesMiddleware(
+ *   defineAiCrawlConfig({
+ *     siteId: "…",           // or read from context.env inside your own wrapper
+ *     authToken: "tt_bot_…",
+ *   }),
+ * );
+ * ```
+ *
+ * Construction throws on invalid config (deploy-time failure); the returned
+ * middleware never throws and never delays the response — delivery goes
+ * through `context.waitUntil` and observes the REAL response status.
+ */
+import { defineAiCrawlConfig, isResolvedConfig } from "../config.js";
+import { factsFromFetchRequest, trackFacts } from "../track.js";
+/**
+ * Build a Pages Functions middleware. THROWS at construction when the config
+ * is invalid — that is deploy time, exactly where a bad token should fail.
+ */
+export function createAICrawlerPagesMiddleware(config) {
+    const resolved = isResolvedConfig(config) ? config : defineAiCrawlConfig(config);
+    // On Pages the platform strips and rewrites cf-connecting-ip, so it is
+    // trustworthy here (and only here / Workers) by default.
+    const cfg = { ...resolved, trustCfConnectingIp: true };
+    return async (context) => {
+        // The customer's response comes first and is never touched. If next()
+        // itself throws, that error propagates untouched — swallowing it would
+        // change the site's behaviour.
+        const response = await context.next();
+        try {
+            trackFacts(cfg, factsFromFetchRequest(context.request), typeof response?.status === "number" ? response.status : undefined, (p) => context.waitUntil(p));
+        }
+        catch {
+            /* silent by contract */
+        }
+        return response;
+    };
+}
+//# sourceMappingURL=cloudflare-pages.js.map

package/dist/adapters/cloudflare-pages.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cloudflare-pages.js","sourceRoot":"","sources":["../../src/adapters/cloudflare-pages.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAEH,OAAO,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAC;AACrE,OAAO,EAAE,qBAAqB,EAAE,UAAU,EAAyB,MAAM,aAAa,CAAC;AAYvF;;;GAGG;AACH,MAAM,UAAU,8BAA8B,CAC5C,MAA6C;IAE7C,MAAM,QAAQ,GAAG,gBAAgB,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,mBAAmB,CAAC,MAAM,CAAC,CAAC;IACjF,uEAAuE;IACvE,yDAAyD;IACzD,MAAM,GAAG,GAA0B,EAAE,GAAG,QAAQ,EAAE,mBAAmB,EAAE,IAAI,EAAE,CAAC;IAE9E,OAAO,KAAK,EAAE,OAAqB,EAAE,EAAE;QACrC,sEAAsE;QACtE,uEAAuE;QACvE,+BAA+B;QAC/B,MAAM,QAAQ,GAAG,MAAM,OAAO,CAAC,IAAI,EAAE,CAAC;QACtC,IAAI,CAAC;YACH,UAAU,CACR,GAAG,EACH,qBAAqB,CAAC,OAAO,CAAC,OAAO,CAAC,EACtC,OAAO,QAAQ,EAAE,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,SAAS,EAClE,CAAC,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,CAAC,CAC5B,CAAC;QACJ,CAAC;QAAC,MAAM,CAAC;YACP,wBAAwB;QAC1B,CAAC;QACD,OAAO,QAAQ,CAAC;IAClB,CAAC,CAAC;AACJ,CAAC"}

package/dist/adapters/cloudflare-workers.d.ts

@@ -0,0 +1,41 @@
+/**
+ * Cloudflare Workers adapter — wrap your fetch handler.
+ *
+ * For a customer already behind Cloudflare this is copy-paste with no
+ * application deploy, which is why this adapter matters disproportionately.
+ *
+ * ```ts
+ * import { defineAiCrawlConfig } from "@traceten/ai-crawl";
+ * import { withAICrawlerTracking } from "@traceten/ai-crawl/cloudflare-workers";
+ *
+ * const config = defineAiCrawlConfig({ siteId: "…", authToken: "tt_bot_…" });
+ *
+ * export default {
+ *   fetch: withAICrawlerTracking(async (request, env, ctx) => {
+ *     return await handle(request);
+ *   }, config),
+ * };
+ * ```
+ *
+ * Construction throws on invalid config; the wrapper never throws its own
+ * errors, captures the REAL status code from the handler's response, and
+ * schedules delivery via `ctx.waitUntil`. Handler errors propagate untouched.
+ */
+import { type FetchLikeRequest } from "../track.js";
+import type { AiCrawlConfig, ResolvedAiCrawlConfig } from "../types.js";
+/** Structural subset of the Workers `ExecutionContext`. */
+export interface WorkersExecutionContext {
+    waitUntil(promise: Promise<unknown>): void;
+}
+type WorkersHandler<Env, Res extends {
+    status: number;
+}> = (request: FetchLikeRequest, env: Env, ctx: WorkersExecutionContext) => Promise<Res> | Res;
+/**
+ * Wrap a Workers fetch handler with AI crawler tracking. THROWS at
+ * construction (module scope — deploy time) when the config is invalid.
+ */
+export declare function withAICrawlerTracking<Env, Res extends {
+    status: number;
+}>(handler: WorkersHandler<Env, Res>, config: AiCrawlConfig | ResolvedAiCrawlConfig): WorkersHandler<Env, Res>;
+export {};
+//# sourceMappingURL=cloudflare-workers.d.ts.map

package/dist/adapters/cloudflare-workers.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cloudflare-workers.d.ts","sourceRoot":"","sources":["../../src/adapters/cloudflare-workers.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAGH,OAAO,EAAqC,KAAK,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACvF,OAAO,KAAK,EAAE,aAAa,EAAE,qBAAqB,EAAE,MAAM,aAAa,CAAC;AAExE,2DAA2D;AAC3D,MAAM,WAAW,uBAAuB;IACtC,SAAS,CAAC,OAAO,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,IAAI,CAAC;CAC5C;AAED,KAAK,cAAc,CAAC,GAAG,EAAE,GAAG,SAAS;IAAE,MAAM,EAAE,MAAM,CAAA;CAAE,IAAI,CACzD,OAAO,EAAE,gBAAgB,EACzB,GAAG,EAAE,GAAG,EACR,GAAG,EAAE,uBAAuB,KACzB,OAAO,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC;AAExB;;;GAGG;AACH,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,GAAG,SAAS;IAAE,MAAM,EAAE,MAAM,CAAA;CAAE,EACvE,OAAO,EAAE,cAAc,CAAC,GAAG,EAAE,GAAG,CAAC,EACjC,MAAM,EAAE,aAAa,GAAG,qBAAqB,GAC5C,cAAc,CAAC,GAAG,EAAE,GAAG,CAAC,CAuB1B"}

package/dist/adapters/cloudflare-workers.js

@@ -0,0 +1,49 @@
+/**
+ * Cloudflare Workers adapter — wrap your fetch handler.
+ *
+ * For a customer already behind Cloudflare this is copy-paste with no
+ * application deploy, which is why this adapter matters disproportionately.
+ *
+ * ```ts
+ * import { defineAiCrawlConfig } from "@traceten/ai-crawl";
+ * import { withAICrawlerTracking } from "@traceten/ai-crawl/cloudflare-workers";
+ *
+ * const config = defineAiCrawlConfig({ siteId: "…", authToken: "tt_bot_…" });
+ *
+ * export default {
+ *   fetch: withAICrawlerTracking(async (request, env, ctx) => {
+ *     return await handle(request);
+ *   }, config),
+ * };
+ * ```
+ *
+ * Construction throws on invalid config; the wrapper never throws its own
+ * errors, captures the REAL status code from the handler's response, and
+ * schedules delivery via `ctx.waitUntil`. Handler errors propagate untouched.
+ */
+import { defineAiCrawlConfig, isResolvedConfig } from "../config.js";
+import { factsFromFetchRequest, trackFacts } from "../track.js";
+/**
+ * Wrap a Workers fetch handler with AI crawler tracking. THROWS at
+ * construction (module scope — deploy time) when the config is invalid.
+ */
+export function withAICrawlerTracking(handler, config) {
+    const resolved = isResolvedConfig(config) ? config : defineAiCrawlConfig(config);
+    // On Workers the platform strips and rewrites cf-connecting-ip, so it is
+    // trustworthy here (and only here / Pages) by default.
+    const cfg = { ...resolved, trustCfConnectingIp: true };
+    return async (request, env, ctx) => {
+        // Handler errors propagate untouched — tracking must not change behaviour.
+        const response = await handler(request, env, ctx);
+        try {
+            trackFacts(cfg, factsFromFetchRequest(request), typeof response?.status === "number" ? response.status : undefined, ctx !== undefined && typeof ctx.waitUntil === "function"
+                ? (p) => ctx.waitUntil(p)
+                : undefined);
+        }
+        catch {
+            /* silent by contract */
+        }
+        return response;
+    };
+}
+//# sourceMappingURL=cloudflare-workers.js.map

package/dist/adapters/cloudflare-workers.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cloudflare-workers.js","sourceRoot":"","sources":["../../src/adapters/cloudflare-workers.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAEH,OAAO,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAC;AACrE,OAAO,EAAE,qBAAqB,EAAE,UAAU,EAAyB,MAAM,aAAa,CAAC;AAcvF;;;GAGG;AACH,MAAM,UAAU,qBAAqB,CACnC,OAAiC,EACjC,MAA6C;IAE7C,MAAM,QAAQ,GAAG,gBAAgB,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,mBAAmB,CAAC,MAAM,CAAC,CAAC;IACjF,yEAAyE;IACzE,uDAAuD;IACvD,MAAM,GAAG,GAA0B,EAAE,GAAG,QAAQ,EAAE,mBAAmB,EAAE,IAAI,EAAE,CAAC;IAE9E,OAAO,KAAK,EAAE,OAAO,EAAE,GAAG,EAAE,GAAG,EAAE,EAAE;QACjC,2EAA2E;QAC3E,MAAM,QAAQ,GAAG,MAAM,OAAO,CAAC,OAAO,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC;QAClD,IAAI,CAAC;YACH,UAAU,CACR,GAAG,EACH,qBAAqB,CAAC,OAAO,CAAC,EAC9B,OAAO,QAAQ,EAAE,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,SAAS,EAClE,GAAG,KAAK,SAAS,IAAI,OAAO,GAAG,CAAC,SAAS,KAAK,UAAU;gBACtD,CAAC,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;gBACzB,CAAC,CAAC,SAAS,CACd,CAAC;QACJ,CAAC;QAAC,MAAM,CAAC;YACP,wBAAwB;QAC1B,CAAC;QACD,OAAO,QAAQ,CAAC;IAClB,CAAC,CAAC;AACJ,CAAC"}

package/dist/adapters/express.d.ts

@@ -0,0 +1,49 @@
+/**
+ * Express adapter (also works for Connect-style and bare Node servers with a
+ * compatible `(req, res, next)` signature).
+ *
+ * ```ts
+ * import { createTracetenAICrawlerMiddleware } from "@traceten/ai-crawl/express";
+ *
+ * app.use(
+ *   createTracetenAICrawlerMiddleware({
+ *     siteId: process.env.TRACETEN_SITE_ID!,
+ *     authToken: process.env.TRACETEN_CRAWL_TOKEN!,
+ *   }),
+ * );
+ * ```
+ *
+ * Construction throws on invalid config (server boot — where it should).
+ * The middleware calls `next()` IMMEDIATELY and attaches a `finish`
+ * listener, so the real status code is reported and the request is never
+ * delayed. Everything inside is wrapped; failure is silent.
+ *
+ * Structural types only — no dependency on Express, not even for types.
+ */
+import type { AiCrawlConfig, ResolvedAiCrawlConfig } from "../types.js";
+/** Structural subset of `express.Request` / Node's `IncomingMessage`. */
+export interface NodeLikeRequest {
+    method?: string | undefined;
+    /** Path + query as received (`/docs/x?y=1`). */
+    url?: string | undefined;
+    /** Express keeps the pre-router-mount path here; prefer it when present. */
+    originalUrl?: string | undefined;
+    headers: Record<string, string | string[] | undefined>;
+    /** Express convenience; used for URL scheme when present. */
+    protocol?: string | undefined;
+    socket?: {
+        remoteAddress?: string | undefined;
+    } | undefined;
+}
+/** Structural subset of `express.Response` / Node's `ServerResponse`. */
+export interface NodeLikeResponse {
+    statusCode?: number | undefined;
+    on(event: "finish", listener: () => void): unknown;
+}
+export type NodeLikeNext = (err?: unknown) => void;
+/**
+ * Create the middleware. THROWS at construction (server boot) when the
+ * config is invalid; the returned middleware itself never throws.
+ */
+export declare function createTracetenAICrawlerMiddleware(config: AiCrawlConfig | ResolvedAiCrawlConfig): (req: NodeLikeRequest, res: NodeLikeResponse, next: NodeLikeNext) => void;
+//# sourceMappingURL=express.d.ts.map

package/dist/adapters/express.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"express.d.ts","sourceRoot":"","sources":["../../src/adapters/express.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;GAqBG;AAKH,OAAO,KAAK,EAAE,aAAa,EAAgB,qBAAqB,EAAE,MAAM,aAAa,CAAC;AAEtF,yEAAyE;AACzE,MAAM,WAAW,eAAe;IAC9B,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC5B,gDAAgD;IAChD,GAAG,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACzB,4EAA4E;IAC5E,WAAW,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACjC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,SAAS,CAAC,CAAC;IACvD,6DAA6D;IAC7D,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B,MAAM,CAAC,EAAE;QAAE,aAAa,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;KAAE,GAAG,SAAS,CAAC;CAC7D;AAED,yEAAyE;AACzE,MAAM,WAAW,gBAAgB;IAC/B,UAAU,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAChC,EAAE,CAAC,KAAK,EAAE,QAAQ,EAAE,QAAQ,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC;CACpD;AAED,MAAM,MAAM,YAAY,GAAG,CAAC,GAAG,CAAC,EAAE,OAAO,KAAK,IAAI,CAAC;AA0BnD;;;GAGG;AACH,wBAAgB,iCAAiC,CAC/C,MAAM,EAAE,aAAa,GAAG,qBAAqB,GAC5C,CAAC,GAAG,EAAE,eAAe,EAAE,GAAG,EAAE,gBAAgB,EAAE,IAAI,EAAE,YAAY,KAAK,IAAI,CAiC3E"}

package/dist/adapters/express.js

@@ -0,0 +1,91 @@
+/**
+ * Express adapter (also works for Connect-style and bare Node servers with a
+ * compatible `(req, res, next)` signature).
+ *
+ * ```ts
+ * import { createTracetenAICrawlerMiddleware } from "@traceten/ai-crawl/express";
+ *
+ * app.use(
+ *   createTracetenAICrawlerMiddleware({
+ *     siteId: process.env.TRACETEN_SITE_ID!,
+ *     authToken: process.env.TRACETEN_CRAWL_TOKEN!,
+ *   }),
+ * );
+ * ```
+ *
+ * Construction throws on invalid config (server boot — where it should).
+ * The middleware calls `next()` IMMEDIATELY and attaches a `finish`
+ * listener, so the real status code is reported and the request is never
+ * delayed. Everything inside is wrapped; failure is silent.
+ *
+ * Structural types only — no dependency on Express, not even for types.
+ */
+import { defineAiCrawlConfig, isResolvedConfig } from "../config.js";
+import { evaluateRequest } from "../track.js";
+import { sendReport } from "../report.js";
+function headerLookup(req) {
+    return (name) => {
+        try {
+            const value = req.headers[name.toLowerCase()];
+            if (value === undefined)
+                return null;
+            return Array.isArray(value) ? (value[0] ?? null) : value;
+        }
+        catch {
+            return null;
+        }
+    };
+}
+/** Build an absolute URL for the request, preferring `publicOrigin`. */
+function requestUrl(cfg, req) {
+    const path = req.originalUrl ?? req.url ?? "/";
+    if (path.indexOf("://") !== -1)
+        return path; // already absolute (rare)
+    if (cfg.publicOrigin !== undefined)
+        return cfg.publicOrigin + path;
+    const hostHeader = req.headers["host"];
+    const host = Array.isArray(hostHeader) ? hostHeader[0] : hostHeader;
+    if (host === undefined || host === "")
+        return path;
+    const proto = req.protocol === "http" ? "http" : "https";
+    return `${proto}://${host}${path}`;
+}
+/**
+ * Create the middleware. THROWS at construction (server boot) when the
+ * config is invalid; the returned middleware itself never throws.
+ */
+export function createTracetenAICrawlerMiddleware(config) {
+    const cfg = isResolvedConfig(config) ? config : defineAiCrawlConfig(config);
+    return (req, res, next) => {
+        try {
+            const facts = {
+                method: req.method ?? "GET",
+                url: requestUrl(cfg, req),
+                header: headerLookup(req),
+                socketAddr: req.socket?.remoteAddress,
+            };
+            // Evaluate up-front so non-crawler traffic attaches no listener at
+            // all; report on `finish` so the REAL status code is captured.
+            const payload = evaluateRequest(cfg, facts);
+            if (payload !== null) {
+                res.on("finish", () => {
+                    try {
+                        const status = res.statusCode;
+                        if (typeof status === "number" && status >= 100 && status <= 599) {
+                            payload.status = status;
+                        }
+                        void sendReport(cfg, payload);
+                    }
+                    catch {
+                        /* silent by contract */
+                    }
+                });
+            }
+        }
+        catch {
+            /* silent by contract */
+        }
+        next();
+    };
+}
+//# sourceMappingURL=express.js.map