@devtune/ai-traffic 0.1.2 → 0.2.1

package/README.md

@@ -1,8 +1,6 @@
 # @devtune/ai-traffic
 
-`@devtune/ai-traffic` is the filtered edge-push sensor for DevTune AI Traffic. It matches incoming requests against DevTune's AI bot registry and forwards only matching machine hits to DevTune's ingest endpoint.
-
-Cloudflare-proxied sites should prefer the Cloudflare pull integration when available. For Vercel and self-hosted sites, this package keeps volume low by filtering at your edge instead of shipping full request logs.
+Measure which AI crawlers visit your site with a dependency-free sensor that filters at the edge and forwards only matched machine traffic to DevTune.
 
 ## Install
 
@@ -10,118 +8,214 @@ Cloudflare-proxied sites should prefer the Cloudflare pull integration when avai
 pnpm add @devtune/ai-traffic
 ```
 
-Set your project-scoped server-side ingest key:
+Create a project-scoped server-side ingest key and expose it only to your server runtime:
 
 ```bash
 DEVTUNE_AI_TRAFFIC_INGEST_KEY=dt_ingest_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 ```
 
-## Next.js Proxy / Middleware
+## Quickstarts
+
+### Next.js
 
-For Next.js 16, add `proxy.ts` and export `proxy()`. For older Next.js projects, use the same body from `middleware.ts` and export `middleware()` instead.
+For Next.js 16, add `proxy.ts`. For older Next.js projects, use the same body in `middleware.ts` and export `middleware()` instead.
 
 ```ts
 // proxy.ts
-import { createDevTuneAiTraffic } from "@devtune/ai-traffic";
+import { createDevTuneAiTrafficMiddleware } from "@devtune/ai-traffic";
 import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
 
-const aiTraffic = createDevTuneAiTraffic({
+const trackAiTraffic = createDevTuneAiTrafficMiddleware({
   ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
-  batchSize: 1,
-  defaultStatus: null,
-  flushIntervalMs: 0,
-  waitUntilRegistryRefresh: false,
 });
 
 export function proxy(request: NextRequest, event: NextFetchEvent) {
-  const response = NextResponse.next();
-
-  aiTraffic.trackRequest(request, event);
+  trackAiTraffic(request, event);
 
-  return response;
+  return NextResponse.next();
 }
 ```
 
-Use `defaultStatus: null` for pass-through proxy or middleware requests. Next.js proxy cannot observe the final status after route resolution, so this records the AI bot hit without guessing that every pass-through is a 200. Use `batchSize: 1` and `flushIntervalMs: 0` in middleware so Vercel does not hold middleware duration open for a delayed batch timer. Use `waitUntilRegistryRefresh: false` in middleware so hourly registry refreshes do not contribute to `waitUntil` duration. The package still schedules matched ingest work through `event.waitUntil()` when available, so the request path can return immediately after matching. Ordinary browser user agents are ignored before registry refresh work is scheduled.
-
-When your proxy or middleware returns a response directly, pass that exact status:
+### Express
 
 ```ts
-export function proxy(request: NextRequest, event: NextFetchEvent) {
-  const response = new Response("Forbidden", { status: 403 });
+import express from "express";
+import { createDevTuneAiTrafficExpressMiddleware } from "@devtune/ai-traffic/express";
 
-  aiTraffic.trackRequest(request, event, response.status);
+const app = express();
 
-  return response;
-}
+app.use(
+  createDevTuneAiTrafficExpressMiddleware({
+    ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
+  }),
+);
 ```
 
-If you only need a minimal best-effort hook, the middleware helper is also available:
+The middleware calls `next()` immediately and records the actual response status after Express emits `finish`.
+
+### Node HTTP
 
 ```ts
-import { createDevTuneAiTrafficMiddleware } from "@devtune/ai-traffic";
-import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
+import { createServer } from "node:http";
 
-const trackAiTraffic = createDevTuneAiTrafficMiddleware({
+import { createDevTuneAiTrafficNodeHandler } from "@devtune/ai-traffic/node";
+
+const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
   ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
 });
 
-export function proxy(request: NextRequest, event: NextFetchEvent) {
-  trackAiTraffic(request, event);
+createServer((request, response) => {
+  trackAiTraffic(request, response);
 
-  return NextResponse.next();
-}
+  response.statusCode = 200;
+  response.end("ok");
+}).listen(3000);
 ```
 
-The middleware helper applies the middleware-safe defaults shown above: unknown pass-through status, immediate single-hit sends, and registry refreshes that are not attached to `waitUntil`. The package does not import `next/server`; it uses the standard request shape and the `waitUntil` method that Next.js passes to proxy and middleware functions.
+The hook records the final `response.statusCode` without blocking or changing the response.
 
-## Status Capture
+## Privacy by Design
 
-Next.js proxy and middleware can report statuses they return directly, such as redirects, denials, and configured misses. They still cannot observe the final page response status after Next.js route resolution, so pass-through requests should use `defaultStatus: null` unless you intentionally want a best-effort default. When status is unknown, DevTune still counts the matched machine hit and treats status-specific reporting as unavailable for that event.
+The sensor filters requests where your application runs. It uses a cheap user-agent hint before consulting the AI bot registry, and only matched machine hits are eligible to be forwarded. It uses no cookies, no fingerprinting, and no full request logs.
 
-Use the route wrapper where you own the handler response and need exact status capture:
+For a matched crawler, DevTune receives only:
+
+- `path`
+- origin plus path, with query parameters removed
+- user agent
+- response status when the adapter can know it
+- timestamp
+
+Request bodies, cookies, IP addresses, unrelated headers, and query-derived identifiers are not sent.
+
+## Detect Without Sending
+
+The standalone classifiers need no ingest key and never make a network request or push data to DevTune:
 
 ```ts
-// app/api/example/route.ts
-import { withDevTuneAiTrafficRoute } from "@devtune/ai-traffic";
-
-export const GET = withDevTuneAiTrafficRoute(
-  async function GET() {
-    return Response.json({ ok: true }, { status: 200 });
-  },
-  {
-    ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
-  },
-);
+import { detectAiCrawler, detectAiReferrer } from "@devtune/ai-traffic";
+
+const crawler = detectAiCrawler(request);
+const referrer = detectAiReferrer(request);
+```
+
+Each function also accepts the relevant string directly: a user-agent for `detectAiCrawler()` or a referrer URL for `detectAiReferrer()`.
+
+`detectAiCrawler()` returns the matched bot registry entry or `null`:
+
+```ts
+type AiCrawlerDetection = {
+  uaPattern: string;
+  llmPlatform: string;
+  botClass: "training_crawler" | "index_bot" | "answer_fetcher" | "acting_agent";
+  label: string;
+  status?: "active" | "retired";
+  asnHints?: number[] | null;
+  notes?: string | null;
+};
+```
+
+`detectAiReferrer()` returns the matched referrer registry entry or `null`:
+
+```ts
+type AiReferrerDetection = {
+  hostname: string;
+  llmPlatform: string;
+  label: string;
+};
+```
+
+Both use bundled registry snapshots by default, so they work offline and in CI. Pass `registryEntries` as the second argument when you need to classify against a pinned or private registry:
+
+```ts
+const match = detectAiCrawler(userAgent, {
+  registryEntries: myRegistryEntries,
+});
 ```
 
-For known misses in middleware, mark them explicitly:
+## Machine Traffic and Human Referrals
+
+The sensor deliberately measures machine traffic only. DevTune gets human visits from AI products through its GA4 integration, where sessions, engagement, and conversions provide a richer picture than middleware referrer matching. The referrer detector is available for local classification, but the sensor adapters never forward human referral visits.
+
+## Production Notes
+
+### Batching and Runtime Lifetime
+
+The default client configuration sends up to 10 matched events per unchanged ingest payload, with a 250 ms flush window for low-volume traffic. Set `batchSize: 1` and `flushIntervalMs: 0` to opt out and send each matched event immediately.
+
+Express and Node servers are long-lived enough to benefit directly from the default. Next.js proxy and middleware pass the delayed send to `event.waitUntil()` when available, which prevents the response from waiting but may keep the middleware invocation alive for the short flush window. Use the explicit opt-out in short-lived runtimes that cannot reliably preserve delayed work, or when minimizing middleware duration matters more than request coalescing:
 
 ```ts
 const trackAiTraffic = createDevTuneAiTrafficMiddleware({
   ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
-  notFoundPathPatterns: [/^\/docs\/removed\//],
+  batchSize: 1,
+  flushIntervalMs: 0,
 });
 ```
 
-## What Gets Sent
+The middleware helper leaves registry refresh work outside `waitUntil` by default. Ordinary browser user agents are rejected before refresh or ingest work is scheduled.
 
-For matched AI bot requests only, the package sends:
+### Rate-Limit Retries
 
-- `path`
-- `url`
-- `ua`
-- `status` when known
-- `ts`
+A `429` from the ingest endpoint is the one failure the server tells us is temporary, so the batch is redelivered rather than dropped. The client waits for the response's `Retry-After` — delay-seconds or an HTTP date — and falls back to exponential backoff from 250 ms when the header is absent or unusable. The batch being retried is held intact, so events that arrive during a backoff are sent separately rather than folded into it.
+
+Retries are bounded so a short-lived runtime cannot be held open by a throttled endpoint. The budget belongs to one flush, not to each batch: `maxRetryAttempts` defaults to 3 and `maxRetryDelayMs` caps each wait at 5000 ms, including a `Retry-After` longer than the cap, so a flush adds at most 15 s no matter how deep the queue is. When the budget is spent the current batch is dropped with the usual sampled warning and the drain stops; whatever is behind it stays queued for the next flush, which gets a fresh budget. `maxRetryAttempts: 0` disables waiting entirely: a 429 drops the batch on its first rejection, as it did before this behaviour existed. The drain still stops there rather than dropping every batch behind it, so the rest stays queued. Tune both:
+
+```ts
+const trackAiTraffic = createDevTuneAiTrafficMiddleware({
+  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
+  maxRetryAttempts: 2,
+  maxRetryDelayMs: 2_000,
+});
+```
+
+Because backing off lets the queue drain slower than traffic arrives, the queue itself is bounded. `maxQueuedEvents` defaults to 1000; past it the oldest events are shed with a sampled warning, so a sustained throttle cannot grow memory without limit.
+
+Non-429 responses and network errors are unchanged: the batch is dropped after a single attempt, so a hard outage never keeps the queue alive.
+
+### Status Semantics
+
+Express and Node adapters observe the completed response and report its actual status. Next.js proxy and middleware cannot observe the final route status after pass-through, so the convenience middleware omits status instead of guessing a 200.
+
+When a Next.js proxy returns a response directly, use the client and pass the known status:
 
-It sends the batch to `https://devtune.ai/api/v1/llm-traffic/ingest` with `Authorization: Bearer <ingest key>`. It does not send cookies, request bodies, query-derived user identifiers, IP addresses, headers other than the user agent, or DevTune's client-side snippet key.
+```ts
+import { createDevTuneAiTraffic } from "@devtune/ai-traffic";
+import type { NextFetchEvent, NextRequest } from "next/server";
+
+const aiTraffic = createDevTuneAiTraffic({
+  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
+  defaultStatus: null,
+});
+
+export function proxy(request: NextRequest, event: NextFetchEvent) {
+  const response = new Response("Forbidden", { status: 403 });
+
+  aiTraffic.trackRequest(request, event, response.status);
 
-## Registry Cache
+  return response;
+}
+```
+
+Use `withDevTuneAiTrafficRoute()` where you own a Fetch-compatible route handler and want its exact response status captured automatically. `notFoundPathPatterns` and `statusResolver` remain available for applications that can provide additional status knowledge.
+
+### Registry Refresh and Fallback
+
+Clients start with the bundled AI bot snapshot, refresh from `https://devtune.ai/api/v1/llm-traffic/registry`, cache active entries for about an hour, and use `ETag` revalidation. Failed refreshes are guarded and sampled; they never fail the application response, and the bundled snapshot remains usable.
 
-The package fetches `https://devtune.ai/api/v1/llm-traffic/registry`, caches the active registry for about an hour, and uses `ETag` revalidation. If the registry cannot be fetched, it falls back to a bundled snapshot so known AI bots still match.
+User-agent matching is a conservative signal. Some agents spoof ordinary browsers or require network-level signals, so reported counts are a floor rather than exact bot truth. Cloudflare-proxied sites should prefer DevTune's Cloudflare pull integration when available because it can combine request, bot-score, and network signals without running middleware.
 
-User-agent matching is a v1 signal. Some agents spoof ordinary browsers or need ASN/TLS hints, so counts are floors rather than exact bot truth. Cloudflare pull should be used when available because Cloudflare can combine request, bot-score, and network-level signals.
+### Forwarded Origins
+
+Express and Node adapters build event URLs from the request protocol and host, preferring the first value in standard `X-Forwarded-Proto` and `X-Forwarded-Host` chains. Only accept those headers from a trusted proxy. If they are unavailable or not trustworthy in your deployment, pass a fixed `origin`:
+
+```ts
+const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
+  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
+  origin: "https://www.example.com",
+});
+```
 
-## Vercel Pull Status
+### Failure Behavior
 
-Vercel Observability exposes bot and AI-crawler insights in the dashboard query builder, but the public Observability REST API currently covers Observability Plus project settings rather than event reads. Until Vercel ships a supported Observability read/query API, this middleware is the Vercel path.
+Registry refresh and ingest sends are fire-and-forget and guarded. Network failures may drop telemetry, but they do not block, throw into, or alter the customer response. The ingest endpoint and batched wire format are unchanged.