@devtune/ai-traffic 0.12.0 → 0.13.0

package/README.md

@@ -1,6 +1,6 @@
 # @devtune/ai-traffic
 
-Measure which AI crawlers visit your site with a dependency-free sensor that filters at the edge and forwards only matched machine traffic to DevTune.
+Measure which AI crawlers visit your site, and the visitors AI answers send you, with a dependency-free sensor that filters at the edge and forwards only matched requests to DevTune.
 
 ## Install
 
@@ -14,7 +14,7 @@ Create a project-scoped server-side ingest key and expose it only to your server
 DEVTUNE_AI_TRAFFIC_INGEST_KEY=dt_ingest_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 ```
 
-On Vercel you can use a project ID instead of a key. See [Vercel without an ingest key](#vercel-without-an-ingest-key).
+On Vercel you can use a project ID instead of a key. See [Vercel OIDC without an ingest key](#vercel-oidc-without-an-ingest-key).
 
 ## Quickstarts
 
@@ -76,9 +76,9 @@ createServer((request, response) => {
 
 The hook records the final `response.statusCode` without blocking or changing the response.
 
-## Vercel without an ingest key
+## Vercel OIDC without an ingest key
 
-On Vercel, the sensor can authenticate with each deployment's Vercel OIDC token instead of an ingest key, so there is no long-lived secret to store. Vercel linking must be enabled for your DevTune account while it is in beta.
+On Vercel, the sensor can authenticate with each deployment's Vercel OIDC token instead of an ingest key, so there is no long-lived secret to store.
 
 1. In DevTune, open **Settings › Integrations › Edge middleware** and select **Add Vercel project**. Enter the Vercel team ID, shown as **Team ID** in the team's **Settings › General**, and the Vercel project ID, shown as **Project ID** in the project's **Settings › General**.
 2. Copy the **DevTune project ID** from the same page and add it to the Vercel project's environment variables. It is not a secret.
@@ -100,13 +100,15 @@ On Vercel, the sensor can authenticate with each deployment's Vercel OIDC token
 
 The same `projectId` option works for the Express and Node HTTP adapters running on Vercel.
 
+If you write your own adapter with `projectId` set, and it skips some requests or calls `trackRequest()` after the response, call the client's `captureVercelOidcToken(request)` on every request as it arrives, before any filtering. Vercel puts a current token on every request, so browser requests keep the token for pending sends current.
+
 When both options are set, the sensor sends with the Vercel token and falls back to the ingest key if Vercel or DevTune cannot verify it. With only `projectId`, nothing is sent from a host that is not Vercel, and the sensor logs one warning per process. If a link stays **Not seen yet** after a deploy that received crawler traffic, check both Vercel IDs.
 
 The token is read from each incoming request, never from the environment, and is sent only to Vercel to obtain a token for DevTune. That token is cached in memory until shortly before it expires and is sent only to the ingest endpoint.
 
 ## Privacy by Design
 
-The sensor filters requests where your application runs. It uses a cheap user-agent hint before consulting the AI bot registry, and only matched machine hits are eligible to be forwarded. It uses no cookies, no fingerprinting, and no full request logs.
+The sensor filters requests where your application runs. It forwards two kinds of request and nothing else: a request whose user agent matches the AI bot registry, checked after a cheap user-agent hint, and a page load that arrived from an AI answer (see [AI Referral Visits](#ai-referral-visits)). It uses no cookies, no fingerprinting, and no full request logs.
 
 Requests that a page makes for itself are never forwarded. A crawler that renders pages runs their JavaScript, and its browser then fires prefetches, client navigations and other `fetch()` calls from inside each page. Counting them would report crawls of pages it only fetched in the background; on DevTune's own site, one rendering crawler made three such requests to `/` for every page it loaded. The sensor recognises them from the fetch metadata the browser attaches, so no configuration is needed:
 
@@ -138,8 +140,22 @@ For a matched crawler, DevTune receives only:
 - timestamp
 - the address the request arrived from, used to verify the crawler and then discarded
 - for a signed agent, its Web Bot Auth signature material, used to verify the agent and then discarded (see [Signed Agents](#signed-agents))
+- whether the request came through a CDN (see [CDN Flag](#cdn-flag))
+
+For an AI referral visit, DevTune receives only:
+
+- `path`
+- origin plus path, with query parameters removed
+- user agent
+- response status when the adapter can know it
+- timestamp
+- the referring site's origin, never the rest of the `Referer`
+- the `utm_source` and `utm_medium` values, and whether an `oppref` parameter was present, never its value
+- whether the request came through a CDN
+
+The page path is sent as it is, so a path that carries a token or an identifier, such as `/invite/abc123`, is sent too. A referral visit never carries an address or signature material.
 
-Request bodies, cookies, unrelated headers, and query strings are never sent. Signature material is only forwarded for a request without a query, so a reset link, OAuth callback or search URL a signed agent follows never leaves the site. Only these request headers are ever forwarded with a signature: `accept`, `accept-encoding`, `accept-language`, `cache-control`, `content-length`, `content-type`, `date`, `from`, `host` and `user-agent`, exported as `WEB_BOT_AUTH_FORWARDED_HEADERS`. A request whose signature covers any other header, such as `cookie`, `authorization` or an API key, forwards no signature material at all, so credentials never leave the site. An address is sent only for a request whose user agent matches the AI bot registry, so ordinary browser traffic never reaches DevTune at all. Note the limit of that statement: the match is on the user agent, and anyone can send a crawler's user agent, so a person whose client does that would have their address sent. That is the same fact the verification exists to handle, and it is why the address is checked and discarded rather than stored. DevTune checks it against the claimed operator's published networks on receipt and keeps the verdict, not the address. Set `trustedProxy: "none"` to send no address at all; DevTune then records that traffic as unverified.
+Request bodies, cookies, unrelated headers, and other query parameters are never sent. Signature material is only forwarded for a request without a query, so a reset link, OAuth callback or search URL a signed agent follows never leaves the site. Only these request headers are ever forwarded with a signature: `accept`, `accept-encoding`, `accept-language`, `cache-control`, `content-length`, `content-type`, `date`, `from`, `host` and `user-agent`, exported as `WEB_BOT_AUTH_FORWARDED_HEADERS`. A request whose signature covers any other header, such as `cookie`, `authorization` or an API key, forwards no signature material at all, so credentials never leave the site. An address is sent only for a request whose user agent matches the AI bot registry. Referral visits never carry one, and other browser traffic never reaches DevTune at all. Note the limit of that statement: the match is on the user agent, and anyone can send a crawler's user agent, so a person whose client does that would have their address sent. That is the same fact the verification exists to handle, and it is why the address is checked and discarded rather than stored. DevTune checks it against the claimed operator's published networks on receipt and keeps the verdict, not the address. Set `trustedProxy: "none"` to send no address at all; DevTune then records that traffic as unverified.
 
 ## Bot Verification
 
@@ -241,9 +257,54 @@ const match = detectAiCrawler(userAgent, {
 Ingest and registry requests include an `x-devtune-sdk-version` header with the package version, so
 DevTune can show an upgrade notice when a newer release is available. It carries no visitor data.
 
-## Machine Traffic and Human Referrals
+## AI Referral Visits
+
+From 0.13.0 the sensor also records people who click through to your site from an AI answer. Because it runs on your server, it counts visitors whose browser blocks DevTune's tracking snippet.
+
+Set `captureReferrals: false` to record AI bot requests only, as before 0.13.0. Every adapter takes it with its other options:
+
+```ts
+const trackAiTraffic = createDevTuneAiTrafficMiddleware({
+  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
+  captureReferrals: false,
+});
+```
+
+A request is a referral visit when all of these hold:
 
-The sensor deliberately measures machine traffic only. DevTune gets human visits from AI products through its GA4 integration, where sessions, engagement, and conversions provide a richer picture than middleware referrer matching. The referrer detector is available for local classification, but the sensor adapters never forward human referral visits.
+- it is a `GET`;
+- it came from another site: its `Sec-Fetch-Site` is not `same-origin` or `same-site`;
+- it is a top-level page load: its `Sec-Fetch-Dest` is `document` or it has no fetch metadata, so a frame load is checked as a crawler only;
+- it is not a request made from inside a page and not a framework data request, the same rules as for crawlers (see [Privacy by Design](#privacy-by-design));
+- its `Referer` host, or failing that its `utm_source`, is one of the registry's AI referral sources, such as `chatgpt.com` or `utm_source=perplexity`.
+
+A `Referer` host also matches its subdomains. A `utm_source` matches case-insensitively, either as a listed source or as a hostname on the hosts list. A request whose user agent matches the AI bot registry is always recorded as a bot request, whatever its `Referer`.
+
+The referral sources arrive with the bot patterns in each registry refresh, so a new source needs no upgrade. The package bundles DevTune's lists, which it uses before the first refresh. A refresh whose referral sources are missing or unusable, as from a registry older than referral capture, keeps the last good lists.
+
+DevTune classifies every event again on receipt. A paid click, one whose `utm_medium` is a paid medium such as `cpc` or `paid_ai`, or that carries OpenAI's `oppref` ad click ID, is not counted as an AI referral. Where the adapter knows the response status, a page load answered with a 301, 302, 303, 307 or 308 is not recorded, because the visit is counted where the redirect lands. Next.js middleware runs before the response and sends no status, so see the [limit on redirects](https://docs.devtune.ai/ai-traffic/edge-middleware#limits) there.
+
+`trackRequest()` returns `{ kind: "referral", platform, event }` for a referral visit and `{ kind: "bot", entry, event }` for a crawler, or `null`:
+
+```ts
+const match = aiTraffic.trackRequest(request, event);
+
+if (match?.kind === "bot") {
+  console.log(match.entry.label);
+}
+```
+
+Referral visits wait in their own queue, bounded by `maxQueuedReferralEvents` (default 200), and have their own retry budget, so a burst of them never pushes out crawler events or spends their retries. Crawler events are sent first, and a batch never mixes the two (see [Rate-Limit Retries](#rate-limit-retries)).
+
+If you write your own adapter, pass `method` and the full request `url`, query included. Without either the sensor records crawlers only. Fetch requests and the Express and Node adapters already provide both.
+
+For when DevTune treats the sensor's referral count for a day as complete, and its limits, see [AI referral visits](https://docs.devtune.ai/ai-traffic/edge-middleware#ai-referral-visits).
+
+### CDN Flag
+
+Every event carries `cdn: true` when its request came through a CDN that marks what it forwards: Cloudflare (`cf-ray`), CloudFront (`x-amz-cf-id`) or Fastly (any `fastly-*` header). A cache in front of your app can answer page loads the sensor never sees, so DevTune does not treat the sensor's referral count for that host and day as complete. A cache that adds none of these headers is not detected.
+
+A custom adapter's `HeadersLike` can provide `keys()` so any `fastly-*` header is found. Without it, only `fastly-client-ip`, `fastly-ff` and `fastly-ssl` are checked.
 
 ## Production Notes
 
@@ -267,7 +328,11 @@ Registry refreshes use `waitUntil` when provided, with a five-second fetch timeo
 
 A `429` from the ingest endpoint is the one failure the server tells us is temporary, so the batch is redelivered rather than dropped. The client waits for the response's `Retry-After` — delay-seconds or an HTTP date — and falls back to exponential backoff from 250 ms when the header is absent or unusable. The batch being retried is held intact, so events that arrive during a backoff are sent separately rather than folded into it.
 
-Retries are bounded so a short-lived runtime cannot be held open by a throttled endpoint. The budget belongs to one flush, not to each batch: `maxRetryAttempts` defaults to 3 and `maxRetryDelayMs` caps each wait at 5000 ms, including a `Retry-After` longer than the cap, so a flush adds at most 15 s no matter how deep the queue is. When the budget is spent the current batch is dropped with the usual sampled warning and the drain stops; whatever is behind it stays queued for the next flush, which gets a fresh budget. `maxRetryAttempts: 0` disables waiting entirely: a 429 drops the batch on its first rejection, as it did before this behaviour existed. The drain still stops there rather than dropping every batch behind it, so the rest stays queued. Tune both:
+Retries are bounded so a short-lived runtime cannot be held open by a throttled endpoint. The budget belongs to one flush, not to each batch, and crawler events and referral visits each get their own: `maxRetryAttempts` defaults to 3 and `maxRetryDelayMs` caps each wait at 5000 ms, including a `Retry-After` longer than the cap, so a flush adds at most 15 s per kind, 30 s in all, no matter how deep the queues are. When a budget is spent the current batch is dropped with the usual sampled warning. If it was the crawler budget the flush ends there, because referral visits would meet the same limit; if it was the referral budget, crawler events keep draining. A referral batch rate-limited while crawler events are waiting goes back to the front of the referral queue, and the crawler events go first.
+
+What a spent budget leaves queued gets one follow-up flush with fresh budgets, crawler events first, after `maxRetryDelayMs` or `flushIntervalMs`, whichever is longer. If the follow-up is throttled too it schedules none, and the rest waits for the next tracked request. The follow-up runs only in a long-lived process: its timer does not keep a Node process open and is not tied to `waitUntil`, so it does not survive a Vercel Function, Edge or Workers response. A tracked request that fills a batch still flushes immediately.
+
+`maxRetryAttempts: 0` disables waiting entirely: a 429 drops the batch on its first rejection, as it did before this behaviour existed. The drain still stops there rather than dropping every batch behind it, so the rest stays queued. Tune both:
 
 ```ts
 const trackAiTraffic = createDevTuneAiTrafficMiddleware({
@@ -277,7 +342,7 @@ const trackAiTraffic = createDevTuneAiTrafficMiddleware({
 });
 ```
 
-Because backing off lets the queue drain slower than traffic arrives, the queue itself is bounded. `maxQueuedEvents` defaults to 1000; past it the oldest events are shed with a sampled warning, so a sustained throttle cannot grow memory without limit.
+Because backing off lets the queue drain slower than traffic arrives, the queue itself is bounded. `maxQueuedEvents` defaults to 1000; past it the oldest events are shed with a sampled warning, so a sustained throttle cannot grow memory without limit. Referral visits have their own bound, `maxQueuedReferralEvents`, which defaults to 200 and sheds only referral visits.
 
 Non-429 responses and network errors are unchanged: the batch is dropped after a single attempt, so a hard outage never keeps the queue alive. The one exception is a `503` to a batch sent with a Vercel token and no ingest key configured: DevTune could not reach Vercel to verify the token, so it is retried like a `429` within the same budget.
 
@@ -318,7 +383,7 @@ an AI client or its model provider.
 
 From 0.5.0, new platform names and traffic classes can arrive through the remote registry without a package upgrade. DevTune handles classification and identity verification on its servers. The SDK keeps validating match patterns before using them.
 
-Clients start with the bundled snapshot. The first request, then the first request after the one-hour cache expires, triggers a background refresh from `https://devtune.ai/api/v1/llm-traffic/registry`. This includes ordinary browser requests, but the registry GET contains no visitor data and only matching machine requests are sent to ingest. Node, Express and middleware use the same policy. Concurrent requests share one refresh, and `ETag` avoids downloading an unchanged registry.
+Clients start with the bundled snapshot. The first request, then the first request after the one-hour cache expires, triggers a background refresh from `https://devtune.ai/api/v1/llm-traffic/registry`. This includes ordinary browser requests, but the registry GET contains no visitor data and only matched requests are sent to ingest. The refresh brings the AI referral sources along with the bot patterns. Node, Express and middleware use the same policy. Concurrent requests share one refresh, and `ETag` avoids downloading an unchanged registry.
 
 A failed refresh keeps the last working registry, or the bundled snapshot before the first successful refresh. Retries wait for the next cache interval. Requests use the currently cached rules while a refresh is in progress, so a newly added bot may be missed until that refresh completes. Create the client or middleware once per process or runtime instance to reuse its cache.
 
@@ -339,7 +404,7 @@ const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
 
 ### Failure Behavior
 
-Registry refresh and ingest sends are fire-and-forget and guarded. Network failures may drop telemetry, but they do not block, throw into, or alter the customer response. The ingest endpoint and batched wire format are unchanged.
+Registry refresh and ingest sends are fire-and-forget and guarded. Network failures may drop telemetry, but they do not block, throw into, or alter the customer response.
 
 ## Documentation