@apideck/agent-analytics 0.10.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +76 -0
- package/dist/adapters/posthog.d.cts +1 -1
- package/dist/adapters/posthog.d.ts +1 -1
- package/dist/adapters/webhook.d.cts +1 -1
- package/dist/adapters/webhook.d.ts +1 -1
- package/dist/index.cjs +598 -0
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +100 -2
- package/dist/index.d.ts +100 -2
- package/dist/index.js +589 -1
- package/dist/index.js.map +1 -1
- package/dist/{types-B7jSKtLz.d.cts → types-DKqlfVz6.d.cts} +18 -0
- package/dist/{types-B7jSKtLz.d.ts → types-DKqlfVz6.d.ts} +18 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -306,6 +306,82 @@ Full middleware example: [`README.md → Markdown mirror helpers`](./README.md#m
|
|
|
306
306
|
|
|
307
307
|
---
|
|
308
308
|
|
|
309
|
+
## Advanced: verifying crawler identity against published IP ranges
|
|
310
|
+
|
|
311
|
+
User agents are trivially forged — `curl -A "ChatGPT-User"` is indistinguishable
|
|
312
|
+
from the real thing at the UA layer. Set `verifyIdentity: true` to check the
|
|
313
|
+
client IP against the vendor's published crawler ranges:
|
|
314
|
+
|
|
315
|
+
```ts
|
|
316
|
+
void trackVisit(request, {
|
|
317
|
+
analytics,
|
|
318
|
+
verifyIdentity: true,
|
|
319
|
+
captureIp: true // not required, but useful for auditing a 'spoofed' verdict
|
|
320
|
+
})
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
Three properties land on the event:
|
|
324
|
+
|
|
325
|
+
| property | values |
|
|
326
|
+
| --- | --- |
|
|
327
|
+
| `bot_verification` | `verified` \| `spoofed` \| `unverifiable` \| `not-claimed` |
|
|
328
|
+
| `bot_verified` | `true` \| `false` \| `null` — tri-state, for quick filtering |
|
|
329
|
+
| `bot_verification_reason` | why, when the verdict is `unverifiable` |
|
|
330
|
+
|
|
331
|
+
### What can actually be verified
|
|
332
|
+
|
|
333
|
+
Only vendors that publish a machine-readable range feed: **OpenAI**,
|
|
334
|
+
**Anthropic**, **Perplexity**, and **Apple**. Bytespider, Amazonbot, Meta and
|
|
335
|
+
the rest report `unverifiable` — never `spoofed`. Collapsing "we can't check"
|
|
336
|
+
into "impostor" would be a false accusation, which is why `bot_verified` is
|
|
337
|
+
tri-state rather than a boolean.
|
|
338
|
+
|
|
339
|
+
### Server-side crawlers vs client-side agents
|
|
340
|
+
|
|
341
|
+
A published range list covers a vendor's **crawler fleet**, not its products
|
|
342
|
+
that fetch from the end user's device. Claude Code runs on a developer's
|
|
343
|
+
laptop, so the request carries *their* IP and will never appear in Anthropic's
|
|
344
|
+
ranges. Measured over 30 days of production traffic:
|
|
345
|
+
|
|
346
|
+
| user agent | events | distinct IPs | in published range |
|
|
347
|
+
| --- | ---: | ---: | ---: |
|
|
348
|
+
| `ClaudeBot` | 13,671 | 236 | 96% |
|
|
349
|
+
| `PerplexityBot` | 6,897 | 158 | 91% |
|
|
350
|
+
| `ChatGPT-User` | ~72,000 | 43 | 99% |
|
|
351
|
+
| `Claude-User` (claude-code CLI) | 6,492 | 4,486 | **0%** |
|
|
352
|
+
| `Perplexity-User` | 493 | 148 | **0%** |
|
|
353
|
+
|
|
354
|
+
A naive vendor-level check would brand the bottom two rows — roughly 7,000
|
|
355
|
+
legitimate fetches a month — as impersonation. So the library gates verdicts on
|
|
356
|
+
the *product*, returning `unverifiable` with reason `client-side-agent` for
|
|
357
|
+
those. Note the distinction is not a `-User` suffix: OpenAI's `ChatGPT-User`
|
|
358
|
+
fetches server-side from Azure and verifies at ~99%.
|
|
359
|
+
|
|
360
|
+
### Keeping the ranges fresh
|
|
361
|
+
|
|
362
|
+
The bundled snapshot is in `src/bot-ranges.ts`, stamped with
|
|
363
|
+
`BOT_RANGES_CAPTURED_AT`. Refresh it on a schedule:
|
|
364
|
+
|
|
365
|
+
```bash
|
|
366
|
+
node scripts/refresh-bot-ranges.mjs
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
Freshness is the whole game. Nearly every OpenAI prefix is an Azure block and
|
|
370
|
+
Anthropic's are GCP, so "came from a datacenter" proves nothing on its own —
|
|
371
|
+
only membership in the *current* published list does. A stale snapshot produces
|
|
372
|
+
false `spoofed` verdicts on real crawlers, so the refresh script refuses to
|
|
373
|
+
write a list that shrinks by more than half or when any feed errors.
|
|
374
|
+
|
|
375
|
+
### Trusting the client IP
|
|
376
|
+
|
|
377
|
+
The verdict is only as good as the IP. On Vercel and Cloudflare the edge
|
|
378
|
+
overwrites `x-forwarded-for`, so the first hop is trustworthy. Behind a proxy
|
|
379
|
+
that passes a client-supplied header through, an attacker controls the value
|
|
380
|
+
and `verified` means nothing — confirm your proxy's behaviour before acting on
|
|
381
|
+
this data.
|
|
382
|
+
|
|
383
|
+
---
|
|
384
|
+
|
|
309
385
|
## Advanced: Peec.ai crawl-insights export
|
|
310
386
|
|
|
311
387
|
[Peec.ai](https://peec.ai)'s **Agent analytics** product ingests a CSV/CLF access log and produces dashboards on top of it. The Peec docs assume you have a Vercel Log Drain → Axiom (or similar) pipeline that emits these eight columns: `timestamp, request_method, request_url, response_status, client_ip, user_agent, country_code, referer`.
|