@osqd/bothandlerjs 0.3.0

package/docs/course/06-identity.md

@@ -0,0 +1,161 @@
+# Lesson 6 — Identity and verification
+
+**Goal:** recognise a crawler by name, then prove or refute the claim — and understand why
+a name alone is worth so little.
+
+← [Course](index.md) · Prev: [The detectors](05-detectors.md) · Next: [Actors and behaviour](07-actors.md)
+
+---
+
+## A name is a claim, not a fact
+
+`self-identified` matches 161 signatures across search, AI, SEO, social, monitoring,
+feeds, archives, security tooling, HTTP libraries and headless runtimes. When it fires you
+get an `identity` — `"googlebot"`, `"gptbot"` — and a `category`.
+
+For a client that has nothing to gain by lying, that is enough: `curl` saying it is `curl`
+is proof, because no honest client is harmed by being believed.
+
+**But `Googlebot` is worth impersonating.** A name that buys privileged treatment cannot be
+taken on trust, which is what verification is for.
+
+## Do this
+
+```js
+const forged = await detector.assess(
+  createFacts({
+    method: "GET", url: "/", ip: "203.0.113.200",
+    headers: { host: "serif.example", "user-agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" },
+  }),
+);
+
+console.log(`verdict=${forged.verdict} class=${forged.botClass} identity=${forged.identity} certain=${forged.certain}`);
+for (const e of forged.evidence) console.log(`  [${e.certainty}] ${e.detector}: ${e.summary}`);
+```
+
+> This one makes a real DNS lookup. It is the only example in the course that touches the
+> network.
+
+### Checkpoint
+
+```
+verdict=confirmed-bot class=impersonator identity=googlebot certain=true
+  [certain] self-identified: User-Agent identifies Googlebot
+  [certain] crawler-verification: Client claims to be Googlebot, but DNS refutes it:
+            address has no PTR record, which every operator of a verifiable crawler publishes
+```
+
+Two pieces of proof, pointing in opposite directions about the same client — it *is*
+declared automation, and it is *not* what it declared. The class is `impersonator`, and
+that is a verdict you can act on with confidence, because it rests on an external authority
+rather than on a pattern.
+
+## How forward-confirmed reverse DNS works
+
+Three steps, and the third is the one people skip:
+
+1. **Reverse.** Look up the `PTR` record for the client's address → `crawl-66-249-66-1.googlebot.com`
+2. **Check the domain.** Does it end in a domain the operator publishes? → `googlebot.com` ✓
+3. **Forward.** Resolve that hostname back to an address. Does it match the one you started
+   with?
+
+Without step 3, anyone who controls reverse DNS for their own address can claim any name
+they like. With it, the claim can only be made by somebody who controls the operator's
+forward DNS too.
+
+| Outcome | Verdict |
+| ------- | ------- |
+| forward-confirmed, domain matches | `verified-bot` — proof, used to **allow** |
+| resolves, domain does not match | `impersonator` — proof, used to **refuse** |
+| no `PTR`, or forward does not match | `impersonator` |
+| resolver error, timeout, no answer | **silence** — no evidence either way |
+
+That last row matters. A resolver having a bad afternoon must never look like an
+accusation, so a lookup that fails produces nothing rather than a refutation.
+
+## Verifying by published address ranges
+
+Twelve shipped signatures — every AI crawler among them — verify by address rather than by
+DNS. It is better where available: a lookup instead of a round trip on the request path,
+immune to somebody else's DNS, and it works for crawlers that publish ranges and no useful
+`PTR` record.
+
+**The library ships no address data**, deliberately. A range baked into a release is wrong
+by the time you install it, and being wrong here means verifying whoever has since been
+handed the address. What it ships is the URL each operator publishes:
+
+```js
+import { startCrawlerRangeRefresh } from "@osqd/bothandlerjs";
+
+const stop = startCrawlerRangeRefresh(detector);   // twice a day
+```
+
+Opt-in, because it makes outbound requests and a dependency-free package quietly fetching
+URLs on a timer is not something to inherit by accident. It fails open per source: one
+publisher being down leaves every other crawler's ranges as they were.
+
+Two things it refuses outright, because these ranges do not merely *describe* a crawler,
+they **verify** one: a list containing a block wider than any crawler owns, and an empty
+list. Either would hand verified status — which most policies allow — to whatever it
+covered.
+
+Supplying them yourself, for a mirror you control:
+
+```js
+detector.updateCrawlerRanges("gptbot", ["203.0.113.0/24"]);
+```
+
+## Categories, and why they are the useful handle
+
+Every signature has a category: `search`, `ai`, `seo`, `social`, `monitoring`, `feed`,
+`archive`, `security`, `library`, `headless`, `advertising`.
+
+Categories are how a policy expresses a *business* decision without naming thirty crawlers:
+
+```js
+{ id: "ai-decline", match: { category: "ai", certain: true }, action: "block" }
+```
+
+You will use this in [lesson 10](10-actions-and-presets.md), where the
+`decline-ai-training` preset splits the AI fleet by job — training crawlers declined,
+fetch-because-a-person-asked served.
+
+## Exercise
+
+Serif wants Googlebot and Bingbot allowed, and anything forging them refused. Write the two
+rules — you have not learned rule syntax yet, so write them as sentences and check your
+reasoning.
+
+<details>
+<summary>Answer</summary>
+
+```js
+{ id: "verified-crawler-allow", match: { verdict: "verified-bot" }, action: "allow" }
+{ id: "impersonator-block", match: { botClass: "impersonator", certain: true }, action: "block" }
+```
+
+Two things worth noticing.
+
+**The allow rule does not name Googlebot.** `verified-bot` already means "an external
+authority confirmed this", so naming crawlers individually only creates a list to maintain.
+
+**The block rule keeps `certain: true`** even though `impersonator` is only ever reached
+through proof. It costs nothing, and it means the rule still says what it depends on if
+somebody later adds a probabilistic route to that class.
+</details>
+
+## What you learned
+
+- A name is a claim; for clients with nothing to gain by lying, that is enough
+- FCrDNS is three steps, and the forward step is what makes it proof
+- Verification confirms *and* refutes, and stays silent when DNS says nothing
+- The library ships no address data on purpose; refreshing is opt-in and fails open
+- Categories are the handle for business decisions about crawlers
+
+## Reference
+
+- [The signature database](../detection/signatures.md)
+- [Verifying a crawler](../detection/verification.md)
+- [Runtime changes](../operations/runtime-changes.md) — keeping ranges fresh
+
+Next: [Actors and behaviour](07-actors.md).

package/docs/course/07-actors.md

@@ -0,0 +1,178 @@
+# Lesson 7 — Actors and behaviour over time
+
+**Goal:** watch suspicion accumulate across requests, and understand why the identity you
+choose decides how good every behavioural signal is.
+
+← [Course](index.md) · Prev: [Identity and verification](06-identity.md) · Next: [Traps](08-traps.md)
+
+---
+
+## One request tells you less than six
+
+Everything so far judged a single request. Four detectors need more than that:
+`rate-anomaly`, `cadence`, `crawl-breadth` and `session-integrity` all read an **actor** —
+the library's word for "the same client, seen again".
+
+## Do this
+
+Forty requests from one address, exactly 250 ms apart — a metronome:
+
+```js
+import { BotHandler, createFacts } from "@osqd/bothandlerjs";
+
+const detector = new BotHandler();
+const UA = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36";
+
+for (let i = 0; i < 40; i++) {
+  const a = await detector.assess(
+    createFacts({
+      method: "GET",
+      url: `/books/${i}`,
+      ip: "203.0.113.91",
+      headers: { host: "serif.example", "user-agent": UA, accept: "text/html,application/xhtml+xml", "accept-language": "en-GB,en;q=0.9", "accept-encoding": "gzip, deflate, br" },
+      timestamp: 1_700_000_000_000 + i * 250,
+    }),
+  );
+
+  if ([9, 19, 39].includes(i)) {
+    console.log(`after ${i + 1}: score=${a.score} verdict=${a.verdict} requests=${a.actor.requests} paths=${a.actor.distinctPaths}`);
+    for (const e of a.evidence) console.log(`    [${e.certainty}] ${e.detector}: ${e.summary}`);
+  }
+}
+```
+
+Passing an explicit `timestamp` is what makes this reproducible — you are simulating time
+rather than waiting for it.
+
+### Checkpoint
+
+```
+after 10: score=45 verdict=unknown requests=10 paths=10
+    [moderate] cadence: Arrivals are machine-regular: 9 gaps averaging 250ms with a coefficient of variation of 0.000
+
+after 20: score=69 verdict=suspected-bot requests=20 paths=20
+    [moderate] cadence: Arrivals are machine-regular: 19 gaps averaging 250ms with a coefficient of variation of 0.000
+    [moderate] session-integrity: 20 requests from this actor, none carrying any cookie
+    [weak] rate-anomaly: 20 requests in 10s (2/s) from this actor
+
+after 40: score=80 verdict=suspected-bot requests=40 paths=40
+    [moderate] cadence: Arrivals are machine-regular: 31 gaps averaging 250ms with a coefficient of variation of 0.000
+    [moderate] rate-anomaly: 32 requests in 10s (3.2/s) from this actor
+    [moderate] session-integrity: 40 requests from this actor, none carrying any cookie
+    [weak] crawl-breadth: 40 distinct paths across 40 requests (100% never revisited)
+```
+
+**The headers never changed.** Every request looked like a perfectly ordinary browser on
+its own; the case was built entirely out of the relationship between them.
+
+Note `cadence` — a coefficient of variation of exactly 0.000. That is the signal that
+catches the *polite* scraper: one pacing itself deliberately under your rate limit is
+invisible to rate counting and obvious here, because people are irregular and loops are
+not.
+
+## The saturation you can see
+
+At 40 requests, `cadence` reports "31 gaps" and `rate-anomaly` "32 requests". Not a bug —
+the arrival ring holds **32 timestamps**, deliberately.
+
+Per-actor state is a fixed budget: 32 arrival timestamps, 64 path *hashes* (not strings),
+up to 4 User-Agents, and at most 20,000 actors in a bounded LRU. Every structure keyed by
+something a client controls has a ceiling, because an unbounded map keyed by IP is a remote
+OOM.
+
+The trade-off is stated rather than hidden: `requestsWithin` saturates at 32 and reports
+`undercounted: true` rather than a true rate. That series exists to *describe* an actor
+cheaply; exact counting belongs to the `rate-limit` action, which uses a store.
+
+## The actor key is the most valuable thing you can change
+
+By default an actor is the client address. An address is a poor identity: shared by a whole
+office, changed by a phone every few minutes, and behind CGNAT it is thousands of people.
+
+**Every behavioural detector is only as good as this function.**
+
+```js
+new BotHandler({
+  actorKey: (facts) => facts.session ?? facts.ip,
+});
+```
+
+A session id, an authenticated user id, or an address plus a TLS fingerprint all make the
+same detectors sharper — sharp enough that `identity-rotation` becomes worth enabling,
+which under an IP key would fire on every corporate NAT.
+
+This is also why the behavioural signals are *capped* where they are. Under an address key
+"one actor" may be a university, so nothing here may exceed `moderate` and none of it can
+deny anybody on its own. The [guard](04-the-guard.md) guarantees that structurally.
+
+## What an actor remembers
+
+```js
+console.log(a.actor);
+```
+
+| Field | |
+| ----- | - |
+| `key` | what it is tracked under |
+| `requests`, `distinctPaths` | inside the behavioural window |
+| `firstSeen`, `lastSeen`, `sinceLastMs` | |
+| `priorConfirmations` | assessments in the window that concluded `confirmed-bot` |
+| `unsolvedChallenges` | outstanding, not cumulative — solving one clears it |
+
+`unsolvedChallenges` is deliberately **not evidence**. One abandoned challenge is a person;
+what repeated abandonment means is a judgement about your traffic that only you can make,
+so it is exposed as something a *rule* can read. You will use it in
+[lesson 11](11-the-challenge.md).
+
+Idle actors are forgotten after `actorWindowMs` (default 15 minutes), and past `maxActors`
+the least recently seen is evicted — never one still sending traffic.
+
+## Forgetting one, by hand
+
+The support-ticket path. Somebody is being challenged, you have looked at their requests,
+and you are satisfied:
+
+```js
+detector.forgetActor("203.0.113.91", { by: "you@serif.example" });
+detector.clearActor("203.0.113.91", 60 * 60_000, { by: "you@serif.example" });
+```
+
+`clearActor` exempts them for a stated number of milliseconds, so the exemption expires on
+its own rather than becoming a permanent hole nobody remembers opening.
+
+## Exercise
+
+Re-run the forty requests with a *human* rhythm — random gaps between 400 ms and 6 s — and
+have each request revisit one of five paths rather than a new one each time.
+
+<details>
+<summary>What you should see, and what it means</summary>
+
+`cadence` stops firing, because the coefficient of variation is no longer near zero.
+`crawl-breadth` stops firing, because 5 distinct paths across 40 requests is reading rather
+than enumerating. `session-integrity` still fires — no cookie is still no cookie — and the
+score settles far below the threshold.
+
+That is the honest limit of behavioural detection, and it is the top of the evasion ladder
+in the [threat model](../concepts/threat-model.md): a scraper paced like a person, taking a
+few pages per address, is **not caught at all**. What defeats that is cost — a challenge,
+or an account — not detection.
+
+Knowing precisely where your detection stops is more useful than believing it does not.
+</details>
+
+## What you learned
+
+- Four detectors read an actor rather than a request, and build a case across time
+- `cadence` catches the polite scraper that rate counting cannot
+- Per-actor state is a fixed, bounded budget, and it saturates visibly rather than lying
+- `actorKey` is the highest-value thing you can replace
+- Behavioural signals are capped at `moderate` because an address is a poor identity
+
+## Reference
+
+- [Actors and behavioural memory](../concepts/actors.md)
+- [The detectors](../detection/detectors.md) — the behavioural four in detail
+- [Threat model](../concepts/threat-model.md) — where this stops working
+
+Next: [Traps](08-traps.md) — the one detector that needs no statistics at all.

package/docs/course/08-traps.md

@@ -0,0 +1,183 @@
+# Lesson 8 — Traps
+
+**Goal:** lay a honeypot properly — including the POST-body case that silently does
+nothing if you skip it.
+
+← [Course](index.md) · Prev: [Actors and behaviour](07-actors.md) · Next: [Rules](09-rules.md)
+
+---
+
+## Why a trap is proof
+
+Every other probabilistic detector asks *"does this look like automation?"*. A trap asks
+*"did you fetch a thing that no person can reach?"* — and that is detection by
+**construction** rather than by inference.
+
+A trap link is positioned off-screen, marked `aria-hidden` and `tabindex="-1"` so assistive
+technology skips it, `rel="nofollow noindex"`, and disallowed in `robots.txt`. There is no
+sequence of user input that reaches it. A client that fetched it either ignored every one
+of those signals or never rendered the page at all.
+
+That is why `trap` is one of the five things that earn `certain`, and it is the cheapest
+proof in the library — no statistics, no DNS, no history.
+
+## Do this
+
+```js
+import { BotHandler, createFacts, renderTrapLink, DEFAULT_TRAP_PATHS } from "@osqd/bothandlerjs";
+
+console.log(DEFAULT_TRAP_PATHS);
+console.log(renderTrapLink("/internal/export.csv"));
+
+const detector = new BotHandler();
+const caught = await detector.assess(
+  createFacts({
+    method: "GET",
+    url: "/internal/export.csv",
+    ip: "203.0.113.150",
+    headers: { host: "serif.example", "user-agent": CHROME["user-agent"], accept: "text/html" },
+  }),
+);
+console.log(`verdict=${caught.verdict} certain=${caught.certain} score=${caught.score}`);
+for (const e of caught.evidence) console.log(`  [${e.certainty}] ${e.detector}: ${e.summary}`);
+console.log("basis:", caught.evidence[0].deterministicBasis);
+```
+
+Notice the request carries a **complete, believable Chrome header set**. It is still
+proven, because the trap does not care what you claim to be — only where you went.
+
+## Putting one in a page
+
+```js
+app.get("/", (req, res) => {
+  res.send(`
+    <main>…your page…</main>
+    ${renderTrapLink("/internal/export.csv", { label: "Archive index" })}
+  `);
+});
+```
+
+And tell the well-behaved crawlers to stay away, so that only the ones ignoring
+`robots.txt` are ever caught:
+
+```js
+import { trapRobotsEntries } from "@osqd/bothandlerjs";
+console.log(trapRobotsEntries());
+// User-agent: *
+// Disallow: /internal/export.csv
+// Disallow: /api/v1/all-users
+// Disallow: /sitemap-index-full.xml
+```
+
+This is not a courtesy. **Publishing the disallow is what makes the trap proof.** Without
+it, Googlebot follows the link — it has no way to know it should not — and you have proven
+something false about a crawler you wanted.
+
+## The form-field trap, and the mistake everyone makes
+
+`renderTrapField` gives you a hidden input to drop into a form:
+
+```js
+import { renderTrapField } from "@osqd/bothandlerjs";
+res.send(`<form method="post" action="/signup">${renderTrapField("company_url")}…</form>`);
+```
+
+A bot filling every input it finds fills this one. A person never sees it.
+
+**But the engine reads no request body.** Doing so would consume the stream before your own
+parser saw it. So a hidden field on a `method="post"` form arrives somewhere this library
+cannot see — and the forms worth protecting are POSTs.
+
+Skip the next step and the field is rendered, filled by a bot, and **silently ignored**:
+
+```js
+import { BotHandler, TRAP_FIELD_SOURCE, defaultDetectors, trapDetector } from "@osqd/bothandlerjs";
+import { botHandler } from "@osqd/bothandlerjs/adapters";
+
+const detector = new BotHandler({
+  // Register the field name — the detector cannot recognise a honeypot it was never told about.
+  detectors: defaultDetectors().map((d) =>
+    d.id === "trap" ? trapDetector({ formFields: ["company_url"] }) : d,
+  ),
+});
+
+app.use(express.urlencoded({ extended: false }));   // your parser runs first
+app.use(
+  botHandler(detector, {
+    enrich: (request, facts) => ({ ...facts, extra: { [TRAP_FIELD_SOURCE]: request.body } }),
+  }),
+);
+```
+
+Two things have to line up: the detector must be **told the field name**, and the parsed
+body must be **handed over** with `enrich`. A field arriving in the query string is read
+without any of this.
+
+## Choosing paths
+
+The defaults are `/internal/export.csv`, `/api/v1/all-users` and
+`/sitemap-index-full.xml` — chosen to look like something worth fetching. Replace them with
+paths that fit your site:
+
+```js
+trapDetector({ paths: ["/admin/backup.sql", "/customers/export"] })
+```
+
+Two rules:
+
+**Do not use a path you might one day build.** A trap that becomes a real endpoint proves
+things about your own users.
+
+**Keep them out of public view.** A trap works because no person can reach it — publishing
+the path in a public repository or a client-side comment turns proof back into a guess.
+This is the one part of your configuration worth treating as a secret.
+
+## Exercise
+
+Serif has a signup form being hit by a registration bot. Lay a trap that catches it, and
+list everything that has to be true for it to work.
+
+<details>
+<summary>Answer</summary>
+
+```js
+// 1. Tell the detector the field name.
+const detector = new BotHandler({
+  detectors: defaultDetectors().map((d) =>
+    d.id === "trap" ? trapDetector({ formFields: ["serif_referral_code"] }) : d,
+  ),
+  rules: [{ id: "trap-block", match: { detector: "trap", certain: true }, action: "block" }],
+});
+
+// 2. Render it in the form.
+`<form method="post" action="/signup">${renderTrapField("serif_referral_code")}…</form>`
+
+// 3. Parse the body first, then hand it over.
+app.use(express.urlencoded({ extended: false }));
+app.use(botHandler(detector, {
+  enrich: (request, facts) => ({ ...facts, extra: { [TRAP_FIELD_SOURCE]: request.body } }),
+}));
+```
+
+Four things must all hold: the field is **rendered**, its name is **registered** with the
+detector, your parser runs **before** the middleware, and `enrich` **hands the body over**.
+Miss any one and the trap fails open silently — no error, no warning, just a honeypot that
+never catches anything.
+
+That silence is why this lesson exists.
+</details>
+
+## What you learned
+
+- A trap is proof by construction, and needs no statistics at all
+- Publishing the disallow in `robots.txt` is what makes it proof rather than a mistake
+- A POST field trap needs the name registered *and* the body handed over with `enrich`
+- Trap paths are the one part of your config to keep quiet about
+
+## Reference
+
+- [The detectors](../detection/detectors.md) — `trap` in full
+- [Adapters](../integration/adapters.md) — `enrich`
+- [robots.txt](../policy/robots.md)
+
+Next: [Rules](09-rules.md) — writing the policy.