@osqd/bothandlerjs 0.3.0

package/docs/course/12-going-live.md

@@ -0,0 +1,211 @@
+# Lesson 12 — Going live
+
+**Goal:** mount Serif behind a real framework, and get the one setting right that is
+dangerous to get wrong.
+
+← [Course](index.md) · Prev: [The challenge](11-the-challenge.md) · Next: [Operating it](13-operating-it.md)
+
+---
+
+## Three lines
+
+```js
+import { BotHandler } from "@osqd/bothandlerjs";
+import { botHandler } from "@osqd/bothandlerjs/adapters";
+
+const detector = new BotHandler({ preset: "monitor-only" });
+app.use(botHandler(detector));
+```
+
+Four adapters ship:
+
+```js
+import { botHandler, fastifyBotHandler, koaBotHandler, withBotHandler } from "@osqd/bothandlerjs/adapters";
+
+app.use(botHandler(detector));                                  // Express / Connect / node:http
+fastify.addHook("onRequest", fastifyBotHandler(detector));
+app.use(koaBotHandler(detector));
+export default { fetch: withBotHandler(detector, myHandler) };  // Workers / Deno / Bun / Edge
+```
+
+**Hono and Next.js need no adapter of their own.** Both speak the platform's `Request` and
+`Response`, which is what `createFetchAdapter` takes and returns:
+
+```js
+const guard = createFetchAdapter(detector, { ipHeaders: ["cf-connecting-ip"] });
+
+// Hono
+app.use(async (c, next) => {
+  const decision = await guard(c.req.raw, c.env);
+  if (decision.response) return decision.response;
+  await next();
+});
+
+// Next.js — middleware.ts
+export async function middleware(request) {
+  const decision = await guard(request);
+  if (decision.response) return decision.response;
+  return NextResponse.next({ request: { headers: decision.request.headers } });
+}
+```
+
+Passing `decision.request.headers` on is the part worth doing: it carries the verdict to
+routes further in, so they need not assess anything themselves.
+
+## Why the engine never touches a response
+
+`handle` returns an `ActionOutcome` — `continue`, `respond`, or `drop` — and the adapter
+applies it. That is why the same policy behaves identically on Express and on a Worker, why
+`assess` is safe to run over a log file, and why writing your own adapter is about thirty
+lines.
+
+It also **fails open**. An unexpected failure inside detection serves the request; the error
+goes to your `onError` and the visitor gets their page. A bot filter that fails closed is an
+outage with extra steps.
+
+## Now the important part
+
+### The client IP
+
+Stop here if Serif sits behind anything — a load balancer, a CDN, nginx, a service mesh.
+
+The client address becomes the [actor key](07-actors.md). Every rate limit, every allowlist
+entry, every behavioural signal depends on it.
+
+`X-Forwarded-For` is a **client-supplied header**. Trust it without knowing how many proxies
+sit in front of you and anyone can prepend a fake hop and choose the address you rate-limit,
+allowlist and block on. **The failure is silent** — nothing errors, nothing logs, and every
+per-actor mechanism becomes an attacker input.
+
+There is deliberately no convenient default.
+
+```js
+proxy: { trustProxy: true, trustedProxies: ["10.0.0.0/8"] }   // recommended
+proxy: { trustProxy: true, hops: 2 }                          // correct only while the count is
+proxy: { trustProxy: false }                                  // default — header ignored entirely
+```
+
+**Use `trustedProxies`.** The chain is walked from the right, discarding your own
+infrastructure, and the first address outside it is the client — robust against an extra hop
+appearing when somebody adds a CDN and forgets to tell you.
+
+`hops` is right until the topology changes, and then it is wrong in the direction that lets
+clients choose their own address.
+
+Three details worth knowing:
+
+**Addresses are compared as bytes.** `::ffff:127.0.0.1`, `0177.0.0.1` and `127.0.0.001` are
+the same address, and all slip past an allowlist that compares strings. Invalid CIDRs throw
+at construction rather than matching nothing silently.
+
+**The connecting peer is checked too.** On a server reachable both through the load balancer
+and directly, a request arriving from outside your trusted ranges did not come through your
+proxies — so its forwarded header is not evidence, and the socket address is used. Without
+that check, anyone who finds the origin address picks their own client IP.
+
+**Do not allowlist loopback.** The moment you sit behind nginx or beside a sidecar, every
+request in the world arrives from `127.0.0.1`.
+
+### On Fetch runtimes
+
+There is no socket, so the address comes from a header. Only `cf-connecting-ip` and
+`x-real-ip` are trusted by default — both single-valued, written by the edge that terminated
+the connection. `x-forwarded-for` is **not**, because a proxy *appends* to it.
+
+Better still, ask the platform:
+
+```js
+withBotHandler(detector, myHandler, {
+  clientIp: (request, env) => env.cf?.connectingIp,
+});
+```
+
+If no address can be found, every visitor is tracked under one empty actor key — which
+applies your rate limits to the whole site at once. The adapter warns the first time.
+
+On Workers, enable `nodejs_compat`; the engine uses `node:crypto`.
+
+## Tell it about your own people
+
+The only conclusive human signal in this library is your assertion:
+
+```js
+new BotHandler({
+  isHuman: (facts) => Boolean(sessions.get(facts.session ?? "")?.authenticated),
+});
+```
+
+That produces `certain` human evidence, which is what the `cleared-human-allow` rule at the
+top of most presets is for. Without it that rule never fires, and your signed-in customers
+are judged like strangers.
+
+## Serif, mounted
+
+```js
+import express from "express";
+import { BotHandler } from "@osqd/bothandlerjs";
+import { botHandler } from "@osqd/bothandlerjs/adapters";
+
+const site = new BotHandler({
+  preset: "monitor-only",                     // week one. Enforce later.
+  proxy: { trustProxy: true, trustedProxies: ["10.0.0.0/8"] },
+  allowlist: ["10.0.0.0/8"],                  // monitors and CI
+  ignorePaths: ["/healthz", "/metrics"],
+  isHuman: (facts) => Boolean(sessions.get(facts.session ?? "")?.authenticated),
+  challenge: { secrets: [process.env.SERIF_CHALLENGE_SECRET], contactHtml: "<p>…</p>" },
+  onWarning: (message) => console.warn("[bot]", message),
+  onDowngrade: ({ decision }) => console.warn(`[bot] guard stopped rule ${decision.rule}`),
+});
+
+const auth = new BotHandler({ preset: "protect-auth", proxy: { trustProxy: true, trustedProxies: ["10.0.0.0/8"] } });
+
+const app = express();
+app.use(botHandler(site));
+app.use("/login", botHandler(auth));
+app.use("/checkout", botHandler(auth));
+```
+
+Note `allowlist` and `ignorePaths` are different things. An **ignored path** is not
+assessed. An **allowlisted address** is not judged *at all* — not judged leniently, not
+judged. It is the strongest setting in the library, and the one most worth alerting on when
+it changes.
+
+## Exercise
+
+Serif runs on Cloudflare in front of a Node origin. A colleague suggests
+`proxy: { trustProxy: true, hops: 1 }`. What is wrong with it, and what breaks?
+
+<details>
+<summary>Answer</summary>
+
+`hops: 1` takes the **last** entry in `X-Forwarded-For`, which is what Cloudflare appended —
+so today it is right. It breaks the moment anything is added in front: another CDN, a
+regional load balancer, a WAF. Then the last entry is that hop's, and Cloudflare's entry —
+the real client — is one further left.
+
+Worse, the failure is silent and the wrong direction: the address you rate-limit becomes one
+your own infrastructure controls, so all traffic collapses onto one actor.
+
+```js
+proxy: { trustProxy: true, trustedProxies: [...cloudflareRanges, "10.0.0.0/8"] }
+```
+
+Walking from the right and discarding known infrastructure survives an extra hop appearing.
+And on the Cloudflare edge itself, prefer `clientIp: (req, env) => env.cf?.connectingIp` —
+ask the platform, not a header.
+</details>
+
+## What you learned
+
+- Four adapters, plus Hono and Next.js for free; the engine never touches a response
+- It fails open, on purpose
+- `trustedProxies` over `hops`, always; the failure mode of getting it wrong is silent
+- `isHuman` is the only conclusive human signal that exists
+- Allowlisting stops detection entirely — it is not leniency
+
+## Reference
+
+- [Adapters](../integration/adapters.md) · [The client IP](../integration/client-ip.md)
+- [Configuration](../reference/configuration.md)
+
+Next: [Operating it](13-operating-it.md).

package/docs/course/13-operating-it.md

@@ -0,0 +1,238 @@
+# Lesson 13 — Operating it
+
+**Goal:** see what the policy is doing to real traffic, and know which two numbers to alert
+on.
+
+← [Course](index.md) · Prev: [Going live](12-going-live.md) · Next: [Scaling](14-scaling.md)
+
+---
+
+## The dashboard
+
+Counters tell you *how much*. The dashboard tells you **which requests, and why** — every
+assessment as it lands, and on any row you open, the individual evidence with its certainty
+tier and, for proven ones, the written basis.
+
+```js
+const dashboard = await detector.serveDashboard({
+  port: 9674,
+  title: "serif",
+  auth: { username: "ops", password: process.env.DASHBOARD_PASSWORD },
+});
+console.log(dashboard.url);
+```
+
+That is the whole integration: it listens on a port of its own, subscribes to the handler
+you called it on, and returns a handle with a `close()`.
+
+Four screens — the live feed, actors, statistics, and the policy editor. Open a row and you
+see the evidence that produced the verdict; that is the view that answers "why was this
+customer challenged?".
+
+### Before it goes anywhere real
+
+**It reports on one process.** Behind a load balancer with eight pods you are looking at an
+eighth of your traffic. The page names the instance, which is the honest amount of help it
+can give.
+
+**Its feed is memory-only** and bounded. It is a live view, not a log.
+
+**Everything it can *do* is behind a flag**, and each flag is separate:
+
+```js
+controls: { reset: true, editPolicy: true, editGuard: true, editRanges: true }
+```
+
+`editGuard` is separate from `editPolicy` deliberately: a rule that overreaches is stopped
+by the guard, and this changes whether anything stops it. They read like similar powers and
+are not.
+
+**What it *shows* is separate again** — `sections` — and enforced on the server, so a
+switched-off section's endpoint answers 403 and its fields never leave the process:
+
+```js
+// What an analyst gets: the shape of the traffic, without naming individuals.
+await detector.serveDashboard({
+  port: 9684,
+  sections: { evidence: false, policy: false },
+  redact: { maskIp: true },
+});
+```
+
+That combination is worth understanding. The dashboard people watch all day need not be the
+one that names individuals, or the one that explains your detection to whoever is scraping
+you.
+
+## Metrics
+
+On by default; a handful of integer increments per request.
+
+```js
+app.get("/internal/metrics", (_req, res) => res.type("text/plain").send(detector.prometheus()));
+```
+
+**Serve it where only you can reach it.** The detector-firing series describe how your
+detection behaves, which is exactly what somebody tuning a scraper would like to read.
+
+### The two to alert on
+
+**`bothandler_downgrades_total`** — rules that asked to deny and were refused for lack of
+proof. You met it in [lesson 4](04-the-guard.md). A rising count means your policy is asking
+for something the evidence does not support.
+
+**`bothandler_verdicts_total{verdict="unknown"}`** — ordinary traffic. If this falls, either
+your traffic changed or your detection did, and you want to know which.
+
+### The one to read before moving a threshold
+
+**`bothandler_score_bucket`** — how suspicion is distributed across everything scored, in
+ten buckets of ten points. Proven assessments are counted separately by
+`bothandler_proven_total`; their score is 100 by definition and would put a meaningless
+spike at the top.
+
+Moving `suspectThreshold` or a rule's `minScore` without looking at this is guessing. The
+histogram tells you how many requests sit in the ten points you are about to cross.
+
+### Per-detector timing is available and off
+
+Timing each detector costs two clock reads per detector per request — forty on a
+twenty-detector set, to measure work counted in microseconds. Worth paying while you tune:
+
+```js
+new BotHandler({ metrics: { perDetectorTiming: true } });
+```
+
+It is how you find out that one `io` detector costs more than the other nineteen together.
+
+## The traffic audit
+
+Counters cannot tell you a number is *unusual*, and bot traffic is not a level — it is an
+event. The number that matters is not "12% of requests are bots" but "12% now, 2% for the
+hour before".
+
+```js
+new BotHandler({
+  audit: {
+    windowMs: 5 * 60_000,
+    baselineMs: 60 * 60_000,
+    minSamples: 50,
+    cooldownMs: 15 * 60_000,
+  },
+  onAnomaly: (anomaly) => pager.send(anomaly.severity, anomaly.summary),
+});
+```
+
+**The baseline ends where the window begins.** A baseline containing the window would be
+partly made of the thing being measured, and a large enough spike would raise its own bar
+until it stopped looking like one.
+
+| Check | Fires when |
+| ----- | ---------- |
+| `bot-share-spike` | automation is a much larger share than the baseline |
+| `traffic-spike` | volume far above the baseline rate |
+| `denial-spike` | a much larger share is being denied — **read these before assuming they are all bots** |
+| `guard-stop-spike` | the guard is refusing far more rules than usual |
+| `human-share-drop` | traffic reading as human has fallen away |
+| `detector-failures` | detectors erroring or timing out |
+| `challenge-solve-rate` | nearly everything challenged is passing — **high is the bad direction** |
+
+That last one is the closest to this library's thesis. Every other check asks whether the
+*traffic* changed shape; this asks whether the mitigation is landing on the right
+population. A proof of work is trivial for a browser and trivial for a competent scraper —
+what it actually costs is a few seconds of somebody's afternoon. So when nearly everything
+challenged goes on to pass, the challenges are not filtering bots out, they are **taxing
+people**. From every other angle a solved challenge looks like a challenge that worked,
+which is why it needs saying out loud.
+
+Every check has a **floor** as well as a ratio — a quiet site at 3am produces "800% more
+bots" from four requests, and an alerting system that cries wolf at 3am gets muted — and a
+**cooldown**, because a spike lasting an hour is one event, not sixty.
+
+## Events
+
+Everything the engine concludes is available as a callback or a subscription:
+
+```js
+const stop = detector.on("denial", ({ assessment, decision }) =>
+  log.info({ actor: assessment.actor.key, rule: decision.rule }),
+);
+```
+
+| Event | Fires when |
+| ----- | ---------- |
+| `assessment` | every assessment. The firehose |
+| `decision` | every decision |
+| `denial` | a request was actually refused |
+| `downgrade` | the guard replaced a terminal action |
+| `challenge` | one was issued, solved or rejected |
+| `detector-failure` | a detector threw or timed out |
+| `policy-change` / `guard-change` / `range-change` / `actor-change` | somebody changed something at runtime |
+| `anomaly` | the audit noticed a change of shape |
+| `warning` / `error` | misconfiguration; a component failed |
+
+**None of them can hurt a request.** A handler that throws is caught and reported once
+through `onError`, the rest still run, and none is awaited — returning a promise is fine and
+the response never waits for your webhook.
+
+## Notifications
+
+```js
+notifications: {
+  sinks: [consoleNotifier(), slackNotifier({ url }), webhookNotifier({ url, secret })],
+  filter: { types: ["denial", "downgrade", "anomaly"], minScore: 70 },
+  redaction: { maskIp: true },
+  dedupeWindowMs: 60_000,
+  maxPerWindow: 200,
+}
+```
+
+Two properties decide whether alerting is an asset or a liability under bot load. **It never
+blocks a request** — a wedged webhook slows nothing. **It has a ceiling** — repeats collapse,
+a global cap catches distributed traffic where every event is genuinely distinct, and the
+suppressed count is reported when the window rolls, so a quiet channel is never mistaken for
+quiet traffic.
+
+Addresses are masked to a `/24` or `/64` on the way *out*, so detection still sees
+everything.
+
+## Exercise
+
+Serif has been live on `monitor-only` for a week. What do you look at, in what order, to
+decide whether to enforce?
+
+<details>
+<summary>Answer</summary>
+
+1. **`bothandler_verdicts_total`** — what is the actual mix? If `unknown` is not the large
+   majority, something is misconfigured, most likely the client IP.
+2. **The dashboard's live feed, filtered to anything not `unknown`.** Open rows and read the
+   evidence. You are looking for the integration you forgot: a partner's nightly sync, a
+   status prober, the marketing team's link checker, your own renderer.
+3. **`bothandler_score_bucket`** — where does suspicion actually sit? That tells you what
+   `minScore: 70` would catch on *your* traffic rather than in general.
+4. **`bothandler_downgrades_total`** — on `monitor-only` this should be zero, because
+   nothing asks for a terminal action. If it is not, your rules are not what you think.
+5. **`npx @osqd/bothandlerjs replay` over the week's access log** with the policy you are
+   considering, and read every request it would have refused.
+
+Only then enforce — and allowlist what step 2 found *first*.
+
+Step 5 is [lesson 16](16-proving-it.md), and it is the one that catches what the others
+miss.
+</details>
+
+## What you learned
+
+- The dashboard shows which requests and why; it reports on one process and is bounded
+- `sections` and `controls` are separate powers, enforced server-side
+- Alert on `downgrades_total` and on `unknown`; read `score_bucket` before moving a threshold
+- The audit compares a window against the baseline before it, with floors and cooldowns
+- A high challenge solve rate is the bad direction
+- Nothing in the observation path can slow or break a request
+
+## Reference
+
+- [The dashboard](../operations/dashboard.md) · [Metrics](../operations/metrics.md)
+- [The audit](../operations/audit.md) · [Notifications](../operations/notifications.md)
+
+Next: [Scaling and changing it live](14-scaling.md).

package/docs/course/14-scaling.md

@@ -0,0 +1,195 @@
+# Lesson 14 — Scaling, and changing it live
+
+**Goal:** run Serif on more than one process without breaking your rate limits, and change
+a rule during an incident without a deploy.
+
+← [Course](index.md) · Prev: [Operating it](13-operating-it.md) · Next: [Extending it](15-extending.md)
+
+---
+
+## What breaks at two replicas
+
+Two things, and both are correctness rather than optimisation.
+
+**Single-use challenge nonces.** Without sharing, a scraper retries a solved nonce against
+another replica until one has not seen it.
+
+**Rate limits.** A limit of 100/minute enforced independently by four replicas is a limit of
+400/minute.
+
+```js
+import { RedisStore } from "@osqd/bothandlerjs";
+new BotHandler({ store: new RedisStore(redis) });
+```
+
+`RedisStore` takes a client you already have. The package stays dependency-free — it uses
+the handful of methods `ioredis` and `node-redis` share, and imports neither.
+
+## What deliberately does not scale, and why
+
+Behavioural state — arrival rates, cadence, path breadth, User-Agent history — stays in
+memory per process, on purpose.
+
+A round trip per request would buy accuracy for signals that are only ever allowed to
+**raise suspicion**, never to deny anybody: the [guard](04-the-guard.md) sees to that.
+Behind four replicas each sees a quarter of an actor's traffic and is correspondingly less
+sure, which is the right trade for something that cannot close a door on its own.
+
+Spending a network round trip on the request path to sharpen a signal that cannot act alone
+is a bad bargain, and the request path is where this library refuses to make bad bargains.
+
+## Except proof, which travels
+
+A `confirmed-bot` verdict is *proven* — something declared itself, forged an identity, or
+walked into a trap. That is a fact about the client rather than a judgement about it:
+
+```js
+new BotHandler({ store: new RedisStore(redis), shareConfirmations: true });
+```
+
+Without it, a client proven to be a bot on one replica is a stranger to the other seven, and
+a rule reading `minPriorConfirmations: 1` fires about an eighth as often as it reads.
+
+**Proof travels; suspicion stays home.** That sentence is the whole design.
+
+The cost is one store read the **first time each instance sees an actor** — not one per
+request — and it is never awaited, so nothing joins the request path. A store outage means
+the count falls back to what that process saw itself.
+
+## Changing things without a deploy
+
+Two things go stale between deploys, and both hurt.
+
+**Published crawler ranges.** A stale list turns a verified crawler into an accused
+impersonator.
+
+**A rule set.** Sometimes a rule is wrong in a way you want to fix *now* — which is exactly
+when the deploy pipeline feels longest.
+
+### Rules
+
+```js
+detector.updatePolicy(
+  [...detector.policy.rules, { id: "allow-healthz", match: { path: "/healthz" }, action: "allow" }],
+  { by: "you@serif.example" },
+);
+```
+
+Validated first, swapped **atomically**, announced through `onWarning` and `policy-change`.
+Invalid input throws and leaves the previous set standing.
+
+**What it cannot change:** `falsePositivePolicy`, `fallbackAction` and
+`terminalScoreThreshold`. No runtime edit can relax the guard through this door.
+
+### The guard
+
+Its own method, its own event, its own dashboard flag:
+
+```js
+detector.updateGuard({ falsePositivePolicy: "balanced" }, { by: "you@serif.example" });
+// warning: Guard settings changed at runtime by you@serif.example:
+//          falsePositivePolicy strict → balanced.
+```
+
+`onGuardChange` carries the before and the after. **If you page on one thing in this
+lesson, page on this.**
+
+### Ranges and actors
+
+```js
+detector.updateCrawlerRanges("gptbot", await fetchRanges());
+detector.updateRanges("allowlist", [...current, "198.51.100.0/24"]);
+detector.listRanges();                      // [{ name: "crawler:gptbot", size: 42 }, …]
+
+detector.forgetActor(key, { by: "you@serif.example" });
+detector.clearActor(key, 60 * 60_000, { by: "you@serif.example" });
+```
+
+`range-change` is worth alerting on: the allowlist is the one list that stops detection
+*running*.
+
+### `by`, and the audit trail
+
+Every mutating method takes `{ by }`. The dashboard fills it from its own `auth` — a basic
+credential names itself, and a custom `authorize` can return an identity instead of `true`.
+
+The library has no user model and does not want one. It carries the name it was given into
+the warning and the event, so your trail can say *who* rather than only *what*:
+
+```js
+detector.on("policy-change", ({ by, rules }) => auditLog.write({ who: by, count: rules.length }));
+```
+
+Nothing verifies the name. It is exactly as trustworthy as whatever supplied it — which is
+your authentication, the same thing that decides whether the change is allowed at all.
+
+## The incident runbook
+
+Put this somewhere you can find it at 3am.
+
+```js
+// 1. See it.            dashboard → live feed, filter to denials
+// 2. Slow it, uniformly.
+detector.updatePolicy([
+  { id: "incident-ceiling", match: {}, action: "rate-limit",
+    params: { limit: { max: 30, windowMs: 60_000 } } },
+  ...detector.policy.rules,
+], { by: "you@serif.example" });
+
+// 3. Only if that is not enough — and read the preset's warning first.
+//    It refuses proven automation, so allowlist your webhooks BEFORE switching over.
+```
+
+A uniform ceiling is the one mitigation that cannot be wrong about who somebody is. Reach
+for it before you reach for anything that singles clients out.
+
+And note what is *not* in the runbook: `falsePositivePolicy: "aggressive"`. An incident is
+precisely when people reach for it, and precisely when the population getting caught is at
+its most unusual.
+
+## Exercise
+
+Serif runs eight pods behind a load balancer. A rule reads
+`{ minPriorConfirmations: 1, action: "block" }`. It fires far less than expected. Why, and
+what are the two fixes?
+
+<details>
+<summary>Answer</summary>
+
+Each pod only knows about the confirmations *it* saw. A client proven on pod 3 is a stranger
+to the other seven, so with traffic spread evenly the rule fires roughly one time in eight.
+
+**Fix one — share the proof:**
+
+```js
+new BotHandler({ store: new RedisStore(redis), shareConfirmations: true });
+```
+
+This is what that flag is for. Proof is a fact about the client, so it can cross replicas;
+suspicion stays home.
+
+**Fix two — give the actors a better key.** If the load balancer has session affinity, or if
+you key on a session id rather than an address, the same client lands where its history is:
+
+```js
+actorKey: (facts) => facts.session ?? facts.ip
+```
+
+Both are worth doing, and the second helps every behavioural detector as well — see
+[lesson 7](07-actors.md).
+</details>
+
+## What you learned
+
+- Nonces and rate limits need a shared store; that is correctness, not tuning
+- Behavioural state stays process-local because it can never deny anybody alone
+- `shareConfirmations` lets *proof* cross replicas, unawaited and once per actor
+- `updatePolicy` cannot relax the guard; `updateGuard` is a separate, alertable power
+- A uniform ceiling is the safest first move in an incident
+
+## Reference
+
+- [Stores](../integration/stores.md) · [Runtime changes](../operations/runtime-changes.md)
+- [Presets](../policy/presets.md#under-attack)
+
+Next: [Extending it](15-extending.md).