@osqd/bothandlerjs 0.3.0

package/docs/detection/index.md

@@ -0,0 +1,134 @@
+# How detection works
+
+The pipeline a request goes through, what it costs, and what comes out the other end.
+
+← [Documentation](../index.md)
+
+---
+
+```
+facts → bypass checks → cheap detectors ─┐
+                        io detectors ────┼→ combine → assessment → policy → decision
+                        confirming ──────┘
+```
+
+Three calls, and you can stop after any of them:
+
+```ts
+const facts = createFacts({ method, url, headers, ip });   // normalise
+const assessment = await detector.assess(facts);           // what is this?
+const decision = detector.decide(assessment);              // what do we do?
+const { outcome } = await detector.handle(facts);          // all three, plus the action
+```
+
+`assess` never decides and `decide` never inspects. That separation is what makes the
+[replay](../testing/replay.md), the [corpus](../testing/corpus.md) and the dashboard's
+policy preview possible: `decide` is **pure**, so a candidate policy can be run over
+recorded assessments as many times as you like and nothing about the running system moves.
+
+## Before any detector runs
+
+Two checks short-circuit everything:
+
+| | |
+| --- | --- |
+| `ignorePaths` | Health checks, static assets, your own instrumentation. |
+| `allowlist` | Addresses that are **not judged at all** — see [design decisions](../design/decisions.md). |
+
+Either produces an assessment with `bypass` set and no evidence. It is counted (so the
+dashboard can say how much traffic detection actually ran on) and nothing else happens.
+
+## The three stages
+
+**Cheap detectors** are synchronous by contract and run *without* an `await`. That is not
+micro-optimisation: `await` on a non-promise still yields a microtask turn, so awaiting
+each of a dozen detectors put a dozen scheduler round-trips on every request to your site
+— which dominated the cost of a clean browser request. A detector that returns a promise
+anyway is collected and awaited with the rest, so the contract is enforced by behaviour
+rather than by trust.
+
+**`io` detectors** run concurrently under `detectorTimeoutMs` (default 300 ms). Mislabelling
+one as `cheap` would put an unbounded await on the request path, so the engine times out
+any promise a `cheap` detector returns as well.
+
+**Confirming detectors** run only when a signature matched. With no claimed identity there
+is nothing to confirm and no lookup to make, which is why a normal browser request never
+touches DNS.
+
+## Failure is not the visitor's problem
+
+A detector that throws or times out is recorded as a `DetectorFailure`, reported through
+`onDetectorFailure`, and the assessment continues with the evidence it has. A resolver
+being down degrades detection; it must never take down the site the detection protects.
+
+```ts
+assessment.failures;   // [{ detector, reason: "timeout" | "error", message }]
+```
+
+The same rule holds one level up: an unexpected failure inside an
+[adapter](../integration/adapters.md) serves the request. A bot filter that fails closed
+is an outage with extra steps.
+
+## What comes out
+
+```ts
+interface Assessment {
+  requestId: string;        // random per request, safe to log and echo
+  verdict: Verdict;         // confirmed-bot | verified-bot | suspected-bot | human | unknown
+  botClass: BotClass;
+  identity?: string;        // "googlebot", when something named itself
+  score: number;            // 0–99, or 100 for proven
+  confidence: number;       // 0–1, unrounded
+  certain: boolean;         // ← the field that gates terminal actions
+  evidence: Evidence[];     // bot-pointing, strongest first
+  humanEvidence: Evidence[];
+  actor: ActorSnapshot;     // history at the time of this request
+  durationMs: number;
+  failures: DetectorFailure[];
+  facts: RequestFacts;
+  bypass?: "allowlist" | "ignored-path";
+}
+```
+
+See [verdicts, classes and scores](../concepts/verdicts.md) for which field to read when,
+and [evidence and certainty](../concepts/evidence.md) for how the evidence became a score.
+
+## Asking about a request that is not happening
+
+```ts
+const assessment = await detector.assess(facts, { record: false });
+```
+
+A **dry run**. Every detector runs and the verdict is real — and nothing is written down:
+no counter moves, no actor state changes, no `assessment` event fires, no notification is
+sent. Asking what the engine thinks of a request does not become part of the answer to
+"what is my traffic doing?".
+
+Reach for it wherever you want an opinion about a request nobody made: a support ticket
+("why is this customer being challenged?"), a rule you are drafting, a test. It is what
+`bothandlerjs explain` and the dashboard's request tester run on.
+
+The one thing it cannot see is history. It gets an actor with no past, so `cadence`,
+`crawl-breadth` and `rate-anomaly` have nothing to read. What it answers precisely is
+*what would this look like as a first request* — which is what a support ticket is asking
+anyway.
+
+## What it costs
+
+Measured on the clean-browser path, which is the overwhelmingly common case and the one
+where nothing short-circuits because nothing fires:
+
+```
+assess — clean browser     ~9 us     ~104,000 ops/s
+handle — clean browser    ~12 us      ~83,000 ops/s
+```
+
+`npm run bench` reproduces it; `npm run bench:guard` is the CI ratchet that stops it
+quietly getting worse. Turn on `metrics.perDetectorTiming` while tuning to see where the
+time goes, and turn it off afterwards — it is two clock reads per detector per request.
+
+## Related
+
+- [The detectors](detectors.md) — all twenty in detail
+- [Writing a detector](writing-a-detector.md)
+- [Policy](../policy/index.md) — what happens to an assessment next

package/docs/detection/signatures.md

@@ -0,0 +1,111 @@
+# The signature database
+
+How a client is recognised by name, and what a name is worth.
+
+← [Documentation](../index.md) · [The detectors](detectors.md)
+
+---
+
+161 signatures, 389 tokens, matched in a single Aho–Corasick pass over the lower-cased
+User-Agent. One pass regardless of how many signatures there are, which is why adding to
+this costs nothing measurable.
+
+```ts
+interface BotSignature {
+  id: string;              // stable; appears in rules, logs and metrics
+  name: string;            // "Googlebot"
+  tokens: readonly string[];   // lower-case literals; any one identifies it
+  category: BotCategory;
+  benign: boolean;
+  robotsAgent?: string;    // the name to write in robots.txt
+  verification: Verification;
+  docs?: string;           // the operator's own documentation
+}
+```
+
+## Categories
+
+`category` is what most policies actually match on, because it carries intent in a way an
+individual name does not.
+
+| Category | Examples | Typical policy |
+| -------- | -------- | -------------- |
+| `search` | Googlebot, Bingbot, DuckDuckBot, Yandex, Baidu, Seznam, Naver | allow |
+| `ai` | GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Amazonbot | a business decision |
+| `seo` | AhrefsBot, Semrush, Majestic, Moz | usually rate-limit |
+| `social` | facebookexternalhit, Twitterbot, Slackbot, Discord, Bluesky | allow — these are people sharing links |
+| `monitoring` | UptimeRobot, Pingdom, Checkly, Better Uptime | allow |
+| `archive` | ia_archiver, Common Crawl | your call |
+| `feed` | Feedly, podcast clients, RSS readers | allow |
+| `scanner` | sqlmap, Nikto, Nuclei, masscan | block |
+| `library` | curl, wget, python-requests, Go-http-client, okhttp | usually challenge |
+| `headless` | HeadlessChrome, Playwright, Puppeteer, Selenium | usually challenge |
+| `embedded` | Smart TVs, set-top boxes, game consoles | allow |
+
+```ts
+{ id: "no-ai", match: { category: "ai" }, action: "block", reason: "Not for model training." }
+```
+
+## Verification
+
+What, if anything, can check the claim:
+
+```ts
+type Verification =
+  | { kind: "fcrdns"; domains: readonly string[] }   // reverse DNS, forward-confirmed
+  | { kind: "ip-ranges"; publishedAt?: string }      // an address list the operator publishes
+  | { kind: "none" };                                // no published mechanism
+```
+
+`none` is honest rather than lazy: a great many crawlers publish nothing that can confirm
+them, and for those the claim is **unfalsifiable**. The library neither confirms nor
+accuses — it records what the client said and lets the policy decide what a self-declared
+identity is worth. See [verifying a crawler](verification.md).
+
+## What a name is worth
+
+A matched signature makes `self-identified` produce `certain` evidence, and this is the
+part worth being precise about: **the certainty is about the declaration, not about the
+identity**.
+
+`Googlebot/2.1` in a User-Agent proves that something *claimed to be Googlebot*. It is
+`confirmed-bot` — automation, certainly, because no person's browser sends that string —
+and it becomes `verified-bot` only once `crawler-verification` confirms it. A forgery
+that is refuted becomes an `impersonator`, which is the strongest thing the library ever
+concludes about anybody.
+
+So a rule matching `identity: ["googlebot"]` alone matches forgeries too. Match on
+proof when it matters:
+
+```ts
+{ id: "trust-google", match: { identity: ["googlebot"], verdict: "verified-bot" }, action: "allow" }
+```
+
+## Adding your own
+
+```ts
+new BotHandler({
+  extraSignatures: [
+    {
+      id: "acme-partner",
+      name: "Acme partner integration",
+      tokens: ["acme-partner-sync"],
+      category: "library",
+      benign: true,
+      verification: { kind: "ip-ranges" },
+    },
+  ],
+});
+
+// and, if you know where they call from:
+detector.updateCrawlerRanges("acme-partner", ["198.51.100.0/24"]);
+```
+
+`signatures` replaces the shipped set entirely; `extraSignatures` adds to it. Tokens are
+lower-case literals rather than patterns, which is what keeps the match linear.
+
+## Related
+
+- [Verifying a crawler](verification.md) — turning a claim into a verdict
+- [robots.txt](../policy/robots.md) — `robotsAgent` is what makes this generatable
+- [The detectors](detectors.md#self-identified)

package/docs/detection/verification.md

@@ -0,0 +1,123 @@
+# Verifying a crawler
+
+Turning "I am Googlebot" into a verdict — or into a refutation.
+
+← [Documentation](../index.md) · [The detectors](detectors.md)
+
+---
+
+A User-Agent is a claim. Two mechanisms can check one, depending on what the operator
+publishes, and both are run by
+[`crawler-verification`](detectors.md#crawler-verification) — which only runs at all when
+a [signature](signatures.md) matched, because with no claim there is nothing to check.
+
+## Forward-confirmed reverse DNS
+
+The mechanism Google, Bing, Yandex, Baidu and Apple document for their own crawlers:
+
+1. `PTR` the client address → `crawl-66-249-66-1.googlebot.com`
+2. Check the name ends in a domain the signature lists.
+3. Resolve that name forward → does it come back to the same address?
+
+Step 3 is the one that matters. Reverse DNS alone is controlled by whoever owns the
+address block; forward-confirming it means the *crawler's* DNS has to agree.
+
+```ts
+{ verification: { kind: "fcrdns", domains: ["googlebot.com", "google.com"] } }
+```
+
+| Outcome | Evidence |
+| ------- | -------- |
+| Forward-confirmed under a listed domain | `certain` → `verified-bot` |
+| Resolves, but to a name outside the domains | `certain` refutation → `impersonator` |
+| No answer, timeout, resolver error | **nothing** |
+
+That last row is the important one. A resolver having a bad afternoon must not look like
+an accusation, so an unhappy lookup produces no evidence at all — not weak evidence, none.
+
+DNS results are cached (`cachingResolver`), and the whole thing runs under
+`detectorTimeoutMs`.
+
+```ts
+new BotHandler({
+  resolver: cachingResolver(nodeDnsResolver(), { ttlMs: 600_000, maxEntries: 5_000 }),
+});
+```
+
+### `treatMissingPtrAsForgery`
+
+Off by default, and it should stay off unless you know what you are doing. Some legitimate
+crawlers have no `PTR` at all; treating absence as forgery converts a gap in somebody
+else's DNS into an accusation.
+
+## Published address ranges
+
+Twelve shipped signatures verify by address instead — every AI crawler among them. Where
+it is available this is better than reverse DNS in three ways: a lookup instead of a
+network round trip on the request path, immune to somebody else's DNS having a bad
+afternoon, and it works for the several crawlers that publish ranges and no useful `PTR`.
+
+**The library ships no address data**, and that is deliberate: a range baked into a
+release is wrong by the time somebody installs it, and being wrong here means verifying
+whoever has since been handed the address. What it ships is the URL each operator
+publishes.
+
+```ts
+import { startCrawlerRangeRefresh } from "@osqd/bothandlerjs";
+
+const stop = startCrawlerRangeRefresh(detector);          // twice a day by default
+```
+
+Opt-in, because it makes outbound requests, and a dependency-free package quietly fetching
+URLs on a timer is not something to inherit by accident.
+
+### What it will and will not accept
+
+It reads the two formats anybody publishes: a JSON document with a `prefixes` array of
+`{ ipv4Prefix }` / `{ ipv6Prefix }` objects — the shape Google standardised and the AI
+crawlers copied — and a plain-text list of one address or CIDR per line.
+
+Two things are refused **whole**, because these ranges do not merely describe a crawler,
+they *verify* one — and an address inside them is a `verified-bot`, which most policies
+allow:
+
+- a list containing a block bigger than any crawler owns (`/8` or wider for IPv4);
+- an empty list.
+
+A partially-parsed list is refused too. The operation replaces a set, and a set that half
+arrived is worse than the one already installed.
+
+### It fails open, per source
+
+One publisher being down, having moved its file, or serving something unrecognisable
+leaves every other crawler's ranges as they were — and leaves *that* crawler's ranges as
+they were too, which is the state it was in before you called this. Failures are raised as
+warnings, so they land in the [dashboard's notices](../operations/dashboard.md).
+
+```ts
+const result = await refreshCrawlerRanges(detector, { by: "the nightly job" });
+result.updated;   // [{ id: "googlebot", prefixes: 42 }]
+result.failed;    // [{ id: "gptbot", reason: "503 Service Unavailable" }]
+```
+
+### Supplying your own
+
+```ts
+await refreshCrawlerRanges(detector, {
+  sources: [{ id: "gptbot", url: "https://internal.example/mirrors/gptbot.json" }],
+});
+
+// or set them directly, from any source you like
+detector.updateCrawlerRanges("gptbot", ["203.0.113.0/24"]);
+```
+
+Mirroring the lists internally is a reasonable thing to do: it removes an outbound
+dependency from your servers and lets you review a change before it takes effect.
+`PUBLISHED_CRAWLER_RANGES` is the shipped list of pointers, and nothing stops you pinning
+all of them.
+
+## Related
+
+- [The signature database](signatures.md) — what is being verified
+- [Design decisions](../design/decisions.md) — why no address data ships
+- [Runtime changes](../operations/runtime-changes.md) — ranges are a runtime change like any other

package/docs/detection/writing-a-detector.md

@@ -0,0 +1,145 @@
+# Writing a detector
+
+The contract, and the one rule you cannot bend.
+
+← [Documentation](../index.md) · [The detectors](detectors.md)
+
+---
+
+A detector returns **evidence** and never a verdict. It does not know what will be done
+with what it found, and that is deliberate: the same observation is worth a tag on a
+documentation site and a challenge on a checkout.
+
+```ts
+import type { Detector, Evidence } from "@osqd/bothandlerjs";
+
+export function checkoutVelocity(): Detector {
+  return {
+    id: "checkout-velocity",
+    description: "More checkout attempts in a minute than a person makes",
+    cost: "cheap",
+    stage: "always",
+
+    inspect(ctx): Evidence | undefined {
+      if (!ctx.facts.path.startsWith("/checkout")) return undefined;
+
+      const attempts = ctx.state.requestsWithin(60_000, ctx.facts.timestamp);
+      if (attempts < 8) return undefined;
+
+      return {
+        detector: "checkout-velocity",
+        summary: `${attempts} checkout attempts in a minute`,
+        direction: "bot",
+        certainty: "strong",   // not `certain` — a shared address explains it too
+        weight: 0.6,
+        botClass: "automation",
+      };
+    },
+  };
+}
+
+new BotHandler({ extraDetectors: [checkoutVelocity()] });
+```
+
+## The context
+
+```ts
+interface DetectionContext {
+  facts: RequestFacts;          // the normalised request
+  ua: ParsedUserAgent;          // parsed once, shared by every detector
+  actor: ActorSnapshot;         // history *before* this request
+  state: ActorState;            // the live object: requestsWithin, intervalStats, …
+  clock: Clock;
+  signatures: MultiPatternMatcher;
+  signatureMatches: BotSignature[];
+  resolver: DnsResolver;
+  ranges: Map<string, IpRangeSet>;
+  shared: Map<string, unknown>;  // scratch space for one request
+}
+```
+
+`shared` exists so two detectors can avoid doing the same expensive thing twice within a
+single assessment. It is discarded afterwards.
+
+## The rule about certainty
+
+**`certain` requires a written `deterministicBasis`.** Not a strong feeling — an
+explanation of why no legitimate client produces this:
+
+```ts
+{
+  certainty: "certain",
+  deterministicBasis:
+    "HTTP/2 forbids connection-specific headers outright. A compliant client cannot " +
+    "send one, so its presence is a protocol violation rather than an unusual choice.",
+}
+```
+
+`strictEvidence` (on by default) enforces it: evidence marked `certain` with no basis is
+**rejected** and reported through `onWarning`. The mechanism is there because "is this
+really proof?" is a question people answer optimistically at 2am, and a downgrade is a
+much better failure than a false accusation.
+
+If you are unsure, you want `strong`. A `strong` signal still tags, delays, rate-limits
+and challenges; the only thing it cannot do is deny somebody service, and if you cannot
+write the sentence then it should not be able to.
+
+## Cost and stage
+
+| | |
+| --- | --- |
+| `cost: "cheap"` | Synchronous, in-memory. Run without an await. |
+| `cost: "io"` | May touch the network or a store. Run concurrently under a timeout. |
+| `stage: "always"` | Every request. |
+| `stage: "confirming"` | Only when a signature matched — there is a claim to check. |
+
+Mislabelling an `io` detector as `cheap` would put an unbounded await on the request path.
+The engine times out any promise a `cheap` detector returns anyway, so the failure mode is
+a timeout rather than a hung request — but label it correctly.
+
+## Human-pointing evidence
+
+`direction: "human"` puts the evidence on the other side of the scale, where it
+*discounts* suspicion. Use it for positive marks of a real session, and hold it to the
+same standard: a `certain` human signal means somebody's own code asserted it, not that a
+cookie looked plausible.
+
+## Families
+
+If your detector fires on the same root cause as another, declare a `family`. Within a
+family the engine takes the strongest observation rather than compounding them — see
+[families](../concepts/evidence.md#families).
+
+## Replacing a shipped detector
+
+`defaultDetectors()` returns the standard set as an array, so swapping one for a
+configured version is a `map`:
+
+```ts
+import { defaultDetectors, probeSignatureDetector } from "@osqd/bothandlerjs";
+
+new BotHandler({
+  detectors: defaultDetectors().map((d) =>
+    d.id === "probe-signature" ? probeSignatureDetector({ ignore: ["/wp-login.php"] }) : d,
+  ),
+});
+```
+
+## Testing it
+
+Put your handler in front of the [corpus](../testing/corpus.md). A new detector that
+scores an extra ten points on every headless browser is doing its job; one that also
+scores three points on every Safari user is not, and only the corpus will tell you which
+you wrote.
+
+```ts
+import { runCorpus } from "@osqd/bothandlerjs/corpus";
+
+const scorecard = await runCorpus({
+  create: ({ resolver, clock }) => new BotHandler({ extraDetectors: [mine()], resolver, clock }),
+  assertActions: false,
+});
+expect(scorecard.falsePositives).toEqual([]);
+```
+
+See `examples/custom-detector.ts` for a complete one.

package/docs/index.md

@@ -0,0 +1,100 @@
+# Documentation
+
+**bothandlerjs** — bot traffic detection and handling for TypeScript.
+
+← [Back to the README](../README.md)
+
+---
+
+## Start here
+
+**New to this?** [**The course**](course/index.md) teaches every capability in sixteen
+lessons, in the order that makes each one make sense, with something to run at every step.
+The pages below are the reference: they answer "how does X work?" rather than "what do I do
+next?".
+
+| | |
+| --- | --- |
+| [The course](course/index.md) | Sixteen lessons, one running example, from first assessment to a policy you can defend. |
+| [Installation](start/installation.md) | Install it, and what it needs from your runtime. |
+| [Your first integration](start/first-integration.md) | Ten lines that assess traffic, and what each one does. |
+| [Choosing a policy](start/choosing-a-policy.md) | Pick a preset, prove it against your own traffic, then deploy it. |
+| [Upgrading](start/upgrading.md) | What changed between versions, and what you have to do about it. |
+
+## The ideas the library is built on
+
+Read these once and everything else follows from them. They are short.
+
+| | |
+| --- | --- |
+| [Evidence and certainty](concepts/evidence.md) | The two tiers, why they combine by different rules, and what "proof" means here. |
+| [Verdicts, classes and scores](concepts/verdicts.md) | What the engine concludes, and which field you should actually be reading. |
+| [The safety guard](concepts/the-guard.md) | The thing that stops a guess closing a door. The central mechanism. |
+| [Actors and behavioural memory](concepts/actors.md) | Who "the same client" is, what is remembered about them, and for how long. |
+| [Threat model](concepts/threat-model.md) | What this defends against, what it does not, and what it costs to be wrong. |
+
+## Detection
+
+| | |
+| --- | --- |
+| [How detection works](detection/index.md) | The pipeline: stages, budgets, failure, and what an assessment contains. |
+| [The detectors](detection/detectors.md) | All twenty, each with what it reads, why it exists, and what it costs. |
+| [The signature database](detection/signatures.md) | How a client is recognised by name, and what a name is worth. |
+| [Verifying a crawler](detection/verification.md) | Reverse DNS, published address ranges, and refuting a forgery. |
+| [Browser signals](detection/client-signals.md) | The optional page script, and the ceiling on anything it reports. |
+| [Writing a detector](detection/writing-a-detector.md) | The contract, and the rules about certainty you have to keep. |
+
+## Policy
+
+| | |
+| --- | --- |
+| [Policy overview](policy/index.md) | Rules, order, and how a decision is reached. |
+| [Matching requests](policy/rules.md) | Every field a rule can match on, with examples. |
+| [Actions](policy/actions.md) | All ten, ordered by what each costs a client that turns out to be a person. |
+| [Presets](policy/presets.md) | The eight shipped policies, and what each is for. |
+| [robots.txt](policy/robots.md) | Generating the file your policy implies. |
+
+## The challenge
+
+| | |
+| --- | --- |
+| [The challenge](challenge/index.md) | Proof of work, clearance, and exactly what it buys. |
+| [Languages](challenge/localisation.md) | Writing the interstitial in a language the visitor reads. |
+
+## Running it
+
+| | |
+| --- | --- |
+| [Operations overview](operations/index.md) | What to watch, and what to do when it moves. |
+| [The dashboard](operations/dashboard.md) | What it shows, what it refuses to do, and every option it takes. |
+| [Metrics](operations/metrics.md) | Counters, histograms and the Prometheus exposition. |
+| [The traffic audit](operations/audit.md) | Watching the shape of your traffic rather than any one request. |
+| [Notifications](operations/notifications.md) | Getting told, without being told a thousand times. |
+| [Runtime changes](operations/runtime-changes.md) | Changing rules, the guard, ranges and actors without a deploy. |
+
+## Integrating it
+
+| | |
+| --- | --- |
+| [Integration overview](integration/index.md) | Where the library sits in a request. |
+| [Adapters](integration/adapters.md) | Express, Fastify, Koa, Fetch, Hono, Next.js — and writing your own. |
+| [Getting the client address right](integration/client-ip.md) | The highest-consequence setting in the library. |
+| [Shared state](integration/stores.md) | What has to be shared between replicas, and what deliberately is not. |
+
+## Proving it before it meets anybody
+
+| | |
+| --- | --- |
+| [Testing overview](testing/index.md) | The three ways to find out what a policy does before it does it. |
+| [The command line](testing/cli.md) | `replay`, `check`, `explain`, `robots`, `detectors`. |
+| [The traffic corpus](testing/corpus.md) | 526 shapes of real traffic, and how to run your own config against them. |
+| [Replaying your own logs](testing/replay.md) | The most useful thing you can do before deploying anything. |
+| [Try it locally](testing/try-it.md) | A demo site, a live dashboard and eighteen scripted clients. |
+
+## Reference
+
+| | |
+| --- | --- |
+| [Configuration](reference/configuration.md) | Every option, with its default and its consequence. |
+| [API](reference/api.md) | Every export, grouped by what it is for. |
+| [Design decisions](design/decisions.md) | The things this library refuses to do, and why. |