@osqd/bothandlerjs 0.3.0

package/docs/concepts/threat-model.md

@@ -0,0 +1,120 @@
+# Threat model
+
+What this library stops, what it raises the cost of, and what it cannot touch.
+
+← [Documentation](../index.md)
+
+---
+
+Most bot tooling is vague about this, which is how people end up trusting it for things it
+cannot do. This page is the honest version, and every claim on it is
+[measured](../testing/corpus.md) rather than asserted.
+
+---
+
+## The populations
+
+| Who | What they send | What happens |
+| --- | -------------- | ------------ |
+| **Honest automation** | `curl/8.4.0`, `python-requests`, `Googlebot` | proven by [self-declaration](evidence.md); you decide by policy |
+| **Verifiable crawlers** | a named identity backed by DNS or published ranges | [proven either way](../detection/verification.md) — confirmed or refuted |
+| **Careless scrapers** | a copied User-Agent and nothing else | caught by four independent signals |
+| **Competent scrapers** | a copied header set, in the right order, with Client Hints | not caught from one request; caught by [behaviour](../detection/detectors.md) if they are quick |
+| **Patient adversaries** | real Chrome, residential proxies, human pace | **not caught, and this library says so** |
+| **Scanners** | `/.env`, `/.git/config`, JNDI, `TRACE` | [`probe-signature`](../detection/detectors.md), one request at a time |
+| **People who look odd** | Tor, corporate proxies, screen readers, old phones | suspected, never denied — this is what [the guard](the-guard.md) is for |
+
+The last two rows are the same problem seen from both ends, and the reason the whole design
+is organised around proof rather than points.
+
+## The evasion ladder
+
+Five cases in the corpus, running from crude to undetectable. **The top rungs are expected
+to fail**, and they are kept so that nobody can claim otherwise:
+
+| | | |
+|-|-|-|
+| 1 | Copied the User-Agent only | caught, four independent signals |
+| 2 | Copied the header set, not the order | caught, weakly |
+| 3 | Copied the order and the Client Hints | **not caught** from one request |
+| 4 | …at a machine-perfect rhythm | caught by `cadence`, and only by `cadence` |
+| 5 | …paced like a person, a few pages per address | **not caught at all** |
+
+At level 5 the difference from a person has stopped being technical. What defeats it is
+**cost** — a [proof of work](../challenge/index.md), or an account — not detection.
+
+---
+
+## What this library cannot do
+
+**Stop a determined, well-resourced adversary.** Someone running real Chrome through a
+residential proxy pool, at human pace, with correct headers, solving the proof of work, is
+indistinguishable from a person at the HTTP layer. What this raises is the *cost*.
+
+**Prove somebody is human.** No signal here does that and none claims to. Proof of work
+proves CPU. `navigator.webdriver` proves what the client chose to report. The only
+conclusive human signal is [your own application's assertion](../reference/configuration.md).
+
+**Replace authentication, authorisation or a WAF.** It classifies traffic. It is not a
+security boundary and nothing about it should be load-bearing for access control. This is
+the most important sentence on the page.
+
+**Stop a DDoS.** It runs inside your process, after the connection is accepted. Volume that
+hurts you at the network layer needs handling at the network layer.
+
+**Ship IP intelligence.** Address-to-operator mappings go stale within weeks, and a stale
+mapping is a false positive with a long half-life. Bring your own, from a source you refresh
+and can audit.
+
+**Be right about a shared address.** Behind CGNAT, "one [actor](actors.md)" is thousands of
+people — which is exactly why the behavioural signals are capped where they are.
+
+**Escalate on a wordlist walk.** `probe-signature` reads one request at a time, so a scanner
+working through five hundred paths produces five hundred separate observations rather than a
+mounting case. That is the price of a detector that runs unchanged over a log file;
+enumeration over time is what `rate-anomaly`, `cadence` and `crawl-breadth` are for.
+
+---
+
+## The adversary's view of the library itself
+
+Worth thinking about, because a detector that explains itself to the client is a detector
+being tuned against.
+
+**Verdict headers are off by default.** An `X-Bot-Score` in the response is a live feedback
+signal: change one header, watch the number fall, iterate. Request-side tagging tells your
+application the same thing and the client nothing.
+
+**Metrics and the dashboard describe your detection.** The detector-firing series is exactly
+what somebody tuning a scraper would like to read. Serve them where only you can reach them,
+and use [the dashboard's redaction and sections](../operations/dashboard.md) when more people
+need to watch than need to know.
+
+**The challenge is public by design.** Its difficulty and its mechanism are visible to
+everyone. That is fine — it is a cost, not a secret, and a cost that only works while hidden
+is not a cost.
+
+**Traps are the one thing to keep quiet about.** A trap works because no person can reach
+it. Publishing the path in a public repository, a `robots.txt` `Allow` line, or a client-side
+comment turns proof back into a guess.
+
+---
+
+## What it is genuinely good at
+
+Being clear about the limits above is what makes this list credible:
+
+- **Identifying honest automation exactly**, so you can decide about it by policy rather
+  than by suspicion — including the [AI crawlers, split by job](../policy/presets.md#decline-ai-training).
+- **Refuting forged identities**, with proof rather than inference.
+- **Making bulk extraction expensive** without touching anybody who is not doing it.
+- **Never denying a person on a guess** — enforced by [the guard](the-guard.md), checked by
+  [the corpus](../testing/corpus.md), and visible as `bothandler_downgrades_total`.
+- **Telling you what your policy would do**, [before it does it](../testing/index.md).
+
+## Related
+
+- [Evidence](evidence.md) — proof versus suspicion, in detail
+- [The guard](the-guard.md) — the mechanism the last claim rests on
+- [The corpus](../testing/corpus.md) — where the ladder above lives
+- [Detectors](../detection/detectors.md) — what each signal is actually worth

package/docs/concepts/verdicts.md

@@ -0,0 +1,108 @@
+# Verdicts, classes and scores
+
+What the engine concludes, and which field you should actually be reading.
+
+← [Documentation](../index.md)
+
+---
+
+An assessment carries three different summaries of the same request. They answer
+different questions and are not interchangeable.
+
+## `verdict` — what this is
+
+```ts
+type Verdict = "confirmed-bot" | "verified-bot" | "suspected-bot" | "human" | "unknown";
+```
+
+| Verdict | Proven? | Means |
+| ------- | ------- | ----- |
+| `confirmed-bot` | yes | Proven automation. It said so, walked into a trap, or violated a protocol. |
+| `verified-bot` | yes | Proven automation **and** proven to be who it claims — a crawler that checked out. |
+| `suspected-bot` | no | The score crossed `suspectThreshold`. A judgement call. |
+| `human` | no | Human evidence outweighs bot evidence. Also a judgement. |
+| `unknown` | no | Nothing conclusive either way. **This is what ordinary traffic looks like.** |
+
+`unknown` being the common case is not a failure. Most requests carry no strong signal in
+either direction, and a library that concluded something about all of them would be
+guessing about most of them.
+
+## `certain` — whether it is proven
+
+```ts
+if (assessment.certain) { /* there is at least one `certain` piece of evidence */ }
+```
+
+**This is the field to read**, not the score. `certain` is what the
+[guard](the-guard.md) consults before allowing a terminal action, and it is the only
+thing that distinguishes "we know" from "we think".
+
+A proven **human** sets `certain` too — an operator assertion or a granted clearance is
+proof in the same sense. Check `verdict` alongside it when the direction matters.
+
+## `score` — how suspicious, when it is a judgement
+
+An integer. `0–99` for a probabilistic verdict; a flat `100` for anything proven, for the
+benefit of dashboards that chart one.
+
+The score exists to be *compared*, not to be trusted absolutely. It is the output of a
+saturating function over weighted evidence, so the difference between 40 and 60 is real
+and the difference between 96 and 98 is noise.
+
+```ts
+suspectThreshold: 60   // the score at which a verdict becomes `suspected-bot`
+```
+
+Before you move that number, look at the score distribution on the
+[dashboard](../operations/dashboard.md) or in `bothandler_score_bucket`: the question is
+not "is 60 right in the abstract" but "how close does *my* ordinary traffic run to it".
+
+## `botClass` — what kind of thing this is
+
+```ts
+type BotClass =
+  | "human" | "verified-bot" | "declared-bot" | "automation"
+  | "http-client" | "scanner" | "scraper" | "impersonator" | "unknown";
+```
+
+This is the field policies usually want, because it carries intent in a way a verdict
+does not. `verified-bot` and `impersonator` are both proven; one is Googlebot and the
+other is something pretending to be it.
+
+| Class | Typical example |
+| ----- | --------------- |
+| `verified-bot` | Googlebot, confirmed by reverse DNS or a published range |
+| `declared-bot` | GPTBot, ClaudeBot — honest, unverified, and a business decision |
+| `http-client` | curl, python-requests, Go-http-client |
+| `automation` | Headless Chrome, Playwright, Selenium |
+| `scraper` | Behaviourally a scraper: breadth, cadence, no session |
+| `scanner` | sqlmap, Nikto, probes for `/wp-admin` and `.env` |
+| `impersonator` | Claimed an identity that was refuted |
+| `human` | Proven or strongly indicated to be a person |
+
+## `confidence`
+
+`0–1`, the raw probability before the threshold is applied. Useful if you are charting
+distributions or building your own thresholds; `score` is the same number scaled and
+rounded.
+
+## Putting it together
+
+```ts
+const { assessment, decision } = await detector.handle(facts);
+
+// The two questions worth asking, in this order:
+if (assessment.certain && assessment.botClass === "impersonator") {
+  // Proven forgery. Safe to refuse.
+}
+if (assessment.verdict === "suspected-bot") {
+  // A judgement. Challenge, tag, slow — never refuse on this alone,
+  // and the guard will stop you if a rule tries.
+}
+```
+
+## Related
+
+- [Evidence and certainty](evidence.md) — where all of this comes from
+- [The safety guard](the-guard.md) — why `certain` matters more than `score`
+- [Matching requests](../policy/rules.md) — matching rules on any of these fields

package/docs/course/01-first-assessment.md

@@ -0,0 +1,178 @@
+# Lesson 1 — Your first assessment
+
+**Goal:** turn an HTTP request into a verdict, and understand every field that comes back.
+
+← [Course](index.md) · Next: [Proof and suspicion](02-proof-and-suspicion.md)
+
+---
+
+## The two calls
+
+Everything in this library is downstream of two functions.
+
+`createFacts` normalises a request into a `RequestFacts` — headers lowercased, path decoded
+once and resolved, query into a null-prototype bag, everything length-bounded. `assess`
+reads those facts and returns an `Assessment`.
+
+Neither touches a response. That is why you can run `assess` over a log file, and why the
+rest of this course can happen in a plain script with no server.
+
+## Do this
+
+`serif/lesson-01.mjs`:
+
+```js
+import { BotHandler, createFacts } from "@osqd/bothandlerjs";
+
+const detector = new BotHandler();
+
+const request = createFacts({
+  method: "GET",
+  url: "/books/1",
+  headers: { host: "serif.example", "user-agent": "curl/8.4.0", accept: "*/*" },
+  ip: "203.0.113.10",
+});
+
+const assessment = await detector.assess(request);
+
+console.log("verdict   ", assessment.verdict);
+console.log("class     ", assessment.botClass);
+console.log("score     ", assessment.score);
+console.log("certain   ", assessment.certain);
+console.log("confidence", assessment.confidence);
+for (const piece of assessment.evidence) {
+  console.log(`  [${piece.certainty}] ${piece.detector}: ${piece.summary}`);
+}
+```
+
+```bash
+node lesson-01.mjs
+```
+
+### Checkpoint
+
+```
+verdict    confirmed-bot
+class      http-client
+score      100
+certain    true
+confidence 1
+  [certain] self-identified: User-Agent identifies curl
+```
+
+If you got that, the library is installed and working.
+
+## What each field means
+
+| Field | |
+| ----- | - |
+| `verdict` | the conclusion: `confirmed-bot`, `verified-bot`, `suspected-bot`, `human`, `unknown` |
+| `botClass` | *what kind* of client: `http-client`, `scraper`, `scanner`, `impersonator`, `declared-bot`, `verified-bot`, `automation`, `human`, `unknown` |
+| `score` | suspicion, 0–100, from probabilistic signals only |
+| `certain` | whether at least one piece of **proof** fired. **This, not the score, is the important one** |
+| `confidence` | how much to trust the verdict, 0–1; exactly 1 when `certain` |
+| `evidence` | every bot-pointing observation, strongest first |
+| `humanEvidence` | every person-pointing observation; these *subtract* |
+| `actor` | who this client is and what has been seen from them — [lesson 7](07-actors.md) |
+| `requestId` | random per request, safe to log |
+| `durationMs` | time spent in detection |
+
+Note what happened above: `curl` did not get a score of 100 because many signals agreed.
+It got `certain: true` because the client **said** it was curl, and the score follows from
+proof rather than the other way round.
+
+## Now try a real browser
+
+Replace the headers with a full Chrome set — the Client Hints, the Fetch Metadata, the
+negotiation headers:
+
+```js
+const CHROME = {
+  host: "serif.example",
+  "user-agent":
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
+  accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
+  "accept-language": "en-GB,en;q=0.9",
+  "accept-encoding": "gzip, deflate, br",
+  "sec-ch-ua": '"Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"',
+  "sec-ch-ua-mobile": "?0",
+  "sec-ch-ua-platform": '"macOS"',
+  "sec-fetch-site": "none",
+  "sec-fetch-mode": "navigate",
+  "sec-fetch-dest": "document",
+};
+```
+
+Also print the human evidence:
+
+```js
+for (const piece of assessment.humanEvidence) {
+  console.log(`  (human) [${piece.certainty}] ${piece.detector}: ${piece.summary}`);
+}
+```
+
+### Checkpoint
+
+```
+verdict    unknown
+class      unknown
+score      0
+certain    false
+confidence 1
+  (human) [weak] browsing-coherence: Fetch Metadata, Client Hints and negotiation headers are all present and mutually consistent
+```
+
+**`unknown` is the correct answer for a person**, and it is the resting state of ordinary
+traffic. The library does not claim to have proved a human — it has nothing conclusive, and
+says so. Only your own application can assert that somebody is a person, which you will do
+in [lesson 12](12-going-live.md).
+
+## Exercise
+
+Take the Chrome headers and delete everything except `host`, `user-agent` and
+`accept: */*` — a scraper that copied a User-Agent string and nothing else. Assess it.
+
+<details>
+<summary>Checkpoint</summary>
+
+```
+verdict    unknown
+class      unknown
+score      45
+certain    false
+confidence 0.553
+  [moderate] header-integrity: Client claims to be a browser but sent no Accept-Language header
+  [moderate] header-integrity: Client claims to be a browser but sent no Accept-Encoding header
+  [weak] accept-signature: Client claiming a browser sent Accept: */* with no Fetch Metadata to explain it
+```
+
+Three signals, a score of 45, and still `unknown` — not enough to call it a bot, and
+nowhere near enough to refuse it. `confidence` dropped to 0.553, which is the library
+saying it is genuinely unsure.
+
+Notice what it did **not** do: claim to be certain. A copied User-Agent is suspicious and
+is not proof, and lesson 2 is about why that distinction is the whole design.
+</details>
+
+## Common mistake
+
+**Passing a forwarded address as `ip`.** `createFacts` wants the **socket** address; the
+forwarded chain is resolved separately and carefully, because getting it wrong lets clients
+pick their own identity. [Lesson 12](12-going-live.md) covers it, and it is the single
+most consequential setting in the library.
+
+## What you learned
+
+- `createFacts` + `assess` is the whole read path, and neither touches a response
+- Five fields describe a client, and `certain` is the one that matters most
+- `unknown` is what ordinary human traffic looks like
+- Proof and suspicion arrive by different routes
+
+## Reference
+
+- [How detection works](../detection/index.md) — the pipeline behind `assess`
+- [Verdicts, classes and scores](../concepts/verdicts.md)
+- [Configuration](../reference/configuration.md) — every constructor option
+
+Next: [Proof and suspicion](02-proof-and-suspicion.md) — the idea the rest of the course
+depends on.

package/docs/course/02-proof-and-suspicion.md

@@ -0,0 +1,164 @@
+# Lesson 2 — Proof and suspicion
+
+**Goal:** understand the distinction the entire library is built on, and why every other
+design decision follows from it.
+
+← [Course](index.md) · Prev: [First assessment](01-first-assessment.md) · Next: [Verdicts and scores](03-verdicts-and-scores.md)
+
+---
+
+## The problem every bot detector has
+
+Every signal that catches sophisticated automation is *probabilistic*: header
+consistency, timing regularity, missing cookies, TLS fingerprints. And every one of them
+has a population of real people who trip it.
+
+Someone on a privacy-hardened browser. Someone behind a corporate proxy that strips
+headers. Someone using a screen reader. Someone on a hotel network, or a five-year-old
+phone, or a carrier that transcodes pages.
+
+The usual answer is to add the signals into a score and block above a threshold. **That is
+the mistake this library exists to avoid.** Points do not compose into proof. Two unrelated
+suspicions about an unusual but entirely real browser reach 100 as readily as two
+well-founded ones — and the people who get caught are disproportionately the ones with the
+strongest reasons for their unusual setup.
+
+So evidence here lives in two compartments that never mix.
+
+| | Deterministic (`certain`) | Probabilistic (`strong` / `moderate` / `weak`) |
+| --- | --- | --- |
+| Rests on | a declaration, a contradiction, a trap, an external authority | a pattern automation usually shows |
+| Can it be wrong? | only if the client lied about itself | yes, about real people |
+| How it combines | short-circuits to a verdict | noisy-OR into a score of 0–99 |
+| Can it deny service? | **yes** | **no** |
+| What it can still do | anything | tag, log, delay, rate-limit, challenge, alert |
+
+## See it
+
+Print the basis of the curl evidence from lesson 1:
+
+```js
+const assessment = await detector.assess(
+  createFacts({ method: "GET", url: "/", headers: { host: "serif.example", "user-agent": "curl/8.4.0" }, ip: "203.0.113.10" }),
+);
+console.log(assessment.evidence[0].deterministicBasis);
+```
+
+### Checkpoint
+
+```
+The product token "curl/" is emitted by an HTTP library or an automation runtime and by
+no browser. Nothing a person does in a browser produces it.
+```
+
+**Every piece of `certain` evidence has to carry one of those sentences**, in writing, and
+the library refuses evidence that claims certainty without one. It is a forcing function:
+if you cannot write the sentence, your evidence is `strong`. When you write your own
+detector in [lesson 15](15-extending.md) you will meet this rule from the other side.
+
+## The five things that earn `certain`
+
+1. **Self-declaration.** The client says it is a bot: `curl/8.4.0`, `python-requests`,
+   `Googlebot/2.1`, `HeadlessChrome` in a Client Hints brand list. Not an inference — you
+   are believing the client's own statement about itself, and no honest client is ever
+   harmed by being believed. If the statement is a lie, the misclassification is the
+   client's doing.
+2. **A refuted third-party identity.** It claimed to be Googlebot and DNS says otherwise.
+   Note how narrow that is: a privacy extension rewriting a User-Agent to a *generic*
+   browser string never lands here, because it never claims to be a named, verifiable
+   third party.
+3. **A confirmed third-party identity.** The same check passing. Used to *allow*.
+4. **A trap.** A link hidden from layout and from assistive technology and excluded in
+   `robots.txt`. Detection by construction rather than by inference — no sequence of user
+   input reaches it. [Lesson 8](08-traps.md).
+5. **A protocol violation.** Three of them, each a rule a recipient is *required* to
+   enforce: a connection-specific header on HTTP/2 ([RFC 9113 §8.2.2](https://www.rfc-editor.org/rfc/rfc9113#section-8.2.2)),
+   `Content-Length` beside `Transfer-Encoding` ([RFC 9112 §6.1](https://www.rfc-editor.org/rfc/rfc9112#section-6.1)),
+   and a repeated `Host` ([RFC 9112 §3.2](https://www.rfc-editor.org/rfc/rfc9112#section-3.2)).
+
+Plus one on the human side: **your own application's assertion** that a request belongs to
+a person.
+
+## The rule that catches people out
+
+> **No argument from absence may ever be `certain`.**
+
+A header missing from your *facts* is not a header missing from the *request*. An HTTP/1.1
+request with no `Host` violates RFC 9112 as plainly as anything above — and it is
+deliberately not proven here, because somebody building facts from a log line, a WAF event
+or a partial adapter would otherwise manufacture proof against every request in the file.
+
+You will rely on this in [lesson 16](16-proving-it.md) when you replay your own access
+logs, where most headers genuinely are missing from the record.
+
+## How suspicion adds up
+
+Probabilistic evidence combines by **noisy-OR** — `1 − Π(1 − wᵢ)` — not by a sum. The
+weights are `strong` 0.60, `moderate` 0.35, `weak` 0.15.
+
+It is bounded without clamping, and it has the right shape: many weak signals do add up,
+but asymptotically. **The probabilistic score is capped at 99**, because 100 means proof.
+
+Human evidence subtracts: `score = pBot × (1 − pHuman)`.
+
+## One cause, counted once
+
+Noisy-OR is only sound over *independent* signals, and several of these are not. A
+corporate proxy that strips `Sec-Fetch-*` also strips the Client Hints and the
+`Accept-Language` — so three detectors fire at once about one person behind one appliance,
+and the arithmetic reads their agreement as corroboration when it is an echo.
+
+Evidence may therefore declare a **`family`**: a shared root cause. Within a family the
+engine takes the strongest observation instead of compounding.
+
+You saw this in lesson 1's exercise without noticing. Print the families:
+
+```js
+for (const piece of assessment.evidence) {
+  console.log(piece.certainty.padEnd(9), (piece.family ?? "—").padEnd(18), piece.summary);
+}
+```
+
+Run it against the copied-User-Agent request. The two `header-integrity` observations share
+a family, so they count once. That scraper scores 45 rather than the ~60 it would reach if
+its two absences were treated as independent — and the reason is that a real corporate
+network produces exactly the same pair, every day.
+
+Families are a scoring correction and nothing more. **They never touch the proven path.**
+
+## Exercise
+
+Build a request that is proven a bot *and* carries a lot of suspicion, and one that carries
+suspicion alone. Compare `certain`, `score` and `confidence` on each.
+
+Then answer, without running anything: a client scores 97 from six independent
+probabilistic signals. Under the default configuration, can a rule block it?
+
+<details>
+<summary>Answer</summary>
+
+**No.** Not at 97, not at 99. The score is not what gates a terminal action — `certain` is.
+Six probabilistic signals are six things a real person can trip, and the population that
+trips six of them is disproportionately the population with the strongest reasons for an
+unusual setup.
+
+That is [lesson 4](04-the-guard.md), and it is enforced by a mechanism you cannot forget to
+apply.
+</details>
+
+## What you learned
+
+- Probabilistic signals all have real people who trip them; that is why points never become
+  proof
+- Exactly five things earn `certain`, and each must carry a written basis
+- No argument from absence may ever be certain
+- Noisy-OR bounds suspicion at 99; human evidence subtracts
+- One root cause is counted once, so a stripped-header proxy is not three reasons
+
+## Reference
+
+- [Evidence and certainty](../concepts/evidence.md) — the full model
+- [Design decisions](../design/decisions.md) — this choice, and what it costs
+- [Threat model](../concepts/threat-model.md) — who this catches and who it does not
+
+Next: [Verdicts, classes and scores](03-verdicts-and-scores.md).