@osqd/bothandlerjs 0.3.0

package/docs/challenge/index.md

@@ -0,0 +1,152 @@
+# The challenge
+
+A proof-of-work interstitial: what it buys, what it does not, and how to wire it up.
+
+← [Documentation](../index.md)
+
+---
+
+## What it actually buys, stated plainly
+
+The client must find a counter such that `SHA-256(nonce + ":" + counter)` begins with
+`difficulty` zero bits. Verification is one hash; solving takes on average `2^difficulty`
+of them.
+
+**It does not identify anyone and it does not prove a human is present.** A headless
+Chrome solves it as readily as a person's phone, just paying for the CPU.
+
+What it does is convert a scrape from free into merely cheap, and change the *shape* of the
+attack: a stateless scraper pulling a million pages must now run a JavaScript engine and
+burn CPU on every one. That is often enough to make bulk extraction not worth doing, and it
+costs a real visitor a fraction of a second, once.
+
+This is why the challenge sits where it does in the [action ladder](../policy/actions.md):
+it is the heaviest thing the [guard](../concepts/the-guard.md) will let a *probabilistic*
+verdict ask for, precisely because a client that is wrongly suspected can pass it on its
+own and carry on.
+
+---
+
+## Turning it on
+
+One thing is required and has no default:
+
+```ts
+import { BotHandler } from "@osqd/bothandlerjs";
+
+const detector = new BotHandler({
+  preset: "protect-content",
+  challenge: {
+    secrets: [process.env.BOT_CHALLENGE_SECRET!],
+    contactHtml: '<p>Locked out? Email <a href="mailto:support@example.com">support@example.com</a>.</p>',
+  },
+});
+```
+
+A library-supplied fallback secret is a library-supplied forgery key, and it would end up
+in production somewhere. Without `secrets`, a rule asking for a challenge degrades to a
+`tag` and says so through [`onWarning`](../operations/notifications.md).
+
+The [adapters](../integration/adapters.md) serve the verification endpoint for you. Nothing
+else to mount.
+
+### Rotating secrets
+
+The first secret signs; all of them verify. Prepend a new one and keep the old for a token
+lifetime, and nobody is logged out:
+
+```ts
+challenge: { secrets: [NEW_SECRET, PREVIOUS_SECRET] }
+```
+
+---
+
+## Every option
+
+| Option | Default | Notes |
+| ------ | ------- | ----- |
+| `secrets` | — | required; first signs, all verify |
+| `difficulty` | `16` | leading zero bits; see below |
+| `challengeTtlMs` | `120_000` | how long a challenge may be solved for |
+| `clearanceTtlMs` | `3_600_000` | how long a granted clearance lasts |
+| `verifyPath` | `/__bothandler/verify` | where the solution is POSTed |
+| `cookieName` | `__bh_clearance` | |
+| `cookieSecure` | `true` | set `false` only for local plaintext development |
+| `cookieSameSite` | `"Lax"` | |
+| `title`, `message` | English defaults | page copy |
+| `contactHtml` | — | **supply something real** |
+| `translations` | — | see [localisation](localisation.md) |
+
+### Difficulty
+
+In *bits*, so each step doubles the work. The default 16 is around 65k hashes — tens of
+milliseconds in any modern browser. It is refused above 24.
+
+Past about 20 you are charging real people a visible delay, and the oldest and slowest
+devices — which disproportionately belong to the users least able to replace them — pay
+the most. Raise it during an incident, not as a posture.
+
+### `contactHtml`
+
+Everyone who sees the no-JavaScript fallback is a person your site just turned away: no
+JavaScript, no WebCrypto, or a device too slow to finish. Put a support address, a phone
+number or a link to a form there. This is the single highest-value line of configuration on
+this page.
+
+---
+
+## What a clearance proves
+
+Solving grants a signed cookie, which the `clearance` [detector](../detection/detectors.md)
+reads on subsequent requests. It carries a **level**, and the level is what a rule should
+key on:
+
+| Level | What was demonstrated | Certainty |
+| ----- | --------------------- | --------- |
+| `pow` | ran JavaScript, has WebCrypto, spent measurable CPU | not conclusive |
+| `interaction` | a trusted input event was observed | stronger, still forgeable by a driven browser |
+| `operator` | **your application** asserted this is a person | `certain` |
+
+Only `operator` is treated as conclusive, because that assertion comes from you rather than
+from the client. See [`isHuman`](../reference/configuration.md) for how to make one.
+
+---
+
+## The lifecycle, and the one piece of state
+
+Deliberately stateless up to the moment of success. A challenge is a signed blob the client
+carries; the server stores nothing while it is being solved, so a flood of unsolved
+challenges costs nothing but the bytes to send them.
+
+Exactly one thing is written, at the one moment it is indispensable: the solved challenge's
+nonce is **claimed atomically**, so a solution cannot be replayed.
+
+That claim lives in the [store](../integration/stores.md). With the default in-memory store
+and several replicas, a scraper retries a solved nonce against other instances until one
+has not seen it. Pass a `RedisStore` if you run more than one process.
+
+## Two refusals worth knowing
+
+**Challenging an actor that already holds valid clearance** is refused and warned about.
+Passing a challenge cannot change a proven verdict, so re-issuing would loop for ever.
+
+**Challenging on an API** is a mistake the [`protect-api` preset](../policy/presets.md)
+deliberately avoids: an API client is not a browser, so a challenge breaks your customers'
+integrations while an attacker solves it once in headless Chrome. Rate-limit instead.
+
+## Unsolved challenges as evidence
+
+Every issued-and-never-solved challenge is counted on the [actor](../concepts/actors.md).
+A client that has been asked five times and never once finished has told you something,
+and rules can read it:
+
+```ts
+{ id: "persistent-refusal", match: { minUnsolvedChallenges: 5 }, action: "block" }
+```
+
+## Related
+
+- [Localisation](localisation.md) — showing the page in a language the visitor reads
+- [Actions](../policy/actions.md) — where `challenge` sits on the ladder
+- [The guard](../concepts/the-guard.md) — why it is the ceiling for unproven verdicts
+- [Stores](../integration/stores.md) — why replay protection needs a shared one

package/docs/challenge/localisation.md

@@ -0,0 +1,105 @@
+# Challenge localisation
+
+Showing the one public-facing page in a language the visitor can read.
+
+← [Documentation](../index.md) · [The challenge](index.md)
+
+---
+
+## Why this exists
+
+The interstitial is the only page this library shows to a member of the public, and it is
+shown because a *probabilistic* verdict went against them.
+
+Serving "Checking your browser" in English to somebody whose browser has been asking for
+Japanese since the first request is the same unfairness [the guard](../concepts/the-guard.md)
+exists to prevent, applied to the one screen where it is most visible: **a person who
+cannot read the page cannot find the contact link on it either.** That turns a check into a
+wall.
+
+## The library ships no translations, and will not
+
+A machine-translated apology on a page that just turned somebody away is worse than an
+honest English one — and only you know which languages your audience actually reads. What
+this does is pick between the translations *you* supply.
+
+```ts
+new BotHandler({
+  challenge: {
+    secrets: [process.env.BOT_CHALLENGE_SECRET!],
+    contactHtml: '<p>Locked out? <a href="/support">Contact us</a>.</p>',
+    translations: {
+      ja: {
+        title: "ブラウザーを確認しています",
+        message: "数秒で完了します。",
+        contactHtml: '<p>お困りですか？<a href="/support">サポート</a>へご連絡ください。</p>',
+      },
+      de: { title: "Browser wird überprüft" },
+      "pt-BR": { title: "Verificando seu navegador" },
+    },
+  },
+});
+```
+
+Anything omitted from a translation falls back to the default text, so a `title`-only entry
+is a perfectly reasonable first step.
+
+## How a language is chosen
+
+`Accept-Language` is parsed, `q` honoured, malformed entries dropped rather than fatal —
+this is a client-supplied header and the page it decides is one somebody is already having
+a bad time with. `*` is dropped too: it means "anything", which is what the default is for.
+Ordering is stable within a `q` value, so a header listing two equally-weighted languages
+means the first one.
+
+Then, in order:
+
+1. **Exact tag.** `pt-BR` matches a `pt-BR` key.
+2. **Primary subtag.** `pt-BR` matches a `pt` key — European Portuguese is far better than
+   English for a Brazilian visitor.
+3. **Nothing.** The default English copy.
+
+## Why `pt` never silently becomes `pt-BR`
+
+Matching stops after the primary subtag. A visitor asking for `pt-PT` will **not** be handed
+`pt-BR`.
+
+That looks unhelpful until you consider the case it protects: serving Simplified Chinese to
+somebody who asked for Traditional is a worse failure than serving English, and no rule can
+tell the two situations apart from the header alone. Whether one regional variant stands in
+for another is a judgement about *your* audience, so it is made by which keys you write
+rather than by a heuristic here.
+
+**Key by the primary tag** — `pt`, `zh`, `de` — unless you genuinely have separate regional
+copy.
+
+## The `lang` attribute
+
+Each translation may set `lang`, defaulting to the key it is filed under.
+
+It matters more than it looks. A screen reader picks its voice and its pronunciation rules
+from this attribute, so Japanese text announced as `lang="en"` is read aloud by an English
+voice and is unintelligible. Getting the copy right and the attribute wrong helps nobody.
+
+```ts
+translations: {
+  "zh-Hant": { lang: "zh-Hant", title: "正在檢查您的瀏覽器" },
+}
+```
+
+## Testing it
+
+`parseAcceptLanguage` and `pickTranslation` are pure and exported, so the choice can be
+tested by calling it:
+
+```ts
+import { parseAcceptLanguage, pickTranslation } from "@osqd/bothandlerjs";
+
+pickTranslation(translations, parseAcceptLanguage("pt-BR,pt;q=0.9,en;q=0.5"));
+// → { tag: "pt-BR", copy: { title: "Verificando seu navegador" } }
+```
+
+## Related
+
+- [The challenge](index.md) — what the page is for
+- [Configuration](../reference/configuration.md) — the rest of `challenge`

package/docs/concepts/actors.md

@@ -0,0 +1,113 @@
+# Actors and behavioural memory
+
+Who "the same client" is, what is remembered about them, and for how long.
+
+← [Documentation](../index.md)
+
+---
+
+Several detectors reason about behaviour over time — rate, rhythm, breadth, rotation —
+and all of them need an answer to one question first: *are these two requests the same
+client?*
+
+## The actor key
+
+```ts
+actorKey: (facts) => facts.ip          // the default
+```
+
+Every behavioural signal in this library is only as good as this function. The default is
+the client address, which is the only thing always available and is also the weakest: a
+corporate NAT, a mobile carrier's CGNAT pool, a university and a VPN exit all present
+hundreds of real people as one actor.
+
+If you can do better, do:
+
+```ts
+actorKey: (facts) => sessionIdFrom(facts) ?? facts.ip,
+```
+
+A session id, an authenticated user id, or an address plus a TLS fingerprint all make the
+same detectors sharper — sharp enough that
+[`identity-rotation`](../detection/detectors.md#identity-rotation) becomes worth enabling,
+which it is not on a shared address.
+
+## What is remembered
+
+```ts
+const snapshot = assessment.actor;
+```
+
+| Field | |
+| ----- | --- |
+| `key` | Whatever `actorKey` returned |
+| `requests` | Requests in the window |
+| `distinctPaths` | Breadth of the crawl |
+| `firstSeen` / `lastSeen` | Epoch ms |
+| `sinceLastMs` | Gap since the previous request |
+| `priorConfirmations` | Times this actor was **proven** a bot before now |
+| `unsolvedChallenges` | Challenges issued that no solution came back for |
+| `cleared` | Holds a currently-valid human clearance |
+
+`priorConfirmations` is a snapshot taken *before* this request's own outcome is recorded,
+so a rule reading `minPriorConfirmations: 1` does not match on an actor's very first
+request.
+
+## Everything here is bounded
+
+Every key in this subsystem is attacker-chosen, so nothing may grow without limit:
+
+- An actor's arrival history is a **fixed-size ring** of timestamps.
+- Its path set is **capped**, and the cap is visible (`pathsSaturated`) so a detector can
+  tell "wide" from "we stopped counting".
+- The registry itself is a **bounded LRU** with a TTL: `maxActors` (default 20,000) and
+  `actorWindowMs` (default 15 minutes).
+
+```ts
+actorWindowMs: 900_000,
+maxActors: 20_000,
+```
+
+An actor still sending traffic is never the one evicted to make room, and eviction is
+plain forgetting: the next request from that client is assessed as a first request.
+
+## It is process-local, deliberately
+
+Rates, cadence and path breadth live in memory rather than in a shared store. A round trip
+per request would buy accuracy for signals that are only ever allowed to *raise suspicion*
+— never to deny anybody — and would put a network dependency on the hot path of every
+request to your site.
+
+Behind four replicas each one sees a quarter of an actor's traffic and is correspondingly
+less sure. That is the right trade for something that cannot close a door on its own.
+
+**A confirmation is not one of those**, and it can travel:
+
+```ts
+new BotHandler({ store: new RedisStore(redis), shareConfirmations: true });
+```
+
+`confirmed-bot` is a *proven* verdict — a fact about the client rather than a judgement
+about it — and without sharing it a client proven to be a bot on one replica is a stranger
+to the other seven. Proof travels; suspicion stays home. See
+[Shared state](../integration/stores.md).
+
+## Acting on one actor by hand
+
+```ts
+detector.forgetActor("203.0.113.9");            // discard its history
+detector.clearActor("203.0.113.9", 3_600_000);  // grant human clearance for an hour
+detector.registry.top(50, Date.now());          // who is here, busiest first
+```
+
+`forgetActor` is the cure for a false positive that has stuck: a person whose actor key
+collected a `confirmed-bot` carries `priorConfirmations` for the rest of the window, and
+before this existed the only remedy was clearing every actor's history to fix one.
+
+These are also on the [dashboard](../operations/dashboard.md), behind a control flag.
+
+## Related
+
+- [The detectors](../detection/detectors.md) — which ones read this state
+- [Shared state](../integration/stores.md) — what crosses replicas
+- [Getting the client address right](../integration/client-ip.md) — the input to all of it

package/docs/concepts/evidence.md

@@ -0,0 +1,133 @@
+# Evidence and certainty
+
+The distinction the whole library rests on: **proof versus suspicion**.
+
+← [Documentation](../index.md)
+
+---
+
+Every detector produces *evidence*, never a verdict. A piece of evidence carries a
+`certainty` tier, and there are two kinds of tier with two different sets of rules.
+
+| Tier | Weight | What it means |
+| ---- | ------ | ------------- |
+| `certain` | — | **Proof.** There is no benign explanation, and the detector has to write down why. |
+| `strong` | 0.60 | A signal with a real population of exceptions. |
+| `moderate` | 0.35 | Suggestive. |
+| `weak` | 0.15 | Worth a point, not worth a sentence. |
+
+`certain` is not "very strong". It is a different kind of claim, and it is the only kind
+that can cost somebody their access.
+
+## What makes something certain
+
+A `certain` piece of evidence must carry a `deterministicBasis` — a written explanation
+of why no legitimate client produces this. Not "no legitimate client we know of": none,
+by construction.
+
+```ts
+{
+  detector: "self-identified",
+  certainty: "certain",
+  summary: 'User-Agent identifies curl',
+  deterministicBasis:
+    'The product token "curl/" is emitted by an HTTP library or an automation ' +
+    "runtime and by no browser. Nothing a person does in a browser produces it.",
+}
+```
+
+There are only a few genuine sources of proof, and they are all of the same shape —
+**the client told us, or the client did something only automation does**:
+
+- **A self-declaration.** `python-requests/2.31.0` in a User-Agent is not an inference.
+  If it is a lie, the misclassification belongs to whoever lied.
+- **A refuted identity.** Something claimed to be Googlebot; the operator's own DNS says
+  the address is not Google's. The claim is disproven by the party entitled to answer.
+- **A trap.** A path no link points at, a form field no rendered browser displays.
+  Reaching it requires reading the page as data rather than as a page.
+- **A protocol violation.** Two `Host` headers. `Connection: keep-alive` on HTTP/2.
+  Something a compliant client cannot emit.
+- **An operator assertion.** Your code said this is a person. We believe you.
+
+Everything else is a judgement. A missing `Accept-Language` is *suspicious* and belongs
+to a real population: privacy browsers, corporate proxies, screen readers, old phones.
+
+## How the two tiers combine
+
+Differently, on purpose.
+
+**Proof does not accumulate.** One `certain` piece is enough, and a second adds nothing:
+the verdict is already `confirmed-bot` or `verified-bot`, the score is reported as 100
+for anything that charts one, and no amount of probabilistic evidence can produce the
+same outcome. Certainty short-circuits scoring entirely.
+
+**Suspicion accumulates by noisy-OR**, not by addition:
+
+```
+pBot = 1 − Π (1 − weight)
+```
+
+Three `moderate` signals reach 0.72, not 1.05. The function saturates, which is the
+point: adding a fourth weak signal to three strong ones should barely move a number that
+is already near certainty, and adding twenty weak ones must never *reach* it.
+
+```ts
+import { combineEvidence, noisyOr } from "@osqd/bothandlerjs";
+
+noisyOr([0.6, 0.35, 0.35]);   // 0.831 — not 1.3
+```
+
+### Families
+
+Signals that are one signal wearing different hats are counted once. A client missing
+`Accept-Language`, `Accept-Encoding` and `Referer` has one property — it is not a
+browser — reported three times, and letting each contribute independently would triple a
+single observation.
+
+```ts
+{ detector: "header-integrity", family: "absent-browser-headers", certainty: "moderate" }
+```
+
+Within a family only the strongest counts. It is a scoring correction and nothing more:
+families never touch the proven path.
+
+### Human evidence subtracts
+
+Almost every detector argues in one direction, which is a problem the scoring model has
+to make visible. A person reading forty pages of documentation from a university's shared
+address accumulates `rate-anomaly`, `cadence`, `crawl-breadth` and `ip-intelligence`
+without a single thing being wrong with their request.
+
+So evidence has a `direction`, and the engine discounts the bot score by whatever human
+evidence it holds:
+
+```
+score = pBot × (1 − pHuman)
+```
+
+[`browsing-coherence`](../detection/detectors.md#browsing-coherence) exists entirely to
+produce that counterweight — a cache validator, a cookie jar, a same-site navigation are
+the marks of a browsing session, and a scraper has none of them.
+
+## Reading it back
+
+```ts
+const assessment = await detector.assess(facts);
+
+assessment.certain;        // true ← this, not the score, gates terminal actions
+assessment.verdict;        // "confirmed-bot"
+assessment.score;          // 100 for proven; 0–99 for a judgement
+assessment.evidence;       // strongest first
+assessment.humanEvidence;  // what argued the other way
+assessment.evidence[0].deterministicBasis;   // why it cannot be wrong
+```
+
+`strictEvidence` (on by default) makes the rule mechanical rather than cultural: a
+detector returning `certain` with no `deterministicBasis` is rejected and reported
+through `onWarning`. See [Writing a detector](../detection/writing-a-detector.md).
+
+## Related
+
+- [Verdicts, classes and scores](verdicts.md) — what the engine does with all this
+- [The safety guard](the-guard.md) — what stops a score from denying anybody
+- [The detectors](../detection/detectors.md) — which tier each one can reach

package/docs/concepts/the-guard.md

@@ -0,0 +1,121 @@
+# The safety guard
+
+The mechanism that stops a guess from closing a door. If you read one page, read this one.
+
+← [Documentation](../index.md)
+
+---
+
+Every bot detector eventually faces the same problem. The signals that catch sophisticated
+automation — header consistency, TLS fingerprints, timing regularity, missing cookies —
+are all *probabilistic*. Each one has a population of real people who trip it: somebody on
+a privacy-hardened browser, behind a corporate proxy, using a screen reader, on a hotel
+network, on a five-year-old phone.
+
+The usual answer is to add the signals into a score and block above a threshold. That is
+exactly the mistake. Points do not compose into proof. Two unrelated suspicions about an
+unusual but entirely real browser reach 100 just as readily as two well-founded ones, and
+the people who get caught are disproportionately the ones with the strongest reasons for
+their unusual setup.
+
+## What the guard does
+
+It runs **after** a rule has been selected and before its action is applied:
+
+```
+rule matches  →  guard  →  decision
+```
+
+Under the default `strict` mode there is exactly one question — *is there proof?* — and
+everything else is a downgrade:
+
+```ts
+if (!TERMINAL_ACTIONS.has(action)) return decision;   // not terminal, nothing to check
+if (mode === "aggressive") return decision;           // the guard is off
+if (assessment.certain) return decision;              // proven; the rule stands
+return downgrade(decision);                           // a guess. Not this.
+```
+
+Terminal actions are `block`, `drop` and `redirect` — the three that end the request
+without a way through. A downgraded decision becomes `fallbackAction` and records what it
+was:
+
+```ts
+{
+  action: "challenge",
+  downgradedFrom: "block",
+  downgradeReason:
+    "Strict mode permits a terminal action only on proven evidence. This request's " +
+    "verdict (suspected-bot, score 78) rests on probabilistic signals, any of which " +
+    "a real client can trip.",
+}
+```
+
+## Why it lives here and not in your rules
+
+It cannot be forgotten in a rule, cannot be bypassed by a cleverly-worded predicate, and
+does not depend on whoever wrote the rules understanding the certainty model. A rule that
+says `action: "block"` on a probabilistic match is not a bug to be caught in review — it
+is a thing the engine will simply decline to do.
+
+Turning that off is a single, explicit, greppable line.
+
+## The three modes
+
+```ts
+falsePositivePolicy: "strict"      // default
+```
+
+| Mode | A terminal action survives when |
+| ---- | ------------------------------- |
+| `strict` | There is `certain` evidence. Nothing else. |
+| `balanced` | There is proof, **or** the score clears `terminalScoreThreshold` with at least two *independent strong* signals. |
+| `aggressive` | Always. The guard is off. |
+
+`balanced` requires independence deliberately: three signals from the same family are one
+observation, and `independentStrongSignals` counts families rather than pieces. Even so,
+real people do trip two independent strong signals — a hardened browser behind a corporate
+proxy is the usual pair — so `balanced` will eventually deny somebody who should have been
+served. That is the trade it is; make it knowingly.
+
+`aggressive` means every rule does exactly what it says, on proof or on suspicion alike.
+The people it turns away first are the ones with the most unusual and most legitimate
+setups.
+
+## The fallback
+
+```ts
+fallbackAction: "challenge"   // default when a challenge is configured, otherwise "tag"
+```
+
+What a stopped rule becomes. **It cannot be terminal**, and that is enforced rather than
+advised: a `block` fallback would make every downgrade deny the request the downgrade
+existed to protect — and the decision would still be *recorded* as a guard stop, so the
+metric that exists to catch this would report success.
+
+## Watching it
+
+`bothandler_downgrades_total` is one of the two series worth alerting on. A rising count
+means your rules are asking to deny requests the evidence does not support — the guard
+working, and your policy needing attention.
+
+The [dashboard](../operations/dashboard.md) breaks guard stops down by the rule that
+overreached, which is usually enough to find the one at fault in a few seconds.
+
+## Changing it
+
+The guard is fixed at construction unless something opts in:
+
+```ts
+detector.updateGuard({ falsePositivePolicy: "balanced" }, { by: "ada@example.com" });
+```
+
+See [Runtime changes](../operations/runtime-changes.md). The dashboard exposes this behind
+a control flag separate from its rule editor, because "which rules exist" and "how far a
+rule may go" are different powers.
+
+## Related
+
+- [Evidence and certainty](evidence.md) — what "proof" means, precisely
+- [Actions](../policy/actions.md) — which are terminal and which are not
+- [Presets](../policy/presets.md) — every shipped policy keeps `strict`