@osqd/bothandlerjs 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (186) hide show
  1. package/CHANGELOG.md +516 -0
  2. package/LICENSE +102 -0
  3. package/README.md +308 -0
  4. package/bin/bothandlerjs.mjs +14 -0
  5. package/dist/actions/index.d.ts +22 -0
  6. package/dist/actions/types.d.ts +59 -0
  7. package/dist/adapters/fastify.d.ts +43 -0
  8. package/dist/adapters/fetch.d.ts +51 -0
  9. package/dist/adapters/index.cjs +723 -0
  10. package/dist/adapters/index.cjs.map +1 -0
  11. package/dist/adapters/index.d.ts +8 -0
  12. package/dist/adapters/index.js +692 -0
  13. package/dist/adapters/index.js.map +1 -0
  14. package/dist/adapters/koa.d.ts +27 -0
  15. package/dist/adapters/node.d.ts +29 -0
  16. package/dist/adapters/shared.d.ts +39 -0
  17. package/dist/audit.d.ts +172 -0
  18. package/dist/challenge/index.d.ts +176 -0
  19. package/dist/challenge/language.d.ts +57 -0
  20. package/dist/challenge/page.d.ts +42 -0
  21. package/dist/challenge/pow.d.ts +33 -0
  22. package/dist/challenge/token.d.ts +72 -0
  23. package/dist/cli.cjs +13388 -0
  24. package/dist/cli.cjs.map +1 -0
  25. package/dist/cli.d.ts +5 -0
  26. package/dist/cli.js +13358 -0
  27. package/dist/cli.js.map +1 -0
  28. package/dist/client/index.cjs +139 -0
  29. package/dist/client/index.cjs.map +1 -0
  30. package/dist/client/index.d.ts +63 -0
  31. package/dist/client/index.js +112 -0
  32. package/dist/client/index.js.map +1 -0
  33. package/dist/config.d.ts +316 -0
  34. package/dist/core.d.ts +443 -0
  35. package/dist/corpus/adversarial.d.ts +2 -0
  36. package/dist/corpus/advertising-email.d.ts +2 -0
  37. package/dist/corpus/ai-crawlers.d.ts +2 -0
  38. package/dist/corpus/benign-bots.d.ts +2 -0
  39. package/dist/corpus/cdn-gateways.d.ts +2 -0
  40. package/dist/corpus/cookies.d.ts +49 -0
  41. package/dist/corpus/crawlers-regional.d.ts +2 -0
  42. package/dist/corpus/crawlers-vertical.d.ts +2 -0
  43. package/dist/corpus/headers.d.ts +403 -0
  44. package/dist/corpus/humans-apps.d.ts +2 -0
  45. package/dist/corpus/humans-browsers.d.ts +2 -0
  46. package/dist/corpus/humans.d.ts +2 -0
  47. package/dist/corpus/index.cjs +4196 -0
  48. package/dist/corpus/index.cjs.map +1 -0
  49. package/dist/corpus/index.d.ts +37 -0
  50. package/dist/corpus/index.js +4132 -0
  51. package/dist/corpus/index.js.map +1 -0
  52. package/dist/corpus/infrastructure.d.ts +17 -0
  53. package/dist/corpus/libraries-extended.d.ts +2 -0
  54. package/dist/corpus/ranges.d.ts +31 -0
  55. package/dist/corpus/reputation.d.ts +21 -0
  56. package/dist/corpus/runner.d.ts +130 -0
  57. package/dist/corpus/schema.d.ts +185 -0
  58. package/dist/corpus/tooling.d.ts +2 -0
  59. package/dist/corpus/unwanted.d.ts +2 -0
  60. package/dist/crawler-ranges.d.ts +113 -0
  61. package/dist/dashboard/client.generated.d.ts +7 -0
  62. package/dist/dashboard/feed.d.ts +186 -0
  63. package/dist/dashboard/index.d.ts +5 -0
  64. package/dist/dashboard/page.d.ts +51 -0
  65. package/dist/dashboard/parse-request.d.ts +28 -0
  66. package/dist/dashboard/preview.d.ts +45 -0
  67. package/dist/dashboard/server.d.ts +70 -0
  68. package/dist/dashboard/types.d.ts +714 -0
  69. package/dist/detectors/accept-signature.d.ts +6 -0
  70. package/dist/detectors/browsing-coherence.d.ts +44 -0
  71. package/dist/detectors/cadence.d.ts +28 -0
  72. package/dist/detectors/clearance.d.ts +25 -0
  73. package/dist/detectors/client-hints.d.ts +19 -0
  74. package/dist/detectors/client-signals.d.ts +27 -0
  75. package/dist/detectors/crawl-breadth.d.ts +28 -0
  76. package/dist/detectors/crawler-verification.d.ts +39 -0
  77. package/dist/detectors/fetch-metadata.d.ts +20 -0
  78. package/dist/detectors/header-integrity.d.ts +26 -0
  79. package/dist/detectors/header-order.d.ts +27 -0
  80. package/dist/detectors/identity-rotation.d.ts +34 -0
  81. package/dist/detectors/index.d.ts +56 -0
  82. package/dist/detectors/ip-intelligence.d.ts +30 -0
  83. package/dist/detectors/known-bots.d.ts +94 -0
  84. package/dist/detectors/probe-signature.d.ts +50 -0
  85. package/dist/detectors/rate-anomaly.d.ts +36 -0
  86. package/dist/detectors/self-identified.d.ts +33 -0
  87. package/dist/detectors/session-integrity.d.ts +24 -0
  88. package/dist/detectors/tls-fingerprint.d.ts +34 -0
  89. package/dist/detectors/trap.d.ts +98 -0
  90. package/dist/detectors/types.d.ts +70 -0
  91. package/dist/detectors/ua-coherence.d.ts +32 -0
  92. package/dist/evidence.d.ts +45 -0
  93. package/dist/facts.d.ts +46 -0
  94. package/dist/index.cjs +9600 -0
  95. package/dist/index.cjs.map +1 -0
  96. package/dist/index.d.ts +63 -0
  97. package/dist/index.js +9444 -0
  98. package/dist/index.js.map +1 -0
  99. package/dist/internal/async.d.ts +16 -0
  100. package/dist/internal/clock.d.ts +17 -0
  101. package/dist/internal/crypto.d.ts +26 -0
  102. package/dist/internal/dns.d.ts +100 -0
  103. package/dist/internal/emitter.d.ts +18 -0
  104. package/dist/internal/http.d.ts +22 -0
  105. package/dist/internal/ip.d.ts +73 -0
  106. package/dist/internal/lru.d.ts +34 -0
  107. package/dist/internal/matcher.d.ts +41 -0
  108. package/dist/internal/pattern.d.ts +14 -0
  109. package/dist/internal/ua.d.ts +92 -0
  110. package/dist/metrics.d.ts +120 -0
  111. package/dist/notify/hub.d.ts +70 -0
  112. package/dist/notify/index.d.ts +7 -0
  113. package/dist/notify/redact.d.ts +48 -0
  114. package/dist/notify/sinks.d.ts +62 -0
  115. package/dist/notify/types.d.ts +56 -0
  116. package/dist/policy/index.d.ts +7 -0
  117. package/dist/policy/match.d.ts +20 -0
  118. package/dist/policy/policy.d.ts +123 -0
  119. package/dist/policy/presets.d.ts +161 -0
  120. package/dist/policy/types.d.ts +155 -0
  121. package/dist/robots.d.ts +83 -0
  122. package/dist/state.d.ts +161 -0
  123. package/dist/stores/index.d.ts +5 -0
  124. package/dist/stores/memory.d.ts +33 -0
  125. package/dist/stores/redis.d.ts +59 -0
  126. package/dist/stores/types.d.ts +37 -0
  127. package/dist/types.d.ts +250 -0
  128. package/docs/challenge/index.md +152 -0
  129. package/docs/challenge/localisation.md +105 -0
  130. package/docs/concepts/actors.md +113 -0
  131. package/docs/concepts/evidence.md +133 -0
  132. package/docs/concepts/the-guard.md +121 -0
  133. package/docs/concepts/threat-model.md +120 -0
  134. package/docs/concepts/verdicts.md +108 -0
  135. package/docs/course/01-first-assessment.md +178 -0
  136. package/docs/course/02-proof-and-suspicion.md +164 -0
  137. package/docs/course/03-verdicts-and-scores.md +160 -0
  138. package/docs/course/04-the-guard.md +200 -0
  139. package/docs/course/05-detectors.md +178 -0
  140. package/docs/course/06-identity.md +161 -0
  141. package/docs/course/07-actors.md +178 -0
  142. package/docs/course/08-traps.md +183 -0
  143. package/docs/course/09-rules.md +224 -0
  144. package/docs/course/10-actions-and-presets.md +216 -0
  145. package/docs/course/11-the-challenge.md +227 -0
  146. package/docs/course/12-going-live.md +211 -0
  147. package/docs/course/13-operating-it.md +238 -0
  148. package/docs/course/14-scaling.md +195 -0
  149. package/docs/course/15-extending.md +259 -0
  150. package/docs/course/16-proving-it.md +266 -0
  151. package/docs/course/index.md +112 -0
  152. package/docs/design/decisions.md +216 -0
  153. package/docs/detection/client-signals.md +96 -0
  154. package/docs/detection/detectors.md +376 -0
  155. package/docs/detection/index.md +134 -0
  156. package/docs/detection/signatures.md +111 -0
  157. package/docs/detection/verification.md +123 -0
  158. package/docs/detection/writing-a-detector.md +145 -0
  159. package/docs/index.md +100 -0
  160. package/docs/integration/adapters.md +138 -0
  161. package/docs/integration/client-ip.md +114 -0
  162. package/docs/integration/index.md +55 -0
  163. package/docs/integration/stores.md +76 -0
  164. package/docs/operations/audit.md +116 -0
  165. package/docs/operations/dashboard.md +725 -0
  166. package/docs/operations/index.md +101 -0
  167. package/docs/operations/metrics.md +74 -0
  168. package/docs/operations/notifications.md +72 -0
  169. package/docs/operations/runtime-changes.md +118 -0
  170. package/docs/policy/actions.md +196 -0
  171. package/docs/policy/index.md +84 -0
  172. package/docs/policy/presets.md +191 -0
  173. package/docs/policy/robots.md +128 -0
  174. package/docs/policy/rules.md +134 -0
  175. package/docs/reference/api.md +126 -0
  176. package/docs/reference/configuration.md +175 -0
  177. package/docs/start/choosing-a-policy.md +127 -0
  178. package/docs/start/first-integration.md +127 -0
  179. package/docs/start/installation.md +63 -0
  180. package/docs/start/upgrading.md +72 -0
  181. package/docs/testing/cli.md +119 -0
  182. package/docs/testing/corpus.md +129 -0
  183. package/docs/testing/index.md +61 -0
  184. package/docs/testing/replay.md +102 -0
  185. package/docs/testing/try-it.md +189 -0
  186. package/package.json +145 -0
@@ -0,0 +1,259 @@
1
+ # Lesson 15 — Extending it
2
+
3
+ **Goal:** add a detector, a signature, an action and a browser signal of your own — and
4
+ meet the certainty rule from the other side.
5
+
6
+ ← [Course](index.md) · Prev: [Scaling](14-scaling.md) · Next: [Proving it](16-proving-it.md)
7
+
8
+ ---
9
+
10
+ ## A detector of your own
11
+
12
+ A detector returns **evidence**, never a verdict. It has no idea what the policy will do
13
+ with what it finds, and that separation is what keeps the certainty model intact.
14
+
15
+ Serif's checkout is being hammered. Nothing shipped knows what a checkout is:
16
+
17
+ ```js
18
+ function checkoutVelocity() {
19
+ return {
20
+ id: "checkout-velocity",
21
+ description: "More checkout attempts in a minute than a person makes",
22
+ cost: "cheap",
23
+ stage: "always",
24
+ inspect(ctx) {
25
+ if (!ctx.facts.path.startsWith("/checkout")) return undefined;
26
+
27
+ const attempts = ctx.state.requestsWithin(60_000, ctx.facts.timestamp);
28
+ if (attempts < 5) return undefined;
29
+
30
+ return {
31
+ detector: "checkout-velocity",
32
+ summary: `${attempts} checkout attempts in a minute`,
33
+ direction: "bot",
34
+ certainty: "strong", // not `certain` — a shared address explains it too
35
+ botClass: "automation",
36
+ };
37
+ },
38
+ };
39
+ }
40
+
41
+ const detector = new BotHandler({ extraDetectors: [checkoutVelocity()] });
42
+ ```
43
+
44
+ ### Checkpoint
45
+
46
+ Six POSTs to `/checkout` from one address, 800 ms apart:
47
+
48
+ ```
49
+ checkout burst: score=66 verdict=suspected-bot
50
+ [strong] checkout-velocity: 6 checkout attempts in a minute
51
+ [weak] accept-signature: Accept-Language is a single bare tag ("en-GB") with no fallback chain
52
+ ```
53
+
54
+ ## The certainty rule, from the inside
55
+
56
+ Look again at that comment: `certainty: "strong"`, not `certain`.
57
+
58
+ Six checkout attempts in a minute is a *strong* signal and it is not proof. An office
59
+ behind one NAT address, a family sharing a connection, a customer whose payment kept
60
+ failing — all produce it. To mark it `certain` you would have to write a
61
+ `deterministicBasis` sentence explaining why no benign explanation exists, and you cannot,
62
+ because one does.
63
+
64
+ **That is the forcing function.** If you cannot write the sentence, your evidence is
65
+ `strong`. Outside production the library *enforces* it: `certain` evidence with no basis is
66
+ rejected. Inside production it downgrades to a warning instead, so a third-party detector
67
+ with a missing basis cannot take a live site down.
68
+
69
+ ```js
70
+ return {
71
+ detector: "checkout-velocity",
72
+ certainty: "certain",
73
+ deterministicBasis: "…", // you would have to justify it here, in writing
74
+ // …
75
+ };
76
+ ```
77
+
78
+ Try it without the basis and watch it get refused. That refusal is the library defending the
79
+ one guarantee it makes.
80
+
81
+ ## Cost and stage, honestly
82
+
83
+ Mark a detector `cost: "io"` if it touches the network or a shared store, and it runs
84
+ concurrently under a timeout. Mislabelling one would put an unbounded await on the request
85
+ path — so the engine times out any promise a `cheap` detector returns anyway.
86
+
87
+ Use `stage: "confirming"` when your work is only worth doing if something else already found
88
+ an identity to confirm.
89
+
90
+ ## Replacing a shipped detector
91
+
92
+ `detectors` replaces the whole set, so map over the default list:
93
+
94
+ ```js
95
+ import { defaultDetectors, probeSignatureDetector } from "@osqd/bothandlerjs";
96
+
97
+ new BotHandler({
98
+ detectors: defaultDetectors().map((d) =>
99
+ d.id === "probe-signature" ? probeSignatureDetector({ ignore: ["/wp-login.php"] }) : d,
100
+ ),
101
+ });
102
+ ```
103
+
104
+ ## A signature of your own
105
+
106
+ Serif's partner sends a nightly sync. You want it recognised and allowed by name:
107
+
108
+ ```js
109
+ const partner = {
110
+ id: "serif-partner-sync",
111
+ name: "Serif Partner Sync",
112
+ tokens: ["serifpartnersync"], // lowercase — see below
113
+ category: "library",
114
+ benign: true,
115
+ verification: { kind: "none" },
116
+ };
117
+
118
+ const detector = new BotHandler({
119
+ extraSignatures: [partner],
120
+ rules: [{ id: "partner-allow", match: { identity: "serif-partner-sync" }, action: "allow",
121
+ reason: "Our own partner integration." }],
122
+ });
123
+ ```
124
+
125
+ ### Checkpoint
126
+
127
+ ```
128
+ verdict=confirmed-bot class=http-client identity=serif-partner-sync certain=true
129
+ [certain] self-identified: User-Agent identifies Serif Partner Sync
130
+ decision: allow via rule "partner-allow"
131
+ ```
132
+
133
+ **`tokens` must be lowercase.** Matching lowercases the User-Agent first, so an uppercase
134
+ token silently never matches — no error, no warning, just a signature that does nothing. It
135
+ is the single easiest mistake to make here, and it costs an afternoon.
136
+
137
+ `verification` is required. Use `{ kind: "none" }` for a client with no verifiable identity,
138
+ `{ kind: "fcrdns", domains: [...] }` where the operator publishes reverse DNS, or
139
+ `{ kind: "ip-ranges" }` where they publish addresses.
140
+
141
+ Set `conclusive: false` with a `caveat` when a token can legitimately appear on a request a
142
+ person made. Electron is the motivating case: that User-Agent comes from VS Code's browser,
143
+ Slack, Discord and Postman — real Chromium instances with a human driving them.
144
+
145
+ ## An action of your own
146
+
147
+ ```js
148
+ import { defineHandler } from "@osqd/bothandlerjs";
149
+
150
+ const emptyResults = defineHandler({
151
+ id: "empty-results",
152
+ description: "Serves an empty result set rather than an error",
153
+ execute: ({ assessment }) => ({
154
+ kind: "respond",
155
+ status: 200,
156
+ headers: { "content-type": "application/json" },
157
+ body: JSON.stringify({ results: [] }),
158
+ }),
159
+ });
160
+
161
+ new BotHandler({
162
+ handlers: [emptyResults],
163
+ rules: [{ id: "shadow", match: { botClass: "scraper" }, action: "custom",
164
+ params: { handler: "empty-results" } }],
165
+ });
166
+ ```
167
+
168
+ Return `{ kind: "continue" }` — optionally with `requestHeaders`, `responseHeaders` or
169
+ `delayMs` — or `{ kind: "respond", status, headers, body }`, or `{ kind: "drop" }`. It may
170
+ be async.
171
+
172
+ **Remember the guard does not apply here.** It cannot know what your handler does. A handler
173
+ that denies service is a decision you own entirely — which is fine, as long as you know you
174
+ are making it.
175
+
176
+ A handler named in a rule but not registered serves the request and warns.
177
+
178
+ ## Browser signals
179
+
180
+ An optional page script reports what only the browser can see — `navigator.webdriver`,
181
+ automation properties, timing:
182
+
183
+ ```js
184
+ import { renderClientScript } from "@osqd/bothandlerjs/client";
185
+
186
+ // `endpoint` is required: it is your route, and it should be cheap and rate-limited.
187
+ app.get("/", (req, res) =>
188
+ res.send(`<body>…${renderClientScript({ endpoint: "/__signals", nonce: res.locals.nonce })}</body>`),
189
+ );
190
+ ```
191
+
192
+ It posts back to that endpoint, and the `client-signals` detector reads what arrives. Supply
193
+ the `nonce` if your Content-Security-Policy needs one.
194
+
195
+ **It is capped at `moderate`, permanently, whatever it reports.** The browser is the one
196
+ place an adversary has complete control: `navigator.webdriver` proves what the client *chose
197
+ to report*, and a scraper that wants to lie about it has already won that argument. Treating
198
+ it as conclusive would move the whole guarantee inside the attacker's process.
199
+
200
+ So it is real evidence, weighted honestly, and it can never on its own deny anybody.
201
+
202
+ ## Exercise
203
+
204
+ Serif wants to catch scripted gift-card redemption: many distinct codes tried from one
205
+ actor, few succeeding. Write the detector, and decide its tier.
206
+
207
+ <details>
208
+ <summary>Answer</summary>
209
+
210
+ ```js
211
+ function giftCardProbing(redemptions) {
212
+ return {
213
+ id: "gift-card-probing",
214
+ description: "Many distinct gift-card codes tried from one actor",
215
+ cost: "cheap",
216
+ stage: "always",
217
+ inspect(ctx) {
218
+ if (ctx.facts.path !== "/gift-cards/redeem") return undefined;
219
+ const tried = redemptions.recentFailures(ctx.actorKey, 10 * 60_000);
220
+ if (tried < 12) return undefined;
221
+ return {
222
+ detector: "gift-card-probing",
223
+ summary: `${tried} failed gift-card redemptions in ten minutes`,
224
+ direction: "bot",
225
+ certainty: "strong",
226
+ botClass: "automation",
227
+ };
228
+ },
229
+ };
230
+ }
231
+ ```
232
+
233
+ **`strong`, not `certain`.** Twelve failed redemptions is a strong signal about a *shared
234
+ address* as much as about a script — a busy office at Christmas produces it. You cannot
235
+ write the basis sentence, so it is not certain.
236
+
237
+ **`cheap`, only if `redemptions` is in memory.** If it queries a database, it is `io` — and
238
+ mislabelling it would put a database round trip on the request path of every checkout.
239
+
240
+ And note that a `strong` signal is enough to challenge or rate-limit, which is the right
241
+ response to card probing anyway. You lose nothing by being honest about the tier.
242
+ </details>
243
+
244
+ ## What you learned
245
+
246
+ - A detector returns evidence, never a verdict
247
+ - The `deterministicBasis` requirement is a forcing function: if you cannot write it, it is
248
+ `strong`
249
+ - `cost` must be honest, or you put an await on the request path
250
+ - Signature `tokens` must be lowercase, and the failure is silent
251
+ - Custom handlers sit outside the guard, deliberately
252
+ - Browser signals are capped at `moderate` for ever, because that is the attacker's process
253
+
254
+ ## Reference
255
+
256
+ - [Writing a detector](../detection/writing-a-detector.md) · [Signatures](../detection/signatures.md)
257
+ - [Browser signals](../detection/client-signals.md) · [Actions](../policy/actions.md)
258
+
259
+ Next: [Proving it, and the capstone](16-proving-it.md).
@@ -0,0 +1,266 @@
1
+ # Lesson 16 — Proving it, and the capstone
2
+
3
+ **Goal:** find out who your policy would hurt *before* it hurts them, then assemble
4
+ everything into a Serif you can defend.
5
+
6
+ ← [Course](index.md) · Prev: [Extending it](15-extending.md)
7
+
8
+ ---
9
+
10
+ ## The question this library is organised around
11
+
12
+ *If I point this configuration at the actual internet, who gets hurt?*
13
+
14
+ Three tools answer it, and they answer different halves.
15
+
16
+ ## `check` — against 526 shapes of real traffic
17
+
18
+ ```bash
19
+ npx @osqd/bothandlerjs check --preset protect-content
20
+ ```
21
+
22
+ ```
23
+ protect-content against 522 shapes of real traffic
24
+
25
+ human 181 cases 3 tag, 175 allow, 3 challenge
26
+ benign-bot 144 cases 6 allow, 86 tag, 48 block, 2 challenge, 2 rate-limit
27
+ declared-bot 32 cases 18 tag, 2 rate-limit, 3 allow, 9 block
28
+ unwanted-bot 105 cases 8 tag, 11 rate-limit, 3 block, 70 challenge, 13 allow
29
+ hostile 27 cases 6 challenge, 17 block, 4 allow
30
+ infrastructure 33 cases 7 allow, 12 tag, 14 challenge
31
+
32
+ No case marked as a person was denied service.
33
+ ```
34
+
35
+ **It exits non-zero if any case marked `human` is denied**, which is what makes it a CI step
36
+ rather than a report.
37
+
38
+ The corpus is 526 cases with provenance: 182 of them people — 30 browser profiles, 40 in-app
39
+ WebViews, Tor, screen readers, IE11, a car's infotainment screen, corporate proxies, CGNAT,
40
+ an author signing in at `/wp-login.php`. Header order is reproduced rather than invented, and
41
+ DNS is controlled rather than mocked away, so *"the operator's DNS disproves this"* and
42
+ *"our resolver was unhappy"* can be told apart.
43
+
44
+ `--audience human` shows only the part that matters. `--strict` also fails on differing
45
+ actions.
46
+
47
+ ### Against *your* configuration
48
+
49
+ The corpus is a published entry point, not a test fixture:
50
+
51
+ ```js
52
+ import { runCorpus } from "@osqd/bothandlerjs/corpus";
53
+
54
+ const scorecard = await runCorpus({
55
+ create: ({ resolver, clock }) => new BotHandler({ ...serifConfig, resolver, clock }),
56
+ assertActions: false, // your actions are yours; the invariants are not
57
+ });
58
+
59
+ if (scorecard.falsePositives.length > 0) {
60
+ throw new Error(`Serif's policy denies ${scorecard.falsePositives.length} people`);
61
+ }
62
+ ```
63
+
64
+ Put that in your test suite. It runs offline and deterministically.
65
+
66
+ ## `replay` — against traffic that is actually yours
67
+
68
+ The corpus knows what the internet looks like. Only your logs know what *your* visitors look
69
+ like.
70
+
71
+ ```bash
72
+ npx @osqd/bothandlerjs replay /var/log/nginx/access.log --preset protect-content
73
+ ```
74
+
75
+ ```
76
+ 116 request(s) would have been DENIED, in 1 distinct kind(s). Read them.
77
+ ─────────────────────────────────────────────────────────────────────
78
+ 116x block by rule "scanner-block" — confirmed-bot, proven
79
+ sqlmap/1.7.2#stable (https://sqlmap.org)
80
+ [certain] self-identified: User-Agent identifies sqlmap
81
+ ```
82
+
83
+ That block is the point of the exercise. If any of those is a person, the policy is wrong —
84
+ and you found out from a log file rather than from a support ticket.
85
+
86
+ **Prefer JSON Lines over CLF**, and the reason is the rule from [lesson 2](02-proof-and-suspicion.md):
87
+
88
+ > A header missing from a *record* is not a header missing from the *request*.
89
+
90
+ An nginx line carries the User-Agent and the Referer and nothing else. Detectors that reason
91
+ from absence would fire on everything, so `partialHeaders` marks a header-poor source and
92
+ they stand down. A CLF replay therefore **under-reports**, and its silence is not a clean
93
+ bill of health.
94
+
95
+ ```js
96
+ createFacts({ method, url, headers, ip, partialHeaders: true });
97
+ ```
98
+
99
+ ## `explain` — one request, by ticket
100
+
101
+ ```bash
102
+ npx @osqd/bothandlerjs explain "curl/8.4.0"
103
+ pbpaste | npx @osqd/bothandlerjs explain --preset protect-data
104
+ ```
105
+
106
+ A User-Agent, a curl command out of devtools, or a raw header block. It runs as a **dry
107
+ run** — nothing recorded, so asking does not change the answer:
108
+
109
+ ```js
110
+ const assessment = await detector.assess(facts, { record: false });
111
+ ```
112
+
113
+ No counter moves, no actor state changes, no event fires. It has no history by construction,
114
+ so what it answers is *what would this look like as a first request* — which is what a
115
+ ticket is asking anyway.
116
+
117
+ ## The order to use them in
118
+
119
+ 1. **`explain`** — one request, when somebody complains
120
+ 2. **`check`** — in CI, on every change
121
+ 3. **`replay`** — before enforcing anything, over a real week
122
+ 4. **`monitor-only` in production** — for a week, watching the dashboard
123
+
124
+ Step 4 is not optional and the first three do not replace it.
125
+
126
+ ---
127
+
128
+ # Capstone: Serif
129
+
130
+ Everything from sixteen lessons, in one configuration.
131
+
132
+ ```js
133
+ import express from "express";
134
+ import { BotHandler, RedisStore, declineAiTraining, robotsFromRules, renderTrapLink } from "@osqd/bothandlerjs";
135
+ import { botHandler } from "@osqd/bothandlerjs/adapters";
136
+
137
+ const store = new RedisStore(redis);
138
+ const proxy = { trustProxy: true, trustedProxies: ["10.0.0.0/8"] }; // lesson 12
139
+
140
+ const site = new BotHandler({
141
+ preset: "decline-ai-training", // lesson 10 — the business decision
142
+ proxy, // lesson 12 — the dangerous setting
143
+ store, // lesson 14 — nonces and rate limits
144
+ shareConfirmations: true, // lesson 14 — proof travels
145
+ actorKey: (facts) => facts.session ?? facts.ip, // lesson 7
146
+ isHuman: (facts) => Boolean(sessions.get(facts.session ?? "")?.authenticated), // lesson 12
147
+ allowlist: ["10.0.0.0/8"], // monitors and CI
148
+ ignorePaths: ["/healthz", "/metrics"],
149
+ extraDetectors: [checkoutVelocity()], // lesson 15
150
+ extraSignatures: [partner], // lesson 15
151
+ challenge: { // lesson 11
152
+ secrets: [process.env.SERIF_CHALLENGE_SECRET, process.env.SERIF_CHALLENGE_SECRET_PREVIOUS].filter(Boolean),
153
+ contactHtml: '<p>Locked out? Email <a href="mailto:help@serif.example">help@serif.example</a>.</p>',
154
+ translations: { "pt-BR": { lang: "pt-BR", title: "Verificando seu navegador" }, ja: { lang: "ja", title: "ブラウザーを確認しています" } },
155
+ },
156
+ rules: [ // lesson 9 — yours run before the preset's
157
+ { id: "partner-allow", match: { identity: "serif-partner-sync" }, action: "allow",
158
+ reason: "Our own nightly partner sync." },
159
+ { id: "persistent-refusal", match: { minUnsolvedChallenges: 5 }, action: "block",
160
+ reason: "Issued five challenges, finished none." },
161
+ ],
162
+ audit: { windowMs: 5 * 60_000, baselineMs: 60 * 60_000, minSamples: 50 }, // lesson 13
163
+ onAnomaly: (a) => pager.send(a.severity, a.summary),
164
+ onDowngrade: ({ decision }) => metrics.increment("bot.downgrade", { rule: decision.rule }),
165
+ onWarning: (message) => log.warn({ message }, "bothandler"),
166
+ });
167
+
168
+ // Auth routes get their own policy, mounted only there. lesson 10
169
+ const auth = new BotHandler({ preset: "protect-auth", proxy, store });
170
+
171
+ const app = express();
172
+ app.use(botHandler(site));
173
+ app.use("/login", botHandler(auth));
174
+ app.use("/checkout", botHandler(auth));
175
+
176
+ // Say out loud what the policy does. lesson 10
177
+ const { robotsTxt, unreadable } = robotsFromRules(declineAiTraining(), {
178
+ disallowPaths: ["/internal/export.csv"],
179
+ sitemap: "https://serif.example/sitemap.xml",
180
+ });
181
+ if (unreadable.length > 0) log.warn({ unreadable }, "not reflected in robots.txt");
182
+ app.get("/robots.txt", (_req, res) => res.type("text/plain").send(robotsTxt));
183
+
184
+ // A trap no person can reach. lesson 8
185
+ app.get("/", (_req, res) => res.send(`<main>…</main>${renderTrapLink("/internal/export.csv")}`));
186
+
187
+ // Watch it. lesson 13
188
+ app.get("/internal/metrics", (_req, res) => res.type("text/plain").send(site.prometheus()));
189
+ await site.serveDashboard({ port: 9674, title: "serif", auth: { username: "ops", password: process.env.DASHBOARD_PASSWORD } });
190
+ ```
191
+
192
+ And the test that stops it hurting anybody:
193
+
194
+ ```js
195
+ import { runCorpus } from "@osqd/bothandlerjs/corpus";
196
+
197
+ it("never denies a person", async () => {
198
+ const scorecard = await runCorpus({
199
+ create: ({ resolver, clock }) => new BotHandler({ ...serifConfig, resolver, clock }),
200
+ assertActions: false,
201
+ });
202
+ expect(scorecard.falsePositives).toEqual([]);
203
+ });
204
+ ```
205
+
206
+ ## Final exercise
207
+
208
+ Answer these without looking anything up. They are the whole course.
209
+
210
+ 1. A client scores 99 from probabilistic signals. Can a rule block it? Why?
211
+ 2. Your policy denies 4% of traffic. What do you check first?
212
+ 3. `bothandler_downgrades_total` doubled after a deploy. What happened?
213
+ 4. `challenge-solve-rate` is 96%. Is that good?
214
+ 5. You add a detector that catches a scraper perfectly. What tier is its evidence?
215
+
216
+ <details>
217
+ <summary>Answers</summary>
218
+
219
+ 1. **No.** The score is not what gates a terminal action — `certain` is. Ninety-nine
220
+ probabilistic points are ninety-nine things a real person can trip, and the guard
221
+ substitutes something recoverable. [Lesson 4](04-the-guard.md).
222
+
223
+ 2. **Who the 4% are.** Run `replay` over the log and read every denial. A `denial-spike`
224
+ anomaly is not evidence they were bots — it is evidence that something changed, and the
225
+ expensive possibility is that it changed about people. [Lesson 13](13-operating-it.md).
226
+
227
+ 3. **Your policy now asks for more than the evidence supports.** Either a rule changed, or
228
+ the traffic did. The guard absorbed the difference, which is why nobody was wrongly
229
+ refused — but the two have drifted apart and the number is telling you so.
230
+ [Lesson 4](04-the-guard.md).
231
+
232
+ 4. **No — it is the bad direction.** A proof of work is trivial for a browser and trivial
233
+ for a competent scraper. When nearly everything challenged passes, the challenges are not
234
+ filtering bots out, they are taxing people. [Lesson 13](13-operating-it.md).
235
+
236
+ 5. **`strong`, almost certainly.** Ask whether you can write a `deterministicBasis`
237
+ sentence explaining why no benign explanation exists. Unless the client *declared*
238
+ itself, *contradicted* itself, walked into a *trap*, was *refuted by an external
239
+ authority*, or committed a *protocol violation* — you cannot, and it is not certain.
240
+ [Lesson 15](15-extending.md).
241
+ </details>
242
+
243
+ ## Where to go now
244
+
245
+ - **[The threat model](../concepts/threat-model.md).** Read it before you promise anybody
246
+ anything. It names the two rungs of the evasion ladder that defeat this library, and it is
247
+ the most useful page in the documentation once you know how everything works.
248
+ - **[Design decisions](../design/decisions.md).** Fifteen choices with what each one cost.
249
+ - **[The reference](../index.md).** Thirty pages, one per question.
250
+
251
+ ## What you learned in this course
252
+
253
+ - Proof and suspicion are different things, and only one of them may close a door
254
+ - Twenty detectors, each with a ceiling it may not exceed
255
+ - A name is a claim until an external authority confirms it
256
+ - Behaviour over time catches what a single request cannot, and stops at a paced scraper
257
+ - First match wins, and order is the policy
258
+ - Rank actions by what they cost somebody who did nothing wrong
259
+ - The challenge imposes cost; it proves neither identity nor humanity
260
+ - The client IP is the setting that is silent when it is wrong
261
+ - Alert on downgrades; read the score histogram before moving a threshold
262
+ - Proof travels between replicas; suspicion stays home
263
+ - If you cannot write why it admits no benign explanation, it is not certain
264
+ - Find out who your policy hurts before it hurts them
265
+
266
+ ← [Back to the course](index.md)
@@ -0,0 +1,112 @@
1
+ # The BotHandler course
2
+
3
+ Sixteen lessons that build one real integration, from a first assessment to a policy you
4
+ can defend.
5
+
6
+ ← [Documentation](../index.md)
7
+
8
+ ---
9
+
10
+ ## What this is
11
+
12
+ The [reference documentation](../index.md) answers *"how does X work?"*. This answers
13
+ *"what do I do, and in what order?"* — every capability of the library, taught in the
14
+ order that makes each one make sense, with something to run at every step.
15
+
16
+ It is written to be worked through rather than read. Every lesson has code you paste and
17
+ output you can check yours against, and every checkpoint in it was produced by actually
18
+ running the code rather than by imagining what it would print.
19
+
20
+ **Time:** about four hours to do properly. Lessons 1–4 are the foundation and are worth
21
+ slowing down for; everything after them assumes you have those.
22
+
23
+ ## Who it is for
24
+
25
+ A TypeScript or JavaScript developer who has a site or an API and a bot problem — or who
26
+ suspects they are about to have one. You need no background in bot detection. You do need
27
+ to be comfortable running `node` and reading a stack trace.
28
+
29
+ ## The running example
30
+
31
+ You are building **Serif**, an online bookshop. It has a catalogue worth scraping, a
32
+ checkout worth attacking, and readers worth not annoying. Every lesson adds one thing to
33
+ Serif, and by lesson 16 you have a complete, tested, production-shaped configuration.
34
+
35
+ ## Set up once
36
+
37
+ ```bash
38
+ mkdir serif && cd serif
39
+ npm init -y && npm pkg set type=module
40
+ npm install @osqd/bothandlerjs
41
+ ```
42
+
43
+ Every lesson's code goes in a file you run with `node`. Nothing needs a server until
44
+ lesson 11.
45
+
46
+ ---
47
+
48
+ ## Part 1 — The ideas everything rests on
49
+
50
+ | | | |
51
+ |-|-|-|
52
+ | 1 | [Your first assessment](01-first-assessment.md) | Turn a request into a verdict, and read what comes back. |
53
+ | 2 | [Proof and suspicion](02-proof-and-suspicion.md) | The distinction the whole library is built on. **The most important lesson here.** |
54
+ | 3 | [Verdicts, classes and scores](03-verdicts-and-scores.md) | Four fields describe a client. Which one should you act on? |
55
+ | 4 | [The safety guard](04-the-guard.md) | The mechanism that stops a guess closing a door. |
56
+
57
+ ## Part 2 — Detection
58
+
59
+ | | | |
60
+ |-|-|-|
61
+ | 5 | [The detectors](05-detectors.md) | All twenty: what each reads, what each costs, what each may conclude. |
62
+ | 6 | [Identity and verification](06-identity.md) | Recognising a crawler by name — and proving or refuting the claim. |
63
+ | 7 | [Actors and behaviour](07-actors.md) | Who "the same client" is, and what watching one over time tells you. |
64
+ | 8 | [Traps](08-traps.md) | The one detector that needs no statistics, and how to lay one properly. |
65
+
66
+ ## Part 3 — Deciding what to do
67
+
68
+ | | | |
69
+ |-|-|-|
70
+ | 9 | [Rules](09-rules.md) | Matching requests, in first-match-wins order. |
71
+ | 10 | [Actions and presets](10-actions-and-presets.md) | Ten responses, ordered by what each costs a person who did nothing wrong. |
72
+ | 11 | [The challenge](11-the-challenge.md) | Proof of work: what it buys, what it cannot, and who it must not exclude. |
73
+
74
+ ## Part 4 — Production
75
+
76
+ | | | |
77
+ |-|-|-|
78
+ | 12 | [Going live](12-going-live.md) | Adapters, and the one setting that is dangerous to get wrong. |
79
+ | 13 | [Operating it](13-operating-it.md) | The dashboard, metrics, the traffic audit, notifications. |
80
+ | 14 | [Scaling and changing it live](14-scaling.md) | Several replicas, shared state, and editing policy without a deploy. |
81
+
82
+ ## Part 5 — Making it yours, and proving it
83
+
84
+ | | | |
85
+ |-|-|-|
86
+ | 15 | [Extending it](15-extending.md) | Your own detector, action, signatures and browser signals. |
87
+ | 16 | [Proving it, and the capstone](16-proving-it.md) | The corpus, log replay, and a finished Serif you can defend. |
88
+
89
+ ---
90
+
91
+ ## How to get the most out of it
92
+
93
+ **Type the code, do not paste it.** The examples are short on purpose.
94
+
95
+ **Do the exercises before reading the answer.** Each one has a checkpoint so you know
96
+ whether you got it.
97
+
98
+ **When something surprises you, chase it.** Every lesson ends with links into the
99
+ reference documentation for the thing you just used. The surprises are usually where the
100
+ library is protecting you from something.
101
+
102
+ ## A promise, and its consequence
103
+
104
+ One sentence holds this library together, and it will explain most of what surprises you:
105
+
106
+ > **Nothing is ever denied service on the strength of a guess.**
107
+
108
+ You will meet it first in [lesson 4](04-the-guard.md), where a rule you wrote asking to
109
+ block somebody quietly does something gentler instead. That is not a bug, and the lesson
110
+ explains why it is the most valuable behaviour here.
111
+
112
+ Start with [lesson 1](01-first-assessment.md).