@osqd/bothandlerjs 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +516 -0
- package/LICENSE +102 -0
- package/README.md +308 -0
- package/bin/bothandlerjs.mjs +14 -0
- package/dist/actions/index.d.ts +22 -0
- package/dist/actions/types.d.ts +59 -0
- package/dist/adapters/fastify.d.ts +43 -0
- package/dist/adapters/fetch.d.ts +51 -0
- package/dist/adapters/index.cjs +723 -0
- package/dist/adapters/index.cjs.map +1 -0
- package/dist/adapters/index.d.ts +8 -0
- package/dist/adapters/index.js +692 -0
- package/dist/adapters/index.js.map +1 -0
- package/dist/adapters/koa.d.ts +27 -0
- package/dist/adapters/node.d.ts +29 -0
- package/dist/adapters/shared.d.ts +39 -0
- package/dist/audit.d.ts +172 -0
- package/dist/challenge/index.d.ts +176 -0
- package/dist/challenge/language.d.ts +57 -0
- package/dist/challenge/page.d.ts +42 -0
- package/dist/challenge/pow.d.ts +33 -0
- package/dist/challenge/token.d.ts +72 -0
- package/dist/cli.cjs +13388 -0
- package/dist/cli.cjs.map +1 -0
- package/dist/cli.d.ts +5 -0
- package/dist/cli.js +13358 -0
- package/dist/cli.js.map +1 -0
- package/dist/client/index.cjs +139 -0
- package/dist/client/index.cjs.map +1 -0
- package/dist/client/index.d.ts +63 -0
- package/dist/client/index.js +112 -0
- package/dist/client/index.js.map +1 -0
- package/dist/config.d.ts +316 -0
- package/dist/core.d.ts +443 -0
- package/dist/corpus/adversarial.d.ts +2 -0
- package/dist/corpus/advertising-email.d.ts +2 -0
- package/dist/corpus/ai-crawlers.d.ts +2 -0
- package/dist/corpus/benign-bots.d.ts +2 -0
- package/dist/corpus/cdn-gateways.d.ts +2 -0
- package/dist/corpus/cookies.d.ts +49 -0
- package/dist/corpus/crawlers-regional.d.ts +2 -0
- package/dist/corpus/crawlers-vertical.d.ts +2 -0
- package/dist/corpus/headers.d.ts +403 -0
- package/dist/corpus/humans-apps.d.ts +2 -0
- package/dist/corpus/humans-browsers.d.ts +2 -0
- package/dist/corpus/humans.d.ts +2 -0
- package/dist/corpus/index.cjs +4196 -0
- package/dist/corpus/index.cjs.map +1 -0
- package/dist/corpus/index.d.ts +37 -0
- package/dist/corpus/index.js +4132 -0
- package/dist/corpus/index.js.map +1 -0
- package/dist/corpus/infrastructure.d.ts +17 -0
- package/dist/corpus/libraries-extended.d.ts +2 -0
- package/dist/corpus/ranges.d.ts +31 -0
- package/dist/corpus/reputation.d.ts +21 -0
- package/dist/corpus/runner.d.ts +130 -0
- package/dist/corpus/schema.d.ts +185 -0
- package/dist/corpus/tooling.d.ts +2 -0
- package/dist/corpus/unwanted.d.ts +2 -0
- package/dist/crawler-ranges.d.ts +113 -0
- package/dist/dashboard/client.generated.d.ts +7 -0
- package/dist/dashboard/feed.d.ts +186 -0
- package/dist/dashboard/index.d.ts +5 -0
- package/dist/dashboard/page.d.ts +51 -0
- package/dist/dashboard/parse-request.d.ts +28 -0
- package/dist/dashboard/preview.d.ts +45 -0
- package/dist/dashboard/server.d.ts +70 -0
- package/dist/dashboard/types.d.ts +714 -0
- package/dist/detectors/accept-signature.d.ts +6 -0
- package/dist/detectors/browsing-coherence.d.ts +44 -0
- package/dist/detectors/cadence.d.ts +28 -0
- package/dist/detectors/clearance.d.ts +25 -0
- package/dist/detectors/client-hints.d.ts +19 -0
- package/dist/detectors/client-signals.d.ts +27 -0
- package/dist/detectors/crawl-breadth.d.ts +28 -0
- package/dist/detectors/crawler-verification.d.ts +39 -0
- package/dist/detectors/fetch-metadata.d.ts +20 -0
- package/dist/detectors/header-integrity.d.ts +26 -0
- package/dist/detectors/header-order.d.ts +27 -0
- package/dist/detectors/identity-rotation.d.ts +34 -0
- package/dist/detectors/index.d.ts +56 -0
- package/dist/detectors/ip-intelligence.d.ts +30 -0
- package/dist/detectors/known-bots.d.ts +94 -0
- package/dist/detectors/probe-signature.d.ts +50 -0
- package/dist/detectors/rate-anomaly.d.ts +36 -0
- package/dist/detectors/self-identified.d.ts +33 -0
- package/dist/detectors/session-integrity.d.ts +24 -0
- package/dist/detectors/tls-fingerprint.d.ts +34 -0
- package/dist/detectors/trap.d.ts +98 -0
- package/dist/detectors/types.d.ts +70 -0
- package/dist/detectors/ua-coherence.d.ts +32 -0
- package/dist/evidence.d.ts +45 -0
- package/dist/facts.d.ts +46 -0
- package/dist/index.cjs +9600 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.ts +63 -0
- package/dist/index.js +9444 -0
- package/dist/index.js.map +1 -0
- package/dist/internal/async.d.ts +16 -0
- package/dist/internal/clock.d.ts +17 -0
- package/dist/internal/crypto.d.ts +26 -0
- package/dist/internal/dns.d.ts +100 -0
- package/dist/internal/emitter.d.ts +18 -0
- package/dist/internal/http.d.ts +22 -0
- package/dist/internal/ip.d.ts +73 -0
- package/dist/internal/lru.d.ts +34 -0
- package/dist/internal/matcher.d.ts +41 -0
- package/dist/internal/pattern.d.ts +14 -0
- package/dist/internal/ua.d.ts +92 -0
- package/dist/metrics.d.ts +120 -0
- package/dist/notify/hub.d.ts +70 -0
- package/dist/notify/index.d.ts +7 -0
- package/dist/notify/redact.d.ts +48 -0
- package/dist/notify/sinks.d.ts +62 -0
- package/dist/notify/types.d.ts +56 -0
- package/dist/policy/index.d.ts +7 -0
- package/dist/policy/match.d.ts +20 -0
- package/dist/policy/policy.d.ts +123 -0
- package/dist/policy/presets.d.ts +161 -0
- package/dist/policy/types.d.ts +155 -0
- package/dist/robots.d.ts +83 -0
- package/dist/state.d.ts +161 -0
- package/dist/stores/index.d.ts +5 -0
- package/dist/stores/memory.d.ts +33 -0
- package/dist/stores/redis.d.ts +59 -0
- package/dist/stores/types.d.ts +37 -0
- package/dist/types.d.ts +250 -0
- package/docs/challenge/index.md +152 -0
- package/docs/challenge/localisation.md +105 -0
- package/docs/concepts/actors.md +113 -0
- package/docs/concepts/evidence.md +133 -0
- package/docs/concepts/the-guard.md +121 -0
- package/docs/concepts/threat-model.md +120 -0
- package/docs/concepts/verdicts.md +108 -0
- package/docs/course/01-first-assessment.md +178 -0
- package/docs/course/02-proof-and-suspicion.md +164 -0
- package/docs/course/03-verdicts-and-scores.md +160 -0
- package/docs/course/04-the-guard.md +200 -0
- package/docs/course/05-detectors.md +178 -0
- package/docs/course/06-identity.md +161 -0
- package/docs/course/07-actors.md +178 -0
- package/docs/course/08-traps.md +183 -0
- package/docs/course/09-rules.md +224 -0
- package/docs/course/10-actions-and-presets.md +216 -0
- package/docs/course/11-the-challenge.md +227 -0
- package/docs/course/12-going-live.md +211 -0
- package/docs/course/13-operating-it.md +238 -0
- package/docs/course/14-scaling.md +195 -0
- package/docs/course/15-extending.md +259 -0
- package/docs/course/16-proving-it.md +266 -0
- package/docs/course/index.md +112 -0
- package/docs/design/decisions.md +216 -0
- package/docs/detection/client-signals.md +96 -0
- package/docs/detection/detectors.md +376 -0
- package/docs/detection/index.md +134 -0
- package/docs/detection/signatures.md +111 -0
- package/docs/detection/verification.md +123 -0
- package/docs/detection/writing-a-detector.md +145 -0
- package/docs/index.md +100 -0
- package/docs/integration/adapters.md +138 -0
- package/docs/integration/client-ip.md +114 -0
- package/docs/integration/index.md +55 -0
- package/docs/integration/stores.md +76 -0
- package/docs/operations/audit.md +116 -0
- package/docs/operations/dashboard.md +725 -0
- package/docs/operations/index.md +101 -0
- package/docs/operations/metrics.md +74 -0
- package/docs/operations/notifications.md +72 -0
- package/docs/operations/runtime-changes.md +118 -0
- package/docs/policy/actions.md +196 -0
- package/docs/policy/index.md +84 -0
- package/docs/policy/presets.md +191 -0
- package/docs/policy/robots.md +128 -0
- package/docs/policy/rules.md +134 -0
- package/docs/reference/api.md +126 -0
- package/docs/reference/configuration.md +175 -0
- package/docs/start/choosing-a-policy.md +127 -0
- package/docs/start/first-integration.md +127 -0
- package/docs/start/installation.md +63 -0
- package/docs/start/upgrading.md +72 -0
- package/docs/testing/cli.md +119 -0
- package/docs/testing/corpus.md +129 -0
- package/docs/testing/index.md +61 -0
- package/docs/testing/replay.md +102 -0
- package/docs/testing/try-it.md +189 -0
- package/package.json +145 -0
|
@@ -0,0 +1,259 @@
|
|
|
1
|
+
# Lesson 15 — Extending it
|
|
2
|
+
|
|
3
|
+
**Goal:** add a detector, a signature, an action and a browser signal of your own — and
|
|
4
|
+
meet the certainty rule from the other side.
|
|
5
|
+
|
|
6
|
+
← [Course](index.md) · Prev: [Scaling](14-scaling.md) · Next: [Proving it](16-proving-it.md)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## A detector of your own
|
|
11
|
+
|
|
12
|
+
A detector returns **evidence**, never a verdict. It has no idea what the policy will do
|
|
13
|
+
with what it finds, and that separation is what keeps the certainty model intact.
|
|
14
|
+
|
|
15
|
+
Serif's checkout is being hammered. Nothing shipped knows what a checkout is:
|
|
16
|
+
|
|
17
|
+
```js
|
|
18
|
+
function checkoutVelocity() {
|
|
19
|
+
return {
|
|
20
|
+
id: "checkout-velocity",
|
|
21
|
+
description: "More checkout attempts in a minute than a person makes",
|
|
22
|
+
cost: "cheap",
|
|
23
|
+
stage: "always",
|
|
24
|
+
inspect(ctx) {
|
|
25
|
+
if (!ctx.facts.path.startsWith("/checkout")) return undefined;
|
|
26
|
+
|
|
27
|
+
const attempts = ctx.state.requestsWithin(60_000, ctx.facts.timestamp);
|
|
28
|
+
if (attempts < 5) return undefined;
|
|
29
|
+
|
|
30
|
+
return {
|
|
31
|
+
detector: "checkout-velocity",
|
|
32
|
+
summary: `${attempts} checkout attempts in a minute`,
|
|
33
|
+
direction: "bot",
|
|
34
|
+
certainty: "strong", // not `certain` — a shared address explains it too
|
|
35
|
+
botClass: "automation",
|
|
36
|
+
};
|
|
37
|
+
},
|
|
38
|
+
};
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
const detector = new BotHandler({ extraDetectors: [checkoutVelocity()] });
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
### Checkpoint
|
|
45
|
+
|
|
46
|
+
Six POSTs to `/checkout` from one address, 800 ms apart:
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
checkout burst: score=66 verdict=suspected-bot
|
|
50
|
+
[strong] checkout-velocity: 6 checkout attempts in a minute
|
|
51
|
+
[weak] accept-signature: Accept-Language is a single bare tag ("en-GB") with no fallback chain
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## The certainty rule, from the inside
|
|
55
|
+
|
|
56
|
+
Look again at that comment: `certainty: "strong"`, not `certain`.
|
|
57
|
+
|
|
58
|
+
Six checkout attempts in a minute is a *strong* signal and it is not proof. An office
|
|
59
|
+
behind one NAT address, a family sharing a connection, a customer whose payment kept
|
|
60
|
+
failing — all produce it. To mark it `certain` you would have to write a
|
|
61
|
+
`deterministicBasis` sentence explaining why no benign explanation exists, and you cannot,
|
|
62
|
+
because one does.
|
|
63
|
+
|
|
64
|
+
**That is the forcing function.** If you cannot write the sentence, your evidence is
|
|
65
|
+
`strong`. Outside production the library *enforces* it: `certain` evidence with no basis is
|
|
66
|
+
rejected. Inside production it downgrades to a warning instead, so a third-party detector
|
|
67
|
+
with a missing basis cannot take a live site down.
|
|
68
|
+
|
|
69
|
+
```js
|
|
70
|
+
return {
|
|
71
|
+
detector: "checkout-velocity",
|
|
72
|
+
certainty: "certain",
|
|
73
|
+
deterministicBasis: "…", // you would have to justify it here, in writing
|
|
74
|
+
// …
|
|
75
|
+
};
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Try it without the basis and watch it get refused. That refusal is the library defending the
|
|
79
|
+
one guarantee it makes.
|
|
80
|
+
|
|
81
|
+
## Cost and stage, honestly
|
|
82
|
+
|
|
83
|
+
Mark a detector `cost: "io"` if it touches the network or a shared store, and it runs
|
|
84
|
+
concurrently under a timeout. Mislabelling one would put an unbounded await on the request
|
|
85
|
+
path — so the engine times out any promise a `cheap` detector returns anyway.
|
|
86
|
+
|
|
87
|
+
Use `stage: "confirming"` when your work is only worth doing if something else already found
|
|
88
|
+
an identity to confirm.
|
|
89
|
+
|
|
90
|
+
## Replacing a shipped detector
|
|
91
|
+
|
|
92
|
+
`detectors` replaces the whole set, so map over the default list:
|
|
93
|
+
|
|
94
|
+
```js
|
|
95
|
+
import { defaultDetectors, probeSignatureDetector } from "@osqd/bothandlerjs";
|
|
96
|
+
|
|
97
|
+
new BotHandler({
|
|
98
|
+
detectors: defaultDetectors().map((d) =>
|
|
99
|
+
d.id === "probe-signature" ? probeSignatureDetector({ ignore: ["/wp-login.php"] }) : d,
|
|
100
|
+
),
|
|
101
|
+
});
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
## A signature of your own
|
|
105
|
+
|
|
106
|
+
Serif's partner sends a nightly sync. You want it recognised and allowed by name:
|
|
107
|
+
|
|
108
|
+
```js
|
|
109
|
+
const partner = {
|
|
110
|
+
id: "serif-partner-sync",
|
|
111
|
+
name: "Serif Partner Sync",
|
|
112
|
+
tokens: ["serifpartnersync"], // lowercase — see below
|
|
113
|
+
category: "library",
|
|
114
|
+
benign: true,
|
|
115
|
+
verification: { kind: "none" },
|
|
116
|
+
};
|
|
117
|
+
|
|
118
|
+
const detector = new BotHandler({
|
|
119
|
+
extraSignatures: [partner],
|
|
120
|
+
rules: [{ id: "partner-allow", match: { identity: "serif-partner-sync" }, action: "allow",
|
|
121
|
+
reason: "Our own partner integration." }],
|
|
122
|
+
});
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
### Checkpoint
|
|
126
|
+
|
|
127
|
+
```
|
|
128
|
+
verdict=confirmed-bot class=http-client identity=serif-partner-sync certain=true
|
|
129
|
+
[certain] self-identified: User-Agent identifies Serif Partner Sync
|
|
130
|
+
decision: allow via rule "partner-allow"
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
**`tokens` must be lowercase.** Matching lowercases the User-Agent first, so an uppercase
|
|
134
|
+
token silently never matches — no error, no warning, just a signature that does nothing. It
|
|
135
|
+
is the single easiest mistake to make here, and it costs an afternoon.
|
|
136
|
+
|
|
137
|
+
`verification` is required. Use `{ kind: "none" }` for a client with no verifiable identity,
|
|
138
|
+
`{ kind: "fcrdns", domains: [...] }` where the operator publishes reverse DNS, or
|
|
139
|
+
`{ kind: "ip-ranges" }` where they publish addresses.
|
|
140
|
+
|
|
141
|
+
Set `conclusive: false` with a `caveat` when a token can legitimately appear on a request a
|
|
142
|
+
person made. Electron is the motivating case: that User-Agent comes from VS Code's browser,
|
|
143
|
+
Slack, Discord and Postman — real Chromium instances with a human driving them.
|
|
144
|
+
|
|
145
|
+
## An action of your own
|
|
146
|
+
|
|
147
|
+
```js
|
|
148
|
+
import { defineHandler } from "@osqd/bothandlerjs";
|
|
149
|
+
|
|
150
|
+
const emptyResults = defineHandler({
|
|
151
|
+
id: "empty-results",
|
|
152
|
+
description: "Serves an empty result set rather than an error",
|
|
153
|
+
execute: ({ assessment }) => ({
|
|
154
|
+
kind: "respond",
|
|
155
|
+
status: 200,
|
|
156
|
+
headers: { "content-type": "application/json" },
|
|
157
|
+
body: JSON.stringify({ results: [] }),
|
|
158
|
+
}),
|
|
159
|
+
});
|
|
160
|
+
|
|
161
|
+
new BotHandler({
|
|
162
|
+
handlers: [emptyResults],
|
|
163
|
+
rules: [{ id: "shadow", match: { botClass: "scraper" }, action: "custom",
|
|
164
|
+
params: { handler: "empty-results" } }],
|
|
165
|
+
});
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Return `{ kind: "continue" }` — optionally with `requestHeaders`, `responseHeaders` or
|
|
169
|
+
`delayMs` — or `{ kind: "respond", status, headers, body }`, or `{ kind: "drop" }`. It may
|
|
170
|
+
be async.
|
|
171
|
+
|
|
172
|
+
**Remember the guard does not apply here.** It cannot know what your handler does. A handler
|
|
173
|
+
that denies service is a decision you own entirely — which is fine, as long as you know you
|
|
174
|
+
are making it.
|
|
175
|
+
|
|
176
|
+
A handler named in a rule but not registered serves the request and warns.
|
|
177
|
+
|
|
178
|
+
## Browser signals
|
|
179
|
+
|
|
180
|
+
An optional page script reports what only the browser can see — `navigator.webdriver`,
|
|
181
|
+
automation properties, timing:
|
|
182
|
+
|
|
183
|
+
```js
|
|
184
|
+
import { renderClientScript } from "@osqd/bothandlerjs/client";
|
|
185
|
+
|
|
186
|
+
// `endpoint` is required: it is your route, and it should be cheap and rate-limited.
|
|
187
|
+
app.get("/", (req, res) =>
|
|
188
|
+
res.send(`<body>…${renderClientScript({ endpoint: "/__signals", nonce: res.locals.nonce })}</body>`),
|
|
189
|
+
);
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
It posts back to that endpoint, and the `client-signals` detector reads what arrives. Supply
|
|
193
|
+
the `nonce` if your Content-Security-Policy needs one.
|
|
194
|
+
|
|
195
|
+
**It is capped at `moderate`, permanently, whatever it reports.** The browser is the one
|
|
196
|
+
place an adversary has complete control: `navigator.webdriver` proves what the client *chose
|
|
197
|
+
to report*, and a scraper that wants to lie about it has already won that argument. Treating
|
|
198
|
+
it as conclusive would move the whole guarantee inside the attacker's process.
|
|
199
|
+
|
|
200
|
+
So it is real evidence, weighted honestly, and it can never on its own deny anybody.
|
|
201
|
+
|
|
202
|
+
## Exercise
|
|
203
|
+
|
|
204
|
+
Serif wants to catch scripted gift-card redemption: many distinct codes tried from one
|
|
205
|
+
actor, few succeeding. Write the detector, and decide its tier.
|
|
206
|
+
|
|
207
|
+
<details>
|
|
208
|
+
<summary>Answer</summary>
|
|
209
|
+
|
|
210
|
+
```js
|
|
211
|
+
function giftCardProbing(redemptions) {
|
|
212
|
+
return {
|
|
213
|
+
id: "gift-card-probing",
|
|
214
|
+
description: "Many distinct gift-card codes tried from one actor",
|
|
215
|
+
cost: "cheap",
|
|
216
|
+
stage: "always",
|
|
217
|
+
inspect(ctx) {
|
|
218
|
+
if (ctx.facts.path !== "/gift-cards/redeem") return undefined;
|
|
219
|
+
const tried = redemptions.recentFailures(ctx.actorKey, 10 * 60_000);
|
|
220
|
+
if (tried < 12) return undefined;
|
|
221
|
+
return {
|
|
222
|
+
detector: "gift-card-probing",
|
|
223
|
+
summary: `${tried} failed gift-card redemptions in ten minutes`,
|
|
224
|
+
direction: "bot",
|
|
225
|
+
certainty: "strong",
|
|
226
|
+
botClass: "automation",
|
|
227
|
+
};
|
|
228
|
+
},
|
|
229
|
+
};
|
|
230
|
+
}
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
**`strong`, not `certain`.** Twelve failed redemptions is a strong signal about a *shared
|
|
234
|
+
address* as much as about a script — a busy office at Christmas produces it. You cannot
|
|
235
|
+
write the basis sentence, so it is not certain.
|
|
236
|
+
|
|
237
|
+
**`cheap`, only if `redemptions` is in memory.** If it queries a database, it is `io` — and
|
|
238
|
+
mislabelling it would put a database round trip on the request path of every checkout.
|
|
239
|
+
|
|
240
|
+
And note that a `strong` signal is enough to challenge or rate-limit, which is the right
|
|
241
|
+
response to card probing anyway. You lose nothing by being honest about the tier.
|
|
242
|
+
</details>
|
|
243
|
+
|
|
244
|
+
## What you learned
|
|
245
|
+
|
|
246
|
+
- A detector returns evidence, never a verdict
|
|
247
|
+
- The `deterministicBasis` requirement is a forcing function: if you cannot write it, it is
|
|
248
|
+
`strong`
|
|
249
|
+
- `cost` must be honest, or you put an await on the request path
|
|
250
|
+
- Signature `tokens` must be lowercase, and the failure is silent
|
|
251
|
+
- Custom handlers sit outside the guard, deliberately
|
|
252
|
+
- Browser signals are capped at `moderate` for ever, because that is the attacker's process
|
|
253
|
+
|
|
254
|
+
## Reference
|
|
255
|
+
|
|
256
|
+
- [Writing a detector](../detection/writing-a-detector.md) · [Signatures](../detection/signatures.md)
|
|
257
|
+
- [Browser signals](../detection/client-signals.md) · [Actions](../policy/actions.md)
|
|
258
|
+
|
|
259
|
+
Next: [Proving it, and the capstone](16-proving-it.md).
|
|
@@ -0,0 +1,266 @@
|
|
|
1
|
+
# Lesson 16 — Proving it, and the capstone
|
|
2
|
+
|
|
3
|
+
**Goal:** find out who your policy would hurt *before* it hurts them, then assemble
|
|
4
|
+
everything into a Serif you can defend.
|
|
5
|
+
|
|
6
|
+
← [Course](index.md) · Prev: [Extending it](15-extending.md)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## The question this library is organised around
|
|
11
|
+
|
|
12
|
+
*If I point this configuration at the actual internet, who gets hurt?*
|
|
13
|
+
|
|
14
|
+
Three tools answer it, and they answer different halves.
|
|
15
|
+
|
|
16
|
+
## `check` — against 526 shapes of real traffic
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
npx @osqd/bothandlerjs check --preset protect-content
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
protect-content against 522 shapes of real traffic
|
|
24
|
+
|
|
25
|
+
human 181 cases 3 tag, 175 allow, 3 challenge
|
|
26
|
+
benign-bot 144 cases 6 allow, 86 tag, 48 block, 2 challenge, 2 rate-limit
|
|
27
|
+
declared-bot 32 cases 18 tag, 2 rate-limit, 3 allow, 9 block
|
|
28
|
+
unwanted-bot 105 cases 8 tag, 11 rate-limit, 3 block, 70 challenge, 13 allow
|
|
29
|
+
hostile 27 cases 6 challenge, 17 block, 4 allow
|
|
30
|
+
infrastructure 33 cases 7 allow, 12 tag, 14 challenge
|
|
31
|
+
|
|
32
|
+
No case marked as a person was denied service.
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
**It exits non-zero if any case marked `human` is denied**, which is what makes it a CI step
|
|
36
|
+
rather than a report.
|
|
37
|
+
|
|
38
|
+
The corpus is 526 cases with provenance: 182 of them people — 30 browser profiles, 40 in-app
|
|
39
|
+
WebViews, Tor, screen readers, IE11, a car's infotainment screen, corporate proxies, CGNAT,
|
|
40
|
+
an author signing in at `/wp-login.php`. Header order is reproduced rather than invented, and
|
|
41
|
+
DNS is controlled rather than mocked away, so *"the operator's DNS disproves this"* and
|
|
42
|
+
*"our resolver was unhappy"* can be told apart.
|
|
43
|
+
|
|
44
|
+
`--audience human` shows only the part that matters. `--strict` also fails on differing
|
|
45
|
+
actions.
|
|
46
|
+
|
|
47
|
+
### Against *your* configuration
|
|
48
|
+
|
|
49
|
+
The corpus is a published entry point, not a test fixture:
|
|
50
|
+
|
|
51
|
+
```js
|
|
52
|
+
import { runCorpus } from "@osqd/bothandlerjs/corpus";
|
|
53
|
+
|
|
54
|
+
const scorecard = await runCorpus({
|
|
55
|
+
create: ({ resolver, clock }) => new BotHandler({ ...serifConfig, resolver, clock }),
|
|
56
|
+
assertActions: false, // your actions are yours; the invariants are not
|
|
57
|
+
});
|
|
58
|
+
|
|
59
|
+
if (scorecard.falsePositives.length > 0) {
|
|
60
|
+
throw new Error(`Serif's policy denies ${scorecard.falsePositives.length} people`);
|
|
61
|
+
}
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Put that in your test suite. It runs offline and deterministically.
|
|
65
|
+
|
|
66
|
+
## `replay` — against traffic that is actually yours
|
|
67
|
+
|
|
68
|
+
The corpus knows what the internet looks like. Only your logs know what *your* visitors look
|
|
69
|
+
like.
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
npx @osqd/bothandlerjs replay /var/log/nginx/access.log --preset protect-content
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
```
|
|
76
|
+
116 request(s) would have been DENIED, in 1 distinct kind(s). Read them.
|
|
77
|
+
─────────────────────────────────────────────────────────────────────
|
|
78
|
+
116x block by rule "scanner-block" — confirmed-bot, proven
|
|
79
|
+
sqlmap/1.7.2#stable (https://sqlmap.org)
|
|
80
|
+
[certain] self-identified: User-Agent identifies sqlmap
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
That block is the point of the exercise. If any of those is a person, the policy is wrong —
|
|
84
|
+
and you found out from a log file rather than from a support ticket.
|
|
85
|
+
|
|
86
|
+
**Prefer JSON Lines over CLF**, and the reason is the rule from [lesson 2](02-proof-and-suspicion.md):
|
|
87
|
+
|
|
88
|
+
> A header missing from a *record* is not a header missing from the *request*.
|
|
89
|
+
|
|
90
|
+
An nginx line carries the User-Agent and the Referer and nothing else. Detectors that reason
|
|
91
|
+
from absence would fire on everything, so `partialHeaders` marks a header-poor source and
|
|
92
|
+
they stand down. A CLF replay therefore **under-reports**, and its silence is not a clean
|
|
93
|
+
bill of health.
|
|
94
|
+
|
|
95
|
+
```js
|
|
96
|
+
createFacts({ method, url, headers, ip, partialHeaders: true });
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## `explain` — one request, by ticket
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
npx @osqd/bothandlerjs explain "curl/8.4.0"
|
|
103
|
+
pbpaste | npx @osqd/bothandlerjs explain --preset protect-data
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
A User-Agent, a curl command out of devtools, or a raw header block. It runs as a **dry
|
|
107
|
+
run** — nothing recorded, so asking does not change the answer:
|
|
108
|
+
|
|
109
|
+
```js
|
|
110
|
+
const assessment = await detector.assess(facts, { record: false });
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
No counter moves, no actor state changes, no event fires. It has no history by construction,
|
|
114
|
+
so what it answers is *what would this look like as a first request* — which is what a
|
|
115
|
+
ticket is asking anyway.
|
|
116
|
+
|
|
117
|
+
## The order to use them in
|
|
118
|
+
|
|
119
|
+
1. **`explain`** — one request, when somebody complains
|
|
120
|
+
2. **`check`** — in CI, on every change
|
|
121
|
+
3. **`replay`** — before enforcing anything, over a real week
|
|
122
|
+
4. **`monitor-only` in production** — for a week, watching the dashboard
|
|
123
|
+
|
|
124
|
+
Step 4 is not optional and the first three do not replace it.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
# Capstone: Serif
|
|
129
|
+
|
|
130
|
+
Everything from sixteen lessons, in one configuration.
|
|
131
|
+
|
|
132
|
+
```js
|
|
133
|
+
import express from "express";
|
|
134
|
+
import { BotHandler, RedisStore, declineAiTraining, robotsFromRules, renderTrapLink } from "@osqd/bothandlerjs";
|
|
135
|
+
import { botHandler } from "@osqd/bothandlerjs/adapters";
|
|
136
|
+
|
|
137
|
+
const store = new RedisStore(redis);
|
|
138
|
+
const proxy = { trustProxy: true, trustedProxies: ["10.0.0.0/8"] }; // lesson 12
|
|
139
|
+
|
|
140
|
+
const site = new BotHandler({
|
|
141
|
+
preset: "decline-ai-training", // lesson 10 — the business decision
|
|
142
|
+
proxy, // lesson 12 — the dangerous setting
|
|
143
|
+
store, // lesson 14 — nonces and rate limits
|
|
144
|
+
shareConfirmations: true, // lesson 14 — proof travels
|
|
145
|
+
actorKey: (facts) => facts.session ?? facts.ip, // lesson 7
|
|
146
|
+
isHuman: (facts) => Boolean(sessions.get(facts.session ?? "")?.authenticated), // lesson 12
|
|
147
|
+
allowlist: ["10.0.0.0/8"], // monitors and CI
|
|
148
|
+
ignorePaths: ["/healthz", "/metrics"],
|
|
149
|
+
extraDetectors: [checkoutVelocity()], // lesson 15
|
|
150
|
+
extraSignatures: [partner], // lesson 15
|
|
151
|
+
challenge: { // lesson 11
|
|
152
|
+
secrets: [process.env.SERIF_CHALLENGE_SECRET, process.env.SERIF_CHALLENGE_SECRET_PREVIOUS].filter(Boolean),
|
|
153
|
+
contactHtml: '<p>Locked out? Email <a href="mailto:help@serif.example">help@serif.example</a>.</p>',
|
|
154
|
+
translations: { "pt-BR": { lang: "pt-BR", title: "Verificando seu navegador" }, ja: { lang: "ja", title: "ブラウザーを確認しています" } },
|
|
155
|
+
},
|
|
156
|
+
rules: [ // lesson 9 — yours run before the preset's
|
|
157
|
+
{ id: "partner-allow", match: { identity: "serif-partner-sync" }, action: "allow",
|
|
158
|
+
reason: "Our own nightly partner sync." },
|
|
159
|
+
{ id: "persistent-refusal", match: { minUnsolvedChallenges: 5 }, action: "block",
|
|
160
|
+
reason: "Issued five challenges, finished none." },
|
|
161
|
+
],
|
|
162
|
+
audit: { windowMs: 5 * 60_000, baselineMs: 60 * 60_000, minSamples: 50 }, // lesson 13
|
|
163
|
+
onAnomaly: (a) => pager.send(a.severity, a.summary),
|
|
164
|
+
onDowngrade: ({ decision }) => metrics.increment("bot.downgrade", { rule: decision.rule }),
|
|
165
|
+
onWarning: (message) => log.warn({ message }, "bothandler"),
|
|
166
|
+
});
|
|
167
|
+
|
|
168
|
+
// Auth routes get their own policy, mounted only there. lesson 10
|
|
169
|
+
const auth = new BotHandler({ preset: "protect-auth", proxy, store });
|
|
170
|
+
|
|
171
|
+
const app = express();
|
|
172
|
+
app.use(botHandler(site));
|
|
173
|
+
app.use("/login", botHandler(auth));
|
|
174
|
+
app.use("/checkout", botHandler(auth));
|
|
175
|
+
|
|
176
|
+
// Say out loud what the policy does. lesson 10
|
|
177
|
+
const { robotsTxt, unreadable } = robotsFromRules(declineAiTraining(), {
|
|
178
|
+
disallowPaths: ["/internal/export.csv"],
|
|
179
|
+
sitemap: "https://serif.example/sitemap.xml",
|
|
180
|
+
});
|
|
181
|
+
if (unreadable.length > 0) log.warn({ unreadable }, "not reflected in robots.txt");
|
|
182
|
+
app.get("/robots.txt", (_req, res) => res.type("text/plain").send(robotsTxt));
|
|
183
|
+
|
|
184
|
+
// A trap no person can reach. lesson 8
|
|
185
|
+
app.get("/", (_req, res) => res.send(`<main>…</main>${renderTrapLink("/internal/export.csv")}`));
|
|
186
|
+
|
|
187
|
+
// Watch it. lesson 13
|
|
188
|
+
app.get("/internal/metrics", (_req, res) => res.type("text/plain").send(site.prometheus()));
|
|
189
|
+
await site.serveDashboard({ port: 9674, title: "serif", auth: { username: "ops", password: process.env.DASHBOARD_PASSWORD } });
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
And the test that stops it hurting anybody:
|
|
193
|
+
|
|
194
|
+
```js
|
|
195
|
+
import { runCorpus } from "@osqd/bothandlerjs/corpus";
|
|
196
|
+
|
|
197
|
+
it("never denies a person", async () => {
|
|
198
|
+
const scorecard = await runCorpus({
|
|
199
|
+
create: ({ resolver, clock }) => new BotHandler({ ...serifConfig, resolver, clock }),
|
|
200
|
+
assertActions: false,
|
|
201
|
+
});
|
|
202
|
+
expect(scorecard.falsePositives).toEqual([]);
|
|
203
|
+
});
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
## Final exercise
|
|
207
|
+
|
|
208
|
+
Answer these without looking anything up. They are the whole course.
|
|
209
|
+
|
|
210
|
+
1. A client scores 99 from probabilistic signals. Can a rule block it? Why?
|
|
211
|
+
2. Your policy denies 4% of traffic. What do you check first?
|
|
212
|
+
3. `bothandler_downgrades_total` doubled after a deploy. What happened?
|
|
213
|
+
4. `challenge-solve-rate` is 96%. Is that good?
|
|
214
|
+
5. You add a detector that catches a scraper perfectly. What tier is its evidence?
|
|
215
|
+
|
|
216
|
+
<details>
|
|
217
|
+
<summary>Answers</summary>
|
|
218
|
+
|
|
219
|
+
1. **No.** The score is not what gates a terminal action — `certain` is. Ninety-nine
|
|
220
|
+
probabilistic points are ninety-nine things a real person can trip, and the guard
|
|
221
|
+
substitutes something recoverable. [Lesson 4](04-the-guard.md).
|
|
222
|
+
|
|
223
|
+
2. **Who the 4% are.** Run `replay` over the log and read every denial. A `denial-spike`
|
|
224
|
+
anomaly is not evidence they were bots — it is evidence that something changed, and the
|
|
225
|
+
expensive possibility is that it changed about people. [Lesson 13](13-operating-it.md).
|
|
226
|
+
|
|
227
|
+
3. **Your policy now asks for more than the evidence supports.** Either a rule changed, or
|
|
228
|
+
the traffic did. The guard absorbed the difference, which is why nobody was wrongly
|
|
229
|
+
refused — but the two have drifted apart and the number is telling you so.
|
|
230
|
+
[Lesson 4](04-the-guard.md).
|
|
231
|
+
|
|
232
|
+
4. **No — it is the bad direction.** A proof of work is trivial for a browser and trivial
|
|
233
|
+
for a competent scraper. When nearly everything challenged passes, the challenges are not
|
|
234
|
+
filtering bots out, they are taxing people. [Lesson 13](13-operating-it.md).
|
|
235
|
+
|
|
236
|
+
5. **`strong`, almost certainly.** Ask whether you can write a `deterministicBasis`
|
|
237
|
+
sentence explaining why no benign explanation exists. Unless the client *declared*
|
|
238
|
+
itself, *contradicted* itself, walked into a *trap*, was *refuted by an external
|
|
239
|
+
authority*, or committed a *protocol violation* — you cannot, and it is not certain.
|
|
240
|
+
[Lesson 15](15-extending.md).
|
|
241
|
+
</details>
|
|
242
|
+
|
|
243
|
+
## Where to go now
|
|
244
|
+
|
|
245
|
+
- **[The threat model](../concepts/threat-model.md).** Read it before you promise anybody
|
|
246
|
+
anything. It names the two rungs of the evasion ladder that defeat this library, and it is
|
|
247
|
+
the most useful page in the documentation once you know how everything works.
|
|
248
|
+
- **[Design decisions](../design/decisions.md).** Fifteen choices with what each one cost.
|
|
249
|
+
- **[The reference](../index.md).** Thirty pages, one per question.
|
|
250
|
+
|
|
251
|
+
## What you learned in this course
|
|
252
|
+
|
|
253
|
+
- Proof and suspicion are different things, and only one of them may close a door
|
|
254
|
+
- Twenty detectors, each with a ceiling it may not exceed
|
|
255
|
+
- A name is a claim until an external authority confirms it
|
|
256
|
+
- Behaviour over time catches what a single request cannot, and stops at a paced scraper
|
|
257
|
+
- First match wins, and order is the policy
|
|
258
|
+
- Rank actions by what they cost somebody who did nothing wrong
|
|
259
|
+
- The challenge imposes cost; it proves neither identity nor humanity
|
|
260
|
+
- The client IP is the setting that is silent when it is wrong
|
|
261
|
+
- Alert on downgrades; read the score histogram before moving a threshold
|
|
262
|
+
- Proof travels between replicas; suspicion stays home
|
|
263
|
+
- If you cannot write why it admits no benign explanation, it is not certain
|
|
264
|
+
- Find out who your policy hurts before it hurts them
|
|
265
|
+
|
|
266
|
+
← [Back to the course](index.md)
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# The BotHandler course
|
|
2
|
+
|
|
3
|
+
Sixteen lessons that build one real integration, from a first assessment to a policy you
|
|
4
|
+
can defend.
|
|
5
|
+
|
|
6
|
+
← [Documentation](../index.md)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## What this is
|
|
11
|
+
|
|
12
|
+
The [reference documentation](../index.md) answers *"how does X work?"*. This answers
|
|
13
|
+
*"what do I do, and in what order?"* — every capability of the library, taught in the
|
|
14
|
+
order that makes each one make sense, with something to run at every step.
|
|
15
|
+
|
|
16
|
+
It is written to be worked through rather than read. Every lesson has code you paste and
|
|
17
|
+
output you can check yours against, and every checkpoint in it was produced by actually
|
|
18
|
+
running the code rather than by imagining what it would print.
|
|
19
|
+
|
|
20
|
+
**Time:** about four hours to do properly. Lessons 1–4 are the foundation and are worth
|
|
21
|
+
slowing down for; everything after them assumes you have those.
|
|
22
|
+
|
|
23
|
+
## Who it is for
|
|
24
|
+
|
|
25
|
+
A TypeScript or JavaScript developer who has a site or an API and a bot problem — or who
|
|
26
|
+
suspects they are about to have one. You need no background in bot detection. You do need
|
|
27
|
+
to be comfortable running `node` and reading a stack trace.
|
|
28
|
+
|
|
29
|
+
## The running example
|
|
30
|
+
|
|
31
|
+
You are building **Serif**, an online bookshop. It has a catalogue worth scraping, a
|
|
32
|
+
checkout worth attacking, and readers worth not annoying. Every lesson adds one thing to
|
|
33
|
+
Serif, and by lesson 16 you have a complete, tested, production-shaped configuration.
|
|
34
|
+
|
|
35
|
+
## Set up once
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
mkdir serif && cd serif
|
|
39
|
+
npm init -y && npm pkg set type=module
|
|
40
|
+
npm install @osqd/bothandlerjs
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Every lesson's code goes in a file you run with `node`. Nothing needs a server until
|
|
44
|
+
lesson 11.
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## Part 1 — The ideas everything rests on
|
|
49
|
+
|
|
50
|
+
| | | |
|
|
51
|
+
|-|-|-|
|
|
52
|
+
| 1 | [Your first assessment](01-first-assessment.md) | Turn a request into a verdict, and read what comes back. |
|
|
53
|
+
| 2 | [Proof and suspicion](02-proof-and-suspicion.md) | The distinction the whole library is built on. **The most important lesson here.** |
|
|
54
|
+
| 3 | [Verdicts, classes and scores](03-verdicts-and-scores.md) | Four fields describe a client. Which one should you act on? |
|
|
55
|
+
| 4 | [The safety guard](04-the-guard.md) | The mechanism that stops a guess closing a door. |
|
|
56
|
+
|
|
57
|
+
## Part 2 — Detection
|
|
58
|
+
|
|
59
|
+
| | | |
|
|
60
|
+
|-|-|-|
|
|
61
|
+
| 5 | [The detectors](05-detectors.md) | All twenty: what each reads, what each costs, what each may conclude. |
|
|
62
|
+
| 6 | [Identity and verification](06-identity.md) | Recognising a crawler by name — and proving or refuting the claim. |
|
|
63
|
+
| 7 | [Actors and behaviour](07-actors.md) | Who "the same client" is, and what watching one over time tells you. |
|
|
64
|
+
| 8 | [Traps](08-traps.md) | The one detector that needs no statistics, and how to lay one properly. |
|
|
65
|
+
|
|
66
|
+
## Part 3 — Deciding what to do
|
|
67
|
+
|
|
68
|
+
| | | |
|
|
69
|
+
|-|-|-|
|
|
70
|
+
| 9 | [Rules](09-rules.md) | Matching requests, in first-match-wins order. |
|
|
71
|
+
| 10 | [Actions and presets](10-actions-and-presets.md) | Ten responses, ordered by what each costs a person who did nothing wrong. |
|
|
72
|
+
| 11 | [The challenge](11-the-challenge.md) | Proof of work: what it buys, what it cannot, and who it must not exclude. |
|
|
73
|
+
|
|
74
|
+
## Part 4 — Production
|
|
75
|
+
|
|
76
|
+
| | | |
|
|
77
|
+
|-|-|-|
|
|
78
|
+
| 12 | [Going live](12-going-live.md) | Adapters, and the one setting that is dangerous to get wrong. |
|
|
79
|
+
| 13 | [Operating it](13-operating-it.md) | The dashboard, metrics, the traffic audit, notifications. |
|
|
80
|
+
| 14 | [Scaling and changing it live](14-scaling.md) | Several replicas, shared state, and editing policy without a deploy. |
|
|
81
|
+
|
|
82
|
+
## Part 5 — Making it yours, and proving it
|
|
83
|
+
|
|
84
|
+
| | | |
|
|
85
|
+
|-|-|-|
|
|
86
|
+
| 15 | [Extending it](15-extending.md) | Your own detector, action, signatures and browser signals. |
|
|
87
|
+
| 16 | [Proving it, and the capstone](16-proving-it.md) | The corpus, log replay, and a finished Serif you can defend. |
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## How to get the most out of it
|
|
92
|
+
|
|
93
|
+
**Type the code, do not paste it.** The examples are short on purpose.
|
|
94
|
+
|
|
95
|
+
**Do the exercises before reading the answer.** Each one has a checkpoint so you know
|
|
96
|
+
whether you got it.
|
|
97
|
+
|
|
98
|
+
**When something surprises you, chase it.** Every lesson ends with links into the
|
|
99
|
+
reference documentation for the thing you just used. The surprises are usually where the
|
|
100
|
+
library is protecting you from something.
|
|
101
|
+
|
|
102
|
+
## A promise, and its consequence
|
|
103
|
+
|
|
104
|
+
One sentence holds this library together, and it will explain most of what surprises you:
|
|
105
|
+
|
|
106
|
+
> **Nothing is ever denied service on the strength of a guess.**
|
|
107
|
+
|
|
108
|
+
You will meet it first in [lesson 4](04-the-guard.md), where a rule you wrote asking to
|
|
109
|
+
block somebody quietly does something gentler instead. That is not a bug, and the lesson
|
|
110
|
+
explains why it is the most valuable behaviour here.
|
|
111
|
+
|
|
112
|
+
Start with [lesson 1](01-first-assessment.md).
|