@osqd/bothandlerjs 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +516 -0
- package/LICENSE +102 -0
- package/README.md +308 -0
- package/bin/bothandlerjs.mjs +14 -0
- package/dist/actions/index.d.ts +22 -0
- package/dist/actions/types.d.ts +59 -0
- package/dist/adapters/fastify.d.ts +43 -0
- package/dist/adapters/fetch.d.ts +51 -0
- package/dist/adapters/index.cjs +723 -0
- package/dist/adapters/index.cjs.map +1 -0
- package/dist/adapters/index.d.ts +8 -0
- package/dist/adapters/index.js +692 -0
- package/dist/adapters/index.js.map +1 -0
- package/dist/adapters/koa.d.ts +27 -0
- package/dist/adapters/node.d.ts +29 -0
- package/dist/adapters/shared.d.ts +39 -0
- package/dist/audit.d.ts +172 -0
- package/dist/challenge/index.d.ts +176 -0
- package/dist/challenge/language.d.ts +57 -0
- package/dist/challenge/page.d.ts +42 -0
- package/dist/challenge/pow.d.ts +33 -0
- package/dist/challenge/token.d.ts +72 -0
- package/dist/cli.cjs +13388 -0
- package/dist/cli.cjs.map +1 -0
- package/dist/cli.d.ts +5 -0
- package/dist/cli.js +13358 -0
- package/dist/cli.js.map +1 -0
- package/dist/client/index.cjs +139 -0
- package/dist/client/index.cjs.map +1 -0
- package/dist/client/index.d.ts +63 -0
- package/dist/client/index.js +112 -0
- package/dist/client/index.js.map +1 -0
- package/dist/config.d.ts +316 -0
- package/dist/core.d.ts +443 -0
- package/dist/corpus/adversarial.d.ts +2 -0
- package/dist/corpus/advertising-email.d.ts +2 -0
- package/dist/corpus/ai-crawlers.d.ts +2 -0
- package/dist/corpus/benign-bots.d.ts +2 -0
- package/dist/corpus/cdn-gateways.d.ts +2 -0
- package/dist/corpus/cookies.d.ts +49 -0
- package/dist/corpus/crawlers-regional.d.ts +2 -0
- package/dist/corpus/crawlers-vertical.d.ts +2 -0
- package/dist/corpus/headers.d.ts +403 -0
- package/dist/corpus/humans-apps.d.ts +2 -0
- package/dist/corpus/humans-browsers.d.ts +2 -0
- package/dist/corpus/humans.d.ts +2 -0
- package/dist/corpus/index.cjs +4196 -0
- package/dist/corpus/index.cjs.map +1 -0
- package/dist/corpus/index.d.ts +37 -0
- package/dist/corpus/index.js +4132 -0
- package/dist/corpus/index.js.map +1 -0
- package/dist/corpus/infrastructure.d.ts +17 -0
- package/dist/corpus/libraries-extended.d.ts +2 -0
- package/dist/corpus/ranges.d.ts +31 -0
- package/dist/corpus/reputation.d.ts +21 -0
- package/dist/corpus/runner.d.ts +130 -0
- package/dist/corpus/schema.d.ts +185 -0
- package/dist/corpus/tooling.d.ts +2 -0
- package/dist/corpus/unwanted.d.ts +2 -0
- package/dist/crawler-ranges.d.ts +113 -0
- package/dist/dashboard/client.generated.d.ts +7 -0
- package/dist/dashboard/feed.d.ts +186 -0
- package/dist/dashboard/index.d.ts +5 -0
- package/dist/dashboard/page.d.ts +51 -0
- package/dist/dashboard/parse-request.d.ts +28 -0
- package/dist/dashboard/preview.d.ts +45 -0
- package/dist/dashboard/server.d.ts +70 -0
- package/dist/dashboard/types.d.ts +714 -0
- package/dist/detectors/accept-signature.d.ts +6 -0
- package/dist/detectors/browsing-coherence.d.ts +44 -0
- package/dist/detectors/cadence.d.ts +28 -0
- package/dist/detectors/clearance.d.ts +25 -0
- package/dist/detectors/client-hints.d.ts +19 -0
- package/dist/detectors/client-signals.d.ts +27 -0
- package/dist/detectors/crawl-breadth.d.ts +28 -0
- package/dist/detectors/crawler-verification.d.ts +39 -0
- package/dist/detectors/fetch-metadata.d.ts +20 -0
- package/dist/detectors/header-integrity.d.ts +26 -0
- package/dist/detectors/header-order.d.ts +27 -0
- package/dist/detectors/identity-rotation.d.ts +34 -0
- package/dist/detectors/index.d.ts +56 -0
- package/dist/detectors/ip-intelligence.d.ts +30 -0
- package/dist/detectors/known-bots.d.ts +94 -0
- package/dist/detectors/probe-signature.d.ts +50 -0
- package/dist/detectors/rate-anomaly.d.ts +36 -0
- package/dist/detectors/self-identified.d.ts +33 -0
- package/dist/detectors/session-integrity.d.ts +24 -0
- package/dist/detectors/tls-fingerprint.d.ts +34 -0
- package/dist/detectors/trap.d.ts +98 -0
- package/dist/detectors/types.d.ts +70 -0
- package/dist/detectors/ua-coherence.d.ts +32 -0
- package/dist/evidence.d.ts +45 -0
- package/dist/facts.d.ts +46 -0
- package/dist/index.cjs +9600 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.ts +63 -0
- package/dist/index.js +9444 -0
- package/dist/index.js.map +1 -0
- package/dist/internal/async.d.ts +16 -0
- package/dist/internal/clock.d.ts +17 -0
- package/dist/internal/crypto.d.ts +26 -0
- package/dist/internal/dns.d.ts +100 -0
- package/dist/internal/emitter.d.ts +18 -0
- package/dist/internal/http.d.ts +22 -0
- package/dist/internal/ip.d.ts +73 -0
- package/dist/internal/lru.d.ts +34 -0
- package/dist/internal/matcher.d.ts +41 -0
- package/dist/internal/pattern.d.ts +14 -0
- package/dist/internal/ua.d.ts +92 -0
- package/dist/metrics.d.ts +120 -0
- package/dist/notify/hub.d.ts +70 -0
- package/dist/notify/index.d.ts +7 -0
- package/dist/notify/redact.d.ts +48 -0
- package/dist/notify/sinks.d.ts +62 -0
- package/dist/notify/types.d.ts +56 -0
- package/dist/policy/index.d.ts +7 -0
- package/dist/policy/match.d.ts +20 -0
- package/dist/policy/policy.d.ts +123 -0
- package/dist/policy/presets.d.ts +161 -0
- package/dist/policy/types.d.ts +155 -0
- package/dist/robots.d.ts +83 -0
- package/dist/state.d.ts +161 -0
- package/dist/stores/index.d.ts +5 -0
- package/dist/stores/memory.d.ts +33 -0
- package/dist/stores/redis.d.ts +59 -0
- package/dist/stores/types.d.ts +37 -0
- package/dist/types.d.ts +250 -0
- package/docs/challenge/index.md +152 -0
- package/docs/challenge/localisation.md +105 -0
- package/docs/concepts/actors.md +113 -0
- package/docs/concepts/evidence.md +133 -0
- package/docs/concepts/the-guard.md +121 -0
- package/docs/concepts/threat-model.md +120 -0
- package/docs/concepts/verdicts.md +108 -0
- package/docs/course/01-first-assessment.md +178 -0
- package/docs/course/02-proof-and-suspicion.md +164 -0
- package/docs/course/03-verdicts-and-scores.md +160 -0
- package/docs/course/04-the-guard.md +200 -0
- package/docs/course/05-detectors.md +178 -0
- package/docs/course/06-identity.md +161 -0
- package/docs/course/07-actors.md +178 -0
- package/docs/course/08-traps.md +183 -0
- package/docs/course/09-rules.md +224 -0
- package/docs/course/10-actions-and-presets.md +216 -0
- package/docs/course/11-the-challenge.md +227 -0
- package/docs/course/12-going-live.md +211 -0
- package/docs/course/13-operating-it.md +238 -0
- package/docs/course/14-scaling.md +195 -0
- package/docs/course/15-extending.md +259 -0
- package/docs/course/16-proving-it.md +266 -0
- package/docs/course/index.md +112 -0
- package/docs/design/decisions.md +216 -0
- package/docs/detection/client-signals.md +96 -0
- package/docs/detection/detectors.md +376 -0
- package/docs/detection/index.md +134 -0
- package/docs/detection/signatures.md +111 -0
- package/docs/detection/verification.md +123 -0
- package/docs/detection/writing-a-detector.md +145 -0
- package/docs/index.md +100 -0
- package/docs/integration/adapters.md +138 -0
- package/docs/integration/client-ip.md +114 -0
- package/docs/integration/index.md +55 -0
- package/docs/integration/stores.md +76 -0
- package/docs/operations/audit.md +116 -0
- package/docs/operations/dashboard.md +725 -0
- package/docs/operations/index.md +101 -0
- package/docs/operations/metrics.md +74 -0
- package/docs/operations/notifications.md +72 -0
- package/docs/operations/runtime-changes.md +118 -0
- package/docs/policy/actions.md +196 -0
- package/docs/policy/index.md +84 -0
- package/docs/policy/presets.md +191 -0
- package/docs/policy/robots.md +128 -0
- package/docs/policy/rules.md +134 -0
- package/docs/reference/api.md +126 -0
- package/docs/reference/configuration.md +175 -0
- package/docs/start/choosing-a-policy.md +127 -0
- package/docs/start/first-integration.md +127 -0
- package/docs/start/installation.md +63 -0
- package/docs/start/upgrading.md +72 -0
- package/docs/testing/cli.md +119 -0
- package/docs/testing/corpus.md +129 -0
- package/docs/testing/index.md +61 -0
- package/docs/testing/replay.md +102 -0
- package/docs/testing/try-it.md +189 -0
- package/package.json +145 -0
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
# Design decisions
|
|
2
|
+
|
|
3
|
+
The choices that shaped the library, and what each one costs.
|
|
4
|
+
|
|
5
|
+
← [Documentation](../index.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
Everything here is a trade rather than a free win. They are recorded together so that a
|
|
10
|
+
future change can weigh what it is undoing.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Proof and suspicion are separate compartments
|
|
15
|
+
|
|
16
|
+
**The decision.** Evidence is tiered. `certain` short-circuits to a verdict; everything else
|
|
17
|
+
combines by noisy-OR into a 0–99 score. They never mix, and only the first can close a door.
|
|
18
|
+
|
|
19
|
+
**Why.** Every signal that catches sophisticated automation is probabilistic, and every one
|
|
20
|
+
has a population of real people who trip it. The usual answer — add the signals up, block
|
|
21
|
+
above a threshold — is exactly the mistake: points do not compose into proof. Two unrelated
|
|
22
|
+
suspicions about an unusual but entirely real browser reach 100 as readily as two
|
|
23
|
+
well-founded ones, and the people who get caught are disproportionately the ones with the
|
|
24
|
+
strongest reasons for their unusual setup.
|
|
25
|
+
|
|
26
|
+
**The cost.** Sophisticated scrapers that trip four probabilistic signals are still not
|
|
27
|
+
blocked. That is the intended behaviour, and it is why the [challenge](../challenge/index.md)
|
|
28
|
+
exists as a ceiling for suspicion.
|
|
29
|
+
|
|
30
|
+
See [evidence](../concepts/evidence.md).
|
|
31
|
+
|
|
32
|
+
## The guard runs after rule selection, not inside a rule
|
|
33
|
+
|
|
34
|
+
**The decision.** A separate pass, after the first matching rule is chosen, that replaces a
|
|
35
|
+
terminal action when the verdict is not proven.
|
|
36
|
+
|
|
37
|
+
**Why.** Anywhere else it can be forgotten. Inside a rule it has to be written correctly by
|
|
38
|
+
every author of every rule, including at 3am during an incident. After selection it cannot
|
|
39
|
+
be forgotten in a rule, worked around by a clever predicate, or bypassed by someone who has
|
|
40
|
+
not read the documentation. Relaxing it is one explicit, greppable setting.
|
|
41
|
+
|
|
42
|
+
**The cost.** A rule can say `block` and get `challenge`, which surprises people until they
|
|
43
|
+
read `bothandler_downgrades_total`. That surprise is the feature.
|
|
44
|
+
|
|
45
|
+
See [the guard](../concepts/the-guard.md).
|
|
46
|
+
|
|
47
|
+
## No argument from absence may ever be `certain`
|
|
48
|
+
|
|
49
|
+
**The decision.** A header missing from the facts never produces proof — including protocol
|
|
50
|
+
violations that would otherwise qualify.
|
|
51
|
+
|
|
52
|
+
**Why.** A header missing from a *record* is not a header missing from the *request*. A
|
|
53
|
+
caller building facts from a log line, a WAF event or a partial adapter would otherwise
|
|
54
|
+
manufacture proof against every request in the file. An HTTP/1.1 request with no `Host`
|
|
55
|
+
violates RFC 9112 as plainly as anything on the certain list, and it is deliberately not
|
|
56
|
+
proven here.
|
|
57
|
+
|
|
58
|
+
**The cost.** Some genuinely conclusive observations are demoted to `strong`. Worth it: the
|
|
59
|
+
failure mode it prevents is silent and enormous.
|
|
60
|
+
|
|
61
|
+
## `deterministicBasis` is required in writing
|
|
62
|
+
|
|
63
|
+
**The decision.** `certain` evidence must carry a written explanation of why it admits no
|
|
64
|
+
benign explanation; enforced at runtime outside production, a warning inside it.
|
|
65
|
+
|
|
66
|
+
**Why.** It is a forcing function. If you cannot write the sentence, your evidence is
|
|
67
|
+
`strong`. Nearly every case where somebody wanted to mark something `certain` has failed at
|
|
68
|
+
this step, which is the point.
|
|
69
|
+
|
|
70
|
+
**The cost.** Slightly more ceremony to write a detector. The warning-not-throw behaviour in
|
|
71
|
+
production means a third-party detector with a missing basis cannot take a live site down.
|
|
72
|
+
|
|
73
|
+
## The engine never touches a response object
|
|
74
|
+
|
|
75
|
+
**The decision.** `assess` returns an `Assessment`, `decide` returns a `Decision`, `handle`
|
|
76
|
+
returns an `ActionOutcome`. Adapters apply it.
|
|
77
|
+
|
|
78
|
+
**Why.** The same policy then behaves identically on Express and on a Worker; the engine is
|
|
79
|
+
testable without a server; `assess()` is safe to run over a log file; and the
|
|
80
|
+
[corpus](../testing/corpus.md), the [request tester](../operations/dashboard.md) and
|
|
81
|
+
[`explain`](../testing/cli.md) all become possible for free.
|
|
82
|
+
|
|
83
|
+
**The cost.** One more layer, and a new framework needs thirty lines of adapter.
|
|
84
|
+
|
|
85
|
+
## Nothing may put an unbounded await on the request path
|
|
86
|
+
|
|
87
|
+
**The decision.** `io` detectors run concurrently under a timeout, and a `cheap` detector's
|
|
88
|
+
promise is timed out too. Event handlers and notification sinks are never awaited. The
|
|
89
|
+
[audit](../operations/audit.md) runs on a timer, not in-band. `shareConfirmations` reads the
|
|
90
|
+
store once per actor per instance, unawaited.
|
|
91
|
+
|
|
92
|
+
**Why.** This code runs inline on every request to the site it protects. A slow resolver, a
|
|
93
|
+
wedged webhook or a Redis failover must cost latency, not availability.
|
|
94
|
+
|
|
95
|
+
**The cost.** Detection is sometimes less complete than it could be — a timed-out detector
|
|
96
|
+
simply does not contribute. Correct: a bot filter that fails closed is an outage with extra
|
|
97
|
+
steps.
|
|
98
|
+
|
|
99
|
+
## Behavioural state is process-local; confirmations are not
|
|
100
|
+
|
|
101
|
+
**The decision.** Rates, cadence, path breadth and UA history live in memory per process.
|
|
102
|
+
`shareConfirmations` lets a *proven* verdict cross replicas.
|
|
103
|
+
|
|
104
|
+
**Why.** A round trip per request would buy accuracy for signals that are only ever allowed
|
|
105
|
+
to raise suspicion. A confirmation is not one of those: it is a fact about the client rather
|
|
106
|
+
than a judgement about it. **Proof travels; suspicion stays home.**
|
|
107
|
+
|
|
108
|
+
**The cost.** Behind four replicas each sees a quarter of an actor's traffic and is
|
|
109
|
+
correspondingly less sure. The right trade for something that cannot close a door on its own.
|
|
110
|
+
|
|
111
|
+
See [stores](../integration/stores.md).
|
|
112
|
+
|
|
113
|
+
## Everything client-keyed is a bounded LRU
|
|
114
|
+
|
|
115
|
+
**The decision.** 20,000 actors, 32 arrival timestamps, 64 path *hashes*, 4 User-Agents.
|
|
116
|
+
Every cache has a TTL and a ceiling.
|
|
117
|
+
|
|
118
|
+
**Why.** An unbounded map keyed by anything a client controls is a remote OOM.
|
|
119
|
+
|
|
120
|
+
**The cost.** `requestsWithin` saturates at 32 and reports `undercounted: true` rather than a
|
|
121
|
+
true rate. Deliberate — that series describes an actor cheaply, and exact counting belongs to
|
|
122
|
+
[`rate-limit`](../policy/actions.md#rate-limit), which uses the store.
|
|
123
|
+
|
|
124
|
+
## No IP intelligence, and no crawler ranges, ship with the library
|
|
125
|
+
|
|
126
|
+
**The decision.** `datacenterRanges` and `crawlerRanges` are empty. What ships is the *URL*
|
|
127
|
+
each crawler operator publishes.
|
|
128
|
+
|
|
129
|
+
**Why.** A range baked into a release is wrong by the time somebody installs it, and being
|
|
130
|
+
wrong here means verifying whoever has since been handed the address. A stale mapping is a
|
|
131
|
+
false positive with a long half-life.
|
|
132
|
+
|
|
133
|
+
**The cost.** `ip-intelligence` does nothing until you supply data, and crawler verification
|
|
134
|
+
by address needs an opt-in refresher that makes outbound requests.
|
|
135
|
+
|
|
136
|
+
See [verification](../detection/verification.md).
|
|
137
|
+
|
|
138
|
+
## Client signals are capped at `moderate`, permanently
|
|
139
|
+
|
|
140
|
+
**The decision.** Anything reported by JavaScript running in the client can never exceed
|
|
141
|
+
`moderate`, whatever it says.
|
|
142
|
+
|
|
143
|
+
**Why.** It is the one place an adversary has complete control. `navigator.webdriver`
|
|
144
|
+
proves what the client *chose to report*.
|
|
145
|
+
|
|
146
|
+
**The cost.** A genuinely conclusive-looking signal contributes modestly. That is the correct
|
|
147
|
+
weight for a witness the defendant controls.
|
|
148
|
+
|
|
149
|
+
See [client signals](../detection/client-signals.md).
|
|
150
|
+
|
|
151
|
+
## One root cause is counted once
|
|
152
|
+
|
|
153
|
+
**The decision.** Evidence may declare a `family`; within a family the strongest observation
|
|
154
|
+
is taken instead of compounding.
|
|
155
|
+
|
|
156
|
+
**Why.** Noisy-OR is only sound over independent signals. A corporate proxy that strips
|
|
157
|
+
`Sec-Fetch-*` also strips the Client Hints and the `Accept-Language`, so three detectors fire
|
|
158
|
+
about one person behind one appliance and the arithmetic reads their agreement as
|
|
159
|
+
corroboration when it is an echo.
|
|
160
|
+
|
|
161
|
+
**The cost.** Slightly lower scores on the traffic that should have them. A scraper that
|
|
162
|
+
copied a User-Agent and nothing else scores 66 rather than 91 — still comfortably
|
|
163
|
+
`suspected-bot`, still challenged.
|
|
164
|
+
|
|
165
|
+
## Zero runtime dependencies
|
|
166
|
+
|
|
167
|
+
**The decision.** The library imports nothing but `node:` builtins, and CI fails if that
|
|
168
|
+
stops being true. `RedisStore` describes the commands it needs structurally and imports
|
|
169
|
+
neither client.
|
|
170
|
+
|
|
171
|
+
**Why.** Nothing here can hand your project a transitive advisory, an install script, or a
|
|
172
|
+
version conflict with something you already run — for a package that sits on every request.
|
|
173
|
+
|
|
174
|
+
**The cost.** More code written here: an IP parser, an Aho–Corasick matcher, an LRU, an
|
|
175
|
+
emitter. All of them are exported, so at least they are useful twice.
|
|
176
|
+
|
|
177
|
+
## The dashboard client is built, not a template literal
|
|
178
|
+
|
|
179
|
+
**The decision.** 21 typed modules, esbuild-bundled into `client.generated.ts` at build time.
|
|
180
|
+
|
|
181
|
+
**Why.** It was a 2,836-line template literal, with no type checking, no linting and no
|
|
182
|
+
tests — inside a string, where a stray backtick in a CSS comment silently terminates the
|
|
183
|
+
program.
|
|
184
|
+
|
|
185
|
+
**The cost.** A build step, and a generated file in the tree. Worth it the first time the
|
|
186
|
+
type checker catches something.
|
|
187
|
+
|
|
188
|
+
## The corpus is a published entry point, not a test fixture
|
|
189
|
+
|
|
190
|
+
**The decision.** `@osqd/bothandlerjs/corpus` ships `runCorpus` and all 526 cases.
|
|
191
|
+
|
|
192
|
+
**Why.** The claims in this documentation are only worth anything if you can check them
|
|
193
|
+
against *your* configuration. A corpus locked inside the test suite proves things about the
|
|
194
|
+
library's own presets and nothing about yours.
|
|
195
|
+
|
|
196
|
+
**The cost.** Package size, and a public API surface that now has to stay stable.
|
|
197
|
+
|
|
198
|
+
See [the corpus](../testing/corpus.md).
|
|
199
|
+
|
|
200
|
+
## The benchmark measures ratios, not microseconds
|
|
201
|
+
|
|
202
|
+
**The decision.** Every budget is a multiple of a reference loop measured in the same
|
|
203
|
+
process, seconds earlier.
|
|
204
|
+
|
|
205
|
+
**Why.** A committed baseline in microseconds is a statement about the machine that produced
|
|
206
|
+
it. CI runners are shared, throttled and re-provisioned; a suite that fails for reasons
|
|
207
|
+
unrelated to the code is a suite people disable. A runner half the speed of a laptop runs
|
|
208
|
+
both halves at half speed and the ratio is unchanged.
|
|
209
|
+
|
|
210
|
+
**The cost.** The numbers are less immediately meaningful. Budgets are set at roughly double
|
|
211
|
+
current cost — a guard, not a tripwire.
|
|
212
|
+
|
|
213
|
+
## Related
|
|
214
|
+
|
|
215
|
+
- [Evidence](../concepts/evidence.md) · [The guard](../concepts/the-guard.md) · [Threat model](../concepts/threat-model.md)
|
|
216
|
+
- [Configuration](../reference/configuration.md) — where these show up as settings
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# Browser signals
|
|
2
|
+
|
|
3
|
+
An optional page script, and the hard ceiling on anything it reports.
|
|
4
|
+
|
|
5
|
+
← [Documentation](../index.md) · [The detectors](detectors.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
Everything else in this library reads what arrived over the wire. This reads what the
|
|
10
|
+
browser says about itself — which is more informative and much less trustworthy, and the
|
|
11
|
+
design is mostly about holding those two facts together.
|
|
12
|
+
|
|
13
|
+
## What it reports
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
interface ClientSignals {
|
|
17
|
+
webdriver?: boolean; // navigator.webdriver === true
|
|
18
|
+
noLanguages?: boolean; // navigator.languages is empty
|
|
19
|
+
zeroDimensions?: boolean; // screen.width === 0
|
|
20
|
+
inconsistentPlatform?: boolean; // userAgentData.platform disagrees with the UA string
|
|
21
|
+
interacted?: boolean; // a trusted pointer/key event happened
|
|
22
|
+
msToInteraction?: number;
|
|
23
|
+
automationGlobals?: string[]; // _phantom, __nightmare, __selenium_unwrapped, …
|
|
24
|
+
}
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
That is the whole list, and the omissions are the point. No canvas hash, no WebGL
|
|
28
|
+
renderer, no font enumeration, no audio fingerprint. Those would raise accuracy and would
|
|
29
|
+
make this a fingerprinting library, which is a different product with different ethics —
|
|
30
|
+
see [design decisions](../design/decisions.md).
|
|
31
|
+
|
|
32
|
+
## Wiring it up
|
|
33
|
+
|
|
34
|
+
Three pieces: render the script, receive the report, attach it to the request.
|
|
35
|
+
|
|
36
|
+
```ts
|
|
37
|
+
import { renderClientScript, parseClientSignals } from "@osqd/bothandlerjs/client";
|
|
38
|
+
|
|
39
|
+
// 1. On any page you want signals from. The nonce matches your CSP.
|
|
40
|
+
app.get("/", (req, res) =>
|
|
41
|
+
res.send(page + renderClientScript({ endpoint: "/__signals", nonce: res.locals.nonce })),
|
|
42
|
+
);
|
|
43
|
+
|
|
44
|
+
// 2. An endpoint that stores what it receives against the session.
|
|
45
|
+
app.post("/__signals", express.json({ limit: "2kb" }), (req, res) => {
|
|
46
|
+
const signals = parseClientSignals(req.body);
|
|
47
|
+
if (signals) req.session.botSignals = signals;
|
|
48
|
+
res.status(204).end();
|
|
49
|
+
});
|
|
50
|
+
|
|
51
|
+
// 3. Attach it, so the detector can read it.
|
|
52
|
+
app.use(
|
|
53
|
+
botHandler(detector, {
|
|
54
|
+
enrich: (request, facts) => ({ ...facts, extra: { clientSignals: request.session?.botSignals } }),
|
|
55
|
+
}),
|
|
56
|
+
);
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
**Where the report lives between step 2 and step 3 is your decision, not the library's.**
|
|
60
|
+
A session store, a short-lived cache keyed by a cookie, an edge KV — the library does not
|
|
61
|
+
invent a storage mechanism, because where per-session data lives is a question only your
|
|
62
|
+
application can answer.
|
|
63
|
+
|
|
64
|
+
The script reports on `sendBeacon` (falling back to `fetch` with `keepalive`), so a short
|
|
65
|
+
visit still reports as the page goes away, and it reports again after the first trusted
|
|
66
|
+
interaction if one happens within `interactionWindowMs`.
|
|
67
|
+
|
|
68
|
+
## Why it is capped at `moderate`
|
|
69
|
+
|
|
70
|
+
Permanently, and no option changes it.
|
|
71
|
+
|
|
72
|
+
Every value here was produced by JavaScript running inside the client, which is the one
|
|
73
|
+
place an adversary has complete control. A framework that wants `navigator.webdriver` to
|
|
74
|
+
read `false` sets it to `false`, and everything the detector sees afterwards is whatever
|
|
75
|
+
that framework decided to say.
|
|
76
|
+
|
|
77
|
+
What this genuinely catches is **automation that never bothered to hide** — Selenium out
|
|
78
|
+
of the box, a scripted Chrome somebody pointed at your site this afternoon — which is a
|
|
79
|
+
large share of real bot traffic and worth catching. What it must never do is convince you
|
|
80
|
+
that a clean report means a person.
|
|
81
|
+
|
|
82
|
+
`isTrusted` on the interaction event is the most informative bit on the page, and it is
|
|
83
|
+
still forgeable by a framework driving a real browser. Treat it as evidence that somebody
|
|
84
|
+
*probably* touched the page, not as proof that anybody did.
|
|
85
|
+
|
|
86
|
+
## What the absence of a report means
|
|
87
|
+
|
|
88
|
+
Nothing. A visitor with JavaScript disabled, a request that never reached an HTML page, a
|
|
89
|
+
beacon that was blocked — all produce no signals, and the detector stays silent rather
|
|
90
|
+
than treating silence as suspicion. A great many real people browse this way.
|
|
91
|
+
|
|
92
|
+
## Related
|
|
93
|
+
|
|
94
|
+
- [The detectors](detectors.md#client-signals)
|
|
95
|
+
- [Design decisions](../design/decisions.md) — why there is no fingerprinting here
|
|
96
|
+
- [The challenge](../challenge/index.md) — the other thing that runs in a browser
|
|
@@ -0,0 +1,376 @@
|
|
|
1
|
+
# The detectors
|
|
2
|
+
|
|
3
|
+
All twenty, each with what it reads, why it exists, what it can conclude, and what it
|
|
4
|
+
costs.
|
|
5
|
+
|
|
6
|
+
← [Documentation](../index.md) · [How detection works](index.md)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
Every detector returns **evidence**, never a verdict. What each one is allowed to
|
|
11
|
+
conclude is as important as what it looks at, so every entry below names its ceiling.
|
|
12
|
+
|
|
13
|
+
- **cost** — `cheap` is synchronous and in-memory; `io` may make a network call.
|
|
14
|
+
- **stage** — `always` runs on every request; `confirming` runs only when a signature
|
|
15
|
+
matched, because there is nothing to confirm otherwise.
|
|
16
|
+
- **ceiling** — the highest certainty this detector can produce. Only `certain` can cost
|
|
17
|
+
somebody their access; see [the guard](../concepts/the-guard.md).
|
|
18
|
+
|
|
19
|
+
```ts
|
|
20
|
+
import { defaultDetectors } from "@osqd/bothandlerjs";
|
|
21
|
+
|
|
22
|
+
new BotHandler({ detectors: defaultDetectors() }); // the default set
|
|
23
|
+
new BotHandler({ extraDetectors: [myDetector()] }); // add to it
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
`bothandlerjs detectors --preset protect-content` lists what a configuration actually
|
|
27
|
+
installs, which is the reliable answer for your setup.
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## Single-request detectors
|
|
32
|
+
|
|
33
|
+
These need nothing but the request in front of them. They work on a replayed log line as
|
|
34
|
+
well as on live traffic.
|
|
35
|
+
|
|
36
|
+
### `self-identified`
|
|
37
|
+
|
|
38
|
+
**cheap · always · ceiling `certain`**
|
|
39
|
+
|
|
40
|
+
The client told us what it is. This is the backbone of the proven tier.
|
|
41
|
+
|
|
42
|
+
When a request arrives saying `python-requests/2.31.0` or `Googlebot/2.1`, nothing is
|
|
43
|
+
being *inferred*. We are taking the client at its word — and if that word is a lie, the
|
|
44
|
+
misclassification belongs to whoever lied, not to a heuristic that guessed wrong about a
|
|
45
|
+
person. That is a point about responsibility rather than about technology, and it is why
|
|
46
|
+
this tier can do what the others may not.
|
|
47
|
+
|
|
48
|
+
What it recognises: the [signature database](signatures.md) (search crawlers, AI
|
|
49
|
+
crawlers, monitors, scanners, social unfurlers), bare library tokens (`curl/`, `okhttp`,
|
|
50
|
+
`Go-http-client`), and User-Agents that announce automation outright.
|
|
51
|
+
|
|
52
|
+
```ts
|
|
53
|
+
// curl/8.4.0
|
|
54
|
+
{ certainty: "certain", summary: "User-Agent identifies curl",
|
|
55
|
+
deterministicBasis: 'The product token "curl/" is emitted by an HTTP library or an ' +
|
|
56
|
+
"automation runtime and by no browser. Nothing a person does in a browser produces it." }
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
A bare token with no browser preamble is `strong` rather than `certain` when it could
|
|
60
|
+
plausibly be a person's own script; the database's own entries are `certain`.
|
|
61
|
+
|
|
62
|
+
### `ua-coherence`
|
|
63
|
+
|
|
64
|
+
**cheap · always · ceiling `strong`**
|
|
65
|
+
|
|
66
|
+
Does this User-Agent contradict *itself*?
|
|
67
|
+
|
|
68
|
+
Every other single-request check compares the User-Agent against something else — the
|
|
69
|
+
Client Hints, the header set, the header order. This one needs nothing but the string,
|
|
70
|
+
which makes it the only consistency check that still works on a source with no headers at
|
|
71
|
+
all: an nginx access line, a CDN log, a WAF event. That is not a small thing when a
|
|
72
|
+
replay is the main way people evaluate a policy.
|
|
73
|
+
|
|
74
|
+
It reads engines, platforms and versions against each other: a Safari token on Windows, a
|
|
75
|
+
Chrome version that does not exist, a mobile token with a desktop platform.
|
|
76
|
+
|
|
77
|
+
### `header-integrity`
|
|
78
|
+
|
|
79
|
+
**cheap · always · ceiling `certain`**
|
|
80
|
+
|
|
81
|
+
Compares the header set against what the client it claims to be would actually send.
|
|
82
|
+
|
|
83
|
+
The `certain` cases here are **protocol violations** rather than absences: two `Host`
|
|
84
|
+
headers, a second `Content-Length`, a connection-specific header on HTTP/2. RFC 9112
|
|
85
|
+
requires a recipient to reject those; no compliant client emits them.
|
|
86
|
+
|
|
87
|
+
Everything else — a browser claim with no `Accept`, no `Accept-Language`, no
|
|
88
|
+
`Accept-Encoding` — is `moderate` or `strong` and shares a **family**, because a client
|
|
89
|
+
missing three browser headers has one property reported three times. See
|
|
90
|
+
[families](../concepts/evidence.md#families).
|
|
91
|
+
|
|
92
|
+
```ts
|
|
93
|
+
headerIntegrityDetector({ requireAcceptLanguage: false }) // for an API-only origin
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
### `header-order`
|
|
97
|
+
|
|
98
|
+
**cheap · always · ceiling `moderate`**
|
|
99
|
+
|
|
100
|
+
The order headers arrived in, against orderings every mainstream browser respects.
|
|
101
|
+
|
|
102
|
+
Header order is a fingerprint that a scraper copying your browser's headers usually gets
|
|
103
|
+
wrong, because it sets them from a dictionary. The rules are stated as invariants — "`X`
|
|
104
|
+
must not appear after `Y`" — verified against Chrome, Firefox, Safari and Edge, and kept
|
|
105
|
+
deliberately few. The value is in rules that no browser breaks; modelling any browser's
|
|
106
|
+
exact sequence would turn this into a source of false positives on next month's release.
|
|
107
|
+
|
|
108
|
+
Needs `rawHeaders`; see [adapters](../integration/adapters.md).
|
|
109
|
+
|
|
110
|
+
### `client-hints`
|
|
111
|
+
|
|
112
|
+
**cheap · always · ceiling `certain`**
|
|
113
|
+
|
|
114
|
+
Cross-checks `Sec-CH-UA` against the legacy User-Agent string.
|
|
115
|
+
|
|
116
|
+
A real Chromium generates both from the same internal state, so they always agree. A
|
|
117
|
+
client that rewrites one and forgets the other contradicts itself — visible from a single
|
|
118
|
+
request, with no history, no state and no network call, which makes it one of the cheapest
|
|
119
|
+
high-value checks there is.
|
|
120
|
+
|
|
121
|
+
Most of it is `strong` rather than `certain`, because the population that rewrites a
|
|
122
|
+
User-Agent without touching Client Hints includes privacy extensions and corporate
|
|
123
|
+
middleboxes as well as scrapers.
|
|
124
|
+
|
|
125
|
+
### `fetch-metadata`
|
|
126
|
+
|
|
127
|
+
**cheap · always · ceiling `strong`**
|
|
128
|
+
|
|
129
|
+
`Sec-Fetch-Site`, `-Mode`, `-Dest`, `-User`.
|
|
130
|
+
|
|
131
|
+
These are **forbidden headers**: page JavaScript cannot set or alter them, so in a real
|
|
132
|
+
browser they are generated by the network stack from the actual context of the request.
|
|
133
|
+
That makes them unusually hard to forge from inside a page — and their *absence* on an
|
|
134
|
+
engine known to send them is informative.
|
|
135
|
+
|
|
136
|
+
It knows which engine versions ship them (Blink 76+, Gecko 90+), so an old browser is not
|
|
137
|
+
accused of missing something it never sent.
|
|
138
|
+
|
|
139
|
+
### `accept-signature`
|
|
140
|
+
|
|
141
|
+
**cheap · always · ceiling `strong`**
|
|
142
|
+
|
|
143
|
+
`Accept` and `Accept-Language` as a fingerprint of what the client will do with the
|
|
144
|
+
response.
|
|
145
|
+
|
|
146
|
+
A browser asking for a page sends a long, specific `Accept` describing the document
|
|
147
|
+
formats it renders. A scraper sends `*/*` because it will take anything and parse it
|
|
148
|
+
itself. Malformed language lists — values that are not RFC 9110 language ranges — are a
|
|
149
|
+
separate and stronger signal, because a browser cannot produce one.
|
|
150
|
+
|
|
151
|
+
### `probe-signature`
|
|
152
|
+
|
|
153
|
+
**cheap · always · ceiling `strong`**
|
|
154
|
+
|
|
155
|
+
What is this request *asking for*?
|
|
156
|
+
|
|
157
|
+
Every other detector reads the client; this one reads the target, and it exists to close a
|
|
158
|
+
gap the rest cannot: the scanner that does not announce itself. `self-identified` catches
|
|
159
|
+
sqlmap and Nikto because they say so, and a great deal of hostile traffic does say so. The
|
|
160
|
+
rest arrives wearing an ordinary browser User-Agent and asks for `/.env`,
|
|
161
|
+
`/wp-admin/setup-config.php`, `/../../etc/passwd` or a path with a SQL payload in the
|
|
162
|
+
query string.
|
|
163
|
+
|
|
164
|
+
```ts
|
|
165
|
+
probeSignatureDetector({ extraPaths: ["/internal/admin"] })
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
### `tls-fingerprint`
|
|
169
|
+
|
|
170
|
+
**cheap · always · ceiling `strong`**
|
|
171
|
+
|
|
172
|
+
Compares an edge-supplied JA3/JA4 handshake fingerprint against the client the User-Agent
|
|
173
|
+
claims to be.
|
|
174
|
+
|
|
175
|
+
This is the signal that survives a scraper copying every header perfectly: the TLS
|
|
176
|
+
handshake is produced by the client's TLS stack rather than by its HTTP code, so a Go or
|
|
177
|
+
Python program wearing Chrome's headers still shakes hands like Go or Python.
|
|
178
|
+
|
|
179
|
+
The library does not compute fingerprints — it cannot see the handshake — so it reads one
|
|
180
|
+
your edge computed:
|
|
181
|
+
|
|
182
|
+
```ts
|
|
183
|
+
createFetchAdapter(handler, { tlsFingerprintHeader: "cf-ja3-hash" });
|
|
184
|
+
|
|
185
|
+
tlsFingerprintDetector({ profiles: { "<ja3 hash>": { name: "Chrome 122", engine: "blink" } } })
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Without profiles it stays quiet rather than guessing.
|
|
189
|
+
|
|
190
|
+
### `ip-intelligence`
|
|
191
|
+
|
|
192
|
+
**cheap · always · ceiling `certain`**
|
|
193
|
+
|
|
194
|
+
What is known about where this came from. Two range sets, treated very differently.
|
|
195
|
+
|
|
196
|
+
**`denylist` is `certain`** — and the justification is not technical. *You* configured it.
|
|
197
|
+
The library is not inferring anything; it is carrying out an instruction you gave about
|
|
198
|
+
addresses you decided about.
|
|
199
|
+
|
|
200
|
+
**`datacenterRanges` is `moderate`** — a server address is a fact about hosting, not about
|
|
201
|
+
intent. VPNs, corporate egress and privacy relays all live there, and so do plenty of
|
|
202
|
+
people.
|
|
203
|
+
|
|
204
|
+
```ts
|
|
205
|
+
new BotHandler({
|
|
206
|
+
denylist: ["203.0.113.0/24"],
|
|
207
|
+
datacenterRanges: ["198.51.100.0/16"],
|
|
208
|
+
});
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
The library ships **no address data** and refuses to guess any; see
|
|
212
|
+
[design decisions](../design/decisions.md).
|
|
213
|
+
|
|
214
|
+
### `trap`
|
|
215
|
+
|
|
216
|
+
**cheap · always · ceiling `certain`**
|
|
217
|
+
|
|
218
|
+
A path no link points at, a form field no rendered browser shows, a header only a script
|
|
219
|
+
would echo. Reaching one requires reading the page as data rather than as a page.
|
|
220
|
+
|
|
221
|
+
```ts
|
|
222
|
+
import { renderTrapLink, trapRobotsEntries, DEFAULT_TRAP_PATHS } from "@osqd/bothandlerjs";
|
|
223
|
+
|
|
224
|
+
app.get("/", (_req, res) => res.send(page + renderTrapLink()));
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
The trap paths belong in your `robots.txt` as `Disallow`, which is what makes the evidence
|
|
228
|
+
fair: a crawler that obeys robots never sees them, so touching one is a decision.
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
## Behavioural detectors
|
|
233
|
+
|
|
234
|
+
These read [the actor's history](../concepts/actors.md). They are the reason the library
|
|
235
|
+
keeps state at all, and none of them can reach `certain` — behaviour is a judgement.
|
|
236
|
+
|
|
237
|
+
### `cadence`
|
|
238
|
+
|
|
239
|
+
**cheap · always · ceiling `moderate`**
|
|
240
|
+
|
|
241
|
+
Is this actor's *rhythm* human?
|
|
242
|
+
|
|
243
|
+
People generate ragged inter-arrival times: they read, scroll, get distracted, open three
|
|
244
|
+
tabs, then nothing for four minutes. A loop calling `setInterval` produces gaps clustered
|
|
245
|
+
tightly around one value, and the coefficient of variation makes that visible in a single
|
|
246
|
+
number.
|
|
247
|
+
|
|
248
|
+
The check is on *regularity*, not speed, which is what makes it complementary to
|
|
249
|
+
`rate-anomaly`: a slow, polite scraper pacing itself at one request every five seconds is
|
|
250
|
+
invisible to a rate check and obvious to this one.
|
|
251
|
+
|
|
252
|
+
### `rate-anomaly`
|
|
253
|
+
|
|
254
|
+
**cheap · always · ceiling `moderate`**
|
|
255
|
+
|
|
256
|
+
How fast is this actor going?
|
|
257
|
+
|
|
258
|
+
Rate is the signal people reach for first and trust most, and it deserves the least trust
|
|
259
|
+
of anything here. The *actor* behind a high rate is frequently not one client: a corporate
|
|
260
|
+
NAT, a CGNAT pool, a university, a VPN exit and a shared office all present hundreds of
|
|
261
|
+
real people under one key.
|
|
262
|
+
|
|
263
|
+
So it is deliberately conservative, and it is the detector most improved by a better
|
|
264
|
+
[`actorKey`](../concepts/actors.md#the-actor-key).
|
|
265
|
+
|
|
266
|
+
### `crawl-breadth`
|
|
267
|
+
|
|
268
|
+
**cheap · always · ceiling `weak`**
|
|
269
|
+
|
|
270
|
+
Reading the site, or enumerating it?
|
|
271
|
+
|
|
272
|
+
A person revisits: they land on an article, go back to the index, follow a related link,
|
|
273
|
+
return. Their ratio of distinct paths to total requests settles well below one. A crawler
|
|
274
|
+
walking a sitemap almost never revisits, so its ratio sits near one.
|
|
275
|
+
|
|
276
|
+
`weak`, because a *welcome* crawler produces exactly this shape and so does a person on a
|
|
277
|
+
first visit to a documentation site.
|
|
278
|
+
|
|
279
|
+
### `identity-rotation`
|
|
280
|
+
|
|
281
|
+
**cheap · always · ceiling `moderate` · off by default**
|
|
282
|
+
|
|
283
|
+
One actor, several User-Agents. A single client does not change its User-Agent
|
|
284
|
+
mid-session; something that does is cycling through a spoofing list.
|
|
285
|
+
|
|
286
|
+
**Off by default, and think before enabling it.** With the default address-based actor
|
|
287
|
+
key, a corporate NAT presents a hundred people's browsers as one actor with a hundred
|
|
288
|
+
User-Agents — which is this detector's exact signature and is entirely innocent. Enable it
|
|
289
|
+
when your `actorKey` identifies a session rather than a network.
|
|
290
|
+
|
|
291
|
+
### `session-integrity`
|
|
292
|
+
|
|
293
|
+
**cheap · always · ceiling `moderate`**
|
|
294
|
+
|
|
295
|
+
Does this client hold a session?
|
|
296
|
+
|
|
297
|
+
A browser accumulates state: once your server has set anything — a session cookie, a
|
|
298
|
+
consent flag, an A/B bucket — a real browser sends it back on every subsequent request. A
|
|
299
|
+
stateless HTTP client sends nothing back however many times it visits.
|
|
300
|
+
|
|
301
|
+
It only speaks about clients that *claim* to be browsers, and only after your origin has
|
|
302
|
+
had a chance to set something.
|
|
303
|
+
|
|
304
|
+
### `browsing-coherence`
|
|
305
|
+
|
|
306
|
+
**cheap · always · ceiling `moderate` (human)**
|
|
307
|
+
|
|
308
|
+
The one that argues the other way.
|
|
309
|
+
|
|
310
|
+
It reports the marks of a real browsing session — a cache validator (`If-None-Match`), a
|
|
311
|
+
cookie jar, a same-site navigation, a plausible `Referer` chain — as **human** evidence,
|
|
312
|
+
which the scoring model subtracts from the bot score.
|
|
313
|
+
|
|
314
|
+
Without it the model has an asymmetry that shows up as false positives on exactly the
|
|
315
|
+
people least able to afford them: a researcher on a shared university address reading forty
|
|
316
|
+
pages accumulates four suspicions and nothing at all in their defence.
|
|
317
|
+
|
|
318
|
+
### `clearance`
|
|
319
|
+
|
|
320
|
+
**cheap · always · ceiling `certain` (human)**
|
|
321
|
+
|
|
322
|
+
Reads a signed clearance token the client already holds. The library's only source of
|
|
323
|
+
human-pointing evidence that is not a guess about headers, and the levels are exact
|
|
324
|
+
because overstating any of them would undo the design:
|
|
325
|
+
|
|
326
|
+
| Level | Certainty | |
|
|
327
|
+
| ----- | --------- | --- |
|
|
328
|
+
| `operator` | `certain` | Your application said this is a person. We believe you. |
|
|
329
|
+
| `pow` | `moderate` | A proof of work was solved. A headless browser solves it too. |
|
|
330
|
+
|
|
331
|
+
See [the challenge](../challenge/index.md).
|
|
332
|
+
|
|
333
|
+
### `client-signals`
|
|
334
|
+
|
|
335
|
+
**cheap · always · ceiling `moderate`**
|
|
336
|
+
|
|
337
|
+
Reads what the optional [browser script](client-signals.md) reported: `navigator.webdriver`,
|
|
338
|
+
an empty language list, zero screen dimensions, a platform that disagrees with the
|
|
339
|
+
User-Agent, whether a trusted interaction happened.
|
|
340
|
+
|
|
341
|
+
**Capped at `moderate`, hard.** Every value here was produced by JavaScript running inside
|
|
342
|
+
the client, which is the one place an adversary has complete control. A framework that
|
|
343
|
+
wants `navigator.webdriver` to read `false` sets it to `false`. What this genuinely catches
|
|
344
|
+
is automation that never bothered to hide — which is a large share of real bot traffic —
|
|
345
|
+
and what it must never do is convince you that a clean report means a person.
|
|
346
|
+
|
|
347
|
+
---
|
|
348
|
+
|
|
349
|
+
## Confirming detectors
|
|
350
|
+
|
|
351
|
+
### `crawler-verification`
|
|
352
|
+
|
|
353
|
+
**io · confirming · ceiling `certain`**
|
|
354
|
+
|
|
355
|
+
Confirms or refutes a claimed crawler identity. Runs only when a signature matched, because
|
|
356
|
+
with no claim there is nothing to confirm and no lookup to make.
|
|
357
|
+
|
|
358
|
+
Two mechanisms, depending on what the operator publishes:
|
|
359
|
+
|
|
360
|
+
- **Forward-confirmed reverse DNS.** `PTR` the address, check the name is under the
|
|
361
|
+
crawler's domain, then resolve that name forward and check it comes back to the same
|
|
362
|
+
address. A confirmation is `verified-bot`; a **refutation** is `certain` evidence of an
|
|
363
|
+
impersonator, and the strongest thing this library ever concludes about a forgery.
|
|
364
|
+
- **Published address ranges.** A lookup rather than a round trip. See
|
|
365
|
+
[verifying a crawler](verification.md).
|
|
366
|
+
|
|
367
|
+
A resolver that is merely *unhappy* must reach a different verdict from one that
|
|
368
|
+
*disproves* the claim, and it does: a timeout produces no evidence at all.
|
|
369
|
+
|
|
370
|
+
---
|
|
371
|
+
|
|
372
|
+
## Related
|
|
373
|
+
|
|
374
|
+
- [How detection works](index.md) — the pipeline these run in
|
|
375
|
+
- [Writing a detector](writing-a-detector.md) — the contract, and the rules on certainty
|
|
376
|
+
- [Evidence and certainty](../concepts/evidence.md) — what the tiers mean
|