@osqd/bothandlerjs 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +516 -0
- package/LICENSE +102 -0
- package/README.md +308 -0
- package/bin/bothandlerjs.mjs +14 -0
- package/dist/actions/index.d.ts +22 -0
- package/dist/actions/types.d.ts +59 -0
- package/dist/adapters/fastify.d.ts +43 -0
- package/dist/adapters/fetch.d.ts +51 -0
- package/dist/adapters/index.cjs +723 -0
- package/dist/adapters/index.cjs.map +1 -0
- package/dist/adapters/index.d.ts +8 -0
- package/dist/adapters/index.js +692 -0
- package/dist/adapters/index.js.map +1 -0
- package/dist/adapters/koa.d.ts +27 -0
- package/dist/adapters/node.d.ts +29 -0
- package/dist/adapters/shared.d.ts +39 -0
- package/dist/audit.d.ts +172 -0
- package/dist/challenge/index.d.ts +176 -0
- package/dist/challenge/language.d.ts +57 -0
- package/dist/challenge/page.d.ts +42 -0
- package/dist/challenge/pow.d.ts +33 -0
- package/dist/challenge/token.d.ts +72 -0
- package/dist/cli.cjs +13388 -0
- package/dist/cli.cjs.map +1 -0
- package/dist/cli.d.ts +5 -0
- package/dist/cli.js +13358 -0
- package/dist/cli.js.map +1 -0
- package/dist/client/index.cjs +139 -0
- package/dist/client/index.cjs.map +1 -0
- package/dist/client/index.d.ts +63 -0
- package/dist/client/index.js +112 -0
- package/dist/client/index.js.map +1 -0
- package/dist/config.d.ts +316 -0
- package/dist/core.d.ts +443 -0
- package/dist/corpus/adversarial.d.ts +2 -0
- package/dist/corpus/advertising-email.d.ts +2 -0
- package/dist/corpus/ai-crawlers.d.ts +2 -0
- package/dist/corpus/benign-bots.d.ts +2 -0
- package/dist/corpus/cdn-gateways.d.ts +2 -0
- package/dist/corpus/cookies.d.ts +49 -0
- package/dist/corpus/crawlers-regional.d.ts +2 -0
- package/dist/corpus/crawlers-vertical.d.ts +2 -0
- package/dist/corpus/headers.d.ts +403 -0
- package/dist/corpus/humans-apps.d.ts +2 -0
- package/dist/corpus/humans-browsers.d.ts +2 -0
- package/dist/corpus/humans.d.ts +2 -0
- package/dist/corpus/index.cjs +4196 -0
- package/dist/corpus/index.cjs.map +1 -0
- package/dist/corpus/index.d.ts +37 -0
- package/dist/corpus/index.js +4132 -0
- package/dist/corpus/index.js.map +1 -0
- package/dist/corpus/infrastructure.d.ts +17 -0
- package/dist/corpus/libraries-extended.d.ts +2 -0
- package/dist/corpus/ranges.d.ts +31 -0
- package/dist/corpus/reputation.d.ts +21 -0
- package/dist/corpus/runner.d.ts +130 -0
- package/dist/corpus/schema.d.ts +185 -0
- package/dist/corpus/tooling.d.ts +2 -0
- package/dist/corpus/unwanted.d.ts +2 -0
- package/dist/crawler-ranges.d.ts +113 -0
- package/dist/dashboard/client.generated.d.ts +7 -0
- package/dist/dashboard/feed.d.ts +186 -0
- package/dist/dashboard/index.d.ts +5 -0
- package/dist/dashboard/page.d.ts +51 -0
- package/dist/dashboard/parse-request.d.ts +28 -0
- package/dist/dashboard/preview.d.ts +45 -0
- package/dist/dashboard/server.d.ts +70 -0
- package/dist/dashboard/types.d.ts +714 -0
- package/dist/detectors/accept-signature.d.ts +6 -0
- package/dist/detectors/browsing-coherence.d.ts +44 -0
- package/dist/detectors/cadence.d.ts +28 -0
- package/dist/detectors/clearance.d.ts +25 -0
- package/dist/detectors/client-hints.d.ts +19 -0
- package/dist/detectors/client-signals.d.ts +27 -0
- package/dist/detectors/crawl-breadth.d.ts +28 -0
- package/dist/detectors/crawler-verification.d.ts +39 -0
- package/dist/detectors/fetch-metadata.d.ts +20 -0
- package/dist/detectors/header-integrity.d.ts +26 -0
- package/dist/detectors/header-order.d.ts +27 -0
- package/dist/detectors/identity-rotation.d.ts +34 -0
- package/dist/detectors/index.d.ts +56 -0
- package/dist/detectors/ip-intelligence.d.ts +30 -0
- package/dist/detectors/known-bots.d.ts +94 -0
- package/dist/detectors/probe-signature.d.ts +50 -0
- package/dist/detectors/rate-anomaly.d.ts +36 -0
- package/dist/detectors/self-identified.d.ts +33 -0
- package/dist/detectors/session-integrity.d.ts +24 -0
- package/dist/detectors/tls-fingerprint.d.ts +34 -0
- package/dist/detectors/trap.d.ts +98 -0
- package/dist/detectors/types.d.ts +70 -0
- package/dist/detectors/ua-coherence.d.ts +32 -0
- package/dist/evidence.d.ts +45 -0
- package/dist/facts.d.ts +46 -0
- package/dist/index.cjs +9600 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.ts +63 -0
- package/dist/index.js +9444 -0
- package/dist/index.js.map +1 -0
- package/dist/internal/async.d.ts +16 -0
- package/dist/internal/clock.d.ts +17 -0
- package/dist/internal/crypto.d.ts +26 -0
- package/dist/internal/dns.d.ts +100 -0
- package/dist/internal/emitter.d.ts +18 -0
- package/dist/internal/http.d.ts +22 -0
- package/dist/internal/ip.d.ts +73 -0
- package/dist/internal/lru.d.ts +34 -0
- package/dist/internal/matcher.d.ts +41 -0
- package/dist/internal/pattern.d.ts +14 -0
- package/dist/internal/ua.d.ts +92 -0
- package/dist/metrics.d.ts +120 -0
- package/dist/notify/hub.d.ts +70 -0
- package/dist/notify/index.d.ts +7 -0
- package/dist/notify/redact.d.ts +48 -0
- package/dist/notify/sinks.d.ts +62 -0
- package/dist/notify/types.d.ts +56 -0
- package/dist/policy/index.d.ts +7 -0
- package/dist/policy/match.d.ts +20 -0
- package/dist/policy/policy.d.ts +123 -0
- package/dist/policy/presets.d.ts +161 -0
- package/dist/policy/types.d.ts +155 -0
- package/dist/robots.d.ts +83 -0
- package/dist/state.d.ts +161 -0
- package/dist/stores/index.d.ts +5 -0
- package/dist/stores/memory.d.ts +33 -0
- package/dist/stores/redis.d.ts +59 -0
- package/dist/stores/types.d.ts +37 -0
- package/dist/types.d.ts +250 -0
- package/docs/challenge/index.md +152 -0
- package/docs/challenge/localisation.md +105 -0
- package/docs/concepts/actors.md +113 -0
- package/docs/concepts/evidence.md +133 -0
- package/docs/concepts/the-guard.md +121 -0
- package/docs/concepts/threat-model.md +120 -0
- package/docs/concepts/verdicts.md +108 -0
- package/docs/course/01-first-assessment.md +178 -0
- package/docs/course/02-proof-and-suspicion.md +164 -0
- package/docs/course/03-verdicts-and-scores.md +160 -0
- package/docs/course/04-the-guard.md +200 -0
- package/docs/course/05-detectors.md +178 -0
- package/docs/course/06-identity.md +161 -0
- package/docs/course/07-actors.md +178 -0
- package/docs/course/08-traps.md +183 -0
- package/docs/course/09-rules.md +224 -0
- package/docs/course/10-actions-and-presets.md +216 -0
- package/docs/course/11-the-challenge.md +227 -0
- package/docs/course/12-going-live.md +211 -0
- package/docs/course/13-operating-it.md +238 -0
- package/docs/course/14-scaling.md +195 -0
- package/docs/course/15-extending.md +259 -0
- package/docs/course/16-proving-it.md +266 -0
- package/docs/course/index.md +112 -0
- package/docs/design/decisions.md +216 -0
- package/docs/detection/client-signals.md +96 -0
- package/docs/detection/detectors.md +376 -0
- package/docs/detection/index.md +134 -0
- package/docs/detection/signatures.md +111 -0
- package/docs/detection/verification.md +123 -0
- package/docs/detection/writing-a-detector.md +145 -0
- package/docs/index.md +100 -0
- package/docs/integration/adapters.md +138 -0
- package/docs/integration/client-ip.md +114 -0
- package/docs/integration/index.md +55 -0
- package/docs/integration/stores.md +76 -0
- package/docs/operations/audit.md +116 -0
- package/docs/operations/dashboard.md +725 -0
- package/docs/operations/index.md +101 -0
- package/docs/operations/metrics.md +74 -0
- package/docs/operations/notifications.md +72 -0
- package/docs/operations/runtime-changes.md +118 -0
- package/docs/policy/actions.md +196 -0
- package/docs/policy/index.md +84 -0
- package/docs/policy/presets.md +191 -0
- package/docs/policy/robots.md +128 -0
- package/docs/policy/rules.md +134 -0
- package/docs/reference/api.md +126 -0
- package/docs/reference/configuration.md +175 -0
- package/docs/start/choosing-a-policy.md +127 -0
- package/docs/start/first-integration.md +127 -0
- package/docs/start/installation.md +63 -0
- package/docs/start/upgrading.md +72 -0
- package/docs/testing/cli.md +119 -0
- package/docs/testing/corpus.md +129 -0
- package/docs/testing/index.md +61 -0
- package/docs/testing/replay.md +102 -0
- package/docs/testing/try-it.md +189 -0
- package/package.json +145 -0
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
# Presets
|
|
2
|
+
|
|
3
|
+
Eight rule sets to read, adapt and own.
|
|
4
|
+
|
|
5
|
+
← [Documentation](../index.md) · [Policy](index.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
A preset is not a black box. It is an array of [rules](rules.md) — the same ones you would
|
|
10
|
+
write — exported from `src/policy/presets.ts`, each carrying an `id` and a `reason`. Print
|
|
11
|
+
one, change one line, keep the rest:
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
import { BotHandler, protectContent } from "@osqd/bothandlerjs";
|
|
15
|
+
|
|
16
|
+
new BotHandler({
|
|
17
|
+
rules: protectContent().filter((rule) => rule.id !== "http-client-challenge"),
|
|
18
|
+
});
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Or name it and be done:
|
|
22
|
+
|
|
23
|
+
```ts
|
|
24
|
+
new BotHandler({ preset: "protect-content" });
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Every preset assumes the default `strict` [guard](../concepts/the-guard.md), which is why
|
|
28
|
+
even the harshest of them is safe to try: a rule asking to block cannot deny an unproven
|
|
29
|
+
request. **Choose a preset for the shape of the policy; the guard decides how far it is
|
|
30
|
+
allowed to go.**
|
|
31
|
+
|
|
32
|
+
| Preset | For | Terminal on |
|
|
33
|
+
| ------ | --- | ----------- |
|
|
34
|
+
| [`monitor-only`](#monitor-only) | week one, always | nothing |
|
|
35
|
+
| [`allow-crawlers`](#allow-crawlers) | publishers who want to be found | impersonators, scanners, traps |
|
|
36
|
+
| [`protect-content`](#protect-content) | a public content site | impersonators, scanners, traps |
|
|
37
|
+
| [`decline-ai-training`](#decline-ai-training) | keeping search, declining trainers | + declared AI crawlers |
|
|
38
|
+
| [`protect-data`](#protect-data) | pricing, listings, inventory | + declared AI and SEO crawlers |
|
|
39
|
+
| [`protect-api`](#protect-api) | a JSON API | impersonators, scanners, traps |
|
|
40
|
+
| [`protect-auth`](#protect-auth) | login, signup, checkout — **those routes only** | all proven automation |
|
|
41
|
+
| [`under-attack`](#under-attack) | during an incident, then off again | all proven automation |
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## `monitor-only`
|
|
46
|
+
|
|
47
|
+
Three rules, no action heavier than `log`. Nothing is ever withheld from anybody.
|
|
48
|
+
|
|
49
|
+
```ts
|
|
50
|
+
new BotHandler({ preset: "monitor-only" });
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
**Run this first, for at least a week, on real traffic.** Every bot policy that has caused
|
|
54
|
+
an outage was deployed straight to enforcement by someone who was sure they knew what
|
|
55
|
+
their traffic looked like. Watch the [dashboard](../operations/dashboard.md), find the
|
|
56
|
+
integration you forgot about, and then choose a real preset.
|
|
57
|
+
|
|
58
|
+
## `allow-crawlers`
|
|
59
|
+
|
|
60
|
+
Allows more than it stops, by name. Verified crawlers first; declared benign automation —
|
|
61
|
+
link unfurlers, feed readers, uptime monitors — allowed rather than merely tolerated; and
|
|
62
|
+
the last rule tags everything else, so nothing is withheld at all.
|
|
63
|
+
|
|
64
|
+
Reach for it when a bot policy has already cost you traffic, or when the site's whole
|
|
65
|
+
purpose is to be indexed, quoted and shared. The cost is honest: bulk extraction is
|
|
66
|
+
rate-limited rather than challenged, so a determined scraper gets your content. On a site
|
|
67
|
+
that wants to be read that was always true; what this refuses to do is trade away your
|
|
68
|
+
search traffic to make it slightly less true.
|
|
69
|
+
|
|
70
|
+
## `protect-content`
|
|
71
|
+
|
|
72
|
+
The sensible default for a public site. Keeps the crawlers that bring traffic, slows the
|
|
73
|
+
ones that only take it, challenges what is probably automated, blocks what has proven
|
|
74
|
+
itself.
|
|
75
|
+
|
|
76
|
+
Two rules are worth knowing before you copy it. `http-client-challenge` challenges bare
|
|
77
|
+
HTTP clients rather than blocking them — plenty of those are your own integrations.
|
|
78
|
+
`suspected-challenge` fires at `minScore: 70`, which is the number to move first if you
|
|
79
|
+
are seeing too much or too little.
|
|
80
|
+
|
|
81
|
+
## `decline-ai-training`
|
|
82
|
+
|
|
83
|
+
Keep the search engines. Decline the model trainers.
|
|
84
|
+
|
|
85
|
+
The split it draws is the one the AI crawlers publish themselves. A crawler collecting a
|
|
86
|
+
training corpus and a crawler fetching one page because a person asked about it are
|
|
87
|
+
different jobs, often from the same operator under different product tokens.
|
|
88
|
+
`ChatGPT-User`, `PerplexityBot`, `OAI-SearchBot`, `ClaudeBot`, `Mistral-AI` and
|
|
89
|
+
`DuckDuckBot` are served and tagged; the rest of the `ai` category is blocked with a body
|
|
90
|
+
that says so.
|
|
91
|
+
|
|
92
|
+
**`robots.txt` is the primary mechanism, not this.** The crawlers named here honour it, and
|
|
93
|
+
a rule that blocks a crawler nobody told is load with no compliance. Generate the file from
|
|
94
|
+
the policy and publish it — see [robots.txt](robots.md):
|
|
95
|
+
|
|
96
|
+
```ts
|
|
97
|
+
import { declineAiTraining, robotsFromRules } from "@osqd/bothandlerjs";
|
|
98
|
+
const { robotsTxt } = robotsFromRules(declineAiTraining(), { sitemap: "https://example.com/sitemap.xml" });
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## `protect-data`
|
|
102
|
+
|
|
103
|
+
For an application whose value is in its data. Like `protect-content`, plus: AI and SEO
|
|
104
|
+
crawlers are blocked on declaration, all proven automation is challenged, and
|
|
105
|
+
`/api/` and `/search` carry a 120/minute ceiling that applies to everyone equally.
|
|
106
|
+
|
|
107
|
+
The `any-proven-automation-challenge` rule matches on **verdicts** rather than bare
|
|
108
|
+
`{ certain: true }` — which would also match a proven human, and challenging a customer you
|
|
109
|
+
just vouched for is worse than useless. That subtlety is worth carrying into your own
|
|
110
|
+
rules.
|
|
111
|
+
|
|
112
|
+
## `protect-api`
|
|
113
|
+
|
|
114
|
+
One difference drives the whole shape: **a challenge is useless here.** A proof-of-work
|
|
115
|
+
interstitial is solved by a browser running JavaScript, and an API client is not one.
|
|
116
|
+
Challenging your customers' integrations does not slow an attacker down; it breaks the
|
|
117
|
+
integrations and leaves the attacker to solve it once in a headless browser. So the
|
|
118
|
+
escalation ladder is rate limiting.
|
|
119
|
+
|
|
120
|
+
The second difference is about what detection is *for* on an API. Your authentication is
|
|
121
|
+
the control that matters and already knows who the caller is. This preset tags everything
|
|
122
|
+
so your handlers can combine a verdict with a key, a plan and a quota; it does not try to
|
|
123
|
+
be the access control. A bare HTTP client is the normal case here, so the rule that
|
|
124
|
+
challenges one on a content site is deliberately absent.
|
|
125
|
+
|
|
126
|
+
## `protect-auth`
|
|
127
|
+
|
|
128
|
+
**Mount this on those routes only.**
|
|
129
|
+
|
|
130
|
+
```ts
|
|
131
|
+
app.use("/login", botHandler(authDetector));
|
|
132
|
+
app.use("/checkout", botHandler(authDetector));
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Applied site-wide it blocks your payment webhooks, your own server-side renderer and every
|
|
136
|
+
honest crawler you have — all proven automation, which is exactly what this refuses. That
|
|
137
|
+
is correct on a login form and an outage anywhere else. The [corpus](../testing/corpus.md)
|
|
138
|
+
catches it; the symptom otherwise is a support ticket about missing orders three days
|
|
139
|
+
later.
|
|
140
|
+
|
|
141
|
+
The unusual choice is `delay` on merely-suspected traffic: 250 ms is imperceptible to a
|
|
142
|
+
person filling in a form and ruinous to a credential stuffer working through a list — and,
|
|
143
|
+
unlike a challenge, it excludes nobody.
|
|
144
|
+
|
|
145
|
+
## `under-attack`
|
|
146
|
+
|
|
147
|
+
A deliberately impatient posture for an incident. Everything proven is refused, suspicion
|
|
148
|
+
is challenged at `minScore: 40`, and *everyone* — people included — is held to 30 requests
|
|
149
|
+
a minute, because a uniform ceiling is the one mitigation that cannot single anybody out.
|
|
150
|
+
|
|
151
|
+
Three things to be clear about first.
|
|
152
|
+
|
|
153
|
+
**It is temporary.** The low threshold will interrupt real people on unusual browsers.
|
|
154
|
+
Put it behind a switch you can flip without a deploy — `updatePolicy()` and the
|
|
155
|
+
[dashboard's editor](../operations/runtime-changes.md) exist for this.
|
|
156
|
+
|
|
157
|
+
**It still cannot deny anyone on a guess.** The guard applies here as everywhere. An
|
|
158
|
+
incident is precisely when people reach for `falsePositivePolicy: "aggressive"`, and
|
|
159
|
+
precisely when the population getting caught is at its most unusual.
|
|
160
|
+
|
|
161
|
+
**It is not DDoS protection.** This runs in your process, after the connection is
|
|
162
|
+
accepted. Volume that hurts at the network layer needs handling at the network layer;
|
|
163
|
+
what this reduces is the *usefulness* of the traffic to whoever is sending it.
|
|
164
|
+
|
|
165
|
+
And the practical warning the corpus makes concrete: like `protect-auth`, this refuses
|
|
166
|
+
proven automation, so your own webhooks, health probes and renderer are refused too —
|
|
167
|
+
thirteen of the corpus's infrastructure cases are. Allowlist their addresses *before* you
|
|
168
|
+
switch it on, not during the incident when you notice.
|
|
169
|
+
|
|
170
|
+
## Choosing between them
|
|
171
|
+
|
|
172
|
+
By name, for config-driven setups:
|
|
173
|
+
|
|
174
|
+
```ts
|
|
175
|
+
import { PRESETS, type PresetName } from "@osqd/bothandlerjs";
|
|
176
|
+
const rules = PRESETS[process.env.BOT_PRESET as PresetName]();
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
Whichever you pick, run it against the corpus before it reaches production — the run tells
|
|
180
|
+
you which of your own traffic the policy refuses:
|
|
181
|
+
|
|
182
|
+
```bash
|
|
183
|
+
npx @osqd/bothandlerjs check --preset protect-auth
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
## Related
|
|
187
|
+
|
|
188
|
+
- [Rules](rules.md) — the grammar these are written in
|
|
189
|
+
- [Actions](actions.md) — what each rule can ask for
|
|
190
|
+
- [Choosing a policy](../start/choosing-a-policy.md) — the decision, start to finish
|
|
191
|
+
- [The corpus](../testing/corpus.md) — 526 cases, including the ones these presets get wrong
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# robots.txt
|
|
2
|
+
|
|
3
|
+
Saying out loud what your policy already does.
|
|
4
|
+
|
|
5
|
+
← [Documentation](../index.md) · [Policy](index.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Why this is here
|
|
10
|
+
|
|
11
|
+
Declining a crawler and not saying so is the worst of both worlds: it keeps coming, wastes
|
|
12
|
+
your bandwidth rediscovering that it is unwelcome on every request, and you get no credit
|
|
13
|
+
for having a policy. `robots.txt` is where you say it — and for the well-behaved crawlers,
|
|
14
|
+
saying it is the *only* thing you need to do, because they will simply stop.
|
|
15
|
+
|
|
16
|
+
Two things this will not pretend. `robots.txt` is a request, not enforcement; everything
|
|
17
|
+
that ignores it is exactly the population this library exists for. And a generated file can
|
|
18
|
+
only reflect rules it can **read**.
|
|
19
|
+
|
|
20
|
+
References: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) is the standard;
|
|
21
|
+
[robotstxt.org](https://www.robotstxt.org/) is the older convention it formalises.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## From your policy
|
|
26
|
+
|
|
27
|
+
The useful entry point. It reads your rules and writes the file that matches them:
|
|
28
|
+
|
|
29
|
+
```ts
|
|
30
|
+
import { declineAiTraining, robotsFromRules } from "@osqd/bothandlerjs";
|
|
31
|
+
|
|
32
|
+
const { robotsTxt, declined, served, unreadable } = robotsFromRules(declineAiTraining(), {
|
|
33
|
+
disallowPaths: ["/internal/", "/admin-console"], // your trap paths belong here
|
|
34
|
+
sitemap: "https://example.com/sitemap.xml",
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
app.get("/robots.txt", (_req, res) => res.type("text/plain").send(robotsTxt));
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
It reads every rule whose action [denies service](actions.md) and whose `match` names an
|
|
41
|
+
`identity` or a `category`, and declines exactly those crawlers.
|
|
42
|
+
|
|
43
|
+
### The three fields you must look at
|
|
44
|
+
|
|
45
|
+
**`declined`** — the signature ids in the file, with categories expanded to the crawlers
|
|
46
|
+
they cover. Reporting only the explicitly-named ids would say "0 declined" for a policy
|
|
47
|
+
that turns away an entire category.
|
|
48
|
+
|
|
49
|
+
**`served`** — crawlers a later rule would have declined, but an earlier rule serves.
|
|
50
|
+
Policies are first-match-wins, so these are correctly *absent* from the file.
|
|
51
|
+
`decline-ai-training` is the case that found this: it serves `ChatGPT-User` and blocks the
|
|
52
|
+
rest of the `ai` category, and the generated file was telling `ChatGPT-User` to go away.
|
|
53
|
+
"Why is GPTBot in my robots.txt but ChatGPT-User is not" has an answer, and it is your own
|
|
54
|
+
rule order.
|
|
55
|
+
|
|
56
|
+
**`unreadable`** — rules that could not be read, with the reason. Two kinds:
|
|
57
|
+
|
|
58
|
+
- *the match is a predicate function* — it can be run, but not asked which crawlers it is
|
|
59
|
+
about.
|
|
60
|
+
- *the match is scoped to a path* — the rule denies service **there**, but a named group in
|
|
61
|
+
`robots.txt` gets `Disallow: /`, the whole site. Reported rather than passed over,
|
|
62
|
+
because the error runs in the expensive direction: a crawler told to stay away entirely
|
|
63
|
+
stops fetching the pages you wanted indexed, and that surfaces weeks later as a ranking
|
|
64
|
+
drop with nothing in the logs pointing at this file.
|
|
65
|
+
|
|
66
|
+
Check `unreadable` before publishing and add anything it names by hand. A `robots.txt` that
|
|
67
|
+
silently omits something you block is worse than no generated file at all — it tells
|
|
68
|
+
crawlers they are welcome where they are not.
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## By hand
|
|
73
|
+
|
|
74
|
+
When you want a file that is not derived from a policy:
|
|
75
|
+
|
|
76
|
+
```ts
|
|
77
|
+
import { generateRobotsTxt } from "@osqd/bothandlerjs";
|
|
78
|
+
|
|
79
|
+
generateRobotsTxt({
|
|
80
|
+
header: ["# Automated collection is declined. Contact abuse@example.com."],
|
|
81
|
+
disallowCategories: ["ai"],
|
|
82
|
+
disallowBots: ["semrushbot", "ahrefsbot"],
|
|
83
|
+
disallowPaths: ["/internal/", "/cart"],
|
|
84
|
+
allowPaths: ["/blog/"],
|
|
85
|
+
crawlDelay: 5,
|
|
86
|
+
sitemap: ["https://example.com/sitemap.xml"],
|
|
87
|
+
});
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
| Option | Effect |
|
|
91
|
+
| ------ | ------ |
|
|
92
|
+
| `disallowBots` | signature ids to decline entirely |
|
|
93
|
+
| `disallowCategories` | whole [categories](../detection/signatures.md) — `ai`, `seo`, … |
|
|
94
|
+
| `disallowPaths` | disallowed for every crawler; put your [trap](../detection/detectors.md) paths here |
|
|
95
|
+
| `allowPaths` | rendered ahead of the disallows |
|
|
96
|
+
| `sitemap` | absolute URLs, per the specification |
|
|
97
|
+
| `crawlDelay` | seconds, wildcard group only; not honoured by every crawler |
|
|
98
|
+
| `header` | verbatim lines at the top, already comment-prefixed |
|
|
99
|
+
| `signatures` | the signature database to resolve ids against |
|
|
100
|
+
|
|
101
|
+
### The grouping rule this gets right for you
|
|
102
|
+
|
|
103
|
+
Under RFC 9309 a crawler obeys the most specific group that names it and **ignores every
|
|
104
|
+
other group**. So a named `User-agent: GPTBot` group replaces the wildcard group outright
|
|
105
|
+
for GPTBot — including your `Disallow: /internal/` lines.
|
|
106
|
+
|
|
107
|
+
The renderer repeats global path rules inside every named group. Forgetting that is the
|
|
108
|
+
classic way a `robots.txt` accidentally *un*-blocks a trap path for exactly the crawlers
|
|
109
|
+
you were most careful about.
|
|
110
|
+
|
|
111
|
+
The wildcard group is emitted first purely as a courtesy to the person reading the file;
|
|
112
|
+
crawlers pick their group by specificity, not by position.
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## From the command line
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
npx @osqd/bothandlerjs robots --preset decline-ai-training --sitemap https://example.com/sitemap.xml
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
See [the CLI](../testing/cli.md).
|
|
123
|
+
|
|
124
|
+
## Related
|
|
125
|
+
|
|
126
|
+
- [Presets](presets.md) — `decline-ai-training`, which this pairs with
|
|
127
|
+
- [Signatures](../detection/signatures.md) — the ids and categories being named
|
|
128
|
+
- [Detectors](../detection/detectors.md) — the trap paths worth disallowing here
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# Matching requests
|
|
2
|
+
|
|
3
|
+
Every field a rule can match on, and what each one is good for.
|
|
4
|
+
|
|
5
|
+
← [Documentation](../index.md) · [Policy](index.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
interface Rule {
|
|
11
|
+
id: string; // stable; names every decision and log line it produces
|
|
12
|
+
match: MatchSpec | ((assessment: Assessment) => boolean);
|
|
13
|
+
action: ActionName;
|
|
14
|
+
params?: ActionParams;
|
|
15
|
+
reason?: string;
|
|
16
|
+
}
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Every field in a `MatchSpec` must match — it is an `AND`. Fields accepting a list match if
|
|
20
|
+
*any* entry matches. An empty `match: {}` matches everything, which is occasionally what
|
|
21
|
+
you want at the bottom of a list.
|
|
22
|
+
|
|
23
|
+
## What this request is
|
|
24
|
+
|
|
25
|
+
| Field | Type | |
|
|
26
|
+
| ----- | ---- | --- |
|
|
27
|
+
| `verdict` | one or more [`Verdict`](../concepts/verdicts.md) | `"suspected-bot"`, `["confirmed-bot", "verified-bot"]` |
|
|
28
|
+
| `botClass` | one or more `BotClass` | The field most policies want — it carries intent. |
|
|
29
|
+
| `certain` | `boolean` | `true` matches only proven verdicts, including a proven human. |
|
|
30
|
+
| `minScore` / `maxScore` | `number` | Inclusive. Only meaningful for probabilistic verdicts. |
|
|
31
|
+
|
|
32
|
+
```ts
|
|
33
|
+
// Proven automation of any kind
|
|
34
|
+
{ match: { certain: true, verdict: ["confirmed-bot", "verified-bot"] } }
|
|
35
|
+
|
|
36
|
+
// A judgement, but a confident one
|
|
37
|
+
{ match: { verdict: "suspected-bot", minScore: 70 } }
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Who this claims to be
|
|
41
|
+
|
|
42
|
+
| Field | Type | |
|
|
43
|
+
| ----- | ---- | --- |
|
|
44
|
+
| `identity` | one or more `string` | A [signature](../detection/signatures.md) id: `"googlebot"`, `"gptbot"`. |
|
|
45
|
+
| `category` | one or more `BotCategory` | `"ai"`, `"search"`, `"seo"`, `"scanner"`, … |
|
|
46
|
+
|
|
47
|
+
```ts
|
|
48
|
+
{ id: "no-training", match: { category: "ai" }, action: "block" }
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
**An identity match alone matches forgeries too**, because the identity is what the client
|
|
52
|
+
*claimed*. Pair it with `verdict: "verified-bot"` when trust is the point:
|
|
53
|
+
|
|
54
|
+
```ts
|
|
55
|
+
{ id: "trust-google", match: { identity: ["googlebot"], verdict: "verified-bot" }, action: "allow" }
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## What it asked for
|
|
59
|
+
|
|
60
|
+
| Field | Type | |
|
|
61
|
+
| ----- | ---- | --- |
|
|
62
|
+
| `path` | `string \| RegExp` or a list | A string matches as a **prefix**; a regex is tested as written. |
|
|
63
|
+
| `method` | one or more `string` | Upper-case. |
|
|
64
|
+
|
|
65
|
+
```ts
|
|
66
|
+
{ id: "protect-export", match: { path: ["/api/export", "/reports/"], verdict: "suspected-bot" },
|
|
67
|
+
action: "challenge" }
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## What fired
|
|
71
|
+
|
|
72
|
+
| Field | Type | |
|
|
73
|
+
| ----- | ---- | --- |
|
|
74
|
+
| `detector` | one or more `string` | Matches if **any** evidence came from one of these. |
|
|
75
|
+
|
|
76
|
+
```ts
|
|
77
|
+
// Traps are proof. Nothing else needs to be true.
|
|
78
|
+
{ id: "trapped", match: { detector: ["trap"], certain: true }, action: "block" }
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## What this actor has done before
|
|
82
|
+
|
|
83
|
+
| Field | Type | |
|
|
84
|
+
| ----- | ---- | --- |
|
|
85
|
+
| `minPriorConfirmations` | `number` | Times this actor was **proven** a bot before now. |
|
|
86
|
+
| `minUnsolvedChallenges` | `number` | Challenges issued that no solution came back for. |
|
|
87
|
+
|
|
88
|
+
```ts
|
|
89
|
+
{ id: "repeat-offender", match: { minPriorConfirmations: 3 }, action: "block",
|
|
90
|
+
reason: "Proven automation three times from this actor." }
|
|
91
|
+
|
|
92
|
+
{ id: "persistent-refusers", match: { minUnsolvedChallenges: 3 }, action: "rate-limit" }
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
`minUnsolvedChallenges` is **outstanding rather than cumulative**: solving one clears the
|
|
96
|
+
count, so it never accumulates against somebody who came back and proved it. It is a rule
|
|
97
|
+
rather than evidence on purpose — one abandoned challenge is a person having a moment, and
|
|
98
|
+
what repetition means depends on traffic the library cannot see. See
|
|
99
|
+
[actors](../concepts/actors.md).
|
|
100
|
+
|
|
101
|
+
## A predicate, for anything else
|
|
102
|
+
|
|
103
|
+
```ts
|
|
104
|
+
{
|
|
105
|
+
id: "checkout-under-attack",
|
|
106
|
+
match: (assessment) =>
|
|
107
|
+
assessment.facts.path.startsWith("/checkout") &&
|
|
108
|
+
assessment.score > 50 &&
|
|
109
|
+
assessment.actor.requests > 20,
|
|
110
|
+
action: "challenge",
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Two things to know. A predicate that **throws** is skipped rather than matched — "unknown"
|
|
115
|
+
cannot safely mean "yes". And a predicate cannot be serialised, so a rule using one is
|
|
116
|
+
shown read-only in the dashboard's editor and spliced back at its original index on save:
|
|
117
|
+
order is the whole semantics, so "wherever it ends up" is not an option.
|
|
118
|
+
|
|
119
|
+
## Validation
|
|
120
|
+
|
|
121
|
+
`validateRules` runs at construction and reports what is survivable rather than throwing:
|
|
122
|
+
duplicate ids, a rule shadowed by a broader one above it, an action whose parameters are
|
|
123
|
+
missing. Warnings go to `onWarning` and to the dashboard's notices.
|
|
124
|
+
|
|
125
|
+
```ts
|
|
126
|
+
import { validateRules } from "@osqd/bothandlerjs";
|
|
127
|
+
validateRules(myRules); // string[] — empty is good
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
## Related
|
|
131
|
+
|
|
132
|
+
- [Actions](actions.md) — what a matched rule can ask for
|
|
133
|
+
- [The safety guard](../concepts/the-guard.md) — what it is allowed to get
|
|
134
|
+
- [Verdicts, classes and scores](../concepts/verdicts.md) — the vocabularies above
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
# API
|
|
2
|
+
|
|
3
|
+
The methods on `BotHandler`, the entry points, and where each exported name is documented.
|
|
4
|
+
|
|
5
|
+
← [Documentation](../index.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Entry points
|
|
10
|
+
|
|
11
|
+
| Import | Contains |
|
|
12
|
+
| ------ | -------- |
|
|
13
|
+
| `@osqd/bothandlerjs` | the engine, detectors, presets, robots, stores, notifiers, challenge, dashboard |
|
|
14
|
+
| `@osqd/bothandlerjs/adapters` | [Express, Fastify, Koa, Fetch](../integration/adapters.md) |
|
|
15
|
+
| `@osqd/bothandlerjs/client` | the browser-side [client signals](../detection/client-signals.md) script |
|
|
16
|
+
| `@osqd/bothandlerjs/corpus` | [`runCorpus`](../testing/corpus.md) and the cases |
|
|
17
|
+
| `@osqd/bothandlerjs/cli` | the [command line](../testing/cli.md) entry point |
|
|
18
|
+
|
|
19
|
+
Zero runtime dependencies, ESM and CJS, types included.
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## The request path
|
|
24
|
+
|
|
25
|
+
```ts
|
|
26
|
+
const assessment = await detector.assess(facts, { record: false });
|
|
27
|
+
const decision = detector.decide(assessment);
|
|
28
|
+
const result = await detector.handle(facts); // both, plus the action applied
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
| Method | |
|
|
32
|
+
| ------ | - |
|
|
33
|
+
| `assess(facts, options?)` | runs detection. `{ record: false }` is a [dry run](../detection/index.md) |
|
|
34
|
+
| `decide(assessment)` | applies the [policy](../policy/index.md) and [the guard](../concepts/the-guard.md). Synchronous and pure |
|
|
35
|
+
| `handle(facts)` | assess + decide + produce an `ActionOutcome` |
|
|
36
|
+
| `createFacts(input)` | builds `RequestFacts` from method, url, headers, `rawHeaders`, ip |
|
|
37
|
+
|
|
38
|
+
`assess` and `decide` are separate on purpose: *what is this client* and *what should we do
|
|
39
|
+
about it* are different questions with different lifetimes. See [policy](../policy/index.md).
|
|
40
|
+
|
|
41
|
+
## The challenge
|
|
42
|
+
|
|
43
|
+
| Method | |
|
|
44
|
+
| ------ | - |
|
|
45
|
+
| `isChallengeEndpoint(facts)` | is this the verification POST? Adapters call it for you |
|
|
46
|
+
| `verifyChallenge(facts, body)` | verify a solution, grant clearance |
|
|
47
|
+
| `grantClearance(facts, level?)` | issue clearance directly — `"operator"` by default |
|
|
48
|
+
|
|
49
|
+
`grantClearance(facts, "operator")` is how your application vouches for an authenticated
|
|
50
|
+
person. See [the challenge](../challenge/index.md).
|
|
51
|
+
|
|
52
|
+
## Observation
|
|
53
|
+
|
|
54
|
+
| Method | |
|
|
55
|
+
| ------ | - |
|
|
56
|
+
| `on(event, handler)` | subscribe; returns an unsubscribe function |
|
|
57
|
+
| `metrics()` | structured snapshot |
|
|
58
|
+
| `prometheus(options?)` | text exposition format |
|
|
59
|
+
| `runAudit()` | run the [audit](../operations/audit.md) checks now |
|
|
60
|
+
| `audit` | the `TrafficAudit` object, with `summary()` |
|
|
61
|
+
| `serveDashboard(options?)` | start [the dashboard](../operations/dashboard.md) on its own port |
|
|
62
|
+
| `describeDetectors()` | id, description, cost and stage for each installed detector |
|
|
63
|
+
|
|
64
|
+
## Runtime changes
|
|
65
|
+
|
|
66
|
+
Each takes a trailing `{ by }` — see [runtime changes](../operations/runtime-changes.md).
|
|
67
|
+
|
|
68
|
+
| Method | |
|
|
69
|
+
| ------ | - |
|
|
70
|
+
| `updatePolicy(rules, ctx?)` | validated, atomic; cannot touch the guard |
|
|
71
|
+
| `updateGuard(settings, ctx?)` | its own method and its own event, deliberately |
|
|
72
|
+
| `updateRanges(name, entries, ctx?)` | allowlist, denylist, datacenter |
|
|
73
|
+
| `updateCrawlerRanges(id, entries, ctx?)` | one crawler's published ranges |
|
|
74
|
+
| `listRanges()` / `rangeEntries(name)` | what is loaded |
|
|
75
|
+
| `forgetActor(key, ctx?)` | drop one [actor's](../concepts/actors.md) memory |
|
|
76
|
+
| `clearActor(key, forMs, ctx?)` | treat an actor as cleared for a while |
|
|
77
|
+
|
|
78
|
+
## Inspection
|
|
79
|
+
|
|
80
|
+
`resolveIp`, `actorKeyFor`, `isAllowlisted`, `isIgnoredPath`, `policy`, `warn`.
|
|
81
|
+
|
|
82
|
+
`warn(message)` is public so warnings raised *outside* the engine — the fetch adapter, the
|
|
83
|
+
crawler-range refresher — reach the same `warning` event as everything else, rather than
|
|
84
|
+
only the `onWarning` callback.
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
## Where the exported names are documented
|
|
89
|
+
|
|
90
|
+
| Names | Page |
|
|
91
|
+
| ----- | ---- |
|
|
92
|
+
| `BotHandler`, `createFacts`, `resolveConfig`, `validateRules`, `ConfigError` | [Configuration](configuration.md) |
|
|
93
|
+
| `CERTAINTY_WEIGHT`, `combineEvidence`, `noisyOr`, `sortEvidence`, `weightOf` | [Evidence](../concepts/evidence.md) |
|
|
94
|
+
| `VERDICTS`, `BOT_CLASSES` | [Verdicts](../concepts/verdicts.md) |
|
|
95
|
+
| `defaultDetectors`, every `*Detector` factory, `Detector`, `Evidence` | [Detectors](../detection/detectors.md), [writing one](../detection/writing-a-detector.md) |
|
|
96
|
+
| `BOT_SIGNATURES`, `BotSignature`, `BotCategory` | [Signatures](../detection/signatures.md) |
|
|
97
|
+
| `forwardConfirmedReverseDns`, `cachingResolver`, `nodeDnsResolver`, `PUBLISHED_CRAWLER_RANGES`, `refreshCrawlerRanges`, `startCrawlerRangeRefresh` | [Verification](../detection/verification.md) |
|
|
98
|
+
| `PRESETS`, every preset function, `Rule`, `MatchSpec`, `Decision` | [Rules](../policy/rules.md), [presets](../policy/presets.md) |
|
|
99
|
+
| `defineHandler`, `CustomHandler`, `ActionOutcome`, `ACTION_NAMES`, `TERMINAL_ACTIONS` | [Actions](../policy/actions.md) |
|
|
100
|
+
| `generateRobotsTxt`, `robotsFromRules`, `agentFor` | [robots.txt](../policy/robots.md) |
|
|
101
|
+
| `ChallengeService`, `ChallengeOptions`, `ChallengeCopy`, `parseAcceptLanguage`, `pickTranslation` | [The challenge](../challenge/index.md), [localisation](../challenge/localisation.md) |
|
|
102
|
+
| `MemoryStore`, `RedisStore`, `BotHandlerStore` | [Stores](../integration/stores.md) |
|
|
103
|
+
| `consoleNotifier`, `webhookNotifier`, `slackNotifier`, `notifyJsNotifier`, `Notifier` | [Notifications](../operations/notifications.md) |
|
|
104
|
+
| `Metrics`, `toPrometheus`, `SCORE_BUCKETS`, `DURATION_BUCKETS_MS` | [Metrics](../operations/metrics.md) |
|
|
105
|
+
| `TrafficAudit`, `DEFAULT_CHECKS`, `AuditCheck`, `TrafficAnomaly` | [The audit](../operations/audit.md) |
|
|
106
|
+
| `startDashboard`, `createDashboardHandler`, `renderDashboardPage`, `Dashboard*` | [The dashboard](../operations/dashboard.md) |
|
|
107
|
+
| `ActorRegistry`, `ActorState`, `ActorSummary` | [Actors](../concepts/actors.md) |
|
|
108
|
+
| `renderTrapField`, `TRAP_FIELD_SOURCE` | [Detectors](../detection/detectors.md), [adapters](../integration/adapters.md) |
|
|
109
|
+
| `ClientSignals`, the client script | [Client signals](../detection/client-signals.md) |
|
|
110
|
+
|
|
111
|
+
### The utilities
|
|
112
|
+
|
|
113
|
+
Exported because they are useful on their own and because a detector you write will want
|
|
114
|
+
them: `IpRangeSet`, `parseIp`, `parseCidr`, `cidrContains`, `normalizeIp`, `formatIp`,
|
|
115
|
+
`isSpecialUse`, `networkKey`, `SPECIAL_USE_RANGES`; `parseUserAgent`, `claimsBrowser`,
|
|
116
|
+
`sendsModernHeaders`; `parseCookies`, `serializeCookie`; `MultiPatternMatcher`, `TtlLru`,
|
|
117
|
+
`Emitter`, `ManualClock`, `systemClock`.
|
|
118
|
+
|
|
119
|
+
`ManualClock` and a stub `DnsResolver` are what make the [corpus](../testing/corpus.md)
|
|
120
|
+
deterministic, and they will do the same for your tests.
|
|
121
|
+
|
|
122
|
+
## Related
|
|
123
|
+
|
|
124
|
+
- [Configuration](configuration.md) — every constructor option
|
|
125
|
+
- [Detection](../detection/index.md) — what `assess` actually runs
|
|
126
|
+
- [Policy](../policy/index.md) — what `decide` actually applies
|