@osqd/bothandlerjs 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (186) hide show
  1. package/CHANGELOG.md +516 -0
  2. package/LICENSE +102 -0
  3. package/README.md +308 -0
  4. package/bin/bothandlerjs.mjs +14 -0
  5. package/dist/actions/index.d.ts +22 -0
  6. package/dist/actions/types.d.ts +59 -0
  7. package/dist/adapters/fastify.d.ts +43 -0
  8. package/dist/adapters/fetch.d.ts +51 -0
  9. package/dist/adapters/index.cjs +723 -0
  10. package/dist/adapters/index.cjs.map +1 -0
  11. package/dist/adapters/index.d.ts +8 -0
  12. package/dist/adapters/index.js +692 -0
  13. package/dist/adapters/index.js.map +1 -0
  14. package/dist/adapters/koa.d.ts +27 -0
  15. package/dist/adapters/node.d.ts +29 -0
  16. package/dist/adapters/shared.d.ts +39 -0
  17. package/dist/audit.d.ts +172 -0
  18. package/dist/challenge/index.d.ts +176 -0
  19. package/dist/challenge/language.d.ts +57 -0
  20. package/dist/challenge/page.d.ts +42 -0
  21. package/dist/challenge/pow.d.ts +33 -0
  22. package/dist/challenge/token.d.ts +72 -0
  23. package/dist/cli.cjs +13388 -0
  24. package/dist/cli.cjs.map +1 -0
  25. package/dist/cli.d.ts +5 -0
  26. package/dist/cli.js +13358 -0
  27. package/dist/cli.js.map +1 -0
  28. package/dist/client/index.cjs +139 -0
  29. package/dist/client/index.cjs.map +1 -0
  30. package/dist/client/index.d.ts +63 -0
  31. package/dist/client/index.js +112 -0
  32. package/dist/client/index.js.map +1 -0
  33. package/dist/config.d.ts +316 -0
  34. package/dist/core.d.ts +443 -0
  35. package/dist/corpus/adversarial.d.ts +2 -0
  36. package/dist/corpus/advertising-email.d.ts +2 -0
  37. package/dist/corpus/ai-crawlers.d.ts +2 -0
  38. package/dist/corpus/benign-bots.d.ts +2 -0
  39. package/dist/corpus/cdn-gateways.d.ts +2 -0
  40. package/dist/corpus/cookies.d.ts +49 -0
  41. package/dist/corpus/crawlers-regional.d.ts +2 -0
  42. package/dist/corpus/crawlers-vertical.d.ts +2 -0
  43. package/dist/corpus/headers.d.ts +403 -0
  44. package/dist/corpus/humans-apps.d.ts +2 -0
  45. package/dist/corpus/humans-browsers.d.ts +2 -0
  46. package/dist/corpus/humans.d.ts +2 -0
  47. package/dist/corpus/index.cjs +4196 -0
  48. package/dist/corpus/index.cjs.map +1 -0
  49. package/dist/corpus/index.d.ts +37 -0
  50. package/dist/corpus/index.js +4132 -0
  51. package/dist/corpus/index.js.map +1 -0
  52. package/dist/corpus/infrastructure.d.ts +17 -0
  53. package/dist/corpus/libraries-extended.d.ts +2 -0
  54. package/dist/corpus/ranges.d.ts +31 -0
  55. package/dist/corpus/reputation.d.ts +21 -0
  56. package/dist/corpus/runner.d.ts +130 -0
  57. package/dist/corpus/schema.d.ts +185 -0
  58. package/dist/corpus/tooling.d.ts +2 -0
  59. package/dist/corpus/unwanted.d.ts +2 -0
  60. package/dist/crawler-ranges.d.ts +113 -0
  61. package/dist/dashboard/client.generated.d.ts +7 -0
  62. package/dist/dashboard/feed.d.ts +186 -0
  63. package/dist/dashboard/index.d.ts +5 -0
  64. package/dist/dashboard/page.d.ts +51 -0
  65. package/dist/dashboard/parse-request.d.ts +28 -0
  66. package/dist/dashboard/preview.d.ts +45 -0
  67. package/dist/dashboard/server.d.ts +70 -0
  68. package/dist/dashboard/types.d.ts +714 -0
  69. package/dist/detectors/accept-signature.d.ts +6 -0
  70. package/dist/detectors/browsing-coherence.d.ts +44 -0
  71. package/dist/detectors/cadence.d.ts +28 -0
  72. package/dist/detectors/clearance.d.ts +25 -0
  73. package/dist/detectors/client-hints.d.ts +19 -0
  74. package/dist/detectors/client-signals.d.ts +27 -0
  75. package/dist/detectors/crawl-breadth.d.ts +28 -0
  76. package/dist/detectors/crawler-verification.d.ts +39 -0
  77. package/dist/detectors/fetch-metadata.d.ts +20 -0
  78. package/dist/detectors/header-integrity.d.ts +26 -0
  79. package/dist/detectors/header-order.d.ts +27 -0
  80. package/dist/detectors/identity-rotation.d.ts +34 -0
  81. package/dist/detectors/index.d.ts +56 -0
  82. package/dist/detectors/ip-intelligence.d.ts +30 -0
  83. package/dist/detectors/known-bots.d.ts +94 -0
  84. package/dist/detectors/probe-signature.d.ts +50 -0
  85. package/dist/detectors/rate-anomaly.d.ts +36 -0
  86. package/dist/detectors/self-identified.d.ts +33 -0
  87. package/dist/detectors/session-integrity.d.ts +24 -0
  88. package/dist/detectors/tls-fingerprint.d.ts +34 -0
  89. package/dist/detectors/trap.d.ts +98 -0
  90. package/dist/detectors/types.d.ts +70 -0
  91. package/dist/detectors/ua-coherence.d.ts +32 -0
  92. package/dist/evidence.d.ts +45 -0
  93. package/dist/facts.d.ts +46 -0
  94. package/dist/index.cjs +9600 -0
  95. package/dist/index.cjs.map +1 -0
  96. package/dist/index.d.ts +63 -0
  97. package/dist/index.js +9444 -0
  98. package/dist/index.js.map +1 -0
  99. package/dist/internal/async.d.ts +16 -0
  100. package/dist/internal/clock.d.ts +17 -0
  101. package/dist/internal/crypto.d.ts +26 -0
  102. package/dist/internal/dns.d.ts +100 -0
  103. package/dist/internal/emitter.d.ts +18 -0
  104. package/dist/internal/http.d.ts +22 -0
  105. package/dist/internal/ip.d.ts +73 -0
  106. package/dist/internal/lru.d.ts +34 -0
  107. package/dist/internal/matcher.d.ts +41 -0
  108. package/dist/internal/pattern.d.ts +14 -0
  109. package/dist/internal/ua.d.ts +92 -0
  110. package/dist/metrics.d.ts +120 -0
  111. package/dist/notify/hub.d.ts +70 -0
  112. package/dist/notify/index.d.ts +7 -0
  113. package/dist/notify/redact.d.ts +48 -0
  114. package/dist/notify/sinks.d.ts +62 -0
  115. package/dist/notify/types.d.ts +56 -0
  116. package/dist/policy/index.d.ts +7 -0
  117. package/dist/policy/match.d.ts +20 -0
  118. package/dist/policy/policy.d.ts +123 -0
  119. package/dist/policy/presets.d.ts +161 -0
  120. package/dist/policy/types.d.ts +155 -0
  121. package/dist/robots.d.ts +83 -0
  122. package/dist/state.d.ts +161 -0
  123. package/dist/stores/index.d.ts +5 -0
  124. package/dist/stores/memory.d.ts +33 -0
  125. package/dist/stores/redis.d.ts +59 -0
  126. package/dist/stores/types.d.ts +37 -0
  127. package/dist/types.d.ts +250 -0
  128. package/docs/challenge/index.md +152 -0
  129. package/docs/challenge/localisation.md +105 -0
  130. package/docs/concepts/actors.md +113 -0
  131. package/docs/concepts/evidence.md +133 -0
  132. package/docs/concepts/the-guard.md +121 -0
  133. package/docs/concepts/threat-model.md +120 -0
  134. package/docs/concepts/verdicts.md +108 -0
  135. package/docs/course/01-first-assessment.md +178 -0
  136. package/docs/course/02-proof-and-suspicion.md +164 -0
  137. package/docs/course/03-verdicts-and-scores.md +160 -0
  138. package/docs/course/04-the-guard.md +200 -0
  139. package/docs/course/05-detectors.md +178 -0
  140. package/docs/course/06-identity.md +161 -0
  141. package/docs/course/07-actors.md +178 -0
  142. package/docs/course/08-traps.md +183 -0
  143. package/docs/course/09-rules.md +224 -0
  144. package/docs/course/10-actions-and-presets.md +216 -0
  145. package/docs/course/11-the-challenge.md +227 -0
  146. package/docs/course/12-going-live.md +211 -0
  147. package/docs/course/13-operating-it.md +238 -0
  148. package/docs/course/14-scaling.md +195 -0
  149. package/docs/course/15-extending.md +259 -0
  150. package/docs/course/16-proving-it.md +266 -0
  151. package/docs/course/index.md +112 -0
  152. package/docs/design/decisions.md +216 -0
  153. package/docs/detection/client-signals.md +96 -0
  154. package/docs/detection/detectors.md +376 -0
  155. package/docs/detection/index.md +134 -0
  156. package/docs/detection/signatures.md +111 -0
  157. package/docs/detection/verification.md +123 -0
  158. package/docs/detection/writing-a-detector.md +145 -0
  159. package/docs/index.md +100 -0
  160. package/docs/integration/adapters.md +138 -0
  161. package/docs/integration/client-ip.md +114 -0
  162. package/docs/integration/index.md +55 -0
  163. package/docs/integration/stores.md +76 -0
  164. package/docs/operations/audit.md +116 -0
  165. package/docs/operations/dashboard.md +725 -0
  166. package/docs/operations/index.md +101 -0
  167. package/docs/operations/metrics.md +74 -0
  168. package/docs/operations/notifications.md +72 -0
  169. package/docs/operations/runtime-changes.md +118 -0
  170. package/docs/policy/actions.md +196 -0
  171. package/docs/policy/index.md +84 -0
  172. package/docs/policy/presets.md +191 -0
  173. package/docs/policy/robots.md +128 -0
  174. package/docs/policy/rules.md +134 -0
  175. package/docs/reference/api.md +126 -0
  176. package/docs/reference/configuration.md +175 -0
  177. package/docs/start/choosing-a-policy.md +127 -0
  178. package/docs/start/first-integration.md +127 -0
  179. package/docs/start/installation.md +63 -0
  180. package/docs/start/upgrading.md +72 -0
  181. package/docs/testing/cli.md +119 -0
  182. package/docs/testing/corpus.md +129 -0
  183. package/docs/testing/index.md +61 -0
  184. package/docs/testing/replay.md +102 -0
  185. package/docs/testing/try-it.md +189 -0
  186. package/package.json +145 -0
@@ -0,0 +1,191 @@
1
+ # Presets
2
+
3
+ Eight rule sets to read, adapt and own.
4
+
5
+ ← [Documentation](../index.md) · [Policy](index.md)
6
+
7
+ ---
8
+
9
+ A preset is not a black box. It is an array of [rules](rules.md) — the same ones you would
10
+ write — exported from `src/policy/presets.ts`, each carrying an `id` and a `reason`. Print
11
+ one, change one line, keep the rest:
12
+
13
+ ```ts
14
+ import { BotHandler, protectContent } from "@osqd/bothandlerjs";
15
+
16
+ new BotHandler({
17
+ rules: protectContent().filter((rule) => rule.id !== "http-client-challenge"),
18
+ });
19
+ ```
20
+
21
+ Or name it and be done:
22
+
23
+ ```ts
24
+ new BotHandler({ preset: "protect-content" });
25
+ ```
26
+
27
+ Every preset assumes the default `strict` [guard](../concepts/the-guard.md), which is why
28
+ even the harshest of them is safe to try: a rule asking to block cannot deny an unproven
29
+ request. **Choose a preset for the shape of the policy; the guard decides how far it is
30
+ allowed to go.**
31
+
32
+ | Preset | For | Terminal on |
33
+ | ------ | --- | ----------- |
34
+ | [`monitor-only`](#monitor-only) | week one, always | nothing |
35
+ | [`allow-crawlers`](#allow-crawlers) | publishers who want to be found | impersonators, scanners, traps |
36
+ | [`protect-content`](#protect-content) | a public content site | impersonators, scanners, traps |
37
+ | [`decline-ai-training`](#decline-ai-training) | keeping search, declining trainers | + declared AI crawlers |
38
+ | [`protect-data`](#protect-data) | pricing, listings, inventory | + declared AI and SEO crawlers |
39
+ | [`protect-api`](#protect-api) | a JSON API | impersonators, scanners, traps |
40
+ | [`protect-auth`](#protect-auth) | login, signup, checkout — **those routes only** | all proven automation |
41
+ | [`under-attack`](#under-attack) | during an incident, then off again | all proven automation |
42
+
43
+ ---
44
+
45
+ ## `monitor-only`
46
+
47
+ Three rules, no action heavier than `log`. Nothing is ever withheld from anybody.
48
+
49
+ ```ts
50
+ new BotHandler({ preset: "monitor-only" });
51
+ ```
52
+
53
+ **Run this first, for at least a week, on real traffic.** Every bot policy that has caused
54
+ an outage was deployed straight to enforcement by someone who was sure they knew what
55
+ their traffic looked like. Watch the [dashboard](../operations/dashboard.md), find the
56
+ integration you forgot about, and then choose a real preset.
57
+
58
+ ## `allow-crawlers`
59
+
60
+ Allows more than it stops, by name. Verified crawlers first; declared benign automation —
61
+ link unfurlers, feed readers, uptime monitors — allowed rather than merely tolerated; and
62
+ the last rule tags everything else, so nothing is withheld at all.
63
+
64
+ Reach for it when a bot policy has already cost you traffic, or when the site's whole
65
+ purpose is to be indexed, quoted and shared. The cost is honest: bulk extraction is
66
+ rate-limited rather than challenged, so a determined scraper gets your content. On a site
67
+ that wants to be read that was always true; what this refuses to do is trade away your
68
+ search traffic to make it slightly less true.
69
+
70
+ ## `protect-content`
71
+
72
+ The sensible default for a public site. Keeps the crawlers that bring traffic, slows the
73
+ ones that only take it, challenges what is probably automated, blocks what has proven
74
+ itself.
75
+
76
+ Two rules are worth knowing before you copy it. `http-client-challenge` challenges bare
77
+ HTTP clients rather than blocking them — plenty of those are your own integrations.
78
+ `suspected-challenge` fires at `minScore: 70`, which is the number to move first if you
79
+ are seeing too much or too little.
80
+
81
+ ## `decline-ai-training`
82
+
83
+ Keep the search engines. Decline the model trainers.
84
+
85
+ The split it draws is the one the AI crawlers publish themselves. A crawler collecting a
86
+ training corpus and a crawler fetching one page because a person asked about it are
87
+ different jobs, often from the same operator under different product tokens.
88
+ `ChatGPT-User`, `PerplexityBot`, `OAI-SearchBot`, `ClaudeBot`, `Mistral-AI` and
89
+ `DuckDuckBot` are served and tagged; the rest of the `ai` category is blocked with a body
90
+ that says so.
91
+
92
+ **`robots.txt` is the primary mechanism, not this.** The crawlers named here honour it, and
93
+ a rule that blocks a crawler nobody told is load with no compliance. Generate the file from
94
+ the policy and publish it — see [robots.txt](robots.md):
95
+
96
+ ```ts
97
+ import { declineAiTraining, robotsFromRules } from "@osqd/bothandlerjs";
98
+ const { robotsTxt } = robotsFromRules(declineAiTraining(), { sitemap: "https://example.com/sitemap.xml" });
99
+ ```
100
+
101
+ ## `protect-data`
102
+
103
+ For an application whose value is in its data. Like `protect-content`, plus: AI and SEO
104
+ crawlers are blocked on declaration, all proven automation is challenged, and
105
+ `/api/` and `/search` carry a 120/minute ceiling that applies to everyone equally.
106
+
107
+ The `any-proven-automation-challenge` rule matches on **verdicts** rather than bare
108
+ `{ certain: true }` — which would also match a proven human, and challenging a customer you
109
+ just vouched for is worse than useless. That subtlety is worth carrying into your own
110
+ rules.
111
+
112
+ ## `protect-api`
113
+
114
+ One difference drives the whole shape: **a challenge is useless here.** A proof-of-work
115
+ interstitial is solved by a browser running JavaScript, and an API client is not one.
116
+ Challenging your customers' integrations does not slow an attacker down; it breaks the
117
+ integrations and leaves the attacker to solve it once in a headless browser. So the
118
+ escalation ladder is rate limiting.
119
+
120
+ The second difference is about what detection is *for* on an API. Your authentication is
121
+ the control that matters and already knows who the caller is. This preset tags everything
122
+ so your handlers can combine a verdict with a key, a plan and a quota; it does not try to
123
+ be the access control. A bare HTTP client is the normal case here, so the rule that
124
+ challenges one on a content site is deliberately absent.
125
+
126
+ ## `protect-auth`
127
+
128
+ **Mount this on those routes only.**
129
+
130
+ ```ts
131
+ app.use("/login", botHandler(authDetector));
132
+ app.use("/checkout", botHandler(authDetector));
133
+ ```
134
+
135
+ Applied site-wide it blocks your payment webhooks, your own server-side renderer and every
136
+ honest crawler you have — all proven automation, which is exactly what this refuses. That
137
+ is correct on a login form and an outage anywhere else. The [corpus](../testing/corpus.md)
138
+ catches it; the symptom otherwise is a support ticket about missing orders three days
139
+ later.
140
+
141
+ The unusual choice is `delay` on merely-suspected traffic: 250 ms is imperceptible to a
142
+ person filling in a form and ruinous to a credential stuffer working through a list — and,
143
+ unlike a challenge, it excludes nobody.
144
+
145
+ ## `under-attack`
146
+
147
+ A deliberately impatient posture for an incident. Everything proven is refused, suspicion
148
+ is challenged at `minScore: 40`, and *everyone* — people included — is held to 30 requests
149
+ a minute, because a uniform ceiling is the one mitigation that cannot single anybody out.
150
+
151
+ Three things to be clear about first.
152
+
153
+ **It is temporary.** The low threshold will interrupt real people on unusual browsers.
154
+ Put it behind a switch you can flip without a deploy — `updatePolicy()` and the
155
+ [dashboard's editor](../operations/runtime-changes.md) exist for this.
156
+
157
+ **It still cannot deny anyone on a guess.** The guard applies here as everywhere. An
158
+ incident is precisely when people reach for `falsePositivePolicy: "aggressive"`, and
159
+ precisely when the population getting caught is at its most unusual.
160
+
161
+ **It is not DDoS protection.** This runs in your process, after the connection is
162
+ accepted. Volume that hurts at the network layer needs handling at the network layer;
163
+ what this reduces is the *usefulness* of the traffic to whoever is sending it.
164
+
165
+ And the practical warning the corpus makes concrete: like `protect-auth`, this refuses
166
+ proven automation, so your own webhooks, health probes and renderer are refused too —
167
+ thirteen of the corpus's infrastructure cases are. Allowlist their addresses *before* you
168
+ switch it on, not during the incident when you notice.
169
+
170
+ ## Choosing between them
171
+
172
+ By name, for config-driven setups:
173
+
174
+ ```ts
175
+ import { PRESETS, type PresetName } from "@osqd/bothandlerjs";
176
+ const rules = PRESETS[process.env.BOT_PRESET as PresetName]();
177
+ ```
178
+
179
+ Whichever you pick, run it against the corpus before it reaches production — the run tells
180
+ you which of your own traffic the policy refuses:
181
+
182
+ ```bash
183
+ npx @osqd/bothandlerjs check --preset protect-auth
184
+ ```
185
+
186
+ ## Related
187
+
188
+ - [Rules](rules.md) — the grammar these are written in
189
+ - [Actions](actions.md) — what each rule can ask for
190
+ - [Choosing a policy](../start/choosing-a-policy.md) — the decision, start to finish
191
+ - [The corpus](../testing/corpus.md) — 526 cases, including the ones these presets get wrong
@@ -0,0 +1,128 @@
1
+ # robots.txt
2
+
3
+ Saying out loud what your policy already does.
4
+
5
+ ← [Documentation](../index.md) · [Policy](index.md)
6
+
7
+ ---
8
+
9
+ ## Why this is here
10
+
11
+ Declining a crawler and not saying so is the worst of both worlds: it keeps coming, wastes
12
+ your bandwidth rediscovering that it is unwelcome on every request, and you get no credit
13
+ for having a policy. `robots.txt` is where you say it — and for the well-behaved crawlers,
14
+ saying it is the *only* thing you need to do, because they will simply stop.
15
+
16
+ Two things this will not pretend. `robots.txt` is a request, not enforcement; everything
17
+ that ignores it is exactly the population this library exists for. And a generated file can
18
+ only reflect rules it can **read**.
19
+
20
+ References: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) is the standard;
21
+ [robotstxt.org](https://www.robotstxt.org/) is the older convention it formalises.
22
+
23
+ ---
24
+
25
+ ## From your policy
26
+
27
+ The useful entry point. It reads your rules and writes the file that matches them:
28
+
29
+ ```ts
30
+ import { declineAiTraining, robotsFromRules } from "@osqd/bothandlerjs";
31
+
32
+ const { robotsTxt, declined, served, unreadable } = robotsFromRules(declineAiTraining(), {
33
+ disallowPaths: ["/internal/", "/admin-console"], // your trap paths belong here
34
+ sitemap: "https://example.com/sitemap.xml",
35
+ });
36
+
37
+ app.get("/robots.txt", (_req, res) => res.type("text/plain").send(robotsTxt));
38
+ ```
39
+
40
+ It reads every rule whose action [denies service](actions.md) and whose `match` names an
41
+ `identity` or a `category`, and declines exactly those crawlers.
42
+
43
+ ### The three fields you must look at
44
+
45
+ **`declined`** — the signature ids in the file, with categories expanded to the crawlers
46
+ they cover. Reporting only the explicitly-named ids would say "0 declined" for a policy
47
+ that turns away an entire category.
48
+
49
+ **`served`** — crawlers a later rule would have declined, but an earlier rule serves.
50
+ Policies are first-match-wins, so these are correctly *absent* from the file.
51
+ `decline-ai-training` is the case that found this: it serves `ChatGPT-User` and blocks the
52
+ rest of the `ai` category, and the generated file was telling `ChatGPT-User` to go away.
53
+ "Why is GPTBot in my robots.txt but ChatGPT-User is not" has an answer, and it is your own
54
+ rule order.
55
+
56
+ **`unreadable`** — rules that could not be read, with the reason. Two kinds:
57
+
58
+ - *the match is a predicate function* — it can be run, but not asked which crawlers it is
59
+ about.
60
+ - *the match is scoped to a path* — the rule denies service **there**, but a named group in
61
+ `robots.txt` gets `Disallow: /`, the whole site. Reported rather than passed over,
62
+ because the error runs in the expensive direction: a crawler told to stay away entirely
63
+ stops fetching the pages you wanted indexed, and that surfaces weeks later as a ranking
64
+ drop with nothing in the logs pointing at this file.
65
+
66
+ Check `unreadable` before publishing and add anything it names by hand. A `robots.txt` that
67
+ silently omits something you block is worse than no generated file at all — it tells
68
+ crawlers they are welcome where they are not.
69
+
70
+ ---
71
+
72
+ ## By hand
73
+
74
+ When you want a file that is not derived from a policy:
75
+
76
+ ```ts
77
+ import { generateRobotsTxt } from "@osqd/bothandlerjs";
78
+
79
+ generateRobotsTxt({
80
+ header: ["# Automated collection is declined. Contact abuse@example.com."],
81
+ disallowCategories: ["ai"],
82
+ disallowBots: ["semrushbot", "ahrefsbot"],
83
+ disallowPaths: ["/internal/", "/cart"],
84
+ allowPaths: ["/blog/"],
85
+ crawlDelay: 5,
86
+ sitemap: ["https://example.com/sitemap.xml"],
87
+ });
88
+ ```
89
+
90
+ | Option | Effect |
91
+ | ------ | ------ |
92
+ | `disallowBots` | signature ids to decline entirely |
93
+ | `disallowCategories` | whole [categories](../detection/signatures.md) — `ai`, `seo`, … |
94
+ | `disallowPaths` | disallowed for every crawler; put your [trap](../detection/detectors.md) paths here |
95
+ | `allowPaths` | rendered ahead of the disallows |
96
+ | `sitemap` | absolute URLs, per the specification |
97
+ | `crawlDelay` | seconds, wildcard group only; not honoured by every crawler |
98
+ | `header` | verbatim lines at the top, already comment-prefixed |
99
+ | `signatures` | the signature database to resolve ids against |
100
+
101
+ ### The grouping rule this gets right for you
102
+
103
+ Under RFC 9309 a crawler obeys the most specific group that names it and **ignores every
104
+ other group**. So a named `User-agent: GPTBot` group replaces the wildcard group outright
105
+ for GPTBot — including your `Disallow: /internal/` lines.
106
+
107
+ The renderer repeats global path rules inside every named group. Forgetting that is the
108
+ classic way a `robots.txt` accidentally *un*-blocks a trap path for exactly the crawlers
109
+ you were most careful about.
110
+
111
+ The wildcard group is emitted first purely as a courtesy to the person reading the file;
112
+ crawlers pick their group by specificity, not by position.
113
+
114
+ ---
115
+
116
+ ## From the command line
117
+
118
+ ```bash
119
+ npx @osqd/bothandlerjs robots --preset decline-ai-training --sitemap https://example.com/sitemap.xml
120
+ ```
121
+
122
+ See [the CLI](../testing/cli.md).
123
+
124
+ ## Related
125
+
126
+ - [Presets](presets.md) — `decline-ai-training`, which this pairs with
127
+ - [Signatures](../detection/signatures.md) — the ids and categories being named
128
+ - [Detectors](../detection/detectors.md) — the trap paths worth disallowing here
@@ -0,0 +1,134 @@
1
+ # Matching requests
2
+
3
+ Every field a rule can match on, and what each one is good for.
4
+
5
+ ← [Documentation](../index.md) · [Policy](index.md)
6
+
7
+ ---
8
+
9
+ ```ts
10
+ interface Rule {
11
+ id: string; // stable; names every decision and log line it produces
12
+ match: MatchSpec | ((assessment: Assessment) => boolean);
13
+ action: ActionName;
14
+ params?: ActionParams;
15
+ reason?: string;
16
+ }
17
+ ```
18
+
19
+ Every field in a `MatchSpec` must match — it is an `AND`. Fields accepting a list match if
20
+ *any* entry matches. An empty `match: {}` matches everything, which is occasionally what
21
+ you want at the bottom of a list.
22
+
23
+ ## What this request is
24
+
25
+ | Field | Type | |
26
+ | ----- | ---- | --- |
27
+ | `verdict` | one or more [`Verdict`](../concepts/verdicts.md) | `"suspected-bot"`, `["confirmed-bot", "verified-bot"]` |
28
+ | `botClass` | one or more `BotClass` | The field most policies want — it carries intent. |
29
+ | `certain` | `boolean` | `true` matches only proven verdicts, including a proven human. |
30
+ | `minScore` / `maxScore` | `number` | Inclusive. Only meaningful for probabilistic verdicts. |
31
+
32
+ ```ts
33
+ // Proven automation of any kind
34
+ { match: { certain: true, verdict: ["confirmed-bot", "verified-bot"] } }
35
+
36
+ // A judgement, but a confident one
37
+ { match: { verdict: "suspected-bot", minScore: 70 } }
38
+ ```
39
+
40
+ ## Who this claims to be
41
+
42
+ | Field | Type | |
43
+ | ----- | ---- | --- |
44
+ | `identity` | one or more `string` | A [signature](../detection/signatures.md) id: `"googlebot"`, `"gptbot"`. |
45
+ | `category` | one or more `BotCategory` | `"ai"`, `"search"`, `"seo"`, `"scanner"`, … |
46
+
47
+ ```ts
48
+ { id: "no-training", match: { category: "ai" }, action: "block" }
49
+ ```
50
+
51
+ **An identity match alone matches forgeries too**, because the identity is what the client
52
+ *claimed*. Pair it with `verdict: "verified-bot"` when trust is the point:
53
+
54
+ ```ts
55
+ { id: "trust-google", match: { identity: ["googlebot"], verdict: "verified-bot" }, action: "allow" }
56
+ ```
57
+
58
+ ## What it asked for
59
+
60
+ | Field | Type | |
61
+ | ----- | ---- | --- |
62
+ | `path` | `string \| RegExp` or a list | A string matches as a **prefix**; a regex is tested as written. |
63
+ | `method` | one or more `string` | Upper-case. |
64
+
65
+ ```ts
66
+ { id: "protect-export", match: { path: ["/api/export", "/reports/"], verdict: "suspected-bot" },
67
+ action: "challenge" }
68
+ ```
69
+
70
+ ## What fired
71
+
72
+ | Field | Type | |
73
+ | ----- | ---- | --- |
74
+ | `detector` | one or more `string` | Matches if **any** evidence came from one of these. |
75
+
76
+ ```ts
77
+ // Traps are proof. Nothing else needs to be true.
78
+ { id: "trapped", match: { detector: ["trap"], certain: true }, action: "block" }
79
+ ```
80
+
81
+ ## What this actor has done before
82
+
83
+ | Field | Type | |
84
+ | ----- | ---- | --- |
85
+ | `minPriorConfirmations` | `number` | Times this actor was **proven** a bot before now. |
86
+ | `minUnsolvedChallenges` | `number` | Challenges issued that no solution came back for. |
87
+
88
+ ```ts
89
+ { id: "repeat-offender", match: { minPriorConfirmations: 3 }, action: "block",
90
+ reason: "Proven automation three times from this actor." }
91
+
92
+ { id: "persistent-refusers", match: { minUnsolvedChallenges: 3 }, action: "rate-limit" }
93
+ ```
94
+
95
+ `minUnsolvedChallenges` is **outstanding rather than cumulative**: solving one clears the
96
+ count, so it never accumulates against somebody who came back and proved it. It is a rule
97
+ rather than evidence on purpose — one abandoned challenge is a person having a moment, and
98
+ what repetition means depends on traffic the library cannot see. See
99
+ [actors](../concepts/actors.md).
100
+
101
+ ## A predicate, for anything else
102
+
103
+ ```ts
104
+ {
105
+ id: "checkout-under-attack",
106
+ match: (assessment) =>
107
+ assessment.facts.path.startsWith("/checkout") &&
108
+ assessment.score > 50 &&
109
+ assessment.actor.requests > 20,
110
+ action: "challenge",
111
+ }
112
+ ```
113
+
114
+ Two things to know. A predicate that **throws** is skipped rather than matched — "unknown"
115
+ cannot safely mean "yes". And a predicate cannot be serialised, so a rule using one is
116
+ shown read-only in the dashboard's editor and spliced back at its original index on save:
117
+ order is the whole semantics, so "wherever it ends up" is not an option.
118
+
119
+ ## Validation
120
+
121
+ `validateRules` runs at construction and reports what is survivable rather than throwing:
122
+ duplicate ids, a rule shadowed by a broader one above it, an action whose parameters are
123
+ missing. Warnings go to `onWarning` and to the dashboard's notices.
124
+
125
+ ```ts
126
+ import { validateRules } from "@osqd/bothandlerjs";
127
+ validateRules(myRules); // string[] — empty is good
128
+ ```
129
+
130
+ ## Related
131
+
132
+ - [Actions](actions.md) — what a matched rule can ask for
133
+ - [The safety guard](../concepts/the-guard.md) — what it is allowed to get
134
+ - [Verdicts, classes and scores](../concepts/verdicts.md) — the vocabularies above
@@ -0,0 +1,126 @@
1
+ # API
2
+
3
+ The methods on `BotHandler`, the entry points, and where each exported name is documented.
4
+
5
+ ← [Documentation](../index.md)
6
+
7
+ ---
8
+
9
+ ## Entry points
10
+
11
+ | Import | Contains |
12
+ | ------ | -------- |
13
+ | `@osqd/bothandlerjs` | the engine, detectors, presets, robots, stores, notifiers, challenge, dashboard |
14
+ | `@osqd/bothandlerjs/adapters` | [Express, Fastify, Koa, Fetch](../integration/adapters.md) |
15
+ | `@osqd/bothandlerjs/client` | the browser-side [client signals](../detection/client-signals.md) script |
16
+ | `@osqd/bothandlerjs/corpus` | [`runCorpus`](../testing/corpus.md) and the cases |
17
+ | `@osqd/bothandlerjs/cli` | the [command line](../testing/cli.md) entry point |
18
+
19
+ Zero runtime dependencies, ESM and CJS, types included.
20
+
21
+ ---
22
+
23
+ ## The request path
24
+
25
+ ```ts
26
+ const assessment = await detector.assess(facts, { record: false });
27
+ const decision = detector.decide(assessment);
28
+ const result = await detector.handle(facts); // both, plus the action applied
29
+ ```
30
+
31
+ | Method | |
32
+ | ------ | - |
33
+ | `assess(facts, options?)` | runs detection. `{ record: false }` is a [dry run](../detection/index.md) |
34
+ | `decide(assessment)` | applies the [policy](../policy/index.md) and [the guard](../concepts/the-guard.md). Synchronous and pure |
35
+ | `handle(facts)` | assess + decide + produce an `ActionOutcome` |
36
+ | `createFacts(input)` | builds `RequestFacts` from method, url, headers, `rawHeaders`, ip |
37
+
38
+ `assess` and `decide` are separate on purpose: *what is this client* and *what should we do
39
+ about it* are different questions with different lifetimes. See [policy](../policy/index.md).
40
+
41
+ ## The challenge
42
+
43
+ | Method | |
44
+ | ------ | - |
45
+ | `isChallengeEndpoint(facts)` | is this the verification POST? Adapters call it for you |
46
+ | `verifyChallenge(facts, body)` | verify a solution, grant clearance |
47
+ | `grantClearance(facts, level?)` | issue clearance directly — `"operator"` by default |
48
+
49
+ `grantClearance(facts, "operator")` is how your application vouches for an authenticated
50
+ person. See [the challenge](../challenge/index.md).
51
+
52
+ ## Observation
53
+
54
+ | Method | |
55
+ | ------ | - |
56
+ | `on(event, handler)` | subscribe; returns an unsubscribe function |
57
+ | `metrics()` | structured snapshot |
58
+ | `prometheus(options?)` | text exposition format |
59
+ | `runAudit()` | run the [audit](../operations/audit.md) checks now |
60
+ | `audit` | the `TrafficAudit` object, with `summary()` |
61
+ | `serveDashboard(options?)` | start [the dashboard](../operations/dashboard.md) on its own port |
62
+ | `describeDetectors()` | id, description, cost and stage for each installed detector |
63
+
64
+ ## Runtime changes
65
+
66
+ Each takes a trailing `{ by }` — see [runtime changes](../operations/runtime-changes.md).
67
+
68
+ | Method | |
69
+ | ------ | - |
70
+ | `updatePolicy(rules, ctx?)` | validated, atomic; cannot touch the guard |
71
+ | `updateGuard(settings, ctx?)` | its own method and its own event, deliberately |
72
+ | `updateRanges(name, entries, ctx?)` | allowlist, denylist, datacenter |
73
+ | `updateCrawlerRanges(id, entries, ctx?)` | one crawler's published ranges |
74
+ | `listRanges()` / `rangeEntries(name)` | what is loaded |
75
+ | `forgetActor(key, ctx?)` | drop one [actor's](../concepts/actors.md) memory |
76
+ | `clearActor(key, forMs, ctx?)` | treat an actor as cleared for a while |
77
+
78
+ ## Inspection
79
+
80
+ `resolveIp`, `actorKeyFor`, `isAllowlisted`, `isIgnoredPath`, `policy`, `warn`.
81
+
82
+ `warn(message)` is public so warnings raised *outside* the engine — the fetch adapter, the
83
+ crawler-range refresher — reach the same `warning` event as everything else, rather than
84
+ only the `onWarning` callback.
85
+
86
+ ---
87
+
88
+ ## Where the exported names are documented
89
+
90
+ | Names | Page |
91
+ | ----- | ---- |
92
+ | `BotHandler`, `createFacts`, `resolveConfig`, `validateRules`, `ConfigError` | [Configuration](configuration.md) |
93
+ | `CERTAINTY_WEIGHT`, `combineEvidence`, `noisyOr`, `sortEvidence`, `weightOf` | [Evidence](../concepts/evidence.md) |
94
+ | `VERDICTS`, `BOT_CLASSES` | [Verdicts](../concepts/verdicts.md) |
95
+ | `defaultDetectors`, every `*Detector` factory, `Detector`, `Evidence` | [Detectors](../detection/detectors.md), [writing one](../detection/writing-a-detector.md) |
96
+ | `BOT_SIGNATURES`, `BotSignature`, `BotCategory` | [Signatures](../detection/signatures.md) |
97
+ | `forwardConfirmedReverseDns`, `cachingResolver`, `nodeDnsResolver`, `PUBLISHED_CRAWLER_RANGES`, `refreshCrawlerRanges`, `startCrawlerRangeRefresh` | [Verification](../detection/verification.md) |
98
+ | `PRESETS`, every preset function, `Rule`, `MatchSpec`, `Decision` | [Rules](../policy/rules.md), [presets](../policy/presets.md) |
99
+ | `defineHandler`, `CustomHandler`, `ActionOutcome`, `ACTION_NAMES`, `TERMINAL_ACTIONS` | [Actions](../policy/actions.md) |
100
+ | `generateRobotsTxt`, `robotsFromRules`, `agentFor` | [robots.txt](../policy/robots.md) |
101
+ | `ChallengeService`, `ChallengeOptions`, `ChallengeCopy`, `parseAcceptLanguage`, `pickTranslation` | [The challenge](../challenge/index.md), [localisation](../challenge/localisation.md) |
102
+ | `MemoryStore`, `RedisStore`, `BotHandlerStore` | [Stores](../integration/stores.md) |
103
+ | `consoleNotifier`, `webhookNotifier`, `slackNotifier`, `notifyJsNotifier`, `Notifier` | [Notifications](../operations/notifications.md) |
104
+ | `Metrics`, `toPrometheus`, `SCORE_BUCKETS`, `DURATION_BUCKETS_MS` | [Metrics](../operations/metrics.md) |
105
+ | `TrafficAudit`, `DEFAULT_CHECKS`, `AuditCheck`, `TrafficAnomaly` | [The audit](../operations/audit.md) |
106
+ | `startDashboard`, `createDashboardHandler`, `renderDashboardPage`, `Dashboard*` | [The dashboard](../operations/dashboard.md) |
107
+ | `ActorRegistry`, `ActorState`, `ActorSummary` | [Actors](../concepts/actors.md) |
108
+ | `renderTrapField`, `TRAP_FIELD_SOURCE` | [Detectors](../detection/detectors.md), [adapters](../integration/adapters.md) |
109
+ | `ClientSignals`, the client script | [Client signals](../detection/client-signals.md) |
110
+
111
+ ### The utilities
112
+
113
+ Exported because they are useful on their own and because a detector you write will want
114
+ them: `IpRangeSet`, `parseIp`, `parseCidr`, `cidrContains`, `normalizeIp`, `formatIp`,
115
+ `isSpecialUse`, `networkKey`, `SPECIAL_USE_RANGES`; `parseUserAgent`, `claimsBrowser`,
116
+ `sendsModernHeaders`; `parseCookies`, `serializeCookie`; `MultiPatternMatcher`, `TtlLru`,
117
+ `Emitter`, `ManualClock`, `systemClock`.
118
+
119
+ `ManualClock` and a stub `DnsResolver` are what make the [corpus](../testing/corpus.md)
120
+ deterministic, and they will do the same for your tests.
121
+
122
+ ## Related
123
+
124
+ - [Configuration](configuration.md) — every constructor option
125
+ - [Detection](../detection/index.md) — what `assess` actually runs
126
+ - [Policy](../policy/index.md) — what `decide` actually applies