@jimhoyd/urlcode 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (254) hide show
  1. package/CONTRIBUTING.md +80 -0
  2. package/LICENSE +202 -0
  3. package/NOTICE +20 -0
  4. package/README.md +342 -0
  5. package/ROADMAP.md +248 -0
  6. package/SECURITY.md +40 -0
  7. package/data/agents/LICENSES/ai-robots-txt.txt +24 -0
  8. package/data/agents/LICENSES/crawler-user-agents.txt +23 -0
  9. package/data/agents/ai-crawlers.json +1240 -0
  10. package/data/agents/crawlers.json +10515 -0
  11. package/data/agents/index.d.ts +3 -0
  12. package/data/agents/index.js +2453 -0
  13. package/data/agents/monitoring.json +1520 -0
  14. package/data/agents/seo.json +3865 -0
  15. package/dist/BUILD-MANIFEST.json +60 -0
  16. package/dist/adapters.js +59 -0
  17. package/dist/agent-lists.js +59 -0
  18. package/dist/assets.js +129 -0
  19. package/dist/authoring.js +68 -0
  20. package/dist/aws.js +110 -0
  21. package/dist/build-cloudflare.js +164 -0
  22. package/dist/cli.js +185 -0
  23. package/dist/client-address.js +85 -0
  24. package/dist/cloudflare.js +165 -0
  25. package/dist/compliance-rules/baseline.js +98 -0
  26. package/dist/compliance-rules/privacy.js +38 -0
  27. package/dist/compliance-rules/shared.js +62 -0
  28. package/dist/compliance-rules/strict.js +61 -0
  29. package/dist/compliance.js +219 -0
  30. package/dist/config-worker.js +9 -0
  31. package/dist/config.js +172 -0
  32. package/dist/errors.js +8 -0
  33. package/dist/function-sources.js +54 -0
  34. package/dist/function-worker.js +105 -0
  35. package/dist/functions.js +174 -0
  36. package/dist/guest-api.js +114 -0
  37. package/dist/header-validation.js +18 -0
  38. package/dist/http-policy.js +74 -0
  39. package/dist/http-response.js +84 -0
  40. package/dist/index.js +11 -0
  41. package/dist/link-api.js +136 -0
  42. package/dist/link-cli.js +141 -0
  43. package/dist/link-events.js +76 -0
  44. package/dist/link-records.js +31 -0
  45. package/dist/link-store-worker.js +150 -0
  46. package/dist/link-store.js +250 -0
  47. package/dist/logging.js +22 -0
  48. package/dist/management-policy.js +41 -0
  49. package/dist/match.js +124 -0
  50. package/dist/observability.js +242 -0
  51. package/dist/plugins.js +74 -0
  52. package/dist/policies/agents.js +248 -0
  53. package/dist/policies/cache.js +297 -0
  54. package/dist/policies/compression.js +187 -0
  55. package/dist/policies/security.js +0 -0
  56. package/dist/policies/throttle.js +131 -0
  57. package/dist/policies.js +142 -0
  58. package/dist/policy.js +55 -0
  59. package/dist/prerender.js +165 -0
  60. package/dist/project-tests.js +23 -0
  61. package/dist/readiness.js +216 -0
  62. package/dist/router.js +166 -0
  63. package/dist/runtime.js +244 -0
  64. package/dist/scaffold.js +0 -0
  65. package/dist/scripts/operational-drills.js +75 -0
  66. package/dist/server.js +270 -0
  67. package/dist/site.js +0 -0
  68. package/dist/sqlite-version.js +6 -0
  69. package/dist/types/adapters.d.ts +12 -0
  70. package/dist/types/agent-lists.d.ts +5 -0
  71. package/dist/types/assets.d.ts +13 -0
  72. package/dist/types/authoring.d.ts +2 -0
  73. package/dist/types/aws.d.ts +35 -0
  74. package/dist/types/build-cloudflare.d.ts +14 -0
  75. package/dist/types/cli.d.ts +2 -0
  76. package/dist/types/client-address.d.ts +9 -0
  77. package/dist/types/cloudflare.d.ts +75 -0
  78. package/dist/types/compliance-rules/baseline.d.ts +12 -0
  79. package/dist/types/compliance-rules/privacy.d.ts +6 -0
  80. package/dist/types/compliance-rules/shared.d.ts +24 -0
  81. package/dist/types/compliance-rules/strict.d.ts +9 -0
  82. package/dist/types/compliance.d.ts +119 -0
  83. package/dist/types/config-worker.d.ts +1 -0
  84. package/dist/types/config.d.ts +21 -0
  85. package/dist/types/errors.d.ts +7 -0
  86. package/dist/types/function-sources.d.ts +21 -0
  87. package/dist/types/function-worker.d.ts +1 -0
  88. package/dist/types/functions.d.ts +92 -0
  89. package/dist/types/guest-api.d.ts +16 -0
  90. package/dist/types/header-validation.d.ts +2 -0
  91. package/dist/types/http-policy.d.ts +39 -0
  92. package/dist/types/http-response.d.ts +43 -0
  93. package/dist/types/index.d.ts +11 -0
  94. package/dist/types/link-api.d.ts +30 -0
  95. package/dist/types/link-cli.d.ts +36 -0
  96. package/dist/types/link-events.d.ts +27 -0
  97. package/dist/types/link-records.d.ts +11 -0
  98. package/dist/types/link-store-worker.d.ts +1 -0
  99. package/dist/types/link-store.d.ts +130 -0
  100. package/dist/types/logging.d.ts +9 -0
  101. package/dist/types/management-policy.d.ts +9 -0
  102. package/dist/types/match.d.ts +81 -0
  103. package/dist/types/observability.d.ts +104 -0
  104. package/dist/types/plugins.d.ts +27 -0
  105. package/dist/types/policies/agents.d.ts +86 -0
  106. package/dist/types/policies/cache.d.ts +82 -0
  107. package/dist/types/policies/compression.d.ts +41 -0
  108. package/dist/types/policies/security.d.ts +37 -0
  109. package/dist/types/policies/throttle.d.ts +55 -0
  110. package/dist/types/policies.d.ts +36 -0
  111. package/dist/types/policy.d.ts +21 -0
  112. package/dist/types/prerender.d.ts +31 -0
  113. package/dist/types/project-tests.d.ts +13 -0
  114. package/dist/types/readiness.d.ts +121 -0
  115. package/dist/types/router.d.ts +11 -0
  116. package/dist/types/runtime.d.ts +91 -0
  117. package/dist/types/scaffold.d.ts +17 -0
  118. package/dist/types/server.d.ts +37 -0
  119. package/dist/types/site.d.ts +12 -0
  120. package/dist/types/sqlite-version.d.ts +1 -0
  121. package/dist/types/types.d.ts +332 -0
  122. package/dist/types/vercel.d.ts +12 -0
  123. package/dist/types.js +177 -0
  124. package/dist/vercel.js +65 -0
  125. package/docs/AI-AUTHORING.md +113 -0
  126. package/docs/ASSETS.md +106 -0
  127. package/docs/AWS.md +77 -0
  128. package/docs/BEST-PRACTICES.md +267 -0
  129. package/docs/CAPACITY.md +208 -0
  130. package/docs/CLOUDFLARE.md +110 -0
  131. package/docs/COMPLIANCE.md +242 -0
  132. package/docs/DYNAMIC-LINKS.md +561 -0
  133. package/docs/FUNCTION-SECURITY.md +113 -0
  134. package/docs/HTTP.md +129 -0
  135. package/docs/INSTALL.md +90 -0
  136. package/docs/LOAD-TESTING.md +91 -0
  137. package/docs/LOCAL-DEVELOPMENT.md +99 -0
  138. package/docs/MANAGEMENT-SECURITY.md +82 -0
  139. package/docs/MIDDLEWARE.md +83 -0
  140. package/docs/MONITORING.md +132 -0
  141. package/docs/OBSERVABILITY.md +229 -0
  142. package/docs/OPERATIONAL-PROOF.md +42 -0
  143. package/docs/OPERATIONS.md +207 -0
  144. package/docs/ORGANIZATION.md +135 -0
  145. package/docs/PERFORMANCE.md +72 -0
  146. package/docs/PLUGINS.md +234 -0
  147. package/docs/POLICIES.md +489 -0
  148. package/docs/PRERENDER.md +193 -0
  149. package/docs/PROJECT-DIRECTION.md +51 -0
  150. package/docs/READINESS.md +125 -0
  151. package/docs/README.md +58 -0
  152. package/docs/RELEASE-READINESS.md +95 -0
  153. package/docs/RELEASE-SECURITY.md +72 -0
  154. package/docs/RESILIENCE.md +160 -0
  155. package/docs/ROUTING.md +95 -0
  156. package/docs/SANDBOX-REVIEW.md +59 -0
  157. package/docs/SCAFFOLDING.md +72 -0
  158. package/docs/SECURITY-AUDIT.md +124 -0
  159. package/docs/SITE.md +150 -0
  160. package/docs/SPECIFICATION.md +232 -0
  161. package/docs/SPIKE-EXTENSIONS.md +475 -0
  162. package/docs/STANDARDS.md +303 -0
  163. package/docs/STARTERS.md +51 -0
  164. package/docs/TUNNELS.md +75 -0
  165. package/docs/TYPESCRIPT.md +102 -0
  166. package/docs/VERCEL.md +107 -0
  167. package/docs/YAML-GUIDE.md +521 -0
  168. package/docs/YAML-REFERENCE.md +328 -0
  169. package/docs/policies/agents.md +182 -0
  170. package/docs/policies/cache.md +139 -0
  171. package/docs/policies/compression.md +168 -0
  172. package/docs/policies/security.md +160 -0
  173. package/docs/policies/throttle.md +103 -0
  174. package/examples/assets/.env.example +1 -0
  175. package/examples/assets/.gitattributes +2 -0
  176. package/examples/assets/Makefile +30 -0
  177. package/examples/assets/README.md +18 -0
  178. package/examples/assets/functions/hello.mjs +3 -0
  179. package/examples/assets/gitignore.template +7 -0
  180. package/examples/assets/public/about.html +2 -0
  181. package/examples/assets/public/assets/example.txt +1 -0
  182. package/examples/assets/public/guide.txt +1 -0
  183. package/examples/assets/tests/requests.json +44 -0
  184. package/examples/assets/urlcode.yaml +27 -0
  185. package/examples/aws/README.md +13 -0
  186. package/examples/aws/handler.mjs +5 -0
  187. package/examples/aws/public/index.html +3 -0
  188. package/examples/aws/public/notes.txt +1 -0
  189. package/examples/aws/template.yaml +33 -0
  190. package/examples/aws/tests/requests.json +9 -0
  191. package/examples/aws/urlcode.yaml +19 -0
  192. package/examples/cloudflare/README.md +18 -0
  193. package/examples/cloudflare/package.json +14 -0
  194. package/examples/cloudflare/tests/requests.json +9 -0
  195. package/examples/cloudflare/urlcode.yaml +24 -0
  196. package/examples/cloudflare/wrangler.toml +5 -0
  197. package/examples/compliance/README.md +20 -0
  198. package/examples/compliance/rules.mjs +65 -0
  199. package/examples/cookbook/README.md +21 -0
  200. package/examples/cookbook/functions/choice.mjs +4 -0
  201. package/examples/cookbook/functions/echo.mjs +3 -0
  202. package/examples/cookbook/functions/hello.mjs +3 -0
  203. package/examples/cookbook/functions/text.mjs +3 -0
  204. package/examples/cookbook/middleware/headers.mjs +6 -0
  205. package/examples/cookbook/public/about.html +1 -0
  206. package/examples/cookbook/public/assets/index.html +1 -0
  207. package/examples/cookbook/public/assets/site.css +1 -0
  208. package/examples/cookbook/public/favicon.svg +1 -0
  209. package/examples/cookbook/public/guide.txt +1 -0
  210. package/examples/cookbook/public/llms.txt +6 -0
  211. package/examples/cookbook/routes/code.yaml +55 -0
  212. package/examples/cookbook/routes/files.yaml +17 -0
  213. package/examples/cookbook/routes/policies.yaml +28 -0
  214. package/examples/cookbook/routes/redirects.yaml +38 -0
  215. package/examples/cookbook/routes/responses.yaml +26 -0
  216. package/examples/cookbook/tests/requests.json +254 -0
  217. package/examples/cookbook/urlcode.yaml +22 -0
  218. package/examples/live-links/README.md +11 -0
  219. package/examples/live-links/tests/requests.json +6 -0
  220. package/examples/live-links/urlcode.yaml +16 -0
  221. package/examples/monitoring/blackbox-jobs.yaml +37 -0
  222. package/examples/monitoring/prometheus-rules.yaml +88 -0
  223. package/examples/monitoring/prometheus-scrape.yaml +20 -0
  224. package/examples/monitoring/vector.toml +65 -0
  225. package/examples/prerender/README.md +48 -0
  226. package/examples/prerender/functions/page.mjs +6 -0
  227. package/examples/prerender/middleware/template.mjs +29 -0
  228. package/examples/prerender/prerender.d.mts +4 -0
  229. package/examples/prerender/prerender.mjs +63 -0
  230. package/examples/prerender/tests/requests.json +39 -0
  231. package/examples/prerender/urlcode.yaml +50 -0
  232. package/examples/tunnel/dev-with-ngrok.sh +61 -0
  233. package/examples/vercel/README.md +13 -0
  234. package/examples/vercel/api/index.js +5 -0
  235. package/examples/vercel/package.json +8 -0
  236. package/examples/vercel/public/index.html +3 -0
  237. package/examples/vercel/public/notes.txt +1 -0
  238. package/examples/vercel/tests/requests.json +48 -0
  239. package/examples/vercel/urlcode.yaml +19 -0
  240. package/examples/vercel/vercel.json +10 -0
  241. package/llms.txt +46 -0
  242. package/package.json +123 -0
  243. package/schemas/urlcode.schema.json +1230 -0
  244. package/starters/default/.gitattributes +1 -0
  245. package/starters/default/Makefile +30 -0
  246. package/starters/default/README.md +34 -0
  247. package/starters/default/functions/hello.mjs +3 -0
  248. package/starters/default/gitignore.template +8 -0
  249. package/starters/default/middleware/headers.mjs +6 -0
  250. package/starters/default/routes/functions.yaml +20 -0
  251. package/starters/default/routes/marketing/links.yaml +7 -0
  252. package/starters/default/starter.json +5 -0
  253. package/starters/default/tests/requests.json +56 -0
  254. package/starters/default/urlcode.yaml +8 -0
@@ -0,0 +1,328 @@
1
+ # YAML field reference
2
+
3
+ Generated from the bundled JSON Schema by `npm run docs:reference`. Required
4
+ means required within its containing object, not that the object itself must be
5
+ present. `routes.*` means a route path; other `*` markers mean user-selected
6
+ keys. `[]` means an array item. Option rows describe union alternatives.
7
+
8
+ Read the [YAML guide](YAML-GUIDE.md) for examples and [specification](SPECIFICATION.md)
9
+ for semantic validation beyond JSON Schema. Exactly one handler is required per
10
+ route; respond.text/respond.json are mutually exclusive. Runtime defaults include
11
+ GET/HEAD, redirect 302, respond 200, default module export, and asset no-cache.
12
+ Only Set-Cookie accepts response header arrays. This table does not imply all
13
+ schema-valid combinations activate successfully.
14
+
15
+ | Field | Type | Required | Schema constraints |
16
+ |---|---|---|---|
17
+ | `version` | constant | yes | const: "1" |
18
+ | `routes` | object | yes | maxProperties: 100000 |
19
+ | `routes.*` | object | no | unknown keys rejected |
20
+ | `routes.*.methods` | array | no | default: ["GET","HEAD"]; minItems: 1; uniqueItems: true |
21
+ | `routes.*.methods[]` | string | no | enum: ["GET","HEAD","POST","PUT","PATCH","DELETE","OPTIONS"] |
22
+ | `routes.*.enabled` | boolean | no | — |
23
+ | `routes.*.expires` | string | no | — |
24
+ | `routes.*.description` | string | no | maxLength: 1024 |
25
+ | `routes.*.parameters` | array | no | maxItems: 64 |
26
+ | `routes.*.parameters[]` | object | no | unknown keys rejected |
27
+ | `routes.*.parameters[].name` | string | yes | pattern: "^[A-Za-z_][A-Za-z0-9_-]*$" |
28
+ | `routes.*.parameters[].in` | string | yes | enum: ["path","query","header"] |
29
+ | `routes.*.parameters[].required` | boolean | no | — |
30
+ | `routes.*.parameters[].schema` | object | yes | unknown keys rejected |
31
+ | `routes.*.parameters[].schema.type` | string | yes | enum: ["string","integer","number","boolean","array"] |
32
+ | `routes.*.parameters[].schema.enum` | array | no | minItems: 1; uniqueItems: true |
33
+ | `routes.*.parameters[].schema.enum[]` | string / number / boolean | no | — |
34
+ | `routes.*.parameters[].schema.default` | string / number / boolean / array | no | — |
35
+ | `routes.*.parameters[].schema.default[]` | string / number / boolean | no | — |
36
+ | `routes.*.parameters[].schema.minLength` | integer | no | minimum: 0; maximum: 8192 |
37
+ | `routes.*.parameters[].schema.maxLength` | integer | no | minimum: 0; maximum: 8192 |
38
+ | `routes.*.parameters[].schema.minimum` | number | no | — |
39
+ | `routes.*.parameters[].schema.maximum` | number | no | — |
40
+ | `routes.*.parameters[].schema.items` | object | no | unknown keys rejected |
41
+ | `routes.*.parameters[].schema.items.type` | string | yes | enum: ["string","integer","number","boolean"] |
42
+ | `routes.*.parameters[].schema.maxItems` | integer | no | minimum: 0; maximum: 100 |
43
+ | `routes.*.redirect` | object | no | unknown keys rejected |
44
+ | `routes.*.redirect.url` | string | yes | maxLength: 8192 |
45
+ | `routes.*.redirect.status` | number | no | enum: [301,302,303,307,308] |
46
+ | `routes.*.redirect.query` | object | no | unknown keys rejected |
47
+ | `routes.*.redirect.query.pass` | one of the shapes below | no | — |
48
+ | `routes.*.redirect.query.pass (option 1)` | constant | no | const: false |
49
+ | `routes.*.redirect.query.pass (option 2)` | array | no | uniqueItems: true |
50
+ | `routes.*.redirect.query.pass (option 2)[]` | string | no | — |
51
+ | `routes.*.redirect.query.map` | object | no | — |
52
+ | `routes.*.redirect.query.map.*` | object | no | unknown keys rejected |
53
+ | `routes.*.redirect.query.map.*.from` | string | yes | enum: ["path","query","header"] |
54
+ | `routes.*.redirect.query.map.*.name` | string | yes | — |
55
+ | `routes.*.function` | object | no | unknown keys rejected |
56
+ | `routes.*.function.source` | string | yes | maxLength: 1024 |
57
+ | `routes.*.function.export` | string | no | pattern: "^[A-Za-z_][A-Za-z0-9_]*$" |
58
+ | `routes.*.function.args` | object | no | — |
59
+ | `routes.*.function.args.*` | one of the shapes below | no | — |
60
+ | `routes.*.function.args.* (option 1)` | string / number / boolean | no | — |
61
+ | `routes.*.function.args.* (option 2)` | object | no | unknown keys rejected |
62
+ | `routes.*.function.args.* (option 2).from` | string | yes | enum: ["path","query","header"] |
63
+ | `routes.*.function.args.* (option 2).name` | string | yes | — |
64
+ | `routes.*.function.args.* (option 3)` | object | no | unknown keys rejected |
65
+ | `routes.*.function.args.* (option 3).env` | string | yes | pattern: "^[A-Za-z_][A-Za-z0-9_]*$" |
66
+ | `routes.*.function.args.* (option 4)` | object | no | unknown keys rejected |
67
+ | `routes.*.function.args.* (option 4).secret` | string | yes | pattern: "^[A-Za-z_][A-Za-z0-9_]*$" |
68
+ | `routes.*.env` | object | no | — |
69
+ | `routes.*.env.*` | one of the shapes below | no | — |
70
+ | `routes.*.env.* (option 1)` | object | no | unknown keys rejected |
71
+ | `routes.*.env.* (option 1).value` | string | yes | — |
72
+ | `routes.*.env.* (option 2)` | object | no | unknown keys rejected |
73
+ | `routes.*.env.* (option 2).env` | string | yes | pattern: "^[A-Za-z_][A-Za-z0-9_]*$" |
74
+ | `routes.*.secrets` | object | no | — |
75
+ | `routes.*.secrets.*` | object | no | unknown keys rejected |
76
+ | `routes.*.secrets.*.secret` | string | yes | pattern: "^[A-Za-z_][A-Za-z0-9_]*$" |
77
+ | `routes.*.page` | object | no | unknown keys rejected |
78
+ | `routes.*.page.file` | string | yes | minLength: 1; maxLength: 1024 |
79
+ | `routes.*.page.contentType` | string | no | maxLength: 128; pattern: "^[A-Za-z0-9!#$&^_.+-]+/[A-Za-z0-9!#$&^_.+-]+$" |
80
+ | `routes.*.page.cacheControl` | string | no | enum: ["no-cache","no-store","public, max-age=3600","public, max-age=31536000, immutable"] |
81
+ | `routes.*.download` | object | no | unknown keys rejected |
82
+ | `routes.*.download.file` | string | yes | minLength: 1; maxLength: 1024 |
83
+ | `routes.*.download.contentType` | string | no | maxLength: 128; pattern: "^[A-Za-z0-9!#$&^_.+-]+/[A-Za-z0-9!#$&^_.+-]+$" |
84
+ | `routes.*.download.cacheControl` | string | no | enum: ["no-cache","no-store","public, max-age=3600","public, max-age=31536000, immutable"] |
85
+ | `routes.*.download.filename` | string | no | minLength: 1; maxLength: 255 |
86
+ | `routes.*.static` | object | no | unknown keys rejected |
87
+ | `routes.*.static.directory` | string | yes | minLength: 1; maxLength: 1024 |
88
+ | `routes.*.static.contentType` | string | no | maxLength: 128; pattern: "^[A-Za-z0-9!#$&^_.+-]+/[A-Za-z0-9!#$&^_.+-]+$" |
89
+ | `routes.*.static.cacheControl` | string | no | enum: ["no-cache","no-store","public, max-age=3600","public, max-age=31536000, immutable"] |
90
+ | `routes.*.static.index` | string | no | pattern: "^[A-Za-z0-9_-]+\\.html$" |
91
+ | `routes.*.request` | object | no | unknown keys rejected |
92
+ | `routes.*.request.body` | object | no | unknown keys rejected |
93
+ | `routes.*.request.body.required` | boolean | no | — |
94
+ | `routes.*.request.body.maxBytes` | integer | no | minimum: 0; maximum: 1048576 |
95
+ | `routes.*.request.body.contentTypes` | array | no | minItems: 1; maxItems: 16; uniqueItems: true |
96
+ | `routes.*.request.body.contentTypes[]` | string | no | pattern: "^[a-z0-9!#$&^_.+-]+/[a-z0-9!#$&^_.+-]+$" |
97
+ | `routes.*.request.body.format` | string | no | enum: ["text","json"] |
98
+ | `routes.*.response` | object | no | unknown keys rejected |
99
+ | `routes.*.response.headers` | object | no | maxProperties: 64 |
100
+ | `routes.*.response.headers.*` | one of the shapes below | no | — |
101
+ | `routes.*.response.headers.* (option 1)` | string | no | maxLength: 4096 |
102
+ | `routes.*.response.headers.* (option 2)` | array | no | minItems: 1; maxItems: 16 |
103
+ | `routes.*.response.headers.* (option 2)[]` | string | no | maxLength: 4096 |
104
+ | `routes.*.respond` | object | no | unknown keys rejected |
105
+ | `routes.*.respond.status` | integer | no | minimum: 200; maximum: 599 |
106
+ | `routes.*.respond.text` | string | no | maxLength: 1048576 |
107
+ | `routes.*.respond.json` | any JSON value | no | — |
108
+ | `routes.*.middleware` | array | no | maxItems: 16 |
109
+ | `routes.*.middleware[]` | object | no | unknown keys rejected |
110
+ | `routes.*.middleware[].source` | string | yes | maxLength: 1024 |
111
+ | `routes.*.middleware[].export` | string | no | pattern: "^[A-Za-z_][A-Za-z0-9_]*$" |
112
+ | `routes.*.link` | object | no | unknown keys rejected |
113
+ | `routes.*.link.collection` | string | yes | pattern: "^[A-Za-z][A-Za-z0-9_-]{0,63}$" |
114
+ | `routes.*.link.code` | object | yes | unknown keys rejected |
115
+ | `routes.*.link.code.from` | constant | yes | const: "path" |
116
+ | `routes.*.link.code.name` | string | yes | pattern: "^[A-Za-z_][A-Za-z0-9_]*$" |
117
+ | `routes.*.policies` | object | no | unknown keys rejected |
118
+ | `routes.*.policies.profile` | string | no | pattern: "^[a-z][a-z0-9-]{0,63}$" |
119
+ | `routes.*.policies.throttle` | one of the shapes below | no | — |
120
+ | `routes.*.policies.throttle (option 1)` | constant | no | const: false |
121
+ | `routes.*.policies.throttle (option 2)` | object | no | unknown keys rejected |
122
+ | `routes.*.policies.throttle (option 2).quota` | integer | no | minimum: 1; maximum: 1000000 |
123
+ | `routes.*.policies.throttle (option 2).window` | integer | no | minimum: 1; maximum: 86400 |
124
+ | `routes.*.policies.throttle (option 2).partition` | string | no | enum: ["client","route","client-route"]; default: "client" |
125
+ | `routes.*.policies.throttle (option 2).status` | integer | no | default: 429; minimum: 400; maximum: 599 |
126
+ | `routes.*.policies.throttle (option 2).mode` | string | no | enum: ["enforce","report"]; default: "enforce" |
127
+ | `routes.*.policies.throttle (option 2).maxKeys` | integer | no | default: 100000; minimum: 1000; maximum: 10000000 |
128
+ | `routes.*.policies.agents` | one of the shapes below | no | — |
129
+ | `routes.*.policies.agents (option 1)` | constant | no | const: false |
130
+ | `routes.*.policies.agents (option 2)` | object | no | unknown keys rejected |
131
+ | `routes.*.policies.agents (option 2).deny` | array | no | maxItems: 32; uniqueItems: true |
132
+ | `routes.*.policies.agents (option 2).deny[]` | string | no | minLength: 1; maxLength: 1024 |
133
+ | `routes.*.policies.agents (option 2).allow` | array | no | maxItems: 32; uniqueItems: true |
134
+ | `routes.*.policies.agents (option 2).allow[]` | string | no | minLength: 1; maxLength: 1024 |
135
+ | `routes.*.policies.agents (option 2).denyPatterns` | array | no | maxItems: 256; uniqueItems: true |
136
+ | `routes.*.policies.agents (option 2).denyPatterns[]` | string | no | minLength: 1; maxLength: 256 |
137
+ | `routes.*.policies.agents (option 2).allowPatterns` | array | no | maxItems: 256; uniqueItems: true |
138
+ | `routes.*.policies.agents (option 2).allowPatterns[]` | string | no | minLength: 1; maxLength: 256 |
139
+ | `routes.*.policies.agents (option 2).denyEmpty` | boolean | no | default: false |
140
+ | `routes.*.policies.agents (option 2).status` | integer | no | default: 403; minimum: 400; maximum: 599 |
141
+ | `routes.*.policies.agents (option 2).mode` | string | no | enum: ["enforce","report"]; default: "enforce" |
142
+ | `routes.*.policies.security` | one of the shapes below | no | — |
143
+ | `routes.*.policies.security (option 1)` | constant | no | const: false |
144
+ | `routes.*.policies.security (option 2)` | object | no | unknown keys rejected |
145
+ | `routes.*.policies.security (option 2).headers` | string | no | enum: ["oshp","oshp-no-csp","off"]; default: "oshp" |
146
+ | `routes.*.policies.security (option 2).set` | object | no | maxProperties: 32 |
147
+ | `routes.*.policies.security (option 2).set.*` | string | no | maxLength: 4096 |
148
+ | `routes.*.policies.security (option 2).unset` | array | no | maxItems: 32; uniqueItems: true |
149
+ | `routes.*.policies.security (option 2).unset[]` | string | no | minLength: 1; maxLength: 128 |
150
+ | `routes.*.policies.compression` | one of the shapes below | no | — |
151
+ | `routes.*.policies.compression (option 1)` | constant | no | const: false |
152
+ | `routes.*.policies.compression (option 2)` | object | no | unknown keys rejected |
153
+ | `routes.*.policies.compression (option 2).encodings` | array | no | default: ["br","gzip"]; minItems: 1; uniqueItems: true |
154
+ | `routes.*.policies.compression (option 2).encodings[]` | string | no | enum: ["br","gzip","deflate","zstd"] |
155
+ | `routes.*.policies.compression (option 2).minBytes` | integer | no | default: 1024; minimum: 0; maximum: 1048576 |
156
+ | `routes.*.policies.compression (option 2).types` | array | no | maxItems: 64; uniqueItems: true |
157
+ | `routes.*.policies.compression (option 2).types[]` | string | no | minLength: 1; maxLength: 128 |
158
+ | `routes.*.policies.compression (option 2).level` | integer | no | minimum: 1; maximum: 11 |
159
+ | `routes.*.policies.compression (option 2).allowWithSecrets` | boolean | no | default: false |
160
+ | `routes.*.policies.cache` | one of the shapes below | no | — |
161
+ | `routes.*.policies.cache (option 1)` | constant | no | const: false |
162
+ | `routes.*.policies.cache (option 2)` | object | no | unknown keys rejected |
163
+ | `routes.*.policies.cache (option 2).strategy` | string | no | enum: ["no-store","revalidate","public","immutable","swr","sie","micro","cdn-only","private"] |
164
+ | `routes.*.policies.cache (option 2).maxAge` | integer | no | minimum: 0; maximum: 31536000 |
165
+ | `routes.*.policies.cache (option 2).staleWhileRevalidate` | integer | no | minimum: 0; maximum: 31536000 |
166
+ | `routes.*.policies.cache (option 2).staleIfError` | integer | no | minimum: 0; maximum: 31536000 |
167
+ | `routes.*.policies.cache (option 2).cdnMaxAge` | integer | no | minimum: 0; maximum: 31536000 |
168
+ | `routes.*.policies.cache (option 2).originTtl` | integer | no | minimum: 0; maximum: 86400 |
169
+ | `routes.*.policies.cache (option 2).vary` | array | no | maxItems: 8; uniqueItems: true |
170
+ | `routes.*.policies.cache (option 2).vary[]` | string | no | minLength: 1; maxLength: 128 |
171
+ | `routes.*.policies.cache (option 2).statuses` | array | no | maxItems: 16; uniqueItems: true |
172
+ | `routes.*.policies.cache (option 2).statuses[]` | integer | no | minimum: 200; maximum: 599 |
173
+ | `routes.*.policies.cache (option 2).maxBytes` | integer | no | minimum: 0; maximum: 16777216 |
174
+ | `routes.*.policies.cache (option 2).maxEntries` | integer | no | minimum: 1; maximum: 1000000 |
175
+ | `routes.*.policies.cache (option 2).force` | boolean | no | default: false |
176
+ | `includes` | array | no | maxItems: 256; uniqueItems: true |
177
+ | `includes[]` | string | no | maxLength: 1024 |
178
+ | `dynamicLinks` | boolean | no | default: false |
179
+ | `policies` | object | no | unknown keys rejected |
180
+ | `policies.profile` | string | no | pattern: "^[a-z][a-z0-9-]{0,63}$" |
181
+ | `policies.throttle` | one of the shapes below | no | — |
182
+ | `policies.throttle (option 1)` | constant | no | const: false |
183
+ | `policies.throttle (option 2)` | object | no | unknown keys rejected |
184
+ | `policies.throttle (option 2).quota` | integer | no | minimum: 1; maximum: 1000000 |
185
+ | `policies.throttle (option 2).window` | integer | no | minimum: 1; maximum: 86400 |
186
+ | `policies.throttle (option 2).partition` | string | no | enum: ["client","route","client-route"]; default: "client" |
187
+ | `policies.throttle (option 2).status` | integer | no | default: 429; minimum: 400; maximum: 599 |
188
+ | `policies.throttle (option 2).mode` | string | no | enum: ["enforce","report"]; default: "enforce" |
189
+ | `policies.throttle (option 2).maxKeys` | integer | no | default: 100000; minimum: 1000; maximum: 10000000 |
190
+ | `policies.agents` | one of the shapes below | no | — |
191
+ | `policies.agents (option 1)` | constant | no | const: false |
192
+ | `policies.agents (option 2)` | object | no | unknown keys rejected |
193
+ | `policies.agents (option 2).deny` | array | no | maxItems: 32; uniqueItems: true |
194
+ | `policies.agents (option 2).deny[]` | string | no | minLength: 1; maxLength: 1024 |
195
+ | `policies.agents (option 2).allow` | array | no | maxItems: 32; uniqueItems: true |
196
+ | `policies.agents (option 2).allow[]` | string | no | minLength: 1; maxLength: 1024 |
197
+ | `policies.agents (option 2).denyPatterns` | array | no | maxItems: 256; uniqueItems: true |
198
+ | `policies.agents (option 2).denyPatterns[]` | string | no | minLength: 1; maxLength: 256 |
199
+ | `policies.agents (option 2).allowPatterns` | array | no | maxItems: 256; uniqueItems: true |
200
+ | `policies.agents (option 2).allowPatterns[]` | string | no | minLength: 1; maxLength: 256 |
201
+ | `policies.agents (option 2).denyEmpty` | boolean | no | default: false |
202
+ | `policies.agents (option 2).status` | integer | no | default: 403; minimum: 400; maximum: 599 |
203
+ | `policies.agents (option 2).mode` | string | no | enum: ["enforce","report"]; default: "enforce" |
204
+ | `policies.security` | one of the shapes below | no | — |
205
+ | `policies.security (option 1)` | constant | no | const: false |
206
+ | `policies.security (option 2)` | object | no | unknown keys rejected |
207
+ | `policies.security (option 2).headers` | string | no | enum: ["oshp","oshp-no-csp","off"]; default: "oshp" |
208
+ | `policies.security (option 2).set` | object | no | maxProperties: 32 |
209
+ | `policies.security (option 2).set.*` | string | no | maxLength: 4096 |
210
+ | `policies.security (option 2).unset` | array | no | maxItems: 32; uniqueItems: true |
211
+ | `policies.security (option 2).unset[]` | string | no | minLength: 1; maxLength: 128 |
212
+ | `policies.compression` | one of the shapes below | no | — |
213
+ | `policies.compression (option 1)` | constant | no | const: false |
214
+ | `policies.compression (option 2)` | object | no | unknown keys rejected |
215
+ | `policies.compression (option 2).encodings` | array | no | default: ["br","gzip"]; minItems: 1; uniqueItems: true |
216
+ | `policies.compression (option 2).encodings[]` | string | no | enum: ["br","gzip","deflate","zstd"] |
217
+ | `policies.compression (option 2).minBytes` | integer | no | default: 1024; minimum: 0; maximum: 1048576 |
218
+ | `policies.compression (option 2).types` | array | no | maxItems: 64; uniqueItems: true |
219
+ | `policies.compression (option 2).types[]` | string | no | minLength: 1; maxLength: 128 |
220
+ | `policies.compression (option 2).level` | integer | no | minimum: 1; maximum: 11 |
221
+ | `policies.compression (option 2).allowWithSecrets` | boolean | no | default: false |
222
+ | `policies.cache` | one of the shapes below | no | — |
223
+ | `policies.cache (option 1)` | constant | no | const: false |
224
+ | `policies.cache (option 2)` | object | no | unknown keys rejected |
225
+ | `policies.cache (option 2).strategy` | string | no | enum: ["no-store","revalidate","public","immutable","swr","sie","micro","cdn-only","private"] |
226
+ | `policies.cache (option 2).maxAge` | integer | no | minimum: 0; maximum: 31536000 |
227
+ | `policies.cache (option 2).staleWhileRevalidate` | integer | no | minimum: 0; maximum: 31536000 |
228
+ | `policies.cache (option 2).staleIfError` | integer | no | minimum: 0; maximum: 31536000 |
229
+ | `policies.cache (option 2).cdnMaxAge` | integer | no | minimum: 0; maximum: 31536000 |
230
+ | `policies.cache (option 2).originTtl` | integer | no | minimum: 0; maximum: 86400 |
231
+ | `policies.cache (option 2).vary` | array | no | maxItems: 8; uniqueItems: true |
232
+ | `policies.cache (option 2).vary[]` | string | no | minLength: 1; maxLength: 128 |
233
+ | `policies.cache (option 2).statuses` | array | no | maxItems: 16; uniqueItems: true |
234
+ | `policies.cache (option 2).statuses[]` | integer | no | minimum: 200; maximum: 599 |
235
+ | `policies.cache (option 2).maxBytes` | integer | no | minimum: 0; maximum: 16777216 |
236
+ | `policies.cache (option 2).maxEntries` | integer | no | minimum: 1; maximum: 1000000 |
237
+ | `policies.cache (option 2).force` | boolean | no | default: false |
238
+ | `profiles` | object | no | maxProperties: 32 |
239
+ | `profiles.*` | object | no | unknown keys rejected |
240
+ | `profiles.*.throttle` | one of the shapes below | no | — |
241
+ | `profiles.*.throttle (option 1)` | constant | no | const: false |
242
+ | `profiles.*.throttle (option 2)` | object | no | unknown keys rejected |
243
+ | `profiles.*.throttle (option 2).quota` | integer | no | minimum: 1; maximum: 1000000 |
244
+ | `profiles.*.throttle (option 2).window` | integer | no | minimum: 1; maximum: 86400 |
245
+ | `profiles.*.throttle (option 2).partition` | string | no | enum: ["client","route","client-route"]; default: "client" |
246
+ | `profiles.*.throttle (option 2).status` | integer | no | default: 429; minimum: 400; maximum: 599 |
247
+ | `profiles.*.throttle (option 2).mode` | string | no | enum: ["enforce","report"]; default: "enforce" |
248
+ | `profiles.*.throttle (option 2).maxKeys` | integer | no | default: 100000; minimum: 1000; maximum: 10000000 |
249
+ | `profiles.*.agents` | one of the shapes below | no | — |
250
+ | `profiles.*.agents (option 1)` | constant | no | const: false |
251
+ | `profiles.*.agents (option 2)` | object | no | unknown keys rejected |
252
+ | `profiles.*.agents (option 2).deny` | array | no | maxItems: 32; uniqueItems: true |
253
+ | `profiles.*.agents (option 2).deny[]` | string | no | minLength: 1; maxLength: 1024 |
254
+ | `profiles.*.agents (option 2).allow` | array | no | maxItems: 32; uniqueItems: true |
255
+ | `profiles.*.agents (option 2).allow[]` | string | no | minLength: 1; maxLength: 1024 |
256
+ | `profiles.*.agents (option 2).denyPatterns` | array | no | maxItems: 256; uniqueItems: true |
257
+ | `profiles.*.agents (option 2).denyPatterns[]` | string | no | minLength: 1; maxLength: 256 |
258
+ | `profiles.*.agents (option 2).allowPatterns` | array | no | maxItems: 256; uniqueItems: true |
259
+ | `profiles.*.agents (option 2).allowPatterns[]` | string | no | minLength: 1; maxLength: 256 |
260
+ | `profiles.*.agents (option 2).denyEmpty` | boolean | no | default: false |
261
+ | `profiles.*.agents (option 2).status` | integer | no | default: 403; minimum: 400; maximum: 599 |
262
+ | `profiles.*.agents (option 2).mode` | string | no | enum: ["enforce","report"]; default: "enforce" |
263
+ | `profiles.*.security` | one of the shapes below | no | — |
264
+ | `profiles.*.security (option 1)` | constant | no | const: false |
265
+ | `profiles.*.security (option 2)` | object | no | unknown keys rejected |
266
+ | `profiles.*.security (option 2).headers` | string | no | enum: ["oshp","oshp-no-csp","off"]; default: "oshp" |
267
+ | `profiles.*.security (option 2).set` | object | no | maxProperties: 32 |
268
+ | `profiles.*.security (option 2).set.*` | string | no | maxLength: 4096 |
269
+ | `profiles.*.security (option 2).unset` | array | no | maxItems: 32; uniqueItems: true |
270
+ | `profiles.*.security (option 2).unset[]` | string | no | minLength: 1; maxLength: 128 |
271
+ | `profiles.*.compression` | one of the shapes below | no | — |
272
+ | `profiles.*.compression (option 1)` | constant | no | const: false |
273
+ | `profiles.*.compression (option 2)` | object | no | unknown keys rejected |
274
+ | `profiles.*.compression (option 2).encodings` | array | no | default: ["br","gzip"]; minItems: 1; uniqueItems: true |
275
+ | `profiles.*.compression (option 2).encodings[]` | string | no | enum: ["br","gzip","deflate","zstd"] |
276
+ | `profiles.*.compression (option 2).minBytes` | integer | no | default: 1024; minimum: 0; maximum: 1048576 |
277
+ | `profiles.*.compression (option 2).types` | array | no | maxItems: 64; uniqueItems: true |
278
+ | `profiles.*.compression (option 2).types[]` | string | no | minLength: 1; maxLength: 128 |
279
+ | `profiles.*.compression (option 2).level` | integer | no | minimum: 1; maximum: 11 |
280
+ | `profiles.*.compression (option 2).allowWithSecrets` | boolean | no | default: false |
281
+ | `profiles.*.cache` | one of the shapes below | no | — |
282
+ | `profiles.*.cache (option 1)` | constant | no | const: false |
283
+ | `profiles.*.cache (option 2)` | object | no | unknown keys rejected |
284
+ | `profiles.*.cache (option 2).strategy` | string | no | enum: ["no-store","revalidate","public","immutable","swr","sie","micro","cdn-only","private"] |
285
+ | `profiles.*.cache (option 2).maxAge` | integer | no | minimum: 0; maximum: 31536000 |
286
+ | `profiles.*.cache (option 2).staleWhileRevalidate` | integer | no | minimum: 0; maximum: 31536000 |
287
+ | `profiles.*.cache (option 2).staleIfError` | integer | no | minimum: 0; maximum: 31536000 |
288
+ | `profiles.*.cache (option 2).cdnMaxAge` | integer | no | minimum: 0; maximum: 31536000 |
289
+ | `profiles.*.cache (option 2).originTtl` | integer | no | minimum: 0; maximum: 86400 |
290
+ | `profiles.*.cache (option 2).vary` | array | no | maxItems: 8; uniqueItems: true |
291
+ | `profiles.*.cache (option 2).vary[]` | string | no | minLength: 1; maxLength: 128 |
292
+ | `profiles.*.cache (option 2).statuses` | array | no | maxItems: 16; uniqueItems: true |
293
+ | `profiles.*.cache (option 2).statuses[]` | integer | no | minimum: 200; maximum: 599 |
294
+ | `profiles.*.cache (option 2).maxBytes` | integer | no | minimum: 0; maximum: 16777216 |
295
+ | `profiles.*.cache (option 2).maxEntries` | integer | no | minimum: 1; maximum: 1000000 |
296
+ | `profiles.*.cache (option 2).force` | boolean | no | default: false |
297
+ | `site` | object | no | unknown keys rejected |
298
+ | `site.robots` | object | no | unknown keys rejected |
299
+ | `site.robots.disallow` | array | no | maxItems: 1024; uniqueItems: true |
300
+ | `site.robots.disallow[]` | string | no | minLength: 1; maxLength: 2048 |
301
+ | `site.robots.allow` | array | no | maxItems: 1024; uniqueItems: true |
302
+ | `site.robots.allow[]` | string | no | minLength: 1; maxLength: 2048 |
303
+ | `site.robots.sitemap` | boolean | no | — |
304
+ | `site.robots.extra` | array | no | maxItems: 1024 |
305
+ | `site.robots.extra[]` | string | no | maxLength: 2048 |
306
+ | `site.sitemap` | one of the shapes below | no | — |
307
+ | `site.sitemap (option 1)` | constant | no | const: true |
308
+ | `site.sitemap (option 2)` | object | no | unknown keys rejected |
309
+ | `site.sitemap (option 2).exclude` | array | no | maxItems: 1024; uniqueItems: true |
310
+ | `site.sitemap (option 2).exclude[]` | string | no | minLength: 1; maxLength: 2048 |
311
+ | `site.sitemap (option 2).changefreq` | string | no | enum: ["always","hourly","daily","weekly","monthly","yearly","never"] |
312
+ | `site.sitemap (option 2).priority` | number | no | minimum: 0; maximum: 1 |
313
+ | `site.favicon` | string | no | minLength: 1; maxLength: 1024 |
314
+ | `site.securityTxt` | object | no | unknown keys rejected |
315
+ | `site.securityTxt.contact` | array | yes | minItems: 1; maxItems: 64 |
316
+ | `site.securityTxt.contact[]` | string | no | minLength: 1; maxLength: 2048 |
317
+ | `site.securityTxt.expires` | string | yes | pattern: "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(?:\\.\\d{3})?Z$" |
318
+ | `site.securityTxt.policy` | array | no | maxItems: 64 |
319
+ | `site.securityTxt.policy[]` | string | no | maxLength: 2048; pattern: "^https://" |
320
+ | `site.securityTxt.acknowledgments` | array | no | maxItems: 64 |
321
+ | `site.securityTxt.acknowledgments[]` | string | no | maxLength: 2048; pattern: "^https://" |
322
+ | `site.securityTxt.preferredLanguages` | array | no | minItems: 1; maxItems: 64 |
323
+ | `site.securityTxt.preferredLanguages[]` | string | no | minLength: 2; maxLength: 35 |
324
+ | `site.securityTxt.canonical` | array | no | maxItems: 64 |
325
+ | `site.securityTxt.canonical[]` | string | no | maxLength: 2048; pattern: "^https://" |
326
+ | `site.securityTxt.encryption` | array | no | maxItems: 64 |
327
+ | `site.securityTxt.encryption[]` | string | no | minLength: 1; maxLength: 2048 |
328
+ | `site.llms` | string | no | minLength: 1; maxLength: 1024 |
@@ -0,0 +1,182 @@
1
+ # The `agents` policy
2
+
3
+ Denies or reports requests by their `User-Agent`, before anything else runs.
4
+ It is the cheapest refusal in the request chain (`agents`, then `throttle`,
5
+ then the cache lookup), so a denied crawler never counts against a quota,
6
+ never reaches the sandbox and never fills the origin cache.
7
+
8
+ ```yaml
9
+ version: "1"
10
+ policies: # project defaults, or per route under routes.<pattern>.policies
11
+ agents:
12
+ deny: [ai-crawlers] # bundled list names, or project-relative .json files
13
+ allow: [monitoring] # allow always wins over deny
14
+ denyPatterns: ["^curl/"] # linear-time regex subset, matched case-insensitively
15
+ allowPatterns: ["^Mozilla/5\\.0 \\(compatible; Googlebot"]
16
+ denyEmpty: false # deny a missing or blank User-Agent
17
+ status: 403 # 400-599
18
+ mode: enforce # enforce | report
19
+ ```
20
+
21
+ The `hardened` profile sets `deny: [ai-crawlers]` with status 403. A route can
22
+ override any key, or set `agents: false` to switch the policy off for itself.
23
+
24
+ ## Semantics
25
+
26
+ - Matching is against the `User-Agent` request header only and is
27
+ case-insensitive. Nothing else about the request (address, path, other
28
+ headers) takes part.
29
+ - Evaluation order: if any `allow` list or `allowPatterns` entry matches, the
30
+ request passes and nothing is logged. Otherwise the first `deny` list (in
31
+ the order written) or `denyPatterns` entry that matches denies it.
32
+ - `urlcode audit`, `test` and `benchmark` send generated probes as
33
+ `Mozilla/5.0 (compatible; RouteProbe/0.1)`;
34
+ a fixture may set its own `user-agent` header. Upstream lists include
35
+ short unanchored names (the `ai-crawlers` list carries `Code`, which
36
+ matches any agent containing that word), so a custom allow pattern for
37
+ your own tooling is worth declaring before denying a whole list.
38
+ - `denyEmpty: true` denies a request with no `User-Agent`, or one that is only
39
+ whitespace. Allow rules cannot match an empty header, so this always wins
40
+ for empty headers.
41
+ - A denial answers `status` (default 403) with `content-type:
42
+ text/plain; charset=utf-8`, `cache-control: no-store` and the body
43
+ `Forbidden\n`. Response policies that also run at request time (cache,
44
+ compression) are skipped on the early response; `security` headers are
45
+ still applied.
46
+ - Every denial is logged as `{ event: 'agents', route, list, outcome }`, where
47
+ `list` is the list name, the file path as written in YAML, `pattern` for
48
+ `denyPatterns`, or `empty` for `denyEmpty`. The raw header value is never
49
+ logged: a `User-Agent` is attacker-controlled text.
50
+ - `mode: report` logs the same event with `outcome: 'reported'` and never
51
+ denies. Run a new deny list in report mode for a release, read the log, then
52
+ switch to `enforce`.
53
+
54
+ ## Allow before deny
55
+
56
+ Broad deny lists without an explicit allow are the most common self-inflicted
57
+ outage in this space. `crawlers` contains every search engine; `ai-crawlers`
58
+ contains agents some operators want (for example `ChatGPT-User` or
59
+ `Applebot-Extended` when a site chooses to appear in AI search). Keep an
60
+ explicit `allow` or `allowPatterns` entry for the agents you depend on, and
61
+ anchor it: `^Mozilla/5\.0 \(compatible; Googlebot` cannot be satisfied by a
62
+ `Googlebot` token dropped in the middle of an unrelated string.
63
+
64
+ Matching is string matching. A client can claim any `User-Agent`, and the
65
+ genuine search crawlers publish the way to check a claim (reverse DNS for
66
+ Googlebot, bingbot and Applebot; the IETF `web-bot-auth` HTTP Message
67
+ Signature drafts for newer agents). That verification needs network calls and
68
+ vendor data, so it belongs in a [host plugin](../SPIKE-EXTENSIONS.md) that
69
+ runs after this policy, not in the runtime.
70
+
71
+ ## Bundled lists
72
+
73
+ | Name | Contents | Upstream | Licence | Refresh |
74
+ | --- | --- | --- | --- | --- |
75
+ | `ai-crawlers` | every agent in `robots.json` (AI training, AI search and assistant crawlers) | [ai-robots-txt/ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) | MIT | pinned tag in `scripts/sync-agent-lists.ts` |
76
+ | `crawlers` | every crawler, bot and automated client in `crawler-user-agents.json` | [monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents) | MIT (revisions after 2016-11-07 only) | same |
77
+ | `seo` | `crawlers` entries tagged `seo` (backlink and rank trackers such as AhrefsBot, SemrushBot, MJ12bot) | derived from crawler-user-agents | MIT | same |
78
+ | `monitoring` | `crawlers` entries tagged `monitoring` (UptimeRobot, Pingdom, StatusCake, ...) | derived from crawler-user-agents | MIT | same |
79
+
80
+ Each list lives in `data/agents/<name>.json` as
81
+ `{ name, description, source: { repository, url, license, tag, commit, file, fetchedAt }, entries }`
82
+ with one entry per pattern: `{ name, pattern, source, sourceRevision, addedAt }`.
83
+ The generated `data/agents/index.js` mirrors the `[name, pattern]` pairs so the
84
+ policy module (which also runs inside the Cloudflare Worker) needs no
85
+ filesystem. The upstream licences are reproduced verbatim under
86
+ `data/agents/LICENSES/` and named in `NOTICE`, as Apache-2.0 section 4(d)
87
+ requires. `urlcode audit` and the runtime's policy inventory report the list
88
+ names, pattern counts and the pinned revision each list was built from, so a
89
+ deploy carries a known list version and a rollback rolls the list back too.
90
+
91
+ ### Refreshing
92
+
93
+ ```sh
94
+ node scripts/sync-agent-lists.ts # fetch pinned upstreams, validate, write data/agents/
95
+ node scripts/sync-agent-lists.ts --check # exit 1 when the committed files are stale
96
+ ```
97
+
98
+ The script fetches each upstream at the tag and commit pinned in its `sources`
99
+ table, normalises entries to the schema above (an ai.robots.txt agent name
100
+ becomes an escaped literal pattern; a crawler-user-agents pattern is kept and
101
+ named by its literal prefix), validates every pattern against the subset below
102
+ (rewriting `{n,}` to `{n,64}`, and dropping and printing anything else),
103
+ preserves `addedAt` from the previous file or the upstream `addition_date`, and
104
+ writes the JSON lists, the index and the licence copies. To move to a newer
105
+ upstream release, change the `tag` and `commit` pins and rerun; the revision
106
+ appears in every entry and in the audit output. Refresh through an ordinary
107
+ pull request so the list diff is reviewed like code. Behind an HTTPS proxy,
108
+ set `NODE_USE_ENV_PROXY=1` so `fetch` honours `HTTPS_PROXY` and
109
+ `NODE_EXTRA_CA_CERTS`.
110
+
111
+ ## Project lists
112
+
113
+ `deny` and `allow` also accept a project-relative path ending in `.json`:
114
+
115
+ ```yaml
116
+ policies:
117
+ agents:
118
+ deny: [agents/deny.json]
119
+ ```
120
+
121
+ The file is either an array of entries or `{ "entries": [...] }`, each entry
122
+ `{ "name": "curl", "pattern": "^curl/" }` (`name` optional; the other fields
123
+ of the bundled schema are ignored). It must stay inside the project, hold at
124
+ most 4096 entries and pass the same pattern validation as YAML patterns. The
125
+ log names the path as written. The file is read once at activation; the
126
+ Cloudflare build embeds its entries in the artifact so the Worker never reads a
127
+ file.
128
+
129
+ ## The pattern subset
130
+
131
+ Patterns in `denyPatterns`, `allowPatterns` and every list file are validated
132
+ at activation. A pattern outside the subset fails activation with a
133
+ `ConfigError` naming the route, the key and the reason, so a project cannot
134
+ turn the matcher into a denial-of-service vector by editing YAML. Allowed:
135
+
136
+ - anchors `^` and `$`; literals; `.`
137
+ - escapes: `\d \w \s \D \W \S \b \B \t \n \r \f \v \0`, `\xHH`, `\uHHHH`, and
138
+ a backslash before any punctuation (`\.`, `\/`, `\(`, `\-`, ...)
139
+ - character classes `[...]` and `[^...]` with ranges and the escapes above
140
+ - groups `(...)` and `(?:...)`, and alternation `|`
141
+ - quantifiers `*`, `+`, `?`, `{n}` and `{n,m}` with `m <= 64`, on a single
142
+ atom (a literal, escape, class or `.`); `?` may also follow a group that
143
+ contains no quantifier, for optional words such as `(?:bot)?`
144
+ - at most 256 bytes
145
+
146
+ Rejected: backreferences (`\1`, `\k<name>`), lookahead and lookbehind, named
147
+ groups, unicode property escapes, `\c` control escapes, nested character
148
+ classes, `{n,}` and bounds above 64, lazy or stacked quantifiers (`+?`, `**`),
149
+ quantifiers on anchors, and `*`, `+` or `{n,m}` on a group (so `(a+)+` and
150
+ `(ab)*` fail). Each list is compiled into one alternated `RegExp` with the `i`
151
+ flag, so a request costs one pass per list rather than one per pattern.
152
+
153
+ ## robots.txt
154
+
155
+ Denying an agent is not the same as asking it to stay away. Well-behaved
156
+ crawlers read `/robots.txt` (RFC 9309) before fetching anything, and the AI
157
+ crawlers in `ai-crawlers` are the ones that upstream tracks as respecting or
158
+ ignoring it. Generate one from the same bundled list with
159
+ [`site.robots`](../SITE.md) so the file and the policy cannot drift apart:
160
+
161
+ ```yaml
162
+ site:
163
+ robots:
164
+ disallow: [ai-crawlers]
165
+ policies:
166
+ agents:
167
+ deny: [ai-crawlers]
168
+ ```
169
+
170
+ This serves `/robots.txt` with one `User-agent:` line per list entry followed
171
+ by `Disallow: /`, then `User-agent: *` / `Allow: /`. A hand-written
172
+ `/robots.txt` `respond` route still works and takes precedence over the
173
+ generated one. The `agents` policy then enforces the same decision for clients
174
+ that ignore the file.
175
+
176
+ ## Targets
177
+
178
+ | Target | Support |
179
+ | --- | --- |
180
+ | self-hosted (`urlcode serve`) | native |
181
+ | Vercel, AWS | native |
182
+ | Cloudflare | compiled: the build validates the policy, embeds the effective configuration (and any project list entries) in the artifact, and the Worker compiles it at startup without filesystem access |
@@ -0,0 +1,139 @@
1
+ # `policies.cache`
2
+
3
+ Named HTTP caching strategies, enforced on the host. A route picks one
4
+ strategy from a fixed catalogue; every row is a pattern with a name outside
5
+ this project and a defined header output, so a browser, a CDN and the
6
+ runtime's own origin memory cache all read the same thing. Explicit fields
7
+ (`maxAge`, `staleWhileRevalidate`, `staleIfError`, `cdnMaxAge`, `originTtl`)
8
+ override what a strategy implies.
9
+
10
+ ```yaml
11
+ policies:
12
+ cache: { strategy: revalidate } # project default
13
+ routes:
14
+ /feed:
15
+ function: { source: feed.mjs }
16
+ policies:
17
+ cache: { strategy: swr, maxAge: 30, staleWhileRevalidate: 300, vary: [Accept-Language] }
18
+ ```
19
+
20
+ ## Strategies
21
+
22
+ | `strategy` | Emitted headers | Origin memory cache | Typical use |
23
+ |---|---|---|---|
24
+ | `no-store` | `Cache-Control: no-store` | off | personalized, secret-bearing |
25
+ | `revalidate` | `Cache-Control: no-cache`, `ETag` (a strong SHA-256 tag is computed for results without one); answers `304` to `If-None-Match`, and to `If-Modified-Since` when the result carries `Last-Modified` | off | HTML, anything that must be fresh but is cheap to validate |
26
+ | `public` | `Cache-Control: public, max-age=N` (`maxAge` required) | on only when `originTtl` > 0 | stable API answers, feeds |
27
+ | `immutable` | `Cache-Control: public, max-age=31536000, immutable` (RFC 8246; `maxAge` overrides the year) | off | content-hashed URLs only; refused elsewhere unless `force: true` |
28
+ | `swr` | `Cache-Control: public, max-age=N, stale-while-revalidate=M` (RFC 5861; both required) | on: fresh for `originTtl` (default `maxAge`), then stale served once | hot functions, link previews |
29
+ | `sie` | as `swr` plus `stale-if-error=K` (`staleIfError` required; `staleWhileRevalidate` optional) | on, as `swr` | keep answering during an upstream failure (headers only at the origin; see below) |
30
+ | `micro` | `Cache-Control: no-store` to clients | on: `originTtl` default 1 s, at most 5 s unless `force: true` | the NGINX micro-cache: absorb a thundering herd without changing what a browser sees |
31
+ | `cdn-only` | `Cache-Control: no-store` plus `CDN-Cache-Control: max-age=N` (RFC 9213; `cdnMaxAge` required) | off | let the CDN cache while browsers do not |
32
+ | `private` | `Cache-Control: private, max-age=N` (`maxAge` required) | off | per-user data a browser may keep |
33
+
34
+ `immutable` accepts a route whose pattern has a segment with eight or more
35
+ hex characters (`/app.3f2a9c1d.js`) or a parameter named like a digest
36
+ (`{hash}`, `{digest}`, `{sha}`, `{version}`, `{build}`, `{rev}`,
37
+ `{fingerprint}`). Anything else fails configuration with the route named
38
+ unless `force: true`.
39
+
40
+ ## Who owns `Cache-Control`
41
+
42
+ Explicit beats strategy, in this order:
43
+
44
+ 1. A `response.headers.cache-control` declared in YAML on the route is kept
45
+ as written; the strategy does not touch it.
46
+ 2. An asset handler's `cacheControl` (`page`, `download`, `static`) is kept
47
+ when the cache policy is only inherited from the project or a profile.
48
+ When the route itself declares `policies.cache`, the strategy overrides it.
49
+ 3. A handler result that already says `no-store` or `private` is kept and is
50
+ never stored: a personalized answer under a `public` route stays private.
51
+ 4. Otherwise the strategy's header replaces whatever the handler emitted.
52
+
53
+ `Vary` is merged with the declared `vary` names (no duplicates, `*` left
54
+ alone), for every strategy, so the origin key and the wire header agree.
55
+ Compression adds `Accept-Encoding` to `Vary` after this policy.
56
+
57
+ ## Origin memory cache
58
+
59
+ On for `swr`, `sie`, `micro`, and `public` with `originTtl` set. It is an
60
+ origin-side micro-cache, so a client's own `Cache-Control: no-cache` or
61
+ `max-age=0` request directive does not bypass it (RFC 9111 §5.2.1 lets an
62
+ origin decide that); conditional and range requests do, as below. Rules:
63
+
64
+ - Only `GET` and `HEAD` are looked up; only `GET` results are stored, and a
65
+ `HEAD` hit serves the `GET` entry's headers with an empty body and the
66
+ entry's `Content-Length`. A route whose `methods` exclude `GET` has the
67
+ origin cache off.
68
+ - Only statuses in `statuses` (default `200, 301, 302, 404, 410`) are stored.
69
+ - A request carrying `If-None-Match`, `If-Modified-Since`, `If-Match`,
70
+ `If-Unmodified-Since` or `Range` is never answered from the origin cache:
71
+ a stored entry is a full `200` representation and the handler owns
72
+ validators and ranges, so such requests always reach it (an asset route
73
+ keeps its own `304`, `412` and `206` answers).
74
+ - Never stored: results carrying `Set-Cookie`; routes declaring `secrets`;
75
+ results whose handler `Cache-Control` says `private` or `no-store`; bodies
76
+ larger than `maxBytes` (default 1 MiB).
77
+ - Key: route pattern, request path, query string and the values of the
78
+ declared `vary` request headers. The method is not part of the key so
79
+ `HEAD` shares the `GET` entry.
80
+ - Hits carry `Age` (RFC 9111). No non-standard headers are added.
81
+ - Concurrent misses for one key coalesce: the first request reaches the
82
+ handler, up to 64 others wait for its result, and any beyond that proceed
83
+ to the handler themselves. If the fill fails, waiters fall through to the
84
+ handler rather than receiving the error.
85
+ - Bounds: `maxEntries` per route configuration (default 10000) and 64 MiB of
86
+ bodies across the whole runtime; the least recently used entry is evicted
87
+ first. The store belongs to one runtime instance and is dropped on close and
88
+ reload, so a deploy never serves the previous code's output.
89
+ - Stored entries hold the handler's headers as they were after YAML
90
+ `response.headers` and this policy ran, before security headers and
91
+ compression. Those run again on every hit, so a hit is compressed and
92
+ hardened the same way as a miss. Bodies are stored uncompressed.
93
+
94
+ ### `swr` at the origin: stale served once, next request refreshes
95
+
96
+ A policy has no handle to the route's handler, so it cannot revalidate in the
97
+ background. The origin-side approximation is: a request that finds an entry
98
+ past `originTtl` but within `staleWhileRevalidate` is answered from the stale
99
+ entry immediately, and the entry is flagged so the next request for that key
100
+ goes to the handler and replaces it. If that refresh fails the flag is
101
+ cleared and the stale entry may be served once more, as long as it is within
102
+ the window. Beyond `max-age + stale-while-revalidate` nothing is served from
103
+ memory. Clients and CDNs that honour RFC 5861 do their own background
104
+ revalidation from the emitted header.
105
+
106
+ ### `sie` limitation
107
+
108
+ `stale-if-error` is header-only at the origin. Policies observe errors but
109
+ cannot replace a thrown error with a result (the runtime rethrows), so a
110
+ handler failure is not answered from a stale entry by this runtime. Downstream
111
+ caches honouring RFC 5861 still serve stale on a 5xx. Origin-side
112
+ stale-if-error needs the plugin API or a runtime change that lets a policy's
113
+ error hook return a result.
114
+
115
+ ## Targets
116
+
117
+ | Target | Support | Note |
118
+ |---|---|---|
119
+ | node | native | headers and origin cache per runtime instance |
120
+ | vercel | native | headers; the origin cache is per function instance, so hit rates depend on instance reuse |
121
+ | aws | native | headers; the origin cache is per Lambda instance |
122
+ | cloudflare | refused | the Worker build has no policy runtime; activation fails with the route named |
123
+
124
+ ## Diagnostics
125
+
126
+ `describe()` (in `urlcode test`, the readiness plan and plugin activation)
127
+ reports `strategy`, the emitted `cacheControl` string (or `explicit response
128
+ header` / `asset handler` when something else owns it), `cdnCacheControl`,
129
+ `originTtl`, `staleWhileRevalidate`, `staleIfError`, `vary` and whether the
130
+ origin cache is `on`.
131
+
132
+ Log events, through the runtime's `log`:
133
+
134
+ ```json
135
+ { "event": "cache", "route": "/feed", "outcome": "hit" }
136
+ ```
137
+
138
+ `outcome` is `hit`, `stale`, `miss` or `store`. The route is the configured
139
+ pattern, never the request path or key.