@jimhoyd/urlcode 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (254) hide show
  1. package/CONTRIBUTING.md +80 -0
  2. package/LICENSE +202 -0
  3. package/NOTICE +20 -0
  4. package/README.md +342 -0
  5. package/ROADMAP.md +248 -0
  6. package/SECURITY.md +40 -0
  7. package/data/agents/LICENSES/ai-robots-txt.txt +24 -0
  8. package/data/agents/LICENSES/crawler-user-agents.txt +23 -0
  9. package/data/agents/ai-crawlers.json +1240 -0
  10. package/data/agents/crawlers.json +10515 -0
  11. package/data/agents/index.d.ts +3 -0
  12. package/data/agents/index.js +2453 -0
  13. package/data/agents/monitoring.json +1520 -0
  14. package/data/agents/seo.json +3865 -0
  15. package/dist/BUILD-MANIFEST.json +60 -0
  16. package/dist/adapters.js +59 -0
  17. package/dist/agent-lists.js +59 -0
  18. package/dist/assets.js +129 -0
  19. package/dist/authoring.js +68 -0
  20. package/dist/aws.js +110 -0
  21. package/dist/build-cloudflare.js +164 -0
  22. package/dist/cli.js +185 -0
  23. package/dist/client-address.js +85 -0
  24. package/dist/cloudflare.js +165 -0
  25. package/dist/compliance-rules/baseline.js +98 -0
  26. package/dist/compliance-rules/privacy.js +38 -0
  27. package/dist/compliance-rules/shared.js +62 -0
  28. package/dist/compliance-rules/strict.js +61 -0
  29. package/dist/compliance.js +219 -0
  30. package/dist/config-worker.js +9 -0
  31. package/dist/config.js +172 -0
  32. package/dist/errors.js +8 -0
  33. package/dist/function-sources.js +54 -0
  34. package/dist/function-worker.js +105 -0
  35. package/dist/functions.js +174 -0
  36. package/dist/guest-api.js +114 -0
  37. package/dist/header-validation.js +18 -0
  38. package/dist/http-policy.js +74 -0
  39. package/dist/http-response.js +84 -0
  40. package/dist/index.js +11 -0
  41. package/dist/link-api.js +136 -0
  42. package/dist/link-cli.js +141 -0
  43. package/dist/link-events.js +76 -0
  44. package/dist/link-records.js +31 -0
  45. package/dist/link-store-worker.js +150 -0
  46. package/dist/link-store.js +250 -0
  47. package/dist/logging.js +22 -0
  48. package/dist/management-policy.js +41 -0
  49. package/dist/match.js +124 -0
  50. package/dist/observability.js +242 -0
  51. package/dist/plugins.js +74 -0
  52. package/dist/policies/agents.js +248 -0
  53. package/dist/policies/cache.js +297 -0
  54. package/dist/policies/compression.js +187 -0
  55. package/dist/policies/security.js +0 -0
  56. package/dist/policies/throttle.js +131 -0
  57. package/dist/policies.js +142 -0
  58. package/dist/policy.js +55 -0
  59. package/dist/prerender.js +165 -0
  60. package/dist/project-tests.js +23 -0
  61. package/dist/readiness.js +216 -0
  62. package/dist/router.js +166 -0
  63. package/dist/runtime.js +244 -0
  64. package/dist/scaffold.js +0 -0
  65. package/dist/scripts/operational-drills.js +75 -0
  66. package/dist/server.js +270 -0
  67. package/dist/site.js +0 -0
  68. package/dist/sqlite-version.js +6 -0
  69. package/dist/types/adapters.d.ts +12 -0
  70. package/dist/types/agent-lists.d.ts +5 -0
  71. package/dist/types/assets.d.ts +13 -0
  72. package/dist/types/authoring.d.ts +2 -0
  73. package/dist/types/aws.d.ts +35 -0
  74. package/dist/types/build-cloudflare.d.ts +14 -0
  75. package/dist/types/cli.d.ts +2 -0
  76. package/dist/types/client-address.d.ts +9 -0
  77. package/dist/types/cloudflare.d.ts +75 -0
  78. package/dist/types/compliance-rules/baseline.d.ts +12 -0
  79. package/dist/types/compliance-rules/privacy.d.ts +6 -0
  80. package/dist/types/compliance-rules/shared.d.ts +24 -0
  81. package/dist/types/compliance-rules/strict.d.ts +9 -0
  82. package/dist/types/compliance.d.ts +119 -0
  83. package/dist/types/config-worker.d.ts +1 -0
  84. package/dist/types/config.d.ts +21 -0
  85. package/dist/types/errors.d.ts +7 -0
  86. package/dist/types/function-sources.d.ts +21 -0
  87. package/dist/types/function-worker.d.ts +1 -0
  88. package/dist/types/functions.d.ts +92 -0
  89. package/dist/types/guest-api.d.ts +16 -0
  90. package/dist/types/header-validation.d.ts +2 -0
  91. package/dist/types/http-policy.d.ts +39 -0
  92. package/dist/types/http-response.d.ts +43 -0
  93. package/dist/types/index.d.ts +11 -0
  94. package/dist/types/link-api.d.ts +30 -0
  95. package/dist/types/link-cli.d.ts +36 -0
  96. package/dist/types/link-events.d.ts +27 -0
  97. package/dist/types/link-records.d.ts +11 -0
  98. package/dist/types/link-store-worker.d.ts +1 -0
  99. package/dist/types/link-store.d.ts +130 -0
  100. package/dist/types/logging.d.ts +9 -0
  101. package/dist/types/management-policy.d.ts +9 -0
  102. package/dist/types/match.d.ts +81 -0
  103. package/dist/types/observability.d.ts +104 -0
  104. package/dist/types/plugins.d.ts +27 -0
  105. package/dist/types/policies/agents.d.ts +86 -0
  106. package/dist/types/policies/cache.d.ts +82 -0
  107. package/dist/types/policies/compression.d.ts +41 -0
  108. package/dist/types/policies/security.d.ts +37 -0
  109. package/dist/types/policies/throttle.d.ts +55 -0
  110. package/dist/types/policies.d.ts +36 -0
  111. package/dist/types/policy.d.ts +21 -0
  112. package/dist/types/prerender.d.ts +31 -0
  113. package/dist/types/project-tests.d.ts +13 -0
  114. package/dist/types/readiness.d.ts +121 -0
  115. package/dist/types/router.d.ts +11 -0
  116. package/dist/types/runtime.d.ts +91 -0
  117. package/dist/types/scaffold.d.ts +17 -0
  118. package/dist/types/server.d.ts +37 -0
  119. package/dist/types/site.d.ts +12 -0
  120. package/dist/types/sqlite-version.d.ts +1 -0
  121. package/dist/types/types.d.ts +332 -0
  122. package/dist/types/vercel.d.ts +12 -0
  123. package/dist/types.js +177 -0
  124. package/dist/vercel.js +65 -0
  125. package/docs/AI-AUTHORING.md +113 -0
  126. package/docs/ASSETS.md +106 -0
  127. package/docs/AWS.md +77 -0
  128. package/docs/BEST-PRACTICES.md +267 -0
  129. package/docs/CAPACITY.md +208 -0
  130. package/docs/CLOUDFLARE.md +110 -0
  131. package/docs/COMPLIANCE.md +242 -0
  132. package/docs/DYNAMIC-LINKS.md +561 -0
  133. package/docs/FUNCTION-SECURITY.md +113 -0
  134. package/docs/HTTP.md +129 -0
  135. package/docs/INSTALL.md +90 -0
  136. package/docs/LOAD-TESTING.md +91 -0
  137. package/docs/LOCAL-DEVELOPMENT.md +99 -0
  138. package/docs/MANAGEMENT-SECURITY.md +82 -0
  139. package/docs/MIDDLEWARE.md +83 -0
  140. package/docs/MONITORING.md +132 -0
  141. package/docs/OBSERVABILITY.md +229 -0
  142. package/docs/OPERATIONAL-PROOF.md +42 -0
  143. package/docs/OPERATIONS.md +207 -0
  144. package/docs/ORGANIZATION.md +135 -0
  145. package/docs/PERFORMANCE.md +72 -0
  146. package/docs/PLUGINS.md +234 -0
  147. package/docs/POLICIES.md +489 -0
  148. package/docs/PRERENDER.md +193 -0
  149. package/docs/PROJECT-DIRECTION.md +51 -0
  150. package/docs/READINESS.md +125 -0
  151. package/docs/README.md +58 -0
  152. package/docs/RELEASE-READINESS.md +95 -0
  153. package/docs/RELEASE-SECURITY.md +72 -0
  154. package/docs/RESILIENCE.md +160 -0
  155. package/docs/ROUTING.md +95 -0
  156. package/docs/SANDBOX-REVIEW.md +59 -0
  157. package/docs/SCAFFOLDING.md +72 -0
  158. package/docs/SECURITY-AUDIT.md +124 -0
  159. package/docs/SITE.md +150 -0
  160. package/docs/SPECIFICATION.md +232 -0
  161. package/docs/SPIKE-EXTENSIONS.md +475 -0
  162. package/docs/STANDARDS.md +303 -0
  163. package/docs/STARTERS.md +51 -0
  164. package/docs/TUNNELS.md +75 -0
  165. package/docs/TYPESCRIPT.md +102 -0
  166. package/docs/VERCEL.md +107 -0
  167. package/docs/YAML-GUIDE.md +521 -0
  168. package/docs/YAML-REFERENCE.md +328 -0
  169. package/docs/policies/agents.md +182 -0
  170. package/docs/policies/cache.md +139 -0
  171. package/docs/policies/compression.md +168 -0
  172. package/docs/policies/security.md +160 -0
  173. package/docs/policies/throttle.md +103 -0
  174. package/examples/assets/.env.example +1 -0
  175. package/examples/assets/.gitattributes +2 -0
  176. package/examples/assets/Makefile +30 -0
  177. package/examples/assets/README.md +18 -0
  178. package/examples/assets/functions/hello.mjs +3 -0
  179. package/examples/assets/gitignore.template +7 -0
  180. package/examples/assets/public/about.html +2 -0
  181. package/examples/assets/public/assets/example.txt +1 -0
  182. package/examples/assets/public/guide.txt +1 -0
  183. package/examples/assets/tests/requests.json +44 -0
  184. package/examples/assets/urlcode.yaml +27 -0
  185. package/examples/aws/README.md +13 -0
  186. package/examples/aws/handler.mjs +5 -0
  187. package/examples/aws/public/index.html +3 -0
  188. package/examples/aws/public/notes.txt +1 -0
  189. package/examples/aws/template.yaml +33 -0
  190. package/examples/aws/tests/requests.json +9 -0
  191. package/examples/aws/urlcode.yaml +19 -0
  192. package/examples/cloudflare/README.md +18 -0
  193. package/examples/cloudflare/package.json +14 -0
  194. package/examples/cloudflare/tests/requests.json +9 -0
  195. package/examples/cloudflare/urlcode.yaml +24 -0
  196. package/examples/cloudflare/wrangler.toml +5 -0
  197. package/examples/compliance/README.md +20 -0
  198. package/examples/compliance/rules.mjs +65 -0
  199. package/examples/cookbook/README.md +21 -0
  200. package/examples/cookbook/functions/choice.mjs +4 -0
  201. package/examples/cookbook/functions/echo.mjs +3 -0
  202. package/examples/cookbook/functions/hello.mjs +3 -0
  203. package/examples/cookbook/functions/text.mjs +3 -0
  204. package/examples/cookbook/middleware/headers.mjs +6 -0
  205. package/examples/cookbook/public/about.html +1 -0
  206. package/examples/cookbook/public/assets/index.html +1 -0
  207. package/examples/cookbook/public/assets/site.css +1 -0
  208. package/examples/cookbook/public/favicon.svg +1 -0
  209. package/examples/cookbook/public/guide.txt +1 -0
  210. package/examples/cookbook/public/llms.txt +6 -0
  211. package/examples/cookbook/routes/code.yaml +55 -0
  212. package/examples/cookbook/routes/files.yaml +17 -0
  213. package/examples/cookbook/routes/policies.yaml +28 -0
  214. package/examples/cookbook/routes/redirects.yaml +38 -0
  215. package/examples/cookbook/routes/responses.yaml +26 -0
  216. package/examples/cookbook/tests/requests.json +254 -0
  217. package/examples/cookbook/urlcode.yaml +22 -0
  218. package/examples/live-links/README.md +11 -0
  219. package/examples/live-links/tests/requests.json +6 -0
  220. package/examples/live-links/urlcode.yaml +16 -0
  221. package/examples/monitoring/blackbox-jobs.yaml +37 -0
  222. package/examples/monitoring/prometheus-rules.yaml +88 -0
  223. package/examples/monitoring/prometheus-scrape.yaml +20 -0
  224. package/examples/monitoring/vector.toml +65 -0
  225. package/examples/prerender/README.md +48 -0
  226. package/examples/prerender/functions/page.mjs +6 -0
  227. package/examples/prerender/middleware/template.mjs +29 -0
  228. package/examples/prerender/prerender.d.mts +4 -0
  229. package/examples/prerender/prerender.mjs +63 -0
  230. package/examples/prerender/tests/requests.json +39 -0
  231. package/examples/prerender/urlcode.yaml +50 -0
  232. package/examples/tunnel/dev-with-ngrok.sh +61 -0
  233. package/examples/vercel/README.md +13 -0
  234. package/examples/vercel/api/index.js +5 -0
  235. package/examples/vercel/package.json +8 -0
  236. package/examples/vercel/public/index.html +3 -0
  237. package/examples/vercel/public/notes.txt +1 -0
  238. package/examples/vercel/tests/requests.json +48 -0
  239. package/examples/vercel/urlcode.yaml +19 -0
  240. package/examples/vercel/vercel.json +10 -0
  241. package/llms.txt +46 -0
  242. package/package.json +123 -0
  243. package/schemas/urlcode.schema.json +1230 -0
  244. package/starters/default/.gitattributes +1 -0
  245. package/starters/default/Makefile +30 -0
  246. package/starters/default/README.md +34 -0
  247. package/starters/default/functions/hello.mjs +3 -0
  248. package/starters/default/gitignore.template +8 -0
  249. package/starters/default/middleware/headers.mjs +6 -0
  250. package/starters/default/routes/functions.yaml +20 -0
  251. package/starters/default/routes/marketing/links.yaml +7 -0
  252. package/starters/default/starter.json +5 -0
  253. package/starters/default/tests/requests.json +56 -0
  254. package/starters/default/urlcode.yaml +8 -0
@@ -0,0 +1,489 @@
1
+ # Policies
2
+
3
+ Policies are host-enforced behavior declared in YAML around a route: a
4
+ per-client request budget, a User-Agent deny list, a security-header profile,
5
+ content-coding negotiation and an HTTP caching strategy. They run in the host
6
+ process, outside the sandbox, so they can see the client address, keep counters
7
+ across requests and touch the transport, which route middleware by design
8
+ cannot ([middleware](MIDDLEWARE.md), [function security](FUNCTION-SECURITY.md)).
9
+
10
+ Everything here is optional and off by default. A project with no `policies`
11
+ key and no `profiles` key behaves exactly as before: no policy module is
12
+ compiled and the request path is unchanged. Each policy has its own page:
13
+
14
+ | Policy | Page | What it does |
15
+ |---|---|---|
16
+ | `throttle` | [throttle](policies/throttle.md) | Sliding-window request budget with `RateLimit`/`RateLimit-Policy` headers, 429 and `Retry-After` |
17
+ | `agents` | [agents](policies/agents.md) | Deny or allow by `User-Agent`, from bundled lists, project files and bounded patterns |
18
+ | `security` | [security](policies/security.md) | OWASP Secure Headers Project response headers, with per-header `set`/`unset` |
19
+ | `compression` | [compression](policies/compression.md) | `Accept-Encoding` negotiation; assets precompressed at snapshot time |
20
+ | `cache` | [cache](policies/cache.md) | Named strategies from RFC 9111/5861/8246/9213 plus an origin memory cache |
21
+
22
+ The design and the reasoning behind each choice are in the
23
+ [extensions spike](SPIKE-EXTENSIONS.md). Operators who need behavior the
24
+ declarative block cannot express pass host [plugins](PLUGINS.md) in code; plugins
25
+ are never named in YAML.
26
+
27
+ ## Declaring policies
28
+
29
+ ```yaml
30
+ version: "1"
31
+ policies: # project defaults; every key optional
32
+ profile: hardened # built-in, or a name under `profiles`
33
+ throttle: { quota: 60, window: 60 }
34
+ routes:
35
+ /api/lookup/{id}:
36
+ function: { source: functions/lookup.mjs }
37
+ policies:
38
+ throttle: { quota: 10, window: 60 }
39
+ cache: false
40
+ ```
41
+
42
+ `policies` at the top level sets project defaults; `routes.<path>.policies`
43
+ adjusts them for one route. Both accept the same keys: `profile` plus one entry
44
+ per policy, each either an object or `false`. Unknown keys fail validation, as
45
+ everywhere in the project format. The
46
+ [field reference](YAML-REFERENCE.md) lists every accepted field with its
47
+ bounds; the [JSON Schema](../schemas/urlcode.schema.json) is the source.
48
+
49
+ Values are behavior, never infrastructure: a quota is a number a proxy or CDN
50
+ can restate, while trusted proxy ranges, storage URLs and vendor rule
51
+ identifiers stay in operator configuration. That is what keeps a project
52
+ portable between a laptop, a container and an adapter.
53
+
54
+ ## Where policies run
55
+
56
+ Policies run once the route is known and before its contract is checked. A
57
+ denied agent or an exhausted budget is answered without reading a body or
58
+ starting the sandbox. The self-hosted pipeline:
59
+
60
+ ```
61
+ socket limits → admission → body read
62
+ → runtime.handle(): match → enabled/expires
63
+ → plugins onRequest (outermost, first plugin first)
64
+ → agents → throttle → cache lookup (request phase; may short-circuit)
65
+ → methods (405) → request.body checks
66
+ → native handler | guest chain
67
+ → YAML response.headers
68
+ → cache store → throttle headers → security → compression (response phase)
69
+ → plugins onResponse (reverse order, first plugin last)
70
+ → writeResponse
71
+ ```
72
+
73
+ Request order is fixed: agents first because a denial is the cheapest
74
+ outcome, then throttle, then the cache lookup, so a denied or refused request
75
+ neither counts against a budget nor touches the cache. Response order is the
76
+ cache store, then the throttle's `RateLimit` headers (after the store, so a
77
+ cached copy is never stamped with one client's remaining budget), then
78
+ security headers, then compression last so every header it depends on is
79
+ already final. YAML `response.headers` are applied by the runtime before the
80
+ response phase, so an explicit header always beats a profile default.
81
+
82
+ A result produced by a request-phase policy (an agent denial, a throttle
83
+ refusal, a cache hit) skips that policy's own response hook and passes through
84
+ the others: a hit is not stored twice but still carries the client's
85
+ rate-limit headers, and a denial is not stored because its status is not
86
+ cacheable. A throttle refusal carries its own `RateLimit` and `Retry-After`
87
+ headers. A plugin short-circuit ran before any policy, so it skips the
88
+ response hook of every policy that has a request phase; security headers and
89
+ compression still apply to it.
90
+
91
+ Two responses bypass the response phase entirely: the `405` the runtime returns
92
+ for an undeclared method, and any error the runtime throws (404 for no match,
93
+ 410 for an expired route, 413/415 from body checks, sandbox 502/504). Those
94
+ receive the runtime's standard headers only. Policies with an `onError` hook
95
+ and plugin `onError` hooks observe thrown errors; they cannot change them.
96
+
97
+ Adapters call `handle()` directly and delegate socket limits and admission to
98
+ the provider; the policy order inside `handle()` is the same. The Cloudflare
99
+ Worker runs the same request and response order with the two policies it can
100
+ carry.
101
+
102
+ ## Portability and the per-target table
103
+
104
+ A project is portable when a second person can run it elsewhere and get the
105
+ same declared behavior or an explicit refusal. A target that cannot honor a
106
+ policy refuses activation naming the route and the policy, exactly as adapters
107
+ refuse functions and middleware; nothing degrades silently. One exception is
108
+ stated rather than hidden: a policy the platform itself already provides is
109
+ **delegated**, meaning accepted and dropped, and reported as such in the
110
+ inventory, so one YAML can serve a Node host and a serverless host without
111
+ edits. The self-hosted
112
+ message reads `/path declares policies.throttle, which the vercel target
113
+ cannot enforce`; the Cloudflare build reports
114
+ `/path: policies.throttle cannot be compiled for this target`.
115
+
116
+ | Policy | Self-hosted (`node`) | Vercel / AWS | Cloudflare build |
117
+ |---|---|---|---|
118
+ | `agents` | native | native | compiled into the artifact |
119
+ | `security` | native | native | compiled into the artifact |
120
+ | `throttle` | native, in-process counters | native only with `partition: route`; `client` and `client-route` refused because a client fans across instances and the budget would silently be quota × instances | refused |
121
+ | `compression` | native | delegated: the provider compresses | delegated |
122
+ | `cache` | native: headers plus origin memory cache | native | refused |
123
+
124
+ "Compiled" means the effective configuration for every route is validated at
125
+ build time and carried in the Worker artifact; the Worker has no filesystem and
126
+ no Node imports, so only modules free of both qualify. The Worker's client
127
+ identity is the platform's `cf-connecting-ip`; the serverless adapters use the
128
+ platform-set source address (`sourceIp` on Lambda, the leftmost
129
+ `X-Forwarded-For` entry the platform writes on Vercel). None of these read a
130
+ forwarded header a client could have set.
131
+
132
+ The cross-request state a policy keeps (throttle counters, the origin cache) is
133
+ per runtime instance on every target, never shared between replicas or
134
+ serverless instances. Given identical request bytes and project, every target
135
+ answers with the same status and headers; only that state may differ.
136
+
137
+ ## Merge semantics
138
+
139
+ The effective configuration for a route is built from four layers, each
140
+ merged over the one below:
141
+
142
+ 1. the project `profile` (built-in or custom),
143
+ 2. the project `policies` keys,
144
+ 3. the route `profile`, if the route names one,
145
+ 4. the route `policies` keys.
146
+
147
+ Within a layer, `false` removes a policy declared below it; an object merges
148
+ shallowly over what is there, so a route can tighten one number without
149
+ restating the rest. Only top-level keys of each policy merge: a route that
150
+ writes `agents: { deny: [crawlers] }` replaces the whole `deny` array, not one
151
+ entry of it.
152
+
153
+ ```yaml
154
+ policies:
155
+ profile: hardened
156
+ throttle: { quota: 5 } # hardened's window and partition remain
157
+ profiles:
158
+ mine:
159
+ security: { headers: oshp-no-csp }
160
+ routes:
161
+ /feed:
162
+ policies:
163
+ profile: mine # merges over the project layer, not instead of it
164
+ throttle: false # removed for this route only
165
+ cache: { strategy: swr, maxAge: 3 }
166
+ ```
167
+
168
+ Here `/feed` ends up with `security: { headers: oshp-no-csp }`, the
169
+ `hardened` agents and compression entries, no throttle, and the route's cache
170
+ entry. Any other route gets `hardened` with `throttle.quota` at 5.
171
+
172
+ `profiles` is a top-level map of reusable policy sets, each a `policies`
173
+ object without a `profile` key. A custom profile whose name matches a built-in
174
+ shadows it, so a project can redefine `hardened` and every reference to it
175
+ resolves to the project's version. Naming an unknown profile fails validation.
176
+
177
+ When any policy is declared anywhere in the project, every route is compiled
178
+ against the effective table; a route with nothing effective simply has empty
179
+ chains. When none is declared, no policy code runs.
180
+
181
+ ## The built-in `hardened` profile
182
+
183
+ `policies.profile: hardened` expands to the following and nothing else, so it
184
+ can be read in one place and overridden key by key. This is
185
+ `builtinProfiles.hardened` in `src/policies.ts`:
186
+
187
+ ```yaml
188
+ policies:
189
+ security: { headers: oshp }
190
+ agents: { deny: [ai-crawlers], status: 403 }
191
+ throttle: { quota: 120, window: 60, partition: client, status: 429 }
192
+ compression: { encodings: [br, gzip], minBytes: 1024 }
193
+ cache: { strategy: revalidate }
194
+ ```
195
+
196
+ The numbers are starting points chosen to be safe for a single small instance;
197
+ they are not tuned for any workload and not a security assessment of your
198
+ deployment. Note what the profile implies per target: on Vercel and AWS it
199
+ refuses activation as written, because `throttle.partition: client` is
200
+ refused there; set `throttle: false` (or `partition: route`) at the project
201
+ or route level. `compression` is delegated to the platform. On Cloudflare
202
+ only `agents` and `security` survive and `compression` is delegated, so the
203
+ profile must also drop `throttle` and `cache` there.
204
+ There is no `strict` profile: anything stricter is a per-project decision.
205
+
206
+ ## The policy contract in TypeScript
207
+
208
+ The policies are modules of one shape, `PolicyModule<Config, State>` in
209
+ `src/types.ts`: `targets`, `compile`, `onRequest`, `onResponse`, optional
210
+ `onError`, `describe` and `close`. `@jimhoyd/urlcode/policies` exports that type with
211
+ `PolicyRequest`, `PolicyContext`, `PolicyChain`, `PolicyShared` and
212
+ `PolicyRegistry` (the five built-ins keyed by name), and the declarations ship
213
+ with the package. A per-policy configuration is typed as the YAML it accepts,
214
+ and a `profiles` layer may hold a partial one:
215
+
216
+ ```ts
217
+ import { registry, targets, type PolicyRegistry, type PolicyRequest } from '@jimhoyd/urlcode/policies';
218
+
219
+ const throttle: PolicyRegistry['throttle'] = registry.throttle; // PolicyModule<ThrottleConfig, ThrottleState>
220
+ const support = throttle.targets({ quota: 120, window: 60, partition: 'client', status: 429 }); // per-target support for this config
221
+ console.log(support.vercel, support.cloudflare);
222
+ function inspect(request: PolicyRequest): void { console.log(request.route, request.client, targets); }
223
+ ```
224
+
225
+ The registry is read-only: a project cannot add a policy from YAML, and an
226
+ operator adds behavior through [plugins](PLUGINS.md), not by editing it.
227
+
228
+ ## Supplying your own patterns
229
+
230
+ The runtime ships mechanisms and one named profile, not an opinion about who
231
+ should be blocked. Ways to express your own:
232
+
233
+ - **Custom profiles.** Define any number under `profiles` and select one per
234
+ project or per route. Profiles are plain data and travel with the YAML.
235
+ - **Per-route overrides.** Any key can be tightened, replaced or set to
236
+ `false` on a route.
237
+ - **Own agent lists.** `agents.deny` and `agents.allow` accept bundled list
238
+ names and project-relative `.json` files in the same schema, so a list you
239
+ do not want to redistribute stays yours. `denyPatterns`/`allowPatterns`
240
+ take a bounded, linear-time pattern subset. See [agents](policies/agents.md).
241
+ - **Header by header.** `security.set` adds or overrides a header and wins over
242
+ the profile, YAML `response.headers` and handler output; `security.unset`
243
+ drops one the profile would emit. Headers the runtime or a handler owns
244
+ (`content-type`, `cache-control`, `set-cookie`, `etag`, `location`, and the
245
+ rest listed in `src/policies/security.ts`) cannot be `set`.
246
+ See [security](policies/security.md).
247
+ - **Explicit cache fields.** A strategy sets defaults; `maxAge`,
248
+ `staleWhileRevalidate`, `staleIfError`, `cdnMaxAge`, `originTtl`, `vary`,
249
+ `statuses`, `maxBytes` and `maxEntries` override what it implies.
250
+ See [cache](policies/cache.md).
251
+ - **Plugins.** Verified-bot checks, shared-store throttling, purge endpoints
252
+ and anything vendor-specific are host code an operator passes in;
253
+ see [plugins](PLUGINS.md).
254
+
255
+ ## Client identity and `--trusted-proxies`
256
+
257
+ `throttle` partitions by `client`. On the self-hosted server the client is the
258
+ socket peer unless `urlcode serve --trusted-proxies 10.0.0.0/8,fd00::/8`
259
+ names the addresses allowed to speak for a client. Then `X-Forwarded-For` is
260
+ walked from the right, skipping trusted hops, and the first untrusted address is
261
+ the client; a chain made only of trusted proxies yields its leftmost entry, and
262
+ a malformed entry is skipped. A forwarded header from a peer
263
+ outside the trusted set is ignored, as is a request carrying more than one
264
+ `X-Forwarded-For` field. Ranges are IPv4 or IPv6 CIDRs (at most 256);
265
+ IPv4-mapped IPv6 peers match IPv4 ranges. `startServer({ trustedProxies })`
266
+ takes the same list.
267
+
268
+ A request whose client cannot be resolved (an adapter without a peer, an
269
+ embedding caller that passes none) shares one bucket rather than being exempt,
270
+ so a misconfigured proxy fails closed. The throttle summary in
271
+ `testPlan().policies` records this as `unresolvedClient: "shared key"`. The runtime still
272
+ never trusts forwarded headers for its public origin; set `--origin`
273
+ explicitly, as [resilience](RESILIENCE.md) already requires.
274
+
275
+ ## What `routes` and `audit` report
276
+
277
+ `urlcode routes` prints the inventory with a `policies` array per route naming
278
+ the policies effective on it (`testPlan().inventory[].policies`) and the full
279
+ `policies` map. The embedding API and a plugin's `onActivate` see
280
+ `testPlan().policies`, a map from route pattern to each policy's summary with
281
+ its `target` value (`native`, `compiled` or `delegated`), the per-route
282
+ capability table the portability rule calls for. `urlcode audit` prints the
283
+ same table under `policies` and, with `--compliance`, checks the declared
284
+ configuration against standards-referenced rules; see
285
+ [compliance](COMPLIANCE.md). `urlcode doctor` lists the policy names this
286
+ runtime knows.
287
+
288
+ ## Logging
289
+
290
+ Policies log through the runtime's request log with one-line events:
291
+ `{ event: 'throttle', route, outcome: 'exceeded' | 'allowed', remaining }`,
292
+ `{ event: 'agents', route, list, outcome: 'denied' | 'reported' }` and the
293
+ cache events described on the [cache page](policies/cache.md). Events name the
294
+ configured route pattern and the list or strategy, never a client address, a
295
+ User-Agent string or request text. A logging failure never changes a response.
296
+
297
+ ## Hardened configuration guidance
298
+
299
+ Advice, not defaults, condensed from the spike's section 6.
300
+
301
+ 1. **Network and edge first.** Volumetric protection, TLS termination and
302
+ per-client connection budgets stay with the provider or the reverse proxy.
303
+ Runtime policies are a second layer, never the first.
304
+ 2. **Ingress to origin.** Bind privately; allow only the proxy's addresses;
305
+ pass `--trusted-proxies` so `client` partitioning sees the real peer.
306
+ 3. **Request policies.** Agents before throttle: denials are cheaper than
307
+ counting. Start throttle in `mode: report` for a release to see real
308
+ quotas in the headers and logs, then switch to `enforce`.
309
+ 4. **Allow before deny.** Keep an explicit allow for the crawlers you need
310
+ indexed; a broad deny without one is the common self-inflicted outage.
311
+ 5. **Route contract.** Exact methods, `request.body` limits and `expires` on
312
+ campaign routes still do most of the work.
313
+ 6. **Response policies.** Security headers on every route; compression only
314
+ on listed types and never on secret-bearing responses (the BREACH class of
315
+ attack, which is why compression is skipped where a route declares secrets
316
+ unless `allowWithSecrets` says otherwise); caching only with a strategy
317
+ whose semantics you can state, `immutable` only on content-hashed paths,
318
+ `no-store` everywhere else.
319
+ 7. **Lists as pinned data.** Bundled agent lists ship with the release, so a
320
+ rollback rolls the list back too.
321
+ 8. **Read the table.** Check `urlcode routes` on each target you deploy to;
322
+ the same YAML is refused where it cannot be enforced, and that is the
323
+ point.
324
+
325
+ ## Interoperability
326
+
327
+ Every pair of policies, and every policy against the runtime's own responses,
328
+ was reviewed by reading the code and by exercising a server. The rules below
329
+ are the ones the current code enforces; where a combination misbehaves, the
330
+ rule says so and names what to do instead.
331
+
332
+ ### Cannot coexist
333
+
334
+ Hard conflicts: the runtime refuses activation, or the combination produces
335
+ a wrong answer today.
336
+
337
+ - **Origin cache and conditional or range requests.** A request carrying
338
+ `If-None-Match`, `If-Modified-Since`, `If-Match`, `If-Unmodified-Since` or
339
+ `Range` is never answered from the origin cache: a stored entry is a full
340
+ `200` representation and the handler owns validators and ranges, so such
341
+ requests always reach it and get the handler's `304`, `412` or `206`.
342
+ Unconditional requests are served from the entry.
343
+ - **`security.unset` and a route that switches profile.** `unset` is
344
+ validated against the profile in effect on that route. A project-level
345
+ `unset` merges into a route that names a profile without that header
346
+ (`headers: off`, or `oshp-no-csp` with `unset: [Content-Security-Policy]`)
347
+ and activation fails with `unset names "...", which the off profile does
348
+ not emit`. Write `unset: []` on that route: a route key replaces the whole
349
+ array.
350
+ - **`security.set` of `Cache-Control`, `Content-Encoding`, `ETag`,
351
+ `Content-Type`, `Set-Cookie`, `Location` and the other reserved names** is
352
+ refused at activation (`is owned by the runtime or handler`). `Vary`,
353
+ `RateLimit`, `RateLimit-Policy`, `Retry-After` and `Age` are reserved for
354
+ the same reason: the cache and throttle policies own them, and a `set` of
355
+ any of the five is refused at activation with the route named. Declare
356
+ `vary` on the cache policy instead.
357
+ - **`agents.denyEmpty`, `throttle` and the generated probes.** `urlcode
358
+ audit`, `urlcode test` and `urlcode benchmark` send generated probes as
359
+ `Mozilla/5.0 (compatible; RouteProbe/0.1)`,
360
+ so `denyEmpty` does not fail them; a deny pattern that matches that string
361
+ would. A tight `throttle` fails an audit once the probes exceed `quota`
362
+ (the audit runs from one address, so a `static` tree with more files than
363
+ the quota trips it): keep the quota at least the number of generated cases
364
+ or run the audit in `mode: report`.
365
+ - **`hardened` on Vercel and AWS** is refused as written, because
366
+ `throttle.partition: client` cannot be honoured there; **on Cloudflare**
367
+ the build refuses `throttle` and `cache`. Override the offending keys with
368
+ `false` or `partition: route` as the profile section above describes.
369
+ - **`compression.encodings: [zstd]` on a Node without `zlib.zstdCompressSync`**
370
+ fails activation with the route named rather than serving identity.
371
+
372
+ ### Coexist with defined precedence
373
+
374
+ Pairs that work, with the rule the code applies.
375
+
376
+ - **Request order is `agents`, `throttle`, cache lookup.** A denied agent is
377
+ never counted and never looked up; a throttled request is never looked up.
378
+ An agent on an `allow` list passes `agents` and is still throttled. An
379
+ `agents` denial and a `throttle` refusal both pass through the cache
380
+ policy's response hook (they carry no flight, so nothing is stored) and
381
+ pick up its declared `vary` names, then security headers, then
382
+ `Vary: Accept-Encoding` from compression; their bodies are below `minBytes`
383
+ so they are never encoded.
384
+ - **Cache hit and the response phase.** A hit skips only the cache's own
385
+ response hook: it still carries the client's `RateLimit` headers (the
386
+ request was counted), the security profile, and is compressed on the way
387
+ out. Stored entries are the handler's bytes and headers after YAML
388
+ `response.headers` and the cache policy ran, before throttle, security and
389
+ compression: bodies are stored uncompressed and encoded again on every hit,
390
+ and a `RateLimit` value is never stored. A cache hit on an asset serves the
391
+ same snapshot buffer, so a precompressed variant is used for `GET` and
392
+ reported at the variant's length for `HEAD`, exactly as when uncached.
393
+ - **Cache `Vary` and compression `Vary`.** The cache merges its declared
394
+ `vary` names first; compression appends `Accept-Encoding` without
395
+ duplicating it and leaves a `Vary: *` alone. The origin key uses only the
396
+ cache's names: because bodies are stored uncompressed, `Accept-Encoding` is
397
+ not part of the key and need not be.
398
+ - **ETags across `cache` and `compression`.** `revalidate` keeps a handler or
399
+ asset `ETag` and computes a strong SHA-256 tag for a `200` without one.
400
+ Compression then weakens a dynamically encoded body's tag (`W/"..."`) and
401
+ suffixes a precompressed asset's (`"...-gz"`). Revalidation works in every
402
+ combination: the cache's `304` compares weakly, the asset handler compares
403
+ weakly against the identity tag, and compression answers `304` for a
404
+ suffixed tag it produced. The `304` that `revalidate` produces keeps
405
+ `Content-Type`, so compression adds `Vary: Accept-Encoding` to it as it
406
+ does to the asset handler's own `304`.
407
+ - **Who owns `Cache-Control`.** YAML `response.headers` first, then an asset
408
+ handler's `cacheControl` when the cache policy is only inherited, then a
409
+ handler's `private` or `no-store`, then the strategy; `security.set` cannot
410
+ name it. A `no-store` or `private` handler answer is never stored whatever
411
+ the strategy says.
412
+ - **Security headers and everything else.** Profile headers fill gaps only:
413
+ YAML `response.headers`, function, asset, redirect and early-denial headers
414
+ keep their values. `set` overrides all of them. `Strict-Transport-Security`
415
+ needs an `https` origin. The profile is applied on cache hits and on early
416
+ denials (including a plugin short-circuit), on the self-hosted server and
417
+ in the Worker alike.
418
+ - **Compression and secrets or cookies.** A route with `secrets` or a
419
+ response with `Set-Cookie` is sent as identity (still with `Vary`) unless
420
+ `allowWithSecrets: true`. A route with `secrets` also never enters the
421
+ origin cache, so the two policies agree on what a secret-bearing route is.
422
+ - **Throttle and the 405.** The request phase runs before the method check,
423
+ so a request that ends as `405` was counted, and the `405` passes through
424
+ the response phase: it carries the `RateLimit` headers, the security
425
+ profile and plugin `onResponse` rewrites. Nothing stores it.
426
+ - **Plugins and policies.** `onRequest` runs before every policy; a
427
+ short-circuit result skips `agents`, `throttle` and the cache lookup
428
+ (nothing is counted, matched or stored) and skips the cache store and the
429
+ throttle headers, then receives security headers and compression, then
430
+ every plugin's `onResponse`. On a thrown error the policies' `onError`
431
+ hooks run first (in the order the policies were declared) and then plugin
432
+ `onError` hooks in reverse. Only the cache policy has an error hook today
433
+ and it never returns a fallback (`stale-if-error` is header-only); if a
434
+ future policy did, that fallback would pass through the response phase and
435
+ every plugin `onResponse`, and plugin `onError` would not run for that
436
+ request.
437
+ - **Reload.** Every reload builds a new runtime with fresh shared state:
438
+ throttle counters start empty and the origin cache and precompressed
439
+ variants are rebuilt; a client mid-window gets a fresh budget. The plugin
440
+ objects are the ones the operator passed and persist across reloads:
441
+ `onActivate` runs for the new runtime before `onClose` runs for the old,
442
+ so a plugin's own state (a shared-store connection, a `WeakMap`) survives a
443
+ reload unless its `onClose` discards it. A plugin whose `onActivate`
444
+ throws makes the reload fail and the old runtime keeps serving.
445
+ - **Merge edge cases.** A route `profile` that lacks a key the project layer
446
+ had leaves that key in force (layers merge, they do not replace); `false`
447
+ at the project level followed by an object at the route level re-enables
448
+ the policy with the route's object alone, so it must be complete
449
+ (`throttle` needs `quota` and `window`); a custom profile named like a
450
+ built-in shadows it everywhere.
451
+ - **Memory bounds add up.** Per runtime: 64 MiB of asset snapshot, up to
452
+ 64 MiB of precompressed variants, up to 64 MiB of origin-cache bodies (a
453
+ stored asset references the snapshot buffer rather than copying it, but
454
+ is counted against the cache budget), and the throttle table at the
455
+ largest `maxKeys` (100,000 keys by default). Two routes serving the same
456
+ file compress it separately and both count. During a reload both
457
+ generations exist at once. [Capacity](CAPACITY.md) states the asset,
458
+ throttle and cache figures; the precompressed budget belongs in that table
459
+ too.
460
+ - **Cloudflare.** The artifact carries the effective `agents` and `security`
461
+ configuration per route, with the entries of any project list file
462
+ embedded under the reference as written in YAML (no filesystem path); a
463
+ route-level `agents: false` leaves the artifact without that key. The
464
+ Worker runs `agents` on the request and `security` on the response, so an
465
+ early denial carries the profile exactly as on the self-hosted server.
466
+
467
+ ### Not covered by policies
468
+
469
+ - **Every error the runtime throws** (404 for no match, disabled route or
470
+ missing link, 410, 400/413/415 from body checks, 502/503/504 from the
471
+ sandbox or a link store) bypasses the response phase: no `Vary`, no
472
+ `RateLimit`, no compression, no plugin `onResponse`, on every target.
473
+ What they do get is the `security` policy: the matched route's effective
474
+ profile when the error came after routing (so a route with
475
+ `security: false` answers its 410 bare), otherwise the project-level
476
+ profile, including a host-side error such as an oversized body or shed
477
+ admission and the Worker's own 404. The runtime's fixed headers
478
+ (`Content-Type`, `Cache-Control: no-store`, `Content-Length`,
479
+ `X-Request-Id`, `X-Content-Type-Options`) can never be replaced by it.
480
+ Plugins keep `onError` for observation; a policy error hook may answer with
481
+ a fallback, and none does today.
482
+ - **The audit's probes** share one address and one `User-Agent`, so they do
483
+ not exercise `agents` or `throttle` the way real traffic does;
484
+ the policy table in `testPlan().policies` is the audit's evidence for
485
+ those two.
486
+ - **`stale-if-error` at the origin**: header-only, as the
487
+ [cache page](policies/cache.md) states.
488
+ - **Cross-instance state**: counters and the cache are per runtime on every
489
+ target; a shared budget or a shared cache is a plugin.