@jimhoyd/urlcode 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +80 -0
- package/LICENSE +202 -0
- package/NOTICE +20 -0
- package/README.md +342 -0
- package/ROADMAP.md +248 -0
- package/SECURITY.md +40 -0
- package/data/agents/LICENSES/ai-robots-txt.txt +24 -0
- package/data/agents/LICENSES/crawler-user-agents.txt +23 -0
- package/data/agents/ai-crawlers.json +1240 -0
- package/data/agents/crawlers.json +10515 -0
- package/data/agents/index.d.ts +3 -0
- package/data/agents/index.js +2453 -0
- package/data/agents/monitoring.json +1520 -0
- package/data/agents/seo.json +3865 -0
- package/dist/BUILD-MANIFEST.json +60 -0
- package/dist/adapters.js +59 -0
- package/dist/agent-lists.js +59 -0
- package/dist/assets.js +129 -0
- package/dist/authoring.js +68 -0
- package/dist/aws.js +110 -0
- package/dist/build-cloudflare.js +164 -0
- package/dist/cli.js +185 -0
- package/dist/client-address.js +85 -0
- package/dist/cloudflare.js +165 -0
- package/dist/compliance-rules/baseline.js +98 -0
- package/dist/compliance-rules/privacy.js +38 -0
- package/dist/compliance-rules/shared.js +62 -0
- package/dist/compliance-rules/strict.js +61 -0
- package/dist/compliance.js +219 -0
- package/dist/config-worker.js +9 -0
- package/dist/config.js +172 -0
- package/dist/errors.js +8 -0
- package/dist/function-sources.js +54 -0
- package/dist/function-worker.js +105 -0
- package/dist/functions.js +174 -0
- package/dist/guest-api.js +114 -0
- package/dist/header-validation.js +18 -0
- package/dist/http-policy.js +74 -0
- package/dist/http-response.js +84 -0
- package/dist/index.js +11 -0
- package/dist/link-api.js +136 -0
- package/dist/link-cli.js +141 -0
- package/dist/link-events.js +76 -0
- package/dist/link-records.js +31 -0
- package/dist/link-store-worker.js +150 -0
- package/dist/link-store.js +250 -0
- package/dist/logging.js +22 -0
- package/dist/management-policy.js +41 -0
- package/dist/match.js +124 -0
- package/dist/observability.js +242 -0
- package/dist/plugins.js +74 -0
- package/dist/policies/agents.js +248 -0
- package/dist/policies/cache.js +297 -0
- package/dist/policies/compression.js +187 -0
- package/dist/policies/security.js +0 -0
- package/dist/policies/throttle.js +131 -0
- package/dist/policies.js +142 -0
- package/dist/policy.js +55 -0
- package/dist/prerender.js +165 -0
- package/dist/project-tests.js +23 -0
- package/dist/readiness.js +216 -0
- package/dist/router.js +166 -0
- package/dist/runtime.js +244 -0
- package/dist/scaffold.js +0 -0
- package/dist/scripts/operational-drills.js +75 -0
- package/dist/server.js +270 -0
- package/dist/site.js +0 -0
- package/dist/sqlite-version.js +6 -0
- package/dist/types/adapters.d.ts +12 -0
- package/dist/types/agent-lists.d.ts +5 -0
- package/dist/types/assets.d.ts +13 -0
- package/dist/types/authoring.d.ts +2 -0
- package/dist/types/aws.d.ts +35 -0
- package/dist/types/build-cloudflare.d.ts +14 -0
- package/dist/types/cli.d.ts +2 -0
- package/dist/types/client-address.d.ts +9 -0
- package/dist/types/cloudflare.d.ts +75 -0
- package/dist/types/compliance-rules/baseline.d.ts +12 -0
- package/dist/types/compliance-rules/privacy.d.ts +6 -0
- package/dist/types/compliance-rules/shared.d.ts +24 -0
- package/dist/types/compliance-rules/strict.d.ts +9 -0
- package/dist/types/compliance.d.ts +119 -0
- package/dist/types/config-worker.d.ts +1 -0
- package/dist/types/config.d.ts +21 -0
- package/dist/types/errors.d.ts +7 -0
- package/dist/types/function-sources.d.ts +21 -0
- package/dist/types/function-worker.d.ts +1 -0
- package/dist/types/functions.d.ts +92 -0
- package/dist/types/guest-api.d.ts +16 -0
- package/dist/types/header-validation.d.ts +2 -0
- package/dist/types/http-policy.d.ts +39 -0
- package/dist/types/http-response.d.ts +43 -0
- package/dist/types/index.d.ts +11 -0
- package/dist/types/link-api.d.ts +30 -0
- package/dist/types/link-cli.d.ts +36 -0
- package/dist/types/link-events.d.ts +27 -0
- package/dist/types/link-records.d.ts +11 -0
- package/dist/types/link-store-worker.d.ts +1 -0
- package/dist/types/link-store.d.ts +130 -0
- package/dist/types/logging.d.ts +9 -0
- package/dist/types/management-policy.d.ts +9 -0
- package/dist/types/match.d.ts +81 -0
- package/dist/types/observability.d.ts +104 -0
- package/dist/types/plugins.d.ts +27 -0
- package/dist/types/policies/agents.d.ts +86 -0
- package/dist/types/policies/cache.d.ts +82 -0
- package/dist/types/policies/compression.d.ts +41 -0
- package/dist/types/policies/security.d.ts +37 -0
- package/dist/types/policies/throttle.d.ts +55 -0
- package/dist/types/policies.d.ts +36 -0
- package/dist/types/policy.d.ts +21 -0
- package/dist/types/prerender.d.ts +31 -0
- package/dist/types/project-tests.d.ts +13 -0
- package/dist/types/readiness.d.ts +121 -0
- package/dist/types/router.d.ts +11 -0
- package/dist/types/runtime.d.ts +91 -0
- package/dist/types/scaffold.d.ts +17 -0
- package/dist/types/server.d.ts +37 -0
- package/dist/types/site.d.ts +12 -0
- package/dist/types/sqlite-version.d.ts +1 -0
- package/dist/types/types.d.ts +332 -0
- package/dist/types/vercel.d.ts +12 -0
- package/dist/types.js +177 -0
- package/dist/vercel.js +65 -0
- package/docs/AI-AUTHORING.md +113 -0
- package/docs/ASSETS.md +106 -0
- package/docs/AWS.md +77 -0
- package/docs/BEST-PRACTICES.md +267 -0
- package/docs/CAPACITY.md +208 -0
- package/docs/CLOUDFLARE.md +110 -0
- package/docs/COMPLIANCE.md +242 -0
- package/docs/DYNAMIC-LINKS.md +561 -0
- package/docs/FUNCTION-SECURITY.md +113 -0
- package/docs/HTTP.md +129 -0
- package/docs/INSTALL.md +90 -0
- package/docs/LOAD-TESTING.md +91 -0
- package/docs/LOCAL-DEVELOPMENT.md +99 -0
- package/docs/MANAGEMENT-SECURITY.md +82 -0
- package/docs/MIDDLEWARE.md +83 -0
- package/docs/MONITORING.md +132 -0
- package/docs/OBSERVABILITY.md +229 -0
- package/docs/OPERATIONAL-PROOF.md +42 -0
- package/docs/OPERATIONS.md +207 -0
- package/docs/ORGANIZATION.md +135 -0
- package/docs/PERFORMANCE.md +72 -0
- package/docs/PLUGINS.md +234 -0
- package/docs/POLICIES.md +489 -0
- package/docs/PRERENDER.md +193 -0
- package/docs/PROJECT-DIRECTION.md +51 -0
- package/docs/READINESS.md +125 -0
- package/docs/README.md +58 -0
- package/docs/RELEASE-READINESS.md +95 -0
- package/docs/RELEASE-SECURITY.md +72 -0
- package/docs/RESILIENCE.md +160 -0
- package/docs/ROUTING.md +95 -0
- package/docs/SANDBOX-REVIEW.md +59 -0
- package/docs/SCAFFOLDING.md +72 -0
- package/docs/SECURITY-AUDIT.md +124 -0
- package/docs/SITE.md +150 -0
- package/docs/SPECIFICATION.md +232 -0
- package/docs/SPIKE-EXTENSIONS.md +475 -0
- package/docs/STANDARDS.md +303 -0
- package/docs/STARTERS.md +51 -0
- package/docs/TUNNELS.md +75 -0
- package/docs/TYPESCRIPT.md +102 -0
- package/docs/VERCEL.md +107 -0
- package/docs/YAML-GUIDE.md +521 -0
- package/docs/YAML-REFERENCE.md +328 -0
- package/docs/policies/agents.md +182 -0
- package/docs/policies/cache.md +139 -0
- package/docs/policies/compression.md +168 -0
- package/docs/policies/security.md +160 -0
- package/docs/policies/throttle.md +103 -0
- package/examples/assets/.env.example +1 -0
- package/examples/assets/.gitattributes +2 -0
- package/examples/assets/Makefile +30 -0
- package/examples/assets/README.md +18 -0
- package/examples/assets/functions/hello.mjs +3 -0
- package/examples/assets/gitignore.template +7 -0
- package/examples/assets/public/about.html +2 -0
- package/examples/assets/public/assets/example.txt +1 -0
- package/examples/assets/public/guide.txt +1 -0
- package/examples/assets/tests/requests.json +44 -0
- package/examples/assets/urlcode.yaml +27 -0
- package/examples/aws/README.md +13 -0
- package/examples/aws/handler.mjs +5 -0
- package/examples/aws/public/index.html +3 -0
- package/examples/aws/public/notes.txt +1 -0
- package/examples/aws/template.yaml +33 -0
- package/examples/aws/tests/requests.json +9 -0
- package/examples/aws/urlcode.yaml +19 -0
- package/examples/cloudflare/README.md +18 -0
- package/examples/cloudflare/package.json +14 -0
- package/examples/cloudflare/tests/requests.json +9 -0
- package/examples/cloudflare/urlcode.yaml +24 -0
- package/examples/cloudflare/wrangler.toml +5 -0
- package/examples/compliance/README.md +20 -0
- package/examples/compliance/rules.mjs +65 -0
- package/examples/cookbook/README.md +21 -0
- package/examples/cookbook/functions/choice.mjs +4 -0
- package/examples/cookbook/functions/echo.mjs +3 -0
- package/examples/cookbook/functions/hello.mjs +3 -0
- package/examples/cookbook/functions/text.mjs +3 -0
- package/examples/cookbook/middleware/headers.mjs +6 -0
- package/examples/cookbook/public/about.html +1 -0
- package/examples/cookbook/public/assets/index.html +1 -0
- package/examples/cookbook/public/assets/site.css +1 -0
- package/examples/cookbook/public/favicon.svg +1 -0
- package/examples/cookbook/public/guide.txt +1 -0
- package/examples/cookbook/public/llms.txt +6 -0
- package/examples/cookbook/routes/code.yaml +55 -0
- package/examples/cookbook/routes/files.yaml +17 -0
- package/examples/cookbook/routes/policies.yaml +28 -0
- package/examples/cookbook/routes/redirects.yaml +38 -0
- package/examples/cookbook/routes/responses.yaml +26 -0
- package/examples/cookbook/tests/requests.json +254 -0
- package/examples/cookbook/urlcode.yaml +22 -0
- package/examples/live-links/README.md +11 -0
- package/examples/live-links/tests/requests.json +6 -0
- package/examples/live-links/urlcode.yaml +16 -0
- package/examples/monitoring/blackbox-jobs.yaml +37 -0
- package/examples/monitoring/prometheus-rules.yaml +88 -0
- package/examples/monitoring/prometheus-scrape.yaml +20 -0
- package/examples/monitoring/vector.toml +65 -0
- package/examples/prerender/README.md +48 -0
- package/examples/prerender/functions/page.mjs +6 -0
- package/examples/prerender/middleware/template.mjs +29 -0
- package/examples/prerender/prerender.d.mts +4 -0
- package/examples/prerender/prerender.mjs +63 -0
- package/examples/prerender/tests/requests.json +39 -0
- package/examples/prerender/urlcode.yaml +50 -0
- package/examples/tunnel/dev-with-ngrok.sh +61 -0
- package/examples/vercel/README.md +13 -0
- package/examples/vercel/api/index.js +5 -0
- package/examples/vercel/package.json +8 -0
- package/examples/vercel/public/index.html +3 -0
- package/examples/vercel/public/notes.txt +1 -0
- package/examples/vercel/tests/requests.json +48 -0
- package/examples/vercel/urlcode.yaml +19 -0
- package/examples/vercel/vercel.json +10 -0
- package/llms.txt +46 -0
- package/package.json +123 -0
- package/schemas/urlcode.schema.json +1230 -0
- package/starters/default/.gitattributes +1 -0
- package/starters/default/Makefile +30 -0
- package/starters/default/README.md +34 -0
- package/starters/default/functions/hello.mjs +3 -0
- package/starters/default/gitignore.template +8 -0
- package/starters/default/middleware/headers.mjs +6 -0
- package/starters/default/routes/functions.yaml +20 -0
- package/starters/default/routes/marketing/links.yaml +7 -0
- package/starters/default/starter.json +5 -0
- package/starters/default/tests/requests.json +56 -0
- package/starters/default/urlcode.yaml +8 -0
package/docs/POLICIES.md
ADDED
|
@@ -0,0 +1,489 @@
|
|
|
1
|
+
# Policies
|
|
2
|
+
|
|
3
|
+
Policies are host-enforced behavior declared in YAML around a route: a
|
|
4
|
+
per-client request budget, a User-Agent deny list, a security-header profile,
|
|
5
|
+
content-coding negotiation and an HTTP caching strategy. They run in the host
|
|
6
|
+
process, outside the sandbox, so they can see the client address, keep counters
|
|
7
|
+
across requests and touch the transport, which route middleware by design
|
|
8
|
+
cannot ([middleware](MIDDLEWARE.md), [function security](FUNCTION-SECURITY.md)).
|
|
9
|
+
|
|
10
|
+
Everything here is optional and off by default. A project with no `policies`
|
|
11
|
+
key and no `profiles` key behaves exactly as before: no policy module is
|
|
12
|
+
compiled and the request path is unchanged. Each policy has its own page:
|
|
13
|
+
|
|
14
|
+
| Policy | Page | What it does |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| `throttle` | [throttle](policies/throttle.md) | Sliding-window request budget with `RateLimit`/`RateLimit-Policy` headers, 429 and `Retry-After` |
|
|
17
|
+
| `agents` | [agents](policies/agents.md) | Deny or allow by `User-Agent`, from bundled lists, project files and bounded patterns |
|
|
18
|
+
| `security` | [security](policies/security.md) | OWASP Secure Headers Project response headers, with per-header `set`/`unset` |
|
|
19
|
+
| `compression` | [compression](policies/compression.md) | `Accept-Encoding` negotiation; assets precompressed at snapshot time |
|
|
20
|
+
| `cache` | [cache](policies/cache.md) | Named strategies from RFC 9111/5861/8246/9213 plus an origin memory cache |
|
|
21
|
+
|
|
22
|
+
The design and the reasoning behind each choice are in the
|
|
23
|
+
[extensions spike](SPIKE-EXTENSIONS.md). Operators who need behavior the
|
|
24
|
+
declarative block cannot express pass host [plugins](PLUGINS.md) in code; plugins
|
|
25
|
+
are never named in YAML.
|
|
26
|
+
|
|
27
|
+
## Declaring policies
|
|
28
|
+
|
|
29
|
+
```yaml
|
|
30
|
+
version: "1"
|
|
31
|
+
policies: # project defaults; every key optional
|
|
32
|
+
profile: hardened # built-in, or a name under `profiles`
|
|
33
|
+
throttle: { quota: 60, window: 60 }
|
|
34
|
+
routes:
|
|
35
|
+
/api/lookup/{id}:
|
|
36
|
+
function: { source: functions/lookup.mjs }
|
|
37
|
+
policies:
|
|
38
|
+
throttle: { quota: 10, window: 60 }
|
|
39
|
+
cache: false
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
`policies` at the top level sets project defaults; `routes.<path>.policies`
|
|
43
|
+
adjusts them for one route. Both accept the same keys: `profile` plus one entry
|
|
44
|
+
per policy, each either an object or `false`. Unknown keys fail validation, as
|
|
45
|
+
everywhere in the project format. The
|
|
46
|
+
[field reference](YAML-REFERENCE.md) lists every accepted field with its
|
|
47
|
+
bounds; the [JSON Schema](../schemas/urlcode.schema.json) is the source.
|
|
48
|
+
|
|
49
|
+
Values are behavior, never infrastructure: a quota is a number a proxy or CDN
|
|
50
|
+
can restate, while trusted proxy ranges, storage URLs and vendor rule
|
|
51
|
+
identifiers stay in operator configuration. That is what keeps a project
|
|
52
|
+
portable between a laptop, a container and an adapter.
|
|
53
|
+
|
|
54
|
+
## Where policies run
|
|
55
|
+
|
|
56
|
+
Policies run once the route is known and before its contract is checked. A
|
|
57
|
+
denied agent or an exhausted budget is answered without reading a body or
|
|
58
|
+
starting the sandbox. The self-hosted pipeline:
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
socket limits → admission → body read
|
|
62
|
+
→ runtime.handle(): match → enabled/expires
|
|
63
|
+
→ plugins onRequest (outermost, first plugin first)
|
|
64
|
+
→ agents → throttle → cache lookup (request phase; may short-circuit)
|
|
65
|
+
→ methods (405) → request.body checks
|
|
66
|
+
→ native handler | guest chain
|
|
67
|
+
→ YAML response.headers
|
|
68
|
+
→ cache store → throttle headers → security → compression (response phase)
|
|
69
|
+
→ plugins onResponse (reverse order, first plugin last)
|
|
70
|
+
→ writeResponse
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Request order is fixed: agents first because a denial is the cheapest
|
|
74
|
+
outcome, then throttle, then the cache lookup, so a denied or refused request
|
|
75
|
+
neither counts against a budget nor touches the cache. Response order is the
|
|
76
|
+
cache store, then the throttle's `RateLimit` headers (after the store, so a
|
|
77
|
+
cached copy is never stamped with one client's remaining budget), then
|
|
78
|
+
security headers, then compression last so every header it depends on is
|
|
79
|
+
already final. YAML `response.headers` are applied by the runtime before the
|
|
80
|
+
response phase, so an explicit header always beats a profile default.
|
|
81
|
+
|
|
82
|
+
A result produced by a request-phase policy (an agent denial, a throttle
|
|
83
|
+
refusal, a cache hit) skips that policy's own response hook and passes through
|
|
84
|
+
the others: a hit is not stored twice but still carries the client's
|
|
85
|
+
rate-limit headers, and a denial is not stored because its status is not
|
|
86
|
+
cacheable. A throttle refusal carries its own `RateLimit` and `Retry-After`
|
|
87
|
+
headers. A plugin short-circuit ran before any policy, so it skips the
|
|
88
|
+
response hook of every policy that has a request phase; security headers and
|
|
89
|
+
compression still apply to it.
|
|
90
|
+
|
|
91
|
+
Two responses bypass the response phase entirely: the `405` the runtime returns
|
|
92
|
+
for an undeclared method, and any error the runtime throws (404 for no match,
|
|
93
|
+
410 for an expired route, 413/415 from body checks, sandbox 502/504). Those
|
|
94
|
+
receive the runtime's standard headers only. Policies with an `onError` hook
|
|
95
|
+
and plugin `onError` hooks observe thrown errors; they cannot change them.
|
|
96
|
+
|
|
97
|
+
Adapters call `handle()` directly and delegate socket limits and admission to
|
|
98
|
+
the provider; the policy order inside `handle()` is the same. The Cloudflare
|
|
99
|
+
Worker runs the same request and response order with the two policies it can
|
|
100
|
+
carry.
|
|
101
|
+
|
|
102
|
+
## Portability and the per-target table
|
|
103
|
+
|
|
104
|
+
A project is portable when a second person can run it elsewhere and get the
|
|
105
|
+
same declared behavior or an explicit refusal. A target that cannot honor a
|
|
106
|
+
policy refuses activation naming the route and the policy, exactly as adapters
|
|
107
|
+
refuse functions and middleware; nothing degrades silently. One exception is
|
|
108
|
+
stated rather than hidden: a policy the platform itself already provides is
|
|
109
|
+
**delegated**, meaning accepted and dropped, and reported as such in the
|
|
110
|
+
inventory, so one YAML can serve a Node host and a serverless host without
|
|
111
|
+
edits. The self-hosted
|
|
112
|
+
message reads `/path declares policies.throttle, which the vercel target
|
|
113
|
+
cannot enforce`; the Cloudflare build reports
|
|
114
|
+
`/path: policies.throttle cannot be compiled for this target`.
|
|
115
|
+
|
|
116
|
+
| Policy | Self-hosted (`node`) | Vercel / AWS | Cloudflare build |
|
|
117
|
+
|---|---|---|---|
|
|
118
|
+
| `agents` | native | native | compiled into the artifact |
|
|
119
|
+
| `security` | native | native | compiled into the artifact |
|
|
120
|
+
| `throttle` | native, in-process counters | native only with `partition: route`; `client` and `client-route` refused because a client fans across instances and the budget would silently be quota × instances | refused |
|
|
121
|
+
| `compression` | native | delegated: the provider compresses | delegated |
|
|
122
|
+
| `cache` | native: headers plus origin memory cache | native | refused |
|
|
123
|
+
|
|
124
|
+
"Compiled" means the effective configuration for every route is validated at
|
|
125
|
+
build time and carried in the Worker artifact; the Worker has no filesystem and
|
|
126
|
+
no Node imports, so only modules free of both qualify. The Worker's client
|
|
127
|
+
identity is the platform's `cf-connecting-ip`; the serverless adapters use the
|
|
128
|
+
platform-set source address (`sourceIp` on Lambda, the leftmost
|
|
129
|
+
`X-Forwarded-For` entry the platform writes on Vercel). None of these read a
|
|
130
|
+
forwarded header a client could have set.
|
|
131
|
+
|
|
132
|
+
The cross-request state a policy keeps (throttle counters, the origin cache) is
|
|
133
|
+
per runtime instance on every target, never shared between replicas or
|
|
134
|
+
serverless instances. Given identical request bytes and project, every target
|
|
135
|
+
answers with the same status and headers; only that state may differ.
|
|
136
|
+
|
|
137
|
+
## Merge semantics
|
|
138
|
+
|
|
139
|
+
The effective configuration for a route is built from four layers, each
|
|
140
|
+
merged over the one below:
|
|
141
|
+
|
|
142
|
+
1. the project `profile` (built-in or custom),
|
|
143
|
+
2. the project `policies` keys,
|
|
144
|
+
3. the route `profile`, if the route names one,
|
|
145
|
+
4. the route `policies` keys.
|
|
146
|
+
|
|
147
|
+
Within a layer, `false` removes a policy declared below it; an object merges
|
|
148
|
+
shallowly over what is there, so a route can tighten one number without
|
|
149
|
+
restating the rest. Only top-level keys of each policy merge: a route that
|
|
150
|
+
writes `agents: { deny: [crawlers] }` replaces the whole `deny` array, not one
|
|
151
|
+
entry of it.
|
|
152
|
+
|
|
153
|
+
```yaml
|
|
154
|
+
policies:
|
|
155
|
+
profile: hardened
|
|
156
|
+
throttle: { quota: 5 } # hardened's window and partition remain
|
|
157
|
+
profiles:
|
|
158
|
+
mine:
|
|
159
|
+
security: { headers: oshp-no-csp }
|
|
160
|
+
routes:
|
|
161
|
+
/feed:
|
|
162
|
+
policies:
|
|
163
|
+
profile: mine # merges over the project layer, not instead of it
|
|
164
|
+
throttle: false # removed for this route only
|
|
165
|
+
cache: { strategy: swr, maxAge: 3 }
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Here `/feed` ends up with `security: { headers: oshp-no-csp }`, the
|
|
169
|
+
`hardened` agents and compression entries, no throttle, and the route's cache
|
|
170
|
+
entry. Any other route gets `hardened` with `throttle.quota` at 5.
|
|
171
|
+
|
|
172
|
+
`profiles` is a top-level map of reusable policy sets, each a `policies`
|
|
173
|
+
object without a `profile` key. A custom profile whose name matches a built-in
|
|
174
|
+
shadows it, so a project can redefine `hardened` and every reference to it
|
|
175
|
+
resolves to the project's version. Naming an unknown profile fails validation.
|
|
176
|
+
|
|
177
|
+
When any policy is declared anywhere in the project, every route is compiled
|
|
178
|
+
against the effective table; a route with nothing effective simply has empty
|
|
179
|
+
chains. When none is declared, no policy code runs.
|
|
180
|
+
|
|
181
|
+
## The built-in `hardened` profile
|
|
182
|
+
|
|
183
|
+
`policies.profile: hardened` expands to the following and nothing else, so it
|
|
184
|
+
can be read in one place and overridden key by key. This is
|
|
185
|
+
`builtinProfiles.hardened` in `src/policies.ts`:
|
|
186
|
+
|
|
187
|
+
```yaml
|
|
188
|
+
policies:
|
|
189
|
+
security: { headers: oshp }
|
|
190
|
+
agents: { deny: [ai-crawlers], status: 403 }
|
|
191
|
+
throttle: { quota: 120, window: 60, partition: client, status: 429 }
|
|
192
|
+
compression: { encodings: [br, gzip], minBytes: 1024 }
|
|
193
|
+
cache: { strategy: revalidate }
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The numbers are starting points chosen to be safe for a single small instance;
|
|
197
|
+
they are not tuned for any workload and not a security assessment of your
|
|
198
|
+
deployment. Note what the profile implies per target: on Vercel and AWS it
|
|
199
|
+
refuses activation as written, because `throttle.partition: client` is
|
|
200
|
+
refused there; set `throttle: false` (or `partition: route`) at the project
|
|
201
|
+
or route level. `compression` is delegated to the platform. On Cloudflare
|
|
202
|
+
only `agents` and `security` survive and `compression` is delegated, so the
|
|
203
|
+
profile must also drop `throttle` and `cache` there.
|
|
204
|
+
There is no `strict` profile: anything stricter is a per-project decision.
|
|
205
|
+
|
|
206
|
+
## The policy contract in TypeScript
|
|
207
|
+
|
|
208
|
+
The policies are modules of one shape, `PolicyModule<Config, State>` in
|
|
209
|
+
`src/types.ts`: `targets`, `compile`, `onRequest`, `onResponse`, optional
|
|
210
|
+
`onError`, `describe` and `close`. `@jimhoyd/urlcode/policies` exports that type with
|
|
211
|
+
`PolicyRequest`, `PolicyContext`, `PolicyChain`, `PolicyShared` and
|
|
212
|
+
`PolicyRegistry` (the five built-ins keyed by name), and the declarations ship
|
|
213
|
+
with the package. A per-policy configuration is typed as the YAML it accepts,
|
|
214
|
+
and a `profiles` layer may hold a partial one:
|
|
215
|
+
|
|
216
|
+
```ts
|
|
217
|
+
import { registry, targets, type PolicyRegistry, type PolicyRequest } from '@jimhoyd/urlcode/policies';
|
|
218
|
+
|
|
219
|
+
const throttle: PolicyRegistry['throttle'] = registry.throttle; // PolicyModule<ThrottleConfig, ThrottleState>
|
|
220
|
+
const support = throttle.targets({ quota: 120, window: 60, partition: 'client', status: 429 }); // per-target support for this config
|
|
221
|
+
console.log(support.vercel, support.cloudflare);
|
|
222
|
+
function inspect(request: PolicyRequest): void { console.log(request.route, request.client, targets); }
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
The registry is read-only: a project cannot add a policy from YAML, and an
|
|
226
|
+
operator adds behavior through [plugins](PLUGINS.md), not by editing it.
|
|
227
|
+
|
|
228
|
+
## Supplying your own patterns
|
|
229
|
+
|
|
230
|
+
The runtime ships mechanisms and one named profile, not an opinion about who
|
|
231
|
+
should be blocked. Ways to express your own:
|
|
232
|
+
|
|
233
|
+
- **Custom profiles.** Define any number under `profiles` and select one per
|
|
234
|
+
project or per route. Profiles are plain data and travel with the YAML.
|
|
235
|
+
- **Per-route overrides.** Any key can be tightened, replaced or set to
|
|
236
|
+
`false` on a route.
|
|
237
|
+
- **Own agent lists.** `agents.deny` and `agents.allow` accept bundled list
|
|
238
|
+
names and project-relative `.json` files in the same schema, so a list you
|
|
239
|
+
do not want to redistribute stays yours. `denyPatterns`/`allowPatterns`
|
|
240
|
+
take a bounded, linear-time pattern subset. See [agents](policies/agents.md).
|
|
241
|
+
- **Header by header.** `security.set` adds or overrides a header and wins over
|
|
242
|
+
the profile, YAML `response.headers` and handler output; `security.unset`
|
|
243
|
+
drops one the profile would emit. Headers the runtime or a handler owns
|
|
244
|
+
(`content-type`, `cache-control`, `set-cookie`, `etag`, `location`, and the
|
|
245
|
+
rest listed in `src/policies/security.ts`) cannot be `set`.
|
|
246
|
+
See [security](policies/security.md).
|
|
247
|
+
- **Explicit cache fields.** A strategy sets defaults; `maxAge`,
|
|
248
|
+
`staleWhileRevalidate`, `staleIfError`, `cdnMaxAge`, `originTtl`, `vary`,
|
|
249
|
+
`statuses`, `maxBytes` and `maxEntries` override what it implies.
|
|
250
|
+
See [cache](policies/cache.md).
|
|
251
|
+
- **Plugins.** Verified-bot checks, shared-store throttling, purge endpoints
|
|
252
|
+
and anything vendor-specific are host code an operator passes in;
|
|
253
|
+
see [plugins](PLUGINS.md).
|
|
254
|
+
|
|
255
|
+
## Client identity and `--trusted-proxies`
|
|
256
|
+
|
|
257
|
+
`throttle` partitions by `client`. On the self-hosted server the client is the
|
|
258
|
+
socket peer unless `urlcode serve --trusted-proxies 10.0.0.0/8,fd00::/8`
|
|
259
|
+
names the addresses allowed to speak for a client. Then `X-Forwarded-For` is
|
|
260
|
+
walked from the right, skipping trusted hops, and the first untrusted address is
|
|
261
|
+
the client; a chain made only of trusted proxies yields its leftmost entry, and
|
|
262
|
+
a malformed entry is skipped. A forwarded header from a peer
|
|
263
|
+
outside the trusted set is ignored, as is a request carrying more than one
|
|
264
|
+
`X-Forwarded-For` field. Ranges are IPv4 or IPv6 CIDRs (at most 256);
|
|
265
|
+
IPv4-mapped IPv6 peers match IPv4 ranges. `startServer({ trustedProxies })`
|
|
266
|
+
takes the same list.
|
|
267
|
+
|
|
268
|
+
A request whose client cannot be resolved (an adapter without a peer, an
|
|
269
|
+
embedding caller that passes none) shares one bucket rather than being exempt,
|
|
270
|
+
so a misconfigured proxy fails closed. The throttle summary in
|
|
271
|
+
`testPlan().policies` records this as `unresolvedClient: "shared key"`. The runtime still
|
|
272
|
+
never trusts forwarded headers for its public origin; set `--origin`
|
|
273
|
+
explicitly, as [resilience](RESILIENCE.md) already requires.
|
|
274
|
+
|
|
275
|
+
## What `routes` and `audit` report
|
|
276
|
+
|
|
277
|
+
`urlcode routes` prints the inventory with a `policies` array per route naming
|
|
278
|
+
the policies effective on it (`testPlan().inventory[].policies`) and the full
|
|
279
|
+
`policies` map. The embedding API and a plugin's `onActivate` see
|
|
280
|
+
`testPlan().policies`, a map from route pattern to each policy's summary with
|
|
281
|
+
its `target` value (`native`, `compiled` or `delegated`), the per-route
|
|
282
|
+
capability table the portability rule calls for. `urlcode audit` prints the
|
|
283
|
+
same table under `policies` and, with `--compliance`, checks the declared
|
|
284
|
+
configuration against standards-referenced rules; see
|
|
285
|
+
[compliance](COMPLIANCE.md). `urlcode doctor` lists the policy names this
|
|
286
|
+
runtime knows.
|
|
287
|
+
|
|
288
|
+
## Logging
|
|
289
|
+
|
|
290
|
+
Policies log through the runtime's request log with one-line events:
|
|
291
|
+
`{ event: 'throttle', route, outcome: 'exceeded' | 'allowed', remaining }`,
|
|
292
|
+
`{ event: 'agents', route, list, outcome: 'denied' | 'reported' }` and the
|
|
293
|
+
cache events described on the [cache page](policies/cache.md). Events name the
|
|
294
|
+
configured route pattern and the list or strategy, never a client address, a
|
|
295
|
+
User-Agent string or request text. A logging failure never changes a response.
|
|
296
|
+
|
|
297
|
+
## Hardened configuration guidance
|
|
298
|
+
|
|
299
|
+
Advice, not defaults, condensed from the spike's section 6.
|
|
300
|
+
|
|
301
|
+
1. **Network and edge first.** Volumetric protection, TLS termination and
|
|
302
|
+
per-client connection budgets stay with the provider or the reverse proxy.
|
|
303
|
+
Runtime policies are a second layer, never the first.
|
|
304
|
+
2. **Ingress to origin.** Bind privately; allow only the proxy's addresses;
|
|
305
|
+
pass `--trusted-proxies` so `client` partitioning sees the real peer.
|
|
306
|
+
3. **Request policies.** Agents before throttle: denials are cheaper than
|
|
307
|
+
counting. Start throttle in `mode: report` for a release to see real
|
|
308
|
+
quotas in the headers and logs, then switch to `enforce`.
|
|
309
|
+
4. **Allow before deny.** Keep an explicit allow for the crawlers you need
|
|
310
|
+
indexed; a broad deny without one is the common self-inflicted outage.
|
|
311
|
+
5. **Route contract.** Exact methods, `request.body` limits and `expires` on
|
|
312
|
+
campaign routes still do most of the work.
|
|
313
|
+
6. **Response policies.** Security headers on every route; compression only
|
|
314
|
+
on listed types and never on secret-bearing responses (the BREACH class of
|
|
315
|
+
attack, which is why compression is skipped where a route declares secrets
|
|
316
|
+
unless `allowWithSecrets` says otherwise); caching only with a strategy
|
|
317
|
+
whose semantics you can state, `immutable` only on content-hashed paths,
|
|
318
|
+
`no-store` everywhere else.
|
|
319
|
+
7. **Lists as pinned data.** Bundled agent lists ship with the release, so a
|
|
320
|
+
rollback rolls the list back too.
|
|
321
|
+
8. **Read the table.** Check `urlcode routes` on each target you deploy to;
|
|
322
|
+
the same YAML is refused where it cannot be enforced, and that is the
|
|
323
|
+
point.
|
|
324
|
+
|
|
325
|
+
## Interoperability
|
|
326
|
+
|
|
327
|
+
Every pair of policies, and every policy against the runtime's own responses,
|
|
328
|
+
was reviewed by reading the code and by exercising a server. The rules below
|
|
329
|
+
are the ones the current code enforces; where a combination misbehaves, the
|
|
330
|
+
rule says so and names what to do instead.
|
|
331
|
+
|
|
332
|
+
### Cannot coexist
|
|
333
|
+
|
|
334
|
+
Hard conflicts: the runtime refuses activation, or the combination produces
|
|
335
|
+
a wrong answer today.
|
|
336
|
+
|
|
337
|
+
- **Origin cache and conditional or range requests.** A request carrying
|
|
338
|
+
`If-None-Match`, `If-Modified-Since`, `If-Match`, `If-Unmodified-Since` or
|
|
339
|
+
`Range` is never answered from the origin cache: a stored entry is a full
|
|
340
|
+
`200` representation and the handler owns validators and ranges, so such
|
|
341
|
+
requests always reach it and get the handler's `304`, `412` or `206`.
|
|
342
|
+
Unconditional requests are served from the entry.
|
|
343
|
+
- **`security.unset` and a route that switches profile.** `unset` is
|
|
344
|
+
validated against the profile in effect on that route. A project-level
|
|
345
|
+
`unset` merges into a route that names a profile without that header
|
|
346
|
+
(`headers: off`, or `oshp-no-csp` with `unset: [Content-Security-Policy]`)
|
|
347
|
+
and activation fails with `unset names "...", which the off profile does
|
|
348
|
+
not emit`. Write `unset: []` on that route: a route key replaces the whole
|
|
349
|
+
array.
|
|
350
|
+
- **`security.set` of `Cache-Control`, `Content-Encoding`, `ETag`,
|
|
351
|
+
`Content-Type`, `Set-Cookie`, `Location` and the other reserved names** is
|
|
352
|
+
refused at activation (`is owned by the runtime or handler`). `Vary`,
|
|
353
|
+
`RateLimit`, `RateLimit-Policy`, `Retry-After` and `Age` are reserved for
|
|
354
|
+
the same reason: the cache and throttle policies own them, and a `set` of
|
|
355
|
+
any of the five is refused at activation with the route named. Declare
|
|
356
|
+
`vary` on the cache policy instead.
|
|
357
|
+
- **`agents.denyEmpty`, `throttle` and the generated probes.** `urlcode
|
|
358
|
+
audit`, `urlcode test` and `urlcode benchmark` send generated probes as
|
|
359
|
+
`Mozilla/5.0 (compatible; RouteProbe/0.1)`,
|
|
360
|
+
so `denyEmpty` does not fail them; a deny pattern that matches that string
|
|
361
|
+
would. A tight `throttle` fails an audit once the probes exceed `quota`
|
|
362
|
+
(the audit runs from one address, so a `static` tree with more files than
|
|
363
|
+
the quota trips it): keep the quota at least the number of generated cases
|
|
364
|
+
or run the audit in `mode: report`.
|
|
365
|
+
- **`hardened` on Vercel and AWS** is refused as written, because
|
|
366
|
+
`throttle.partition: client` cannot be honoured there; **on Cloudflare**
|
|
367
|
+
the build refuses `throttle` and `cache`. Override the offending keys with
|
|
368
|
+
`false` or `partition: route` as the profile section above describes.
|
|
369
|
+
- **`compression.encodings: [zstd]` on a Node without `zlib.zstdCompressSync`**
|
|
370
|
+
fails activation with the route named rather than serving identity.
|
|
371
|
+
|
|
372
|
+
### Coexist with defined precedence
|
|
373
|
+
|
|
374
|
+
Pairs that work, with the rule the code applies.
|
|
375
|
+
|
|
376
|
+
- **Request order is `agents`, `throttle`, cache lookup.** A denied agent is
|
|
377
|
+
never counted and never looked up; a throttled request is never looked up.
|
|
378
|
+
An agent on an `allow` list passes `agents` and is still throttled. An
|
|
379
|
+
`agents` denial and a `throttle` refusal both pass through the cache
|
|
380
|
+
policy's response hook (they carry no flight, so nothing is stored) and
|
|
381
|
+
pick up its declared `vary` names, then security headers, then
|
|
382
|
+
`Vary: Accept-Encoding` from compression; their bodies are below `minBytes`
|
|
383
|
+
so they are never encoded.
|
|
384
|
+
- **Cache hit and the response phase.** A hit skips only the cache's own
|
|
385
|
+
response hook: it still carries the client's `RateLimit` headers (the
|
|
386
|
+
request was counted), the security profile, and is compressed on the way
|
|
387
|
+
out. Stored entries are the handler's bytes and headers after YAML
|
|
388
|
+
`response.headers` and the cache policy ran, before throttle, security and
|
|
389
|
+
compression: bodies are stored uncompressed and encoded again on every hit,
|
|
390
|
+
and a `RateLimit` value is never stored. A cache hit on an asset serves the
|
|
391
|
+
same snapshot buffer, so a precompressed variant is used for `GET` and
|
|
392
|
+
reported at the variant's length for `HEAD`, exactly as when uncached.
|
|
393
|
+
- **Cache `Vary` and compression `Vary`.** The cache merges its declared
|
|
394
|
+
`vary` names first; compression appends `Accept-Encoding` without
|
|
395
|
+
duplicating it and leaves a `Vary: *` alone. The origin key uses only the
|
|
396
|
+
cache's names: because bodies are stored uncompressed, `Accept-Encoding` is
|
|
397
|
+
not part of the key and need not be.
|
|
398
|
+
- **ETags across `cache` and `compression`.** `revalidate` keeps a handler or
|
|
399
|
+
asset `ETag` and computes a strong SHA-256 tag for a `200` without one.
|
|
400
|
+
Compression then weakens a dynamically encoded body's tag (`W/"..."`) and
|
|
401
|
+
suffixes a precompressed asset's (`"...-gz"`). Revalidation works in every
|
|
402
|
+
combination: the cache's `304` compares weakly, the asset handler compares
|
|
403
|
+
weakly against the identity tag, and compression answers `304` for a
|
|
404
|
+
suffixed tag it produced. The `304` that `revalidate` produces keeps
|
|
405
|
+
`Content-Type`, so compression adds `Vary: Accept-Encoding` to it as it
|
|
406
|
+
does to the asset handler's own `304`.
|
|
407
|
+
- **Who owns `Cache-Control`.** YAML `response.headers` first, then an asset
|
|
408
|
+
handler's `cacheControl` when the cache policy is only inherited, then a
|
|
409
|
+
handler's `private` or `no-store`, then the strategy; `security.set` cannot
|
|
410
|
+
name it. A `no-store` or `private` handler answer is never stored whatever
|
|
411
|
+
the strategy says.
|
|
412
|
+
- **Security headers and everything else.** Profile headers fill gaps only:
|
|
413
|
+
YAML `response.headers`, function, asset, redirect and early-denial headers
|
|
414
|
+
keep their values. `set` overrides all of them. `Strict-Transport-Security`
|
|
415
|
+
needs an `https` origin. The profile is applied on cache hits and on early
|
|
416
|
+
denials (including a plugin short-circuit), on the self-hosted server and
|
|
417
|
+
in the Worker alike.
|
|
418
|
+
- **Compression and secrets or cookies.** A route with `secrets` or a
|
|
419
|
+
response with `Set-Cookie` is sent as identity (still with `Vary`) unless
|
|
420
|
+
`allowWithSecrets: true`. A route with `secrets` also never enters the
|
|
421
|
+
origin cache, so the two policies agree on what a secret-bearing route is.
|
|
422
|
+
- **Throttle and the 405.** The request phase runs before the method check,
|
|
423
|
+
so a request that ends as `405` was counted, and the `405` passes through
|
|
424
|
+
the response phase: it carries the `RateLimit` headers, the security
|
|
425
|
+
profile and plugin `onResponse` rewrites. Nothing stores it.
|
|
426
|
+
- **Plugins and policies.** `onRequest` runs before every policy; a
|
|
427
|
+
short-circuit result skips `agents`, `throttle` and the cache lookup
|
|
428
|
+
(nothing is counted, matched or stored) and skips the cache store and the
|
|
429
|
+
throttle headers, then receives security headers and compression, then
|
|
430
|
+
every plugin's `onResponse`. On a thrown error the policies' `onError`
|
|
431
|
+
hooks run first (in the order the policies were declared) and then plugin
|
|
432
|
+
`onError` hooks in reverse. Only the cache policy has an error hook today
|
|
433
|
+
and it never returns a fallback (`stale-if-error` is header-only); if a
|
|
434
|
+
future policy did, that fallback would pass through the response phase and
|
|
435
|
+
every plugin `onResponse`, and plugin `onError` would not run for that
|
|
436
|
+
request.
|
|
437
|
+
- **Reload.** Every reload builds a new runtime with fresh shared state:
|
|
438
|
+
throttle counters start empty and the origin cache and precompressed
|
|
439
|
+
variants are rebuilt; a client mid-window gets a fresh budget. The plugin
|
|
440
|
+
objects are the ones the operator passed and persist across reloads:
|
|
441
|
+
`onActivate` runs for the new runtime before `onClose` runs for the old,
|
|
442
|
+
so a plugin's own state (a shared-store connection, a `WeakMap`) survives a
|
|
443
|
+
reload unless its `onClose` discards it. A plugin whose `onActivate`
|
|
444
|
+
throws makes the reload fail and the old runtime keeps serving.
|
|
445
|
+
- **Merge edge cases.** A route `profile` that lacks a key the project layer
|
|
446
|
+
had leaves that key in force (layers merge, they do not replace); `false`
|
|
447
|
+
at the project level followed by an object at the route level re-enables
|
|
448
|
+
the policy with the route's object alone, so it must be complete
|
|
449
|
+
(`throttle` needs `quota` and `window`); a custom profile named like a
|
|
450
|
+
built-in shadows it everywhere.
|
|
451
|
+
- **Memory bounds add up.** Per runtime: 64 MiB of asset snapshot, up to
|
|
452
|
+
64 MiB of precompressed variants, up to 64 MiB of origin-cache bodies (a
|
|
453
|
+
stored asset references the snapshot buffer rather than copying it, but
|
|
454
|
+
is counted against the cache budget), and the throttle table at the
|
|
455
|
+
largest `maxKeys` (100,000 keys by default). Two routes serving the same
|
|
456
|
+
file compress it separately and both count. During a reload both
|
|
457
|
+
generations exist at once. [Capacity](CAPACITY.md) states the asset,
|
|
458
|
+
throttle and cache figures; the precompressed budget belongs in that table
|
|
459
|
+
too.
|
|
460
|
+
- **Cloudflare.** The artifact carries the effective `agents` and `security`
|
|
461
|
+
configuration per route, with the entries of any project list file
|
|
462
|
+
embedded under the reference as written in YAML (no filesystem path); a
|
|
463
|
+
route-level `agents: false` leaves the artifact without that key. The
|
|
464
|
+
Worker runs `agents` on the request and `security` on the response, so an
|
|
465
|
+
early denial carries the profile exactly as on the self-hosted server.
|
|
466
|
+
|
|
467
|
+
### Not covered by policies
|
|
468
|
+
|
|
469
|
+
- **Every error the runtime throws** (404 for no match, disabled route or
|
|
470
|
+
missing link, 410, 400/413/415 from body checks, 502/503/504 from the
|
|
471
|
+
sandbox or a link store) bypasses the response phase: no `Vary`, no
|
|
472
|
+
`RateLimit`, no compression, no plugin `onResponse`, on every target.
|
|
473
|
+
What they do get is the `security` policy: the matched route's effective
|
|
474
|
+
profile when the error came after routing (so a route with
|
|
475
|
+
`security: false` answers its 410 bare), otherwise the project-level
|
|
476
|
+
profile, including a host-side error such as an oversized body or shed
|
|
477
|
+
admission and the Worker's own 404. The runtime's fixed headers
|
|
478
|
+
(`Content-Type`, `Cache-Control: no-store`, `Content-Length`,
|
|
479
|
+
`X-Request-Id`, `X-Content-Type-Options`) can never be replaced by it.
|
|
480
|
+
Plugins keep `onError` for observation; a policy error hook may answer with
|
|
481
|
+
a fallback, and none does today.
|
|
482
|
+
- **The audit's probes** share one address and one `User-Agent`, so they do
|
|
483
|
+
not exercise `agents` or `throttle` the way real traffic does;
|
|
484
|
+
the policy table in `testPlan().policies` is the audit's evidence for
|
|
485
|
+
those two.
|
|
486
|
+
- **`stale-if-error` at the origin**: header-only, as the
|
|
487
|
+
[cache page](policies/cache.md) states.
|
|
488
|
+
- **Cross-instance state**: counters and the cache are per runtime on every
|
|
489
|
+
target; a shared budget or a shared cache is a plugin.
|