@wave-rf/cloudflare-md-router 0.2.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +78 -1
- package/bin/cli.mjs +34 -0
- package/bin/dev.mjs +575 -0
- package/bin/lib.mjs +682 -0
- package/package.json +15 -5
- package/src/bots.ts +1 -1
- package/src/index.ts +2 -0
- package/src/worker.ts +134 -41
package/README.md
CHANGED
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
A tiny Cloudflare Worker that serves the `.md` twin of any static page when the request is from a known LLM crawler or explicitly asks for `text/markdown`. Falls back to the HTML response when the `.md` twin doesn't exist.
|
|
4
4
|
|
|
5
|
+
It also ships `cloudflare-md-router dev`, a local dev loop that serves real builds through `wrangler dev`, so the Worker's routing works while you edit (see [Dev loop](#dev-loop-cloudflare-md-router-dev)).
|
|
6
|
+
|
|
5
7
|
If you're building a docs site that already emits a per-page raw-markdown twin (e.g. `/foo/bar` and `/foo/bar.md`), this lets every page do content negotiation transparently — Claude, ChatGPT, Perplexity, etc. fetch the model-friendly version automatically; humans keep getting the styled HTML page.
|
|
6
8
|
|
|
7
9
|
## Behavior
|
|
@@ -24,7 +26,13 @@ This is on by default (only for a `200 text/html` reply to an extension-less GET
|
|
|
24
26
|
|
|
25
27
|
Because the worker negotiates on the `Accept` header, two clients can get different representations of the same URL. If you put a shared cache (a CDN, Cloudflare's own cache) in front of it, set `vary: true` to add a `Vary: Accept` header to the negotiated responses so the cache keys on `Accept` and doesn't serve the HTML page to a client that asked for markdown — or vice-versa. It's off by default (verbatim pass-through stays byte-for-byte; `Vary: User-Agent` is intentionally not added, as it would defeat shared caching).
|
|
26
28
|
|
|
27
|
-
The included bot list covers the common ones: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, CCBot, Applebot-Extended, Google-Extended, cohere-ai, Bytespider, Diffbot. See `src/bots.ts`.
|
|
29
|
+
The included bot list covers the common ones: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, Claude-User, Claude-SearchBot, Amazonbot, DuckAssistBot, anthropic-ai, PerplexityBot, CCBot, Applebot-Extended, Google-Extended, cohere-ai, Bytespider, Diffbot. See `src/bots.ts`.
|
|
30
|
+
|
|
31
|
+
Three more opt-in features, all off by default:
|
|
32
|
+
|
|
33
|
+
- `checkTwin: true` fetches the `.md` twin from `ASSETS` before advertising it, so a route with no twin (an error page, say) gets no dead `Link`. It costs one extra local asset lookup per HTML page.
|
|
34
|
+
- `advertiseLlmsTxt: true` (or a path) adds `Link: </llms.txt>; rel="alternate"; type="text/plain"` to HTML pages.
|
|
35
|
+
- `canonicalOrigin: "https://example.com"` rewrites absolute `og:image` / `twitter:image` URLs under that origin to the origin being browsed when the request host differs (a preview deploy), via `HTMLRewriter`. `canonical` and `og:url` are left alone.
|
|
28
36
|
|
|
29
37
|
## Install
|
|
30
38
|
|
|
@@ -86,9 +94,78 @@ export default createMdRouter({
|
|
|
86
94
|
// Add `Vary: Accept` to negotiated responses so a shared cache doesn't
|
|
87
95
|
// cross-serve the HTML and markdown representations (default: false).
|
|
88
96
|
vary: true,
|
|
97
|
+
|
|
98
|
+
// Only advertise a twin that exists (default: false).
|
|
99
|
+
checkTwin: true,
|
|
100
|
+
|
|
101
|
+
// Also advertise /llms.txt via a Link header (default: false).
|
|
102
|
+
advertiseLlmsTxt: true,
|
|
103
|
+
|
|
104
|
+
// Point og:image / twitter:image at the browsed origin on preview hosts.
|
|
105
|
+
canonicalOrigin: "https://example.com",
|
|
89
106
|
});
|
|
90
107
|
```
|
|
91
108
|
|
|
109
|
+
To run the negotiation inside a larger Worker, use `createMdResponder(options)`, which returns a plain `(request, env) => Promise<Response>`:
|
|
110
|
+
|
|
111
|
+
```ts
|
|
112
|
+
const respond = createMdResponder({ vary: true });
|
|
113
|
+
export default {
|
|
114
|
+
fetch: (request, env) => (isApi(request) ? api(request, env) : respond(request, env)),
|
|
115
|
+
};
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
## Dev loop (`cloudflare-md-router dev`)
|
|
119
|
+
|
|
120
|
+
`astro dev` (or any framework dev server) skips everything the Worker adds in production: the `.md` twins, the `Link` header, and any outputs that only exist in a real build. If your Worker serves something the dev server can't, use this instead of it. If the Worker adds nothing you are iterating on, keep using the dev server — it has HMR and an error overlay; this does not.
|
|
121
|
+
|
|
122
|
+
The command runs a **full build on every save** into a staging directory, copies it into your output directory **only if the build succeeded**, and serves that directory through `wrangler dev --live-reload`. A failed build (or the emptied-output window during one) never reaches the browser: you keep the last good build, with a red banner and a terminal bell. Ctrl-C stops the build (its whole process tree) and wrangler, with SIGTERM and then SIGKILL after 5 s, and the command does not exit until they are gone. Each save costs a real build, with errors in the terminal rather than the page.
|
|
123
|
+
|
|
124
|
+
```sh
|
|
125
|
+
pnpm add -D @wave-rf/cloudflare-md-router wrangler
|
|
126
|
+
pnpm exec cloudflare-md-router dev
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
```jsonc
|
|
130
|
+
// package.json
|
|
131
|
+
{ "scripts": { "dev:worker": "cloudflare-md-router dev" } }
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Needs Node 20+ (recursive `fs.watch`; checked at startup, tested in CI on Node 24 only; POSIX is the supported platform, Windows is best-effort) and a locally-installed `wrangler`. It is never downloaded: the site's own `wrangler` package is resolved the way `require` would (so a workspace-root install works) and its `bin` is run directly with Node, with no package manager or shell in between to swallow Ctrl-C. If the package can't be resolved (Yarn PnP, or not installed) it falls back to your package manager's local exec (`pnpm exec`, `npm exec --no`, `yarn`, `bun x --no-install`). That fallback is best-effort on shutdown: on Debian-based images, where `/bin/sh` is dash, npm and pnpm run it through an `sh` that does not pass signals on, so the CLI also signals wrangler's descendants directly (the smoke test `wrangler run through the <pm> fallback is stopped too` covers this under npm and pnpm). Add the staging directory (`.dev-<outDir>`, so `.dev-dist/` by default) to `.gitignore`.
|
|
135
|
+
|
|
136
|
+
| Option | Default |
|
|
137
|
+
| ------ | ------- |
|
|
138
|
+
| `--build <cmd>` (`MD_ROUTER_DEV_BUILD`) | the site's `build` script, via the package manager its lockfile indicates (pnpm, yarn, bun or npm; the lockfile is searched up the tree, so monorepos work), else the `packageManager` field in `package.json`, else npm. A custom command runs through the shell with `node_modules/.bin` on `PATH`, so `--build "astro build"` works. |
|
|
139
|
+
| `--out-dir <dir>` | `assets.directory` from `wrangler.jsonc` / `wrangler.json` / `wrangler.toml` (JSONC comments and trailing commas are fine), else `dist`. A relative path resolves against the wrangler config's directory. The staging directory is `.dev-<name>` next to it. Both are emptied and refilled on every build, so the command refuses (and says why) an output directory that is: the site root, or any directory outside it (a sibling such as `../shared/dist` too); `src/`, `public/`, `node_modules/` or `.git/`, inside one of them, or enclosing one; or the wrangler config's directory, or one enclosing it. Neither the output nor the staging directory may hold a `package.json` or any git-tracked file. The Worker's directory (the directory of wrangler `main`) depends on where it is: at or above the root or the config's directory, it falls under the rules above (no equal or enclosing output dir); inside `assets.directory`, where the config itself declares the Worker to be build output (Astro's `dist/_worker.js`), only an output dir equal to it or inside it is refused; anywhere else it is treated like `src/`. Directories are compared by identity (device and inode), not by spelling, so `SRC` on a case-insensitive disk is `src` even on Linux over a macOS bind mount, and git-tracked files are matched case-insensitively. |
|
|
140
|
+
| `--out-dir-flag <flag>` | `--outDir`: appended to the build as `<flag> <staging>` so it writes to staging, not the served directory. Pass `""` to disable. Put `{outDir}` anywhere in `--build` to place the path yourself (and skip the flag). The staging path (relative to the site root) is also exported as `MD_ROUTER_DEV_OUT_DIR`. |
|
|
141
|
+
| `--watch <path>` | Added to the defaults: `src/`, `public/`, `astro.config.*`, `vite.config.*`, `tsconfig.json`, `package.json`, `.env*`. Directories recurse; other entries are root-level file names (a trailing `*` matches a prefix). A nested file path (`config/site.json`) can't be watched (watch its directory) and a directory that doesn't exist is skipped; either one, when you passed it, is reported at startup. Repeatable or comma-separated. `--no-default-watch` drops the defaults. The Worker directory is deliberately not watched — wrangler reloads the Worker itself. Changes never trigger a build when they are under the output or staging directory, `node_modules/`, `.git/`, `.wrangler/` or a common framework cache (`.astro/`, `.svelte-kit/`, `.next/`, `.nuxt/`, `.output/`, `.cache/`, `.turbo/`, `.vercel/`, `.parcel-cache/`). In a git work tree, a path below a watched directory that git ignores is dropped too — except under a watched directory that is itself gitignored. Root-level files matching an exact name (`tsconfig.json`, `package.json`), `.env*`, or a name you pass are never git-filtered; the prefix-glob defaults (`astro.config.*`, `vite.config.*`) are, and Vite's temporary `*.timestamp-<ms>[-<hex>].[cm]?js` config files (Vite 3-5) are always skipped, anywhere under a watched path. That makes `--watch .` usable, as long as everything your build writes outside the output directory is in that list or gitignored. Otherwise each build triggers the next. |
|
|
142
|
+
| `--port <n>` (`PORT`) | `4321`. If it is taken, the next free port (up to 19 higher, both IPv4 and IPv6 loopback checked) is used and printed — wrangler itself would just die on a busy explicit port. |
|
|
143
|
+
| `--config <file>` | `wrangler.jsonc`, `wrangler.json`, then `wrangler.toml` in the root. A `--config` that doesn't exist is an error. |
|
|
144
|
+
| `--root <dir>` | the current directory. |
|
|
145
|
+
| `--strict` (`MD_ROUTER_DEV_STRICT=1`) | off. See below. |
|
|
146
|
+
| `-- <args>` | everything after `--` is passed to `wrangler dev`. |
|
|
147
|
+
| `--help`, `-h` | prints the options and exits. |
|
|
148
|
+
|
|
149
|
+
Flags win over env. A failed build, and a build that exits 0 but writes nothing, are both handled: the previous good build stays up. A build that ignores the staging directory and writes straight into the output directory (say `astro build && node postbuild.mjs` without `{outDir}`) is called out as such: what is served is whatever it left there, with no last-good-build protection, until you point it at `{outDir}` or set `--out-dir-flag`. On a cold start with no output directory yet, the first build must succeed or the command exits 1. When wrangler exits, so does the command, with wrangler's exit code (1 if wrangler was killed by a signal); a Ctrl-C you send exits 0.
|
|
150
|
+
|
|
151
|
+
### Relaxing checks in watch builds
|
|
152
|
+
|
|
153
|
+
Every build in the loop runs with `MD_ROUTER_DEV=1` (unless `--strict`). Use it in your site config to turn off checks that are right for CI but wrong mid-edit — typically a link validator, since a page you are still writing links to pages that don't exist yet. The rendered output should be identical; only the gate differs:
|
|
154
|
+
|
|
155
|
+
```js
|
|
156
|
+
// astro.config.mjs
|
|
157
|
+
const watchBuild = Boolean(process.env.MD_ROUTER_DEV);
|
|
158
|
+
|
|
159
|
+
export default defineConfig({
|
|
160
|
+
integrations: [
|
|
161
|
+
// Fails the build on a broken link: keep it for CI and `astro build`, skip it in the dev loop.
|
|
162
|
+
...(watchBuild ? [] : [starlightLinksValidator({ errorOnRelativeLinks: true })]),
|
|
163
|
+
],
|
|
164
|
+
});
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Run `cloudflare-md-router dev --strict` (or `MD_ROUTER_DEV_STRICT=1`) to leave the variable unset and get the CI behaviour in the loop.
|
|
168
|
+
|
|
92
169
|
## Why content-negotiate?
|
|
93
170
|
|
|
94
171
|
Most LLMs do better with raw markdown than with rendered HTML — less DOM noise, no Starlight nav chrome, no script tags. Serving the same content at one URL with two representations means:
|
package/bin/cli.mjs
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/* `cloudflare-md-router <command>` — currently one command: `dev`. */
|
|
3
|
+
import { dev } from "./dev.mjs";
|
|
4
|
+
import { HELP, parseArgs, resolveOptions } from "./lib.mjs";
|
|
5
|
+
|
|
6
|
+
if (Number(process.versions.node.split(".")[0]) < 20) {
|
|
7
|
+
console.error(`cloudflare-md-router dev needs Node 20+ (running ${process.versions.node})`);
|
|
8
|
+
process.exit(1);
|
|
9
|
+
}
|
|
10
|
+
|
|
11
|
+
const [command, ...rest] = process.argv.slice(2);
|
|
12
|
+
|
|
13
|
+
if (command !== "dev") {
|
|
14
|
+
console.log(
|
|
15
|
+
command && command !== "--help" && command !== "-h" ? `unknown command ${command}\n` : ""
|
|
16
|
+
);
|
|
17
|
+
console.log(HELP);
|
|
18
|
+
process.exit(command === undefined || command === "--help" || command === "-h" ? 0 : 1);
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
let opts;
|
|
22
|
+
try {
|
|
23
|
+
const flags = parseArgs(rest);
|
|
24
|
+
if (flags.help) {
|
|
25
|
+
console.log(HELP);
|
|
26
|
+
process.exit(0);
|
|
27
|
+
}
|
|
28
|
+
opts = resolveOptions(flags, process.env, process.cwd());
|
|
29
|
+
} catch (err) {
|
|
30
|
+
console.error(`\x1b[1;31mcloudflare-md-router dev: ${err.message}\x1b[0m\x07`);
|
|
31
|
+
process.exit(1);
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
process.exitCode = await dev(opts);
|