@redirhub/cloudflare-redirects 0.2.0 → 0.3.0

package/README.md

@@ -9,8 +9,8 @@ in front of your origin, from a remote rules file, and monitor its 404s.** Old U
 whole website migrations redirect at Cloudflare's edge; every redirect and every 404 is reported
 back, so broken links show up before your visitors find them.
 
-- Redirects, short links and whole website migrations, live within a minute, without
-  redeploying the Worker.
+- Redirects, short links, frame pages and whole website migrations, live within a minute,
+  without redeploying the Worker.
 - 404 monitoring: the origin's 404s that no rule redirected are reported in the background.
 - Fails open: no rules yet, a download error or a bug never breaks a page; the request goes on to
   the origin.
@@ -148,7 +148,7 @@ the origin unchanged.
 1. **Check path** (`/_redirhub/check`): `200`, `text/plain`, `Cache-Control: no-store`, body
    `token=<check token> etag=<ETag of the rules in use>`. It proves the hostname is yours and
    tells RedirHub which rules you serve.
-2. **Exact rules** redirect before the origin, even when the page exists.
+2. **Exact rules** redirect (or show their frame page) before the origin, even when the page exists.
 3. Otherwise the origin answers. Only when it answers **404** and the hostname has a **catch-all
    rule** (`{host}/*`), the 404 is discarded and the Worker redirects instead; a working page is
    never taken over. Without a catch-all, the 404 goes to the visitor and is reported when the
@@ -158,20 +158,32 @@ Matching follows format v1: the hostname in any case, port ignored; the path cas
 percent-decoded, leading and trailing slashes ignored; scheme and query ignored. Every redirect
 carries `X-Powered-By` when the rules file has `@powered-by`.
 
+**Frame pages.** A rule with status `frame` shows its target in a frame instead of redirecting:
+`200`, `text/html; charset=utf-8`, a page whose one full-window `<iframe>` loads the address a
+redirect would send (flags `p` and `q` applied), titled with the target's hostname; the visitor's
+address bar keeps their URL. It is matched like a redirect (an exact rule before the origin, a
+catch-all only on the origin's 404), carries `X-Powered-By` and is reported, with status `200`.
+HEAD gets the same status and headers without the page.
+
+**Options** in a rule's fifth column (`password=`, `destinations=`, `routing=`, `plugins=`, and any
+other) are ignored, as the format allows: the rule redirects, or shows its frame, to its target.
+
 **Rules download.** The first request of a new isolate starts the download and waits for it at
 most one second; if the rules server is slower, that request goes to the origin without rules and
 the download (5 second timeout) finishes in `ctx.waitUntil`. Requests that arrive meanwhile do not
 wait. After that the copy in memory is used, and once it is older than `@refresh` seconds (60 by
 default) it is revalidated in `ctx.waitUntil` with `If-None-Match`, so no visitor waits. Any
 failure keeps the last good copy and waits 30 seconds before trying again. `@notify` (push) is
-ignored: a Worker keeps no connection open.
+ignored: a Worker keeps no connection open. Rules downloads and reports identify themselves as
+`User-Agent: cloudflare-redirects/<version>`.
 
 **Reports.** A line queued while nothing is being sent goes out at once; lines queued while a
 request is out go together in the next one (at most 100 lines or 64 KB, gzip with
-`CompressionStream`, 2 second timeout). Only the request that starts sending waits for it, in
-`ctx.waitUntil`, for at most 10 seconds; the others only add their line, so no request waits on
-another's work. A failed or rejected batch is dropped, never retried; beyond 10,000 waiting lines,
-lines are dropped.
+`CompressionStream`, 2 second timeout), and each full batch opens another request, up to four at
+once. Only the request that starts sending waits for them, in `ctx.waitUntil`, for at most 10
+seconds; the others only add their line, so no request waits on another's work. A failed or
+rejected batch is dropped, never retried; beyond 10,000 waiting lines, lines are dropped. Dropped
+lines are logged at once, then at most every ten minutes with the total by reason.
 
 ## Privacy
 
@@ -184,9 +196,10 @@ Each report is one Apache combined line with the absolute URL:
 That is the visitor's IP address (`CF-Connecting-IP`, or anonymized with `anonymizeIp`), the time
 (UTC), the method, the requested URL with its query string, the status, the response size, the
 Referer and the User-Agent (each cut to 1 KB); for a redirect also the rule's source and the
-`Location` sent. The protocol is always written `HTTP/1.1`. No cookies, other headers or bodies
-are sent. Only redirects, and 404s of hostnames with 404 reporting on, are reported; all other
-traffic stays between the visitor and your origin. Reports go only to the `@hits` address of your
+`Location` sent, for a frame page the rule's source and the frame's address. The protocol is
+always written `HTTP/1.1`. No cookies, other headers or bodies are sent. Only redirects, frame
+pages, and 404s of hostnames with 404 reporting on, are reported; all other traffic stays between
+the visitor and your origin. Reports go only to the `@hits` address of your
 rules file, with your key.
 
 ## Limits

package/dist/core/redirects.d.ts

@@ -1,7 +1,8 @@
 /**
  * The integration on any runtime with the Fetch API (Request, Response, fetch): answers the check
  * path, redirects exact rules before the site, redirects a hostname's catch-all where the site
- * answers 404, and reports redirects and 404s. The platform entry (`src/index.ts`) only says
+ * answers 404, shows `frame` rules in a frame page instead of redirecting, and reports redirects,
+ * frame pages and 404s. The platform entry (`src/index.ts`) only says
  * where the visitor's address and the background work come from.
  *
  * `src/core/` is the same in every JavaScript implementation (nextjs-redirects,

package/dist/core/redirects.js

@@ -1,14 +1,15 @@
 /**
  * The integration on any runtime with the Fetch API (Request, Response, fetch): answers the check
  * path, redirects exact rules before the site, redirects a hostname's catch-all where the site
- * answers 404, and reports redirects and 404s. The platform entry (`src/index.ts`) only says
+ * answers 404, shows `frame` rules in a frame page instead of redirecting, and reports redirects,
+ * frame pages and 404s. The platform entry (`src/index.ts`) only says
  * where the visitor's address and the background work come from.
  *
  * `src/core/` is the same in every JavaScript implementation (nextjs-redirects,
  * cloudflare-redirects); CI compares it with the others.
  */
 import { hitLine, Reporter } from "./report.js";
-import { buildLocation, matchExact, normalizeHost } from "./rules.js";
+import { buildLocation, framePage, matchExact, normalizeHost } from "./rules.js";
 import { RulesSource } from "./source.js";
 import { PACKAGE } from "../version.js";
 const WARN_EVERY_MS = 10 * 60_000;
@@ -60,14 +61,23 @@ export function createRedirects(options) {
             redirect,
         }, now()), keepAlive);
     };
+    /** A rule's answer: its redirect, or for a `frame` rule a `200` page framing the same address. */
     const redirect = (request, url, rules, host, rule, keepAlive) => {
         const target = buildLocation(rule, url);
         const location = options.location ? options.location(target, request) : target;
-        const headers = new Headers({ Location: byteString(location) });
+        const headers = new Headers();
         if (rules.poweredBy) {
             headers.set('X-Powered-By', rules.poweredBy);
         }
-        report(request, url, host.name, rule.status, 0, keepAlive, { source: rule.source, location });
+        // A frame page is reported like a redirect, with status 200 and size 0 as the format says.
+        report(request, url, host.name, rule.status === 'frame' ? 200 : rule.status, 0, keepAlive, { source: rule.source, location });
+        if (rule.status === 'frame') {
+            const page = framePage(location, rule.target);
+            headers.set('Content-Type', 'text/html; charset=utf-8');
+            headers.set('Content-Length', String(new TextEncoder().encode(page).length));
+            return new Response(request.method === 'HEAD' ? null : page, { status: 200, headers });
+        }
+        headers.set('Location', byteString(location));
         return new Response(null, { status: rule.status, headers });
     };
     const handle = async (request, context) => {

package/dist/core/report.d.ts

@@ -1,6 +1,6 @@
 /**
- * Visit reports to the rules file's `@hits` address: one Apache combined line per redirect or
- * 404, as specified in redirhub/integration-spec (spec/v1/reports.txt is the golden sample).
+ * Visit reports to the rules file's `@hits` address: one Apache combined line per redirect, frame
+ * page or 404, as specified in redirhub/integration-spec (spec/v1/reports.txt is the golden sample).
  */
 import type { Logger } from './log.ts';
 export interface Hit {
@@ -14,11 +14,11 @@ export interface Hit {
     /** The request URL; its escaped path and query are the request target. */
     url: URL;
     status: number;
-    /** Body bytes sent: 0 for a redirect. */
+    /** Body bytes sent: 0 for a redirect and a frame page. */
     size: number;
     referer: string | null;
     userAgent: string | null;
-    /** For a redirect: the rule's source column and the `Location` sent. */
+    /** For a redirect or frame page: the rule's source column and the `Location` sent, or the frame's `src`. */
     redirect?: {
         source: string;
         location: string;
@@ -47,15 +47,18 @@ export interface ReporterOptions {
     maxQueue?: number;
     /** How long one invocation keeps sending before it leaves the rest to the next; default 10,000. */
     budgetMs?: number;
+    /** Requests out at the same time; default 4. */
+    maxInFlight?: number;
 }
 /**
  * Sends report lines without delaying a response and without a timer.
  *
  * A line queued while nothing is being sent goes out at once, so a site with little traffic
- * reports each visit immediately and keeps nothing waiting. Lines queued while a request is out
- * wait for it and go together in the next one (at most 100 lines, 64 KB, gzip), so a burst costs
- * one request per hundred visits. The invocation that starts sending keeps itself alive (through
- * `keepAlive`) until the queue is empty; the others only add their line, so no request waits on
+ * reports each visit immediately and keeps nothing waiting. Lines queued while requests are out
+ * go together in the next one (at most 100 lines, 64 KB, gzip), so a burst costs one request per
+ * hundred visits; up to `maxInFlight` (4) requests are out at once, so a busy instance keeps up.
+ * The invocation that starts sending keeps itself alive (through `keepAlive`) until the queue is
+ * empty and its requests are back; the others only add their line, so no request waits on
  * another request's work. A failed or rejected batch is dropped and counted, never retried.
  */
 export declare class Reporter {

package/dist/core/report.js

@@ -1,6 +1,6 @@
 /**
- * Visit reports to the rules file's `@hits` address: one Apache combined line per redirect or
- * 404, as specified in redirhub/integration-spec (spec/v1/reports.txt is the golden sample).
+ * Visit reports to the rules file's `@hits` address: one Apache combined line per redirect, frame
+ * page or 404, as specified in redirhub/integration-spec (spec/v1/reports.txt is the golden sample).
  */
 import { PACKAGE, USER_AGENT } from "../version.js";
 /** The Referer and User-Agent are cut to this many bytes. */
@@ -73,17 +73,25 @@ const DROP_LOG_EVERY_MS = 10 * 60_000;
  * Sends report lines without delaying a response and without a timer.
  *
  * A line queued while nothing is being sent goes out at once, so a site with little traffic
- * reports each visit immediately and keeps nothing waiting. Lines queued while a request is out
- * wait for it and go together in the next one (at most 100 lines, 64 KB, gzip), so a burst costs
- * one request per hundred visits. The invocation that starts sending keeps itself alive (through
- * `keepAlive`) until the queue is empty; the others only add their line, so no request waits on
+ * reports each visit immediately and keeps nothing waiting. Lines queued while requests are out
+ * go together in the next one (at most 100 lines, 64 KB, gzip), so a burst costs one request per
+ * hundred visits; up to `maxInFlight` (4) requests are out at once, so a busy instance keeps up.
+ * The invocation that starts sending keeps itself alive (through `keepAlive`) until the queue is
+ * empty and its requests are back; the others only add their line, so no request waits on
  * another request's work. A failed or rejected batch is dropped and counted, never retried.
  */
 export class Reporter {
     #options;
     #queue = [];
+    /** The sender: the invocation that started it keeps it alive until it is done. */
     #sending = null;
-    #dropped = 0;
+    /** The sender's requests out now, and when it started (for the budget); #running once it has. */
+    #out = new Set();
+    #started = 0;
+    #running = false;
+    /** Lines dropped since the last log, by reason. */
+    #dropped = new Map();
+    #droppedSince = 0;
     #lastDropLog = Number.NEGATIVE_INFINITY;
     constructor(options) {
         // `??` per field, not a spread: an option passed as undefined keeps its default.
@@ -95,6 +103,7 @@ export class Reporter {
             timeoutMs: options.timeoutMs ?? 2000,
             maxQueue: options.maxQueue ?? 10_000,
             budgetMs: options.budgetMs ?? 10_000,
+            maxInFlight: options.maxInFlight ?? 4,
         };
     }
     /** Queues one line; `keepAlive` holds the invocation open while this call has to send. */
@@ -102,6 +111,7 @@ export class Reporter {
         if (this.#options.key === '') {
             return;
         }
+        this.#logDrops();
         if (this.#queue.length >= this.#options.maxQueue) {
             this.#drop(1, 'queue full');
             return;
@@ -111,6 +121,9 @@ export class Reporter {
             this.#sending = this.#send();
             keepAlive?.(this.#sending);
         }
+        else if (this.#running) {
+            this.#fill();
+        }
     }
     /** Resolves once everything queued so far has been sent or dropped. */
     async flush() {
@@ -125,14 +138,31 @@ export class Reporter {
     async #send() {
         // Lines queued in the same turn of the event loop go in the first request.
         await Promise.resolve();
-        const started = Date.now();
+        this.#started = Date.now();
+        this.#running = true;
         for (;;) {
-            if (this.#queue.length === 0 || Date.now() - started >= this.#options.budgetMs) {
+            this.#fill();
+            if (this.#out.size === 0) {
                 // Checked and cleared in one step: a line queued after this starts a new sender.
+                this.#running = false;
                 this.#sending = null;
                 return;
             }
-            await this.#post(this.#take());
+            await Promise.race(this.#out);
+        }
+    }
+    /**
+     * Starts requests for the waiting lines, within the budget: one when none is out, and more,
+     * up to maxInFlight, only for full batches, so lines that arrive while a request is out wait
+     * for it and go together. Called by the sender and on each line queued while it runs.
+     */
+    #fill() {
+        while (this.#out.size < this.#options.maxInFlight &&
+            this.#queue.length > 0 &&
+            (this.#out.size === 0 || this.#queue.length >= MAX_BATCH_LINES) &&
+            Date.now() - this.#started < this.#options.budgetMs) {
+            const post = this.#post(this.#take()).finally(() => this.#out.delete(post));
+            this.#out.add(post);
         }
     }
     #take() {
@@ -180,15 +210,27 @@ export class Reporter {
             this.#drop(lines.length, error instanceof Error ? error.name : 'error');
         }
     }
-    /** Counts dropped lines and logs them at most once per ten minutes. */
+    /** Counts dropped lines by reason; see #logDrops. */
     #drop(lines, reason) {
-        this.#dropped += lines;
-        const now = Date.now();
-        if (now - this.#lastDropLog < DROP_LOG_EVERY_MS) {
+        if (this.#dropped.size === 0) {
+            this.#droppedSince = Date.now();
+        }
+        this.#dropped.set(reason, (this.#dropped.get(reason) ?? 0) + lines);
+        this.#logDrops();
+    }
+    /**
+     * Logs the dropped lines: the first drop at once, then at most once per ten minutes with the
+     * total since the last log, by reason. Called on every drop and every line queued, so a count
+     * is logged even after the drops stop, without a timer.
+     */
+    #logDrops() {
+        if (this.#dropped.size === 0 || Date.now() - this.#lastDropLog < DROP_LOG_EVERY_MS) {
             return;
         }
-        this.#options.logger?.warn(`${PACKAGE}: ${this.#dropped} report lines dropped (${reason})`);
-        this.#dropped = 0;
-        this.#lastDropLog = now;
+        const total = [...this.#dropped.values()].reduce((sum, n) => sum + n, 0);
+        const reasons = [...this.#dropped].map(([reason, n]) => `${reason} ${n}`).join(', ');
+        this.#options.logger?.warn(`${PACKAGE}: ${total} report lines dropped since ${new Date(this.#droppedSince).toISOString()} (${reasons})`);
+        this.#dropped.clear();
+        this.#lastDropLog = Date.now();
     }
 }

package/dist/core/rules.d.ts

@@ -12,6 +12,8 @@ export declare const DEFAULT_REFRESH = 60;
 export declare const MIN_REFRESH = 30;
 export declare const MAX_REFRESH = 3600;
 export type RedirectStatus = 301 | 302 | 307 | 308;
+/** A rule's status column: a redirect status, or `frame` (a page that shows the target in a frame). */
+export type RuleStatus = RedirectStatus | 'frame';
 /** A rules file whose `@format` major version is not 1: the caller keeps its last good copy. */
 export declare class UnknownFormatError extends Error {
     name: string;
@@ -20,7 +22,7 @@ export interface Rule {
     /** The source column as written; reports name the rule with it (`file=`). */
     source: string;
     target: string;
-    status: RedirectStatus;
+    status: RuleStatus;
     /** `p`: append the request path (catch-all only). */
     appendPath: boolean;
     /** `q`: keep the request query string. */
@@ -46,7 +48,7 @@ export interface RuleSet {
     /** Where redirects and 404s are reported; null disables reporting. */
     hitsUrl: string | null;
     checkPath: string;
-    /** Sent as `X-Powered-By` on every redirect from the rules; null sends none. */
+    /** Sent as `X-Powered-By` on every redirect and frame page from the rules; null sends none. */
     poweredBy: string | null;
     /** The `@notify` push channel. Parsed for completeness; this package polls. */
     notifyUrl: string | null;
@@ -75,8 +77,16 @@ export declare function cleanEtag(etag: string): string;
 /** The exact rule for a request URL on a listed hostname, or null. */
 export declare function matchExact(host: HostRules, url: URL): Rule | null;
 /**
- * The `Location` of a rule for a request URL. With `appendPath` the escaped request path is
- * appended to the target's path without a double slash; with `keepQuery` the request query is
- * added with `?` or `&`, otherwise dropped. A fragment of the target stays last.
+ * The `Location` of a rule for a request URL, and a `frame` rule's iframe `src`. With `appendPath`
+ * the escaped request path is appended to the target's path without a double slash; with
+ * `keepQuery` the request query is added with `?` or `&`, otherwise dropped. A fragment of the
+ * target stays last.
  */
 export declare function buildLocation(rule: Rule, url: URL): string;
+/**
+ * The page of a `frame` rule: one full-window `<iframe>` showing `src`, titled with the target's
+ * hostname. Both are HTML-escaped.
+ */
+export declare function framePage(src: string, target: string): string;
+/** `&`, `<`, `>`, `"` and `'` as character references. */
+export declare function escapeHtml(value: string): string;

package/dist/core/rules.js

@@ -143,7 +143,10 @@ function parseRuleLine(line) {
     }
     const rule = { source, target, status: 301, appendPath: false, keepQuery: false };
     const status = columns[2]?.trim() ?? '';
-    if (status !== '') {
+    if (status === 'frame') {
+        rule.status = 'frame';
+    }
+    else if (status !== '') {
         if (!/^[+-]?\d+$/.test(status) || !STATUSES.has(Number(status))) {
             return null;
         }
@@ -263,9 +266,10 @@ export function matchExact(host, url) {
     return host.exact.get(matchPath(url.pathname)) ?? null;
 }
 /**
- * The `Location` of a rule for a request URL. With `appendPath` the escaped request path is
- * appended to the target's path without a double slash; with `keepQuery` the request query is
- * added with `?` or `&`, otherwise dropped. A fragment of the target stays last.
+ * The `Location` of a rule for a request URL, and a `frame` rule's iframe `src`. With `appendPath`
+ * the escaped request path is appended to the target's path without a double slash; with
+ * `keepQuery` the request query is added with `?` or `&`, otherwise dropped. A fragment of the
+ * target stays last.
  */
 export function buildLocation(rule, url) {
     const hash = rule.target.indexOf('#');
@@ -283,3 +287,28 @@ export function buildLocation(rule, url) {
     }
     return hash < 0 ? location : `${location}#${rule.target.slice(hash + 1)}`;
 }
+/**
+ * The page of a `frame` rule: one full-window `<iframe>` showing `src`, titled with the target's
+ * hostname. Both are HTML-escaped.
+ */
+export function framePage(src, target) {
+    return [
+        '<!DOCTYPE html>',
+        '<html>',
+        '<head>',
+        '<meta charset="utf-8">',
+        '<meta name="viewport" content="width=device-width, initial-scale=1">',
+        `<title>${escapeHtml(new URL(target).hostname)}</title>`,
+        '<style>html,body{margin:0;height:100%;overflow:hidden}iframe{display:block;width:100%;height:100%;border:0}</style>',
+        '</head>',
+        '<body>',
+        `<iframe src="${escapeHtml(src)}" allowfullscreen></iframe>`,
+        '</body>',
+        '</html>',
+        '',
+    ].join('\n');
+}
+/** `&`, `<`, `>`, `"` and `'` as character references. */
+export function escapeHtml(value) {
+    return value.replace(/[&<>"']/g, (c) => `&#${c.charCodeAt(0)};`);
+}

package/dist/version.d.ts

@@ -1,5 +1,5 @@
 /** This package: names its log lines and the User-Agent of rules downloads and reports. */
 export declare const PACKAGE = "cloudflare-redirects";
 /** Equal to package.json's version (a test checks it). */
-export declare const VERSION = "0.2.0";
-export declare const USER_AGENT = "cloudflare-redirects/0.2.0";
+export declare const VERSION = "0.3.0";
+export declare const USER_AGENT = "cloudflare-redirects/0.3.0";

package/dist/version.js

@@ -1,5 +1,5 @@
 /** This package: names its log lines and the User-Agent of rules downloads and reports. */
 export const PACKAGE = 'cloudflare-redirects';
 /** Equal to package.json's version (a test checks it). */
-export const VERSION = '0.2.0';
+export const VERSION = '0.3.0';
 export const USER_AGENT = `${PACKAGE}/${VERSION}`;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@redirhub/cloudflare-redirects",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Bulk redirects and 404 monitoring for Cloudflare Workers from a remote rules file: thousands of 301/302 redirects, short links and website migrations in front of your origin, live within a minute without a redeploy. RedirHub Integration Mode.",
   "keywords": [
     "cloudflare",