webrecipe 0.1.1 → 0.1.3

package/README.md

@@ -18,9 +18,8 @@ search results. Runs locally. No hosted account, no API key.
 ```
 $ webrecipe fetch hn/list
 title	url
-ECB to assess feasibility of interlinking with Brazil instant payment system Pix	https://www.ecb.europa.eu/...
-Special agents' blood and urine test results stolen in FBI hack	https://www.bbc.co.uk/news/...
-Show HN: Xyp RSS – hold to play, swipe to skip, highlight a word for graph	https://xyp.app
+Special agents' blood and urine test results…	https://www.bbc.co.uk/news/...
+Show HN: Xyp RSS – hold to play, swipe to skip…	https://xyp.app
 ...
 ```
 
@@ -55,7 +54,7 @@ webrecipe fetch hn/list            # TSV on stdout
 webrecipe fetch hn/list --json     # one JSON object on stdout
 ```
 
-Run it again tomorrow and you get tomorrow's front page.
+Run it again tomorrow and you get tomorrow's latest submissions.
 
 ## Use with an agent
 
@@ -128,10 +127,20 @@ How to choose:
 - Scores do not tell you meaning. Above, the vote link scores as well as the
   story link. Read the sample column to tell them apart.
 
-**2. Save** your choice under a name, `site/intent`, where intent is `list`,
+**2. Save** your choice. Replace `NAME` with the output field name you want,
+such as `title` or `url`.
+
+```sh
+webrecipe save hn/list --url https://news.ycombinator.com/newest \
+  --items 'tr.athing.submission' \
+  --field 'title=span.titleline > a' 'url=span.titleline > a@href'
+```
+
+The name is `site/intent`. `site` is any name you choose, not necessarily the
+domain, so `hn`, `hn-jobs` and `my-shop` are all fine. `intent` is `list`,
 `search` or `detail`. Saving the same name again replaces it.
 
-**3. Fetch** by that name.
+**3. Fetch** by that name: `webrecipe fetch hn/list`.
 
 ### Pages with a parameter
 
@@ -197,6 +206,7 @@ A failure prints one object and exits 1:
 | --- | --- |
 | `NOT_TAUGHT` | nothing saved under that `site/intent` |
 | `INVALID_INPUT` | bad arguments, or an input the recipe does not take |
+| `ROBOTS_DISALLOWED` | robots.txt disallows the page or a redirect target; the disallowed URL was not requested |
 | `UNVERIFIED_RESULT` | zero rows, or a selected field missing from some rows |
 | `EXECUTION_FAILED` | network, browser or storage error |
 
@@ -226,8 +236,13 @@ Every `inspect`, `save`, `fetch` and `read` appends one line to a local log.
 timing and outcome, never page content, cookies or headers. Nothing is
 uploaded anywhere. `--no-log` skips logging for one command.
 
-Requests identify themselves with a `webrecipe/0.1` user agent, respect
-robots.txt, and wait between requests to the same host.
+HTTP requests identify themselves with a `webrecipe/0.1` user agent and wait
+between requests to the same host, honouring robots.txt `Crawl-delay`.
+`fetch` refuses a URL that robots.txt disallows before requesting it, and
+checks every redirect target the same way, over HTTP and in the browser.
+`read` refuses a disallowed URL before requesting it. `inspect` and `save` load the page in a
+browser without checking robots.txt. Check a site's terms before saving a
+recipe for it.
 
 ## Also: `read`
 
@@ -259,9 +274,11 @@ politeness waits and Node startup
 | Steam store page | 0.3 s | 2.9 s | 162 KB | 33.6 MB |
 
 **The first read is not free.** Inspect plus save took 5.4 s on Hacker News,
-52 s on Remote OK and 15 s on Steam, not counting the time to choose
-selectors. Counting that setup, repeat fetches overtook the browser at
-repetition 2, 18 and 6 respectively. If you will read a page once, use `read`.
+52 s on Remote OK and 15 s on Steam. Those figures exclude the time to choose
+selectors and any politeness wait; Hacker News added 27 s of waiting between
+inspect and save. Counting setup and waits in cumulative wall time, repeat fetches
+overtook the browser at repetition 2, 18 and 6 respectively. If you will read
+a page once, use `read`.
 
 Hacker News asks crawlers to wait 30 s between page loads. With that wait
 included, both approaches are dominated by it: a median 28.0 s per fetch for
@@ -275,10 +292,11 @@ hand. But of the 23 answers checked by hand, 12 had reached `verified` on the
 structural checks alone. That is where a wrong answer would hide, and one
 field there was ambiguous in exactly that way.
 
-**Sites drift.** One site that saved cleanly one day answered the next with a
-human-verification page ([notes](benchmark/results/drift/)). The recipe failed
-because its selectors matched nothing, which is the defence this tool relies
-on.
+**Sites drift.** A page that saved cleanly one day failed to save the next,
+because a required field was missing ([notes](benchmark/results/drift/)).
+Separate requests that day also got a human-verification page. The cause of
+the original failure was not established. The save failed loudly rather than
+storing a recipe with an empty field.
 
 ## Development
 

package/dist/src/benchmark/ground-truth.js

@@ -1,5 +1,13 @@
 import { DATASET, PAGE_SIZE, search } from '../../fixtures/data.js';
-const identity = (item) => String(item.url ?? '');
+const identity = (item) => {
+    const url = String(item.url ?? '');
+    try {
+        return new URL(url).pathname;
+    }
+    catch {
+        return url;
+    }
+};
 const KNOWN = new Set(DATASET.map((r) => `/item/${r.id}`));
 const PAGES = Math.ceil(DATASET.length / PAGE_SIZE);
 /** Every match, not just page one: a different ranking is not a wrong answer. */

package/dist/src/browser/navigate.js

@@ -1,3 +1,4 @@
+import { UserError } from '../local.js';
 const NAVIGATION_TIMEOUT_MS = 30_000;
 const SELECTOR_TIMEOUT_MS = 10_000;
 /** After the items appear, give late XHR a moment to land before reading. */
@@ -12,11 +13,104 @@ export const SETTLE_MS = 800;
  * the selector, with a short settle for anything still in flight, and a
  * network-idle attempt only as a best effort that is allowed to fail.
  */
-export async function navigateAndSettle(page, url, itemSelector) {
-    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS });
-    await page
-        .waitForSelector(itemSelector, { timeout: SELECTOR_TIMEOUT_MS })
-        .catch(() => undefined);
-    await page.waitForLoadState('networkidle', { timeout: SETTLE_MS }).catch(() => undefined);
-    await page.waitForTimeout(SETTLE_MS);
+export async function navigateAndSettle(page, url, itemSelector, guard) {
+    if (guard)
+        await guard.goto(url);
+    else
+        await page.goto(url, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS });
+    // Waited for again while the page is still moving: the items worth reading
+    // are the ones on the page it ends on, not the one it left.
+    for (let moves = 0;; moves++) {
+        await page
+            .waitForSelector(itemSelector, { timeout: SELECTOR_TIMEOUT_MS })
+            .catch(() => undefined);
+        await page.waitForLoadState('networkidle', { timeout: SETTLE_MS }).catch(() => undefined);
+        await page.waitForTimeout(SETTLE_MS);
+        if (!guard || moves >= MAX_REDIRECTS || !(await guard.arrived()))
+            break;
+    }
+    guard?.check();
+}
+const MAX_REDIRECTS = 20;
+/**
+ * Checks every main-frame navigation of a page, for as long as the page lives.
+ *
+ * A route handler alone cannot check redirects: Chromium follows a redirect
+ * without routing the next hop, so the disallowed page would be requested
+ * before any handler saw it. Instead the handler fetches each main-frame
+ * document without following redirects, stops the navigation at a redirect,
+ * and starts a fresh one to the target, which passes through the check again.
+ *
+ * This happens for the page's whole life, not only inside `goto`: a page can
+ * move itself after loading, and a redirect then has to be followed, or
+ * refused, rather than dropped with the old page left to be read.
+ */
+export async function guardPage(page, allowed) {
+    let refused = null;
+    let failure = null;
+    let hops = 0;
+    const following = new Set();
+    const refusal = () => new UserError('ROBOTS_DISALLOWED', `robots.txt disallows ${refused}`);
+    const check = () => {
+        if (refused !== null)
+            throw refusal();
+        if (failure !== null)
+            throw failure;
+    };
+    const follow = (target) => {
+        if (++hops > MAX_REDIRECTS) {
+            failure ??= new Error('too many redirects');
+            return;
+        }
+        const moving = page.goto(target, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS })
+            .then(() => undefined, () => undefined)
+            .finally(() => { following.delete(moving); });
+        following.add(moving);
+    };
+    await page.route('**/*', async (route) => {
+        try {
+            const request = route.request();
+            if (!request.isNavigationRequest() || request.frame() !== page.mainFrame())
+                return await route.continue();
+            if (!(await allowed(request.url()))) {
+                refused ??= request.url();
+                return await route.abort('blockedbyclient');
+            }
+            const response = await route.fetch({ maxRedirects: 0 });
+            const location = response.headers()['location'];
+            if (response.status() >= 300 && response.status() < 400 && location) {
+                await route.abort('aborted');
+                follow(new URL(location, request.url()).toString());
+                return;
+            }
+            await route.fulfill({ response });
+        }
+        catch {
+            // The page closed under a pending request; there is no one left to answer.
+        }
+    });
+    let seen = 0;
+    const arrived = async () => {
+        while (following.size > 0)
+            await Promise.all(following);
+        check();
+        const moved = hops !== seen;
+        seen = hops;
+        return moved;
+    };
+    return {
+        async goto(url) {
+            // A navigation stopped at a redirect rejects; what happens next is `follow`'s, and `arrived` waits for it.
+            try {
+                await page.goto(url, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS });
+            }
+            catch (error) {
+                if (refused === null && hops === 0)
+                    throw error;
+            }
+            await arrived();
+        },
+        arrived,
+        check,
+    };
 }

package/dist/src/compiler/html.js

@@ -101,7 +101,7 @@ function compileFrom(trace, plan, source) {
     const untemplated = required.find((name) => !templated.has(name));
     if (untemplated !== undefined)
         return { refused: `required input "${untemplated}" not templated` };
-    const items = extractHtmlItems(draft, body);
+    const items = extractHtmlItems(draft, body, source.url);
     if (!verifyAgainstBrowser(trace, plan, items).equivalent) {
         return { refused: equivalenceRefusal(items.length, browserItemsOf(trace, plan).length) };
     }

package/dist/src/compiler/verify.js

@@ -10,7 +10,7 @@ import { diffGolden } from '../benchmark/golden.js';
  */
 /** What the browser extracted from this visit, using the plan's own selectors. */
 export function browserItemsOf(trace, plan) {
-    return extractBySelector(trace.finalHtml, plan.itemSelector, plan.fields);
+    return extractBySelector(trace.finalHtml, plan.itemSelector, plan.fields, trace.finalUrl);
 }
 export function verifyAgainstBrowser(trace, plan, recipeItems) {
     const browserItems = browserItemsOf(trace, plan);

package/dist/src/executor/extract.js

@@ -29,10 +29,10 @@ export function extractJsonItems(recipe, payload) {
             : renderRowTemplate(spec, row).trim(),
     ])));
 }
-export function extractHtmlItems(recipe, html) {
+export function extractHtmlItems(recipe, html, pageUrl) {
     if (recipe.output.type !== 'html')
         throw new Error('extractHtmlItems requires an html recipe');
-    return extractBySelector(html, recipe.output.items.selector, recipe.output.items.fields);
+    return extractBySelector(html, recipe.output.items.selector, recipe.output.items.fields, pageUrl);
 }
 /**
  * The structural signature of an HTML response: which of the recipe's declared
@@ -90,16 +90,20 @@ export function parseHtmlFragment(html) {
     const wrapper = FRAGMENT_WRAPPERS.find(([pattern]) => pattern.test(head));
     return cheerio.load(wrapper ? wrapper[1](head) : html);
 }
+/** Attributes holding a URL, reported absolute the way the browser's `a.href` reports them. */
+const URL_ATTRIBUTES = new Set(['href', 'src']);
 /** One interpretation of a field spec, shared by cheerio and the browser. */
 export function planFields(fields) {
     return Object.entries(fields).map(([name, spec]) => {
         if (spec === '')
             return { name, mode: 'own-text' };
-        if (spec.startsWith('@'))
-            return { name, mode: 'own-attr', attribute: spec.slice(1) };
+        if (spec.startsWith('@')) {
+            const attribute = spec.slice(1);
+            return { name, mode: 'own-attr', attribute, url: URL_ATTRIBUTES.has(attribute) };
+        }
         const suffix = attributeSuffix(spec);
         return suffix
-            ? { name, mode: 'find-attr', selector: suffix.selector, attribute: suffix.attribute }
+            ? { name, mode: 'find-attr', ...suffix, url: URL_ATTRIBUTES.has(suffix.attribute) }
             : { name, mode: 'find-text', selector: spec };
     });
 }
@@ -136,15 +140,39 @@ function attributeSuffix(spec) {
         return null;
     return { selector, attribute: match[1] };
 }
+/** Resolves a URL attribute as the browser does; an empty or unparsable value is left alone. */
+function absolute(value, base) {
+    if (value.trim() === '')
+        return value;
+    try {
+        return new URL(value, base).href;
+    }
+    catch {
+        return value;
+    }
+}
+/** Makes href and src fields absolute against the document's base URL, in place. */
+export function resolveUrlFields(items, plans, base) {
+    const urlFields = plans.filter((f) => (f.mode === 'own-attr' || f.mode === 'find-attr') && f.url).map((f) => f.name);
+    for (const item of items) {
+        for (const name of urlFields) {
+            const value = item[name];
+            if (typeof value === 'string')
+                item[name] = absolute(value, base);
+        }
+    }
+    return items;
+}
 /**
  * Extracts items from HTML with a bare selector and field map, independent of a
  * recipe. Used to reconstruct what the browser saw so a candidate recipe can be
- * checked against it.
+ * checked against it. Given the page's URL, href and src come back absolute;
+ * without it they come back as written.
  */
-export function extractBySelector(html, selector, fields) {
+export function extractBySelector(html, selector, fields, pageUrl) {
     const $ = parseHtmlFragment(html);
     const plans = planFields(fields);
-    return $(selector).toArray().map((element) => {
+    const items = $(selector).toArray().map((element) => {
         const item = $(element);
         const row = {};
         // Field candidates ask for `a`, `a@href` and `a@id` of the same item; one find serves all three.
@@ -176,4 +204,8 @@ export function extractBySelector(html, selector, fields) {
         }
         return row;
     });
+    if (pageUrl === undefined)
+        return items;
+    const baseHref = $('base[href]').first().attr('href');
+    return resolveUrlFields(items, plans, baseHref === undefined ? pageUrl : absolute(baseHref, pageUrl));
 }

package/dist/src/executor/index.js

@@ -1,5 +1,6 @@
 import { measureResult } from '../measurement.js';
 import { validate } from '../validator/index.js';
+import { isRobotsRefusal } from '../local.js';
 import { addMeta } from '../types.js';
 /** How long a site stays off the recipe path after it refused one. */
 export const BLOCK_COOLDOWN_MS = 10 * 60_000;
@@ -45,6 +46,8 @@ export class Executor {
                 return { ...result, recipeUsed: false, fellBack: true, reasons };
             }
             catch (err) {
+                if (isRobotsRefusal(err))
+                    throw err;
                 lastError = err;
                 reasons.push(err instanceof Error ? err.message : String(err));
             }
@@ -92,6 +95,9 @@ export class Executor {
                 }
             }
             catch (err) {
+                // Not a reason to try the browser: it would request the page robots.txt excluded.
+                if (isRobotsRefusal(err))
+                    throw err;
                 reasons.push(err instanceof Error ? err.message : String(err));
             }
         }

package/dist/src/executor/strategies/browser.js

@@ -1,7 +1,7 @@
 import { measureResult } from '../../measurement.js';
 import { openSession } from '../../browser/session.js';
-import { navigateAndSettle } from '../../browser/navigate.js';
-import { planFields } from '../extract.js';
+import { navigateAndSettle, guardPage } from '../../browser/navigate.js';
+import { planFields, resolveUrlFields } from '../extract.js';
 import { countTokens } from '../tokens.js';
 import { emptyMeta } from '../../types.js';
 export const BROWSER_PLANS = {};
@@ -9,13 +9,15 @@ export class BrowserStrategy {
     sites;
     plans;
     countAgentTokens;
+    guard;
     name = 'browser';
     constructor(sites, plans = BROWSER_PLANS, 
     /** The token count costs a page read of its own; a timing benchmark can decline to pay it. */
-    countAgentTokens = true) {
+    countAgentTokens = true, guard) {
         this.sites = sites;
         this.plans = plans;
         this.countAgentTokens = countAgentTokens;
+        this.guard = guard;
     }
     async execute(recipe, task) {
         return measureResult(this.name, () => this.executeAttempt(recipe, task));
@@ -29,7 +31,8 @@ export class BrowserStrategy {
         const session = await openSession();
         meta.browserLaunches = 1;
         try {
-            await navigateAndSettle(session.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector);
+            const guard = this.guard ? await guardPage(session.page, this.guard) : undefined;
+            await navigateAndSettle(session.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector, guard);
             // The field specs are interpreted once, in node, so that the browser and
             // cheerio cannot drift apart on what a spec means.
             const items = (await session.page.$$eval(plan.itemSelector, (elements, plans) => elements.map((el) => Object.fromEntries(plans.map((f) => {
@@ -40,6 +43,8 @@ export class BrowserStrategy {
                     default: return [f.name, el.querySelector(f.selector)?.textContent?.trim() ?? null];
                 }
             }))), planFields(plan.fields)));
+            // Resolved in node rather than in the page, by the same rule the http path uses.
+            resolveUrlFields(items, planFields(plan.fields), await session.page.evaluate(() => document.baseURI));
             meta.pageNavigations = session.cost.pageNavigations;
             meta.networkRequests = session.cost.networkRequests;
             meta.bytesDownloaded = session.cost.bytesDownloaded;
@@ -51,6 +56,8 @@ export class BrowserStrategy {
             // graded run into a thrown one.
             const snapshot = this.countAgentTokens ? await session.page.locator('body').ariaSnapshot().catch(() => '') : '';
             meta.llmTokens = countTokens(snapshot);
+            // After reading, so a page that moved somewhere disallowed while it was read is not answered from.
+            guard?.check();
             return { items, meta };
         }
         finally {

package/dist/src/executor/strategies/http-html.js

@@ -35,7 +35,7 @@ export class HttpHtmlStrategy {
         if (res.status !== recipe.validation.status) {
             return { items: [], meta, status: res.status, payload: undefined };
         }
-        const items = extractHtmlItems(recipe, res.body);
+        const items = extractHtmlItems(recipe, res.body, res.url);
         meta.llmTokens = countTokens(formatItems(items, 'tsv'));
         return { items, meta, payload: htmlSignature(recipe, items), status: res.status };
     }

package/dist/src/executor/strategies/warm-browser.js

@@ -1,7 +1,7 @@
 import { measureResult } from '../../measurement.js';
 import { BrowserPool } from '../../browser/pool.js';
-import { navigateAndSettle } from '../../browser/navigate.js';
-import { planFields } from '../extract.js';
+import { navigateAndSettle, guardPage } from '../../browser/navigate.js';
+import { planFields, resolveUrlFields } from '../extract.js';
 import { countTokens } from '../tokens.js';
 import { BROWSER_PLANS } from './browser.js';
 import { emptyMeta } from '../../types.js';
@@ -9,11 +9,13 @@ import { emptyMeta } from '../../types.js';
 export class WarmBrowserStrategy {
     sites;
     plans;
+    guard;
     name = 'warm-browser';
     pool = new BrowserPool();
-    constructor(sites, plans = BROWSER_PLANS) {
+    constructor(sites, plans = BROWSER_PLANS, guard) {
         this.sites = sites;
         this.plans = plans;
+        this.guard = guard;
     }
     async execute(recipe, task) {
         return measureResult(this.name, () => this.executeAttempt(recipe, task));
@@ -26,7 +28,8 @@ export class WarmBrowserStrategy {
         const started = performance.now();
         const warm = await this.pool.acquire();
         try {
-            await navigateAndSettle(warm.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector);
+            const guard = this.guard ? await guardPage(warm.page, this.guard) : undefined;
+            await navigateAndSettle(warm.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector, guard);
             const items = (await warm.page.$$eval(plan.itemSelector, (elements, plans) => elements.map((el) => Object.fromEntries(plans.map((f) => {
                 switch (f.mode) {
                     case 'own-text': return [f.name, el.textContent?.trim() ?? null];
@@ -35,6 +38,8 @@ export class WarmBrowserStrategy {
                     default: return [f.name, el.querySelector(f.selector)?.textContent?.trim() ?? null];
                 }
             }))), planFields(plan.fields)));
+            // Resolved in node rather than in the page, by the same rule the http path uses.
+            resolveUrlFields(items, planFields(plan.fields), await warm.page.evaluate(() => document.baseURI));
             // Zero only when the pool already had a browser. The first call starts
             // Chromium like any cold run does, and reporting none said otherwise.
             meta.browserLaunches = warm.launched ? 1 : 0;
@@ -45,6 +50,8 @@ export class WarmBrowserStrategy {
             // After the latency line, and allowed to fail, for the reasons in browser.ts.
             const snapshot = await warm.page.locator('body').ariaSnapshot().catch(() => '');
             meta.llmTokens = countTokens(snapshot);
+            // After reading, so a page that moved somewhere disallowed while it was read is not answered from.
+            guard?.check();
             return { items, meta };
         }
         finally {

package/dist/src/healing/index.js

@@ -70,7 +70,7 @@ export class SelfHealer {
         if (!plan)
             return { healed: false, recipe: null, changes: [], refused: null, meta: null };
         const started = performance.now();
-        const trace = await record(plan, event.task, this.opts.sites);
+        const trace = await record(plan, event.task, this.opts.sites, this.opts.guard);
         try {
             const meta = costOf(trace, Math.round(performance.now() - started));
             // Every compiler that ran and refused, not only the last. The html one

package/dist/src/local.js

@@ -109,6 +109,14 @@ export class UserError extends Error {
         this.code = code;
     }
 }
+/** A robots.txt refusal ends the task: no fallback, no retry on another path, no relearn. */
+export function isRobotsRefusal(error) {
+    for (let e = error; e instanceof Error; e = e.cause) {
+        if (e instanceof UserError && e.code === 'ROBOTS_DISALLOWED')
+            return true;
+    }
+    return false;
+}
 export function siteName(value) {
     if (!/^[a-zA-Z0-9][a-zA-Z0-9._-]*$/.test(value))
         throw new UserError('INVALID_INPUT', 'site must be a name such as news.ycombinator.com, not a URL or path');

package/dist/src/net/politeness.js

@@ -1,5 +1,6 @@
 import { costSink } from '../measurement.js';
 import { parseRobots, isPathAllowed } from './robots.js';
+import { UserError } from '../local.js';
 export const DEFAULT_USER_AGENT = 'webrecipe/0.1 (+https://github.com/Pillsoon/webrecipe)';
 const LOOPBACK = /^(localhost|127\.0\.0\.1|\[::1\])(:\d+)?$/;
 export class PolitenessLayer {
@@ -10,6 +11,7 @@ export class PolitenessLayer {
     now;
     sleep;
     fetchImpl;
+    enforceRobots;
     hosts = new Map();
     constructor(opts = {}) {
         this.userAgent = opts.userAgent ?? DEFAULT_USER_AGENT;
@@ -19,6 +21,7 @@ export class PolitenessLayer {
         this.now = opts.now ?? (() => Date.now());
         this.sleep = opts.sleep ?? ((ms) => new Promise((r) => setTimeout(r, ms)));
         this.fetchImpl = opts.fetchImpl ?? fetch;
+        this.enforceRobots = opts.enforceRobots ?? false;
     }
     state(host) {
         let s = this.hosts.get(host);
@@ -49,14 +52,20 @@ export class PolitenessLayer {
         await this.robotsFor(u);
         return this.state(u.host).intervalMs;
     }
+    async refuseDisallowed(url) {
+        if (!(await this.isAllowed(url)))
+            throw new UserError('ROBOTS_DISALLOWED', `robots.txt disallows ${url}`);
+    }
     /** Bypasses the rate limiter; used only to fetch robots.txt itself. */
-    async raw(url, opts = {}) {
+    async raw(url, opts = {}, check) {
         const charge = costSink();
         let current = url;
         let method = opts.method ?? 'GET';
         let requestBody = opts.body;
         const headers = new Headers({ 'user-agent': this.userAgent, ...(opts.headers ?? {}) });
         for (let redirects = 0;; redirects++) {
+            // Before every hop, not only the first: an allowed URL may redirect to a disallowed one.
+            await check?.(current);
             charge({ networkRequests: 1 });
             const res = await this.fetchImpl(current, {
                 method, headers: Object.fromEntries(headers.entries()), body: requestBody, redirect: 'manual', signal: AbortSignal.timeout(this.timeoutMs),
@@ -92,7 +101,7 @@ export class PolitenessLayer {
                 continue;
             }
             return {
-                status: res.status, headers: Object.fromEntries(res.headers.entries()),
+                status: res.status, headers: Object.fromEntries(res.headers.entries()), url: current,
                 body: buffer.toString('utf8'), bytesDownloaded: buffer.byteLength, waitedMs: 0,
             };
         }
@@ -124,7 +133,7 @@ export class PolitenessLayer {
                 }
             }
             s.lastRequestAt = this.now();
-            const res = await this.raw(u.toString(), opts);
+            const res = await this.raw(u.toString(), opts, this.enforceRobots ? (url) => this.refuseDisallowed(url) : undefined);
             if (res.status !== 429 || attempt >= this.maxRetries)
                 return { ...res, waitedMs };
             const retryAfter = Number(res.headers['retry-after']);

package/dist/src/recorder/index.js

@@ -1,6 +1,6 @@
 import { openSession } from '../browser/session.js';
 import { BodyStore } from './body.js';
-import { navigateAndSettle } from '../browser/navigate.js';
+import { navigateAndSettle, guardPage } from '../browser/navigate.js';
 /** A DOM mutation this soon after a response is treated as caused by it. */
 const DOM_SETTLE_MS = 800;
 /**
@@ -84,7 +84,7 @@ export function attributeMutations(requests, mutations) {
  * A trace owns the files its large bodies were spilled into, so a caller that
  * keeps the trace past the call must dispose of it.
  */
-export async function record(plan, task, sites) {
+export async function record(plan, task, sites, allowed) {
     const origin = sites.origin(task.site);
     const session = await openSession();
     const actions = [];
@@ -124,7 +124,8 @@ export async function record(plan, task, sites) {
         });
         const target = plan.url(origin, task);
         actions.push({ index: 0, type: 'navigate', value: target, at: Date.now() });
-        await navigateAndSettle(session.page, target, plan.itemSelector);
+        const guard = allowed ? await guardPage(session.page, allowed) : undefined;
+        await navigateAndSettle(session.page, target, plan.itemSelector, guard);
         const mutations = (await session.page.evaluate('window.__fwaMutations || []'));
         const completions = (await session.page.evaluate('window.__fwaCompletions || []'));
         // Align first: attribution compares these timestamps against each other.
@@ -132,6 +133,7 @@ export async function record(plan, task, sites) {
         attributeMutations(pending, mutations);
         const finalHtml = await session.page.content();
         await Promise.all(bodyReads);
+        guard?.check();
         return {
             site: task.site,
             intent: task.intent,
@@ -140,6 +142,7 @@ export async function record(plan, task, sites) {
             actions,
             requests: pending,
             finalHtml,
+            finalUrl: session.page.url(),
             ...(session.page.viewportSize() === null ? {} : { viewport: session.page.viewportSize() }),
             cost: session.cost,
             dispose: () => bodies.dispose(),

package/dist/src/tasks.js

@@ -3,6 +3,8 @@ import { buildEngine } from './wiring.js';
 import { loadLearnedPlans, mergePlans } from './authoring/plans.js';
 import { PLANS } from '../benchmark/plans.js';
 import { WILD_ORIGINS } from './sites.js';
+import { buildUrl } from './executor/strategies/http-json.js';
+import { measureResult } from './measurement.js';
 /** The hand-written plans, with anything saved layered over them. */
 export async function allPlans(planDir) {
     const learned = await loadLearnedPlans(planDir);
@@ -21,11 +23,26 @@ export async function fetchTask(site, intent, input, opts = {}) {
     const plan = plans[site]?.[intent];
     if (!plan)
         throw new UserError('NOT_TAUGHT', `No recipe saved as ${site}/${intent}. Run inspect on the page, then save.`);
+    const origin = origins[site] ?? WILD_ORIGINS[site];
+    const task = { id: opts.runId ?? 'fetch', site, intent, input };
     // Reject missing placeholders before starting a browser or making a request.
-    plan.url(origins[site] ?? WILD_ORIGINS[site], { id: 'check', site, intent, input });
-    const engine = buildEngine({ recipeDir: paths.recipes, plans, origins, heal: opts.heal });
+    const pageUrl = plan.url(origin, task);
+    const engine = buildEngine({ recipeDir: paths.recipes, plans, origins, heal: opts.heal, enforceRobots: true });
     try {
-        const outcome = await engine.executor.run({ id: opts.runId ?? 'fetch', site, intent, input });
+        // One measurement around both, so the robots.txt request is counted in meta
+        // whether the fetch goes ahead or is refused.
+        const outcome = await measureResult('browser', async () => {
+            // Decided here, before the executor, because the executor treats an HTTP
+            // failure as a reason to try the browser: a refusal raised any lower would
+            // be retried on the very page robots.txt excluded, and then relearned.
+            const recipe = await engine.registry.load(site, intent);
+            const urls = [pageUrl, ...(recipe && recipe.strategy.type !== 'browser' ? [buildUrl(origin, recipe, task)] : [])];
+            for (const url of urls) {
+                if (!(await engine.net.isAllowed(url)))
+                    throw new UserError('ROBOTS_DISALLOWED', `robots.txt disallows ${url}`);
+            }
+            return engine.executor.run(task);
+        });
         const verification = verifyReadable(outcome.items, Object.keys(plan.fields), verifications[site]?.[intent]);
         // Derived from the checks, so a run that proves more says less.
         const warnings = [...outcome.reasons, ...verificationWarnings(verification)];

package/dist/src/wiring.js

@@ -11,12 +11,14 @@ export function buildEngine(opts) {
     const net = new PolitenessLayer({
         userAgent: DEFAULT_USER_AGENT,
         minIntervalMs: opts.minIntervalMs,
+        enforceRobots: opts.enforceRobots,
     });
+    const guard = opts.enforceRobots ? (url) => net.isAllowed(url) : undefined;
     const sites = new StaticSiteResolver({ ...WILD_ORIGINS, ...(opts.origins ?? {}) });
     const registry = new RecipeRegistry(opts.recipeDir);
-    const browser = new BrowserStrategy(sites, opts.plans);
-    const warm = new WarmBrowserStrategy(sites, opts.plans);
-    const healer = new SelfHealer({ registry, sites, plans: opts.plans });
+    const browser = new BrowserStrategy(sites, opts.plans, true, guard);
+    const warm = new WarmBrowserStrategy(sites, opts.plans, guard);
+    const healer = new SelfHealer({ registry, sites, plans: opts.plans, guard });
     const executor = new Executor({
         registry,
         strategies: [new HttpJsonStrategy(net, sites), new HttpHtmlStrategy(net, sites), warm, browser],

package/package.json

@@ -1,12 +1,12 @@
 {
   "name": "webrecipe",
-  "version": "0.1.1",
+  "version": "0.1.3",
   "type": "module",
   "engines": {
     "node": ">=22"
   },
   "bin": {
-    "webrecipe": "./dist/src/cli.js"
+    "webrecipe": "dist/src/cli.js"
   },
   "scripts": {
     "build": "tsc -p tsconfig.build.json",
@@ -41,7 +41,7 @@
   "license": "MIT",
   "repository": {
     "type": "git",
-    "url": "https://github.com/Pillsoon/webrecipe.git"
+    "url": "git+https://github.com/Pillsoon/webrecipe.git"
   },
   "keywords": [
     "web",