webrecipe 0.1.1 → 0.1.2

package/README.md

@@ -18,9 +18,8 @@ search results. Runs locally. No hosted account, no API key.
 ```
 $ webrecipe fetch hn/list
 title	url
-ECB to assess feasibility of interlinking with Brazil instant payment system Pix	https://www.ecb.europa.eu/...
-Special agents' blood and urine test results stolen in FBI hack	https://www.bbc.co.uk/news/...
-Show HN: Xyp RSS – hold to play, swipe to skip, highlight a word for graph	https://xyp.app
+Special agents' blood and urine test results…	https://www.bbc.co.uk/news/...
+Show HN: Xyp RSS – hold to play, swipe to skip…	https://xyp.app
 ...
 ```
 
@@ -55,7 +54,7 @@ webrecipe fetch hn/list            # TSV on stdout
 webrecipe fetch hn/list --json     # one JSON object on stdout
 ```
 
-Run it again tomorrow and you get tomorrow's front page.
+Run it again tomorrow and you get tomorrow's latest submissions.
 
 ## Use with an agent
 
@@ -128,10 +127,20 @@ How to choose:
 - Scores do not tell you meaning. Above, the vote link scores as well as the
   story link. Read the sample column to tell them apart.
 
-**2. Save** your choice under a name, `site/intent`, where intent is `list`,
+**2. Save** your choice. Replace `NAME` with the output field name you want,
+such as `title` or `url`.
+
+```sh
+webrecipe save hn/list --url https://news.ycombinator.com/newest \
+  --items 'tr.athing.submission' \
+  --field 'title=span.titleline > a' 'url=span.titleline > a@href'
+```
+
+The name is `site/intent`. `site` is any name you choose, not necessarily the
+domain, so `hn`, `hn-jobs` and `my-shop` are all fine. `intent` is `list`,
 `search` or `detail`. Saving the same name again replaces it.
 
-**3. Fetch** by that name.
+**3. Fetch** by that name: `webrecipe fetch hn/list`.
 
 ### Pages with a parameter
 
@@ -197,6 +206,7 @@ A failure prints one object and exits 1:
 | --- | --- |
 | `NOT_TAUGHT` | nothing saved under that `site/intent` |
 | `INVALID_INPUT` | bad arguments, or an input the recipe does not take |
+| `ROBOTS_DISALLOWED` | robots.txt disallows the page or a redirect target; the disallowed URL was not requested |
 | `UNVERIFIED_RESULT` | zero rows, or a selected field missing from some rows |
 | `EXECUTION_FAILED` | network, browser or storage error |
 
@@ -226,8 +236,13 @@ Every `inspect`, `save`, `fetch` and `read` appends one line to a local log.
 timing and outcome, never page content, cookies or headers. Nothing is
 uploaded anywhere. `--no-log` skips logging for one command.
 
-Requests identify themselves with a `webrecipe/0.1` user agent, respect
-robots.txt, and wait between requests to the same host.
+HTTP requests identify themselves with a `webrecipe/0.1` user agent and wait
+between requests to the same host, honouring robots.txt `Crawl-delay`.
+`fetch` refuses a URL that robots.txt disallows before requesting it, and
+checks every redirect target the same way, over HTTP and in the browser.
+`read` refuses a disallowed URL before requesting it. `inspect` and `save` load the page in a
+browser without checking robots.txt. Check a site's terms before saving a
+recipe for it.
 
 ## Also: `read`
 
@@ -259,9 +274,11 @@ politeness waits and Node startup
 | Steam store page | 0.3 s | 2.9 s | 162 KB | 33.6 MB |
 
 **The first read is not free.** Inspect plus save took 5.4 s on Hacker News,
-52 s on Remote OK and 15 s on Steam, not counting the time to choose
-selectors. Counting that setup, repeat fetches overtook the browser at
-repetition 2, 18 and 6 respectively. If you will read a page once, use `read`.
+52 s on Remote OK and 15 s on Steam. Those figures exclude the time to choose
+selectors and any politeness wait; Hacker News added 27 s of waiting between
+inspect and save. Counting setup and waits in cumulative wall time, repeat fetches
+overtook the browser at repetition 2, 18 and 6 respectively. If you will read
+a page once, use `read`.
 
 Hacker News asks crawlers to wait 30 s between page loads. With that wait
 included, both approaches are dominated by it: a median 28.0 s per fetch for
@@ -275,10 +292,11 @@ hand. But of the 23 answers checked by hand, 12 had reached `verified` on the
 structural checks alone. That is where a wrong answer would hide, and one
 field there was ambiguous in exactly that way.
 
-**Sites drift.** One site that saved cleanly one day answered the next with a
-human-verification page ([notes](benchmark/results/drift/)). The recipe failed
-because its selectors matched nothing, which is the defence this tool relies
-on.
+**Sites drift.** A page that saved cleanly one day failed to save the next,
+because a required field was missing ([notes](benchmark/results/drift/)).
+Separate requests that day also got a human-verification page. The cause of
+the original failure was not established. The save failed loudly rather than
+storing a recipe with an empty field.
 
 ## Development
 

package/dist/src/browser/navigate.js

@@ -1,3 +1,4 @@
+import { UserError } from '../local.js';
 const NAVIGATION_TIMEOUT_MS = 30_000;
 const SELECTOR_TIMEOUT_MS = 10_000;
 /** After the items appear, give late XHR a moment to land before reading. */
@@ -12,11 +13,104 @@ export const SETTLE_MS = 800;
  * the selector, with a short settle for anything still in flight, and a
  * network-idle attempt only as a best effort that is allowed to fail.
  */
-export async function navigateAndSettle(page, url, itemSelector) {
-    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS });
-    await page
-        .waitForSelector(itemSelector, { timeout: SELECTOR_TIMEOUT_MS })
-        .catch(() => undefined);
-    await page.waitForLoadState('networkidle', { timeout: SETTLE_MS }).catch(() => undefined);
-    await page.waitForTimeout(SETTLE_MS);
+export async function navigateAndSettle(page, url, itemSelector, guard) {
+    if (guard)
+        await guard.goto(url);
+    else
+        await page.goto(url, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS });
+    // Waited for again while the page is still moving: the items worth reading
+    // are the ones on the page it ends on, not the one it left.
+    for (let moves = 0;; moves++) {
+        await page
+            .waitForSelector(itemSelector, { timeout: SELECTOR_TIMEOUT_MS })
+            .catch(() => undefined);
+        await page.waitForLoadState('networkidle', { timeout: SETTLE_MS }).catch(() => undefined);
+        await page.waitForTimeout(SETTLE_MS);
+        if (!guard || moves >= MAX_REDIRECTS || !(await guard.arrived()))
+            break;
+    }
+    guard?.check();
+}
+const MAX_REDIRECTS = 20;
+/**
+ * Checks every main-frame navigation of a page, for as long as the page lives.
+ *
+ * A route handler alone cannot check redirects: Chromium follows a redirect
+ * without routing the next hop, so the disallowed page would be requested
+ * before any handler saw it. Instead the handler fetches each main-frame
+ * document without following redirects, stops the navigation at a redirect,
+ * and starts a fresh one to the target, which passes through the check again.
+ *
+ * This happens for the page's whole life, not only inside `goto`: a page can
+ * move itself after loading, and a redirect then has to be followed, or
+ * refused, rather than dropped with the old page left to be read.
+ */
+export async function guardPage(page, allowed) {
+    let refused = null;
+    let failure = null;
+    let hops = 0;
+    const following = new Set();
+    const refusal = () => new UserError('ROBOTS_DISALLOWED', `robots.txt disallows ${refused}`);
+    const check = () => {
+        if (refused !== null)
+            throw refusal();
+        if (failure !== null)
+            throw failure;
+    };
+    const follow = (target) => {
+        if (++hops > MAX_REDIRECTS) {
+            failure ??= new Error('too many redirects');
+            return;
+        }
+        const moving = page.goto(target, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS })
+            .then(() => undefined, () => undefined)
+            .finally(() => { following.delete(moving); });
+        following.add(moving);
+    };
+    await page.route('**/*', async (route) => {
+        try {
+            const request = route.request();
+            if (!request.isNavigationRequest() || request.frame() !== page.mainFrame())
+                return await route.continue();
+            if (!(await allowed(request.url()))) {
+                refused ??= request.url();
+                return await route.abort('blockedbyclient');
+            }
+            const response = await route.fetch({ maxRedirects: 0 });
+            const location = response.headers()['location'];
+            if (response.status() >= 300 && response.status() < 400 && location) {
+                await route.abort('aborted');
+                follow(new URL(location, request.url()).toString());
+                return;
+            }
+            await route.fulfill({ response });
+        }
+        catch {
+            // The page closed under a pending request; there is no one left to answer.
+        }
+    });
+    let seen = 0;
+    const arrived = async () => {
+        while (following.size > 0)
+            await Promise.all(following);
+        check();
+        const moved = hops !== seen;
+        seen = hops;
+        return moved;
+    };
+    return {
+        async goto(url) {
+            // A navigation stopped at a redirect rejects; what happens next is `follow`'s, and `arrived` waits for it.
+            try {
+                await page.goto(url, { waitUntil: 'domcontentloaded', timeout: NAVIGATION_TIMEOUT_MS });
+            }
+            catch (error) {
+                if (refused === null && hops === 0)
+                    throw error;
+            }
+            await arrived();
+        },
+        arrived,
+        check,
+    };
 }

package/dist/src/executor/index.js

@@ -1,5 +1,6 @@
 import { measureResult } from '../measurement.js';
 import { validate } from '../validator/index.js';
+import { isRobotsRefusal } from '../local.js';
 import { addMeta } from '../types.js';
 /** How long a site stays off the recipe path after it refused one. */
 export const BLOCK_COOLDOWN_MS = 10 * 60_000;
@@ -45,6 +46,8 @@ export class Executor {
                 return { ...result, recipeUsed: false, fellBack: true, reasons };
             }
             catch (err) {
+                if (isRobotsRefusal(err))
+                    throw err;
                 lastError = err;
                 reasons.push(err instanceof Error ? err.message : String(err));
             }
@@ -92,6 +95,9 @@ export class Executor {
                 }
             }
             catch (err) {
+                // Not a reason to try the browser: it would request the page robots.txt excluded.
+                if (isRobotsRefusal(err))
+                    throw err;
                 reasons.push(err instanceof Error ? err.message : String(err));
             }
         }

package/dist/src/executor/strategies/browser.js

@@ -1,6 +1,6 @@
 import { measureResult } from '../../measurement.js';
 import { openSession } from '../../browser/session.js';
-import { navigateAndSettle } from '../../browser/navigate.js';
+import { navigateAndSettle, guardPage } from '../../browser/navigate.js';
 import { planFields } from '../extract.js';
 import { countTokens } from '../tokens.js';
 import { emptyMeta } from '../../types.js';
@@ -9,13 +9,15 @@ export class BrowserStrategy {
     sites;
     plans;
     countAgentTokens;
+    guard;
     name = 'browser';
     constructor(sites, plans = BROWSER_PLANS, 
     /** The token count costs a page read of its own; a timing benchmark can decline to pay it. */
-    countAgentTokens = true) {
+    countAgentTokens = true, guard) {
         this.sites = sites;
         this.plans = plans;
         this.countAgentTokens = countAgentTokens;
+        this.guard = guard;
     }
     async execute(recipe, task) {
         return measureResult(this.name, () => this.executeAttempt(recipe, task));
@@ -29,7 +31,8 @@ export class BrowserStrategy {
         const session = await openSession();
         meta.browserLaunches = 1;
         try {
-            await navigateAndSettle(session.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector);
+            const guard = this.guard ? await guardPage(session.page, this.guard) : undefined;
+            await navigateAndSettle(session.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector, guard);
             // The field specs are interpreted once, in node, so that the browser and
             // cheerio cannot drift apart on what a spec means.
             const items = (await session.page.$$eval(plan.itemSelector, (elements, plans) => elements.map((el) => Object.fromEntries(plans.map((f) => {
@@ -51,6 +54,8 @@ export class BrowserStrategy {
             // graded run into a thrown one.
             const snapshot = this.countAgentTokens ? await session.page.locator('body').ariaSnapshot().catch(() => '') : '';
             meta.llmTokens = countTokens(snapshot);
+            // After reading, so a page that moved somewhere disallowed while it was read is not answered from.
+            guard?.check();
             return { items, meta };
         }
         finally {

package/dist/src/executor/strategies/warm-browser.js

@@ -1,6 +1,6 @@
 import { measureResult } from '../../measurement.js';
 import { BrowserPool } from '../../browser/pool.js';
-import { navigateAndSettle } from '../../browser/navigate.js';
+import { navigateAndSettle, guardPage } from '../../browser/navigate.js';
 import { planFields } from '../extract.js';
 import { countTokens } from '../tokens.js';
 import { BROWSER_PLANS } from './browser.js';
@@ -9,11 +9,13 @@ import { emptyMeta } from '../../types.js';
 export class WarmBrowserStrategy {
     sites;
     plans;
+    guard;
     name = 'warm-browser';
     pool = new BrowserPool();
-    constructor(sites, plans = BROWSER_PLANS) {
+    constructor(sites, plans = BROWSER_PLANS, guard) {
         this.sites = sites;
         this.plans = plans;
+        this.guard = guard;
     }
     async execute(recipe, task) {
         return measureResult(this.name, () => this.executeAttempt(recipe, task));
@@ -26,7 +28,8 @@ export class WarmBrowserStrategy {
         const started = performance.now();
         const warm = await this.pool.acquire();
         try {
-            await navigateAndSettle(warm.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector);
+            const guard = this.guard ? await guardPage(warm.page, this.guard) : undefined;
+            await navigateAndSettle(warm.page, plan.url(this.sites.origin(task.site), task), plan.itemSelector, guard);
             const items = (await warm.page.$$eval(plan.itemSelector, (elements, plans) => elements.map((el) => Object.fromEntries(plans.map((f) => {
                 switch (f.mode) {
                     case 'own-text': return [f.name, el.textContent?.trim() ?? null];
@@ -45,6 +48,8 @@ export class WarmBrowserStrategy {
             // After the latency line, and allowed to fail, for the reasons in browser.ts.
             const snapshot = await warm.page.locator('body').ariaSnapshot().catch(() => '');
             meta.llmTokens = countTokens(snapshot);
+            // After reading, so a page that moved somewhere disallowed while it was read is not answered from.
+            guard?.check();
             return { items, meta };
         }
         finally {

package/dist/src/healing/index.js

@@ -70,7 +70,7 @@ export class SelfHealer {
         if (!plan)
             return { healed: false, recipe: null, changes: [], refused: null, meta: null };
         const started = performance.now();
-        const trace = await record(plan, event.task, this.opts.sites);
+        const trace = await record(plan, event.task, this.opts.sites, this.opts.guard);
         try {
             const meta = costOf(trace, Math.round(performance.now() - started));
             // Every compiler that ran and refused, not only the last. The html one

package/dist/src/local.js

@@ -109,6 +109,14 @@ export class UserError extends Error {
         this.code = code;
     }
 }
+/** A robots.txt refusal ends the task: no fallback, no retry on another path, no relearn. */
+export function isRobotsRefusal(error) {
+    for (let e = error; e instanceof Error; e = e.cause) {
+        if (e instanceof UserError && e.code === 'ROBOTS_DISALLOWED')
+            return true;
+    }
+    return false;
+}
 export function siteName(value) {
     if (!/^[a-zA-Z0-9][a-zA-Z0-9._-]*$/.test(value))
         throw new UserError('INVALID_INPUT', 'site must be a name such as news.ycombinator.com, not a URL or path');

package/dist/src/net/politeness.js

@@ -1,5 +1,6 @@
 import { costSink } from '../measurement.js';
 import { parseRobots, isPathAllowed } from './robots.js';
+import { UserError } from '../local.js';
 export const DEFAULT_USER_AGENT = 'webrecipe/0.1 (+https://github.com/Pillsoon/webrecipe)';
 const LOOPBACK = /^(localhost|127\.0\.0\.1|\[::1\])(:\d+)?$/;
 export class PolitenessLayer {
@@ -10,6 +11,7 @@ export class PolitenessLayer {
     now;
     sleep;
     fetchImpl;
+    enforceRobots;
     hosts = new Map();
     constructor(opts = {}) {
         this.userAgent = opts.userAgent ?? DEFAULT_USER_AGENT;
@@ -19,6 +21,7 @@ export class PolitenessLayer {
         this.now = opts.now ?? (() => Date.now());
         this.sleep = opts.sleep ?? ((ms) => new Promise((r) => setTimeout(r, ms)));
         this.fetchImpl = opts.fetchImpl ?? fetch;
+        this.enforceRobots = opts.enforceRobots ?? false;
     }
     state(host) {
         let s = this.hosts.get(host);
@@ -49,14 +52,20 @@ export class PolitenessLayer {
         await this.robotsFor(u);
         return this.state(u.host).intervalMs;
     }
+    async refuseDisallowed(url) {
+        if (!(await this.isAllowed(url)))
+            throw new UserError('ROBOTS_DISALLOWED', `robots.txt disallows ${url}`);
+    }
     /** Bypasses the rate limiter; used only to fetch robots.txt itself. */
-    async raw(url, opts = {}) {
+    async raw(url, opts = {}, check) {
         const charge = costSink();
         let current = url;
         let method = opts.method ?? 'GET';
         let requestBody = opts.body;
         const headers = new Headers({ 'user-agent': this.userAgent, ...(opts.headers ?? {}) });
         for (let redirects = 0;; redirects++) {
+            // Before every hop, not only the first: an allowed URL may redirect to a disallowed one.
+            await check?.(current);
             charge({ networkRequests: 1 });
             const res = await this.fetchImpl(current, {
                 method, headers: Object.fromEntries(headers.entries()), body: requestBody, redirect: 'manual', signal: AbortSignal.timeout(this.timeoutMs),
@@ -124,7 +133,7 @@ export class PolitenessLayer {
                 }
             }
             s.lastRequestAt = this.now();
-            const res = await this.raw(u.toString(), opts);
+            const res = await this.raw(u.toString(), opts, this.enforceRobots ? (url) => this.refuseDisallowed(url) : undefined);
             if (res.status !== 429 || attempt >= this.maxRetries)
                 return { ...res, waitedMs };
             const retryAfter = Number(res.headers['retry-after']);

package/dist/src/recorder/index.js

@@ -1,6 +1,6 @@
 import { openSession } from '../browser/session.js';
 import { BodyStore } from './body.js';
-import { navigateAndSettle } from '../browser/navigate.js';
+import { navigateAndSettle, guardPage } from '../browser/navigate.js';
 /** A DOM mutation this soon after a response is treated as caused by it. */
 const DOM_SETTLE_MS = 800;
 /**
@@ -84,7 +84,7 @@ export function attributeMutations(requests, mutations) {
  * A trace owns the files its large bodies were spilled into, so a caller that
  * keeps the trace past the call must dispose of it.
  */
-export async function record(plan, task, sites) {
+export async function record(plan, task, sites, allowed) {
     const origin = sites.origin(task.site);
     const session = await openSession();
     const actions = [];
@@ -124,7 +124,8 @@ export async function record(plan, task, sites) {
         });
         const target = plan.url(origin, task);
         actions.push({ index: 0, type: 'navigate', value: target, at: Date.now() });
-        await navigateAndSettle(session.page, target, plan.itemSelector);
+        const guard = allowed ? await guardPage(session.page, allowed) : undefined;
+        await navigateAndSettle(session.page, target, plan.itemSelector, guard);
         const mutations = (await session.page.evaluate('window.__fwaMutations || []'));
         const completions = (await session.page.evaluate('window.__fwaCompletions || []'));
         // Align first: attribution compares these timestamps against each other.
@@ -132,6 +133,7 @@ export async function record(plan, task, sites) {
         attributeMutations(pending, mutations);
         const finalHtml = await session.page.content();
         await Promise.all(bodyReads);
+        guard?.check();
         return {
             site: task.site,
             intent: task.intent,

package/dist/src/tasks.js

@@ -3,6 +3,8 @@ import { buildEngine } from './wiring.js';
 import { loadLearnedPlans, mergePlans } from './authoring/plans.js';
 import { PLANS } from '../benchmark/plans.js';
 import { WILD_ORIGINS } from './sites.js';
+import { buildUrl } from './executor/strategies/http-json.js';
+import { measureResult } from './measurement.js';
 /** The hand-written plans, with anything saved layered over them. */
 export async function allPlans(planDir) {
     const learned = await loadLearnedPlans(planDir);
@@ -21,11 +23,26 @@ export async function fetchTask(site, intent, input, opts = {}) {
     const plan = plans[site]?.[intent];
     if (!plan)
         throw new UserError('NOT_TAUGHT', `No recipe saved as ${site}/${intent}. Run inspect on the page, then save.`);
+    const origin = origins[site] ?? WILD_ORIGINS[site];
+    const task = { id: opts.runId ?? 'fetch', site, intent, input };
     // Reject missing placeholders before starting a browser or making a request.
-    plan.url(origins[site] ?? WILD_ORIGINS[site], { id: 'check', site, intent, input });
-    const engine = buildEngine({ recipeDir: paths.recipes, plans, origins, heal: opts.heal });
+    const pageUrl = plan.url(origin, task);
+    const engine = buildEngine({ recipeDir: paths.recipes, plans, origins, heal: opts.heal, enforceRobots: true });
     try {
-        const outcome = await engine.executor.run({ id: opts.runId ?? 'fetch', site, intent, input });
+        // One measurement around both, so the robots.txt request is counted in meta
+        // whether the fetch goes ahead or is refused.
+        const outcome = await measureResult('browser', async () => {
+            // Decided here, before the executor, because the executor treats an HTTP
+            // failure as a reason to try the browser: a refusal raised any lower would
+            // be retried on the very page robots.txt excluded, and then relearned.
+            const recipe = await engine.registry.load(site, intent);
+            const urls = [pageUrl, ...(recipe && recipe.strategy.type !== 'browser' ? [buildUrl(origin, recipe, task)] : [])];
+            for (const url of urls) {
+                if (!(await engine.net.isAllowed(url)))
+                    throw new UserError('ROBOTS_DISALLOWED', `robots.txt disallows ${url}`);
+            }
+            return engine.executor.run(task);
+        });
         const verification = verifyReadable(outcome.items, Object.keys(plan.fields), verifications[site]?.[intent]);
         // Derived from the checks, so a run that proves more says less.
         const warnings = [...outcome.reasons, ...verificationWarnings(verification)];

package/dist/src/wiring.js

@@ -11,12 +11,14 @@ export function buildEngine(opts) {
     const net = new PolitenessLayer({
         userAgent: DEFAULT_USER_AGENT,
         minIntervalMs: opts.minIntervalMs,
+        enforceRobots: opts.enforceRobots,
     });
+    const guard = opts.enforceRobots ? (url) => net.isAllowed(url) : undefined;
     const sites = new StaticSiteResolver({ ...WILD_ORIGINS, ...(opts.origins ?? {}) });
     const registry = new RecipeRegistry(opts.recipeDir);
-    const browser = new BrowserStrategy(sites, opts.plans);
-    const warm = new WarmBrowserStrategy(sites, opts.plans);
-    const healer = new SelfHealer({ registry, sites, plans: opts.plans });
+    const browser = new BrowserStrategy(sites, opts.plans, true, guard);
+    const warm = new WarmBrowserStrategy(sites, opts.plans, guard);
+    const healer = new SelfHealer({ registry, sites, plans: opts.plans, guard });
     const executor = new Executor({
         registry,
         strategies: [new HttpJsonStrategy(net, sites), new HttpHtmlStrategy(net, sites), warm, browser],

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webrecipe",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "type": "module",
   "engines": {
     "node": ">=22"