@wrenautomation/weaver 0.0.0-stage → 0.1.0

package/LICENSE.md

@@ -0,0 +1,91 @@
+# PolyForm Shield License 1.0.0
+
+<https://polyformproject.org/licenses/shield/1.0.0>
+
+## Acceptance
+
+In order to get any license under these terms, you must agree to them as both strict obligations and conditions to all your licenses.
+
+## Copyright License
+
+The licensor grants you a copyright license for the software to do everything you might do with the software that would otherwise infringe the licensor's copyright in it for any permitted purpose.  However, you may only distribute the software according to [Distribution License](#distribution-license) and make changes or new works based on the software according to [Changes and New Works License](#changes-and-new-works-license).
+
+## Distribution License
+
+The licensor grants you an additional copyright license to distribute copies of the software.  Your license to distribute covers distributing the software with changes and new works permitted by [Changes and New Works License](#changes-and-new-works-license).
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms or the URL for them above, as well as copies of any plain-text lines beginning with `Required Notice:` that the licensor provided with the software.  For example:
+
+> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
+
+## Changes and New Works License
+
+The licensor grants you an additional copyright license to make changes and new works based on the software for any permitted purpose.
+
+## Patent License
+
+The licensor grants you a patent license for the software that covers patent claims the licensor can license, or becomes able to license, that you would infringe by using the software.
+
+## Noncompete
+
+Any purpose is a permitted purpose, except for providing any product that competes with the software or any product the licensor or any of its affiliates provides using the software.
+
+## Competition
+
+Goods and services compete even when they provide functionality through different kinds of interfaces or for different technical platforms.  Applications can compete with services, libraries with plugins, frameworks with development tools, and so on, even if they're written in different programming languages or for different computer architectures.  Goods and services compete even when provided free of charge.  If you market a product as a practical substitute for the software or another product, it definitely competes.
+
+## New Products
+
+If you are using the software to provide a product that does not compete, but the licensor or any of its affiliates brings your product into competition by providing a new version of the software or another product using the software, you may continue using versions of the software available under these terms beforehand to provide your competing product, but not any later versions.
+
+## Discontinued Products
+
+You may begin using the software to compete with a product or service that the licensor or any of its affiliates has stopped providing, unless the licensor includes a plain-text line beginning with `Licensor Line of Business:` with the software that mentions that line of business.  For example:
+
+> Licensor Line of Business: YoyodyneCMS Content Management System (http://example.com/cms)
+
+## Sales of Business
+
+If the licensor or any of its affiliates sells a line of business developing the software or using the software to provide a product, the buyer can also enforce [Noncompete](#noncompete) for that product.
+
+## Fair Use
+
+You may have "fair use" rights for the software under the law. These terms do not limit them.
+
+## No Other Rights
+
+These terms do not allow you to sublicense or transfer any of your licenses to anyone else, or prevent the licensor from granting licenses to anyone else.  These terms do not imply any other licenses.
+
+## Patent Defense
+
+If you make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.
+
+## Violations
+
+The first time you are notified in writing that you have violated any of these terms, or done anything with the software not covered by your licenses, your licenses can nonetheless continue if you come into full compliance with these terms, and take practical steps to correct past violations, within 32 days of receiving notice.  Otherwise, all your licenses end immediately.
+
+## No Liability
+
+***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.***
+
+## Definitions
+
+The **licensor** is the individual or entity offering these terms, and the **software** is the software the licensor makes available under these terms.
+
+A **product** can be a good or service, or a combination of them.
+
+**You** refers to the individual or entity agreeing to these terms.
+
+**Your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all its affiliates.
+
+**Affiliates** means the other organizations than an organization has control over, is under the control of, or is under common control with.
+
+**Control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise.  Control can be direct or indirect.
+
+**Your licenses** are all the licenses granted to you for the software under these terms.
+
+**Use** means anything you do with the software requiring one of your licenses.
+
+Required Notice: Copyright 2026 William Jin (https://wrenautomation.com)

package/README.md

@@ -1,3 +1,52 @@
-# Temporary Holding Version
+# weaver
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+The web in. Documents out. No database, no accounts, no business rules: a
+library that fetches, and a small CLI over it. Carved out of
+[falcon](https://github.com/wrenautomation/falcon) (then `@wrenautomation/scribe`)
+in 2026-10: everything that takes a URL or spawns yt-dlp lives here, everything
+that reads a file, bytes or text stays there.
+
+Two doors:
+
+| Import | Does |
+|---|---|
+| `@wrenautomation/weaver/fetch` | `PoliteFetcher` (one request at a time per host, a contact in the User-Agent, retries on 429 and 5xx, a byte cap), `robots` in warn mode, `fetchLadder` (plain GET up to a browser through a `LadderClient`) |
+| `@wrenautomation/weaver/video` | `readYoutube` (captions and chapters by yt-dlp, plus Gemini's read of the screen), `readYoutubeByUrl` (Gemini alone, no download), `readVideo` (any other video: yt-dlp, then falcon's `watch`), `readAudio` (a podcast by its audio URL, then falcon's `hearAudio`), `watch` over a URL or a file, `flatPlaylist` (what a profile, channel or search holds), `download` |
+
+Every reader returns falcon's one `VideoRead` shape. What turns a page or a file into text is
+falcon's (`@wrenautomation/falcon`). What searches by a source interface, runs a plan and keeps
+the ledger comes next (phase 3.5c of wren's research engine design).
+
+## CLI
+
+Needs Node 24+. `read` needs yt-dlp and ffmpeg.
+
+```sh
+npm i @wrenautomation/weaver
+npx weaver fetch https://example.com/report --contact you@example.com   # a page as text
+GEMINI_API_KEY=... npx weaver read https://www.youtube.com/watch?v=...    # captions + screen
+COHERE_API_KEY=... npx weaver read https://www.instagram.com/reel/...     # watched: speech, screen, cuts
+npx weaver list 'ytsearch10:cold email' --to 10                           # hits as JSON, no downloads
+```
+
+## Library
+
+```ts
+import { PoliteFetcher, userAgent } from "@wrenautomation/weaver/fetch";
+import { readPage } from "@wrenautomation/falcon/text";
+
+const fetcher = new PoliteFetcher(userAgent("you@example.com"));
+const res = await fetcher.get(url);
+const page = readPage(res.text, res.url);
+```
+
+## Develop
+
+```sh
+pnpm install
+pnpm lint && pnpm typecheck && pnpm test
+```
+
+A `v*` tag publishes to npm (`.github/workflows/release.yml`, trusted publishing: no stored token).
+
+License: PolyForm Shield 1.0.0 (`LICENSE.md`).

package/dist/app/cli.d.ts

@@ -0,0 +1,4 @@
+#!/usr/bin/env node
+import type { VideoRead } from "@wrenautomation/falcon/video";
+/** The read as one markdown document: its facts as front matter, its title, its sections. */
+export declare function renderRead(r: VideoRead): string;

package/dist/app/cli.js

@@ -0,0 +1,92 @@
+#!/usr/bin/env node
+/**
+ * `weaver`: the library from a shell. `fetch <url>` prints a page as text, `read <url>` a video
+ * as one markdown document, `list <target>` what a profile, channel or search holds. A file on
+ * disk is falcon's (`npx falcon text|page|watch`). Keys come from the environment:
+ * COHERE_API_KEY (or COHERE_API_KEY_1..n), GEMINI_API_KEY.
+ */
+import { readPage } from "@wrenautomation/falcon/text";
+import { Command } from "commander";
+import { PoliteFetcher, userAgent } from "../fetch/fetcher.js";
+import { flatPlaylist } from "../video/list.js";
+import { readVideo, readYoutube } from "../video/read.js";
+/** `NAME`, then `NAME_1`, `NAME_2`, ... in order: a fleet of keys for one vendor. */
+function keys(env, name) {
+    const out = [];
+    const one = env[name];
+    if (one)
+        out.push(one);
+    for (let i = 1;; i++) {
+        const k = env[`${name}_${i}`];
+        if (!k)
+            break;
+        out.push(k);
+    }
+    return out;
+}
+const program = new Command().name("weaver").description("pages and videos on the web as text");
+program
+    .command("fetch <url>")
+    .description("a page as text (polite: one request at a time per host, a contact in the UA)")
+    .option("--contact <who>", "an email address or URL a site operator could reach you at (sent in the User-Agent)", process.env.WEAVER_CONTACT)
+    .option("--json", "title, text, links, mailtos and tels as JSON")
+    .action(async (url, o) => {
+    const fetcher = new PoliteFetcher(userAgent(o.contact));
+    const res = await fetcher.get(url);
+    const page = readPage(res.text, res.url);
+    if (o.json)
+        console.log(JSON.stringify({ status: res.status, url: res.url, ...page }, null, 2));
+    else
+        console.log(page.title ? `${page.title}\n\n${page.text}` : page.text);
+});
+/** The read as one markdown document: its facts as front matter, its title, its sections. */
+export function renderRead(r) {
+    const facts = [
+        ["source", `${r.kind}:${r.id}`],
+        ["title", r.title],
+        ["url", r.url],
+        ["channel", r.channel],
+        ["uploaded", r.uploaded],
+        ["duration", r.duration],
+        ["thumbnail", r.thumbnail],
+    ];
+    const front = facts
+        .filter((f) => f[1] !== undefined)
+        .map(([k, v]) => `${k}: ${typeof v === "string" ? JSON.stringify(v) : v}`)
+        .join("\n");
+    return `---\n${front}\n---\n\n# ${r.title}\n\n${r.body}`;
+}
+program
+    .command("read <url>")
+    .description("a video as one markdown document: YouTube from its captions (plus the screen with GEMINI_API_KEY), any other video by watching it (COHERE_API_KEY)")
+    .option("--json", "the read as JSON")
+    .option("--ffmpeg <path>", "the ffmpeg binary", "ffmpeg")
+    .option("--yt-dlp <cmd>", "the yt-dlp command", "yt-dlp")
+    .action(async (url, o) => {
+    const geminiKeys = keys(process.env, "GEMINI_API_KEY");
+    let read;
+    if (/^https?:\/\/(www\.|m\.)?(youtube\.com|youtu\.be)\//.test(url))
+        read = await readYoutube(url, { ytDlp: o.ytDlp, geminiKeys });
+    else {
+        const cohereKeys = keys(process.env, "COHERE_API_KEY");
+        if (!cohereKeys.length)
+            throw new Error("read needs COHERE_API_KEY for a video with no captions");
+        read = await readVideo(url, { ytDlp: o.ytDlp, ffmpeg: o.ffmpeg, cohereKeys, geminiKeys });
+    }
+    console.log(o.json ? JSON.stringify(read, null, 2) : renderRead(read));
+});
+program
+    .command("list <target>")
+    .description("what a profile, channel or search (ytsearch10:<words>) holds, as JSON, no downloads")
+    .option("--from <n>", "first entry, 1-based", Number)
+    .option("--to <n>", "last entry, inclusive", Number)
+    .option("--yt-dlp <cmd>", "the yt-dlp command", "yt-dlp")
+    .action(async (target, o) => {
+    const range = o.from || o.to ? { start: o.from ?? 1, end: o.to ?? Number.MAX_SAFE_INTEGER } : undefined;
+    console.log(JSON.stringify(await flatPlaylist(o.ytDlp, target, range), null, 2));
+});
+program.parseAsync(process.argv).catch((err) => {
+    console.error(err instanceof Error ? err.message : String(err));
+    process.exit(1);
+});
+//# sourceMappingURL=cli.js.map

package/dist/fetch/fetcher.d.ts

@@ -0,0 +1,84 @@
+/**
+ * A fetch that failed after retries. `status` is the last HTTP status when the
+ * server answered (5xx after retries), null for transport failures: robots handling
+ * needs the difference (a 503 robots.txt means disallow-all; a dead host means the
+ * web's allow default).
+ */
+export declare class FetchError extends Error {
+    name: string;
+    readonly status: number | null;
+    constructor(message: string, status?: number | null);
+}
+/** The declared UA, or a loud failure: contact info is an explicit operator choice, never invented. */
+export declare function userAgent(contact: string | null | undefined): string;
+/** What one GET produced. `url` is the final URL after redirects. */
+export interface FetchResponse {
+    status: number;
+    url: string;
+    text: string;
+    /** The `fetch_tier` the page was kept under, when it was not the caller's own plain fetch. */
+    tier?: string;
+}
+/** The seam crawlers and gates depend on; tests substitute canned pages. */
+export interface Fetcher {
+    readonly userAgent: string;
+    get(url: string): Promise<FetchResponse>;
+}
+export interface DownloadResult {
+    path: string;
+    /** false: already on disk, not re-fetched. */
+    downloaded: boolean;
+    size: number;
+}
+interface PoliteFetcherOptions {
+    /** Seconds between requests to one host. */
+    minInterval?: number;
+    /** Extra uniform(lo, hi) seconds per request so crawls don't tick like a metronome. */
+    jitter?: readonly [number, number] | null;
+    timeout?: number;
+    retries?: number;
+    /** Test seams. */
+    fetch?: typeof globalThis.fetch;
+    sleep?: (ms: number) => Promise<void>;
+    now?: () => number;
+    random?: () => number;
+}
+/**
+ * fetch() wrapper enforcing UA + throttle + retry in one place. The throttle is per
+ * host: a crawl across many small sites is not serialized globally.
+ */
+export declare class PoliteFetcher implements Fetcher {
+    readonly userAgent: string;
+    private readonly minIntervalMs;
+    private readonly jitter;
+    private readonly timeoutMs;
+    private readonly retries;
+    private readonly fetchImpl;
+    private readonly sleep;
+    private readonly now;
+    private readonly random;
+    private readonly lastRequest;
+    /** Requests to one host are serialized so concurrent callers still honor the interval. */
+    private readonly hostQueues;
+    constructor(userAgent: string, opts?: PoliteFetcherOptions);
+    private throttle;
+    private request;
+    /**
+     * One throttled request as the raw `Response`, for a body the caller streams (a file it
+     * filters as it lands) or a method other than GET. `init` wins: its signal replaces the
+     * fetcher's timeout, its headers join the User-Agent. No retries: a half-read stream cannot be
+     * asked again, so the caller decides what a bad status means.
+     */
+    open(url: string, init?: RequestInit): Promise<Response>;
+    /**
+     * GET with throttle and bounded retries. 4xx returns (the caller interprets: a
+     * crawl 404 is data); transport errors and 5xx retry with backoff, then FetchError.
+     */
+    get(url: string): Promise<FetchResponse>;
+    /**
+     * Stream a (possibly large) file to `dest`; skip if already present. Writes to a
+     * .part file and renames, so an interrupted download never masquerades as complete.
+     */
+    download(url: string, dest: string): Promise<DownloadResult>;
+}
+export {};

package/dist/fetch/fetcher.js

@@ -0,0 +1,180 @@
+/**
+ * Polite HTTP fetching, shared by every layer that touches the network: the
+ * discovery ownership gate, the site crawler, bulk downloads. One class holds the
+ * etiquette (declared User-Agent with contact info, per-host throttle, bounded
+ * retries, never re-downloading what is on disk) so it cannot drift between callers.
+ */
+import { createWriteStream } from "node:fs";
+import { mkdir, rename, stat, unlink } from "node:fs/promises";
+import { dirname } from "node:path";
+import { Readable } from "node:stream";
+import { pipeline } from "node:stream/promises";
+const DEFAULT_RETRIES = 3;
+const BACKOFF_MS = 2000;
+/**
+ * A fetch that failed after retries. `status` is the last HTTP status when the
+ * server answered (5xx after retries), null for transport failures: robots handling
+ * needs the difference (a 503 robots.txt means disallow-all; a dead host means the
+ * web's allow default).
+ */
+export class FetchError extends Error {
+    name = "FetchError";
+    status;
+    constructor(message, status = null) {
+        super(message);
+        this.status = status;
+    }
+}
+/** The declared UA, or a loud failure: contact info is an explicit operator choice, never invented. */
+export function userAgent(contact) {
+    if (!contact) {
+        throw new FetchError("outbound fetches need a contact (an email address or URL a remote operator could reach you at)");
+    }
+    return `weaver/0.1 (${contact})`;
+}
+const defaultSleep = (ms) => new Promise((r) => setTimeout(r, ms));
+/**
+ * fetch() wrapper enforcing UA + throttle + retry in one place. The throttle is per
+ * host: a crawl across many small sites is not serialized globally.
+ */
+export class PoliteFetcher {
+    userAgent;
+    minIntervalMs;
+    jitter;
+    timeoutMs;
+    retries;
+    fetchImpl;
+    sleep;
+    now;
+    random;
+    lastRequest = new Map();
+    /** Requests to one host are serialized so concurrent callers still honor the interval. */
+    hostQueues = new Map();
+    constructor(userAgent, opts = {}) {
+        this.userAgent = userAgent;
+        this.minIntervalMs = (opts.minInterval ?? 1.0) * 1000;
+        this.jitter = opts.jitter ?? null;
+        this.timeoutMs = (opts.timeout ?? 30) * 1000;
+        this.retries = Math.max(1, opts.retries ?? DEFAULT_RETRIES);
+        this.fetchImpl = opts.fetch ?? globalThis.fetch;
+        this.sleep = opts.sleep ?? defaultSleep;
+        this.now = opts.now ?? (() => performance.now());
+        this.random = opts.random ?? Math.random;
+    }
+    async throttle(url) {
+        let host = "";
+        try {
+            host = new URL(url).host;
+        }
+        catch {
+            host = "";
+        }
+        const previous = this.hostQueues.get(host) ?? Promise.resolve();
+        const turn = previous.then(async () => {
+            let wait = this.minIntervalMs;
+            if (this.jitter) {
+                const [lo, hi] = this.jitter;
+                wait += (lo + (hi - lo) * this.random()) * 1000;
+            }
+            const elapsed = this.now() - (this.lastRequest.get(host) ?? Number.NEGATIVE_INFINITY);
+            if (elapsed < wait)
+                await this.sleep(wait - elapsed);
+            this.lastRequest.set(host, this.now());
+        });
+        this.hostQueues.set(host, turn.catch(() => { }));
+        await turn;
+    }
+    request(url, init = {}) {
+        const headers = new Headers(init.headers);
+        if (!headers.has("user-agent"))
+            headers.set("User-Agent", this.userAgent);
+        return this.fetchImpl(url, {
+            redirect: "follow",
+            signal: AbortSignal.timeout(this.timeoutMs),
+            ...init,
+            headers,
+        });
+    }
+    /**
+     * One throttled request as the raw `Response`, for a body the caller streams (a file it
+     * filters as it lands) or a method other than GET. `init` wins: its signal replaces the
+     * fetcher's timeout, its headers join the User-Agent. No retries: a half-read stream cannot be
+     * asked again, so the caller decides what a bad status means.
+     */
+    async open(url, init = {}) {
+        await this.throttle(url);
+        return this.request(url, init);
+    }
+    /**
+     * GET with throttle and bounded retries. 4xx returns (the caller interprets: a
+     * crawl 404 is data); transport errors and 5xx retry with backoff, then FetchError.
+     */
+    async get(url) {
+        let lastError = "";
+        let lastStatus = null;
+        for (let attempt = 0; attempt < this.retries; attempt++) {
+            await this.throttle(url);
+            try {
+                const resp = await this.request(url);
+                if (resp.status < 500) {
+                    return { status: resp.status, url: resp.url || url, text: await resp.text() };
+                }
+                lastError = `HTTP ${resp.status}`;
+                lastStatus = resp.status;
+                await resp.body?.cancel().catch(() => { });
+            }
+            catch (err) {
+                lastError = describe(err);
+                lastStatus = null;
+            }
+            if (attempt < this.retries - 1)
+                await this.sleep(BACKOFF_MS * (attempt + 1));
+        }
+        throw new FetchError(`GET ${url} failed after ${this.retries} attempts: ${lastError}`, lastStatus);
+    }
+    /**
+     * Stream a (possibly large) file to `dest`; skip if already present. Writes to a
+     * .part file and renames, so an interrupted download never masquerades as complete.
+     */
+    async download(url, dest) {
+        const existing = await stat(dest).catch(() => null);
+        if (existing)
+            return { path: dest, downloaded: false, size: existing.size };
+        await mkdir(dirname(dest), { recursive: true });
+        const tmp = `${dest}.part`;
+        let lastError = "";
+        for (let attempt = 0; attempt < this.retries; attempt++) {
+            await this.throttle(url);
+            try {
+                const resp = await this.request(url);
+                if (resp.status >= 400) {
+                    lastError = `HTTP ${resp.status}`;
+                    await resp.body?.cancel().catch(() => { });
+                    if (resp.status < 500)
+                        break; // 4xx will not improve on retry
+                    continue;
+                }
+                const body = resp.body ? Readable.fromWeb(resp.body) : Readable.from([]);
+                await pipeline(body, createWriteStream(tmp));
+                await rename(tmp, dest);
+                const size = (await stat(dest)).size;
+                return { path: dest, downloaded: true, size };
+            }
+            catch (err) {
+                lastError = describe(err);
+                if (attempt < this.retries - 1)
+                    await this.sleep(BACKOFF_MS * (attempt + 1));
+            }
+        }
+        await unlink(tmp).catch(() => { });
+        throw new FetchError(`download ${url} failed: ${lastError}`);
+    }
+}
+function describe(err) {
+    if (err instanceof Error) {
+        const cause = err.cause instanceof Error ? `: ${err.cause.message}` : "";
+        return `${err.name}: ${err.message}${cause}`;
+    }
+    return String(err);
+}
+//# sourceMappingURL=fetcher.js.map

package/dist/fetch/index.d.ts

@@ -0,0 +1,9 @@
+/**
+ * The open web by HTTP: a polite fetcher (one request at a time per host, a real User-Agent with a
+ * contact, retries on 429 and 5xx, a byte cap), robots.txt in warn mode, and the fetch ladder
+ * that climbs from a plain GET to a browser (`LadderClient`: autobrowse's `web GET /fetch`).
+ * Turning the page into text is falcon's (`@wrenautomation/falcon/text`). No database here.
+ */
+export * from "./fetcher.js";
+export * from "./ladder.js";
+export * from "./robots.js";

package/dist/fetch/index.js

@@ -0,0 +1,10 @@
+/**
+ * The open web by HTTP: a polite fetcher (one request at a time per host, a real User-Agent with a
+ * contact, retries on 429 and 5xx, a byte cap), robots.txt in warn mode, and the fetch ladder
+ * that climbs from a plain GET to a browser (`LadderClient`: autobrowse's `web GET /fetch`).
+ * Turning the page into text is falcon's (`@wrenautomation/falcon/text`). No database here.
+ */
+export * from "./fetcher.js";
+export * from "./ladder.js";
+export * from "./robots.js";
+//# sourceMappingURL=index.js.map

package/dist/fetch/ladder.d.ts

@@ -0,0 +1,39 @@
+/**
+ * The client of autobrowse's fetch ladder, `web GET /fetch`. A caller asks it for a blocked
+ * page; autobrowse climbs `impersonate`, `render` and `exit`. A block is data in the answer.
+ * A failed call is the other result.
+ */
+/** The rungs a ladder climbs, lowest first. */
+export declare const LADDER_TIERS: readonly ["impersonate", "render", "exit"];
+export type LadderTier = (typeof LADDER_TIERS)[number];
+export declare const isLadderTier: (tier: string | null | undefined) => tier is LadderTier;
+/** Whoever answers `web GET /fetch`: autobrowse's site client, or a test double. */
+export interface LadderClient {
+    call<T>(site: string, method: string, path: string, input?: unknown): Promise<T>;
+}
+/** One rung the ladder tried, and what it met there. */
+export interface FetchStep {
+    tier: string;
+    status: number | null;
+    block: string | null;
+}
+/** What `web GET /fetch` answers (`Fetched`): the fields this side reads. */
+interface Fetched {
+    finalUrl: string;
+    status: number | null;
+    html: string;
+    /** The rung that answered: the last one tried. */
+    tier: LadderTier;
+    /** null: a usable page; else the signal the last rung still hit. */
+    block: string | null;
+    tries: FetchStep[];
+}
+/** `cap`: the day's `fetch` bucket is spent (429). `offline`: any other failed call. */
+type LadderFailure = "cap" | "offline";
+/** One `web GET /fetch`. A failed call is a result, never a throw. */
+export declare function fetchLadder(sites: LadderClient, url: string, from: LadderTier): Promise<{
+    fetched: Fetched;
+} | {
+    failed: LadderFailure;
+}>;
+export {};

package/dist/fetch/ladder.js

@@ -0,0 +1,18 @@
+/**
+ * The client of autobrowse's fetch ladder, `web GET /fetch`. A caller asks it for a blocked
+ * page; autobrowse climbs `impersonate`, `render` and `exit`. A block is data in the answer.
+ * A failed call is the other result.
+ */
+/** The rungs a ladder climbs, lowest first. */
+export const LADDER_TIERS = ["impersonate", "render", "exit"];
+export const isLadderTier = (tier) => LADDER_TIERS.some((t) => t === tier);
+/** One `web GET /fetch`. A failed call is a result, never a throw. */
+export async function fetchLadder(sites, url, from) {
+    try {
+        return { fetched: await sites.call("web", "GET", "/fetch", { url, from }) };
+    }
+    catch (err) {
+        return { failed: err.status === 429 ? "cap" : "offline" };
+    }
+}
+//# sourceMappingURL=ladder.js.map

package/dist/fetch/robots.d.ts

@@ -0,0 +1,14 @@
+import { type Fetcher } from "./fetcher.js";
+export declare class RobotsPolicy {
+    private readonly robot;
+    private readonly disallowAll;
+    private constructor();
+    static allowAll(): RobotsPolicy;
+    static disallowAll(): RobotsPolicy;
+    static parse(text: string): RobotsPolicy;
+    canFetch(userAgent: string, url: string): boolean;
+}
+export type RobotsCache = Map<string, RobotsPolicy>;
+/** The robots policy for this url's scheme+host, fetched once each. */
+export declare function robotsFor(fetcher: Fetcher, url: string, cache: RobotsCache): Promise<RobotsPolicy>;
+export declare function canFetch(fetcher: Fetcher, url: string, cache: RobotsCache): Promise<boolean>;

package/dist/fetch/robots.js

@@ -0,0 +1,92 @@
+/**
+ * robots.txt reading, shared by every page-fetching consumer.
+ *
+ * The policy that binds a fetch is the one on the HOST actually being fetched, over
+ * the SCHEME actually in use: apex and www can serve different files, so policies
+ * are cached per scheme+host and fetched over the URL's own scheme.
+ *
+ * Absence semantics follow RFC 9309: a missing robots.txt (404, or a dead host)
+ * means everything is allowed, but a server that ANSWERED 5xx is saying
+ * "unavailable", which means full disallow.
+ *
+ * Matching is robots-parser's RFC 9309 reading: the group for our product token
+ * (groups naming it merged), else `*`; the longest matching rule wins, Allow on a
+ * tie; `*` and `$` in paths.
+ */
+import parseRobots from "robots-parser";
+import { FetchError } from "./fetcher.js";
+/** robots-parser checks a URL's origin against the file's; ours are read per origin, so one stand-in. */
+const ORIGIN = "https://robots.invalid";
+// A CommonJS module: at runtime the default import is the function its typings
+// declare, which NodeNext resolution types as a namespace instead.
+const robotsParser = parseRobots;
+export class RobotsPolicy {
+    robot;
+    disallowAll;
+    constructor(robot, disallowAll) {
+        this.robot = robot;
+        this.disallowAll = disallowAll;
+    }
+    static allowAll() {
+        return new RobotsPolicy(null, false);
+    }
+    static disallowAll() {
+        return new RobotsPolicy(null, true);
+    }
+    static parse(text) {
+        return new RobotsPolicy(robotsParser(`${ORIGIN}/robots.txt`, text), false);
+    }
+    canFetch(userAgent, url) {
+        if (this.disallowAll)
+            return false;
+        if (!this.robot)
+            return true;
+        return this.robot.isAllowed(onOrigin(url), userAgent) ?? true;
+    }
+}
+/** The url's path and query on the stand-in origin; an unparseable url reads as `/`. */
+function onOrigin(url) {
+    try {
+        const u = new URL(url);
+        return `${ORIGIN}${u.pathname}${u.search}`;
+    }
+    catch {
+        return `${ORIGIN}/`;
+    }
+}
+/** The robots policy for this url's scheme+host, fetched once each. */
+export async function robotsFor(fetcher, url, cache) {
+    let key;
+    try {
+        const u = new URL(url);
+        key = `${u.protocol}//${u.host}`;
+    }
+    catch {
+        key = `https://${url}`;
+    }
+    const hit = cache.get(key);
+    if (hit)
+        return hit;
+    let policy;
+    try {
+        const resp = await fetcher.get(`${key}/robots.txt`);
+        policy = resp.status === 200 ? RobotsPolicy.parse(resp.text) : RobotsPolicy.allowAll();
+    }
+    catch (err) {
+        if (err instanceof FetchError && err.status !== null && err.status >= 500) {
+            policy = RobotsPolicy.disallowAll(); // "unavailable" is not "absent"
+        }
+        else if (err instanceof FetchError) {
+            policy = RobotsPolicy.allowAll(); // unreachable host: the web's allow default
+        }
+        else {
+            throw err;
+        }
+    }
+    cache.set(key, policy);
+    return policy;
+}
+export async function canFetch(fetcher, url, cache) {
+    return (await robotsFor(fetcher, url, cache)).canFetch(fetcher.userAgent, url);
+}
+//# sourceMappingURL=robots.js.map

package/dist/index.d.ts

@@ -0,0 +1,9 @@
+/**
+ * weaver: the web in, documents out. Two doors: `./fetch` (polite HTTP, robots, the fetch ladder
+ * up to a browser) and `./video` (YouTube, any other video and podcasts read from a URL through
+ * yt-dlp and falcon; what a channel or search holds). What reads a file, bytes or text is
+ * falcon's (`@wrenautomation/falcon`). The source interface, registry, runner and ledger come in
+ * phase 3.5c of wren's `designs/2026-10-26-research-engine.md`.
+ */
+export * from "./fetch/index.js";
+export * from "./video/index.js";

package/dist/index.js

@@ -0,0 +1,10 @@
+/**
+ * weaver: the web in, documents out. Two doors: `./fetch` (polite HTTP, robots, the fetch ladder
+ * up to a browser) and `./video` (YouTube, any other video and podcasts read from a URL through
+ * yt-dlp and falcon; what a channel or search holds). What reads a file, bytes or text is
+ * falcon's (`@wrenautomation/falcon`). The source interface, registry, runner and ledger come in
+ * phase 3.5c of wren's `designs/2026-10-26-research-engine.md`.
+ */
+export * from "./fetch/index.js";
+export * from "./video/index.js";
+//# sourceMappingURL=index.js.map

package/dist/video/index.d.ts

@@ -0,0 +1,8 @@
+/**
+ * Video on the web: the readers over a URL (`read.ts`: YouTube by captions or by URL, any other
+ * video by yt-dlp plus falcon's `watch`, a podcast by its audio URL), `watch` over a URL or a
+ * file, and the list reader (`list.ts`: what a profile, channel or search holds). The reading
+ * itself is falcon's. Every caller passes its own keys and tools. No database here.
+ */
+export * from "./list.js";
+export * from "./read.js";

package/dist/video/index.js

@@ -0,0 +1,9 @@
+/**
+ * Video on the web: the readers over a URL (`read.ts`: YouTube by captions or by URL, any other
+ * video by yt-dlp plus falcon's `watch`, a podcast by its audio URL), `watch` over a URL or a
+ * file, and the list reader (`list.ts`: what a profile, channel or search holds). The reading
+ * itself is falcon's. Every caller passes its own keys and tools. No database here.
+ */
+export * from "./list.js";
+export * from "./read.js";
+//# sourceMappingURL=index.js.map

package/dist/video/list.d.ts

@@ -0,0 +1,27 @@
+/** One entry of yt-dlp's playlist view (`--flat-playlist`): a TikTok profile, a YouTube search. */
+export interface YtDlpEntry {
+    id?: string;
+    url?: string;
+    webpage_url?: string;
+    title?: string;
+    description?: string;
+    timestamp?: number;
+    duration?: number;
+    uploader?: string;
+    channel?: string;
+    thumbnails?: {
+        url?: string;
+    }[];
+    view_count?: number;
+    like_count?: number;
+    comment_count?: number;
+}
+/**
+ * yt-dlp's playlist view of `target` (a profile, a channel's videos, `ytsearchN:<words>`), no
+ * downloads: the entries with an id, in yt-dlp's order. `range` is 1-based and inclusive. Some
+ * sites answer only a home IP. `ytDlp` is the command, split on spaces (`uvx yt-dlp`).
+ */
+export declare function flatPlaylist(ytDlp: string, target: string, range?: {
+    start: number;
+    end: number;
+}): Promise<YtDlpEntry[]>;

package/dist/video/list.js

@@ -0,0 +1,24 @@
+/**
+ * What a profile, channel or search holds, through yt-dlp's playlist view: titles, ids and
+ * counts, no downloads. The list reader beside the piece readers in `read.ts`.
+ */
+import { runFor } from "@wrenautomation/falcon/video";
+const LIST_MS = 180_000;
+/**
+ * yt-dlp's playlist view of `target` (a profile, a channel's videos, `ytsearchN:<words>`), no
+ * downloads: the entries with an id, in yt-dlp's order. `range` is 1-based and inclusive. Some
+ * sites answer only a home IP. `ytDlp` is the command, split on spaces (`uvx yt-dlp`).
+ */
+export async function flatPlaylist(ytDlp, target, range) {
+    const [cmd, ...pre] = ytDlp.split(/\s+/);
+    const { stdout } = await runFor(cmd, [
+        ...pre,
+        ...["-J", "--flat-playlist", "--no-warnings"],
+        ...(range
+            ? ["--playlist-start", String(range.start), "--playlist-end", String(range.end)]
+            : []),
+        target,
+    ], LIST_MS, { maxBuffer: 64 * 1024 * 1024 });
+    return (JSON.parse(stdout).entries ?? []).filter((e) => e.id);
+}
+//# sourceMappingURL=list.js.map

package/dist/video/read.d.ts

@@ -0,0 +1,68 @@
+import { type VideoInfo, type VideoRead, type Watched, type WatchOptions } from "@wrenautomation/falcon/video";
+/** What yt-dlp's `-j` prints for a download; only what the readers use. */
+export interface VideoMeta extends Pick<VideoInfo, "id" | "title" | "channel" | "upload_date" | "webpage_url" | "duration" | "thumbnail"> {
+    extractor_key?: string;
+    description?: string;
+    uploader?: string;
+    width?: number;
+    height?: number;
+}
+/** yt-dlp's download of a reel or video: 540p on the short side, remuxed to mp4. */
+export declare function download(url: string, ytDlp: string, dir: string): Promise<{
+    file: string;
+    info: VideoMeta;
+}>;
+/**
+ * Watch a video: a URL yt-dlp fetches first (`ytDlp` is the command, split on spaces:
+ * `uvx yt-dlp`), or a file on disk, which goes straight to falcon's `watch`. A URL's yt-dlp
+ * metadata comes back as `info`; a duration ffmpeg could not read falls back to it.
+ */
+export declare function watch(input: string, opts: WatchOptions & {
+    ytDlp?: string;
+}): Promise<Watched & {
+    info?: VideoMeta;
+}>;
+/**
+ * A YouTube video from its captions and chapters, through `yt-dlp` (`ytDlp` is the command,
+ * split on spaces: `uvx yt-dlp`). With Gemini keys, an `## On screen` section follows: what the
+ * video showed, read from the frames, since the captions carry only the speech.
+ */
+export declare function readYoutube(url: string, opts: {
+    ytDlp: string;
+    geminiKeys?: readonly string[];
+    fetchFn?: typeof fetch;
+}): Promise<VideoRead>;
+/**
+ * A YouTube video read from its URL alone, no download and no home IP: Gemini hears the speech
+ * and reads the screen in `CLIP_S` clips (one clip when the length isn't known). The title,
+ * channel and thumbnail come from YouTube's oEmbed. For a server; a machine with yt-dlp and a
+ * home IP reads the captions instead (`readYoutube`).
+ */
+export declare function readYoutubeByUrl(url: string, opts: {
+    geminiKeys: readonly string[];
+    fetchFn?: typeof fetch;
+    durationS?: number | null | undefined;
+}): Promise<VideoRead>;
+/**
+ * Any other video `yt-dlp` fetches (an Instagram reel, TikTok, X). No captions there, so falcon's
+ * `watch` hears the audio and reads the frames with Cohere (Gemini adds its read of the screen
+ * when a key works). The body holds the caption, then Speech, On screen and Cuts.
+ */
+export declare function readVideo(url: string, opts: WatchOptions & {
+    ytDlp: string;
+}): Promise<VideoRead>;
+/**
+ * A podcast episode heard from its audio URL: downloaded to a temp file, then falcon's
+ * `hearAudio` (ffmpeg clips, Gemini transcribes each). `url` is the episode's page; `title` and
+ * `creator` are what its feed said. The id is a hash of the audio URL.
+ */
+export declare function readAudio(audioUrl: string, opts: {
+    url: string;
+    title?: string | null;
+    creator?: string | null;
+    geminiKeys: readonly string[];
+    fetchFn?: typeof fetch;
+    ffmpeg?: string;
+    /** Seconds per clip; `CLIP_S` unless a test cuts a short tone. */
+    clipS?: number;
+}): Promise<VideoRead>;

package/dist/video/read.js

@@ -0,0 +1,263 @@
+/**
+ * A video or podcast on the web read as one markdown document (falcon's `VideoRead`). Four
+ * readers: a YouTube video from its captions and chapters (yt-dlp) plus Gemini's read of the
+ * screen; a YouTube video from its URL alone (Gemini hears and reads it: no download, no home
+ * IP); any other video yt-dlp fetches (falcon's `watch`: Cohere hears the audio and reads the
+ * frames); a podcast episode from its audio URL (fetched, then falcon's `hearAudio`). Everything
+ * that takes a URL or spawns yt-dlp lives here; the reading over a file is falcon's. Every caller
+ * passes its own keys and tools. Nothing here touches a database.
+ */
+import { createHash } from "node:crypto";
+import { mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { askGemini, askShown, CLIP_S, captionsBody, clock, hearAudio, isoDate, real, renderWatched, runFor, SHOWN, TOOL_MS, watch as watchFile, } from "@wrenautomation/falcon/video";
+/** yt-dlp's download of a reel or video: 540p on the short side, remuxed to mp4. */
+export async function download(url, ytDlp, dir) {
+    const [cmd, ...pre] = ytDlp.split(/\s+/);
+    const { stdout } = await runFor(cmd, [
+        ...pre,
+        ...["-j", "--no-simulate", "--no-warnings", "--no-playlist"],
+        // `res` is the short side, so a vertical reel gets 540x960, not its 1080x1920.
+        ...["-S", "res:540", "-f", "bv*+ba/b", "--remux-video", "mp4"],
+        ...["-o", join(dir, "video.%(ext)s"), url],
+    ], TOOL_MS, { maxBuffer: 256 * 1024 * 1024 });
+    const file = (await readdir(dir)).find((f) => f.endsWith(".mp4"));
+    if (!file)
+        throw new Error(`${url}: yt-dlp wrote no mp4`);
+    return { file: join(dir, file), info: JSON.parse(stdout) };
+}
+/**
+ * Watch a video: a URL yt-dlp fetches first (`ytDlp` is the command, split on spaces:
+ * `uvx yt-dlp`), or a file on disk, which goes straight to falcon's `watch`. A URL's yt-dlp
+ * metadata comes back as `info`; a duration ffmpeg could not read falls back to it.
+ */
+export async function watch(input, opts) {
+    if (!/^https?:\/\//.test(input))
+        return watchFile(input, opts);
+    const dir = await mkdtemp(join(tmpdir(), "weaver-watch-"));
+    try {
+        const { file, info } = await download(input, opts.ytDlp ?? "yt-dlp", dir);
+        const watched = await watchFile(file, opts);
+        return { ...watched, durationS: watched.durationS || info.duration || 0, info };
+    }
+    finally {
+        await rm(dir, { recursive: true, force: true });
+    }
+}
+const captionTracks = (info) => [info.subtitles?.en, info.automatic_captions?.en, info.automatic_captions?.["en-orig"]]
+    .flatMap((t) => t ?? [])
+    .filter((t) => t.ext === "json3");
+const SHOWN_ASK = (start, paraphrase) => `${SHOWN(start, paraphrase)} Markdown. If nothing is shown, reply with the single word none.`;
+/**
+ * What the video shows on screen (documents, slides, tables, templates), read by Gemini straight
+ * from the YouTube URL in `CLIP_S` clips. One markdown block per clip, timestamps absolute.
+ * Empty when `keys` is empty.
+ */
+async function screenText(url, durationS, keys, fetchFn) {
+    if (!keys.length)
+        return "";
+    const clips = Array.from({ length: Math.ceil(durationS / CLIP_S) }, (_, i) => i * CLIP_S);
+    const read = async (start) => {
+        const end = Math.min(start + CLIP_S, durationS);
+        const body = (paraphrase) => ({
+            contents: [
+                {
+                    parts: [
+                        {
+                            fileData: { fileUri: url },
+                            videoMetadata: { startOffset: `${start}s`, endOffset: `${end}s`, fps: 0.5 },
+                        },
+                        { text: SHOWN_ASK(start, paraphrase) },
+                    ],
+                },
+            ],
+            // Low resolution reads a shown document fine at a quarter of the tokens.
+            generationConfig: { mediaResolution: "MEDIA_RESOLUTION_LOW" },
+        });
+        const r = await askShown(body, keys, fetchFn, clock(start));
+        if ("text" in r)
+            return r.text;
+        // One unread clip must not lose the other 23: leave a marker and carry on.
+        console.warn(`gemini ${clock(start)}: unread, last ${r.unread}`);
+        return `[${clock(start)}] (not read: ${r.unread})`;
+    };
+    const out = [];
+    // 4 clips at a time; limits are per key, so go wider if it ever drags.
+    for (let i = 0; i < clips.length; i += 4) {
+        const texts = await Promise.all(clips.slice(i, i + 4).map(read));
+        out.push(...texts.filter(real));
+    }
+    return out.join("\n\n");
+}
+/**
+ * A YouTube video from its captions and chapters, through `yt-dlp` (`ytDlp` is the command,
+ * split on spaces: `uvx yt-dlp`). With Gemini keys, an `## On screen` section follows: what the
+ * video showed, read from the frames, since the captions carry only the speech.
+ */
+export async function readYoutube(url, opts) {
+    const fetchFn = opts.fetchFn ?? fetch;
+    const [cmd, ...pre] = opts.ytDlp.split(/\s+/);
+    const { stdout } = await runFor(cmd, [...pre, "-J", "--no-warnings", "--no-playlist", url], TOOL_MS, { maxBuffer: 256 * 1024 * 1024 });
+    const info = JSON.parse(stdout);
+    const tracks = captionTracks(info);
+    if (!tracks.length)
+        throw new Error(`${url}: no English captions (manual or auto) to read`);
+    // YouTube throttles single caption URLs (429); the same text sits behind several, so try each.
+    let json;
+    let last = 0;
+    for (const t of tracks) {
+        const res = await fetchFn(t.url);
+        last = res.status;
+        if (!res.ok)
+            continue;
+        json = (await res.json());
+        break;
+    }
+    if (!json)
+        throw new Error(`${url}: captions fetch HTTP ${last} on all ${tracks.length} tracks`);
+    let body = captionsBody(info, json);
+    const screen = await screenText(url, info.duration ?? 0, opts.geminiKeys ?? [], fetchFn);
+    if (screen)
+        body += `\n## On screen\n\nWhat the video showed (documents, slides, tables, visuals), read from the frames; the transcript above has only the speech.\n\n${screen}\n`;
+    const uploaded = isoDate(info.upload_date);
+    return {
+        kind: "youtube",
+        id: info.id,
+        title: info.title,
+        url: info.webpage_url ?? `https://www.youtube.com/watch?v=${info.id}`,
+        ...(info.channel ? { channel: info.channel } : {}),
+        ...(uploaded ? { uploaded } : {}),
+        ...(info.duration ? { duration: Math.round(info.duration) } : {}),
+        ...(info.thumbnail ? { thumbnail: info.thumbnail } : {}),
+        body,
+    };
+}
+const SPEECH = "Transcribe the speech in this video: every word spoken, in paragraphs, each starting with its [m:ss] video time. Fix only obvious mishearings. Markdown, no heading. If nobody speaks, reply with the single word none.";
+/**
+ * A YouTube video read from its URL alone, no download and no home IP: Gemini hears the speech
+ * and reads the screen in `CLIP_S` clips (one clip when the length isn't known). The title,
+ * channel and thumbnail come from YouTube's oEmbed. For a server; a machine with yt-dlp and a
+ * home IP reads the captions instead (`readYoutube`).
+ */
+export async function readYoutubeByUrl(url, opts) {
+    const fetchFn = opts.fetchFn ?? fetch;
+    const id = /(?:v=|youtu\.be\/|shorts\/|live\/)([\w-]{11})/.exec(url)?.[1];
+    if (!id)
+        throw new Error(`${url}: not a YouTube video`);
+    const page = `https://www.youtube.com/watch?v=${id}`;
+    const meta = (await fetchFn(`https://www.youtube.com/oembed?format=json&url=${encodeURIComponent(page)}`)
+        .then((r) => (r.ok ? r.json() : {}))
+        .catch(() => ({})));
+    const duration = opts.durationS ?? 0;
+    const starts = duration
+        ? Array.from({ length: Math.ceil(duration / CLIP_S) }, (_, i) => i * CLIP_S)
+        : [null];
+    const readClip = async (start) => {
+        const part = {
+            fileData: { fileUri: page },
+            ...(start === null
+                ? {}
+                : {
+                    videoMetadata: {
+                        startOffset: `${start}s`,
+                        endOffset: `${Math.min(start + CLIP_S, duration)}s`,
+                    },
+                }),
+        };
+        const ask = (text) => ({
+            contents: [{ parts: [part, { text }] }],
+            generationConfig: { mediaResolution: "MEDIA_RESOLUTION_LOW" },
+        });
+        const label = `${id} ${clock(start ?? 0)}`;
+        const [said, screen] = await Promise.all([
+            askGemini(ask(SPEECH), opts.geminiKeys, fetchFn, `${label} speech`),
+            askShown((paraphrase) => ask(SHOWN_ASK(start ?? 0, paraphrase)), opts.geminiKeys, fetchFn, `${label} screen`),
+        ]);
+        if ("unread" in said)
+            throw new Error(`gemini ${label} speech: unread, last ${said.unread}`);
+        return { said: said.text, screen: "text" in screen ? screen.text : "" };
+    };
+    const clips = [];
+    // 4 clips at a time, kept in order: a key fleet spreads them.
+    for (let i = 0; i < starts.length; i += 4)
+        clips.push(...(await Promise.all(starts.slice(i, i + 4).map(readClip))));
+    const speech = clips.map((c) => c.said).filter(real);
+    const shown = clips.map((c) => c.screen).filter(real);
+    return {
+        kind: "youtube",
+        id,
+        title: meta.title?.trim() || page,
+        url: page,
+        ...(meta.author_name ? { channel: meta.author_name } : {}),
+        ...(duration ? { duration: Math.round(duration) } : {}),
+        ...(meta.thumbnail_url ? { thumbnail: meta.thumbnail_url } : {}),
+        body: `## Speech\n\n${speech.join("\n\n") || "(no speech)"}\n${shown.length ? `\n## On screen\n\n${shown.join("\n\n")}\n` : ""}`,
+    };
+}
+/**
+ * Any other video `yt-dlp` fetches (an Instagram reel, TikTok, X). No captions there, so falcon's
+ * `watch` hears the audio and reads the frames with Cohere (Gemini adds its read of the screen
+ * when a key works). The body holds the caption, then Speech, On screen and Cuts.
+ */
+export async function readVideo(url, opts) {
+    const watched = await watch(url, opts);
+    const info = watched.info;
+    const caption = info.description?.trim() ?? "";
+    const by = info.channel ?? info.uploader;
+    const uploaded = isoDate(info.upload_date);
+    const duration = info.duration ?? watched.durationS;
+    return {
+        kind: (info.extractor_key ?? "video").toLowerCase(),
+        id: info.id,
+        title: caption.split("\n")[0]?.slice(0, 100) || info.title,
+        url: info.webpage_url ?? url,
+        ...(by ? { channel: by } : {}),
+        ...(uploaded ? { uploaded } : {}),
+        ...(duration ? { duration: Math.round(duration) } : {}),
+        ...(info.thumbnail ? { thumbnail: info.thumbnail } : {}),
+        body: `${caption ? `## Caption\n\n${caption}\n\n` : ""}${renderWatched(watched)}\n`,
+    };
+}
+const AUDIO_MAX = 600_000_000;
+/**
+ * A podcast episode heard from its audio URL: downloaded to a temp file, then falcon's
+ * `hearAudio` (ffmpeg clips, Gemini transcribes each). `url` is the episode's page; `title` and
+ * `creator` are what its feed said. The id is a hash of the audio URL.
+ */
+export async function readAudio(audioUrl, opts) {
+    if (!opts.geminiKeys.length)
+        throw new Error(`${audioUrl}: needs Gemini keys to transcribe`);
+    const fetchFn = opts.fetchFn ?? fetch;
+    const dir = await mkdtemp(join(tmpdir(), "weaver-audio-"));
+    try {
+        const res = await fetchFn(audioUrl, { signal: AbortSignal.timeout(600_000) });
+        if (!res.ok || !res.body)
+            throw new Error(`${audioUrl}: HTTP ${res.status}`);
+        if (Number(res.headers.get("content-length") ?? 0) > AUDIO_MAX)
+            throw new Error(`${audioUrl}: over ${AUDIO_MAX / 1e6} MB`);
+        const whole = join(dir, "episode");
+        await writeFile(whole, Buffer.from(await res.arrayBuffer()));
+        const id = createHash("sha1").update(audioUrl).digest("hex").slice(0, 12);
+        const heard = await hearAudio(whole, {
+            geminiKeys: opts.geminiKeys,
+            fetchFn,
+            ...(opts.ffmpeg ? { ffmpeg: opts.ffmpeg } : {}),
+            ...(opts.clipS ? { clipS: opts.clipS } : {}),
+            label: id,
+        });
+        return {
+            kind: "podcast",
+            id,
+            title: opts.title?.trim() || opts.url,
+            url: opts.url,
+            ...(opts.creator ? { channel: opts.creator } : {}),
+            ...(heard.duration ? { duration: heard.duration } : {}),
+            body: heard.speech,
+        };
+    }
+    finally {
+        await rm(dir, { recursive: true, force: true });
+    }
+}
+//# sourceMappingURL=read.js.map

package/package.json

@@ -1,6 +1,61 @@
 {
   "name": "@wrenautomation/weaver",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "The web in, documents out. Polite HTTP fetching with robots and a fetch ladder up to a browser; YouTube, any other video and podcasts read from a URL through yt-dlp and @wrenautomation/falcon; what a channel or search holds. The source interface, runner and ledger of Wren's research engine come next.",
+  "license": "PolyForm-Shield-1.0.0",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/wrenautomation/weaver.git"
+  },
+  "type": "module",
+  "files": [
+    "dist",
+    "!dist/**/*.test.*",
+    "!dist/**/*.map"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./fetch": {
+      "types": "./dist/fetch/index.d.ts",
+      "import": "./dist/fetch/index.js"
+    },
+    "./video": {
+      "types": "./dist/video/index.d.ts",
+      "import": "./dist/video/index.js"
+    }
+  },
+  "bin": {
+    "weaver": "dist/app/cli.js"
+  },
+  "engines": {
+    "node": ">=24"
+  },
+  "packageManager": "pnpm@9.15.0",
+  "scripts": {
+    "prepare": "git config core.hooksPath .githooks || true",
+    "lint": "biome check --error-on-warnings .",
+    "lint:fix": "biome check --write .",
+    "typecheck": "tsc --noEmit",
+    "build": "rm -rf dist && tsc && chmod +x dist/app/cli.js",
+    "test": "vitest run",
+    "gates": "lockf -t 900 /tmp/wren_automation-checks.lock sh -c 'pnpm lint && pnpm typecheck && pnpm test'",
+    "prepublishOnly": "pnpm build"
+  },
+  "dependencies": {
+    "@wrenautomation/falcon": "0.3.0",
+    "commander": "15.0.0",
+    "robots-parser": "3.0.1"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "2.5.14",
+    "@types/node": "22.20.3",
+    "typescript": "5.9.3",
+    "vitest": "5.0.1"
+  }
+}