@wrenautomation/birdwatch 0.0.0-stage → 0.29.0

package/LICENSE.md

@@ -0,0 +1,61 @@
+# PolyForm Strict License 1.0.0
+
+<https://polyformproject.org/licenses/strict/1.0.0>
+
+## Acceptance
+
+In order to get any license under these terms, you must agree to them as both strict obligations and conditions to all your licenses.
+
+## Copyright License
+
+The licensor grants you a copyright license for the software to do everything you might do with the software that would otherwise infringe the licensor's copyright in it for any permitted purpose, other than distributing the software or making changes or new works based on the software.
+
+## Patent License
+
+The licensor grants you a patent license for the software that covers patent claims the licensor can license, or becomes able to license, that you would infringe by using the software.
+
+## Noncommercial Purposes
+
+Any noncommercial purpose is a permitted purpose.
+
+## Personal Uses
+
+Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, is use for a permitted purpose.
+
+## Noncommercial Organizations
+
+Use by any charitable organization, educational institution, public research organization, public safety or health organization, environmental protection organization, or government institution is use for a permitted purpose regardless of the source of funding or obligations resulting from the funding.
+
+## Fair Use
+
+You may have "fair use" rights for the software under the law. These terms do not limit them.
+
+## No Other Rights
+
+These terms do not allow you to sublicense or transfer any of your licenses to anyone else, or prevent the licensor from granting licenses to anyone else.  These terms do not imply any other licenses.
+
+## Patent Defense
+
+If you make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.
+
+## Violations
+
+The first time you are notified in writing that you have violated any of these terms, or done anything with the software not covered by your licenses, your licenses can nonetheless continue if you come into full compliance with these terms, and take practical steps to correct past violations, within 32 days of receiving notice.  Otherwise, all your licenses end immediately.
+
+## No Liability
+
+***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.***
+
+## Definitions
+
+The **licensor** is the individual or entity offering these terms, and the **software** is the software the licensor makes available under these terms.
+
+**You** refers to the individual or entity agreeing to these terms.
+
+**Your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization.  **Control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise.  Control can be direct or indirect.
+
+**Your licenses** are all the licenses granted to you for the software under these terms.
+
+**Use** means anything you do with the software requiring one of your licenses.
+
+Required Notice: Copyright 2026 William Jin (https://wrenautomation.com)

package/README.md

@@ -1,3 +1,338 @@
-# Temporary Holding Version
+# birdwatch
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Research from the open web, driven by an agent one step per command. The agent plans, triages,
+reviews and stops; birdwatch searches, opens pages, extracts findings behind a quote gate, checks
+them, and keeps every record in a ledger. Every finding is a sentence with a quote on a page the
+run read. `skills/birdwatch/SKILL.md` (in this repo) is how an agent drives it.
+
+Reading the web and files (the fetch ladder, HTML and PDF to text, video) is
+[`@wrenautomation/read`](https://github.com/wrenautomation/engine/tree/main/packages/read)'s. Every model call goes through
+[`@wrenautomation/models`](https://github.com/wrenautomation/engine/tree/main/packages/models). Until 0.29.0 this was three packages: raven (the planner,
+now `./plan`), falcon's `./document` and weaver (the engine, sources, ledger and CLI).
+
+| Import | Does |
+|---|---|
+| `@wrenautomation/birdwatch/engine` | `Source` (a name, `search` and/or `open`, a cost), `registry` and the plan's allow-list; the agent's calls (`planRound`, `search`, `open`, `extract`, `check`, `add`, `accept`, `brief`), one step each; the judge's passes (`judgeHits`, `judgeDocuments`, `judgeFindings`, `judgeStats`); `run(deps, options)`: plan, fetch, triage, read, extract, check (a sample of findings against their documents), review, round after round until the planner's edge says stop; `fold` rebuilds a run's state from its records, so a run resumes from the step it stopped on; `Transient`, the failure worth another try; the planner's `Ask`, `PlanFile` and record shapes re-exported |
+| `@wrenautomation/birdwatch/sources` | `searxng`, `exa` (search), `youtube` (search and read, yt-dlp), `fetcher` (open, the polite fetcher), `ladder` (open, autobrowse's fetch ladder up to a browser), `reddit` and `facebook` (open a thread or a group post through autobrowse's site routes), `google` (Google's own page through the desk's browser: each result a hit, the AI Overview one hit with its text as the snippet, each page it cites a hit), `arxiv` (search and read: the API inside `BIRDWATCH_ARXIV_CATEGORIES`, each hit led by its OpenAlex citation count and comment; the paper read whole from its HTML, else its PDF; one call per 3 s), `paste` (a person's URLs), `ledger` (hits earlier runs saw). A query with a `site` skips a searcher that cannot keep to it (`Source.keeps`: `youtube` keeps to YouTube only). A follow (`Source.follows`, `follow`) lists a page's author's posts (`youtube`: a channel's uploads, from a video or the channel, and a TikTok profile; `reddit`: a user's top posts and comments) or pages like it (`exa`: findSimilar) |
+| `@wrenautomation/birdwatch/ledger` | the records (`query`, `hit`, `document`, `finding`, `verdict`, `call`, `spend`, `snapshot`, `edge`, `done`, `steer`, `gap`, `check`, `pattern`, `found`, `judge`, `person`), `FilesLedger` (a folder: `runs/<run>/<writer>.jsonl`, `plans/<kind>.json`), `MemoryLedger`, yield rows per source, person, channel and kind |
+| `@wrenautomation/birdwatch/plan` | the planner: `plan` (an ask, proposals and yield rows in; typed queries out, each with `why`, `from`, `by`), `triage` (read or skip per hit), `review` (the edge after a round: `stop`, `again`, `back`), the shapes (`Ask`, `Plan`, `PlanFile`, `Query`, `Edge`, `Snapshot`), a stock plan file per ask kind (`web`, `video`, `channel`, `person`, `site`: prompts, sources, budget, `age`, the query tree's guide), the renderers. Pure functions and prompts; the model is injected. The runner uses all of it; an agent makes the three decisions itself and uses the shapes |
+| `@wrenautomation/birdwatch/document` | the document side: `Document`, `Finding`, `Proof`, `Verdict` (zod), readers by media (`html`, `pdf`, `text`), `claimExtractor` (findings that quote the document, through the gate, and a verdict on the document: its quality and the ask's gap; no cap on findings or their length, and an answer the model's output cap cut is continued from its last kept quote), the `Model` seam it takes and the `Cut` a model throws when its cap cut an answer |
+
+Every record carries `from` (ai, human or tool) and `by` (a model id, a source, a person). A
+person's pasted URL, a person's keep or drop, are the same records as the model's. The plan
+file in `plans/<kind>.json` beside the ledger (prompts, sources, budget) is where a run learns:
+`birdwatch learn <kind>` writes its next version from what a run taught (sources, budget, age, a
+prompt), on the person's agreement; a kind with no tuned file runs the stock one
+(`birdwatch plans <kind>` prints it).
+
+## CLI
+
+Needs Node 24+. `read` and `watch` need yt-dlp and ffmpeg.
+
+```sh
+npm i @wrenautomation/birdwatch
+npx birdwatch fetch https://example.com/report --contact you@example.com   # a page as text
+GEMINI_API_KEY=... npx birdwatch read https://www.youtube.com/watch?v=...    # captions + screen
+COHERE_API_KEY=... npx birdwatch read https://www.instagram.com/reel/...     # watched: speech, screen, cuts
+npx birdwatch list 'ytsearch10:cold email' --to 10                           # hits as JSON, no downloads
+npx birdwatch text report.pdf                                             # a file on disk as text
+npx birdwatch page saved.html --url https://example.com/report            # title, text, links as JSON
+COHERE_API_KEY=... npx birdwatch watch talk.mp4                           # speech, on-screen text, cuts
+npx birdwatch plans web > ledger/plans/web.json                           # a stock plan file, to tune
+```
+
+The engine has two drivers over the same steps. An agent (Claude Code, or any agent with a
+shell) drives it one command at a time: it plans, triages, reviews and stops in its own head and
+the engine does the fixed work (search, open, extract behind the quote gate, check) and keeps the
+records. `run` is the runner: the planner's prompts make those decisions and the loop runs alone. Both
+write the same ledger, so `run --resume` takes over a run an agent started and `brief` reads a
+run the runner made. Every command prints JSON.
+
+```sh
+export BIRDWATCH_AGENT=claude-code                     # the agent's id on every record it writes
+birdwatch plan "how do staffing firms win repeat clients" --known "referrals convert"   # a run; prints its id
+birdwatch plan "who buys growth help" --premise "coaches are the buyer"   # a premise gets a tree facet that tests it
+birdwatch brief 01M5YS...                              # the ask, budget, sources, what is open, hits, findings, cost
+birdwatch recall staffing repeat clients --not 01M5YS... # earlier runs' findings: quote, URL, source date, run; --also <ledger> per extra folder
+birdwatch search 01M5YS... "staffing repeat client rate" --why "the ask's words"       # one query, then every searcher
+echo '[{"terms":"...","why":"..."},{"terms":"...","why":"..."}]' | birdwatch search 01M5YS...   # a batch
+birdwatch search 01M5YS... < tree.json                 # a query tree, one tier: {facets: [{name, why, tries: [[{terms, why}], [...]]}]}
+birdwatch search 01M5YS... < tree.json                 # after triage, the same file: the next try of each facet still thin in kept hits
+birdwatch add call 01M5YS... 01M5YT... keep --why "a survey with numbers"               # triage: a hit kept
+birdwatch add call 01M5YS... 01M5YU... drop --why "a vendor page"
+birdwatch open 01M5YS... 01M5YT... --url https://a.example/report --why "numbers"      # keep these, read every kept hit
+birdwatch open 01M5YS... --with ladder                                                  # an opener the run lacks joins it, to open only
+birdwatch show 01M5YS... 01M5YV...                     # a document's text, for the agent to read itself
+birdwatch add finding 01M5YS... 01M5YV... --text "412 firms were surveyed." --quote "A 2025 survey of 412 firms"
+birdwatch add check 01M5YS... 01M5YW... --ok --why "the quote says it"   # the agent's own check; --fail also drops
+birdwatch extract 01M5YS...                            # or a model behind the quote gate, every document without a verdict
+birdwatch check 01M5YS... -n 3                         # or a model's sample of findings read against their documents
+birdwatch check 01M5YS... --patterns                   # every unchecked finding a pattern rests on
+birdwatch accept 01M5YS... keep --over 0.8 --why "serves the ask"   # the judge's hints in bulk, at the agent's bar
+birdwatch accept 01M5YS... class --min 0.8 --why "head of page"   # also: drop --under 0.2
+birdwatch add gap 01M5YS... "no UK numbers yet"
+birdwatch add steer 01M5YS... "numbers only" --human will                                # the person's word, relayed
+birdwatch add pattern 01M5YS... "Repeat clients come from referrals." --on 01M5YV... 01M5YW...   # what findings say together; sites counted
+birdwatch add edge 01M5YS... again --why "thin on the UK"                                # the review: a new round
+birdwatch plan --run 01M5YS...                         # round 2: known, steer and what round 1 left open, in its snapshot
+birdwatch plan "coaches: who pays for setters" --parent 01M5YS... --why "the best niche of round 1"   # a child run
+birdwatch add dead 01M5YT... spend --why "found consumers, not buyers"   # later tries skip the facet
+birdwatch follow 01M5YS... --from 01M5YV... --as posts --why "the speaker's other talks"   # the author's own posts; --as similar: pages like it; a paper: --as refs (what it cites) or citers (what cites it)
+birdwatch add person 01M5YS... "Bee Speaker" --profile https://www.youtube.com/@bee --aka Bee --why "one speaker"   # one person across sites
+birdwatch learn web --sources searxng exa --rounds 4   # the plan file's next version, after a good run
+birdwatch learn web --thin 2                           # the tree's guide: a try under 2 new hits falls through
+birdwatch learn web --show                             # the current one
+birdwatch add edge 01M5YS... stop --why "enough"
+```
+
+`plan` opens a round: its snapshot carries the ask, what is known (the findings so far), every
+steer, what the last round left open, the allow-list and the budget. No queries: the agent's come
+through `search` or `add query`, never capped. The budget is a guard rail: `brief` shows it, and
+an edge `again` past `budget.rounds` prints a warning, never a refusal. `search`, `open`,
+`extract` and `check` each run one step and print what it wrote (a document's text as `chars`);
+a step that gave up exits 2 with its note. `open` reads every kept hit of the run that has no
+document; `extract` every document without a verdict; `check` a sample across documents of the
+kept findings not yet checked. `add` writes one record by hand: a query, a finding (its quote
+must be on the document word for word, or the add fails with the gate's why), a check (the
+agent's own read of a finding against its document; a fail also drops it), a call (keep or
+drop a hit or a finding), an edge (`again`, `stop`, `back --to <step>`), a gap, or a steer (always
+the person's: `--human [name]`). Every `add` takes `--why` where a decision needs one. A
+`--human` record is the person's word through the agent: `from: human`, `by` their name.
+
+A round's search is a tree when the agent gives one: facets (the sides a researcher covers: a
+number, a how, a who, a counter-view, a primary source; the plan file's `tree` guide names them),
+each with tries in order, each try a list of queries. `search` runs one tier per call, in one
+fetch: the first call, every facet's first try; each call after, the next try of each facet
+with fewer kept hits than the guide's `thin` (3 by default). A facet at `thin` is done; a facet
+with hits no call has named waits for triage; a facet out of tries and still thin is a gap,
+written once. The yield a try falls through on is what triage kept, never what a searcher
+returned: every searcher returns a page. `--all` runs every facet's next try, whatever triage
+kept. A page a second query brings back is a `found` record on the first hit, never a second
+hit, and it counts as kept for every facet that reached it. `brief.hits.uncalled` lists each
+hit with the facets that found it, most first; `brief.tree` shows every try with what it wrote,
+`kept` and `uncalled` per facet. A guide facet no query served is named in the note.
+`learn --thin <n>` or `--tree <file>` tunes the guide.
+
+The agent reads for itself: `show <run> <document>` prints a document's stored text, `add
+finding` writes what it read behind the same quote gate, `add check` its own check. `extract` and
+`check` put a model behind the same gate instead, for bulk or for the runner; both paths write
+the same records.
+
+The runner alone:
+
+```sh
+BIRDWATCH_CONTACT=you@example.com npx birdwatch run "how do staffing firms win repeat clients" --known "referrals convert"
+npx birdwatch run "..." --kind video --propose "cold email teardown" --by will   # your queries, merged with the model's
+npx birdwatch run "..." --url https://a.example/report https://b.example/post    # pages you want read, first
+npx birdwatch run --resume 01M5YS... --url https://a.example/report             # from the step it stopped on; same --url, same --model
+npx birdwatch run "..." --pause                                                   # stop after each round's review
+npx birdwatch add steer 01M5YS... "numbers only; skip vendor pages" --human       # what you say, first in every prompt after
+npx birdwatch run --resume 01M5YS... --pause                                      # the next round, steered
+npx birdwatch add gap 01M5YS... "no UK numbers yet" --human                       # what the run lacks: its review and next plan read it
+npx birdwatch add edge 01M5YS... back --to fetch --why "searxng was down" --human # rerun one step of the round; then --resume
+npx birdwatch trace 01M5YS... --type finding spend                                # the run's records, JSON lines
+npx birdwatch add call 01M5YS... 01M5YT... drop --why "a forecast, not a result" --human   # a call in your name
+npx birdwatch yield --source exa                                                  # hits, reads, proposed, kept
+npx birdwatch sources                                                             # what this machine can run
+npx birdwatch serve --port 4747                                                   # the ledger over HTTP for a viewer, read only
+```
+
+The model is `anthropic` by default (Claude Haiku direct, `ANTHROPIC_API_KEY`; `--model
+anthropic:sonnet|opus|id` for another), `--model gateway[:alias]` for Wren's LLM gateway
+(`WREN_LLM_GATEWAY_URL`, `WREN_LLM_GATEWAY_TOKEN`), or any command that reads a prompt on stdin
+and prints the answer (`claude -p --output-format json` is Claude Code headless). The model does
+the bulk (extract, check); `--decider <spec>` (`BIRDWATCH_DECIDER`) takes the steps that decide what
+comes next (plan, triage, review): `anthropic:sonnet` when the model is Haiku, else the model. Sources come
+from the environment: `SEARXNG_URL` (and `SEARXNG_TOKEN`, a bearer, for an instance behind a
+proxy), `EXA_API_KEY` (or `_1..n`), `GEMINI_API_KEY` for videos without captions,
+`BIRDWATCH_CONTACT` for the fetcher and `arxiv` (no contact, no page opens; `OPENALEX_API_KEY`, free, gives `arxiv` its own citation budget, and `SEMANTIC_SCHOLAR_API_KEY`, free, its own rate for `--as refs`; `--sources arxiv` names it, since no stock plan file does), and for the ladder either
+`AUTOBROWSE_URL` and `AUTOBROWSE_TOKEN` (a worker's HTTP face) or `RESTATE_INGRESS_URL` and
+`RESTATE_AUTH_TOKEN` (the Restate ingress a desk worker sits behind: `POST /desk/call`;
+`AUTOBROWSE_SERVICE` for another service name). The same worker gives the `reddit` and
+`facebook` readers. `BIRDWATCH_ACCOUNTS` names the account each site reader may sign in as
+(`reddit=reddit@alt`); a site with no entry reads signed out only. The ledger folder is `--ledger` or
+`BIRDWATCH_LEDGER` (default `./ledger`); `--writer` or `BIRDWATCH_WRITER` names your file in it.
+The judge is `--judge` or `BIRDWATCH_JUDGE` (`jev`, `jev:<model>`, `fake`, `off`); with neither,
+`jev` when `TYPESAFE_API_KEY` is set, else none. `--no-judge` turns it off for one command.
+
+Each open source claims the URLs it opens (`opens`): `youtube` a YouTube page, `fetcher` and
+`ladder` any other http(s) page, `reddit` a Reddit thread or comment permalink, `facebook` a
+public group post. A site's own reader (`first`) gets its turn before the general openers,
+whatever the allow-list's order: it answers the thread as data, where a fetched page is a wall.
+`reddit` reads signed out first (`reddit-public`, the thread's feed); only when that misses and
+`BIRDWATCH_ACCOUNTS` names an account does it ask again signed in as it. `facebook` is always
+signed out. A spent cap or a refused read is a miss; a busy profile (503) or no answer retries.
+A refusal carries the worker's access code in its reason: `need_login`, `platform_blocked`,
+`rate_limited` or `upstream_broken`. The last two never read signed in: that fixes neither.
+The read step asks before a call, so a source that was never
+going to open a hit writes no spend. A page that was fetched with nothing usable (a 403, a block
+the ladder's last rung still hit, a media no reader takes, a ladder answer that is not HTML such
+as a PDF) is a miss: the source throws, its `spend` says `ok: false` with the reason, the next
+opener gets its turn, and a hit every opener missed is a `gap` that lists each reason. A page
+that answered 200 and still holds nothing (a captcha wall, a "not found" shell, under 200
+characters of text) is the same miss, with the title or the count as the reason. A run's
+snapshot pins its sources at `plan`: `sources`, the allow-list, which search, open and follow;
+`opens`, the registry's other openers (unless `--sources` was given), which open and follow but
+never search. `open --with <source>` writes a new snapshot that adds one to `opens` (the read
+note names them when hits missed). The
+ladder costs browser time and one of the worker's daily `fetch` bucket per call. Exa's search
+asks for one highlight per result (three sentences, billed per result) as the hit's `snippet`.
+
+Every model call is a `spend` record with the model id, the tokens in and out, and `usd` from
+the list price (`src/app/prices.ts`: Claude Haiku, Sonnet and Opus 5.5; the gateway's `free`
+alias is $0; another model keeps its tokens, no price). `run` prints the run's total on stderr
+and as `cost` in its JSON, and the last review's edge (`edge`: next and why) the same way;
+`brief` prints the total as `cost`.
+
+`run` exits 2 when the run stopped on a step that gave up (no model, no source, a block):
+`--resume` continues it once the cause is fixed. A failure worth another try (a 429, a 5xx, a
+dropped connection) is tried three times with a pause, every attempt a `spend` record; a step
+that still cannot go on writes no `done`, and `run` waits and resumes it, five times at most,
+before it exits 2. Searches, reads and extracts fan out (four reads or extracts at a time);
+each document's records land as its batch settles, and a step run again skips what its records
+show done. An extract that fails for good (the model's answer would not parse, a 400) stops the
+step with its findings so far on record and `run` exits 2: `--resume` asks only for the
+documents without a verdict. A model answer the output cap cut is not a failure: the whole
+findings before the cut go through the gate and the model is asked again for the rest of the
+document, with those findings known (`./document`'s `Cut`; the `spend` carries both calls' tokens).
+
+`--pause` stops the run after a review that says `again`, with no exit code: between rounds
+you (or a Claude Code session) read `brief`, keep or drop findings (`add call`), and steer the
+run with free text (`add steer`). Every `steer` record is rendered first in every prompt of the
+rounds after it, under "The person says:". `--resume` goes on. A review that says `again` into
+a round past `budget.rounds` parks the run the same way (`parked` says so, no exit code): the
+budget is a guard rail, not a stop rule. `--resume` goes one more round; `plan --run` opens it
+by hand.
+
+The steps talk in English records. A query every searcher answered with no new hit, and a kept
+hit no source could open (a 403, a block the ladder could not pass), each write a `gap`; so does
+`add gap`. The round's review reads the gaps and a person's keep or drop with its why; the next
+round's plan reads them, and the review's why on `again`, under "The last round left open".
+`add edge back --to <step>` writes a person's edge: the fold takes it like the planner's. A `pattern`
+(`add pattern`) is what several kept findings say together, in the agent's words; its count of
+distinct sites is computed from the findings' proofs, never claimed, and `brief.patterns` lists
+them for the report. A run can be a child (`--parent <run> --why`): its snapshot names the parent, its plan reads
+what the parent and siblings found (recall over those runs), and its budget is the parent's at
+most. The plan file's `runs.depth` (2) is a guard rail: a deeper child warns. Two caps stop: the
+children per parent round (`runs.children`, 6) and the whole tree's reads (`runs.reads`, pinned
+on the root's snapshot as `treeReads`; by default the root's reads × rounds × (children + 1)).
+`open` stops or cuts its hits at the tree's cap; `brief.lineage` and `brief.treeRoom` show the
+ancestors and the room left. A
+stopped child's stop, patterns and dead ends (`add dead`) reach the parent's next plan as
+known lines; a running child is open. A `class` (`add class`) says what kind of source a document is (primary data, survey,
+practitioner, vendor, forum, news, other), with why; its findings take it, and a class on one
+finding overrides it. `brief.patterns[].classes` counts sources per class: distinct sites,
+but each forum thread on its own; `echo` flags a pattern whose classed sources are all vendors,
+and `unchecked` lists its findings no check has read. Only the agent or a person writes a
+class. A tree facet's `needs` names the classes that can prove it; `tree.facets[].lacks` says
+its read pages have none (and the facet is not `enough`), and a facet whose every try pins one site is flagged. A `draft` (`add draft`, JSON items on stdin) is a rough draft over the kept
+findings and patterns: the study's draft gate drops an item that cites nothing kept or has a number
+no cited quote has. `brief.draft` shows the latest. It is never the final write.
+
+The check step reads a sample of the round's kept findings (`budget.checks`, 3 by default, or a
+tenth of the kept findings, whichever is more; 0 is none; spread across documents);
+`check --patterns` reads every unchecked finding a pattern rests on, against the text around each quote. The extract gate proves a quote is
+on the page; the check asks whether it backs the sentence. A finding that fails gets a `check`
+record and a drop call; `add call <finding> keep` puts it back. A fail comes with a fix when one
+fits: a narrower sentence the same quote backs. The fix passes the extract gate and a second
+check, then lands as a new finding with `fixes` naming the old one. The review sees every check, and a finding
+carries its document's date, so the review weighs age as the plan file's `age` line says.
+
+The judge (Jev, from `@wrenautomation/models/judge`) answers typed questions with probabilities
+and no text, so it never writes a call or a check. `search` asks it of each new hit: does it
+serve the ask, whose side wrote it (the people the ask is about, or a seller to them), what
+class. `open` asks each new page its class. `check` asks each kept finding with a quote: does the
+quote back it, does it credit another source. Each answer is a `judge` record and each call a
+spend. They hint and route: `brief.hits.uncalled` sorts by facets then the judge's serves, and
+shows its answers; `documents.unclassed[].guess` shows its class; a facet with `needs` is not
+enough while its kept hits have a class (the agent's, else the judge's) and none is needed;
+`patterns[].cites` lists findings that credit another source, and the vendor echo warning names
+them. `check` reads the sample plus each finding the judge backs under 0.8: a judge's yes never
+takes one out. `accept` turns the hints into records at the agent's own bar, each with the
+judge's number in its why. `brief.judge` counts its calls, cost, and how often the agent and the
+check agreed with it. A trial (0.22.0): `open` also asks each page its `worth` (would a careful
+researcher cite it for the ask). `documents.toExtract[].worth` shows it; `brief.judge.worth`
+gives its AUC against "a finding on it was kept", and what a gate at a half would skip. Nothing
+gates on it yet.
+
+From 0.25.0 the judge reads a page whole: the body to its references (an appendix stays), in
+parts of 64,000 characters, 4 at most, each answer the most yes of any part. Jev's input is
+cheap (about $0.04 per million tokens), so a paper costs about a tenth of a cent. A run's
+criteria (`--criterion name="question"`, on `run` and `plan`) are the person's own yes-or-no
+tests of a good source: the judge asks each one on every hit and every page. Every prompt shows
+them with the person's steer, and the runner's triage sees the judge's odds on each hit (serves
+and each test) as a hint. Their mean sorts `hits.uncalled` and `documents.toExtract`; `brief.judge.criteria` gives each one's AUC against
+"a finding on it was kept", on hits and on pages. A sort and a hint, never a gate.
+
+From 0.27.0 the extractor reads the body too (the reference list holds no finding). Each review
+sees every earlier round's new hits and kept findings, and decides when to stop. A paper's
+`follow --as refs` and `--as citers` list what it cites and what cites it. `--decider` runs plan,
+triage and review on Sonnet while Haiku does the bulk. A check (25 papers, Sonnet as reference):
+on the whole page, Jev's criteria odds rank Sonnet's yes over its no with AUC 0.94 to 1.00; on the
+abstract, 0.77 to 1.00.
+
+A follow is a query record with `follow: {url, as, document?}` and terms `<as> of <url>`. The
+fetch step sends it to the sources whose `follows` takes it, `follow` lists up to 30, and the
+hits go through triage, the read and the caps like a search's. Each query goes to the allow-list
+as it stood when it was written: a source `follow --with` lets in joins `opens`, so it runs
+follows, never searches. A
+`person` names one author across sites (profile URLs, names they post under); a document is
+theirs when a posts follow from their profile found it, when its URL sits under a profile, or
+when its channel (or its hit's) is one of their names. `brief.people` counts each person's
+documents, kept findings and sites; `brief.unfollowed` lists kept pages no follow started from
+and no person owns.
+
+`serve` puts the ledger on 127.0.0.1 for a viewer (the Wren portal's Research page). Read only:
+the CLI stays the one writer. `GET /runs` gives a row per run (ask, parent, round, next step,
+counts, spend). `GET /runs/:run` gives a `RunView` (`v: 1`): the brief, every query with the hits
+it wrote and found again, every hit with its sources, queries, call and judge, every page with its
+class, verdict and person, every finding with its checks, the timeline of decisions and step ends
+with their whys, and spend per source. `GET /runs/:run/documents/:doc` gives a page's stored text.
+`GET /events` sends `event: run` with `{run}` each time a run's files change, and a ping each 25 s;
+the viewer then gets that run again. CORS answers `--origin` (the portal by default) and any
+localhost, and says yes to Chrome's private-network preflight. The types live in
+`src/engine/view.ts`; a host copies them and checks `v`, never imports them.
+
+## Library
+
+```ts
+import { PoliteFetcher, userAgent } from "@wrenautomation/read/fetch";
+import { readPage } from "@wrenautomation/read/text";
+
+const fetcher = new PoliteFetcher(userAgent("you@example.com"));
+const res = await fetcher.get(url);
+const page = readPage(res.text, res.url);
+```
+
+The engine, with your own model and sources:
+
+```ts
+import { claimExtractor } from "@wrenautomation/birdwatch/document";
+import { registry, run } from "@wrenautomation/birdwatch/engine";
+import { FilesLedger, ulids } from "@wrenautomation/birdwatch/ledger";
+import { exa, fetcherSource } from "@wrenautomation/birdwatch/sources";
+
+const ledger = new FilesLedger("./ledger", "will");
+const sources = registry(exa({ keys }), fetcherSource({ fetcher }));
+const result = await run(
+  { sources, ledger, model, extractor: claimExtractor, clock: () => new Date(), ids: ulids(), runner: "will" },
+  { ask: { text: "how do staffing firms win repeat clients", kind: "web", known: [] }, planFile },
+);
+result.state.findings; // each with its proof: a quote on a page the run read
+```
+
+`RunOptions.steps` bounds one call (64 by default): a loop that advances a run a little per pass
+passes a few and calls again with `resume`. A host with its own ledger, sources or model mounts
+the same commands: `addEngineCommands(program, { ledger, sources, model, extractor })` from
+`@wrenautomation/birdwatch` (wren's `research` command is these over Postgres).
+
+## Develop
+
+```sh
+pnpm install
+pnpm lint && pnpm typecheck && pnpm test
+```
+
+A `v*` tag publishes to npm (`.github/workflows/release.yml`, trusted publishing: no stored token).
+
+License: PolyForm Strict 1.0.0 (`LICENSE.md`).

package/dist/app/cli.d.ts

@@ -0,0 +1,4 @@
+#!/usr/bin/env node
+import { type VideoRead } from "@wrenautomation/read/video";
+/** The read as one markdown document: its facts as front matter, its title, its sections. */
+export declare function renderRead(r: VideoRead): string;

package/dist/app/cli.js

@@ -0,0 +1,149 @@
+#!/usr/bin/env node
+/**
+ * `birdwatch`: the engine and its readers from a shell. The readers: `fetch <url>` prints a page
+ * as text, `read <url>` a video as one markdown document, `list <target>` what a profile, channel
+ * or search holds, `text <file>` an HTML or PDF file as text, `page <file>` an HTML file's title,
+ * text and links, `watch <video>` what a video file shows and says. `plans [kind]` prints a stock
+ * plan file to tune. Keys come from the environment: COHERE_API_KEY (or COHERE_API_KEY_1..n),
+ * GEMINI_API_KEY. The engine's commands are in `engine-cli.ts`: the agent's (`plan`, `search`,
+ * `open`, `extract`, `check`, `add`, `brief`), the runner's (`run`), the plan file's (`learn`) and
+ * the ledger's (`trace`, `yield`, `sources`). A person's word goes through `add ... --human`.
+ */
+import { readFile } from "node:fs/promises";
+import { PoliteFetcher, userAgent } from "@wrenautomation/read/fetch";
+import { htmlText, pdfText, readPage } from "@wrenautomation/read/text";
+import { cohereEye, cohereSpeech, flatPlaylist, frameCounts, geminiVideo, readVideo, readYoutube, renderWatched, watchFile, } from "@wrenautomation/read/video";
+import { Command } from "commander";
+import { AskKind, defaultPlanFile, PLAN_KINDS } from "../plan/index.js";
+import { addEngineCommands, keys } from "./engine-cli.js";
+const program = new Command()
+    .name("birdwatch")
+    .description("research from the open web: the engine's steps, its ledger, and its readers");
+program
+    .command("fetch <url>")
+    .description("a page as text (polite: one request at a time per host, a contact in the UA)")
+    .option("--contact <who>", "an email address or URL a site operator could reach you at (sent in the User-Agent)", process.env.BIRDWATCH_CONTACT)
+    .option("--json", "title, text, links, mailtos and tels as JSON")
+    .action(async (url, o) => {
+    const fetcher = new PoliteFetcher(userAgent(o.contact));
+    const res = await fetcher.get(url);
+    const page = readPage(res.text, res.url);
+    if (o.json)
+        console.log(JSON.stringify({ status: res.status, url: res.url, ...page }, null, 2));
+    else
+        console.log(page.title ? `${page.title}\n\n${page.text}` : page.text);
+});
+/** The read as one markdown document: its facts as front matter, its title, its sections. */
+export function renderRead(r) {
+    const facts = [
+        ["source", `${r.kind}:${r.id}`],
+        ["title", r.title],
+        ["url", r.url],
+        ["channel", r.channel],
+        ["uploaded", r.uploaded],
+        ["duration", r.duration],
+        ["thumbnail", r.thumbnail],
+    ];
+    const front = facts
+        .filter((f) => f[1] !== undefined)
+        .map(([k, v]) => `${k}: ${typeof v === "string" ? JSON.stringify(v) : v}`)
+        .join("\n");
+    return `---\n${front}\n---\n\n# ${r.title}\n\n${r.body}`;
+}
+program
+    .command("read <url>")
+    .description("a video as one markdown document: YouTube from its captions (plus the screen with GEMINI_API_KEY), any other video by watching it (COHERE_API_KEY)")
+    .option("--json", "the read as JSON")
+    .option("--ffmpeg <path>", "the ffmpeg binary", "ffmpeg")
+    .option("--yt-dlp <cmd>", "the yt-dlp command", "yt-dlp")
+    .action(async (url, o) => {
+    const video = await geminiVideo(keys(process.env, "GEMINI_API_KEY"));
+    let read;
+    if (/^https?:\/\/(www\.|m\.)?(youtube\.com|youtu\.be)\//.test(url))
+        read = await readYoutube(url, { ytDlp: o.ytDlp, video });
+    else {
+        const cohereKeys = keys(process.env, "COHERE_API_KEY");
+        if (!cohereKeys.length)
+            throw new Error("read needs COHERE_API_KEY for a video with no captions");
+        read = await readVideo(url, {
+            ytDlp: o.ytDlp,
+            ffmpeg: o.ffmpeg,
+            eye: cohereEye(cohereKeys),
+            ear: cohereSpeech(cohereKeys),
+            video,
+        });
+    }
+    console.log(o.json ? JSON.stringify(read, null, 2) : renderRead(read));
+});
+program
+    .command("list <target>")
+    .description("what a profile, channel or search (ytsearch10:<words>) holds, as JSON, no downloads")
+    .option("--from <n>", "first entry, 1-based", Number)
+    .option("--to <n>", "last entry, inclusive", Number)
+    .option("--yt-dlp <cmd>", "the yt-dlp command", "yt-dlp")
+    .action(async (target, o) => {
+    const range = o.from || o.to ? { start: o.from ?? 1, end: o.to ?? Number.MAX_SAFE_INTEGER } : undefined;
+    console.log(JSON.stringify(await flatPlaylist(o.ytDlp, target, range), null, 2));
+});
+program
+    .command("text <file>")
+    .description("an HTML or PDF file as text")
+    .action(async (file) => {
+    const bytes = await readFile(file);
+    const out = /\.pdf$/i.test(file) ? await pdfText(bytes) : htmlText(bytes.toString("utf8"));
+    if (out === null)
+        throw new Error(`${file}: no text (scanned, encrypted or broken)`);
+    console.log(out);
+});
+program
+    .command("page <file>")
+    .description("an HTML file's title, text, links, mailtos and tels as JSON")
+    .option("--url <url>", "the page's address, to resolve its links")
+    .action(async (file, o) => {
+    const html = await readFile(file, "utf8");
+    console.log(JSON.stringify(readPage(html, o.url ?? `file://${file}`), null, 2));
+});
+program
+    .command("watch <video>")
+    .description("a video file's speech, on-screen text and cuts")
+    .option("--json", "the result as JSON")
+    .option("--no-speech", "skip the transcription")
+    .option("--language <code>", "spoken language, ISO 639-1", "en")
+    .option("--every <s>", "seconds between frames", Number, 2)
+    .option("--ffmpeg <path>", "the ffmpeg binary", "ffmpeg")
+    .action(async (input, o) => {
+    const cohereKeys = keys(process.env, "COHERE_API_KEY");
+    if (!cohereKeys.length)
+        throw new Error("watch needs COHERE_API_KEY");
+    const watched = await watchFile(input, {
+        eye: cohereEye(cohereKeys),
+        ear: cohereSpeech(cohereKeys),
+        video: await geminiVideo(keys(process.env, "GEMINI_API_KEY")),
+        ffmpeg: o.ffmpeg,
+        speech: o.speech,
+        language: o.language,
+        everyS: o.every,
+    });
+    if (o.json)
+        console.log(JSON.stringify(watched, null, 2));
+    else
+        console.log(renderWatched(watched));
+    const { read, unread } = frameCounts(watched);
+    console.error(`${watched.durationS}s, ${read} frames read, ${unread} not read, ${watched.cuts.length} cuts, ${watched.speech.length} chars of speech`);
+});
+program
+    .command("plans [kind]")
+    .description("a stock plan file to copy and tune; no kind lists them all")
+    .action((kind) => {
+    const print = (x) => console.log(JSON.stringify(x, null, 2));
+    if (kind)
+        print(defaultPlanFile(AskKind.parse(kind)));
+    else
+        print(PLAN_KINDS.map((k) => defaultPlanFile(k)));
+});
+addEngineCommands(program);
+program.parseAsync(process.argv).catch((err) => {
+    console.error(err instanceof Error ? err.message : String(err));
+    process.exit(1);
+});
+//# sourceMappingURL=cli.js.map

package/dist/app/cli.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.js","sourceRoot":"","sources":["../../src/app/cli.ts"],"names":[],"mappings":";AACA;;;;;;;;;GASG;AACH,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAAE,aAAa,EAAE,SAAS,EAAE,MAAM,4BAA4B,CAAC;AACtE,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,2BAA2B,CAAC;AACxE,OAAO,EACL,SAAS,EACT,YAAY,EACZ,YAAY,EACZ,WAAW,EACX,WAAW,EACX,SAAS,EACT,WAAW,EACX,aAAa,EAEb,SAAS,GACV,MAAM,4BAA4B,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACpC,OAAO,EAAE,OAAO,EAAE,eAAe,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AACxE,OAAO,EAAE,iBAAiB,EAAE,IAAI,EAAE,MAAM,iBAAiB,CAAC;AAE1D,MAAM,OAAO,GAAG,IAAI,OAAO,EAAE;KAC1B,IAAI,CAAC,WAAW,CAAC;KACjB,WAAW,CAAC,6EAA6E,CAAC,CAAC;AAE9F,OAAO;KACJ,OAAO,CAAC,aAAa,CAAC;KACtB,WAAW,CAAC,8EAA8E,CAAC;KAC3F,MAAM,CACL,iBAAiB,EACjB,qFAAqF,EACrF,OAAO,CAAC,GAAG,CAAC,iBAAiB,CAC9B;KACA,MAAM,CAAC,QAAQ,EAAE,8CAA8C,CAAC;KAChE,MAAM,CAAC,KAAK,EAAE,GAAW,EAAE,CAAsC,EAAE,EAAE;IACpE,MAAM,OAAO,GAAG,IAAI,aAAa,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;IACxD,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IACnC,MAAM,IAAI,GAAG,QAAQ,CAAC,GAAG,CAAC,IAAI,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC;IACzC,IAAI,CAAC,CAAC,IAAI;QAAE,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,GAAG,EAAE,GAAG,CAAC,GAAG,EAAE,GAAG,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;;QAC3F,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC7E,CAAC,CAAC,CAAC;AAEL,6FAA6F;AAC7F,MAAM,UAAU,UAAU,CAAC,CAAY;IACrC,MAAM,KAAK,GAA4C;QACrD,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,EAAE,EAAE,CAAC;QAC/B,CAAC,OAAO,EAAE,CAAC,CAAC,KAAK,CAAC;QAClB,CAAC,KAAK,EAAE,CAAC,CAAC,GAAG,CAAC;QACd,CAAC,SAAS,EAAE,CAAC,CAAC,OAAO,CAAC;QACtB,CAAC,UAAU,EAAE,CAAC,CAAC,QAAQ,CAAC;QACxB,CAAC,UAAU,EAAE,CAAC,CAAC,QAAQ,CAAC;QACxB,CAAC,WAAW,EAAE,CAAC,CAAC,SAAS,CAAC;KAC3B,CAAC;IACF,MAAM,KAAK,GAAG,KAAK;SAChB,MAAM,CAAC,CAAC,CAAC,EAAkC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,SAAS,CAAC;SACjE,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;SACzE,IAAI,CAAC,IAAI,CAAC,CAAC;IACd,OAAO,QAAQ,KAAK,cAAc,CAAC,CAAC,KAAK,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC;AAC3D,CAAC;AAED,OAAO;KACJ,OAAO,CAAC,YAAY,CAAC;KACrB,WAAW,CACV,oJAAoJ,CACrJ;KACA,MAAM,CAAC,QAAQ,EAAE,kBAAkB,CAAC;KACpC,MAAM,CAAC,iBAAiB,EAAE,mBAAmB,EAAE,QAAQ,CAAC;KACxD,MAAM,CAAC,gBAAgB,EAAE,oBAAoB,EAAE,QAAQ,CAAC;KACxD,MAAM,CAAC,KAAK,EAAE,GAAW,EAAE,CAAoD,EAAE,EAAE;IAClF,MAAM,KAAK,GAAG,MAAM,WAAW,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,gBAAgB,CAAC,CAAC,CAAC;IACrE,IAAI,IAAe,CAAC;IACpB,IAAI,oDAAoD,CAAC,IAAI,CAAC,GAAG,CAAC;QAChE,IAAI,GAAG,MAAM,WAAW,CAAC,GAAG,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC;SACtD,CAAC;QACJ,MAAM,UAAU,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,gBAAgB,CAAC,CAAC;QACvD,IAAI,CAAC,UAAU,CAAC,MAAM;YACpB,MAAM,IAAI,KAAK,CAAC,wDAAwD,CAAC,CAAC;QAC5E,IAAI,GAAG,MAAM,SAAS,CAAC,GAAG,EAAE;YAC1B,KAAK,EAAE,CAAC,CAAC,KAAK;YACd,MAAM,EAAE,CAAC,CAAC,MAAM;YAChB,GAAG,EAAE,SAAS,CAAC,UAAU,CAAC;YAC1B,GAAG,EAAE,YAAY,CAAC,UAAU,CAAC;YAC7B,KAAK;SACN,CAAC,CAAC;IACL,CAAC;IACD,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC;AACzE,CAAC,CAAC,CAAC;AAEL,OAAO;KACJ,OAAO,CAAC,eAAe,CAAC;KACxB,WAAW,CACV,qFAAqF,CACtF;KACA,MAAM,CAAC,YAAY,EAAE,sBAAsB,EAAE,MAAM,CAAC;KACpD,MAAM,CAAC,UAAU,EAAE,uBAAuB,EAAE,MAAM,CAAC;KACnD,MAAM,CAAC,gBAAgB,EAAE,oBAAoB,EAAE,QAAQ,CAAC;KACxD,MAAM,CAAC,KAAK,EAAE,MAAc,EAAE,CAAgD,EAAE,EAAE;IACjF,MAAM,KAAK,GACT,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,EAAE,GAAG,EAAE,CAAC,CAAC,EAAE,IAAI,MAAM,CAAC,gBAAgB,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC5F,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,YAAY,CAAC,CAAC,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;AACnF,CAAC,CAAC,CAAC;AAEL,OAAO;KACJ,OAAO,CAAC,aAAa,CAAC;KACtB,WAAW,CAAC,6BAA6B,CAAC;KAC1C,MAAM,CAAC,KAAK,EAAE,IAAY,EAAE,EAAE;IAC7B,MAAM,KAAK,GAAG,MAAM,QAAQ,CAAC,IAAI,CAAC,CAAC;IACnC,MAAM,GAAG,GAAG,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,MAAM,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,KAAK,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC;IAC3F,IAAI,GAAG,KAAK,IAAI;QAAE,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,0CAA0C,CAAC,CAAC;IACrF,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;AACnB,CAAC,CAAC,CAAC;AAEL,OAAO;KACJ,OAAO,CAAC,aAAa,CAAC;KACtB,WAAW,CAAC,6DAA6D,CAAC;KAC1E,MAAM,CAAC,aAAa,EAAE,0CAA0C,CAAC;KACjE,MAAM,CAAC,KAAK,EAAE,IAAY,EAAE,CAAmB,EAAE,EAAE;IAClD,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAC1C,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,CAAC,GAAG,IAAI,UAAU,IAAI,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;AAClF,CAAC,CAAC,CAAC;AAUL,OAAO;KACJ,OAAO,CAAC,eAAe,CAAC;KACxB,WAAW,CAAC,gDAAgD,CAAC;KAC7D,MAAM,CAAC,QAAQ,EAAE,oBAAoB,CAAC;KACtC,MAAM,CAAC,aAAa,EAAE,wBAAwB,CAAC;KAC/C,MAAM,CAAC,mBAAmB,EAAE,4BAA4B,EAAE,IAAI,CAAC;KAC/D,MAAM,CAAC,aAAa,EAAE,wBAAwB,EAAE,MAAM,EAAE,CAAC,CAAC;KAC1D,MAAM,CAAC,iBAAiB,EAAE,mBAAmB,EAAE,QAAQ,CAAC;KACxD,MAAM,CAAC,KAAK,EAAE,KAAa,EAAE,CAAa,EAAE,EAAE;IAC7C,MAAM,UAAU,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,gBAAgB,CAAC,CAAC;IACvD,IAAI,CAAC,UAAU,CAAC,MAAM;QAAE,MAAM,IAAI,KAAK,CAAC,4BAA4B,CAAC,CAAC;IACtE,MAAM,OAAO,GAAG,MAAM,SAAS,CAAC,KAAK,EAAE;QACrC,GAAG,EAAE,SAAS,CAAC,UAAU,CAAC;QAC1B,GAAG,EAAE,YAAY,CAAC,UAAU,CAAC;QAC7B,KAAK,EAAE,MAAM,WAAW,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,EAAE,gBAAgB,CAAC,CAAC;QAC7D,MAAM,EAAE,CAAC,CAAC,MAAM;QAChB,MAAM,EAAE,CAAC,CAAC,MAAM;QAChB,QAAQ,EAAE,CAAC,CAAC,QAAQ;QACpB,MAAM,EAAE,CAAC,CAAC,KAAK;KAChB,CAAC,CAAC;IACH,IAAI,CAAC,CAAC,IAAI;QAAE,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;;QACrD,OAAO,CAAC,GAAG,CAAC,aAAa,CAAC,OAAO,CAAC,CAAC,CAAC;IACzC,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,WAAW,CAAC,OAAO,CAAC,CAAC;IAC9C,OAAO,CAAC,KAAK,CACX,GAAG,OAAO,CAAC,SAAS,MAAM,IAAI,iBAAiB,MAAM,cAAc,OAAO,CAAC,IAAI,CAAC,MAAM,UAAU,OAAO,CAAC,MAAM,CAAC,MAAM,kBAAkB,CACxI,CAAC;AACJ,CAAC,CAAC,CAAC;AAEL,OAAO;KACJ,OAAO,CAAC,cAAc,CAAC;KACvB,WAAW,CAAC,4DAA4D,CAAC;KACzE,MAAM,CAAC,CAAC,IAAa,EAAE,EAAE;IACxB,MAAM,KAAK,GAAG,CAAC,CAAU,EAAE,EAAE,CAAC,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;IACtE,IAAI,IAAI;QAAE,KAAK,CAAC,eAAe,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;;QACjD,KAAK,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AACxD,CAAC,CAAC,CAAC;AAEL,iBAAiB,CAAC,OAAO,CAAC,CAAC;AAE3B,OAAO,CAAC,UAAU,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,KAAK,CAAC,CAAC,GAAY,EAAE,EAAE;IACtD,OAAO,CAAC,KAAK,CAAC,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;IAChE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}

package/dist/app/engine-cli.d.ts

@@ -0,0 +1,50 @@
+import { type Model } from "@wrenautomation/models";
+import { type Judge } from "@wrenautomation/models/judge";
+import type { Command } from "commander";
+import type { Extractor } from "../document/index.js";
+import { costOf, type Sources } from "../engine/index.js";
+import { FilesLedger, type Ledger, type Problem } from "../ledger/index.js";
+import { type AskKind, Criterion } from "../plan/index.js";
+/** Resumes `run` makes on its own after a transient failure. */
+export declare const RESUMES = 5;
+export { costOf };
+/** The pause before a resume: to `retryAfter` (five minutes at most), else 15 s per resume so far. */
+export declare function resumeWaitMs(n: number, retryAfter: string | undefined, now?: number): number;
+/** `NAME`, then `NAME_1`, `NAME_2`, ... in order: a fleet of keys for one vendor. */
+export declare function keys(printed: NodeJS.ProcessEnv, name: string): string[];
+/** `name=question` lines as criteria; a name the judge already asks, or a repeat, fails. */
+export declare function criteriaOf(lines: readonly string[] | undefined): Criterion[];
+export interface Common {
+    ledger: string;
+    writer: string;
+}
+/** A ledger a host hands the commands: the contract, and what the files backend adds when it has it. */
+export type HostLedger = Ledger & {
+    problems?: Problem[];
+};
+/** What a host swaps in: the files backend, this machine's sources and `--model` are the defaults. */
+export interface EngineCliDeps {
+    ledger?(o: Common): HostLedger;
+    sources?(o: Common, ledger: HostLedger): Sources;
+    /** The model for a `--model` spec; the default is models' `modelFromSpec`. */
+    model?(spec: string, printed: NodeJS.ProcessEnv): Model;
+    /** The judge for a `--judge` spec; the default is models' `judgeFromSpec`. */
+    judge?(spec: string, printed: NodeJS.ProcessEnv): Judge | null;
+    extractor?: Extractor;
+    /** The pause before a resume; a test passes a stub. */
+    sleep?(ms: number): Promise<void>;
+}
+/** The sources this machine can run, from the environment. */
+export declare function sourcesHere(printed: NodeJS.ProcessEnv, ledger: FilesLedger, ytDlp: string, warn?: (line: string) => void): Sources;
+/**
+ * The run's allow-list: the names wanted that exist here. `opens`: unless the list was given by
+ * hand, every other opener here, let in to open and follow, never to search (arxiv searched every
+ * web run when it joined the list). A name wanted that this machine lacks is dropped with a line on stderr.
+ */
+export declare function allowList(sources: Sources, wanted: readonly string[], byHand: boolean, warn?: (line: string) => void): {
+    sources: string[];
+    opens: string[];
+};
+/** A kind the engine plans for; anything else is the caller's mistake, said in words. */
+export declare const kindOf: (kind: string) => AskKind;
+export declare function addEngineCommands(program: Command, deps?: EngineCliDeps): void;