strom-research 1.18.0 → 1.19.0-beta.1

package/CHANGELOG.cs.md

@@ -5,6 +5,21 @@
 Změny každé verze ve zkratce. Úplné poznámky a instalační řádky jsou na
 [GitHub Releases](https://github.com/ACiDekCZ/strom-research/releases) (anglicky).
 
+## 1.19.0
+
+*Připravuje se: bety do 1.19.0-beta.1. Oddíl bude úplný s vydáním.*
+
+- **Na archiv, který zmlkl, se čeká, místo opakovaných dotazů.** Když archiv přestane odpovídat, strom ho nechá na
+  pokoji hodinu — a když týž den mlčí dál, dvě a pak čtyři hodiny — a agent se dozví, do kdy: mezitím dělá
+  jinou práci, nechá stahování na pozadí na archiv počkat (až 30 minut) a pokračovat samo, nebo sezení ukončí a
+  naváže později. Archiv se už mezitím neobtěžuje dotazy pořád dokola.
+- **První úklid staršího výzkumu se jednou nabídne.** Výzkum, u kterého se nastřádalo hodně dat dřív, než si v tom
+  strom začal sám dělat pořádek (záznamy práce, náhledy snímků a podobně), dostane v menu jednu nabídku: co by se
+  smazalo, kolik a proč — bez souhlasu nic. Po souhlasu se v tom pak udržuje pořádek samo; po odmítnutí menu a
+  nastavení dál říkají, co jde uvolnit.
+- **Digitální knihovny napoprvé správně.** U Krameria Národní knihovny ČR a Moravské zemské knihovny a u ANNO
+  Rakouské národní knihovny dostane agent jejich vlastní vyhledávací adresy — méně neúspěšných dotazů a hádání.
+
 ## 1.18.0
 
 *21. 10. 2026 · patří k aplikaci Strom 3.14.0.*

package/CHANGELOG.de.md

@@ -5,6 +5,23 @@
 Die Änderungen jeder Version in Kürze. Die vollständigen Hinweise und die Installationszeilen stehen bei
 [GitHub Releases](https://github.com/ACiDekCZ/strom-research/releases) (auf Englisch).
 
+## 1.19.0
+
+*In Vorbereitung: die Betas bis 1.19.0-beta.1. Der Abschnitt ist mit dem Release vollständig.*
+
+- **Auf ein verstummtes Archiv wird gewartet, statt es immer wieder zu fragen.** Antwortet ein Archiv nicht mehr,
+  lässt strom es eine Stunde in Ruhe — zwei, dann vier, wenn es am selben Tag still bleibt — und der Agent erfährt, bis
+  wann: Er macht inzwischen andere Arbeit, lässt einen Download im Hintergrund auf das Archiv warten (bis zu 30 Minuten)
+  und von selbst weiterlaufen, oder beendet die Sitzung, um später weiterzumachen. Das Archiv wird inzwischen nicht mehr
+  immer wieder angefragt.
+- **Das erste Aufräumen einer älteren Forschung wird einmal angeboten.** Eine Forschung, neben der sich viel
+  angesammelt hat, bevor strom dort selbst Ordnung hielt (Protokolle der Arbeitssitzungen, Ansichten von Scans und
+  Ähnliches), bekommt im Menü ein Angebot: was ginge, wie viel und warum — ohne Ja geht nichts. Nach einem Ja wird dort
+  von selbst Ordnung gehalten; nach einem Nein sagen Menü und Einstellungen weiter, was sich freigeben lässt.
+- **Digitale Bibliotheken beim ersten Versuch richtig durchsucht.** Für Kramerius der Tschechischen
+  Nationalbibliothek und der Mährischen Landesbibliothek und für ANNO der Österreichischen Nationalbibliothek bekommt
+  der Agent die eigenen Suchadressen der Bibliotheken — weniger fehlgeschlagene Anfragen und weniger Raten.
+
 ## 1.18.0
 
 *21.10.2026 · gehört zur Strom-App 3.14.0.*

package/CHANGELOG.md

@@ -5,6 +5,22 @@ English · [Čeština](CHANGELOG.cs.md) · [Deutsch](CHANGELOG.de.md)
 The changes of each version in short. The full notes and the install lines are on
 [GitHub Releases](https://github.com/ACiDekCZ/strom-research/releases).
 
+## 1.19.0
+
+*In preparation: the betas up to 1.19.0-beta.1. The section is complete with the release.*
+
+- **An archive that went silent is waited for, not asked again and again.** When an archive stops answering, strom
+  leaves it alone for an hour — two, then four when it stays silent the same day — and the agent hears until when:
+  it goes on with other work, lets a download in the background wait for the archive (up to 30 minutes) and go on by
+  itself, or closes the session to go on later. No more asking the archive over and over meanwhile.
+- **The first tidy of an older research is offered once.** A research that gathered much beside it before strom
+  kept order there by itself (logs of work sessions, views of scans and the like) gets one offer in the menu: what
+  would go, how much and why, and nothing goes without a yes. After a yes it is kept in order by itself; after a no,
+  the menu and the settings keep saying what can be freed.
+- **Digital libraries searched right at the first try.** For Kramerius of the Czech National Library and of the
+  Moravian Library and for ANNO of the Austrian National Library, the agent gets the libraries' own search
+  addresses — fewer failed requests and less guessing.
+
 ## 1.18.0
 
 *2026-10-21 · goes with the Strom app 3.14.0.*

package/assets/archives.json

@@ -180,7 +180,11 @@
       "quote": "actions: { pdf: 'available', print: 'available', jpeg: 'available', text: 'available' …",
       "robots": "none",
       "scope": "public-domain works only, through its API (works under copyright or DNNT need an institution's login: never)",
-      "checked": "2026-10-11"
+      "checked": "2026-10-11",
+      "search": [
+        {"what": "full text of its books and newspapers (Solr: q a word, or (word OR word); the years in it; public works only: accessibility)", "url": "https://api.kramerius.mzk.cz/search/api/client/v7.0/search?q=text_ocr%3A{q}%20AND%20date_range_start.year%3A%5B{from}%20TO%20{to}%5D&fl=pid,root.title,date.str,page.number,accessibility&rows=50"},
+        {"what": "the text of a page found (its pid)", "url": "https://api.kramerius.mzk.cz/search/api/client/v7.0/items/{pid}/ocr/text"}
+      ]
     },
     {
       "host": "www.digitalniknihovna.cz",
@@ -190,7 +194,11 @@
       "quote": "actions: { pdf: 'available', print: 'available', jpeg: 'available', text: 'available' …",
       "robots": "none (the application's page)",
       "scope": "public-domain works only, through its API (works under copyright or DNNT need an institution's login: never)",
-      "checked": "2026-10-11"
+      "checked": "2026-10-11",
+      "search": [
+        {"what": "full text of its books and newspapers (Solr: q a word, or (word OR word); the years in it; public works only: accessibility)", "url": "https://api.kramerius.mzk.cz/search/api/client/v7.0/search?q=text_ocr%3A{q}%20AND%20date_range_start.year%3A%5B{from}%20TO%20{to}%5D&fl=pid,root.title,date.str,page.number,accessibility&rows=50"},
+        {"what": "the text of a page found (its pid)", "url": "https://api.kramerius.mzk.cz/search/api/client/v7.0/items/{pid}/ocr/text"}
+      ]
     },
     {
       "host": "api.kramerius.mzk.cz",
@@ -200,7 +208,11 @@
       "quote": "actions: { pdf: 'available', print: 'available', jpeg: 'available', text: 'available' …",
       "robots": "none (404)",
       "scope": "public-domain works only (works under copyright or DNNT need an institution's login: never)",
-      "checked": "2026-10-11"
+      "checked": "2026-10-11",
+      "search": [
+        {"what": "full text of its books and newspapers (Solr: q a word, or (word OR word); the years in it; public works only: accessibility)", "url": "https://api.kramerius.mzk.cz/search/api/client/v7.0/search?q=text_ocr%3A{q}%20AND%20date_range_start.year%3A%5B{from}%20TO%20{to}%5D&fl=pid,root.title,date.str,page.number,accessibility&rows=50"},
+        {"what": "the text of a page found (its pid)", "url": "https://api.kramerius.mzk.cz/search/api/client/v7.0/items/{pid}/ocr/text"}
+      ]
     },
     {
       "host": "kramerius5.nkp.cz",
@@ -210,7 +222,11 @@
       "quote": "actions: { pdf: 'available', print: 'available', jpeg: 'available', text: 'available' …",
       "robots": "none (the application's page)",
       "scope": "public-domain works only, through its API (works under copyright or DNNT need an institution's login: never)",
-      "checked": "2026-10-11"
+      "checked": "2026-10-11",
+      "search": [
+        {"what": "full text of its books and newspapers (Solr: q a word, or (word OR word); fq the years)", "url": "https://kramerius5.nkp.cz/search/api/v5.0/search?q=text_ocr%3A{q}&fq=datum_begin%3A%5B{from}%20TO%20{to}%5D&fl=PID,root_title,datum_str,details,root_pid&rows=30&wt=json"},
+        {"what": "the text of a page found (its PID)", "url": "https://kramerius5.nkp.cz/search/api/v5.0/item/{pid}/streams/TEXT_OCR"}
+      ]
     },
     {
       "host": "ndk.cz",
@@ -220,7 +236,21 @@
       "quote": "actions: { pdf: 'available', print: 'available', jpeg: 'available', text: 'available' …",
       "robots": "none (the application's page)",
       "scope": "public-domain works only, through its API (works under copyright or DNNT need an institution's login: never)",
-      "checked": "2026-10-11"
+      "checked": "2026-10-11",
+      "search": [
+        {"what": "full text of its books and newspapers (Solr: q a word, or (word OR word); fq the years)", "url": "https://kramerius5.nkp.cz/search/api/v5.0/search?q=text_ocr%3A{q}&fq=datum_begin%3A%5B{from}%20TO%20{to}%5D&fl=PID,root_title,datum_str,details,root_pid&rows=30&wt=json"},
+        {"what": "the text of a page found (its PID)", "url": "https://kramerius5.nkp.cz/search/api/v5.0/item/{pid}/streams/TEXT_OCR"}
+      ]
+    },
+    {
+      "host": "anno.onb.ac.at",
+      "name": "Österreichische Nationalbibliothek (ANNO)",
+      "level": "unknown",
+      "robots": "User-agent: * · Disallow: /core/, /profiles/, /admin/, /search/, /user/… (nothing on /anno-suche or /cgi-content)",
+      "checked": "2026-10-21",
+      "search": [
+        {"what": "full text of its newspapers and journals (text a word; the years; from=0 the first hits) — the search page itself is empty without a browser", "url": "https://anno.onb.ac.at/anno-suche/rest/search/complex?text={q}&dateFrom={from}&dateTo={to}&from=0&facets=false"}
+      ]
     },
     {
       "host": "geneteka.genealodzy.pl",

package/assets/lang/cs.json

@@ -1066,6 +1066,7 @@
   "ui.tidy.done": "✓ Uvolněno {size}. Odteď to strom po každém sezení udržuje v pořádku sám.",
   "ui.tidy.no": "Nic se nesmazalo.",
   "ui.consent.tidy": "Uvolnit asi {size} vedle výzkumu {name} (staré záznamy sezení, náhledy skenů, dávno zapsaná poslání)? Výzkum i jeho skeny zůstanou.",
+  "ui.tidy.first": "Vedle výzkumu se nastřádalo, co jde uvolnit, asi {size} (záznamy práce, náhledy snímků a podobně – nikdy výzkum). Ukazuje se tu jednou; později v menu Nastavení → Údržba → Místo na disku.",
   "ui.tidy.hint": "Vedle výzkumu si strom drží {strom}, z toho jde uvolnit {size}: strom tidy (v menu Nastavení → Údržba → Místo na disku)",
   "ui.error.consent": "tady je potřeba souhlas – {question}",
   "ui.error.consent.hint": "spustit ve vlastním terminálu: {set}",

package/assets/lang/de.json

@@ -1066,6 +1066,7 @@
   "ui.tidy.done": "✓ {size} freigegeben. Ab jetzt hält strom es nach jeder Sitzung selbst in Ordnung.",
   "ui.tidy.no": "Nichts wurde gelöscht.",
   "ui.consent.tidy": "Etwa {size} neben der Forschung {name} freigeben (alte Sitzungsprotokolle, Ansichten von Scans, längst geschriebene Sendungen)? Die Forschung und ihre Scans bleiben.",
+  "ui.tidy.first": "Neben der Forschung hat sich angesammelt, was freigegeben werden kann, etwa {size} (Protokolle, Ansichten von Scans und Ähnliches – nie die Forschung). Wird hier einmal gezeigt; später im Menü Einstellungen → Wartung → Speicherplatz.",
   "ui.tidy.hint": "Neben der Forschung hält strom {strom}, davon lassen sich {size} freigeben: strom tidy (im Menü Einstellungen → Wartung → Speicherplatz)",
   "ui.error.consent": "dafür braucht es eine Zustimmung – {question}",
   "ui.error.consent.hint": "im eigenen Terminal ausführen: {set}",

package/assets/method/core.md

@@ -107,6 +107,9 @@ conclusion so the next researcher can follow it.
   Images you will read later in the session: `--background` fetches them
   while you work on, `strom read` reads them as they come. Nothing for a
   later session: the session that reads a book fetches it.
+  An archive unavailable for now (exit 7, "try again at …"): never wait for
+  it in a loop of fetches — other work, `--background` when it frees within
+  30 minutes, or `strom session close --continue`.
 <!-- end -->
 - **What strom or the tree denies stays denied.** Never offer the user to
   change your permissions or settings, strom's or the tree's, to get round it:

package/assets/method/enrich.md

@@ -25,6 +25,8 @@ recorded, also in vain (`strom search add … --method web|catalog|full-text`).
   built right away when the task needs that site (`strom connector new
   <site>`, then `strom fetch`) — never for a site that refused strom or
   restricts automation (`strom net web --check <url>`): another source.
+  A digital library strom knows gives its own search there: fetch its
+  addresses as given, the words URL-encoded — never a guessed API.
 - **The history of a place once in a research**: what is written of it (the
   brief says) is reused, never searched again.
 - **Read the source itself** — the page, the scan. A snippet, an OCR line,

package/dist/cli/menu.js

@@ -9,7 +9,8 @@
 import fs from "node:fs";
 import path from "node:path";
 import { ui } from "./ui.js";
-import { mb, tidyPlan, TIDY_SAID } from "../core/tidy.js";
+import { isAgent } from "../core/which.js";
+import { mb, setTidyAsked, tidyAsked, tidyOn, tidyPlan, TIDY_SAID } from "../core/tidy.js";
 import { calibrationLabel, calibrationOffer } from "../core/viewsizes.js";
 /** A line as a sentence: its first letter capital (a catalog's text that also stands inside a line). */
 const sentence = (s) => s.charAt(0).toUpperCase() + s.slice(1);
@@ -192,8 +193,20 @@ export async function runMenu(ctx, run) {
                 out(line);
             // much kept beside the research that can go (an older strom never tidied it): said, freed in the settings
             const disk = tidyPlan(tree);
-            if (disk.frees >= TIDY_SAID)
-                out(t("ui.tidy.hint", { strom: mb(disk.size.strom, lang), size: mb(disk.frees, lang) }));
+            if (disk.frees >= TIDY_SAID) {
+                // never tidied (an older strom gathered it): its first tidy offered once, shown and asked as strom tidy asks
+                // (Enter says no) — nothing goes unasked; then kept in order by itself. Nobody to ask (no terminal,
+                // STROM_NONINTERACTIVE): only said, the offer kept for the person
+                if (!tidyOn(root) && !tidyAsked(root) && !isAgent(ctx.env) && ctx.interactive) {
+                    setTidyAsked(root);
+                    out(t("ui.tidy.first", { size: mb(disk.frees, lang) }));
+                    out();
+                    await run(["tidy"]);
+                    out();
+                }
+                else
+                    out(t("ui.tidy.hint", { strom: mb(disk.size.strom, lang), size: mb(disk.frees, lang) }));
+            }
             // the scan views tuned before, but for another agent or model: offered again (the settings), never run unasked
             const offer = !archive && ready ? calibrationOffer(ctx.settings, ctx.settings.agent(tree.config).value, tree.config, tree.root) : undefined;
             if (offer)

package/dist/cli/ui.js

@@ -957,6 +957,7 @@ export const UI = {
     "ui.tidy.done": "✓ Freed {size}. From now on strom keeps it in order by itself after each session.",
     "ui.tidy.no": "Nothing was removed.",
     "ui.consent.tidy": "Free about {size} beside the research {name} (old logs, views of scans, sends written long ago)? The research and its images stay.",
+    "ui.tidy.first": "Beside the research strom has gathered what can go, about {size} (logs, views of scans and the like — never the research). Shown once here; later in the menu: Settings → Maintenance → Disk space.",
     "ui.tidy.hint": "Beside the research strom keeps {strom}, {size} of it can go: strom tidy (the menu: Settings → Maintenance → Disk space)",
     "ui.error.consent": "this needs a yes — {question}",
     "ui.error.consent.hint": "run in a terminal of its own: {set}",

package/dist/commands/connectors.js

@@ -30,7 +30,7 @@ import { runGit } from "../core/git.js";
 import { imageSizeOfFile } from "../image/index.js";
 import { Tree } from "../core/tree.js";
 import { syncAgentFiles } from "../agents/files.js";
-import { CAP_WARN, capNear, clearBlock, clock, CookieJar, DEFAULT_PACE, hostAllowed, hostPace, hostState, knownHosts, leftUntil, MIN_INTERVAL_MS, NetError, paceText, politeRequest, REFUSED_FOREVER, refusedBy, reserveSlots, setOwnPace, testHooks } from "../core/net.js";
+import { CAP_WARN, capNear, clearBlock, clock, clockUp, CookieJar, DEFAULT_PACE, hostAllowed, hostPace, hostState, knownHosts, leftAloneFor, leftUntil, MIN_INTERVAL_MS, NetError, paceText, politeRequest, REFUSED_FOREVER, refusedBy, reserveSlots, setOwnPace, testHooks } from "../core/net.js";
 import { match, optionsOf } from "../cli/registry.js";
 import { botCheck, fileBase, findDownload, loadPage, loadPlans, pageKey, pageScript, parseResult, PLAN_TTL_MS, planScript, readSavedPage, removePlan, savePage, savePlan, SCRIPT_MS, claudeInChrome, CLAUDE_IN_CHROME_URL, } from "../core/browser.js";
 import { PROFILES } from "../agents/profiles.js";
@@ -399,7 +399,7 @@ function limitWait(ctx) {
     const waitMs = Math.max(0, Math.min(FETCH_WAIT_MS, left));
     return {
         waitMs,
-        onWait: (w) => ctx.io.stderr(`  · ${w.host}: ${w.why === "cap" ? "its hourly cap is used up" : "it says its limit is used up"} — strom waits until ${clock(w.until)} (${Math.max(1, Math.ceil(w.ms / 60_000))} min), then goes on by itself (it waits at most ${Math.round(waitMs / 60_000)} min); do not stop it\n`),
+        onWait: (w) => ctx.io.stderr(`  · ${w.host}: ${w.why === "cap" ? "its hourly cap is used up" : w.why === "down" ? "left alone for a while (it went silent, or asked to slow down)" : "it says its limit is used up"} — strom waits until ${clockUp(w.until)} (${Math.max(1, Math.ceil(w.ms / 60_000))} min), then goes on by itself (it waits at most ${Math.round(waitMs / 60_000)} min); do not stop it\n`),
     };
 }
 /** The limiter's pauses and requests counted for the measure of a run (core/metrics.ts). */
@@ -439,11 +439,22 @@ function measure(ctx, c, meter, rec, r) {
         // a measurement is no reason to fail
     }
 }
-/** A run that ended at a limit used up: when to try again, and the exit code that says "later". */
-function later(r, again) {
+/**
+ * A run that ended at a limit used up, or at a host left alone for a while (it went silent, asked to slow down): when
+ * to try again (rounded up — never a minute too early), what to do meanwhile, and the exit code that says "later".
+ * `background`: the command of a fetch in the background (it waits for a host that frees within FETCH_WAIT_MS).
+ */
+function later(r, again, background) {
     if (r.later === undefined)
         return {};
-    return { line: `→ the archive's limit is used up: try again at ${clock(r.later)} (${again}); meanwhile other work — no sleeping and asking sooner`, exitCode: EXIT.later };
+    const at = clockUp(r.later);
+    const what = r.laterDown ? `${r.laterDown} is left alone until ${at} (it went silent, or asked to slow down — strom sends it nothing before then)` : "the archive's limit is used up";
+    const soon = background && r.later - Date.now() <= FETCH_WAIT_MS ? `; or let it wait by itself: ${background} --background` : "";
+    return {
+        line: `→ ${what}: try again at ${at}, not sooner (${again})${soon}; meanwhile other work, or close the session for later (strom session close --continue) — no sleeping, no asking again sooner`,
+        exitCode: EXIT.later,
+        data: { retryAt: new Date(Math.ceil(r.later / 60_000) * 60_000).toISOString(), ...(r.laterDown ? { leftAlone: r.laterDown } : {}) },
+    };
 }
 /** Does this strom's task wait take --check <URL> (a check of a site by the person)? Said in the advice only then. */
 function waitChecks() {
@@ -1227,7 +1238,10 @@ register({
     description: "Every request goes through strom's limiter: one at a time, paced, capped per hour; a refusal (401/403)\n" +
         "stops the run and leaves the archive alone for a day. An archive's hourly cap used up: strom waits for it by\n" +
         "itself (up to 30 min, never into a session's last 10) and says until when — no need to sleep and ask again;\n" +
-        "longer, it ends with exit 7 and the time to try again. With --recordset the images are registered as they\n" +
+        "longer, it ends with exit 7 and the time to try again. An archive that went silent (or asked twice to slow\n" +
+        "down) is left alone for an hour (longer when it goes silent again): a fetch then ends at once with exit 7\n" +
+        "and the time — other work meanwhile, or close the session with --continue; a fetch with --background waits\n" +
+        "for it by itself when it frees within 30 min. With --recordset the images are registered as they\n" +
         "come, 10 at a time (strom read of the book may begin meanwhile); Ctrl-C keeps what came, the same command\n" +
         "fetches the rest;\n" +
         "without it they go into the inbox, a folder for the book.\n" +
@@ -2200,6 +2214,17 @@ const BACKGROUND_START_MS = 15_000;
  */
 async function startBackground(ctx, c, request, recordset, owner, noIndex, again) {
     const tree = ctx.tree();
+    // a host of the archive left alone for a while (it went silent, asked to slow down): waited for when it frees in
+    // time, else said when to try again — no strom started only to end at once
+    const alone = leftAloneHosts(ctx, c);
+    const by = Date.now() + limitWait(ctx).waitMs;
+    const tooLong = alone.filter((a) => a.until > by);
+    if (tooLong.length) {
+        const last = tooLong.reduce((a, b) => (b.until > a.until ? b : a));
+        const wait = later({ later: last.until, laterDown: last.host }, `strom fetch ${c.name} ${shellArg(String(request.book))} --images ${imagesArg(request.images)} --recordset ${recordset} --background`);
+        return { text: lines(`nothing started in the background: ${last.host} is left alone longer than a fetch in the background waits (${Math.round(limitWait(ctx).waitMs / 60_000)} min)`, wait.line), data: { started: false, ...wait.data }, exitCode: EXIT.later };
+    }
+    const waits = alone.length ? alone.reduce((a, b) => (b.until > a.until ? b : a)) : undefined;
     const log = backgroundLog(tree.root, recordset);
     fs.mkdirSync(path.dirname(log), { recursive: true });
     const out = fs.openSync(log, "a");
@@ -2232,9 +2257,9 @@ async function startBackground(ctx, c, request, recordset, owner, noIndex, again
     child.on("error", () => (exit = 1));
     child.unref();
     const pid = child.pid ?? 0;
-    const by = Date.now() + BACKGROUND_START_MS;
+    const startBy = Date.now() + BACKGROUND_START_MS;
     let began = false;
-    while (Date.now() < by && exit === undefined) {
+    while (Date.now() < startBy && exit === undefined) {
         if (activeFetches(tree.root, recordset).some((m) => m.pid === pid)) {
             began = true;
             break;
@@ -2258,10 +2283,21 @@ async function startBackground(ctx, c, request, recordset, owner, noIndex, again
     }
     const n = request.images.length;
     return {
-        text: lines(`fetching ${counted(n, "image")} of ${recordset} (images ${runs(request.images)}) in the background — pid ${pid}, for ${owner.task ?? `session ${owner.session}`}`, `  read them as they come: strom read ${recordset} --images ${imagesArg(request.images)} · what came: strom fetch --status · stop it: strom fetch --stop ${recordset}`, `  it stops when the session is asked to finish or closes${isRunWorker(owner.worker) ? " (in this run: when the run ends)" : ""}; what came stays registered`, owner.guessed ? `  for ${owner.task}: the one task this session worked on that names ${recordset}${currentSession(tree, ctx.env)?.task ? ` (the session's own: ${currentSession(tree, ctx.env).task})` : ""} — for another one: --task T…` : undefined),
-        data: { pid, recordset, images: request.images, session: owner.session, ...(owner.task ? { task: owner.task } : {}), log },
+        text: lines(`fetching ${counted(n, "image")} of ${recordset} (images ${runs(request.images)}) in the background — pid ${pid}, for ${owner.task ?? `session ${owner.session}`}`, waits ? `  ${waits.host} is left alone until ${clockUp(waits.until)} (it went silent, or asked to slow down): the fetch waits for it, then goes on by itself — other work meanwhile, no asking again` : undefined, `  read them as they come: strom read ${recordset} --images ${imagesArg(request.images)} · what came: strom fetch --status · stop it: strom fetch --stop ${recordset}`, `  it stops when the session is asked to finish or closes${isRunWorker(owner.worker) ? " (in this run: when the run ends)" : ""}; what came stays registered`, owner.guessed ? `  for ${owner.task}: the one task this session worked on that names ${recordset}${currentSession(tree, ctx.env)?.task ? ` (the session's own: ${currentSession(tree, ctx.env).task})` : ""} — for another one: --task T…` : undefined),
+        data: { pid, recordset, images: request.images, session: owner.session, ...(owner.task ? { task: owner.task } : {}), log, ...(waits ? { waitsUntil: new Date(waits.until).toISOString() } : {}) },
     };
 }
+/** The hosts of a connector left alone now for a while without a refusal (silence, "slow down"), and until when. */
+function leftAloneHosts(ctx, c) {
+    const dir = netDir(ctx);
+    const now = Date.now();
+    return knownHosts(dir)
+        .filter((h) => hostAllowed(h, c.manifest.hosts))
+        .flatMap((host) => {
+        const until = leftAloneFor(hostState(dir, host), now);
+        return until !== undefined ? [{ host, until }] : [];
+    });
+}
 /** A fetch in the foreground whose images another strom is fetching already: waits for them (Ctrl-C stops waiting). */
 async function joinFetch(ctx, recordset, joined, said) {
     const tree = ctx.tree();
@@ -2289,7 +2325,7 @@ async function joinFetch(ctx, recordset, joined, said) {
 /** One line of a fetch at work, for strom fetch --status. */
 function activeLine(m) {
     const since = m.since ? ` · since ${clock(Date.parse(m.since))}` : "";
-    const state = m.waitUntil && m.waitUntil > Date.now() ? ` · waits for the archive's limit until ${clock(m.waitUntil)}` : m.yielding ? " · waits while a fetch in the foreground asks the archive" : "";
+    const state = m.waitUntil && m.waitUntil > Date.now() ? ` · waits for the archive until ${clockUp(m.waitUntil)}, then goes on by itself` : m.yielding ? " · waits while a fetch in the foreground asks the archive" : "";
     return `${m.recordset} (${m.connector} ${m.book}): ${m.registered}/${m.images.length} images registered · ${m.background ? `in the background${m.task ? ` for ${m.task}` : ""}${m.session ? ` (session ${m.session})` : ""}` : "in the foreground"}${since}${state} · pid ${m.pid}`;
 }
 function endedLine(e) {
@@ -2495,7 +2531,7 @@ async function fetchWith(ctx, c, request, recordset, noIndex = false, again = fa
             const wait = later(r, "the same command");
             return {
                 text: lines(describeRun(r, `${request.cmd} (${took})`), r.books.length ? [`books (${r.books.length}):`, ...bookLines(r, c, `strom fetch ${c.name} <id> --list`, tree)].join("\n") : advice ?? (r.stopped ? "no books — the run stopped before any came" : "no books found"), r.books.length ? advice : undefined, wait.line, near),
-                data: r,
+                data: { ...r, ...wait.data },
                 ...(wait.exitCode ? { exitCode: wait.exitCode } : r.stopped && !r.books.length ? { exitCode: 1 } : {}),
             };
         }
@@ -2549,12 +2585,13 @@ async function fetchWith(ctx, c, request, recordset, noIndex = false, again = fa
             text = "no images fetched";
         }
         // the images it got are registered; the rest when the archive takes requests again
-        const wait = later(r, request.cmd === "fetch" ? `strom fetch ${c.name} ${shellArg(String(request.book))} --images ${imagesArg(request.images)}${recordset ? ` --recordset ${recordset}` : ""} — what is here already is not fetched again` : "the same command");
+        const again = request.cmd === "fetch" ? `strom fetch ${c.name} ${shellArg(String(request.book))} --images ${imagesArg(request.images)}${recordset ? ` --recordset ${recordset}` : ""}` : undefined;
+        const wait = later(r, again ? `${again} — what is here already is not fetched again` : "the same command", again && recordset && !child && currentSession(tree, ctx.env) ? again : undefined);
         // images another strom was fetching: waited for once this one's are here (the foreground)
         const after = joined && !batches?.interrupted() && recordset ? await joinFetch(ctx, recordset, joined, `images ${runs(joined.images)} of ${recordset}: the other fetch`) : undefined;
         const code = wait.exitCode ? wait.exitCode : batches?.stopped() ? 0 : batches?.interrupted() ? EXIT.cancelled : r.stopped && !r.images.length ? 1 : after?.exitCode;
         endCode = code ?? 0;
-        return { text: lines(describeRun(r, request.cmd), text, advice, wait.line, near, after?.text), data: { ...data, ...(after ? { joined: after.data } : {}) }, ...(code ? { exitCode: code } : {}) };
+        return { text: lines(describeRun(r, request.cmd), text, advice, wait.line, near, after?.text), data: { ...data, ...wait.data, ...(after ? { joined: after.data } : {}) }, ...(code ? { exitCode: code } : {}) };
     }
     finally {
         // a fetch in the background says how it ended (strom fetch --status, the next brief) before its mark goes

package/dist/commands/grep.js

@@ -62,9 +62,12 @@ register({
         const regex = !!opts.regex;
         if (!patterns.length || (!regex && !patterns.some((p) => foldText(p))))
             throw new UsageError("what to find?", { hint: 'strom grep "<text>"' });
-        const context = opts.context === undefined ? 0 : Number(opts.context);
-        if (!Number.isInteger(context) || context < 0 || context > MAX_CONTEXT)
+        const asked = opts.context === undefined ? 0 : Number(opts.context);
+        if (!Number.isInteger(asked) || asked < 0)
             throw new UsageError(`--context: a number of lines, 0–${MAX_CONTEXT}`);
+        // more than the most: the most, said (no turn lost on it)
+        const context = Math.min(asked, MAX_CONTEXT);
+        const cut = asked > MAX_CONTEXT ? `--context: at most ${MAX_CONTEXT} lines — ${MAX_CONTEXT} shown` : undefined;
         // an input named by its ID: its file in inputs/ (one kept in the shared store is an image or a document)
         const wanted = listOpt(opts.in).map((w) => {
             if (!/^I\p{N}+$/iu.test(w))
@@ -123,7 +126,7 @@ register({
         const tooBig = found.tooBig.length ? `left out, over 64 MB: ${found.tooBig.join(", ")}` : undefined;
         if (!found.hits.length)
             return {
-                text: lines(`not found in ${counted(found.searched, "text file")} of ${where}`, tooBig, "→ the records: strom find <text>"),
+                text: lines(`not found in ${counted(found.searched, "text file")} of ${where}`, tooBig, cut, "→ the records: strom find <text>"),
                 data: { total: 0, searched: found.searched, hits: [], ...(found.tooBig.length ? { tooBig: found.tooBig } : {}) },
             };
         const total = `${found.hits.length}${found.capped ? "+" : ""}`;
@@ -140,7 +143,7 @@ register({
         }
         const page = paginate(found.hits, ctx.limit, ctx.page);
         return {
-            text: lines(head + ":", ...grepLines(page.items), moreLine(page, again) || undefined, tooBig),
+            text: lines(head + ":", ...grepLines(page.items), moreLine(page, again) || undefined, tooBig, cut),
             data: { total: found.hits.length, capped: found.capped, searched: found.searched, files: found.counts.size, hits: page.items, ...(found.tooBig.length ? { tooBig: found.tooBig } : {}) },
         };
     },

package/dist/commands/readings.js

@@ -131,11 +131,13 @@ export function summary(blocks, o = {}) {
         else
             out.push(...ls);
     }
+    // words looked for: the blocks that mention them alone — the book's gaps and parts belong to its overview
+    const overview = !(o.match ?? []).some((m) => m.trim());
     const gaps = blocks.flatMap((b) => (b.gaps ?? []).map((g) => `${label(b)}: ${g}`));
     const parts = blocks.flatMap((b) => (b.section ?? []).map((s) => `${label(b)}: ${s}`));
-    if (gaps.length)
+    if (gaps.length && overview)
         out.push(`gaps in the book (by image): ${gaps.join(" · ")}`);
-    if (parts.length)
+    if (parts.length && overview)
         out.push(`parts of the book (by image): ${parts.join(" · ")}`);
     return { lines: out, more, gaps: blocks.flatMap((b) => b.gaps ?? []), shown };
 }
@@ -389,6 +391,8 @@ export function earlierLines(e) {
         ...(e.readings.length || e.searches.length ? ["(reading them again is right for another question or a blind check — not for the same one)"] : []),
     ];
 }
+/** The readings strom readings --match shows at most unless --limit says otherwise (the newest first). */
+const MATCH_READINGS = 20;
 register({
     path: ["readings"],
     summary: "What earlier readers reported (strom read), compact: finds, unclear entries, what was illegible, gaps in the book",
@@ -397,13 +401,15 @@ register({
     description: "The readers' reports in notes/readings, without reading them whole: each found and unclear block, from the\n" +
         "blocks with nothing found only what the reader could not read (a possible match is often there), the gaps in\n" +
         "the page numbering and the parts of the book they saw. One image's block word for word: --image N.\n" +
-        "Default: the newest reading; with --images, --image or --match all of them. Nothing is written.",
+        "Default: the newest reading; with --images, --image or --match all of them — with --match only the readings\n" +
+        "that mention it (the newest 20; --limit for more), the others counted. Nothing is written.",
     args: [{ name: "which", description: "a record set (B…), a reading: its report's name or the start of it (a day), or a reading of a document (D…)" }],
     options: [
         { name: "images", type: "string", value: "<from-to>", description: "only these image numbers, e.g. 40-69 or 5-113,130-228" },
         { name: "image", type: "string", value: "<n>", description: "that image's whole block, word for word (a few: 35-38)" },
         { name: "match", type: "string", multiple: true, value: "<words>", description: "only blocks that mention it — any spelling of accents (repeatable: variants)" },
         { name: "list", type: "boolean", description: "the readings there are, one line each" },
+        { name: "limit", type: "string", value: "<n>", description: `at most this many readings (with --match: ${MATCH_READINGS} unless said)` },
     ],
     examples: ["strom readings", "strom readings B0001 --images 40-69 --match Novák --match Nowak", "strom readings B0001 --image 57", "strom readings D20260101120000-ab12", "strom readings --list"],
     run(ctx, { args, opts }) {
@@ -443,10 +449,21 @@ register({
         // one image, or a few (35-38, 35..38): each one's block word for word
         const image = opts.image === undefined ? undefined : new Set(imageRange(String(opts.image)));
         const match = (Array.isArray(opts.match) ? opts.match : typeof opts.match === "string" ? [opts.match] : []).map(String).filter((m) => m.trim());
+        const limit = opts.limit === undefined ? undefined : Number(opts.limit);
+        if (limit !== undefined && !(Number.isInteger(limit) && limit > 0))
+            throw new UsageError(`invalid --limit "${String(opts.limit)}": a whole number, 1 or more`, { hint: "e.g. --limit 40" });
         const pick = (r) => r.blocks.filter((b) => (!book || b.recordset === book) && (!ranges || within(b.num, ranges)) && (image === undefined || (b.num !== undefined && image.has(b.num))));
-        const readings = all.map((r) => ({ r, blocks: pick(r) })).filter((x) => x.blocks.length);
+        const picked = all.map((r) => ({ r, blocks: pick(r) })).filter((x) => x.blocks.length);
+        // words looked for: only the readings that mention them (the rest counted), the newest first, at most the limit
+        const readings = match.length ? picked.filter(({ r, blocks }) => select(blocks, { match, question: r.question }).length) : picked;
+        const silent = picked.length - readings.length;
         // nothing asked: the newest reading only
-        const shown = which || ranges || image !== undefined || match.length || opts.list ? readings : readings.slice(0, 1);
+        const every = which || ranges || image !== undefined || match.length || opts.list ? readings : readings.slice(0, 1);
+        const cap = limit ?? (match.length ? MATCH_READINGS : Infinity);
+        const shown = every.slice(0, cap);
+        const beyond = every.length - shown.length;
+        if (shown.length === 0 && match.length && picked.length)
+            return { text: `no reading mentions ${match.map((m) => `"${m}"`).join(" or ")} (${counted(picked.length, "reading")} looked through) · strom grep for the notes too`, data: { readings: [], without: picked.length } };
         if (shown.length === 0)
             return { text: `no reader reported ${image !== undefined ? `image ${runs([...image])}` : "these images"}${book ? ` of ${book}` : ""} · strom readings --list`, data: { readings: [] } };
         if (opts.list) {
@@ -477,7 +494,11 @@ register({
             };
         });
         const at = book ?? (which ? shellArg(which) : shown[0].blocks[0]?.recordset ?? shellArg(shown[0].r.stem));
+        if (beyond > 0)
+            parts.push(`… ${counted(beyond, "more reading")}${match.length ? " mention it" : ""}: --limit ${every.length}`);
+        if (silent > 0)
+            parts.push(`${counted(silent, "other reading")} ${silent === 1 ? "does" : "do"} not mention it`);
         parts.push(`one image's whole block: strom readings ${at} --image <n>`);
-        return { text: lines(...parts), data: { readings: data } };
+        return { text: lines(...parts), data: { readings: data, ...(beyond > 0 ? { more: beyond } : {}), ...(match.length ? { without: silent } : {}) } };
     },
 });

package/dist/commands/sources.js

@@ -147,7 +147,9 @@ register({
     tree: true,
     writes: true,
     description: "An original register entry or certificate is primary evidence; family trees and memories are\n" +
-        "authored/derivative — facts citing only them stay leads. --transcript @file reads a file.",
+        "authored/derivative — facts citing only them stay leads. --transcript @file reads a file.\n" +
+        `One source is one entry (its clips: the entry, and its continuation over a page break — at most ${MAX_CLIPS});\n` +
+        "several entries of one page are several sources.",
     args: [{ name: "title", description: 'short title: "Baptism of Jan Novák, Týnec 1905"', required: true }],
     options: [
         { name: "kind", type: "string", value: "<kind>", description: `${SOURCE_KINDS.slice(0, 8).join(", ")}, … (default other)` },
@@ -185,7 +187,7 @@ register({
             requireRecord(tree, opts.input, "input");
         const clips = clipRefs(tree, listOpt(opts.clip), ctx.settings.shared()?.value);
         if (clips.length > MAX_CLIPS)
-            throw new UsageError(`at most ${MAX_CLIPS} clips: an entry, and its continuation over a page break`);
+            throw new UsageError(`at most ${MAX_CLIPS} clips: an entry, and its continuation over a page break — several entries are several sources (one strom source add each)`);
         const media = [...new Set([...mediaRefs(tree, csvOpt(opts.media)), ...clips.map((c) => c.media)])];
         // a line of an index is a copy made by another hand; a book with its own index holds originals too
         const kind = oneOf(opts.kind, SOURCE_KINDS, "kind", "other");

package/dist/commands/web.js

@@ -7,6 +7,7 @@ import { webDomain, webRequestsOf } from "../core/metrics.js";
 import { currentSession } from "../core/session.js";
 import { requireRecord } from "../core/records.js";
 import { webGate } from "../core/webgate.js";
+import { knownSearch } from "../core/knownterms.js";
 import { termsFor } from "../core/terms.js";
 import { clock } from "../core/net.js";
 import { UsageError } from "../core/errors.js";
@@ -28,7 +29,8 @@ register({
         "(their yes or no to that question stands for it) — counted once it went out (the same hook after the fetch ran).\n" +
         "Without --hook: this session's requests by server (no session open: the last one with web requests, said which).\n" +
         "--check <url>: whether strom would let your web tools fetch that address now, and why — the archive's level, the\n" +
-        "person's permission, the robots.txt line, a refusal of strom's client, the host left alone — read only, nothing sent.\n" +
+        "person's permission, the robots.txt line, a refusal of strom's client, the host left alone — read only, nothing sent;\n" +
+        "a digital library strom knows: its own search addresses too, to fetch as given (the words URL-encoded).\n" +
         "An agent without strom's hook (Codex, OpenCode, Antigravity) runs it before its own web fetch of an archive's site\n" +
         "and keeps to its answer.",
     options: [
@@ -122,6 +124,7 @@ function checkUrl(ctx, url) {
     const session = tree ? currentSession(tree, ctx.env, ctx.refs) : undefined;
     const g = webGate({ env: ctx.env, ...(tree ? { tree } : {}), url: u.href, shared, ...(session?.task ? { task: session.task } : {}) });
     const ok = g.allow && !g.blocked;
+    const search = knownSearch(u.hostname);
     const a = g.level;
     const answer = !g.allow ? "refused" : g.blocked ? "refused now" : g.why === "terms" ? "only this page (its terms)" : "allowed";
     const why = !g.allow
@@ -132,7 +135,9 @@ function checkUrl(ctx, url) {
                 ? `its archive is closed to automated tools; its terms page only, at most a few times a day`
                 : "pages of it through your web tools within the web rule (strom guide); images of an archive through strom fetch";
     return {
-        text: lines(`${u.href}: ${answer}`, a ? `  archive ${a.host}: ${a.level}${a.from.length ? ` (${a.from.join(", ")})` : ""}${a.permission ? ` · the person's permission: ${a.permission}` : ""}${a.refused ? " · refused strom's client" : ""}${a.browserOnly ? " · only through a person's browser" : ""}` : undefined, g.robots ? `  robots.txt: ${g.robots.line} (${g.robots.path})` : undefined, `  ${why}`),
+        text: lines(`${u.href}: ${answer}`, a ? `  archive ${a.host}: ${a.level}${a.from.length ? ` (${a.from.join(", ")})` : ""}${a.permission ? ` · the person's permission: ${a.permission}` : ""}${a.refused ? " · refused strom's client" : ""}${a.browserOnly ? " · only through a person's browser" : ""}` : undefined, g.robots ? `  robots.txt: ${g.robots.line} (${g.robots.path})` : undefined, `  ${why}`, 
+        // its own search, as strom's list gives it: fetched as given, never guessed
+        ...(g.allow && search.length ? ["  its search (fetch as given; {q} the words, a space %20, ( %28, ) %29, [ %5B, ] %5D, : %3A, \" %22):", ...search.map((x) => `    ${x.what}: ${x.url}`)] : [])),
         data: {
             url: u.href,
             host: g.host,
@@ -144,6 +149,7 @@ function checkUrl(ctx, url) {
             ...(g.blocked ? { blocked: { until: new Date(g.blocked.until).toISOString(), ...(g.blocked.reason ? { reason: g.blocked.reason } : {}) } } : {}),
             ...(g.termsToday !== undefined ? { termsToday: g.termsToday } : {}),
             reason: why,
+            ...(g.allow && search.length ? { search } : {}),
         },
     };
 }

package/dist/core/connector.js

@@ -25,7 +25,7 @@ import { loadUserConfig, Settings } from "./config.js";
 import { readJsonIfExists, writeJson } from "./json.js";
 import { readAsset } from "./assets.js";
 import { foldText } from "./text.js";
-import { CookieJar, hostAllowed, hostState, NetError, paceOf, paceText, politeRequest, releaseForeground } from "./net.js";
+import { CookieJar, hostAllowed, hostState, NetError, paceOf, paceText, politeRequest, REFUSED_FOREVER, releaseForeground } from "./net.js";
 import { loginOf, redactor } from "./logins.js";
 import { HYPOTHESIS_LINKS_ORIGIN } from "./hypolinks.js";
 import { decodeImage, encodeImage, imageSize, imageSizeOfFile } from "../image/index.js";
@@ -1052,10 +1052,14 @@ export async function runConnector(c, request, opts) {
             catch (err) {
                 if (!(err instanceof NetError))
                     throw err;
-                if (err.failure === "cap" && err.until !== undefined)
-                    report.later = err.until;
                 // a refusal: why, and the limiter's advice with it (the agent reads it with what strom fetch says next)
                 const refusal = (err.failure === "refused" && (err.status === 401 || err.status === 403)) || (err.failure === "blocked" && !!err.refusedKind) || err.failure === "robots";
+                // a limit used up, or a host left alone for a while (silence, "slow down"): when to try again — exit 7
+                if (!refusal && err.until !== undefined && err.until < REFUSED_FOREVER && (err.failure === "cap" || err.failure === "blocked" || err.failure === "silent" || err.failure === "refused")) {
+                    report.later = err.until;
+                    if (err.failure !== "cap")
+                        report.laterDown = err.host;
+                }
                 if (refusal)
                     report.refused = {
                         host: err.host,

package/dist/core/errors.js

@@ -9,7 +9,10 @@ export const EXIT = {
     locked: 5,
     /** A backup before another channel or an older version could not be made: nothing is switched (core/backup.ts). */
     noBackup: 6,
-    /** An archive's limit is used up (its hourly cap): nothing more goes to it until the time said — other work meanwhile. */
+    /**
+     * An archive's limit is used up (its hourly cap), or it is left alone for a while (it went silent, asked to slow
+     * down): nothing more goes to it until the time said — other work meanwhile.
+     */
     later: 7,
     cancelled: 130,
 };

package/dist/core/knownterms.js

@@ -53,3 +53,21 @@ export function knownPaceMs(host) {
 export function knownDisallow(host) {
     return [...new Set(shippedTerms().filter((k) => within(host, k.host)).flatMap((k) => (Array.isArray(k.disallow) ? k.disallow.filter((d) => typeof d === "string" && d.startsWith("/")) : [])))];
 }
+/**
+ * The search addresses strom's list gives for a host (its own, else its site's): what an agent tried blind before
+ * (a Solr query's brackets and spaces unencoded: HTTP 400; a search page empty without a browser).
+ */
+export function knownSearch(host) {
+    const seen = new Set();
+    return shippedTerms()
+        .filter((k) => within(host, k.host) && Array.isArray(k.search))
+        .flatMap((k) => k.search.filter((x) => !!x && typeof x.url === "string" && /^https:\/\//.test(x.url) && typeof x.what === "string"))
+        .filter((x) => !seen.has(x.url) && !!seen.add(x.url));
+}
+/** The search addresses as lines for the agent (none: undefined). */
+export function searchText(host) {
+    const all = knownSearch(host);
+    if (!all.length)
+        return undefined;
+    return `strom: ${host}'s own search, fetched as given — {q} the words (a word, or (word OR word)), {from} {to} the years, {pid} an item found; every space %20, ( %28, ) %29, [ %5B, ] %5D, : %3A, " %22: ${all.map((x) => `${x.what}: ${x.url}`).join(" · ")}`;
+}

package/dist/core/net.js

@@ -60,6 +60,13 @@ export const AGENT_TOOL_TOKENS = { claude: ["claude-user", "claude-web", "claude
 export const MAX_WAIT_MS = 2 * 60_000;
 /** An hour off after a host kept asking us to slow down, or stopped answering. */
 export const COOL_OFF_MS = 3600_000;
+/**
+ * Off after a host went silent: an hour, and longer each time it goes silent again before it answered (within a day of
+ * the last time) — never shorter than an hour (gentle on archives: a firewall that drops requests looks the same).
+ */
+export const SILENT_COOL_MS = [COOL_OFF_MS, 2 * COOL_OFF_MS, 4 * COOL_OFF_MS];
+/** How long a silence counts towards the next, longer, time off (SILENT_COOL_MS). */
+const SILENT_RUN_MS = 24 * 3600_000;
 const TIMEOUT_MS = 60_000;
 /** Attempts per request: a busy server (5xx) a few, "slow down" (429) and silence only one more. */
 const ATTEMPTS = { busy: 3, slowDown: 2, silent: 2 };
@@ -115,7 +122,7 @@ export function capNear(s, pace, now) {
 /** A limit used up: try again then. */
 function capError(host, why, until, perHour) {
     const what = why === "cap" ? `${perHour} requests to ${host} in the last hour — its hourly cap` : `${host} says its limit is used up`;
-    return new NetError("cap", host, `${what}; try again at ${clock(until)}`, `go on with other work; run it again at ${clock(until)} — not sooner: strom sends nothing to ${host} before then${why === "limit" ? `; the person can lift it early: strom allow host ${host} --unblock` : ""}`, undefined, until);
+    return new NetError("cap", host, `${what}; try again at ${clockUp(until)}`, `go on with other work; run it again at ${clockUp(until)} — not sooner: strom sends nothing to ${host} before then${why === "limit" ? `; the person can lift it early: strom allow host ${host} --unblock` : ""}`, undefined, until);
 }
 /**
  * A fetch in the foreground holds a host this long after its last request (a mark beside the host's state, renewed
@@ -449,6 +456,11 @@ function checkHostState(raw, at) {
         drop("refusedFirst");
     if (has("okAt") && !until(raw.okAt))
         drop("okAt");
+    if (has("silentRun")) {
+        const r = raw.silentRun;
+        if (!(r && typeof r === "object" && typeof r.n === "number" && Number.isInteger(r.n) && r.n >= 1 && until(r.at)))
+            drop("silentRun");
+    }
     if (has("tracked") && !until(raw.tracked))
         drop("tracked");
     if (has("own")) {
@@ -511,7 +523,7 @@ export function stamp(ms) {
 }
 /** Until when a host is left alone, in words: a time, or until the person lifts it (a refusal of strom's client). */
 export function leftUntil(host, until) {
-    return until >= REFUSED_FOREVER ? `until the person lifts it (strom allow host ${host} --unblock)` : `until ${clock(until)}`;
+    return until >= REFUSED_FOREVER ? `until the person lifts it (strom allow host ${host} --unblock)` : `until ${clockUp(until)}`;
 }
 /**
  * The groups of a robots.txt that apply to strom: User-agent strom-research (its product token, any case, a version
@@ -672,7 +684,11 @@ function netError(failure, host, message, hint, status, extra = {}) {
 /** A host left alone (a refusal, silence, "slow down"): what strom says of a request it does not send. */
 function blockedError(host, s) {
     const until = s.blockedUntil;
-    return netError("blocked", host, `${host}: ${s.reason ?? "refused us"} — left alone ${leftUntil(host, until)}`, until >= REFUSED_FOREVER ? `do not try it through your web tools or a browser either; only the person lifts it: strom allow host ${host} --unblock` : `do not retry: go on with other work; the user can lift it early with strom allow host ${host} --unblock`, undefined, { refusedKind: s.refusedKind, until });
+    return netError("blocked", host, `${host}: ${s.reason ?? "refused us"} — left alone ${leftUntil(host, until)}`, until >= REFUSED_FOREVER ? `do not try it through your web tools or a browser either; only the person lifts it: strom allow host ${host} --unblock` : `do not retry before ${clockUp(until)}: strom sends ${host} nothing until then — go on with other work; the user can lift it early with strom allow host ${host} --unblock`, undefined, { refusedKind: s.refusedKind, until });
+}
+/** A host left alone for a while without a refusal (it went silent, asked to slow down): waited for in the background. */
+export function leftAloneFor(s, now) {
+    return s.blockedUntil && s.blockedUntil > now && s.blockedUntil < REFUSED_FOREVER && !s.refusedKind && !(s.reason ?? "").includes("refused") ? s.blockedUntil : undefined;
 }
 /** A time as the clock on the wall shows it here ("14:32"), with the day when it is not today. */
 export function clock(ms, today = Date.now()) {
@@ -683,7 +699,28 @@ export function clock(ms, today = Date.now()) {
     const sameDay = d.getFullYear() === t.getFullYear() && d.getMonth() === t.getMonth() && d.getDate() === t.getDate();
     return `${sameDay ? "" : `${day} `}${two(d.getHours())}:${two(d.getMinutes())}`;
 }
+/** A time something waits until, as clock() says it, rounded up to the minute: "11:04" for 11:03:22 — never too early. */
+export function clockUp(ms, today = Date.now()) {
+    return clock(Math.ceil(ms / 60_000) * 60_000, today);
+}
 /** One request, politely. Throws NetError when the host is not allowed, refused us, went silent, or is capped. */
+/**
+ * A wait that ends early when the run is stopped: its own timer, cleared then (a process stopped never stays for a
+ * timer left behind) — or a caller's sleep (a test's clock) raced with the stop.
+ */
+function stoppable(ms, stop, sleep) {
+    if (stop.aborted)
+        return Promise.resolve();
+    return new Promise((r) => {
+        let timer;
+        const done = () => (stop.removeEventListener("abort", done), timer !== undefined && clearTimeout(timer), r());
+        stop.addEventListener("abort", done, { once: true });
+        if (sleep)
+            void sleep(ms).then(done);
+        else
+            timer = setTimeout(done, ms);
+    });
+}
 /** A pause told to the caller's measure; whatever it does, the request goes on. */
 function paused(opts, p) {
     try {
@@ -880,8 +917,13 @@ export async function politeRequest(url, opts, redirects = 0) {
             pace = hostPace(s, opts.pace, human, host);
             slowed = (s.slowdown ?? 1) > 1;
             const t = now();
-            if (s.blockedUntil && s.blockedUntil > t)
+            if (s.blockedUntil && s.blockedUntil > t) {
+                // in the background a host left alone for a while is waited for (with the host let go), never one that refused
+                const free = opts.lane === "background" ? leftAloneFor(s, t) : undefined;
+                if (free !== undefined && free <= waitBy)
+                    throw new WaitOutside(free, "down");
                 throw blockedError(host, s);
+            }
             s.recent = s.recent.filter((x) => x > t - 3600_000);
             if (s.recent.length >= pace.perHour) {
                 const free = capFree(s.recent, pace.perHour);
@@ -904,7 +946,7 @@ export async function politeRequest(url, opts, redirects = 0) {
             const gap = (s.last ?? 0) + gapOf(s, pace) - now();
             // a pause longer than the caller waits (a site's long Crawl-delay): said when to come back, never sooner
             if (gap > MAX_WAIT_MS && now() + gap > waitBy)
-                throw new NetError("cap", host, `${host}'s pace (${paceText(pace)}${robotsDelayMs(s.robots) >= pace.minIntervalMs ? ", its robots.txt's Crawl-delay" : ""}) — the next request at ${clock(now() + gap)}`, `go on with other work; run it again at ${clock(now() + gap)} — not sooner`, undefined, now() + gap);
+                throw new NetError("cap", host, `${host}'s pace (${paceText(pace)}${robotsDelayMs(s.robots) >= pace.minIntervalMs ? ", its robots.txt's Crawl-delay" : ""}) — the next request at ${clockUp(now() + gap)}`, `go on with other work; run it again at ${clockUp(now() + gap)} — not sooner`, undefined, now() + gap);
             if (gap > 0) {
                 paused(opts, { host, ms: gap, why: "pace" });
                 await sleep(gap);
@@ -937,6 +979,12 @@ export async function politeRequest(url, opts, redirects = 0) {
             took = now() - asked;
             const cool = (ms, reason, kind) => {
                 s = hostState(opts.stateDir, host, now());
+                if (ms === "silent") {
+                    // silent again within a day, no answer since: longer off each time (an hour, two, four)
+                    const run = s.silentRun && now() - s.silentRun.at < SILENT_RUN_MS ? s.silentRun.n : 0;
+                    ms = SILENT_COOL_MS[Math.min(run, SILENT_COOL_MS.length - 1)];
+                    s.silentRun = { n: run + 1, at: now() };
+                }
                 s.blockedUntil = kind === "client" ? REFUSED_FOREVER : now() + ms;
                 s.reason = reason;
                 if (kind)
@@ -957,7 +1005,7 @@ export async function politeRequest(url, opts, redirects = 0) {
             else if (!robotsAsked && res?.status === 429 && tries.slowDown + 1 >= ATTEMPTS.slowDown)
                 cool(COOL_OFF_MS, `it asked us to slow down twice (HTTP 429 at ${stamp(now())})`);
             else if (!robotsAsked && !res && tries.silent + 1 >= ATTEMPTS.silent)
-                cool(COOL_OFF_MS, `no answer at ${stamp(now())} — the server is down, or it blocks this IP`);
+                cool("silent", `no answer at ${stamp(now())} — the server is down, or it blocks this IP`);
         }
         catch (err) {
             if (!(err instanceof WaitOutside))
@@ -970,10 +1018,14 @@ export async function politeRequest(url, opts, redirects = 0) {
         if (outside) {
             // a slot freed may go to another strom first: waited again, a few times at most, then said when to try again
             if (++waitedOutside > 3)
-                throw capError(host, outside.why, outside.until, pace.perHour);
+                throw outside.why === "down" ? blockedError(host, hostState(opts.stateDir, host, now())) : capError(host, outside.why, outside.until, pace.perHour);
             opts.onWait?.({ host, until: outside.until, ms: Math.max(0, outside.until - now()), why: outside.why });
             paused(opts, { host, ms: Math.max(0, outside.until - now()), why: outside.why });
-            await sleep(Math.max(0, outside.until - now()));
+            // a long wait in the background ends when its run is stopped (its session closed)
+            const ms = Math.max(0, outside.until - now());
+            await (opts.stop ? stoppable(ms, opts.stop, opts.sleep ?? testHooks.sleep) : sleep(ms));
+            if (opts.stop?.aborted)
+                throw new NetError("http", host, `stopped while it waited for ${host}`);
             continue;
         }
         // 426 "Upgrade Required": the server speaks HTTP/2 only (Node's fetch speaks HTTP/1.1).
@@ -1026,6 +1078,8 @@ export async function politeRequest(url, opts, redirects = 0) {
                 delete cur.refusedFirst;
             }
             cur.latencyMs = Math.round(cur.latencyMs === undefined ? took : cur.latencyMs * 0.7 + took * 0.3);
+            // it answered: a silence later starts from an hour off again
+            delete cur.silentRun;
             // the pause counts from the answer: a host that works long on each gets as long to rest
             cur.last = Math.max(cur.last ?? 0, now());
             const until = limitUsedUp(res.headers, now());
@@ -1044,13 +1098,17 @@ export async function politeRequest(url, opts, redirects = 0) {
         // Busy, "slow down", or no answer: slow down, and try again — a little.
         const cur = hostState(opts.stateDir, host, now());
         cur.slowdown = Math.min(16, (cur.slowdown ?? 1) * 2);
+        if (res)
+            delete cur.silentRun;
         writeJson(file, cur);
         tries[kind]++;
         if (tries[kind] >= attempts[kind]) {
+            const left = hostState(opts.stateDir, host, now()).blockedUntil;
+            const off = left !== undefined && left > now() ? left : undefined;
             if (kind === "silent")
-                throw new NetError("silent", host, `no answer from ${host} (${failed?.message ?? "timeout"}) — it is down, or it blocks this IP; strom leaves it alone for an hour`, "do not retry: a firewall that drops requests looks exactly like this");
+                throw new NetError("silent", host, `no answer from ${host} (${failed?.message ?? "timeout"}) — it is down, or it blocks this IP; strom leaves it alone${off ? ` until ${clockUp(off)}` : " for an hour"}`, "do not retry: a firewall that drops requests looks exactly like this", undefined, off);
             if (kind === "slowDown")
-                throw new NetError("refused", host, `${host} asked us twice to slow down (429) — strom leaves it alone for an hour`, "go on with other work; a smaller batch next time", 429);
+                throw new NetError("refused", host, `${host} asked us twice to slow down (429) — strom leaves it alone${off ? ` until ${clockUp(off)}` : " for an hour"}`, "go on with other work; a smaller batch next time", 429, off);
             throw new NetError("http", host, `${host} answered ${res.status} ${tries.busy} times — the server has trouble; try later`, undefined, res.status);
         }
         const asked = res ? retryAfterMs(res.headers.get("retry-after"), now()) : undefined;
@@ -1087,7 +1145,7 @@ export function reserveSlots(stateDir, host, asked, count, opts = {}) {
         const first = Math.max(t + (opts.leadMs ?? 0), (s.last ?? 0) + gap, s.waitUntil ?? 0);
         // never further ahead than SLOTS_AHEAD_MS: fewer than asked, the rest next time
         if (first > t + SLOTS_AHEAD_MS)
-            throw new NetError("cap", host, `${host}: strom has planned its requests for the next hour already; try again at ${clock(first - SLOTS_AHEAD_MS)}`, `go on with other work; run it again at ${clock(first - SLOTS_AHEAD_MS)}`, undefined, first - SLOTS_AHEAD_MS);
+            throw new NetError("cap", host, `${host}: strom has planned its requests for the next hour already; try again at ${clockUp(first - SLOTS_AHEAD_MS)}`, `go on with other work; run it again at ${clockUp(first - SLOTS_AHEAD_MS)}`, undefined, first - SLOTS_AHEAD_MS);
         const times = Array.from({ length: Math.min(free, Math.floor((t + SLOTS_AHEAD_MS - first) / gap) + 1) }, (_, i) => first + i * gap);
         s.last = times.at(-1);
         s.recent.push(...times);
@@ -1111,8 +1169,11 @@ export function refusedBy(stateDir, host, status, now = Date.now) {
         const reason = status === 429 ? `it asked the browser to slow down (HTTP 429 at ${stamp(now())})` : `it refused the browser (HTTP ${status} at ${stamp(now())})`;
         s.blockedUntil = Math.max(s.blockedUntil ?? 0, now() + ms);
         s.reason = reason;
+        // a refusal is said as one (strom fetch: refused, not "went silent"); a kind it had already stays
+        if (status !== 429)
+            s.refusedKind ??= "load";
         writeJson(file, s);
-        return `${host}: ${reason} — strom leaves it alone until ${clock(s.blockedUntil)}`;
+        return `${host}: ${reason} — strom leaves it alone until ${clockUp(s.blockedUntil)}`;
     }
     finally {
         release();

package/dist/core/tidy.js

@@ -49,6 +49,25 @@ export function tidyOn(root) {
         return false;
     }
 }
+/**
+ * The first tidy of an old tree offered to the person once (the menu): asked already, whatever the answer — the menu
+ * and the settings go on saying what can be freed.
+ */
+export function tidyAsked(root) {
+    try {
+        return typeof JSON.parse(fs.readFileSync(path.join(root, ".strom", "tidy.json"), "utf8")).asked === "string";
+    }
+    catch {
+        return false;
+    }
+}
+export function setTidyAsked(root) {
+    if (tidyOn(root))
+        return;
+    const file = path.join(root, ".strom", "tidy.json");
+    fs.mkdirSync(path.dirname(file), { recursive: true });
+    fs.writeFileSync(file, JSON.stringify({ auto: false, asked: new Date().toISOString() }) + "\n");
+}
 export function setTidyOn(root) {
     const file = path.join(root, ".strom", "tidy.json");
     fs.mkdirSync(path.dirname(file), { recursive: true });

package/dist/core/tree.js

@@ -27,7 +27,7 @@ import { lastBackup } from "./backup.js";
 export const TREE_FILE = "strom.json";
 // between releases the candidate being made (a bridge run from the sources says what it is: the Strom app goes by it);
 // at a release the release's own (package.json)
-export const VERSION = "1.18.0";
+export const VERSION = "1.19.0-beta.1";
 const GITIGNORE = `# Strom Research — local state and large files are not versioned
 .strom/
 /media/

package/dist/core/web.js

@@ -44,6 +44,7 @@ import { calibrationKey, viewModel } from "./viewsizes.js";
 import { ui } from "../cli/ui.js";
 import { isAgent } from "./which.js";
 import { otherWay, webGate } from "./webgate.js";
+import { searchText } from "./knownterms.js";
 /**
  * The thresholds of an agent's own web fetch, per site (webDomain) and session (core/metrics.ts): up to WEB_SOFT at the
  * host's pace alone; then never refused for their number, one per WEB_SOFT_GAP_MS to the whole site, the connector
@@ -466,10 +467,13 @@ export async function answerWebEvent(ev, o) {
             return {};
         const n = had.domains.get(site) ?? 0;
         const event = ev.phase === "failed" ? "PostToolUseFailure" : "PostToolUse";
+        // a site whose own search strom's list knows: its addresses with the first page of it in the session, and after a
+        // call that failed (what was tried blind before: brackets unencoded, HTTP 400)
+        const search = n === 1 || ev.phase === "failed" ? searchText(ev.netHost ?? host) : undefined;
         if (n < Math.max(1, perHost - WEB_NOTE_BEFORE)) {
             if (!had.advised.has(ev.toolUse))
-                return {};
-            return { context: `${softText(host, site, soft, perHost)} ${wayText(host, way())} ${NEVER}`, event };
+                return search ? { context: search, event } : {};
+            return { context: `${softText(host, site, soft, perHost)} ${wayText(host, way())} ${NEVER}${search ? ` ${search}` : ""}`, event };
         }
         const head = n <= perHost
             ? `strom: ${n} of the ${perHost} requests to ${siteText(host, site)} the web fetch tool takes in this session (web.perHost) used${n === perHost ? " — the next one is refused working alone, and asked of the user once in a conversation" : ""}.`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "strom-research",
-  "version": "1.18.0",
+  "version": "1.19.0-beta.1",
   "description": "Genealogical research toolkit for AI coding agents",
   "type": "module",
   "bin": {
@@ -57,5 +57,8 @@
   "devDependencies": {
     "@types/node": "^22.20.4",
     "typescript": "^7.0.2"
+  },
+  "publishConfig": {
+    "tag": "beta"
   }
 }