@robzilla1738/agentswarm 0.5.0 → 0.7.0

package/README.md

@@ -95,25 +95,32 @@ swarm run "Research the best open-source vector DBs in 2026 and write a recommen
 | `swarm cancel <id>` | Stop a run. It still synthesizes a report from completed work. |
 | `swarm config [list\|get\|set …]` | Manage `~/.agentswarm/config.json`. |
 | `swarm models` | List models from the active provider. |
+| `swarm config unset <key>` | Remove a setting (e.g., `swarm config unset firecrawlApiKey`). |
 | `swarm demo` | Run a self-contained demo mission in an isolated workspace. |
 
 Run options (also on the UI launch form under Options): `--workers N` (parallelism), `--tasks N`, `--steps N` (tool steps per task), `--budget N` (token cap), `--model`, `--conductor`, `--verify off|normal|strict`, `--effort low|medium|high|max`, `--no-thinking`, `--sandbox host|docker|e2b|modal|vercel|auto` (shell runtime for this run), `--cwd <path>` (run against a real directory instead of an isolated workspace), `--fg` (foreground in this process).
 
 ## How it works
 
-The conductor is a model with six tools: `spawn_tasks`, `set_phase`, `update_plan`, `read_report`, `wait`, and `finish`. It reads the mission, spawns self-contained tasks (each with an objective, success criteria, a role, optional dependencies, and an optional `verify` flag), then reacts as reports come back. On long missions it declares phases (`set_phase`) whose goals and exit criteria are pinned into every update — so the plan survives even when old history is trimmed and replaced by a mission ledger (settled tasks, decisions, current phase).
+The conductor is a model with six tools: `spawn_tasks`, `set_phase`, `update_plan`, `read_report`, `wait`, and `finish`. It reads the mission, spawns self-contained tasks (each with an objective, success criteria, a role, optional dependencies, and an optional `verify` flag), then reacts as reports come back. On long missions it declares phases (`set_phase`) whose goals and exit criteria are pinned into every update — so the plan survives even when old history is trimmed and replaced by a mission ledger (settled tasks, decisions, current phase). On resume, the conductor is re-seeded with this ledger so it picks up where it left off without losing context.
 
-Each task becomes an autonomous agent with a tool budget. It works in small steps, posts durable findings to the blackboard (decisions are never trimmed from digests; `search_notes` searches the full history), journals progress checkpoints on long tasks, saves artifacts, and ends by reporting back with structured handoff fields (`key_facts`, `open_questions`, `files_touched`). Dependent tasks receive report excerpts plus those fields, and can pull full text with `read_report`.
+Each task becomes an autonomous agent with a tool budget. It works in small steps, posts durable findings to the blackboard (decisions are never trimmed from digests; `search_notes` now supports `kind` filters to find decisions, context, or source links without noise), journals progress checkpoints on long tasks, saves artifacts, and ends by reporting back with structured handoff fields (`key_facts`, `open_questions`, `files_touched`) plus any sources discovered. Sources flow through to the final report as numbered citations — every source is deduplicated, attributed, and linked inline (`[1]`) with a full bibliography at the end. Dependent tasks receive report excerpts plus those fields, and can pull full text with `read_report`.
 
-**Scale.** A global AIMD limiter (`maxConcurrentCalls`) bounds concurrent model calls per endpoint — a 429 halves the ceiling, successes recover it, and conductor calls always jump the queue, so a 100-agent swarm degrades gracefully instead of melting down. Settles are debounced before waking the conductor; on big runs the task table collapses settled waves (failures stay itemized) and excess reports become one-liners the conductor can expand with `read_report`. Spawn specs take a `model` tier (`cheap` for scouts, `strong` for leads/verifiers via `cheapModel`/`strongModel` config) and `team:true` to run a task as a full sub-swarm — its own conductor decomposes it in parallel and reports one consolidated result, with all activity journaled under its `teamId`.
+**Search & research.** Web search now includes engine rate-limit cooldowns (on a 429, the engine skips it for a while and re-plans); queries reformulate themselves down to keywords if they get zero results (lifting recall without noise); results are freshness-ranked so recent content bubbles up. For academic queries, agents can use `academic_search` to query arXiv and Crossref directly — no API key needed. Fetches from the web pull plain text via `fetch_url`, which extracts text from PDFs (zero runtime dependencies, zlib only), decodes non-UTF-8 charsets, and flags paywall shells so agents know when they hit a wall.
 
-**Long horizon.** The conductor maintains a living `mission-plan.md` (`update_plan`) pinned into every update and restored on resume; every 25 settled tasks a progress snapshot lands in `artifacts/` so multi-day runs always have a partial deliverable; and real-directory runs leave a memory (`~/.agentswarm/memory/`) of missions, outcomes, and decisions that seeds the next swarm in the same workspace.
+**Scale.** A global AIMD limiter (`maxConcurrentCalls`) bounds concurrent model calls per endpoint — a 429 halves the ceiling, successes recover it, and conductor calls always jump the queue, so a 100-agent swarm degrades gracefully instead of melting down. Settles are debounced before waking the conductor; on big runs the task table collapses settled waves (failures stay itemized) and excess reports become one-liners the conductor can expand with `read_report`. Spawn specs take a `model` tier (`cheap` for scouts, `strong` for leads/verifiers via `cheapModel`/`strongModel` config) and `team:true` to run a task as a full sub-swarm — its own conductor decomposes it in parallel and reports one consolidated result, with all activity journaled under its `teamId`. Context windows are configurable per model via `contextWindows` config; the engine respects each model's actual limit and compacts agent context accordingly.
 
-Verified tasks pass two gates: a free mechanical check (claimed artifacts must exist and be non-empty), then a blind LLM verifier that judges the deliverables against the objective with its own tools — it never sees the worker's blackboard. In `--verify strict` mode, a completeness critic reviews the whole run for gaps before synthesis (the conductor gets one round to fill them), and the final report is checked for faithfulness against the task reports.
+**Worker tools.** The toolbelt gained `grep_files` for structured content search and `replace_in_file` with atomic multi-edit batches — both portable across sandboxes (Docker, E2B, Modal, Vercel).
+
+**Verification & quality.** Tasks pass a mechanical format pre-check (JSON/CSV/HTML structure), then a blind LLM verifier with its own tools. Failed verifications retry with structured feedback (problem/evidence/fix). The verifier gets copies of all dependencies' reports for context. In `--verify strict` mode, the verifier must back verdicts with tool-gathered evidence (not just a pass statement), a completeness critic reviews the whole run for gaps before synthesis, and the final report is checked for faithfulness against the task reports.
+
+**Long horizon.** The conductor maintains a living `mission-plan.md` (`update_plan`) pinned into every update and restored on resume; every 25 settled tasks a progress snapshot lands in `artifacts/` so multi-day runs always have a partial deliverable; and real-directory runs leave a memory (`~/.agentswarm/memory/`) of missions, outcomes, and decisions that seeds the next swarm in the same workspace. When tasks fail, the cascade carries the root cause transitively — blocked tasks know why rather than just "dependency did not complete". Failed tasks surface their last failing tool call as diagnostics.
+
+**Planning & steering.** The UI now includes a Plan tab showing the living `mission-plan.md`, and the conductor can update it from an agent note (`swarm note <id> "update the plan: ..."`). The budget sparkline in the run dashboard shows at-a-glance how much token budget remains.
 
 The scheduler starts a task as soon as its dependencies are done, up to the parallelism cap. Tasks whose dependencies failed are blocked and surfaced to the conductor for re-planning.
 
-When the conductor finishes (or the budget forces it), a synthesizer composes the final deliverable from every task report. Deliverables ship in the format the mission calls for — code, `.csv`/`.json` data, styled documents — alongside `final-report.md` and a self-contained `final-report.html` rendering (open it with `swarm report <id> --open`).
+When the conductor finishes (or the budget forces it), a synthesizer composes the final deliverable from every task report. Deliverables ship in the format the mission calls for — code, `.csv`/`.json` data, styled documents — alongside `final-report.md` and a self-contained `final-report.html` rendering (open it with `swarm report <id> --open`). The final report includes an inline-cited Sources section and all findings are preserved.
 
 The journal is the source of truth. Every run is an append-only `events.jsonl`; the terminal dashboard, the web UI, and `swarm ls` all reduce the same file. That's why runs survive crashes and can be resumed or replayed. Runs live under `~/.agentswarm/runs/<id>/`.
 
@@ -137,15 +144,24 @@ src/                         TypeScript engine (zero runtime deps)
   sandbox.ts    sandbox runtimes: host, docker, E2B, Modal, Vercel
   agent.ts      the agent loop: stream → tool calls → results → repeat, with compaction
   executor.ts   the orchestrator: conductor loop, parallel scheduler, verify, synth, budget
-  tools.ts      worker toolbelt (shell, files, web, blackboard, artifacts) + safety
-  webtools.ts   web search/fetch: SearchKit → TinyFish → DuckDuckGo fallback chain
+  tools.ts      worker toolbelt (shell, files, web, blackboard, artifacts) + safety + grep/replace
+  webtools.ts   web search/fetch: SearchKit → TinyFish → DuckDuckGo fallback chain, with cooldowns + reformulation
+  searchcore.ts search ranking (freshness boost, academic intent detection) + academic_search (arXiv/Crossref)
+  pdftext.ts    PDF text extraction (zero deps, zlib only)
+  crawltools.ts crawl backend resolver (firecrawl/context.dev/deepcrawl)
   journal.ts    append-only crash-safe event log (single source of truth)
-  state.ts      pure reducer: events → live run state
-  hub.ts        localhost HTTP API + SSE + static UI server
+  state.ts      pure reducer: events → live run state (with budgetSeries sampling)
+  hub.ts        localhost HTTP API + SSE + static UI server (CORS locked to localhost)
   terminal.ts   live TTY dashboard
   cli.ts        command-line interface
+  memory.ts     atomic runId-keyed cross-run memory + interim snapshots
 ui/             Next.js 15 + Tailwind 4 web app (static-exported, served by the hub)
+  components/SideRail Plan tab showing mission-plan.md
+  app/run/page.tsx Blackboard search with kind filters + budget sparkline
+  app/settings/page.tsx Test buttons for crawl/search backends, key management
 test/           end-to-end test with a scripted mock model (no API key needed)
+  e2e.js        21 phases covering the full pipeline, including citations + force + resume + budget + verify + teams
+  unit/*.test.js individual suites for tools, crawl, memory, pdftext, webtools, searchcore, citations
 ```
 
 ## Testing
@@ -158,11 +174,12 @@ Boots a mock model server and drives real missions through the engine, offline,
 
 ## Safety notes
 
-- Safe mode is on by default. It blocks obviously destructive shell commands and confines writes to the working directory. `--no-safe` turns it off for a run; only do that when you trust the mission.
+- Safe mode is on by default. It blocks obviously destructive shell commands and confines writes to the working directory, plus symlink escapes to parent directories. `--no-safe` turns it off for a run; only do that when you trust the mission.
+- The hub API (started by `swarm serve`) only accepts requests from localhost origins (`http://localhost:*` and `127.0.0.1:*`). The web UI runs in your browser locally and never phones home.
 - Runs default to an isolated per-run workspace on this machine. That's a private directory, not a container. Agents still execute with your user's permissions; the engine strips API keys and sandbox credentials from their environment, and safe mode constrains commands and writes. For untrusted or risky missions, use `--sandbox docker` or a cloud runtime.
 - Use `--cwd <path>` (or Workspace → "A directory on disk" in the UI) to let agents touch a real project. Those runs always execute on the host, since touching your real files is the point.
 - Costs are estimates based on list prices and the token counts the API reports. Models without pricing data show $0. Set a `--budget` either way.
-- Keys are stored in `~/.agentswarm/config.json` (chmod 600) and are only sent to the APIs you configured.
+- Keys are stored in `~/.agentswarm/config.json` (chmod 600) and are only sent to the APIs you configured. Use `swarm config unset <key>` to remove a key, or the Settings UI for test buttons on crawl/search backends.
 
 ## Author
 

package/dist/agent.js

@@ -2,6 +2,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.runAgent = runAgent;
 exports.estimateMessages = estimateMessages;
+const config_1 = require("./config");
 const deepseek_1 = require("./deepseek");
 const prompts_1 = require("./prompts");
 const types_1 = require("./types");
@@ -58,20 +59,10 @@ async function runAgent(p) {
         if (stopReason)
             break;
         steps++;
-        let res;
-        try {
-            res = await callModel();
-        }
-        catch (e) {
-            // The chat client already retries 429/5xx; this catches the rest of the
-            // transient class (connection resets, DNS blips) once per step so a
-            // single network hiccup doesn't burn a whole task attempt.
-            if (p.signal.aborted)
-                throw e;
-            hooks.onLog?.("warn", `${p.agentId}: model call failed (${(0, util_1.errMsg)(e)}); retrying once`);
-            await new Promise((r) => setTimeout(r, 1500));
-            res = await callModel();
-        }
+        // No retry here: the chat client already retries the whole transient
+        // class (429/5xx/network) with backoff; anything that escapes it is
+        // permanent (4xx) and re-sending only doubles the doomed spend.
+        const res = await callModel();
         hooks.onUsage?.(p.model, res.usage);
         usage = (0, types_1.addUsage)(usage, res.usage);
         if (res.toolCalls.length === 0) {
@@ -142,7 +133,7 @@ async function runAgent(p) {
             messages.push({ role: "tool", tool_call_id: call.id, content: result });
         }
         hooks.onTranscript?.(messages);
-        if (estimateMessages(messages) > cfg.contextTokenLimit) {
+        if (estimateMessages(messages) > (0, config_1.contextLimitFor)(cfg, p.model)) {
             messages = await compact(p, messages);
             hooks.onTranscript?.(messages);
             hooks.onLog?.("info", `${p.agentId}: context compacted`);

package/dist/cli.js

@@ -524,15 +524,27 @@ async function cmdConfig(rest, flags) {
         const cfg = (0, config_1.loadConfig)();
         if (sub === "get" && rest[1]) {
             const key = rest[1];
-            const v = key === "apiKey" || key === "tinyfishApiKey" ? (0, config_1.maskKey)(String(cfg[key])) : cfg[key];
+            if (key === "providers") {
+                // Nested per-provider creds — mask every apiKey, never dump raw.
+                const masked = Object.fromEntries(Object.entries(cfg.providers ?? {}).map(([id, c]) => [
+                    id,
+                    { ...c, ...(c?.apiKey ? { apiKey: (0, config_1.maskKey)(c.apiKey) } : {}) },
+                ]));
+                console.log(JSON.stringify(masked, null, 2));
+                return;
+            }
+            const v = (0, config_1.isSecretConfigKey)(key) ? (0, config_1.maskKey)(String(cfg[key] ?? "")) : cfg[key];
             console.log(typeof v === "object" ? JSON.stringify(v, null, 2) : String(v));
             return;
         }
         console.log(util_1.ansi.bold("config") + util_1.ansi.gray(`  (${(0, config_1.configPath)()})`));
         for (const k of config_1.SETTABLE_KEYS) {
             let v = cfg[k];
-            if (k === "apiKey" || k === "tinyfishApiKey")
-                v = v ? (0, config_1.maskKey)(String(v)) : util_1.ansi.red("(not set)");
+            // Every secret-bearing key prints masked — `config list` output ends up
+            // in terminal scrollback and pasted bug reports.
+            if ((0, config_1.isSecretConfigKey)(k)) {
+                v = v ? (0, config_1.maskKey)(String(v)) : k === "apiKey" ? util_1.ansi.red("(not set)") : "(not set)";
+            }
             console.log(`  ${k.padEnd(18)} ${util_1.ansi.gray(String(v))}`);
         }
         return;
@@ -559,11 +571,26 @@ async function cmdConfig(rest, flags) {
             console.log(util_1.ansi.gray("  verify it works: ") + "swarm models");
         return;
     }
+    if (sub === "unset") {
+        const key = rest[1];
+        if (!key)
+            throw new Error("usage: swarm config unset <key>");
+        if (!config_1.SETTABLE_KEYS.includes(key)) {
+            throw new Error(`unknown key. Keys: ${config_1.SETTABLE_KEYS.join(", ")}`);
+        }
+        // apiKey/baseUrl route into the active provider's creds, so clearing
+        // means writing "". Everything else is deleted from the file outright —
+        // the default applies again (unset model:"" would brick every run).
+        const cred = key === "apiKey" || key === "baseUrl";
+        (0, config_1.saveConfig)({ [key]: cred ? "" : undefined });
+        console.log(util_1.ansi.green("✓ ") + `cleared ${key}` + (cred ? "" : util_1.ansi.gray(" — default applies")));
+        return;
+    }
     if (sub === "path") {
         console.log((0, config_1.configPath)());
         return;
     }
-    throw new Error("usage: swarm config [list|get <key>|set <key> <value>|path]");
+    throw new Error("usage: swarm config [list|get <key>|set <key> <value>|unset <key>|path]");
 }
 async function cmdModels() {
     const cfg = (0, config_1.loadConfig)();

package/dist/config.js

@@ -33,13 +33,15 @@ var __importStar = (this && this.__importStar) || (function () {
     };
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.SETTABLE_KEYS = exports.SECRET_ENV_KEYS = exports.DEFAULTS = exports.DEFAULT_PRICING = void 0;
+exports.SETTABLE_KEYS = exports.SECRET_ENV_KEYS = exports.DEFAULTS = exports.DEFAULT_WINDOWS = exports.DEFAULT_PRICING = void 0;
+exports.contextLimitFor = contextLimitFor;
 exports.home = home;
 exports.runsDir = runsDir;
 exports.runDir = runDir;
 exports.configPath = configPath;
 exports.loadConfig = loadConfig;
 exports.saveConfig = saveConfig;
+exports.isSecretConfigKey = isSecretConfigKey;
 exports.maskKey = maskKey;
 exports.coerceConfigValue = coerceConfigValue;
 const fs = __importStar(require("fs"));
@@ -63,6 +65,20 @@ exports.DEFAULT_PRICING = {
     "MiniMax-M2.1": { inMiss: 0.3, inHit: 0.03, out: 1.2 },
     "MiniMax-M2": { inMiss: 0.3, inHit: 0.03, out: 1.2 },
 };
+exports.DEFAULT_WINDOWS = {
+    // tokens (June 2026 published limits; conservative where ranges exist)
+    "deepseek-v4-flash": 128_000,
+    "deepseek-v4-pro": 128_000,
+    "deepseek-chat": 128_000,
+    "deepseek-reasoner": 128_000,
+    "gpt-5.1": 272_000,
+    "gpt-5.1-mini": 272_000,
+    "claude-opus-4-8": 200_000,
+    "claude-sonnet-4-6": 200_000,
+    "claude-haiku-4-5": 200_000,
+    "MiniMax-M2.1": 192_000,
+    "MiniMax-M2": 192_000,
+};
 exports.DEFAULTS = {
     provider: "deepseek",
     providers: {},
@@ -105,7 +121,17 @@ exports.DEFAULTS = {
     hubPort: 7777,
     uiPort: 7780,
     pricing: exports.DEFAULT_PRICING,
+    contextWindows: exports.DEFAULT_WINDOWS,
 };
+/**
+ * Effective compaction/trim threshold for a model: the configured limit,
+ * hard-capped by the model's known context window (15% headroom for output
+ * and estimation error). Models we don't know keep the configured limit.
+ */
+function contextLimitFor(cfg, model) {
+    const win = cfg.contextWindows[model];
+    return win ? Math.min(cfg.contextTokenLimit, Math.floor(win * 0.85)) : cfg.contextTokenLimit;
+}
 /**
  * Env vars that must never leak into agent shell commands when they execute
  * directly on the host: every provider key env plus the search/sandbox
@@ -163,9 +189,17 @@ function loadConfig() {
         provider,
         providers: file.providers || {},
         pricing: { ...exports.DEFAULT_PRICING, ...(file.pricing || {}) },
+        contextWindows: { ...exports.DEFAULT_WINDOWS, ...(file.contextWindows || {}) },
         apiKey: cred.apiKey || "",
         baseUrl: cred.baseUrl || info.baseUrl,
     };
+    // A cleared/hand-edited model must fall back, not brick every run with
+    // model:"" requests. (cheapModel/strongModel legitimately clear to "" —
+    // they mean "use `model`".)
+    if (!cfg.model)
+        cfg.model = info.defaultModel;
+    if (!cfg.conductorModel)
+        cfg.conductorModel = cfg.model;
     // Env overrides: provider-specific key env, plus legacy DEEPSEEK_API_KEY.
     if (info.keyEnv && process.env[info.keyEnv])
         cfg.apiKey = process.env[info.keyEnv];
@@ -223,6 +257,15 @@ function saveConfig(patch) {
     }
     return loadConfig();
 }
+/**
+ * Config keys whose values must never print in cleartext — CLI output ends up
+ * in terminal scrollback and pasted bug reports. `providers` holds nested
+ * per-provider apiKeys, so it counts too. Single source of truth for the CLI
+ * masking sites (the hub's publicConfig is a strict allowlist already).
+ */
+function isSecretConfigKey(key) {
+    return /apikey|token|secret/i.test(key) || key === "providers";
+}
 function maskKey(key) {
     if (!key)
         return "";

package/dist/crawltools.js

@@ -138,7 +138,7 @@ async function firecrawlCrawl(cfg, opts, warnings) {
             warnings.push(`crawl still running after 120s; returning ${partial.length} partial pages`);
             return partial;
         }
-        await sleep(pollMs, opts.signal);
+        await (0, util_1.sleep)(pollMs, opts.signal);
     }
     // Completed: collect pages, following `next` pagination until maxPages.
     const pages = mapFirecrawlPages(last);
@@ -206,18 +206,12 @@ function friendlyHttpError(service, status, body) {
         return new Error(`${service}: rate limited (HTTP 429) — retry later`);
     return new Error(`${service}: HTTP ${status} ${(0, util_1.truncateMiddle)(body, 300, "chars")}`);
 }
-function mergeSignal(timeoutMs, signal) {
-    const t = AbortSignal.timeout(timeoutMs);
-    if (!signal)
-        return t;
-    return typeof AbortSignal.any === "function" ? AbortSignal.any([t, signal]) : signal;
-}
 async function callJson(service, url, key, body, timeoutMs, signal) {
     const res = await fetch(url, {
         method: "POST",
         headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
         body: JSON.stringify(body),
-        signal: mergeSignal(timeoutMs, signal),
+        signal: (0, util_1.mergeSignal)(timeoutMs, signal),
     });
     if (!res.ok)
         throw friendlyHttpError(service, res.status, await res.text().catch(() => ""));
@@ -226,22 +220,9 @@ async function callJson(service, url, key, body, timeoutMs, signal) {
 async function getJson(service, url, key, signal) {
     const res = await fetch(url, {
         headers: { authorization: `Bearer ${key}` },
-        signal: mergeSignal(30_000, signal),
+        signal: (0, util_1.mergeSignal)(30_000, signal),
     });
     if (!res.ok)
         throw friendlyHttpError(service, res.status, await res.text().catch(() => ""));
     return res.json();
 }
-function sleep(ms, signal) {
-    return new Promise((resolve, reject) => {
-        const t = setTimeout(() => {
-            signal?.removeEventListener("abort", onAbort);
-            resolve();
-        }, ms);
-        const onAbort = () => {
-            clearTimeout(t);
-            reject(new Error("aborted"));
-        };
-        signal?.addEventListener("abort", onAbort, { once: true });
-    });
-}