@robzilla1738/agentswarm 0.5.0 → 0.6.0

package/README.md

@@ -95,25 +95,32 @@ swarm run "Research the best open-source vector DBs in 2026 and write a recommen
 | `swarm cancel <id>` | Stop a run. It still synthesizes a report from completed work. |
 | `swarm config [list\|get\|set …]` | Manage `~/.agentswarm/config.json`. |
 | `swarm models` | List models from the active provider. |
+| `swarm config unset <key>` | Remove a setting (e.g., `swarm config unset firecrawlApiKey`). |
 | `swarm demo` | Run a self-contained demo mission in an isolated workspace. |
 
 Run options (also on the UI launch form under Options): `--workers N` (parallelism), `--tasks N`, `--steps N` (tool steps per task), `--budget N` (token cap), `--model`, `--conductor`, `--verify off|normal|strict`, `--effort low|medium|high|max`, `--no-thinking`, `--sandbox host|docker|e2b|modal|vercel|auto` (shell runtime for this run), `--cwd <path>` (run against a real directory instead of an isolated workspace), `--fg` (foreground in this process).
 
 ## How it works
 
-The conductor is a model with six tools: `spawn_tasks`, `set_phase`, `update_plan`, `read_report`, `wait`, and `finish`. It reads the mission, spawns self-contained tasks (each with an objective, success criteria, a role, optional dependencies, and an optional `verify` flag), then reacts as reports come back. On long missions it declares phases (`set_phase`) whose goals and exit criteria are pinned into every update — so the plan survives even when old history is trimmed and replaced by a mission ledger (settled tasks, decisions, current phase).
+The conductor is a model with six tools: `spawn_tasks`, `set_phase`, `update_plan`, `read_report`, `wait`, and `finish`. It reads the mission, spawns self-contained tasks (each with an objective, success criteria, a role, optional dependencies, and an optional `verify` flag), then reacts as reports come back. On long missions it declares phases (`set_phase`) whose goals and exit criteria are pinned into every update — so the plan survives even when old history is trimmed and replaced by a mission ledger (settled tasks, decisions, current phase). On resume, the conductor is re-seeded with this ledger so it picks up where it left off without losing context.
 
-Each task becomes an autonomous agent with a tool budget. It works in small steps, posts durable findings to the blackboard (decisions are never trimmed from digests; `search_notes` searches the full history), journals progress checkpoints on long tasks, saves artifacts, and ends by reporting back with structured handoff fields (`key_facts`, `open_questions`, `files_touched`). Dependent tasks receive report excerpts plus those fields, and can pull full text with `read_report`.
+Each task becomes an autonomous agent with a tool budget. It works in small steps, posts durable findings to the blackboard (decisions are never trimmed from digests; `search_notes` now supports `kind` filters to find decisions, context, or source links without noise), journals progress checkpoints on long tasks, saves artifacts, and ends by reporting back with structured handoff fields (`key_facts`, `open_questions`, `files_touched`) plus any sources discovered. Sources flow through to the final report as numbered citations — every source is deduplicated, attributed, and linked inline (`[1]`) with a full bibliography at the end. Dependent tasks receive report excerpts plus those fields, and can pull full text with `read_report`.
 
-**Scale.** A global AIMD limiter (`maxConcurrentCalls`) bounds concurrent model calls per endpoint — a 429 halves the ceiling, successes recover it, and conductor calls always jump the queue, so a 100-agent swarm degrades gracefully instead of melting down. Settles are debounced before waking the conductor; on big runs the task table collapses settled waves (failures stay itemized) and excess reports become one-liners the conductor can expand with `read_report`. Spawn specs take a `model` tier (`cheap` for scouts, `strong` for leads/verifiers via `cheapModel`/`strongModel` config) and `team:true` to run a task as a full sub-swarm — its own conductor decomposes it in parallel and reports one consolidated result, with all activity journaled under its `teamId`.
+**Search & research.** Web search now includes engine rate-limit cooldowns (on a 429, the engine skips it for a while and re-plans); queries reformulate themselves down to keywords if they get zero results (lifting recall without noise); results are freshness-ranked so recent content bubbles up. For academic queries, agents can use `academic_search` to query arXiv and Crossref directly — no API key needed. Fetches from the web pull plain text via `fetch_url`, which extracts text from PDFs (zero runtime dependencies, zlib only), decodes non-UTF-8 charsets, and flags paywall shells so agents know when they hit a wall.
 
-**Long horizon.** The conductor maintains a living `mission-plan.md` (`update_plan`) pinned into every update and restored on resume; every 25 settled tasks a progress snapshot lands in `artifacts/` so multi-day runs always have a partial deliverable; and real-directory runs leave a memory (`~/.agentswarm/memory/`) of missions, outcomes, and decisions that seeds the next swarm in the same workspace.
+**Scale.** A global AIMD limiter (`maxConcurrentCalls`) bounds concurrent model calls per endpoint — a 429 halves the ceiling, successes recover it, and conductor calls always jump the queue, so a 100-agent swarm degrades gracefully instead of melting down. Settles are debounced before waking the conductor; on big runs the task table collapses settled waves (failures stay itemized) and excess reports become one-liners the conductor can expand with `read_report`. Spawn specs take a `model` tier (`cheap` for scouts, `strong` for leads/verifiers via `cheapModel`/`strongModel` config) and `team:true` to run a task as a full sub-swarm — its own conductor decomposes it in parallel and reports one consolidated result, with all activity journaled under its `teamId`. Context windows are configurable per model via `contextWindows` config; the engine respects each model's actual limit and compacts agent context accordingly.
 
-Verified tasks pass two gates: a free mechanical check (claimed artifacts must exist and be non-empty), then a blind LLM verifier that judges the deliverables against the objective with its own tools — it never sees the worker's blackboard. In `--verify strict` mode, a completeness critic reviews the whole run for gaps before synthesis (the conductor gets one round to fill them), and the final report is checked for faithfulness against the task reports.
+**Worker tools.** The toolbelt gained `grep_files` for structured content search and `replace_in_file` with atomic multi-edit batches — both portable across sandboxes (Docker, E2B, Modal, Vercel).
+
+**Verification & quality.** Tasks pass a mechanical format pre-check (JSON/CSV/HTML structure), then a blind LLM verifier with its own tools. Failed verifications retry with structured feedback (problem/evidence/fix). The verifier gets copies of all dependencies' reports for context. In `--verify strict` mode, the verifier must back verdicts with tool-gathered evidence (not just a pass statement), a completeness critic reviews the whole run for gaps before synthesis, and the final report is checked for faithfulness against the task reports.
+
+**Long horizon.** The conductor maintains a living `mission-plan.md` (`update_plan`) pinned into every update and restored on resume; every 25 settled tasks a progress snapshot lands in `artifacts/` so multi-day runs always have a partial deliverable; and real-directory runs leave a memory (`~/.agentswarm/memory/`) of missions, outcomes, and decisions that seeds the next swarm in the same workspace. When tasks fail, the cascade carries the root cause transitively — blocked tasks know why rather than just "dependency did not complete". Failed tasks surface their last failing tool call as diagnostics.
+
+**Planning & steering.** The UI now includes a Plan tab showing the living `mission-plan.md`, and the conductor can update it from an agent note (`swarm note <id> "update the plan: ..."`). The budget sparkline in the run dashboard shows at-a-glance how much token budget remains.
 
 The scheduler starts a task as soon as its dependencies are done, up to the parallelism cap. Tasks whose dependencies failed are blocked and surfaced to the conductor for re-planning.
 
-When the conductor finishes (or the budget forces it), a synthesizer composes the final deliverable from every task report. Deliverables ship in the format the mission calls for — code, `.csv`/`.json` data, styled documents — alongside `final-report.md` and a self-contained `final-report.html` rendering (open it with `swarm report <id> --open`).
+When the conductor finishes (or the budget forces it), a synthesizer composes the final deliverable from every task report. Deliverables ship in the format the mission calls for — code, `.csv`/`.json` data, styled documents — alongside `final-report.md` and a self-contained `final-report.html` rendering (open it with `swarm report <id> --open`). The final report includes an inline-cited Sources section and all findings are preserved.
 
 The journal is the source of truth. Every run is an append-only `events.jsonl`; the terminal dashboard, the web UI, and `swarm ls` all reduce the same file. That's why runs survive crashes and can be resumed or replayed. Runs live under `~/.agentswarm/runs/<id>/`.
 
@@ -137,15 +144,24 @@ src/                         TypeScript engine (zero runtime deps)
   sandbox.ts    sandbox runtimes: host, docker, E2B, Modal, Vercel
   agent.ts      the agent loop: stream → tool calls → results → repeat, with compaction
   executor.ts   the orchestrator: conductor loop, parallel scheduler, verify, synth, budget
-  tools.ts      worker toolbelt (shell, files, web, blackboard, artifacts) + safety
-  webtools.ts   web search/fetch: SearchKit → TinyFish → DuckDuckGo fallback chain
+  tools.ts      worker toolbelt (shell, files, web, blackboard, artifacts) + safety + grep/replace
+  webtools.ts   web search/fetch: SearchKit → TinyFish → DuckDuckGo fallback chain, with cooldowns + reformulation
+  searchcore.ts search ranking (freshness boost, academic intent detection) + academic_search (arXiv/Crossref)
+  pdftext.ts    PDF text extraction (zero deps, zlib only)
+  crawltools.ts crawl backend resolver (firecrawl/context.dev/deepcrawl)
   journal.ts    append-only crash-safe event log (single source of truth)
-  state.ts      pure reducer: events → live run state
-  hub.ts        localhost HTTP API + SSE + static UI server
+  state.ts      pure reducer: events → live run state (with budgetSeries sampling)
+  hub.ts        localhost HTTP API + SSE + static UI server (CORS locked to localhost)
   terminal.ts   live TTY dashboard
   cli.ts        command-line interface
+  memory.ts     atomic runId-keyed cross-run memory + interim snapshots
 ui/             Next.js 15 + Tailwind 4 web app (static-exported, served by the hub)
+  components/SideRail Plan tab showing mission-plan.md
+  app/run/page.tsx Blackboard search with kind filters + budget sparkline
+  app/settings/page.tsx Test buttons for crawl/search backends, key management
 test/           end-to-end test with a scripted mock model (no API key needed)
+  e2e.js        21 phases covering the full pipeline, including citations + force + resume + budget + verify + teams
+  unit/*.test.js individual suites for tools, crawl, memory, pdftext, webtools, searchcore, citations
 ```
 
 ## Testing
@@ -158,11 +174,12 @@ Boots a mock model server and drives real missions through the engine, offline,
 
 ## Safety notes
 
-- Safe mode is on by default. It blocks obviously destructive shell commands and confines writes to the working directory. `--no-safe` turns it off for a run; only do that when you trust the mission.
+- Safe mode is on by default. It blocks obviously destructive shell commands and confines writes to the working directory, plus symlink escapes to parent directories. `--no-safe` turns it off for a run; only do that when you trust the mission.
+- The hub API (started by `swarm serve`) only accepts requests from localhost origins (`http://localhost:*` and `127.0.0.1:*`). The web UI runs in your browser locally and never phones home.
 - Runs default to an isolated per-run workspace on this machine. That's a private directory, not a container. Agents still execute with your user's permissions; the engine strips API keys and sandbox credentials from their environment, and safe mode constrains commands and writes. For untrusted or risky missions, use `--sandbox docker` or a cloud runtime.
 - Use `--cwd <path>` (or Workspace → "A directory on disk" in the UI) to let agents touch a real project. Those runs always execute on the host, since touching your real files is the point.
 - Costs are estimates based on list prices and the token counts the API reports. Models without pricing data show $0. Set a `--budget` either way.
-- Keys are stored in `~/.agentswarm/config.json` (chmod 600) and are only sent to the APIs you configured.
+- Keys are stored in `~/.agentswarm/config.json` (chmod 600) and are only sent to the APIs you configured. Use `swarm config unset <key>` to remove a key, or the Settings UI for test buttons on crawl/search backends.
 
 ## Author
 

package/dist/agent.js

@@ -2,6 +2,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.runAgent = runAgent;
 exports.estimateMessages = estimateMessages;
+const config_1 = require("./config");
 const deepseek_1 = require("./deepseek");
 const prompts_1 = require("./prompts");
 const types_1 = require("./types");
@@ -142,7 +143,7 @@ async function runAgent(p) {
             messages.push({ role: "tool", tool_call_id: call.id, content: result });
         }
         hooks.onTranscript?.(messages);
-        if (estimateMessages(messages) > cfg.contextTokenLimit) {
+        if (estimateMessages(messages) > (0, config_1.contextLimitFor)(cfg, p.model)) {
             messages = await compact(p, messages);
             hooks.onTranscript?.(messages);
             hooks.onLog?.("info", `${p.agentId}: context compacted`);

package/dist/cli.js

@@ -524,15 +524,18 @@ async function cmdConfig(rest, flags) {
         const cfg = (0, config_1.loadConfig)();
         if (sub === "get" && rest[1]) {
             const key = rest[1];
-            const v = key === "apiKey" || key === "tinyfishApiKey" ? (0, config_1.maskKey)(String(cfg[key])) : cfg[key];
+            const v = /apikey|token|secret/i.test(key) ? (0, config_1.maskKey)(String(cfg[key] ?? "")) : cfg[key];
             console.log(typeof v === "object" ? JSON.stringify(v, null, 2) : String(v));
             return;
         }
         console.log(util_1.ansi.bold("config") + util_1.ansi.gray(`  (${(0, config_1.configPath)()})`));
         for (const k of config_1.SETTABLE_KEYS) {
             let v = cfg[k];
-            if (k === "apiKey" || k === "tinyfishApiKey")
-                v = v ? (0, config_1.maskKey)(String(v)) : util_1.ansi.red("(not set)");
+            // Every secret-bearing key prints masked — `config list` output ends up
+            // in terminal scrollback and pasted bug reports.
+            if (/apikey|token|secret/i.test(k)) {
+                v = v ? (0, config_1.maskKey)(String(v)) : k === "apiKey" ? util_1.ansi.red("(not set)") : "(not set)";
+            }
             console.log(`  ${k.padEnd(18)} ${util_1.ansi.gray(String(v))}`);
         }
         return;
@@ -559,11 +562,25 @@ async function cmdConfig(rest, flags) {
             console.log(util_1.ansi.gray("  verify it works: ") + "swarm models");
         return;
     }
+    if (sub === "unset") {
+        const key = rest[1];
+        if (!key)
+            throw new Error("usage: swarm config unset <key>");
+        // Only string-valued keys can sensibly clear to "" — numbers/enums keep
+        // their defaults via `set`.
+        const clearable = config_1.SETTABLE_KEYS.filter((k) => /apikey|token|secret|url|model/i.test(k));
+        if (!clearable.includes(key)) {
+            throw new Error(`not clearable. Clearable keys: ${clearable.join(", ")}`);
+        }
+        (0, config_1.saveConfig)({ [key]: "" });
+        console.log(util_1.ansi.green("✓ ") + `cleared ${key}`);
+        return;
+    }
     if (sub === "path") {
         console.log((0, config_1.configPath)());
         return;
     }
-    throw new Error("usage: swarm config [list|get <key>|set <key> <value>|path]");
+    throw new Error("usage: swarm config [list|get <key>|set <key> <value>|unset <key>|path]");
 }
 async function cmdModels() {
     const cfg = (0, config_1.loadConfig)();

package/dist/config.js

@@ -33,7 +33,8 @@ var __importStar = (this && this.__importStar) || (function () {
     };
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.SETTABLE_KEYS = exports.SECRET_ENV_KEYS = exports.DEFAULTS = exports.DEFAULT_PRICING = void 0;
+exports.SETTABLE_KEYS = exports.SECRET_ENV_KEYS = exports.DEFAULTS = exports.DEFAULT_WINDOWS = exports.DEFAULT_PRICING = void 0;
+exports.contextLimitFor = contextLimitFor;
 exports.home = home;
 exports.runsDir = runsDir;
 exports.runDir = runDir;
@@ -63,6 +64,20 @@ exports.DEFAULT_PRICING = {
     "MiniMax-M2.1": { inMiss: 0.3, inHit: 0.03, out: 1.2 },
     "MiniMax-M2": { inMiss: 0.3, inHit: 0.03, out: 1.2 },
 };
+exports.DEFAULT_WINDOWS = {
+    // tokens (June 2026 published limits; conservative where ranges exist)
+    "deepseek-v4-flash": 128_000,
+    "deepseek-v4-pro": 128_000,
+    "deepseek-chat": 128_000,
+    "deepseek-reasoner": 128_000,
+    "gpt-5.1": 272_000,
+    "gpt-5.1-mini": 272_000,
+    "claude-opus-4-8": 200_000,
+    "claude-sonnet-4-6": 200_000,
+    "claude-haiku-4-5": 200_000,
+    "MiniMax-M2.1": 192_000,
+    "MiniMax-M2": 192_000,
+};
 exports.DEFAULTS = {
     provider: "deepseek",
     providers: {},
@@ -105,7 +120,17 @@ exports.DEFAULTS = {
     hubPort: 7777,
     uiPort: 7780,
     pricing: exports.DEFAULT_PRICING,
+    contextWindows: exports.DEFAULT_WINDOWS,
 };
+/**
+ * Effective compaction/trim threshold for a model: the configured limit,
+ * hard-capped by the model's known context window (15% headroom for output
+ * and estimation error). Models we don't know keep the configured limit.
+ */
+function contextLimitFor(cfg, model) {
+    const win = cfg.contextWindows[model];
+    return win ? Math.min(cfg.contextTokenLimit, Math.floor(win * 0.85)) : cfg.contextTokenLimit;
+}
 /**
  * Env vars that must never leak into agent shell commands when they execute
  * directly on the host: every provider key env plus the search/sandbox
@@ -163,6 +188,7 @@ function loadConfig() {
         provider,
         providers: file.providers || {},
         pricing: { ...exports.DEFAULT_PRICING, ...(file.pricing || {}) },
+        contextWindows: { ...exports.DEFAULT_WINDOWS, ...(file.contextWindows || {}) },
         apiKey: cred.apiKey || "",
         baseUrl: cred.baseUrl || info.baseUrl,
     };