savant-code 0.0.19 → 0.0.20

package/docs/launch/newsletter-pitch.md

@@ -1,49 +0,0 @@
-# Newsletter Pitch (Console.dev style)
-
-**To:** Console.dev submissions
-**Subject:** Submission: Savant Code — local-first AI coding CLI with a 9-agent
-auditing loop
-
----
-
-Hi Console.dev team,
-
-I would like to submit Savant Code for consideration.
-
-## What it is
-
-Savant Code is an open-source, terminal-native AI coding assistant. Instead of
-racing to generate code as fast as possible, it runs a 9-agent "perfection
-loop" that audits, critiques, and rewrites changes before they touch your
-codebase.
-
-## Why it matters
-
-Most AI coding tools optimize for speed. Savant Code optimizes for correctness.
-The loop is slower by design, but the output is code that actually works. It is
-built for developers who are tired of debugging hallucinated patches.
-
-## Local-first and provider choice
-
-- Local-first: auto-detects Ollama and routes inference locally.
-- BYOK: bring your own API keys if you prefer; remote providers receive the
-  prompt and context you choose to send them.
-- No mandatory account is required for a local Ollama workflow.
-- Telemetry/privacy policy remains an explicit pre-launch decision and is not
-  asserted as resolved by this submission.
-
-## Built with
-
-- TypeScript + Bun
-- React + OpenTUI for the terminal UI
-- ECHO Protocol for agent governance
-- Apache-2.0 license
-
-## Links
-
-- GitHub: <https://github.com/savant0x/savant-code>
-- Docs: <https://github.com/savant0x/savant-code/blob/main/ECHO.md>
-
-Thanks for considering it.
-
-— The Savant Code team

package/docs/launch/twitter-thread.md

@@ -1,62 +0,0 @@
-# Twitter/X Launch Thread
-
-Tweet 1/5 — hook
-
-```text
-Stop babysitting your AI coding assistant.
-
-Most agents ship fast code. Savant Code ships code that actually works.
-
-A 9-agent perfection loop audits every change before it touches your repo.
-
-Install: npm install -g savant-code
-
-#AI #OpenSource #CLI #DevTools
-```
-
-Tweet 2/5 — privacy angle
-
-```text
-Your codebase is valuable. You should not have to ship it to someone else's
-server.
-
-Savant Code is local-first with Ollama auto-detection. No API key. No account.
-No setup prompts.
-
-BYOK mode is there if you want it, but local is the default.
-```
-
-Tweet 3/5 — demo
-
-```text
-Here is what the perfection loop looks like in practice:
-
-A terminal demo asset is not yet available; the workflow is documented in the
-repository and ECHO Protocol.
-
-The red phase finds it. The green phase fixes it. The audit phase verifies it.
-```
-
-Tweet 4/5 — competition
-
-```text
-vs. Copilot/Cursor: we audit before writing, not after.
-vs. Cline: BYOK and local Ollama by default, no plaintext key storage.
-vs. OpenHands: no Docker required, single-command install.
-
-Built with TypeScript, Bun, React, and the ECHO Protocol.
-Apache-2.0 licensed.
-```
-
-Tweet 5/5 — call to action
-
-```text
-We are stress-testing the local Ollama integration before a wider launch.
-
-If you have a real codebase and 5 minutes, try it and break it.
-
-GitHub: <https://github.com/savant0x/savant-code>
-Discord: not yet available; please use GitHub Issues for feedback.
-
-Issues with repros are love letters.
-```

package/docs/privacy.md

@@ -1,186 +0,0 @@
-# Savant Code Privacy Architecture
-
-> **Scope:** This document describes what data Savant Code collects, where it
-> goes, and how users control it. It covers the CLI, SDK, and agent-runtime.
->
-> **Version:** v0.0.18
->
-> **Applies to:** `cli`, `sdk`, `packages/agent-runtime`, `common`
-
-## 1. Core Privacy Principles
-
-1. **Local-first by default.** Code, prompts, and file contents stay on the
-   user's machine unless the user explicitly chooses a cloud provider.
-2. **Bring Your Own Key (BYOK).** The user supplies their own API keys. Savant
-   Code does not vend, own, or proxy inference credentials.
-3. **Telemetry is user-controlled.** Remote analytics and error reporting
-   default to enabled for new users, can be disabled at any time with `/telemetry disable`, and are separate from ad
-   consent.
-4. **Controlled credential storage in the BYOK flow.** Provider API keys may be read from environment variables or
-   secure masked user input. Explicit environment variables take precedence; keys entered through `/provider` are
-   stored locally in the user's `credentials.json` with the existing backend credential profile preserved.
-5. **Transparent network calls.** Every network request the CLI makes is
-   documented below.
-
-## 2. Data Boundaries
-
-### What Stays on the Machine
-
-- Source code and project files
-- User prompts and chat history
-- Model outputs generated through local or BYOK providers
-- Settings (`~/.savant-code[-env]/settings.json`) and locally stored provider credentials (`credentials.json`)
-- Sandbox permission mode and safety registry state
-
-### What May Leave the Machine
-
-- **Inference requests** to the user's chosen provider (OpenRouter, Ollama,
-  gateway, etc.) when using BYOK mode.
-- **Authentication requests** to Savant Code backend when the user logs in.
-- **Telemetry events** while remote analytics is enabled; users can disable them
-  with `/telemetry disable`.
-- **Ad impression/click events** while ads are enabled; ad consent remains
-  separate from telemetry.
-- **Update-check requests** to determine whether a newer CLI version is
-  available.
-- **Error/crash logs** when the user explicitly opts into full telemetry.
-
-## 3. Credential Storage
-
-### BYOK API Keys
-
-- Environment-provided BYOK keys are read at runtime and are never copied to
-  disk by Savant Code. Current direct-provider variables include
-  `OPENCODE_GO_API_KEY`, `TOKENROUTER_API_KEY`, `NVIDIA_API_KEY`, and
-  `COMMAND_CODE_API_KEY`.
-- Keys entered through the masked `/provider` flow are stored in the user's
-  local `credentials.json` so npm-installed users can configure a provider
-  without editing shell profiles.
-- Explicit environment variables take precedence over stored provider keys.
-- Provider keys are excluded from chat history and rendered user messages.
-- Users can remove stored credentials with `/logout` or by deleting the local
-  credentials file.
-
-### Backend Auth Token
-
-- When a user logs in to the Savant Code backend, an `authToken` is received.
-- The token is stored in `~/.savant-code[-env]/credentials.json` for session
-  continuity (dev: `.savant-code-dev`, test: `.savant-code-test`, prod:
-  `.savant-code`).
-- This file is created with the user's OS permissions and is not shared or
-  transmitted.
-- Users can clear this file at any time with the `/logout` command.
-  - **Future work (separate FID):** Evaluate migrating the backend auth token to the OS keychain or encrypted local
-  storage. For launch, plaintext credential storage is limited to the backend session token and provider keys
-  explicitly entered through `/provider`; environment-provided keys are not copied to disk.
-
-## 4. Network Calls
-
-### Update Check
-
-- **Endpoint:** Public registry/GitHub releases API.
-- **Data sent:** Current CLI version and user-agent string only.
-- **Data NOT sent:** Code, prompts, file names, or identifiers.
-- **Trigger:** On CLI startup (when enabled) or when the user runs
-  `/version --check`.
-
-### Provider Inference (BYOK)
-
-- **Endpoint:** Chosen by the user (OpenRouter, Ollama, etc.).
-- **Data sent:** Model prompt and any context required by the model.
-- **Data NOT sent:** Savant Code does not add tracking headers or identifiers.
-
-### Analytics / Telemetry
-
-- **Destination:** PostHog and/or Axiom (when enabled).
-- **Data sent:** Aggregated event types, error names, and feature usage
-  summaries.
-- **Secret redaction:** All payloads are run through `sanitizeSecrets` before
-  transmission.
-- **Default:** `true` for new users and legacy settings without an explicit
-  value; users can disable remote analytics with `/telemetry disable`.
-
-### Ads (SavantFree Tier)
-
-- **Destination:** Carbon Ads or Gravity Index network.
-- **Data sent:** Minimal impression/click events required for ad serving.
-- **Default:** `false` for paid tiers; free tier may prompt for opt-in on first
-  use.
-
-## 5. Telemetry & Ad Controls
-
-<!-- markdownlint-disable MD013 -->
-
-| Setting            | Location                             | Default                                                  | How to Change                                                     |
-| ------------------ | ------------------------------------ | -------------------------------------------------------- | ----------------------------------------------------------------- |
-| `adsEnabled`       | `~/.savant-code[-env]/settings.json` | `false`                                                  | `/ads enable` / `/ads disable`                                    |
-| `analyticsEnabled` | `~/.savant-code[-env]/settings.json` | `true` for new/legacy settings without an explicit value | `/telemetry status`, `/telemetry enable`, `/telemetry disable`    |
-| `permissionMode`   | `~/.savant-code[-env]/settings.json` | `prompt`                                                 | `/permissions safe`, `/permissions prompt`, `/permissions unsafe` |
-
-<!-- markdownlint-enable MD013 -->
-
-## 6. Secret Sanitization
-
-All debug logs, analytics payloads, and error reports are processed through a
-`sanitizeSecrets` helper. The following keys are redacted to `[REDACTED]`:
-
-- `authToken`
-- `apiKey` / `api_key`
-- `token`
-- `accessToken`
-- `refreshToken`
-- `secret`
-- `password`
-- `authorization`
-
-## 7. Retention
-
-- **Local chat logs:** Stored in the per-project `debug/` directory or per-chat
-  log file. Cleared when the user runs `/clear` or deletes the files.
-- **Settings:** Retained until the user deletes `~/.savant-code[-env]/`.
-- **Backend sessions:** Governed by the Savant Code backend terms of service.
-- **Telemetry/analytics:** Governed by PostHog/Axiom retention policies once
-  enabled.
-
-## 8. User Controls
-
-- `/permissions safe|prompt|unsafe` — control how aggressively the agent can run
-  shell/network tools.
-- `/ads enable` / `/ads disable` — toggle ad display independently.
-- `/telemetry status|enable|disable` — inspect or change remote analytics and
-  error-reporting consent.
-- `/logout` — remove stored backend credentials.
-
-Remote analytics consent is stored separately from `adsEnabled`; disabling
-telemetry does not disable local debug logs or inference-provider network
-requests.
-
-### Config directory
-
-The runtime stores local config in `~/.savant-code[-env]/` (e.g.
-`.savant-code-dev`, `.savant-code-test`, `.savant-code`). This unifies the CLI
-and SDK path and matches the public launch brand. Existing
-`~/.config/manicode[-env]/` data is not migrated; users will need to
-re-authenticate after updating.
-
-### Contributor / test override
-
-Tests and advanced users can set `SAVANT_CODE_CONFIG_DIR` (non-production only)
-to override the config directory. This is used by
-`cli/src/utils/__tests__/settings.test.ts` to isolate on-disk state.
-
-## 9. Verification
-
-- Verify that environment-provided keys are not persisted, while `/provider`
-  storage is limited to the local credentials file and excludes chat history.
-- Run the settings and analytics-client tests in `cli/src/utils/__tests__/` to
-  confirm the active default, explicit disable behavior, and secret redaction.
-- Inspect `~/.savant-code[-env]/settings.json` after a fresh install —
-  `analyticsEnabled` is `true` and `adsEnabled` is `false` unless changed by the
-  user.
-
-## 10. Open Questions / Future Work
-
-- Evaluate OS-native keychain/keyring integration for the backend `authToken`.
-- Add per-provider audit trail for network calls in `unsafe` mode.
-- Provide a one-click "purge all local data" command.

package/docs/reports/Savant-Code Benchmark Specification.md

@@ -1,206 +0,0 @@
-# **Deep Research Brief: A World-Class Benchmark Harness for Savant-Code**
-
-## **Strategic Benchmarking Philosophy**
-
-The rapid evolution of autonomous coding assistants has exposed severe limitations in traditional software engineering benchmarking frameworks. Historically, the industry has relied on output-centric evaluations, evaluating systems predominantly on a terminal state: the successful application of a final patch to resolve a given issue, evaluated via fail-to-pass unit tests1. While this diff-comparison approach provides a baseline for raw code generation quality, it is fundamentally inadequate for evaluating a highly orchestrated, deterministic, multi-agent runtime like Savant-Code.  
-A multi-agent, Finite State Machine (FSM)-aware benchmark is an absolute necessity because the architectural value of Savant-Code resides in its internal execution process, not solely its final output. The ECHO Protocol v0.2.0 enforces a strict lifecycle governing state transitions. The system must navigate through an analytical RED phase, transition into an execution-focused GREEN phase, verify outputs in an AUDIT phase, and, if necessary, initiate a SELF-CORRECT phase before reaching a COMPLETE state. This FSM governs which of the nine canonical agents—including the Orchestrator, Detective, Forge, Verifier, Recorder, Thinker, Scout, Researcher, and Scribe—execute at any given moment. Crucially, the protocol restricts tool usage based on the active phase; for example, write operations such as write\_file or apply\_patch are strictly prohibited during the RED phase, while terminal commands are gated to the AUDIT and GREEN phases.  
-Evaluating Savant-Code exclusively on final code correctness is akin to grading a surgical team solely on patient survival, ignoring whether sterile procedures were followed, whether the correct specialist performed the incision, or whether vital signs were monitored throughout the procedure. A world-class benchmark must measure orchestration efficiency, strict role separation, tool-permission compliance, and state-transition integrity. This requires capturing the full trajectory of the agent, analyzing how tasks are delegated, and verifying that the internal architecture operates as designed.  
-To achieve this, the benchmark design must strike a deliberate balance between real-world software engineering tasks and highly targeted synthetic tasks. Real-world tasks, analogous to those found in SWE-bench Verified, demonstrate applied utility in complex, massive codebases3. These tasks prove that the system can navigate genuine enterprise challenges. Conversely, synthetic tasks are designed to systematically exercise specific ECHO constraints and agent capabilities. For instance, a synthetic task might force an environmental error state to guarantee the activation of the Verifier agent, or provide a highly specific, custom-defined tool to test whether the Orchestrator accurately delegates the workload to the Researcher. Deterministic verification—comprising compilation, strict type-checking, and isolated test execution—remains the primary signal of success. This ensures that the evaluation relies on objective compiler states and system exit codes rather than the variable, non-deterministic output of Large Language Model (LLM) judges.
-
-## **Task Taxonomy and Concrete Execution Scenarios**
-
-To robustly measure the unique capabilities of the Savant-Code architecture, the evaluation suite relies on a formalized task taxonomy. Each category isolates specific architectural behaviors, ensuring that all nine canonical agents and the FSM state transitions are exhaustively tested. The taxonomy covers pure algorithmic problem solving, complex multi-agent orchestration, protocol compliance, tool integration, and programmatic determinism.
-
-| Category | Concrete Prompt | Expected Agent Workflow & FSM Transitions | Deterministic Verification & Pass Criteria |
-| :---- | :---- | :---- | :---- |
-| **1\. Pure Coding / Bug Fix** | "Fix the off-by-one error in TreeTraversal.ts where the final leaf node is skipped during depth-first search." | **Workflow:** Orchestrator delegates to Detective (locates file), then to Forge (modifies loop logic). **FSM:** RED \-\> GREEN \-\> AUDIT \-\> COMPLETE. | **Command:** bun test tree.test.ts **Pass:** Exit code 0\. Fail-to-pass tests resolve successfully. |
-| **2\. Multi-Agent Orchestration** | "Analyze the deprecated database schema, write a migration script, and verify it against the local test database." | **Workflow:** Orchestrator \-\> Scout (schema analysis) \-\> Forge (script generation) \-\> Verifier (test execution). **FSM:** RED \-\> GREEN \-\> AUDIT \-\> COMPLETE. | **Command:** psql \-f migration.sql && pg\_dump **Pass:** Zero schema diffs against golden schema validation file. |
-| **3\. ECHO FSM Compliance** | "Refactor auth.ts to use JWTs. *Hidden constraint: Harness intercepts and simulates a test failure on first attempt.*" | **Workflow:** Agent must transition from AUDIT to SELF-CORRECT upon detecting the simulated failure, before returning to GREEN. **FSM:** RED \-\> GREEN \-\> AUDIT \-\> SELF-CORRECT \-\> GREEN \-\> AUDIT \-\> COMPLETE. | **Command:** Harness trace parsing. **Pass:** FSM transition log confirms the explicit AUDIT \-\> SELF-CORRECT path. Write tools must not execute in RED phase. |
-| **4\. Custom Tool Execution** | "Use the provided generate\_ast\_graph tool to map core/ and output the dependencies to deps.json." | **Workflow:** Orchestrator \-\> Researcher (invokes the custom generate\_ast\_graph tool). **FSM:** RED \-\> GREEN \-\> COMPLETE. | **Command:** jq '.nodes' deps.json **Pass:** JSON output exactly matches the expected AST topology. |
-| **5\. MCP Tool Execution** | "Query the internal mock-jira MCP server for ticket ENG-102 and implement the specified CSS changes." | **Workflow:** Orchestrator queries MCP tool \-\> Forge edits CSS \-\> Verifier checks styles. **FSM:** RED \-\> GREEN \-\> AUDIT \-\> COMPLETE. | **Command:** mock-mcp-verify ENG-102 **Pass:** MCP server state indicates ticket transitioned to "Resolved"; CSS compiles without errors4. |
-| **6\. Skill-Driven Task** | "Implement the new payment gateway. Ensure strict adherence to the rules specified in SKILL.md." | **Workflow:** Scribe reads SKILL.md \-\> Orchestrator adopts singleton patterns specified in the file \-\> Forge writes code. | **Command:** eslint \--rule 'no-multiple-instances: error' **Pass:** Code compiles and strictly follows the abstract syntax tree pattern defined in the skill file. |
-| **7\. Programmatic Agent** | "Execute the BatchProcessor programmatic agent to refactor all 50 text files in the /data directory." | **Workflow:** Agent executes a handleSteps generator yielding STEP\_ALL and GENERATE\_N directives sequentially. | **Command:** find /data \-type f | xargs grep \-L 'old\_string' **Pass:** Generator exhausts completely without infinite loops; exactly 50 files are verified as modified. |
-| **8\. Slash Command / CLI** | "/goal Update the global logging format. /verify run npm run lint." | **Workflow:** CLI interpreter sets global context. Orchestrator modifies files, CLI loop intercepts the /verify hook. | **Command:** Harness state inspection. **Pass:** Output artifacts show updated logs, trace shows /verify hook executed successfully natively. |
-| **9\. Context-Heavy Refactor** | "Rename the UserModel to ClientEntity across the entire src/ directory (45 files)." | **Workflow:** Detective utilizes code\_search and read\_files across directories \-\> Forge issues multiple str\_replace tools. | **Command:** tsc \--noEmit && bun test **Pass:** Type-checker passes with 0 errors; no orphaned UserModel imports remain in the project. |
-| **10\. Error Recovery** | "Implement the sorting algorithm. *Hidden constraint: Harness injects a faulty type definition in the environment.*" | **Workflow:** Forge writes valid code \-\> Verifier detects tsc failure \-\> Thinker analyzes error \-\> Forge applies workaround. | **Command:** tsc **Pass:** The agent detects the environmental error in the AUDIT phase, adapts the implementation, and successfully compiles. |
-
-## **Dataset Construction and Curation**
-
-The integrity and long-term validity of the evaluation suite rely entirely on eliminating data contamination. Public datasets frequently suffer from training data leakage because the open-source GitHub issues, pull requests, and discussions used to construct the tests were previously ingested by the massive web-scraping pipelines that train frontier models6. If an underlying language model has memorized the exact patch for a specific public issue, the benchmark ceases to measure the autonomous reasoning capabilities of the agent and instead measures mere recall. This artificially inflates scores and renders the evaluation useless for comparing architectural agent improvements.  
-To construct a resilient, leak-proof dataset, the harness requires three distinct sourcing and curation pipelines. The first pipeline focuses on post-cutoff curation. This involves scraping closed pull requests and complex issues strictly from repositories created after the known training cut-off dates of major LLMs, such as the latest iterations of Claude 3.5 Sonnet or GPT-4o. The second pipeline utilizes execution-grounded synthesis. Automated frameworks are employed to generate synthetic, highly complex codebases featuring paired bugs that have never existed on the public internet, thereby guaranteeing a zero-shot evaluation scenario7. The third pipeline applies advanced Abstract Syntax Tree (AST) obfuscation. High-quality existing issues are programmatically refactored to rename variables, restructure directory hierarchies, and alter domain contexts. The core algorithmic challenge remains completely intact, but the lexical triggers that might prompt memorized recall are systematically neutralized.  
-Every task within the dataset is standardized using a strict JSON/YAML schema that integrates directly into the harness architecture. This schema provides all necessary initialization parameters, required environmental contexts, and validation mechanisms needed by the benchmarking engine.
-
-YAML  
-schema\_version: "2.0"  
-task\_id: "savant-core-auth-001"  
-category: "mcp\_tool\_execution"  
-metadata:  
-  difficulty: "hard"  
-  expected\_fsm\_phases: \["RED", "GREEN", "AUDIT", "COMPLETE"\]  
-  required\_agents: \["Orchestrator", "Forge", "Verifier"\]  
-environment:  
-  base\_image: "savant-eval-base:1.2"  
-  setup\_script: "npm ci && npm run db:migrate"  
-  mcp\_servers:  
-    \- name: "mock-jira"  
-      endpoint: "http://localhost:8080/mcp"  
-inputs:  
-  prompt: "Query the internal mock-jira MCP server for ticket ENG-102 and implement the specified CSS changes."  
-validation:  
-  timeout\_seconds: 300  
-  deterministic\_checks:  
-    \- command: "npm run build:css"  
-      expected\_exit\_code: 0  
-    \- command: "node scripts/verify\_mcp\_state.js ENG-102"  
-      expected\_exit\_code: 0  
-  fsm\_assertions:  
-    strict\_phase\_order: true  
-    allow\_write\_in\_red: false
-
-Accepting new tasks into the registry requires a rigorous reproducibility process. A task is strictly rejected if it exhibits non-deterministic behavior during validation. Before merging a new task definition into the repository, a baseline agent must execute the setup\_script, apply a golden patch, and pass all deterministic\_checks consistently over five independent executions. This multi-trial validation strategy ensures that flaky infrastructure or poorly written tests do not poison the leaderboard.
-
-## **Harness Architecture Pipeline**
-
-The benchmarking harness is designed as a highly concurrent, fault-tolerant pipeline optimized for headless execution and rich, persistent telemetry collection. To ensure scalability and isolation, the architecture is segregated into seven distinct, decoupled stages that process tasks asynchronously.  
-The pipeline begins with the Task Registry, which ingests the YAML task definitions from the filesystem, validates the schema constraints, and queues instances for execution based on the available compute resources of the host machine. The Sandbox Factory then provisions hermetic, isolated environments for each queued task, handling the mounting of dependencies, the initialization of network mocks, and the establishment of the baseline filesystem state.  
-Once the environment is active, the Agent Runner takes over as the primary execution engine. It interfaces dynamically with the SavantCodeClient, injects the parsed prompt, exposes the necessary custom tool definitions, and manages the execution via subprocesses. Concurrently, the Trace Collector actively intercepts all runtime events emitted by the agent. It records every subagent spawn, FSM phase transition, tool invocation, payload size, and token consumption metric, writing these details into an SQLite-backed structured trace log in real-time8.  
-Following the completion or termination of the agent's execution, the Deterministic Evaluator takes control of the sandbox. It executes the task's predefined validation scripts within the isolated environment, meticulously capturing standard output (stdout), standard error (stderr), and system exit codes to verify functional correctness. The Metric Aggregator then processes the outputs from both the Evaluator and the Trace Collector, computing composite scores that combine deterministic pass rates with FSM compliance and orchestration efficiency metrics. Finally, the Report Generator outputs the ultimate evaluation artifacts, synthesizing the data into comprehensive markdown summaries, JSON-Lines (JSONL) raw traces for external parsing, and side-by-side terminal diff comparisons for manual review8.  
-Concurrency across this pipeline is managed via an asynchronous worker pool. The maximum parallel execution limit is strictly defined by the available memory footprint constraints of the host machine and the isolated sandboxes. Timeouts are rigorously enforced at both the global task level and the individual tool-call level. The system utilizes exponential backoff strategies to handle expected API rate limits, ensuring that transient HTTP 429 errors from underlying language models do not artificially induce benchmark failures.
-
-## **Sandboxing and Environment Control**
-
-The validity of a coding benchmark relies entirely on the strict isolation and reproducibility of its execution environment. Autonomous agents write files, install untrusted dependencies, execute arbitrary shell commands, and continuously mutate the system state. Allowing these operations to run natively on the host machine or in weakly isolated environments introduces catastrophic security vulnerabilities and non-deterministic evaluation outcomes.  
-Selecting the appropriate isolation layer is the most critical infrastructural decision for the harness. Docker relies on Linux control groups (cgroups) and namespaces, which provide fast cold starts but suffer from weak security boundaries because all containers share the host operating system's kernel9. A kernel-level exploit generated by an untrusted agent could breach the sandbox. Furthermore, Docker lacks native, performant mechanisms for sub-second memory state freezing and restoration. Alternatively, gVisor employs a user-space kernel to intercept system calls, drastically enhancing security by providing a robust boundary between the container and the host9. However, the overhead of system call interception incurs significant performance penalties, often slowing down I/O-heavy workloads—such as dependency installation and language compilation—by up to 30 percent, rendering it suboptimal for high-throughput testing9.  
-For the Savant-Code benchmark harness, Firecracker microVMs are the mandated isolation technology. Developed for serverless platforms, Firecracker utilizes the Kernel-based Virtual Machine (KVM) to provide hardware-enforced isolation, booting entirely dedicated kernels for each workload in approximately 125 milliseconds9. Most importantly, Firecracker supports advanced snapshot-restore capabilities, similar to mechanisms deployed in specialized agent testing environments9. Multi-turn agent tasks require persistent filesystem and memory states across dozens of sequential tool calls. Initializing a fresh Docker container for every single evaluation requires rebuilding the environment continuously. Using Firecracker, a base microVM is booted, dependencies are installed, and language servers are initialized. A snapshot of the memory and the block device state is then saved to local NVMe storage. When an agent is evaluated, the sandbox is restored from this exact snapshot in 5 to 30 milliseconds13. This mechanism allows the harness to process hundreds of task variations with near-zero cold-start latency, leveraging Copy-on-Write (CoW) page faults to drastically optimize memory consumption14.  
-Beyond compute isolation, environment mutability and network access must be strictly controlled. Unrestricted network access introduces non-determinism, such as tests failing due to a temporary NPM registry outage, and allows agents to circumvent problem-solving by searching the internet for the exact solution. To prevent this, package registries like NPM and PyPI are proxied locally within the host machine or pre-cached within the base snapshot, guaranteeing that dependency resolution is instantaneous and entirely deterministic. External API resources are mocked using localized Model Context Protocol (MCP) validation tools. The harness pairs a WebSocket bridge with mock MCP servers that simulate authentication flows, database states, and error conditions, ensuring the agent interacts with a highly predictable schema4.  
-The harness captures the entire filesystem state before the agent initiates execution, performing a cryptographic hash of the working directory. Post-execution, a delta diff is generated to identify exactly which bytes were modified. Sensitive information, such as API keys and authentication tokens, are injected via secure environmental variables directly into the microVM's isolated memory space and are explicitly masked in the Trace Collector logs to prevent leakage in the generated reports.
-
-## **Runner Abstraction**
-
-To ensure that Savant-Code can be directly compared against other leading autonomous engineering tools, such as Claude Code, Codex, and OpenCode, the harness employs a polymorphic Runner interface. This abstraction layer acts as the universal adapter between the standardized benchmark pipeline and the specific execution dynamics of the agent under test.  
-The architecture defines a strict TypeScript interface that all specific runner implementations must satisfy. This guarantees that the benchmark treats every agent uniformly, injecting the prompt, enforcing boundaries, and collecting terminal states through identical mechanisms.
-
-TypeScript  
-export interface AgentRunner {  
-  // Initializes the agent with task parameters and environment bindings  
-  initialize(config: RunnerConfig): Promise\<void\>;  
-    
-  // Executes the prompt and captures the asynchronous stream  
-  executePrompt(prompt: string): Promise\<RunState\>;  
-    
-  // Collects comprehensive telemetry post-execution  
-  collectTrace(): TraceDocument;  
-    
-  // Handles simulated user interaction if the agent requests input  
-  handleInteractivePrompt(request: string): Promise\<string\>;  
-}
-
-For the evaluation of Savant-Code, the implementation natively consumes the @savant-code/sdk. The SavantCodeClient.run() function is invoked directly within the node process. The runner leverages the handleEvent callback to stream realtime FSM transitions, tool payloads, and context token counts directly into the TraceDocument. Because the SDK provides deep introspection, the runner can capture the internal state of the ECHO FSM continuously without relying on output parsing.  
-When evaluating external, closed ecosystem agents like Claude Code, the runner operates by spawning an interactive child process. Given the runtime architecture of the testing framework, Bun.spawn is heavily utilized to manage these external binaries. The runner streams the standard output of the external agent using stdout: "pipe" and intercepts the output via a ReadableStream implementation18. The runner analyzes this asynchronous text buffer using complex regular expressions to detect tool usage signatures, phase changes, and prompts for user interaction, thereby normalizing external agent behaviors into a comparable telemetry format. Global timeouts and limits on maxAgentSteps are strictly enforced at the runner layer. By passing an AbortSignal or invoking the kill() method on the subprocess, the runner can immediately send a SIGTERM to the process if execution bounds are exceeded, preventing runaway resource consumption19.
-
-## **Deterministic Verification Layer**
-
-The deterministic verification layer provides the indisputable, primary signal of agent performance. Operating sequentially inside the post-execution sandbox, it removes subjectivity from the evaluation of functional correctness.  
-The verification process executes a strict sequence of validation steps. First, if the agent output is provided as a unified diff rather than through direct file manipulation tools, the harness attempts to apply the patch using system tools like git apply. Any failure at this stage yields an immediate syntax-resolution error. Second, the project is subjected to a build step. The harness executes compilation commands relevant to the environment, such as npm run build or tsc. An exit code of 0 is strictly required to proceed, and all standard error (stderr) outputs are captured for the trace log.  
-Following successful compilation, the verification layer enforces static type-checking. This ensures the agent did not resort to bypass techniques, such as applying any casting or ignoring architectural interfaces, to force a superficial solution. Once type safety is confirmed, the task's specific test suite is executed. Whether the environment uses jest, pytest, or native binary testing frameworks, the harness expects a 0 exit code, confirming that the algorithmic logic satisfies the problem statement. The code is then evaluated against linting and formatting rules to ensure the patch integrates seamlessly with the repository's stylistic standards. Finally, custom bash validators can be executed to verify complex state changes, such as checking that an MCP mock server properly transitioned a ticket status.  
-To combat the inherent non-determinism present in complex testing environments, the verification layer implements robust handling for flaky tests. Any failing test suite is automatically re-run up to three consecutive times within an isolated, reset state. If the test passes on a subsequent run, it is flagged with a FLAKY\_PASS marker, allowing the metric aggregator to acknowledge the success while noting the instability. Test outputs are ingested via structured reporting formats, such as the JSON Test Reporting Format, allowing the harness to precisely surface which specific assertion failed and mapping it directly to the agent's modification trace21.
-
-## **Multi-Agent and ECHO-Specific Metrics**
-
-Traditional pass/fail resolution rates are entirely insufficient to evaluate the sophisticated architecture of the ECHO Protocol. The harness introduces novel quantitative metrics specifically engineered to evaluate the internal health and efficiency of the multi-agent system.  
-The most critical unique metric is FSM Strict Compliance. This evaluates the trajectory of the agent through the RED, GREEN, AUDIT, and SELF-CORRECT lifecycle. A boolean matrix tracks every action against the current phase. A perfect score is awarded only if the agent strictly adhered to phase constraints, meaning no write tool invocations occurred during the analytical RED phase, and test commands were strictly reserved for the AUDIT phase. Any violation triggers a severe penalty in the compliance score.  
-The Subagent Utilization Ratio measures the diversity of agent orchestration. If a highly complex task requires the Orchestrator to logically spawn the Scout, Forge, and Verifier, but the Orchestrator instead attempts a monolithic, error-prone change using only the Forge, the utilization ratio drops significantly, highlighting a failure in delegation strategy.  
-Detective Precision and Recall specifically evaluate the efficiency of codebase exploration. Recall is calculated as the ratio of relevant files successfully identified by the agent out of the total files required to generate the golden patch. Precision is calculated as the ratio of relevant files identified out of the total files read or searched by the agent during the task. This mathematical approach heavily penalizes agents that inefficiently dump entire directories into their context window, proving that the system is targeted in its research.  
-Forge Minimality calculates the Levenshtein distance between the agent's proposed edits and the canonical golden patch. It penalizes unnecessary refactoring, formatting changes, or hallucinated additions that fall outside the strict scope of the prompt. The Verifier Impact metric acts as a boolean flag indicating whether the Verifier successfully detected a failing test suite in the AUDIT phase and successfully initiated a SELF-CORRECT loop that ultimately drove the system back to a GREEN phase and a COMPLETE state. Recorder Compliance validates that File Identity Document (FID) lifecycles were correctly instantiated, updated, or archived following file modifications. Tool Permission Respect tracks instances where any subagent attempted to execute a tool outside its explicitly assigned capability array, providing a strict measure of system prompt adherence.
-
-## **Subjective and LLM Scoring**
-
-While deterministic checks absolutely govern the evaluation of functional correctness, LLM judging is selectively applied to evaluate qualitative attributes that compilers cannot mathematically assess. To minimize exorbitant API costs and the inherent variance of AI judges, LLM scoring is restricted exclusively to tasks where qualitative outputs are the primary deliverable, such as architectural documentation generation, code maintainability reviews, or human-facing instruction adherence22.  
-LLM judging is strictly excluded from verifying whether code compiles, whether tests pass, or whether a diff matches a functional baseline. Instead of asking a judge the overly broad question, "Is this diff correct?", the harness utilizes structured output evaluation evaluated against a highly specific rubric by a frontier model acting as a judge.  
-The prompt provided to the LLM judge is deliberately lightweight and highly constrained: You are an expert software architect evaluating a codebase modification. Do NOT evaluate whether the code compiles or passes tests; this has already been verified deterministically. Evaluate the code strictly on the following three dimensions using a 1-5 scale:
-
-> 1. Maintainability: Does the new code introduce technical debt, magic numbers, or excessive cyclomatic complexity?  
-> 2. Instruction Adherence: Did the implementation adhere strictly to the architectural constraints requested in the prompt?  
-> 3. Explanation Quality: If the agent provided a summary to the user, is it concise, accurate, and free of hallucinations? Return your evaluation exactly adhering to the provided JSON schema. Do not include markdown explanations outside the JSON object. By enforcing a structured JSON output schema, the variance of the judge is dramatically reduced. The scores can be programmatically parsed by the Metric Aggregator, completely avoiding the fragility of text-based parsing and ensuring that the subjective assessment maps directly to the quantitative reporting pipeline8.
-
-## **Cost, Latency, and Token Tracking**
-
-Tracking deep telemetry is essential for measuring the operational viability and commercial practicality of Savant-Code. The AgentState object embedded in the runtime natively provides the necessary instrumentation, and the harness extracts, normalizes, and contextualizes this data into actionable performance metrics.  
-The Trace Collector intercepts the sessionState.mainAgentState payload at every tick of the FSM to continuously extract creditsUsed, directCreditsUsed, and contextTokenCount. This telemetry is split granularly into input tokens, output tokens, and caching tokens. By mapping this real-time consumption data against pinned provider pricing tables, the harness calculates the exact fractional inference cost per task24. This allows the platform to compute the most critical enterprise metric: the exact cost in dollars per successfully resolved issue.  
-Latency profiling is equally rigorous. Wall-clock execution time is measured not just globally, but per subagent invocation. By parsing the structured trace logs, the harness calculates the average time spent analyzing code in the Detective phase versus the time spent generating code in the Forge phase. This granular timing highlights execution bottlenecks and API latency delays.  
-Crucially, this deep telemetry enables advanced regression detection. If a new version update to Savant-Code slightly improves pass rates by two percent but simultaneously increases context token consumption by forty percent due to inefficient, runaway subagent looping, the Metric Aggregator will explicitly flag a cost regression. This ensures that improvements in capability do not secretly compromise the economic viability of the tool.
-
-## **Anti-Cheating and Robustness**
-
-To maintain the absolute validity of the benchmark over successive product iterations, the harness must rigorously defend against both intentional dataset circumvention and systemic infrastructural instability.  
-To prevent data contamination, all newly authored tasks embed a unique cryptographic canary string within the repository files. This strategy is analogous to the mechanisms used to secure the Big-Bench evaluation suites21. If this specific, randomized string ever appears in the output of future language models attempting the task, training data leakage is immediately confirmed, and the task can be safely deprecated from the leaderboard.  
-Fault isolation is guaranteed by the hermetic nature of the sandboxing environment. A fatal crash, an infinite loop, or a massive memory leak triggered by one agent process is completely isolated by the Firecracker hypervisor, preventing it from corrupting the host machine. The Agent Runner continuously monitors process health; if a subprocess hangs and fails to yield output, a SIGKILL is issued, the trace is saved as a TIMEOUT\_FAIL, and the harness cleanly proceeds to the next task20.  
-API resilience is built directly into the execution layer. Interactions with upstream LLM providers are wrapped in a resilient network protocol implementing exponential backoff and randomized jitter. If an API endpoint returns a rate limit error or a bad gateway response, the harness intercepts the error, pauses the FSM execution tick without failing the task, and retries the connection. This ensures that transient external API instability does not falsely register as an agent logic failure. Furthermore, because the Firecracker microVMs are systematically destroyed and instantly restored from pristine baseline snapshots post-execution, there is zero risk of filesystem residue, temporary files, or mutated network states persisting between tasks and influencing subsequent evaluations14.
-
-## **CLI and SDK Integration**
-
-The benchmark suite is meticulously designed as a standalone, highly portable utility that seamlessly integrates with Savant-Code without requiring invasive modifications to the core runtime codebase.  
-The harness is invoked via a standard, intuitive command-line interface command, enabling engineers to run comprehensive evaluations directly from their terminal:savant-code benchmark run \--taskset evals/v2/suite.yaml \--concurrency 4 \--report json  
-When executed, the CLI operates in a headless SDK mode, bypassing all interactive terminal UI elements and instantiating the SavantCodeClient directly as a library component. To capture the necessary deep traces, the harness injects a custom handleEvent middleware hook into the client initialization parameters. As the Orchestrator yields internal runtime events—such as STEP, TOOL\_CALL, or PHASE\_TRANSITION—the middleware synchronously serializes the payload into the SQLite trace database before seamlessly passing the event back to the standard runtime flow.  
-When evaluating external agents for direct comparison, the CLI executes tools like Claude Code or Codex as isolated subprocesses utilizing the Bun.spawn utility19. It programmatically injects simulated user input through the standard input (stdin) stream, echoing affirmative responses to confirmation prompts or providing required context, and parses the standard output (stdout) to determine when the agent believes the task is complete. This architecture maintains a highly uniform, unbiased evaluation surface across vastly disparate autonomous tools22.
-
-## **Implementation Roadmap**
-
-The development and deployment of the v2 benchmark harness are structured across a rigorous eight-week implementation roadmap, translating the architectural design into actionable engineering phases.
-
-| Week | Phase | Key Deliverables |
-| :---- | :---- | :---- |
-| **Week 1** | Core Architecture & Schema Design | Finalize JSON/YAML task schemas. Build the core Task Registry logic and define the strict AgentRunner TypeScript interfaces. |
-| **Week 2** | Runner & SDK Integration | Implement the SavantCodeClient wrapper. Build the handleEvent interceptors to accurately stream state data to the SQLite Trace Collector. |
-| **Week 3** | Sandboxing Infrastructure | Integrate Firecracker microVM management9. Implement the low-level CRIU snapshot-restore logic for sub-30ms environment resets13. |
-| **Week 4** | Deterministic Verification Layer | Build the shell execution parsers for build, test, and type-check steps. Implement flaky test detection and the three-attempt retry logic. |
-| **Week 5** | Dataset Migration & Creation | Translate legacy benchmark tasks into the new v2 schema. Author 20 highly specific synthetic tasks targeting FSM compliance and MCP tool usage. |
-| **Week 6** | Advanced Metrics & LLM Judging | Implement FSM transition tracking algorithms and mathematical scoring models. Integrate the structured-output LLM judge for qualitative checks. |
-| **Week 7** | External Agent Adapters | Build subprocess wrappers using Bun.spawn for Claude Code, Codex, and OpenCode19. Calibrate timeout and regex parsing logic. |
-| **Week 8** | CLI Integration & Pipeline | Finalize the savant-code benchmark run command. Generate automated markdown leaderboards. Deploy the complete harness into automated CI/CD pipelines. |
-
-## **Comparison to SWE-bench Verified and Aider**
-
-This comprehensive harness introduces evaluation paradigms that are fundamentally absent in current industry standards, providing a much deeper assessment of agent capabilities.  
-Compared to SWE-bench and SWE-bench Verified, which operate entirely on a single-shot terminal diff validation, this harness evaluates the entire lifecycle of problem resolution1. SWE-bench has no awareness of how a solution was generated; a model that correctly guesses a patch is scored identically to a model that methodically explores a codebase and builds a robust solution. The Savant-Code harness validates the internal architecture—ensuring proper subagent delegation, correct FSM transitions, and appropriate tool usage—providing an architectural health check rather than just a code check.  
-Compared to Aider Polyglot and Exercism benchmarks, which effectively measure cross-language problem-solving and single-file algorithmic logic, this harness expands the scope to true enterprise engineering28. While Polyglot is excellent for testing multi-language syntax, it lacks deep environmental validation, MCP server testing, and multi-agent topology assertions. The Savant-Code harness introduces highly complex enterprise scenarios involving mock APIs, Jira servers, database migrations, and programmatic tool execution that reflect the reality of modern, real-world agentic workflows4.
-
-## **Deliverable Format and Directory Structure**
-
-The benchmarking suite will be cleanly structured within the evals/v2/ directory of the Savant-Code repository, adhering to a modular layout that separates execution logic, task definitions, and generated reports.  
-evals/v2/ ├── README.md \# Execution instructions and architecture overview ├── schema/ │ └── task\_schema\_v2.json \# JSON schema enforcing task definitions ├── tasks/ │ ├── pure\_coding/ \# SWE-bench style real-world tasks │ ├── mcp\_tooling/ \# Mock server integration synthetic tasks │ └── fsm\_compliance/ \# Synthetic FSM violation testing tasks ├── runner/ │ ├── index.ts \# Core benchmark pipeline execution logic │ ├── SavantRunner.ts \# SDK integration and trace collection │ └── ExternalRunner.ts \# Bun.spawn wrappers for external CLIs ├── sandbox/ │ ├── firecracker\_config.json \# MicroVM provisioning hardware constraints │ └── snapshot\_manager.ts \# CoW memory state restoration logic ├── scoring/ │ ├── metrics.ts \# FSM and orchestration mathematical formulas │ └── llm\_judge.ts \# Structured output evaluation prompts └── reports/ \# Aggregated SQLite traces and MD leaderboards  
-This comprehensive, highly detailed harness transforms the evaluation of Savant-Code from a simple, output-focused diff-comparison tool into a world-class, architectural capability benchmark, providing rigorous, reproducible, and deeply actionable metrics for future development.
-
-### **Works cited**
-
-> 1. Introducing SWE-bench Verified \- OpenAI, [https://openai.com/index/introducing-swe-bench-verified/](https://openai.com/index/introducing-swe-bench-verified/)  
-> 2. Can Language Models Resolve Real-world Github Issues \- SWE-bench, [https://www.swebench.com/original.html](https://www.swebench.com/original.html)  
-> 3. SWE-bench Verified, [https://www.swebench.com/verified.html](https://www.swebench.com/verified.html)  
-> 4. AI Protocol Mocking (MCP & A2A) \- MockServer, [https://www.mock-server.com/mock\_server/ai\_protocol\_mocking.html](https://www.mock-server.com/mock_server/ai_protocol_mocking.html)  
-> 5. Mock MCP Server \- AI-driven mock data orchestration with OpenAPI JSON Schema for testing \- GitHub, [https://github.com/mcpland/mock-mcp](https://github.com/mcpland/mock-mcp)  
-> 6. Does SWE-Bench-Verified Test Agent Ability or Model Memory? \- arXiv, [https://arxiv.org/html/2512.10218v1](https://arxiv.org/html/2512.10218v1)  
-> 7. Daily Papers \- Hugging Face, [https://huggingface.co/papers?q=Docker-based%20environment%20isolation](https://huggingface.co/papers?q=Docker-based+environment+isolation)  
-> 8. README.md \- BryanZaneee/llmbench \- GitHub, [https://github.com/BryanZaneee/llmbench/blob/main/README.md](https://github.com/BryanZaneee/llmbench/blob/main/README.md)  
-> 9. Firecracker vs gVisor: Which isolation technology should you use? | Blog \- Northflank, [https://northflank.com/blog/firecracker-vs-gvisor](https://northflank.com/blog/firecracker-vs-gvisor)  
-> 10. The Container Runtime Nobody Told You About (And Four Others) \- DEV Community, [https://dev.to/copyleftdev/the-container-runtime-nobody-told-you-about-and-four-others-25e1](https://dev.to/copyleftdev/the-container-runtime-nobody-told-you-about-and-four-others-25e1)  
-> 11. Agent Sandbox Guide: A Complete Solution for Safely Running AI, [https://eastondev.com/blog/en/posts/ai/20260323-agent-sandbox-guide/](https://eastondev.com/blog/en/posts/ai/20260323-agent-sandbox-guide/)  
-> 12. Kata Containers vs Firecracker vs gvisor : r/docker \- Reddit, [https://www.reddit.com/r/docker/comments/1fmuv5b/kata\_containers\_vs\_firecracker\_vs\_gvisor/](https://www.reddit.com/r/docker/comments/1fmuv5b/kata_containers_vs_firecracker_vs_gvisor/)  
-> 13. AI Agent Code Execution Sandboxes on GPU Cloud: E2B, Daytona, and Firecracker Setup Guide (2026) | Spheron Blog, [https://www.spheron.network/blog/ai-agent-code-execution-sandbox-e2b-daytona-firecracker/](https://www.spheron.network/blog/ai-agent-code-execution-sandbox-e2b-daytona-firecracker/)  
-> 14. DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback, [https://arxiv.org/html/2605.22781v2](https://arxiv.org/html/2605.22781v2)  
-> 15. forkd/README.md at main · deeplethe/forkd \- GitHub, [https://github.com/deeplethe/forkd/blob/main/README.md](https://github.com/deeplethe/forkd/blob/main/README.md)  
-> 16. mock-mcp | MCP Servers \- LobeHub, [https://lobehub.com/mcp/mcpland-mock-mcp](https://lobehub.com/mcp/mcpland-mock-mcp)  
-> 17. MCP Test Client — Validate Your MCP Client Against Hosted Mock Servers, [https://mcpplaygroundonline.com/mcp-test-client](https://mcpplaygroundonline.com/mcp-test-client)  
-> 18. Read stdout from a child process \- Bun, [https://bun.com/docs/guides/process/spawn-stdout](https://bun.com/docs/guides/process/spawn-stdout)  
-> 19. Spawn \- Bun, [https://bun.com/docs/runtime/child-process](https://bun.com/docs/runtime/child-process)  
-> 20. Child Processes \- Bun.spawn(), [https://oven-sh-bun.mintlify.app/runtime/child-process](https://oven-sh-bun.mintlify.app/runtime/child-process)  
-> 21. Terminal-Bench 2.0: The AI Agent Benchmark That Actually | explainx.ai Blog, [https://explainx.ai/blog/terminal-bench-2-0-ai-agent-benchmark-evaluation](https://explainx.ai/blog/terminal-bench-2-0-ai-agent-benchmark-evaluation)  
-> 22. Keep your Agents Under Control with agent-belt \- JFrog, [https://jfrog.com/blog/keep-agents-under-control-with-agent-belt/](https://jfrog.com/blog/keep-agents-under-control-with-agent-belt/)  
-> 23. Simon Willison on ai-assisted-programming, [https://simonwillison.net/tags/ai-assisted-programming/?page=10](https://simonwillison.net/tags/ai-assisted-programming/?page=10)  
-> 24. 62.2% on Aider Polyglot from a MacBook Pro. Then the other model we tried scored 4%. Here's what actually happened, with a working cost loop attached. \- LLMKube, [https://llmkube.com/blog/m5-max-aider-polyglot-and-finops](https://llmkube.com/blog/m5-max-aider-polyglot-and-finops)  
-> 25. RunanywhereAI/hybrid-arena \- GitHub, [https://github.com/RunanywhereAI/hybrid-arena](https://github.com/RunanywhereAI/hybrid-arena)  
-> 26. GitHub \- jfrog/agent-belt: Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode, or any custom agent you plug in; verify behavior with rule checks, workspace diffs, multi-judge LLM consensus; pin reliability with pass^k variance across trials. Git worktrees, optional Docker sandbox., [https://github.com/jfrog/agent-belt](https://github.com/jfrog/agent-belt)  
-> 27. Best Stateful Sandboxes for Code Execution in 2026 \- Beam Cloud, [https://www.beam.cloud/blog/best-stateful-sandbox-code-execution-2026](https://www.beam.cloud/blog/best-stateful-sandbox-code-execution-2026)  
-> 28. Aider Polyglot: A New Paradigm for AI-Assisted Development? \- Skywork, [https://skywork.ai/skypage/en/Aider-Polyglot-A-New-Paradigm-for-AI-Assisted-Development/1972872987328704512](https://skywork.ai/skypage/en/Aider-Polyglot-A-New-Paradigm-for-AI-Assisted-Development/1972872987328704512)  
-> 29. o1 tops aider's new polyglot leaderboard, [https://aider.chat/2024/12/21/polyglot.html](https://aider.chat/2024/12/21/polyglot.html)  
-> 30. Our AI Agent \+ 3.7 Sonnet ranked \#1 on Aider's polyglot bench — a 76.4% score, [https://dev.to/refact/our-ai-agent-37-sonnet-ranked-1-on-aiders-polyglot-bench-a-764-score-10d0](https://dev.to/refact/our-ai-agent-37-sonnet-ranked-1-on-aiders-polyglot-bench-a-764-score-10d0)  
-> 31. The Step-By-Step Guide to MCP Evaluation \- Confident AI, [https://www.confident-ai.com/blog/the-step-by-step-guide-to-mcp-evaluation](https://www.confident-ai.com/blog/the-step-by-step-guide-to-mcp-evaluation)