khabotcli 1.0.0

package/.OLD/STEP-1/RESEARCH-RESULTS/Deep Research: Testing Strategies for AI Agent Runtimes.md

@@ -0,0 +1,240 @@
+# **Deep Research: Testing Strategies for AI Agent Runtimes**
+
+## **The Paradigm Shift in Agentic Quality Assurance**
+
+The deployment of autonomous artificial intelligence agents operating via Large Language Models (LLMs) fundamentally invalidates traditional, deterministic software testing methodologies. In classical software engineering, test suites rely on fixed inputs producing exact, predictable outputs. Conversely, AI agent runtimes, such as the proposed KhaBotCLI, handle non-deterministic LLM outputs, orchestrate complex multi-agent workflows, and interact with dynamic tool execution environments. A single passing test run in an agentic system is statistically insignificant; counterintuitively, re-running the same test with identical inputs can yield a failure due to minor variations in token sampling, model state, or contextual drift.1
+
+To ensure reliability, security, and performance within KhaBotCLI, the quality assurance architecture must transition from binary correctness validation to probabilistic behavioral bounding. This necessitates the implementation of continuous evaluation, semantic similarity tracking, property-based testing, and rigorous cognitive load profiling.2 Modern enterprise guidelines emphasize that distributed AI systems require continuous observability and instrumentation across all operations and handoffs, moving away from static regression tests that penalize agent improvement.2 This exhaustive report delineates a comprehensive testing strategy tailored for an advanced AI agent runtime, establishing architectural specifications for testing non-deterministic outputs, validating the Model Context Protocol (MCP), testing multi-agent workflows, and constructing resilient continuous integration and continuous delivery (CI/CD) pipelines.
+
+## **The KhaBotCLI Testing Pyramid**
+
+To manage the inherent variability of LLM-backed workflows without incurring unsustainable API costs or latency, the KhaBotCLI testing strategy must adopt a specialized, four-tier testing pyramid. This structure isolates reasoning logic from deterministic execution, ensuring that the bulk of the test suite runs rapidly and reliably offline, while reserving expensive network calls for high-level evaluations.
+
+| Testing Layer | Proportion of Suite | Objective and Scope | Execution Environment and Tooling | Example Test Case Specification |
+| :---- | :---- | :---- | :---- | :---- |
+| **Unit Testing** | 50% | Validate deterministic components, tool parsers, configuration loaders, and MCP client logic without invoking network-based LLMs. | Local, isolated, offline environments. Utilizes Vitest/Jest for JavaScript components and go test for Go components.5 | Verify that the read_file tool returns a standardized, stringified error object when encountering a 403 Permission Denied operating system error, rather than crashing the runtime. |
+| **Integration Testing** | 30% | Validate the interaction between the agent runtime, local system tools, and mocked LLM responses to ensure correct tool-calling loops. | Local environments using dependency injection. Relies on recorded LLM trajectories (Cassettes) and Mock MCP servers.7 | Inject a deterministic mocked LLM response requesting a bash command execution; verify the runtime intercepts the payload, evaluates permissions, and successfully executes the command. |
+| **End-to-End (E2E)** | 10% | Validate the full lifecycle of the agent using lightweight local models or constrained API calls to ensure cross-component system integration. | Sandboxed environments (e.g., Docker containers) interacting with live, isolated external file systems and APIs.8 | Provide a prompt instructing the agent to refactor a specific repository file. Validate that the file is read, successfully edited, and subsequently passes a local syntax check. |
+| **Continuous Evaluation** | 10% | Measure output quality, reasoning capabilities, and behavioral drift using probabilistic grading and LLMs-as-judges.2 | Post-deployment shadow mode or nightly CI runs using production-grade models (e.g., Claude 3.5 Sonnet, GPT-4o). | Evaluate the agent's summarization of a 10,000-token log file against a human-graded golden standard using a strict scoring rubric. |
+
+This pyramid ensures that the runtime is tested from the foundational logic up to its emergent reasoning capabilities. Unit and integration tests ensure the "hands" of the agent work perfectly, while E2E and evaluation layers ensure the "brain" is reasoning correctly.
+
+## **1. Testing Non-Deterministic Outputs**
+
+Because KhaBotCLI will inherently produce varying text outputs for the identical prompt, exact string matching must be replaced with statistical, property-based, and semantic validation frameworks. The testing infrastructure must shift its focus toward asserting outcomes and semantic correctness rather than precise phrasing.10
+
+### **Scoring Rubrics**
+
+Defining strict criteria and applying scoring rubrics is essential for evaluating complex, multi-step agent behaviors. Instead of a binary pass/fail, rubrics allow evaluators to assign granular scores across multiple dimensions of an output. For example, a response might be graded on factuality, conciseness, and adherence to constraints. The implementation requires defining a specific prompt for a judge that outlines the grading scale (e.g., assigning a score from 0 to 5) and explicitly detailing what constitutes each tier. The tradeoffs of this approach include the difficulty of designing rubrics that remove subjectivity and the overhead of maintaining these rubrics as the agent's capabilities expand. However, when combined with automated grading, rubrics provide highly actionable feedback for developers, identifying exactly which dimension of the agent's behavior is failing.11
+
+### **LLM-as-a-Judge**
+
+For nuanced validation, utilizing a secondary, highly capable LLM to grade the output of the agent is the industry standard for evaluating non-deterministic text. The implementation involves constructing an evaluation pipeline where the judge LLM is provided with a system prompt containing the rubric, the original user prompt, and the KhaBotCLI agent's generated response. The judge evaluates the response and outputs a structured JSON object containing scores and a reasoning trace.11 The primary tradeoff is that this approach introduces an additional layer of non-determinism and incurs further API costs. Furthermore, the judge model requires periodic calibration against human expert graders to ensure it is not hallucinating scores or exhibiting bias.9 Leading frameworks for implementing this include LangSmith, Braintrust, and Promptfoo, which streamline the execution of these evaluation pipelines and track scores over time to detect capability regressions.5
+
+### **Property-Based Testing**
+
+Property-based testing frameworks, such as Hypothesis in Python or fast-check in JavaScript/TypeScript, transcend traditional example-based testing by automatically generating vast ranges of inputs to verify system invariants. In the context of AI agent tool execution, this methodology is invaluable. The implementation involves defining core properties that must always hold true regardless of the LLM's non-deterministic text generation. For instance, if KhaBotCLI is tasked with generating a JSON configuration file, the property-based test asserts that for any valid prompt, the resulting output file must always parse as valid JSON, must contain specific required keys, and must not exceed a certain file size limit.13 The tradeoff is the steep learning curve required to translate complex agent behaviors into mathematical properties; however, once established, these tests uncover edge cases that manual example-based tests routinely miss.
+
+### **Snapshot Testing**
+
+Snapshot testing in AI runtimes involves recording a "golden" output trajectory, which includes the agent's chain of thought, tool invocations, and final responses. During subsequent test runs, the newly generated output is compared to the recorded snapshot. Because exact string matches will inevitably fail due to non-determinism, the comparison engine must evaluate structural equivalence and semantic similarity. Implementation involves converting both the expected golden response and the agent's actual response into vector embeddings and asserting that the cosine similarity exceeds a specific threshold (e.g., similarity > 0.85).12 This allows the test to pass even if the agent rephrases its answer, so long as the factual content and tool-calling sequence remain identical. The tradeoff is the computational overhead of generating embeddings during test runs, but it provides a highly resilient method for validating conversational agents.
+
+### **Regression Detection**
+
+Regression detection monitors the agent's performance over time to detect when quality drops across versions. Anthropic divides this into "capability evals" and "regression evals." Capability evals target tasks the agent currently struggles with, providing a benchmark for future improvement. Once the agent masters these tasks, they graduate into the regression suite. The regression suite must have a near 100% pass rate and runs continuously to protect against backsliding.9 Implementation involves integrating continuous evaluation metrics into the CI/CD pipeline, tracking task success rates over rolling 7-day and 30-day windows. If the success rate drops by more than 10% below the established baseline, an automated drift detection alert is triggered, preventing the deployment of the degraded agent version.2
+
+## **2. Testing Agent Tools**
+
+KhaBotCLI's ability to manipulate its environment relies entirely on its tools. The testing of these tools must be strictly isolated from the LLM's reasoning engine to ensure deterministic, rapid validation. Tools act as the functional extremities of the agent, and their reliability is paramount.7
+
+### **Testing Individual Tool Implementations**
+
+Testing individual tool implementations, such as bash, read_file, or edit_file, requires robust unit testing frameworks like Vitest or Go test. These tests must be executed without invoking an LLM. The implementation involves passing hardcoded, deterministic arguments to the tool function and asserting the outcome on the host system. For a read_file tool, the test creates a temporary file, invokes the tool with the file path, and asserts that the returned string matches the file's contents exactly. For an edit_file tool, the test provides a path and a diff payload, and asserts that the file on disk is correctly mutated.17
+
+### **Testing Tool Permission Evaluation**
+
+AI agents operating within enterprise environments must adhere to the principle of least privilege. Testing tool permission evaluation ensures that the agent cannot execute destructive actions. The implementation requires injecting payloads where the mock LLM attempts to execute unauthorized commands, such as running rm -rf / via the bash tool or attempting to read sensitive environment variables. The test asserts that the KhaBotCLI policy engine intercepts the request, blocks the execution, and returns a specific unauthorized status code back to the LLM context, preventing any actual system mutation.19
+
+### **Testing Tool Error Handling**
+
+LLMs frequently hallucinate tool arguments or attempt to use tools in invalid contexts. Robust error handling is critical to prevent the runtime from crashing. Testing this involves simulating edge cases by injecting tool-call payloads with missing required parameters, incorrect data types, or requests for non-existent files. The assertions must verify that KhaBotCLI catches the exception gracefully, formats a descriptive error message (e.g., "Error: file not found at path X, please verify the directory structure"), and returns this string back to the LLM context to prompt an autonomous self-correction loop.17
+
+### **Mocking LLM Responses for Deterministic Tool Testing**
+
+To test the runtime's ability to orchestrate tools based on LLM output, the LLM API client must be entirely mocked. Utilizing dependency injection at the architectural level (Level 3 Mocking), the test suite replaces the live API client with a mock client.7
+
+| Testing Goal | Mock Payload Injection | Assertion Strategy |
+| :---- | :---- | :---- |
+| **Verify Tool Invocation** | Inject: {"name": "edit_file", "args": {"path": "main.go", "content": "func main(){}"}} | Assert that the internal edit_file function was called exactly once with the provided arguments. |
+| **Verify Tool Output Routing** | Inject: Mocked success response from a mocked read_file tool. | Assert that the runtime correctly formats the tool output into the proper message schema and appends it to the session history array. |
+| **Verify Multi-Step Chains** | Inject: A sequence of predefined tool calls (e.g., list_dir followed by read_file). | Assert that the runtime executes the sequence in order, maintaining state between calls without dropping context.18 |
+
+This approach allows engineers to validate the complex orchestration logic of the agent framework without incurring the latency, cost, or non-determinism of actual neural network inference.
+
+## **3. Testing Multi-Agent Workflows**
+
+Advanced iterations of KhaBotCLI will employ multi-agent workflows, where specialized sub-agents collaborate to resolve complex tasks. This architectural pattern, moving beyond single-agent paradigms, introduces exponential testing complexity.21
+
+### **Testing Agent-to-Agent Communication**
+
+Agent-to-agent communication relies on specific protocols, such as the Agent-to-Agent (A2A) protocol or framework-specific message queues. Testing this communication requires interceptor frameworks that can monitor and assert the data flowing between agents.22 The implementation involves creating a mock sending agent and a mock receiving agent. The test injects a message from the sender and asserts that the receiver correctly parses the intent, extracts the payload, and updates its internal working context. Furthermore, tests must utilize active translation validation, ensuring that a sub-agent does not misattribute the history of a previous agent to itself during handoffs.22
+
+### **Testing Task Delegation Chains**
+
+Multi-agent topologies rely on hierarchical delegation. For example, a system might utilize a Manager agent (Level 0) that delegates to an Architect agent (Level 1), which in turn delegates to a Builder agent (Level 2).24 Automated test scripts must validate the integrity of these delegation matrices.
+
+| Validation Target | Description and Implementation | Failure Condition |
+| :---- | :---- | :---- |
+| **Hierarchy Levels** | Ensures agents delegate downward to lower levels and escalate upward appropriately. | An L3 agent attempting to delegate to an L2 agent.25 |
+| **Circular Dependencies** | Automatically detects routing loops where agents endlessly pass tasks to one another. | Agent A delegates to Agent B, which delegates to Agent C, which escalates back to Agent A.25 |
+| **Reference Validity** | Confirms that all delegates_to or escalates_to targets defined in the agent configuration files actually exist. | An agent is configured to delegate to a target identifier that is missing from the system registry.25 |
+
+### **Testing Concurrent Agent Execution**
+
+When multiple agents execute tasks in parallel—such as an Information Retrieval Agent querying a database while a Reasoning Agent processes a user prompt—they share underlying state and context. Testing must subject the runtime to rigorous stress tests to identify race conditions where concurrent updates corrupt the session memory.26 Frameworks must utilize thread-sanitizers or isolated concurrency testing libraries to detect deadlocks. The implementation involves simulating highly concurrent workloads and inserting deliberate delays between relevant code statements to artificially expand the race window, making evidence of unexpected behavior easier to detect.27
+
+### **Testing Context Compaction Preservation**
+
+As agent sessions extend over hundreds of turns, the context window fills, leading to "context rot" where performance degrades and API costs surge. KhaBotCLI must implement context compaction—asynchronously summarizing older events and pruning raw logs to maintain dense, relevant working context.22 Testing compaction strategies requires specialized probe tests to measure how much critical context is preserved.
+
+| Probe Type | Testing Objective | Example Evaluation Question |
+| :---- | :---- | :---- |
+| **Recall** | Factual retention of specific details from early in the session. | "What was the exact error code returned by the server on turn 2?" 11 |
+| **Artifact** | Tracking changes to files or system state over time. | "Which files have we modified so far, and what changed in each?" 11 |
+| **Continuation** | Task planning and understanding the current workflow state. | "Based on the previous steps, what is the immediate next action required?" 11 |
+| **Decision** | Retaining the reasoning chain and agreed-upon constraints. | "We previously discussed options for the database schema. What was decided and why?" 11 |
+
+During testing, the judge LLM receives these probe questions, the agent's response based on the compacted context, and a strict rubric. If the agent fails to answer accurately, the compaction algorithm is deemed lossy and requires refinement.11
+
+### **Integration Testing a Full Agent Lifecycle**
+
+Integration testing the full lifecycle involves deploying KhaBotCLI in an isolated environment and evaluating its ability to resolve complete, multi-step problems autonomously. Methodologies like SWE-bench provide a rigorous framework for this. The agent is provided with an execution environment (e.g., a Docker Image) containing a codebase and an issue description. The agent must navigate the repository, utilize tools to investigate the bug, generate a patch, and apply it. The test benchmark evaluates performance by measuring whether the generated patch successfully passes both PASS_TO_PASS and FAIL_TO_PASS unit tests within the repository.8
+
+## **4. CI/CD for AI Agent Projects**
+
+The Continuous Integration and Continuous Delivery (CI/CD) pipeline for an AI agent runtime cannot simply check code syntax; it must validate the emergent behavior resulting from the interplay of code, prompt templates, and model parameters. Altering a system prompt by a single sentence can fundamentally shift agent behavior without altering any programmatic logic, making traditional pipeline architectures insufficient.10
+
+### **CI Practices of Leading Projects**
+
+Leading agentic development environments employ distinct CI practices to maintain reliability.
+
+* **Claude Code CLI:** Anthropic's terminal agent relies heavily on self-testing and integration suites. Tests are organized into deep describe blocks that validate workflows step-by-step. Furthermore, Claude Code encourages the inclusion of explicit verification commands within the agent's prompt context (e.g., instructing the agent to run the test suite after every file edit and loop until tests pass) to ensure the generated code is functional before the task is marked complete.32  
+* **OpenCode:** The OpenCode architecture separates concerns into dual orchestrators (Manager and Architect) and utilizes specialized worker agents. Its test suite, written in Go, relies on a highly modular architecture that validates state recovery, checkpointing, and the ability of agents to hand off tasks within tmux sessions. The suite actively runs tests to detect and kill "slop comments" and ensures strict adherence to routing protocols.24  
+* **Cursor:** While heavily integrated into the IDE, Cursor's backend testing relies on evaluating the model's ability to maintain context across multiple files and apply precise edits without corrupting surrounding syntax, often utilizing custom models to orchestrate these specific codebase interactions.36
+
+### **CI Pipeline Specification for an Agent Runtime**
+
+A robust CI pipeline for KhaBotCLI must follow a strict, multi-stage matrix to catch both deterministic bugs and behavioral drift.10
+
+| Pipeline Stage | Triggers & Execution Constraints | Primary Actions | Failure Threshold |
+| :---- | :---- | :---- | :---- |
+| **1. Static Analysis & Linting** | Runs on every commit. Execution time: < 1 min. | Standard code linting, security scanning, and prompt template validation (ensuring required Jinja/template variables exist). | Any syntax error, memory leak warning, or missing schema definition. |
+| **2. Deterministic Unit Tests** | Runs on every commit. Execution time: < 2 mins. | Validates tool parsers, configuration loading, and pure functions. Tool functions are tested with hardcoded inputs. | < 95% line coverage on the core reasoning engine; any single test failure. |
+| **3. Cassette Integration (Record/Replay)** | Runs on every Pull Request. Execution time: < 5 mins. | Loads recorded LLM trajectories. Verifies the runtime correctly parses tool calls and routes multi-agent delegations without external API calls.38 | Any deviation in the execution path or unhandled exception during mock tool execution. |
+| **4. E2E Capability Evals** | Runs nightly or pre-merge for major features. Execution time: 10-30 mins. | Deploys KhaBotCLI in an isolated Docker sandbox. Assigns 10-20 known tasks. Uses live API calls (e.g., Claude 3.5 Sonnet).31 | Evaluated via LLM-as-judge. Pass rate must exceed the predefined capability baseline. |
+| **5. Regression Evals** | Runs nightly. Execution time: 30+ mins. | Tests previously mastered, critical capabilities using live models.9 | Pass rate must be > 98%. Any regression immediately blocks release pipelines. |
+
+### **Handling Flaky Tests Due to LLM Non-Determinism**
+
+Because End-to-End and evaluation layers rely on live probabilistic models, occasional failures are inevitable and do not always indicate a code defect. To handle this flakiness, CI steps running live evaluations must not enforce a binary pass/fail on a single run. Instead, tests must execute a defined number of iterations (e.g., 5 runs per prompt) and assert that the statistical success rate meets a strict threshold (e.g., passes 4 out of 5 times).1
+
+### **Benchmarking Performance and Gating Releases**
+
+Deploying KhaBotCLI updates directly to production users carries high risk. The final stage of the CI/CD pipeline must enforce strict quality gates and utilize progressive rollout strategies. Version control must encompass prompts, tool schemas, and model configuration temperatures alongside the codebase to ensure true rollback capability.10 New versions should initially be deployed in "shadow mode," processing a sample of live user inputs silently in the background. The outputs are compared against the active production version using semantic similarity scoring. If the new version performs adequately, a canary deployment routes a small percentage of active traffic to the new version, gated by automated drift detection metrics that halt the rollout if error rates spike.10
+
+## **5. Mock LLM Strategies**
+
+To facilitate fast, reliable, and cost-effective CI pipelines, KhaBotCLI requires a sophisticated Mock LLM strategy. Relying solely on live APIs introduces unpredictable network latency, non-determinism, and significant financial overhead, making it unsuitable for the bulk of the testing pyramid.
+
+### **Record/Replay LLM Responses**
+
+The most robust approach for deterministic testing of complex agent trajectories is the Record and Replay mechanism. Academic frameworks conceptualize this as AgentRR, while practical implementations utilize tools like pytest-agentcontract or custom proxies.38
+
+* **Implementation:** During the recording phase, developers run the agent through a task against a live LLM API. A network interceptor captures the exact sequence of HTTP requests and responses, including user prompts, intermediate tool calls, system outputs, and final responses. This trajectory is saved as a structured JSON "cassette".12 During the replay phase in CI, the Mock Client intercepts requests, hashes the prompt, and retrieves the exact corresponding response from the cassette.  
+* **Tradeoffs:** Replay rules must be strict. Routing decisions, fallback evaluations, and model inferences are bypassed. The recorded outputs are returned exactly as captured. This ensures determinism but requires developers to manually re-record cassettes whenever the underlying agent prompt structure changes significantly.41
+
+### **Fake LLM with Scripted Responses**
+
+For unit testing specific agent loops, a Fake LLM can be programmed with stateful, scripted responses based on input patterns.
+
+* **Implementation:** The Fake LLM analyzes the incoming prompt. If the prompt matches a regex pattern (e.g., .*calculate.*), it returns a hardcoded tool call for the calculator. If the input contains an error string, it returns a predefined self-correction payload.  
+* **Tradeoffs:** This isolates the orchestration logic from network calls entirely, making tests execute in milliseconds. However, it fails to simulate the complex, unpredictable ways real LLMs might combine multiple tools or misunderstand context.42
+
+### **Lightweight Local Models for Testing**
+
+When testing the integration of the prompt construction and parsing logic without incurring API costs, lightweight local models (e.g., Llama-3-8B via Ollama) can be spun up during the CI process.
+
+* **Tradeoffs:** While these small models may lack the reasoning capacity to successfully complete complex coding tasks, they are sufficient to verify that KhaBotCLI successfully connects to an endpoint, sends a properly formatted payload, and correctly parses the returned markdown or JSON structure.43
+
+### **When to Use Real LLM Calls vs. Mocks**
+
+| Testing Scenario | Recommended Approach | Justification |
+| :---- | :---- | :---- |
+| **Tool Parser Validation** | Scripted Fake LLM | Requires instant, deterministic payloads to ensure the parser handles both valid and malformed JSON seamlessly. |
+| **Agent Orchestration Logic** | Record/Replay (Cassettes) | Allows testing of deep, multi-turn tool calling loops without non-determinism breaking the test assertions.38 |
+| **Prompt Template Rendering** | Lightweight Local Model | Ensures the generated prompt fits within context limits and is processed correctly by a real inference engine without high API costs. |
+| **Reasoning and Capability Evaluation** | Real LLM (e.g., Claude 3.5 Sonnet) | The only way to accurately assess the agent's emergent intelligence, accuracy, and task success rate against novel problems.9 |
+
+## **6. MCP Server Testing**
+
+The Model Context Protocol (MCP) standardizes how AI agents discover, authenticate, and interact with external tools and data sources. Testing MCP integration requires simulating the full client-server lifecycle to ensure KhaBotCLI can reliably expand its capabilities dynamically.
+
+### **Testing Discovery and Handshake**
+
+Testing the initial connection involves asserting that KhaBotCLI correctly initiates the handshake procedure. The test must verify that when the runtime attempts to connect to a protected server, it correctly processes the 401 Unauthorized response and reads the WWW-Authenticate header to locate the Protected Resource Metadata (PRM) document. The framework must then simulate the Dynamic Client Registration (DCR) flow or OAuth 2.1 exchange, verifying that the client successfully retrieves an access token and establishes the connection.44 Tools like the MCP Inspector and custom Go mock transports are essential for validating this raw protocol exchange.46
+
+### **Creating Mock MCP Servers**
+
+To test integration without relying on external third-party services, developers must utilize mock MCP servers.
+
+* **Implementation:** Tools like mcp-tools-tester provide an interactive API layer that simulates an active MCP server.48 For automated testing in Go, developers can create in-memory transports (mcp.NewInMemoryTransports()) that instantiate a lightweight MCP server directly within the test process. This server registers a dummy tool, allowing the test suite to connect the client, invoke the tool, and assert the response entirely in memory without network overhead.47
+
+### **Testing Tool Registration from MCP**
+
+MCP allows servers to dynamically change the list of available tools at runtime based on authentication state or context.49 Testing this capability requires simulating a runtime event where the mock MCP server emits a notifications/tools/list_changed signal. The test must assert that KhaBotCLI intercepts this event, resets prior tool acceptances to prevent rug-pull attacks, refetches the tool list, and updates its internal routing registry without dropping the active user session or crashing.50
+
+### **Testing MCP Error Handling**
+
+Robust error handling must be verified at the protocol level. Tests must simulate scenarios where the MCP server returns malformed JSON-RPC responses, times out during tool execution, or rejects requests due to invalid scopes or expired JWTs. The test suite must ensure KhaBotCLI handles these errors gracefully, attempts reasonable retries, and ultimately translates the protocol error into a natural language explanation for the user or the agent's internal reasoning loop.44
+
+## **7. Performance & Load Testing**
+
+Traditional load testing, which focuses on requests per second (RPS) against stateless web endpoints, is entirely inadequate for AI agent runtimes. Every query in an agent session carries historical context; thus, a user's 10th message requires exponentially more compute, memory, and token processing than their 1st. AI workloads are probabilistic, and performance degradation is non-linear.3
+
+### **Benchmarking Agent Startup Time**
+
+Benchmarking startup time involves measuring the latency from the moment the user invokes the CLI to the moment the agent is ready to process input. This includes loading configuration files, parsing the system prompt, initializing local tools, and establishing WebSocket or HTTP connections to configured MCP servers. The test should assert that the initialization overhead remains minimal (e.g., under 500ms) to ensure a responsive developer experience, utilizing Go's benchmarking tools (go test -bench) to detect regressions in initialization routines.6
+
+### **Testing with 1000+ Sessions in Storage**
+
+Supporting 1000+ concurrent sessions requires a highly robust memory architecture. Storing massive conversation histories entirely in the runtime's local memory guarantees resource exhaustion and crashes.
+
+* **Tiered Architecture Benchmarking:** Load tests must benchmark the runtime's ability to seamlessly transition data across memory tiers, inspired by cognitive models. This involves testing an L1 Working Memory (e.g., Redis) for sub-millisecond latency caching of active sessions, and an L2 Episodic Memory (e.g., SQLite or PostgreSQL) for long-term storage of events and serialized tool outputs.52  
+* **Simulation:** Automated stress tests must initialize 1000 concurrent mock sessions, simulate rapid read/write operations spanning both memory tiers, and monitor for connection pool exhaustion, database lock contention, and latency degradation.54
+
+### **Testing Concurrent Task Execution Limits**
+
+When an agent attempts to execute multiple tasks concurrently, the runtime must enforce limits to prevent overwhelming the host system or external APIs. Load testing must simulate "spike" scenarios, slamming the system with a rapid increase in concurrent tool invocations.55 The tests must verify that KhaBotCLI implements proper rate limiting, queueing mechanisms, and backpressure handling. If performance degrades beyond an acceptable threshold, the system should fail gracefully rather than corrupting data or crashing the host process.56
+
+### **Memory Usage Profiling Strategies**
+
+Memory leaks in long-running AI services cause gradual performance degradation, increased infrastructure costs, and eventual crashes.57
+
+* **Continuous Profiling:** In Go-based runtimes like KhaBotCLI, continuous profiling via the built-in pprof package is mandatory. This provides deep, low-overhead insights into CPU usage, memory allocation, and goroutine behavior over time, allowing developers to identify unclosed resources, growing caches, or event listener accumulation.57  
+* **OpenTelemetry (OTel) Tracing:** To diagnose performance bottlenecks at the AI interaction level, KhaBotCLI must emit structured traces using OpenTelemetry. This transforms the "black box" of agent reasoning into a transparent, debuggable execution graph.
+
+| OTel Attribute Schema | Description and Purpose |
+| :---- | :---- |
+| gen_ai.usage.input_tokens | Tracks the context window bloat by measuring tokens consumed per LLM call.59 |
+| gen_ai.usage.output_tokens | Measures the generation length, helping calculate exact API costs per step.59 |
+| agent.tool.name | Identifies which tool is currently causing a bottleneck or latency spike.59 |
+| agent.tool.arguments | Captures the exact parameters passed, essential for reproducing errors locally.59 |
+| agent.decision | Records whether the agent opted for a tool_call or provided a final_answer, illuminating the routing logic.59 |
+
+By implementing cognitive load profiling that tracks token consumption and context window utilization rather than simple request volume, engineering teams can accurately assess whether the agent runtime will remain performant and cost-effective under real-world, marathon session conditions.51
+
+## **Conclusion**
+
+Testing an advanced AI agent runtime like KhaBotCLI requires abandoning the comfort of absolute determinism. The inherent unpredictability of language models mandates a paradigm shift toward continuous evaluation, semantic bounding, and sophisticated architectural mocks. By constructing a comprehensive testing pyramid that isolates pure execution logic from probabilistic inference, developers can maintain rapid, reliable CI/CD pipelines.
+
+Leveraging techniques such as LLM-as-a-judge for qualitative evaluation, strict Record and Replay mechanisms for deterministic integration testing, and comprehensive OpenTelemetry tracing for memory profiling ensures that the runtime remains observable and debuggable. Furthermore, the rigorous validation of the Model Context Protocol lifecycle and multi-agent delegation topologies guarantees that KhaBotCLI can scale its capabilities securely. Ultimately, by shifting from traditional exact-match assertions to outcome-based probabilistic evaluations, engineering teams can ensure the agent runtime remains a stable, resilient, and highly performant interface for autonomous task execution across iterative deployments.

package/.OLD/STEP-1/RESEARCH-TODO/deep-research-agent-orchestration.md

@@ -0,0 +1,75 @@
+# Deep Research: Agent Orchestration Patterns for AI Runtimes
+
+## Research Goal
+Provide a comprehensive comparison of multi-agent orchestration patterns used in production AI agent systems (Feb 2026), with specific recommendations for an API-first agent runtime like KhaBotCLI.
+
+## Key Questions
+
+### 1. Orchestration Patterns Comparison
+Microsoft Azure documents 5 patterns. For each, provide:
+- **Sequential** — agents in a pipeline, each processes previous output
+- **Concurrent** — multiple agents work in parallel, results aggregated
+- **Group Chat** — agents discuss in a shared context (with moderator)
+- **Handoff** — one agent transfers control to another based on context
+- **Magentic** — dynamic task assignment by a coordinator agent
+
+For each pattern:
+- When to use it?
+- When NOT to use it?
+- What's the context management strategy?
+- How does state persist between agents?
+- Real-world examples in production systems?
+
+### 2. Claude Code Agent Teams (Feb 2026)
+- How does Claude Code's "Agent Teams" feature work?
+- What is the TeammateTool? How does it coordinate?
+- What is the SendMessageTool protocol?
+- How do agents in a team share context?
+- How is work divided? (by file? by task? by domain?)
+
+### 3. Parallel Agent Execution
+Every major tool shipped multi-agent in Feb 2026:
+- **Cursor 2.0** — subagent system, independent agents handle discrete parts
+- **Windsurf** — 5 parallel agents via git worktrees
+- **Codex CLI** — parallel via OpenAI Agents SDK + git worktrees
+- **Grok Build** — 8 parallel agents
+- **Cline CLI 2.0** — parallel terminal agents
+
+For each:
+- How is work divided between parallel agents?
+- How do they avoid conflicts (file locks? worktrees? scoped permissions)?
+- How are results merged?
+- What's the coordination overhead?
+
+### 4. Task Dependencies & DAGs
+- How do modern agent orchestrators handle task dependencies?
+- Should KhaBotCLI support DAG-based task execution?
+- How do you handle circular dependencies?
+- What happens when a dependency fails?
+
+### 5. Human-in-the-Loop (HITL)
+- Where should human checkpoints be in multi-agent workflows?
+- How to persist state at HITL checkpoints for resumption?
+- Approval gates vs feedback loops?
+- How does this interact with the permission system?
+
+### 6. Cost & Latency Optimization
+- How to minimize total LLM cost in multi-agent orchestrations?
+- When is it cheaper to use one agent vs multiple agents?
+- How to avoid "context explosion" (each agent adds overhead)?
+- Strategies for using cheaper models for sub-tasks
+
+## Sources to Investigate
+- Microsoft Azure "AI Agent Orchestration Patterns" architecture guide
+- Claude Code Agent Teams documentation and system prompts
+- OpenAI Agents SDK documentation
+- Cursor 2.0 subagent architecture
+- AutoGen, CrewAI, LangGraph multi-agent patterns
+- Anthropic's "How we built our multi-agent research system" blog post
+
+## Expected Output
+1. Comparison matrix of all 5 patterns with pros/cons/use-cases
+2. Recommended pattern(s) for KhaBotCLI (likely: Sequential + Concurrent as defaults, Handoff for complex flows)
+3. Detailed design for KhaBotCLI's multi-agent communication protocol
+4. Task DAG specification for KhaBotCLI
+5. Cost optimization recommendations

package/.OLD/STEP-1/RESEARCH-TODO/deep-research-apply-layer.md

@@ -0,0 +1,88 @@
+# Deep Research: Apply Layer & Cheaper Model Delegation
+
+## Research Goal
+Investigate the "apply layer" pattern where AI coding agents use cheaper/faster models for file editing operations, and broader patterns of delegating specific tasks to specialized cheaper models.
+
+## Key Questions
+
+### 1. The Apply Layer Problem
+When an AI agent edits a file, it traditionally:
+1. Reads the full file
+2. Generates the entire new file content (or a diff)
+3. Writes the result
+
+This is expensive (thousands of output tokens) and error-prone (diffs fail when context shifts, search-and-replace misses when code moves, full rewrites waste tokens).
+
+- How do modern agents solve this?
+- What is Morph Fast Apply? How does it work?
+- What other "apply" models exist?
+- What's the accuracy vs cost tradeoff?
+
+### 2. Edit Strategies Comparison
+Compare these file editing strategies used in AI coding agents:
+- **Full rewrite** — LLM outputs entire file (Claude Code's Write tool)
+- **Search-and-replace** — find old string, replace with new (Claude Code's Edit tool)
+- **Unified diff** — LLM outputs diff, apply with patch (Aider's approach)
+- **Edit instructions** — LLM outputs intent, apply model merges (Morph approach)
+- **Line-range edit** — specify start/end lines and new content
+
+For each:
+- Accuracy rate in practice?
+- Token cost?
+- Speed?
+- Failure modes?
+- When to use which?
+
+### 3. Cheaper Model Delegation Patterns
+Beyond file editing, what other tasks can be delegated to cheaper models?
+- **Title generation** — session titles (Claude Code uses separate prompt)
+- **Summarization** — context compaction (Claude Code's compaction agent)
+- **Code search** — finding relevant files (Explore sub-agent)
+- **Tool search** — finding relevant tools
+- **Sentiment analysis** — detecting user frustration
+- **Commit message generation** — git commit messages
+- **PR description** — pull request descriptions
+
+For each:
+- What model tier is appropriate? (e.g., Haiku, GPT-4o-mini, Flash)
+- How much cost savings vs using the main model?
+- Is quality acceptable?
+
+### 4. Speculative Editing
+- What is speculative editing / speculative decoding for code?
+- How does Cursor implement speculative edits?
+- Can the agent predict likely edits and pre-generate them?
+
+### 5. Model Routing
+- How to route different tasks to different models automatically?
+- Should the agent decide which model to use, or should the runtime decide?
+- What heuristics determine task complexity for model selection?
+
+### 6. Cost Optimization
+- What's the typical cost breakdown for an AI coding session?
+  - Main model reasoning: X%
+  - File reading: X%
+  - File editing: X%
+  - Search/exploration: X%
+  - Summarization: X%
+- Where are the biggest cost savings opportunities?
+
+## Sources to Investigate
+- Morph Fast Apply model documentation and API
+- Aider's diff editing approach
+- Claude Code's Edit vs Write tool implementation
+- Cursor's speculative editing
+- Kilo Code's cheaper model delegation
+- OpenRouter pricing for model comparison
+- Anthropic's prompt caching documentation
+
+## Expected Output
+1. Comparison table of edit strategies (accuracy, cost, speed)
+2. Recommended model delegation map for KhaBotCLI:
+   - Main reasoning: Model A
+   - File editing: Model B
+   - Summarization: Model C
+   - Title/meta: Model D
+3. Cost projection: typical session with vs without model delegation
+4. Implementation specification for KhaBotCLI's apply layer
+5. Model routing heuristics

package/.OLD/STEP-1/RESEARCH-TODO/deep-research-context-engineering.md

@@ -0,0 +1,63 @@
+# Deep Research: Context Engineering for AI Agent Runtimes
+
+## Research Goal
+Provide a comprehensive analysis of context window management strategies for AI agent runtimes (like KhaBotCLI), covering compaction, structured note-taking, sub-agent isolation, and tool result clearing.
+
+## Key Questions
+
+### 1. Compaction Algorithms
+- What are the best compaction/summarization strategies used in production AI agents?
+- How does Claude Code implement compaction? What does its compaction prompt look like?
+- How does OpenCode implement auto-compact at 95% threshold?
+- What should be preserved during compaction? (architectural decisions, unresolved bugs, file paths, TODOs)
+- What should be discarded? (raw tool outputs, resolved issues, redundant messages)
+- How do you tune compaction for maximum recall vs precision?
+- What's the optimal compaction threshold (80%? 90%? 95%)?
+
+### 2. Tool Result Clearing
+- Anthropic recommends "tool result clearing" as the lightest compaction. How exactly does this work?
+- When should tool results be cleared vs kept?
+- Should tool results be replaced with summaries or completely removed?
+- How does Anthropic's new "context management" feature on the Developer Platform handle this?
+
+### 3. Structured Note-Taking
+- How do agents like Claude Code use TODO lists and NOTES.md for persistent memory?
+- What's the optimal format for structured notes that survive compaction?
+- How does "Claude playing Pokémon" maintain state across thousands of steps?
+- Should notes be part of the system prompt or injected as user messages?
+
+### 4. Sub-Agent Context Isolation
+- How do sub-agents get clean context windows?
+- What gets passed to a sub-agent (full context? summary? just the task)?
+- How long should a sub-agent's returned summary be? (Anthropic says 1,000-2,000 tokens)
+- When should you use sub-agents vs compaction vs note-taking?
+
+### 5. Token Budgeting
+- How should an agent runtime budget its context window?
+  - System prompt: X%
+  - Tools: X%
+  - History: X%
+  - Current task: X%
+- How do competitors (Claude Code, Cursor, OpenCode) budget their context?
+
+### 6. Multi-Turn Context Optimization
+- How to handle conversations that span 100+ turns?
+- Progressive summarization (summarize every N turns)?
+- Sliding window approaches?
+- When to start a new session vs continue compacting?
+
+## Sources to Investigate
+- Anthropic's "Effective Context Engineering for AI Agents" blog post
+- Claude Code's compaction agent prompt (from Piebald-AI/claude-code-system-prompts)
+- OpenCode's auto-compact implementation (Go source code)
+- Cursor's context management approach
+- JetBrains' research paper on "Efficient Context Management" (NeurIPS 2025)
+- Anthropic's Developer Platform "context management" feature docs
+
+## Expected Output
+A detailed document with:
+1. Comparison table of compaction strategies across tools
+2. Recommended compaction prompt template for KhaBotCLI
+3. Recommended token budgeting strategy
+4. Decision tree: when to compact vs note-take vs spawn sub-agent
+5. Implementation pseudocode for auto-compaction system

package/.OLD/STEP-1/RESEARCH-TODO/deep-research-system-prompts.md

@@ -0,0 +1,84 @@
+# Deep Research: System Prompt Architecture for AI Agent Runtimes
+
+## Research Goal
+Design a modular system prompt architecture for KhaBotCLI based on how Claude Code, OpenCode, Cursor, and other production AI agents structure their system prompts.
+
+## Key Questions
+
+### 1. Claude Code's System Prompt Architecture
+Claude Code v2.1.62 has 110+ system prompt strings. Investigate:
+- How are they organized? (categories: base, task, tool, agent, utility, reminder)
+- How are they assembled? What determines which fragments are included?
+- What's the total token count of a typical system prompt?
+- Which fragments are always included vs conditional?
+- What are "system reminders" and when are they injected mid-conversation?
+- How does the system prompt change for sub-agents (Explore, Plan, Task)?
+
+Specific prompts to analyze (from Piebald-AI/claude-code-system-prompts):
+- Base system prompt (identity, safety)
+- "Doing tasks" fragments (avoid over-engineering, read before modifying, no premature abstractions, etc.)
+- Tool descriptions (18 tools, each with its own prompt)
+- Agent prompts (Explore, Plan, Task, compaction, title, summary)
+- Utility prompts (bash command analysis, memory selection, session search)
+- System reminders (~40 mid-conversation injections)
+
+### 2. OpenCode's System Prompt Approach
+- How does OpenCode structure its system prompts?
+- Per-agent custom prompts via config
+- File-based prompts (`{file:./prompts/code-review.txt}`)
+- How does it differ from Claude Code's approach?
+
+### 3. System Prompt Best Practices
+From Anthropic's documentation and research:
+- What makes a good system prompt for an agent?
+- How to balance specificity vs flexibility?
+- How to handle safety instructions without being overly restrictive?
+- How to include tool usage guidelines effectively?
+- How to manage system prompt size vs context budget?
+
+### 4. Conditional Fragment Inclusion
+- Which fragments should be included based on environment?
+  - Git available → include git status prompt
+  - LSP available → include diagnostics prompt
+  - Hooks configured → include hooks prompt
+  - Images in context → include vision prompt
+  - MCP servers → include MCP tool descriptions
+- How to implement conditional inclusion cleanly?
+
+### 5. System Reminders (Mid-Conversation Injection)
+Claude Code injects ~40 system reminders at specific points during conversation:
+- What triggers a reminder? (e.g., after X tool calls, when context is large, when agent seems stuck)
+- What do they contain? (e.g., "remember to use parallel tool calls", "check your todo list")
+- How effective are they?
+- Should KhaBotCLI implement this pattern?
+
+### 6. Token Budget for System Prompts
+- How many tokens should a system prompt use?
+- Claude Code's system prompt is reportedly 24k+ tokens with tools
+- Is this too much? What's the impact on response quality?
+- How to keep it lean while still being effective?
+- Strategies for reducing system prompt size (abbreviation, reference, lazy loading)
+
+### 7. User-Customizable System Prompts
+- AGENT.md as user's custom addition to system prompt
+- CLAUDE.md / AGENT.md creation assistants
+- How to merge user's prompt with system prompt without conflicts?
+- Priority: user instructions > system instructions?
+
+## Sources to Investigate
+- Piebald-AI/claude-code-system-prompts repository (all 110+ prompts)
+- OpenCode agent configuration docs
+- OpenClaw agent prompts
+- Anthropic's prompt engineering documentation
+- Anthropic's "effective context engineering" blog
+- Cursor's system prompt (if leaked/available)
+- System prompts collection: asgeirtj/system_prompts_leaks
+
+## Expected Output
+1. Complete taxonomy of system prompt categories
+2. Template system prompt for KhaBotCLI's main agent
+3. Template system prompts for built-in sub-agents (explore, plan, execute, compact)
+4. Conditional fragment inclusion rules
+5. System reminder specification (when to inject, what to say)
+6. Token budget recommendation per category
+7. File structure for `./system-prompts/` directory