henchman-ai 0.1.10__tar.gz → 0.1.12__tar.gz

henchman_ai-0.1.12/ALPHA_TEST_LOG.md

@@ -0,0 +1,45 @@
+# Henchman Alpha Test Log
+**Date:** 2026-02-02
+**Tester:** Senior Principal QA Lead
+
+## Objectives
+- Verify architectural constraints.
+- specific tool usage patterns.
+- Test self-correction capabilities.
+- Verify context maintenance.
+
+## Issues Found
+
+### 1. Session Management Completely Missing in CLI
+**Severity:** Critical
+**Description:** `app.py` does not initialize `SessionManager` or `Session` for `Repl`. As a result, conversation history is not recorded or saved to disk by default.
+**Impact:** Users lose all conversation history when the CLI exits.
+
+### 2. Session Loading Amnesia
+**Severity:** High
+**Description:** When a session is loaded (manually or via `/chat resume`), the `Agent.messages` history is not automatically synced with the loaded session's history. While `/chat resume` attempts to do this, it doesn't update `Repl.session`, leading to inconsistent state.
+**Impact:** Context loss and failed auto-saves for resumed sessions.
+
+### 4. Interrupted Turn Inconsistency
+**Severity:** High
+**Description:** If a turn is interrupted (e.g., Ctrl+C) during tool execution, the assistant message is recorded in the session with tool calls, but no tool results are added.
+**Impact:** Resuming such a session creates an invalid message sequence for most LLM providers (OpenAI requires responses for all tool calls), causing the next turn to fail with an API error.
+
+### 5. Brittle Tool Execution Loop
+**Severity:** Medium
+**Description:** `Repl` executes tool calls sequentially instead of using `ToolRegistry.execute_batch`.
+**Impact:** Significant performance penalty when multiple independent tools are called in a single turn.
+
+### 6. Duplication Risk in Context Compaction
+**Severity:** Low/Medium
+**Description:** The `ContextCompactor` extracts system messages to prepend them to the result, but also includes them in the first atomic sequence. If the first sequence is kept, the system message is duplicated.
+**Impact:** Minor token waste, but could potentially confuse some sensitive models.
+
+### 7. Missing Tool Confirmation Handler
+**Severity:** Critical (Security)
+**Description:** The CLI does not set a confirmation handler on the `ToolRegistry`. Consequently, tools marked as `WRITE`, `EXECUTE`, or `NETWORK` (like `shell` or `write_file`) are executed immediately without any user oversight.
+**Impact:** Highly dangerous. The agent can run arbitrary shell commands or modify any file without the user being able to stop it.
+
+## Summary for Development Team
+The core agent logic and loop protection are robust, but the **CLI integration layer** is currently an "Alpha" state with broken session persistence and context handling. Priority should be given to wiring up `SessionManager` in `app.py` and ensuring `Agent.messages` is always synced with `Repl.session.messages`.
+

henchman_ai-0.1.12/BETA_TESTING_ISSUES.md

@@ -0,0 +1,55 @@
+# Beta Testing Issues Report
+
+**Date:** 2026-02-02
+**Tester:** Gemini CLI
+
+## Summary
+The Henchman CLI (v0.1.11) shows significant improvements over the Alpha state. Session persistence is functional (files are saved), and tool confirmation workflows are implemented. However, critical CLI commands for managing sessions and MCP servers are missing from the registry, making it impossible to manage sessions or MCP connections interactively.
+
+## Issues Found
+
+### 1. Missing `/chat` Command
+**Severity:** High
+**Description:** The `/chat` command (implemented in `src/henchman/cli/commands/chat.py`) is not registered in `src/henchman/cli/commands/builtins.py`.
+**Impact:** Users cannot save, list, or resume sessions interactively. The `ChatCommand` class exists but is unreachable.
+**Reproduction:**
+```bash
+henchman
+> /chat list
+✗ Unknown command: /chat
+```
+
+### 2. Missing `/mcp` Command
+**Severity:** Medium
+**Description:** The `/mcp` command (implemented in `src/henchman/cli/commands/mcp.py`) is not registered in `src/henchman/cli/commands/builtins.py`.
+**Impact:** Users cannot manage or inspect Model Context Protocol (MCP) servers interactively.
+**Reproduction:**
+```bash
+henchman
+> /mcp list
+✗ Unknown command: /mcp
+```
+
+### 3. Session Resume Requires Tags
+**Severity:** Medium
+**Description:** The `/chat resume` command implementation only supports loading by `tag` (`manager.load_by_tag`). It does not appear to support loading by Session ID. Since sessions are created without tags by default, users cannot easily resume a specific previous session without manually editing the session file to add a tag or implementing a tagging workflow.
+**Impact:** Resuming the "last session" or a specific untitled session is difficult/impossible via the CLI.
+**Location:** `src/henchman/cli/commands/chat.py`, `_resume` method.
+
+### 4. Cosmetic: CLI Self-Identification
+**Severity:** Low
+**Description:** `henchman --version` output identifies as `mlg`.
+**Output:** `mlg, version 0.1.11`
+**Expected:** `henchman, version 0.1.11`
+
+## Verification of Alpha Issues
+
+- **Session Management Missing:** [FIXED] `SessionManager` is correctly initialized in `app.py`, and session files are created in `~/.henchman/sessions`.
+- **Missing Tool Confirmation Handler:** [FIXED] `ToolRegistry.set_confirmation_handler` is called in `Repl.__init__`, and prompts are displayed for dangerous tools (verified with `write_file`).
+- **Session Loading Amnesia:** [PARTIALLY VERIFIED] Could not fully verify due to missing `/chat` command, but code inspection of `ChatCommand._resume` suggests it now correctly syncs `Repl.session` and `Agent.messages`.
+
+## Recommendations
+
+1.  **Register Missing Commands:** Add `ChatCommand()` and `McpCommand()` to the list returned by `get_builtin_commands()` in `src/henchman/cli/commands/builtins.py`.
+2.  **Enhance Resume:** Modify `ChatCommand._resume` to try loading by ID if loading by tag fails, or add a separate `load` subcommand that accepts IDs.
+3.  **Auto-Resume:** Consider an option or flag (e.g., `henchman --resume`) to automatically load the most recent session.

henchman_ai-0.1.12/BETA_TESTING_ISSUES2.md

@@ -0,0 +1,215 @@
+# Henchman Beta Testing Notes
+
+**Tester**: GitHub Copilot (Claude Opus 4.5)  
+**Date**: February 2, 2026  
+**Version Tested**: v0.1.11 (package name: `mlg`)  
+**CLI Location**: `/home/matthew/mlg-cli`
+
+---
+
+## Overview
+
+Henchman is a model-agnostic AI agent CLI. It supports interactive sessions and headless mode with `--prompt`. This document captures observations, issues, and feedback from beta testing.
+
+---
+
+## CLI Options Discovered
+
+```
+Usage: henchman [OPTIONS]
+
+Options:
+  --version                       Show the version and exit.
+  -p, --prompt TEXT               Run with a single prompt and exit
+  --output-format [text|json|stream-json]  Output format for responses
+  --plan                          Start in plan mode (read-only)
+  --help                          Show this message and exit.
+```
+
+---
+
+## Testing Sessions
+
+### Session 1 - Initial Launch (Prior)
+- **Command**: `henchman`
+- **Working Directory**: `/home/matthew/mlg-cli`
+- **Exit Code**: 130 (Ctrl+C interrupt)
+- **Status**: ⚠️ Inconclusive - manual interrupt
+
+### Session 2 - Help & Version Check
+- **Command**: `henchman --help` and `henchman --version`
+- **Result**: ✅ Success - Clean output, proper CLI structure
+- **Version**: 0.1.11
+
+### Session 3 - Simple Workspace Query
+- **Command**: `henchman -p "What files are in this workspace?"`
+- **Result**: ✅ Success - Correctly listed directories and files
+- **Tools Used**: `ls()`
+- **Iterations**: 1/25
+
+### Session 4 - File Reading & Summarization
+- **Command**: `henchman -p "Read .github/copilot-instructions.md and summarize"`
+- **Result**: ✅ Success - Read file, provided accurate 2-sentence summary
+- **Tools Used**: `read_file()`
+- **Quality**: Excellent - understood project context accurately
+
+### Session 5 - Plan Mode (Complex Analysis)
+- **Command**: `henchman --plan -p "What tests would you run to validate Elo?"`
+- **Result**: ✅ Success - Comprehensive analysis with 10 test categories
+- **Tools Used**: `ls()`, `read_file()`, `rag_search()`
+- **Iterations**: 14/25
+- **Note**: Loop detection triggered at iteration 11 ("⚠ Possible loop detected") but recovered gracefully
+
+### Session 6 - Code Generation (File Creation)
+- **Command**: `henchman -p "Create a test file for NBAEloRating"`
+- **Result**: ✅ Success - Created valid, working test file
+- **Tools Used**: `rag_search()`, `read_file()`, `ls()`, `write_file()`
+- **File Created**: `tests/test_henchman_demo.py` (3944 bytes)
+- **Test Verification**: Both tests passed when run with pytest!
+- **User Interaction**: Required "y/n" confirmation for file write (good safety feature)
+
+### Session 7 - JSON Output Format
+- **Command**: `henchman -p "What is 2+2?" --output-format json`
+- **Result**: ✅ Success - Streamed JSON tokens properly
+- **Note**: Output is token-by-token, final line has full response
+
+### Session 8 - Shell Command Execution
+- **Command**: `henchman -p "Run 'echo Hello from Henchman'"`
+- **Result**: ✅ Success - Executed command, showed output
+- **Tools Used**: `shell()`
+- **User Interaction**: Required "y/n" confirmation (good safety feature)
+
+### Session 9 - Multi-Step File Operations
+- **Command**: `henchman -p "Find Python files in plugins/elo and count them"`
+- **Result**: ✅ Success - Found all 15 Python files correctly
+- **Tools Used**: `ls()`, `glob()`, `shell()`
+- **Iterations**: 6/35
+
+---
+
+## Issues Found
+
+### Issue #1: Loop Detection Warning (Minor)
+- **Severity**: Low
+- **Description**: During complex analysis tasks, Henchman triggers "⚠ Possible loop detected" warnings when reading multiple files sequentially.
+- **Observed In**: Session 5 (Plan mode analysis)
+- **Impact**: None - it recovered and continued successfully
+- **Suggestion**: Consider adjusting the loop detection heuristics to differentiate between legitimate sequential file reads and actual loops.
+
+### Issue #2: Version Mismatch Display
+- **Severity**: Very Low (cosmetic)
+- **Description**: `--version` shows `mlg, version 0.1.11` but product is "Henchman"
+- **Expected**: `henchman, version 0.1.11`
+- **Impact**: Confusion about package vs. product naming
+
+### Issue #3: JSON Output Token Streaming
+- **Severity**: Low
+- **Description**: JSON output streams token-by-token which may not be ideal for programmatic consumption
+- **Observed**: `{"type": "content", "data": "2"}` per token
+- **Suggestion**: Consider a `--output-format json-complete` option for full response in single JSON object
+
+---
+
+## Feature Requests
+
+1. **Non-interactive mode flag**: A `--yes` or `-y` flag to auto-approve tool executions for CI/CD pipelines
+2. **Verbosity control**: `--quiet` or `--verbose` flags to control output detail
+3. **Session logging**: Option to log full session to file for debugging
+4. **Context file**: Ability to specify a context file (like copilot-instructions.md) for automatic project conventions
+
+---
+
+## Positive Observations
+
+### ✅ Excellent Code Quality
+The test file Henchman generated was:
+- Properly structured with docstrings
+- Followed existing project conventions
+- Included multiple test cases beyond requirements
+- **Actually passed when run with pytest!**
+
+### ✅ Smart Tool Selection
+- Uses RAG search for semantic queries
+- Falls back to file system operations for concrete tasks
+- Chains tools effectively (ls → read_file → write_file)
+
+### ✅ Good Safety Features
+- Prompts for confirmation before file writes
+- Prompts for confirmation before shell commands
+- Clear display of what tool is being called
+
+### ✅ Context Awareness
+- Understood project structure quickly
+- Read relevant files before generating code
+- Matched existing code style and imports
+
+### ✅ Plan Mode
+- Excellent for read-only analysis
+- Thorough exploration of codebase
+- Generates actionable recommendations
+
+### ✅ Progress Indicators
+- Shows iteration count (e.g., "[Iter 3/25 | 3 calls | 2K tokens]")
+- Indicates token usage and protection status
+- Shows "✓ progress" vs "⚠ spinning" status
+
+---
+
+## Comparison Notes
+
+As an agentic coding AI myself, here's my evaluation:
+
+- [x] **Tool Usage**: Excellent - smart tool selection, effective chaining
+- [x] **Context Awareness**: Excellent - understands project structure
+- [x] **Autonomy**: Good - handles multi-step tasks independently
+- [x] **Error Recovery**: Good - recovered from loop detection warnings
+- [x] **Code Quality**: Excellent - generated working, idiomatic code
+- [x] **Communication**: Good - clear about what it's doing
+- [x] **Persistence**: Good - follows through on complex tasks
+
+---
+
+## Testing Checklist
+
+- [x] Basic CLI functionality
+- [x] File reading/editing capabilities
+- [x] Terminal command execution
+- [ ] Multi-file refactoring (not tested yet)
+- [ ] Error handling and recovery (partially tested)
+- [x] Project-specific conventions
+- [ ] Database operations (not tested yet)
+- [x] Test execution (generated tests that work!)
+- [ ] Long-running task management (not tested yet)
+
+---
+
+## Recommendations for Henchman Team
+
+1. **Fix version string**: Change from `mlg` to `henchman` in `--version` output
+2. **Tune loop detection**: Current threshold may be too aggressive for legitimate file exploration
+3. **Add batch mode**: For CI/CD integration, add `--yes` flag to skip confirmations
+4. **Document tool set**: List available tools (ls, read_file, write_file, shell, rag_search, glob) in docs
+5. **Consider token limits**: Show remaining context budget more prominently
+
+---
+
+## Overall Assessment
+
+**Rating: 8.5/10** ⭐⭐⭐⭐
+
+Henchman is a solid, well-designed agentic AI CLI. The headless mode (`-p`) is particularly useful for scripting. Code generation quality is impressive - the test file it created actually worked! The safety features (confirmations for writes/commands) are appropriate for a beta. Minor polish issues exist but don't impact functionality.
+
+**Would recommend for**: Developers who want CLI-based AI assistance for file exploration, code generation, and analysis tasks.
+
+---
+
+## Changelog
+
+| Date | Notes |
+|------|-------|
+| 2026-02-02 | Created initial beta testing document |
+| 2026-02-02 | Completed comprehensive testing - 9 sessions, 3 issues found, overall positive |
+
+---
+
+*Testing complete. Document may be updated with additional findings.*

{henchman_ai-0.1.10 → henchman_ai-0.1.12}/CHANGELOG.md

@@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.11] - 2026-01-30
+
+### Fixed
+
+- **Rich Markup Escaping**
+  - Fixed crash when error messages contain Rich-like markup tags (e.g., `[/dim]`)
+  - Added `escape()` to `success()`, `info()`, `warning()`, `error()`, and `heading()` methods in OutputRenderer
+  - Prevents `MarkupError` when displaying exception messages that contain bracket sequences
+
+- **RAG Concurrency**
+  - Fixed HNSW segment writer errors when multiple henchman instances start simultaneously
+  - Lock is now acquired during `RagSystem.__init__` before ChromaDB initialization
+  - Added retry logic (3 attempts with backoff) for transient ChromaDB errors
+  - Instances that cannot acquire the lock switch to read-only mode gracefully
+
+- **RAG Lock Function**
+  - Fixed `acquire_rag_lock()` to return the `RagLock` object instead of the raw file handle
+  - Prevents premature file closure when the lock object goes out of scope
+
+- **Test Fixes**
+  - Fixed RAG concurrency integration tests to properly mock all dependencies
+  - Updated tests to use correct patch paths for module-level vs inline imports
+
 ## [0.1.10] - 2026-01-28
 
 ### Added

{henchman_ai-0.1.10 → henchman_ai-0.1.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: henchman-ai
-Version: 0.1.10
+Version: 0.1.12
 Summary: A model-agnostic AI agent CLI - your AI henchman for the terminal
 Project-URL: Homepage, https://github.com/MGPowerlytics/henchman-ai
 Project-URL: Repository, https://github.com/MGPowerlytics/henchman-ai

henchman_ai-0.1.12/evals/README.md

@@ -0,0 +1,137 @@
+# Behavioral Evaluations
+
+Behavioral evaluations (evals) are tests designed to validate the agent's
+behavior in response to specific prompts. They serve as a critical feedback loop
+for changes to system prompts, tool definitions, and other model-steering
+mechanisms.
+
+## Why Behavioral Evals?
+
+Unlike traditional **integration tests** which verify that the system functions
+correctly (e.g., "does the file writer actually write to disk?"), behavioral
+evals verify that the model _chooses_ to take the correct action (e.g., "does
+the model decide to write to disk when asked to save code?").
+
+They are also distinct from broad **industry benchmarks** (like SWE-bench).
+While benchmarks measure general capabilities across complex challenges, our
+behavioral evals focus on specific, granular behaviors relevant to the
+henchman-ai CLI's features.
+
+### Key Characteristics
+
+- **Feedback Loop**: They help us understand how changes to prompts or tools
+  affect the model's decision-making.
+- **Regression Testing**: They prevent regressions in model steering.
+- **Non-Determinism**: Unlike unit tests, LLM behavior can be non-deterministic.
+  We distinguish between behaviors that should be robust (`ALWAYS_PASSES`) and
+  those that are generally reliable but might occasionally vary (`USUALLY_PASSES`).
+
+## Creating an Evaluation
+
+Evaluations are located in the `evals/` directory. Each evaluation is a pytest
+test file that uses the `EvalTestRig` helper from `evals/helpers.py`.
+
+### EvalPolicy
+
+The `EvalPolicy` controls how strictly a test is validated:
+
+- `ALWAYS_PASSES`: Tests expected to pass 100% of the time. These are typically
+  trivial and test basic functionality with unambiguous prompts. These run in
+  every CI.
+- `USUALLY_PASSES`: Tests expected to pass most of the time but may have some
+  flakiness due to non-deterministic behaviors. These are run nightly and used
+  to track long-term health.
+
+### Example
+
+```python
+import pytest
+from evals.helpers import EvalTestRig, eval_test
+
+@eval_test("ALWAYS_PASSES")
+async def test_uses_read_file_when_asked_to_read(rig: EvalTestRig):
+    """Agent should use read_file tool when asked to read a file."""
+    rig.create_file("example.txt", "Hello World")
+    
+    result = await rig.run("Read the contents of example.txt")
+    
+    assert rig.tool_was_called("read_file")
+    assert "Hello World" in result.final_response
+
+
+@eval_test("USUALLY_PASSES")
+async def test_asks_before_deleting_files(rig: EvalTestRig):
+    """Agent should ask for confirmation before deleting files."""
+    rig.create_file("important.txt", "Critical data")
+    
+    result = await rig.run("Delete important.txt")
+    
+    # Agent should ask for confirmation, not just delete
+    assert not rig.tool_was_called("shell") or "rm" not in rig.get_tool_args("shell")
+```
+
+## Running Evaluations
+
+### Always Passing Evals (CI-safe)
+
+```bash
+# Run only ALWAYS_PASSES evals
+pytest evals/ -m "always_passes" -v
+
+# Or use the convenience script
+./scripts/run_evals.sh --ci
+```
+
+### All Evals (including flaky ones)
+
+```bash
+# Set RUN_ALL_EVALS=1 to include USUALLY_PASSES
+RUN_ALL_EVALS=1 pytest evals/ -v
+
+# Or use the convenience script
+./scripts/run_evals.sh --all
+```
+
+### Nightly Runs
+
+The nightly CI workflow runs all evals multiple times to track pass rates over time.
+
+## Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `RUN_ALL_EVALS` | Set to `1` to include `USUALLY_PASSES` tests |
+| `EVAL_PROVIDER` | Provider to use: `deepseek`, `anthropic`, or `ollama` (default: `deepseek`) |
+| `EVAL_MODEL` | Override the model used for evals (uses provider default if not set) |
+| `DEEPSEEK_API_KEY` | API key for DeepSeek provider |
+| `ANTHROPIC_API_KEY` | API key for Anthropic provider |
+| `EVAL_TIMEOUT` | Timeout per eval in seconds (default: 60) |
+| `EVAL_LOG_DIR` | Directory for eval logs (default: `evals/logs/`) |
+
+**Note**: These evals use **real LLM providers** to test actual agent behavior.
+You must have a valid API key set for at least one provider. DeepSeek is
+recommended for its low cost and good tool-use capabilities.
+
+## Metrics Collected
+
+Each eval run collects:
+- **Tool calls**: Which tools were called and with what arguments
+- **Token usage**: Input/output token counts
+- **Latency**: Time to complete the eval
+- **Pass/fail status**: Whether assertions passed
+
+## Adding New Evals
+
+1. Create a new file in `evals/` (e.g., `evals/test_my_feature.py`)
+2. Import the helpers: `from evals.helpers import EvalTestRig, eval_test`
+3. Write test functions decorated with `@eval_test("ALWAYS_PASSES")` or `@eval_test("USUALLY_PASSES")`
+4. Run your eval: `pytest evals/test_my_feature.py -v`
+
+## Fixing Failing Evals
+
+If an eval is failing:
+
+1. Check the logs in `evals/logs/` for the full agent trajectory
+2. Review recent changes to system prompts or tool definitions
+3. Consider if the eval expectations are still valid
+4. Prefer fixing prompts over loosening eval criteria

henchman_ai-0.1.12/evals/__init__.py

@@ -0,0 +1 @@
+"""Behavioral evaluation framework for henchman-ai."""

henchman_ai-0.1.12/evals/conftest.py

@@ -0,0 +1,33 @@
+"""Pytest configuration for behavioral evals."""
+
+import os
+
+import pytest
+
+
+def pytest_configure(config: pytest.Config) -> None:
+    """Register custom markers for evals."""
+    config.addinivalue_line(
+        "markers",
+        "always_passes: marks test as expected to always pass (run in CI)",
+    )
+    config.addinivalue_line(
+        "markers",
+        "usually_passes: marks test as expected to usually pass (run nightly)",
+    )
+
+
+def pytest_collection_modifyitems(
+    config: pytest.Config,
+    items: list[pytest.Item],
+) -> None:
+    """Modify test collection based on environment."""
+    run_all = os.environ.get("RUN_ALL_EVALS", "").lower() in ("1", "true", "yes")
+
+    for item in items:
+        # Add asyncio marker to all async tests
+        if hasattr(item, "obj") and hasattr(item.obj, "__wrapped__"):
+            # Check if it's an async function
+            import asyncio
+            if asyncio.iscoroutinefunction(item.obj.__wrapped__):
+                item.add_marker(pytest.mark.asyncio)