agent-mini 0.2.0__tar.gz → 0.3.0__tar.gz

{agent_mini-0.2.0 → agent_mini-0.3.0}/.gitignore

@@ -36,3 +36,4 @@ config.json
 .ruff_cache/
 
 archived
+evals/results

{agent_mini-0.2.0 → agent_mini-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agent-mini
-Version: 0.2.0
+Version: 0.3.0
 Summary: Ultra-lightweight personal AI agent — shell, files, web, memory — powered by local or cloud LLMs
 Project-URL: Homepage, https://github.com/mohsinkaleem/agent-mini
 Project-URL: Repository, https://github.com/mohsinkaleem/agent-mini
@@ -46,8 +46,9 @@ A minimal, local-first AI agent you can actually understand and extend.
 - **Local-first** — Ollama as default, OpenAI if you want cloud, or any OpenAI-compatible server
 - **Zero frameworks** — pure `httpx` + `asyncio`, no LangChain, no LiteLLM
 - **Built-in tools** — shell, files, web search, persistent memory
+- **Vision** — drop an image path or URL into any message; works on Ollama, OpenAI, and OpenAI-compatible providers
 - **Extensible** — drop a Python file in `~/.agent-mini/plugins/` and it's a tool
-- **Small-model optimized** — token-aware context pruning, tool call repair, model-tier tuning
+- **Small-model optimized** — tier-scaled system prompt, token-aware context pruning, tool-call repair, and a [task-eval harness](evals/) to measure whether the tuning actually pays off
 
 ## Quick Start
 
@@ -133,7 +134,7 @@ Available out of the box — no API keys needed:
 | `list_directory` | Browse filesystem |
 | `search_files` | Grep / ripgrep across files |
 | `web_search` | DuckDuckGo search (free, no key) |
-| `web_fetch` | Fetch any URL as plain text |
+| `web_fetch` | Fetch a public URL as plain text (private/loopback hosts blocked) |
 | `memory_store` | Save to persistent memory |
 | `memory_recall` | Fuzzy search memory (TF-IDF) |
 
@@ -214,7 +215,9 @@ Control tool access:
 { "tools": { "sandboxLevel": "readonly" } }
 ```
 
-Dangerous shell commands (`rm -rf`, `sudo`, `mkfs`, etc.) are blocked by default.
+Dangerous shell commands (`rm -rf`, `sudo`, `mkfs`, etc.) are blocked by default. Treat the shell blocklist as **friction, not a boundary** — a determined caller can bypass it, so don't run the agent in an untrusted context and expect the blocklist to save you.
+
+`web_fetch` is guarded against basic SSRF: requests to `localhost`, loopback (`::1`), RFC-1918 private ranges, and link-local (including the cloud metadata endpoint `169.254.169.254`) are refused, and the final URL is re-checked after every redirect. DNS rebinding can still bypass — use `sandboxLevel: readonly` if you need a stronger guarantee.
 
 ---
 
@@ -236,11 +239,14 @@ Agent Mini is a **ReAct loop** — the LLM reasons, picks a tool, observes the r
 
 Key design choices for small/local models:
 
+- **Tier-scaled system prompt** — tiny models get a compact rules block and no memory recall; larger tiers get the full descriptive prompt and recent context
+- **Inline tool list** — `<available_tools>` block in the system prompt so small models can see tool names at a glance without inferring from the JSON schema
 - **Token-aware context** — estimates token usage and prunes old tool results when approaching the model's effective context window
-- **Model tier classification** — auto-detects tiny/small/medium/cloud models and adjusts context budgets, iteration limits, and output caps
+- **Model tier classification** — auto-detects tiny/small/medium/cloud (including large open-weight sizes like `:32b`, `:70b`, `8x7b`) and adjusts context budgets, iteration limits, and output caps
 - **Tool call repair** — fixes malformed JSON from small models (trailing commas, single quotes, unquoted keys)
 - **Loop detection** — catches repeated identical tool calls and nudges the LLM to try a different approach
 - **History summarization** — compresses long conversations to stay within context
+- **[Task-eval harness](evals/)** — measures success rate, iterations, tokens, and JSON-repair fire-rate across model tiers so the tuning above is testable, not hand-waved
 
 ---
 
@@ -284,6 +290,7 @@ Key paths:
 | `agent-mini init` | Interactive setup wizard |
 | `agent-mini chat` | Interactive chat |
 | `agent-mini chat -m "..."` | Single message |
+| `agent-mini chat --workspace <dir>` | Override the workspace for a single run (also honours `AGENT_MINI_WORKSPACE`) |
 | `agent-mini gateway` | Start Telegram bot |
 | `agent-mini status` | Show config status |
 
@@ -324,6 +331,21 @@ uv run pytest tests/ -v
 uv run ruff check src/ tests/
 ```
 
+### Task evals
+
+The [`evals/`](evals/) directory contains a small, framework-free task-eval
+harness that runs the agent end-to-end against real fixtures (refactor a
+codebase, fix a failing test, answer a codebase question, etc.). Use it to
+measure whether the small-model optimizations actually pay off:
+
+```bash
+python evals/run.py                        # run all tasks against config model
+python evals/run.py --task refactor_rename # single task
+python evals/run.py --compare results/*.json  # cross-tier comparison table
+```
+
+See [evals/README.md](evals/README.md) for the full workflow.
+
 See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 
 ## License

{agent_mini-0.2.0 → agent_mini-0.3.0}/README.md

@@ -11,8 +11,9 @@ A minimal, local-first AI agent you can actually understand and extend.
 - **Local-first** — Ollama as default, OpenAI if you want cloud, or any OpenAI-compatible server
 - **Zero frameworks** — pure `httpx` + `asyncio`, no LangChain, no LiteLLM
 - **Built-in tools** — shell, files, web search, persistent memory
+- **Vision** — drop an image path or URL into any message; works on Ollama, OpenAI, and OpenAI-compatible providers
 - **Extensible** — drop a Python file in `~/.agent-mini/plugins/` and it's a tool
-- **Small-model optimized** — token-aware context pruning, tool call repair, model-tier tuning
+- **Small-model optimized** — tier-scaled system prompt, token-aware context pruning, tool-call repair, and a [task-eval harness](evals/) to measure whether the tuning actually pays off
 
 ## Quick Start
 
@@ -98,7 +99,7 @@ Available out of the box — no API keys needed:
 | `list_directory` | Browse filesystem |
 | `search_files` | Grep / ripgrep across files |
 | `web_search` | DuckDuckGo search (free, no key) |
-| `web_fetch` | Fetch any URL as plain text |
+| `web_fetch` | Fetch a public URL as plain text (private/loopback hosts blocked) |
 | `memory_store` | Save to persistent memory |
 | `memory_recall` | Fuzzy search memory (TF-IDF) |
 
@@ -179,7 +180,9 @@ Control tool access:
 { "tools": { "sandboxLevel": "readonly" } }
 ```
 
-Dangerous shell commands (`rm -rf`, `sudo`, `mkfs`, etc.) are blocked by default.
+Dangerous shell commands (`rm -rf`, `sudo`, `mkfs`, etc.) are blocked by default. Treat the shell blocklist as **friction, not a boundary** — a determined caller can bypass it, so don't run the agent in an untrusted context and expect the blocklist to save you.
+
+`web_fetch` is guarded against basic SSRF: requests to `localhost`, loopback (`::1`), RFC-1918 private ranges, and link-local (including the cloud metadata endpoint `169.254.169.254`) are refused, and the final URL is re-checked after every redirect. DNS rebinding can still bypass — use `sandboxLevel: readonly` if you need a stronger guarantee.
 
 ---
 
@@ -201,11 +204,14 @@ Agent Mini is a **ReAct loop** — the LLM reasons, picks a tool, observes the r
 
 Key design choices for small/local models:
 
+- **Tier-scaled system prompt** — tiny models get a compact rules block and no memory recall; larger tiers get the full descriptive prompt and recent context
+- **Inline tool list** — `<available_tools>` block in the system prompt so small models can see tool names at a glance without inferring from the JSON schema
 - **Token-aware context** — estimates token usage and prunes old tool results when approaching the model's effective context window
-- **Model tier classification** — auto-detects tiny/small/medium/cloud models and adjusts context budgets, iteration limits, and output caps
+- **Model tier classification** — auto-detects tiny/small/medium/cloud (including large open-weight sizes like `:32b`, `:70b`, `8x7b`) and adjusts context budgets, iteration limits, and output caps
 - **Tool call repair** — fixes malformed JSON from small models (trailing commas, single quotes, unquoted keys)
 - **Loop detection** — catches repeated identical tool calls and nudges the LLM to try a different approach
 - **History summarization** — compresses long conversations to stay within context
+- **[Task-eval harness](evals/)** — measures success rate, iterations, tokens, and JSON-repair fire-rate across model tiers so the tuning above is testable, not hand-waved
 
 ---
 
@@ -249,6 +255,7 @@ Key paths:
 | `agent-mini init` | Interactive setup wizard |
 | `agent-mini chat` | Interactive chat |
 | `agent-mini chat -m "..."` | Single message |
+| `agent-mini chat --workspace <dir>` | Override the workspace for a single run (also honours `AGENT_MINI_WORKSPACE`) |
 | `agent-mini gateway` | Start Telegram bot |
 | `agent-mini status` | Show config status |
 
@@ -289,6 +296,21 @@ uv run pytest tests/ -v
 uv run ruff check src/ tests/
 ```
 
+### Task evals
+
+The [`evals/`](evals/) directory contains a small, framework-free task-eval
+harness that runs the agent end-to-end against real fixtures (refactor a
+codebase, fix a failing test, answer a codebase question, etc.). Use it to
+measure whether the small-model optimizations actually pay off:
+
+```bash
+python evals/run.py                        # run all tasks against config model
+python evals/run.py --task refactor_rename # single task
+python evals/run.py --compare results/*.json  # cross-tier comparison table
+```
+
+See [evals/README.md](evals/README.md) for the full workflow.
+
 See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 
 ## License

agent_mini-0.3.0/evals/README.md

@@ -0,0 +1,154 @@
+# Agent Mini — Task Evals
+
+A tiny, framework-free harness that measures whether the agent can complete
+**real-world tasks** end-to-end, not just pass unit tests. This is where the
+"small-model optimized" claim gets tested for real.
+
+Zero new dependencies. Uses only the standard library + `agent-mini` itself.
+
+---
+
+## Quick start
+
+```bash
+# 1. Have Ollama (or any provider) already running with a model
+ollama pull llama3.1:8b
+
+# 2. Point agent-mini at it (only needed once)
+agent-mini init
+
+# 3. Run the whole suite
+python evals/run.py
+
+# 4. Run against a specific model
+python evals/run.py --model llama3.2:3b
+
+# 5. Run a single task
+python evals/run.py --task refactor_rename
+
+# 6. Compare tiers (the thesis test)
+python evals/run.py --model llama3.2:3b   --out results/tiny.json
+python evals/run.py --model llama3.1:8b   --out results/small.json
+python evals/run.py --model qwen2.5:14b   --out results/medium.json
+python evals/run.py --compare results/*.json
+```
+
+Results are printed as a table and written to `results/<model>_<timestamp>.json`
+(one line per task, so you can `jq` / `grep` them).
+
+---
+
+## Latest results — qwen3.5 tiers (Ollama, 2026-07-28)
+
+Local M-series Mac, `think: false`, default agent config (`maxIterations: 20`).
+
+| Task                       | qwen3.5:4b   | qwen3.5:9b   | qwen3.5:27b   |
+|----------------------------|--------------|--------------|---------------|
+| codebase_qa                | PASS · 18.0s | PASS · 23.8s | PASS · 81.5s  |
+| error_recovery             | PASS ·  8.7s | PASS · 19.5s | PASS · 50.3s  |
+| fix_failing_test           | PASS · 17.9s | PASS · 24.9s | PASS · 85.3s  |
+| list_directory_basic       | FAIL ·  8.3s | PASS · 14.0s | PASS · 47.5s  |
+| multi_file_edit            | PASS ·  9.5s | PASS · 18.3s | PASS · 78.4s  |
+| refactor_rename            | FAIL · 10.1s | FAIL · 61.0s | PASS · 184.9s |
+| search_and_summarize       | PASS ·  6.2s | PASS · 23.2s | PASS · 32.0s  |
+| shell_pipeline             | FAIL · 12.6s | PASS · 32.2s | FAIL · 120.0s (timeout) |
+| write_file_from_scratch    | FAIL · 11.1s | FAIL · 15.7s | PASS · 34.7s  |
+| **Pass rate**              | **5/9**      | **7/9**      | **8/9**       |
+
+Notes:
+
+- `qwen3.5:9b` is the sweet spot on this suite
+  `shell_pipeline` and finishing every task well under the timeout.
+- `write_file_from_scratch` also fails everywhere: models emit a `print`
+  / `__main__` block despite the prompt forbidding it. Good candidate for
+  a small-model instruction-tuning improvement.
+---
+
+## Layout
+
+```
+evals/
+├── README.md          # this file
+├── run.py             # ~200 lines: copy fixture → run agent headless → score
+├── tasks/*.yaml       # task fixtures — prompt + programmatic check
+└── fixtures/          # tiny sample repos used by tasks
+```
+
+Each task is a self-contained YAML file:
+
+```yaml
+id: refactor_rename
+category: refactor
+setup: fixtures/py_project
+prompt: |
+  Rename the function `calc` to `compute` across all .py files in this
+  workspace and make sure the tests still pass.
+timeout_seconds: 180
+checks:
+  - shell: "! grep -rq 'def calc(' ."
+    description: "No `def calc` remains"
+  - shell: "grep -rq 'def compute(' ."
+    description: "`def compute` appears somewhere"
+  - shell: "python -m pytest -q"
+    description: "Tests still pass"
+```
+
+Add a task: drop a YAML file in `tasks/` and (optionally) a matching
+fixture directory in `fixtures/`. The runner picks it up on next run.
+
+---
+
+## Task categories (the surface we actually care about)
+
+| Category | Exercises | Task ID |
+|----------|-----------|---------|
+| File ops | `write_file`, `code_edit`, multi-file rename | `refactor_rename` |
+| Codebase Q&A | `search_files` → `read_file` → summarize | `codebase_qa` |
+| Multi-step | write, run, verify a fix | `fix_failing_test` |
+| Error recovery | retry a different way after failure | `error_recovery` |
+| Memory | store fact turn 1, recall turn 3 (uses `--multi-turn`) | `memory_recall` |
+| Search | grep + read | `search_and_summarize` |
+
+---
+
+## Metrics (per model)
+
+The runner tabulates:
+
+| Metric | Why it matters |
+|--------|----------------|
+| Success rate | Core efficacy. |
+| Median iterations | Loop efficiency. Wandering = bad. |
+| JSON-repair fire rate | Is small-model tool-call repair earning its keep? |
+| Summarize trigger rate | Is context management engaging when needed? |
+| Loop-detection hits | Are small models stalling? |
+| Total tokens / wall time | Cost. |
+
+The **thesis test**: run the same suite across `tiny` / `small` / `medium` /
+`cloud` tiers. If tuned `3B` ≈ untuned `8B`, the optimization is proven. If
+not, it's theater — and this harness is exactly what surfaces that.
+
+---
+
+## Design constraints
+
+- **No new dependencies.** stdlib only (`subprocess`, `shutil`, `tempfile`,
+  `json`, `re`, `argparse`). YAML is parsed by a tiny in-file loader — the
+  task schema is intentionally flat enough not to need PyYAML.
+- **No mocking.** Real provider, real tools, real filesystem. Fixtures are
+  copied to a fresh `tempfile.mkdtemp()` per task, so runs don't leak.
+- **Zero framework bloat.** ~200 LOC in `run.py`. Read it in five minutes.
+- **CI-friendly.** Gate as nightly against a live model; keep unit tests as
+  the fast job. Model latency should not block PRs.
+
+## Adding a task
+
+1. Create `evals/fixtures/my_fixture/` with a minimal set of files.
+2. Create `evals/tasks/my_task.yaml`:
+   - `id`: matches the file stem.
+   - `setup`: path to the fixture (relative to `evals/`).
+   - `prompt`: what to ask the agent.
+   - `checks`: shell one-liners that must exit `0`.
+3. Verify: `python evals/run.py --task my_task`.
+
+Each bug we fix should get a new task. The corpus grows organically.

agent_mini-0.3.0/evals/fixtures/buggy_stack/pytest.ini

@@ -0,0 +1,2 @@
+[pytest]
+testpaths = tests

agent_mini-0.3.0/evals/fixtures/buggy_stack/src/stack.py

@@ -0,0 +1,22 @@
+"""A buggy stack implementation. The `pop` method is off-by-one — the eval
+asks the agent to find and fix the bug so the tests pass."""
+
+
+class Stack:
+    def __init__(self):
+        self._items = []
+
+    def push(self, x):
+        self._items.append(x)
+
+    def pop(self):
+        if not self._items:
+            raise IndexError("pop from empty stack")
+        # BUG: returns the wrong end of the list
+        return self._items.pop(0)
+
+    def peek(self):
+        return self._items[-1]
+
+    def __len__(self):
+        return len(self._items)

agent_mini-0.3.0/evals/fixtures/buggy_stack/tests/test_stack.py

@@ -0,0 +1,24 @@
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
+
+from stack import Stack  # noqa: E402
+
+
+def test_push_pop_is_lifo():
+    s = Stack()
+    s.push(1)
+    s.push(2)
+    s.push(3)
+    assert s.pop() == 3
+    assert s.pop() == 2
+    assert s.pop() == 1
+
+
+def test_peek_does_not_mutate():
+    s = Stack()
+    s.push("a")
+    s.push("b")
+    assert s.peek() == "b"
+    assert len(s) == 2

agent_mini-0.3.0/evals/fixtures/py_project/pytest.ini

@@ -0,0 +1,2 @@
+[pytest]
+testpaths = tests

agent_mini-0.3.0/evals/fixtures/py_project/src/mymath.py

@@ -0,0 +1,11 @@
+"""A tiny arithmetic helper — used by the eval fixture."""
+
+
+def calc(a: int, b: int) -> int:
+    """Add two ints. The eval will ask the agent to rename this."""
+    return a + b
+
+
+def double_calc(x: int) -> int:
+    """Uses `calc` — the rename must update the call site too."""
+    return calc(x, x)

agent_mini-0.3.0/evals/fixtures/py_project/tests/test_mymath.py

@@ -0,0 +1,16 @@
+"""Tests for mymath — must still pass after the rename."""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
+
+from mymath import calc, double_calc  # noqa: E402
+
+
+def test_calc_adds():
+    assert calc(2, 3) == 5
+
+
+def test_double_uses_calc():
+    assert double_calc(4) == 8

agent_mini-0.3.0/evals/fixtures/small_repo/README.md

@@ -0,0 +1,5 @@
+# Wonderland Widget
+
+A small internal library shared by three teams. The project's port is **9137**.
+
+Owner: alice@example.org

agent_mini-0.3.0/evals/fixtures/small_repo/src/config.py

@@ -0,0 +1,2 @@
+SERVICE_NAME = "wonderland-widget"
+VERSION = "0.4.2"

agent_mini-0.3.0/evals/fixtures/small_repo/src/server.py

@@ -0,0 +1,8 @@
+"""Wonderland widget server."""
+
+PORT = 9137
+HOST = "0.0.0.0"
+
+
+def start_server():
+    print(f"Serving on {HOST}:{PORT}")