kidwit 0.1.0rc1__tar.gz

kidwit-0.1.0rc1/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+All notable changes to kidwit are recorded here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow
+[Semantic Versioning](https://semver.org/). Until 1.0, a minor version may change behaviour.
+
+## [Unreleased]
+
+## [0.1.0] - not yet released
+
+First release.
+
+### Known limitations
+- Tested live only with the OpenAI-compatible adapter (DeepSeek) on macOS with Python 3.14.
+- The Anthropic adapter is covered by offline tests but has not been run against the live API.
+- Windows is untested. Linux and Python 3.9 to 3.13 are untested until CI runs.
+
+### Added
+- A coding agent in one file of standard-library Python, with Anthropic and OpenAI-compatible providers.
+- Six tools: `list_files`, `search`, `read_file`, `write_file`, `edit_file` and `run`.
+- Safety: a path jail for the file tools, an approval prompt before every write, edit and command,
+  `--read-only`, a guard that always asks for dangerous commands, `/undo`, and an audit log.
+- `--json` event protocol for building a UI on top of the engine.
+- Context management: old tool output is dropped and, if needed, the older conversation is summarized.
+- A small eval suite in `evals/`.

kidwit-0.1.0rc1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SM Sabbir Amin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

kidwit-0.1.0rc1/MANIFEST.in

@@ -0,0 +1,6 @@
+include LICENSE README.md CHANGELOG.md
+graft tests
+graft evals
+prune evals/results
+include scripts/check.sh
+global-exclude __pycache__ *.py[cod] .DS_Store

kidwit-0.1.0rc1/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: kidwit
+Version: 0.1.0rc1
+Summary: A minimal coding agent in one file of standard-library Python. It works inside one directory and asks before it changes anything.
+Author-email: SM Sabbir Amin <sabbiramin.cse11ruet@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://kidwit.sabbiramin.dev
+Project-URL: Source, https://github.com/sabbiramin113008/kidwit
+Project-URL: Issues, https://github.com/sabbiramin113008/kidwit/issues
+Project-URL: Changelog, https://github.com/sabbiramin113008/kidwit/blob/main/CHANGELOG.md
+Keywords: coding-agent,ai-agent,llm,cli,anthropic,openai,deepseek,ollama
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# kidwit
+
+A complete coding-agent engine in one file of standard-library Python, around 500 lines of code. It works inside one directory and asks before it changes anything.
+
+kidwit reads your code, edits files, runs commands and reports back, using native tool calling with Anthropic or any OpenAI-compatible API. It is a from-scratch build and a case study in how little a coding agent needs. It is not a replacement for mature agents.
+
+## Install
+
+You need Python 3.9 or newer. kidwit has no dependencies. From a clone of this repository:
+
+```
+pipx install .            # or: pip install .
+export ANTHROPIC_API_KEY=sk-ant-...
+kidwit "fix the failing test"
+```
+
+The whole engine is one file, so you can also run it with no install at all:
+
+```
+python3 src/kidwit.py "fix the failing test"
+```
+
+Without a task, kidwit starts an interactive session and, on a color terminal, shows a small banner. The banner never appears with `--json`, with a task on the command line, or when output is piped. `/clear` resets the conversation, `/verbose` toggles detailed output, and `/exit` (or `/quit`, or end of input) leaves.
+
+## Status
+
+kidwit is early software. This is what has and has not been tested:
+
+1. **Tested live:** the OpenAI-compatible adapter with DeepSeek (`deepseek-chat`) on macOS with Python 3.14. The 8 tasks in `evals/` all pass end to end.
+2. **Not yet run against the real service:** the Anthropic adapter. It is covered by the offline tests, but nobody has run it against the live Anthropic API yet.
+3. **Untested:** Windows. The command runner uses Python's default shell there (`cmd.exe`), and killing a timed-out command's child processes is only verified on macOS and Linux-style systems. Linux and Python 3.9 to 3.13 are also untested for now.
+
+If you try it somewhere on this list, please open an issue with what happened.
+
+## Configuration
+
+Environment variables:
+
+| Variable | Meaning |
+|---|---|
+| `ANTHROPIC_API_KEY` | Key for the Anthropic Messages API. If set, Anthropic is the default provider. |
+| `OPENAI_API_KEY` | Key for an OpenAI-compatible API. Optional for local servers. |
+| `AGENT_PROVIDER` | `anthropic` or `openai`. |
+| `AGENT_MODEL` | Model name. Defaults to `claude-sonnet-5-5` for Anthropic. Required for `openai`. |
+| `AGENT_BASE_URL` | API base URL, for OpenRouter, Ollama, LM Studio, vLLM or a proxy. |
+
+Provider choice follows this order: `--provider`, then `AGENT_PROVIDER`, then Anthropic if `ANTHROPIC_API_KEY` is set, then OpenAI-compatible.
+
+Local model example:
+
+```
+AGENT_BASE_URL=http://localhost:11434/v1 AGENT_MODEL=qwen2.5-coder kidwit "add a docstring to main()"
+```
+
+Flags:
+
+| Flag | Meaning |
+|---|---|
+| `--root DIR` | Project directory. Default: the current directory. |
+| `--provider anthropic\|openai` | Choose the API. |
+| `--model M`, `--base-url URL` | Override `AGENT_MODEL` and `AGENT_BASE_URL`. |
+| `--yes` | Never ask for approval. |
+| `--read-only` | Offer only the read tools. Writes and commands are not available. |
+| `--json` | Print JSON-lines events instead of text. |
+| `--verbose` | Show live command output and full tool arguments. By default kidwit prints one short dim line per tool call and then the answer. |
+| `--no-log` | Do not write `.agent/audit.jsonl`. |
+| `--version` | Print the version. |
+
+A task given on the command line runs once. The exit code is 0 on success and 1 on error, so it works in scripts and benchmarks.
+
+## Reading the terminal
+
+On a color terminal each kind of text looks different. Your prompt is a bold green `>>`. The model's replies have a magenta `▌` gutter. Tool calls are one dim line with the tool name in cyan. Permission requests start with a yellow `┌ permission needed` line, followed by the diff or command and a yellow `└ allow?` prompt. Errors are red. With `NO_COLOR`, piped output or `--json`, none of this styling is printed.
+
+## Tools
+
+The model gets six tools: `list_files`, `search`, `read_file`, `write_file`, `edit_file` and `run`. Inside a git repository, `list_files` and `search` honour `.gitignore`, so build output and similar files stay out of the way. Edits are exact-string replacement: a match that differs only in trailing whitespace is accepted, and when nothing matches the error names the closest line so the model can correct itself. `old_text` must match exactly once unless `replace_all` is set, and files with Windows line endings keep them.
+
+If the project root has an `AGENTS.md` (or else a `CLAUDE.md`), its first 20,000 characters are added to the system prompt.
+
+## Approval modes
+
+1. `ask` (default). Before every `write_file`, `edit_file` and `run`, kidwit shows the diff or command and waits for `y`, `a`, `n`, or any other text. Other text is sent to the model as the reason for the denial. `a` (always) approves that kind of action for the rest of the session: file changes, or commands, but not both.
+2. `auto` (`--yes`). No prompts, except for dangerous commands.
+3. `read-only` (`--read-only`). The write tools are not offered to the model, and are refused if called anyway.
+
+Some things are handled for you in every mode:
+
+1. In `ask` mode, harmless commands run without a prompt: `ls`, `pwd`, `git status`, `git diff`, `git log`, `git show`, `git branch`, and test runners such as `python3 -m unittest`, `pytest`, `npm test`, `go test` and `cargo test`. They must use plain relative arguments with no shell operators. `cat`, `grep` and `head` are not on the list on purpose: the path jail does not cover `run`, so they could read files outside the project.
+2. Dangerous commands always ask, even with `--yes` or after `a`: `rm -rf`, `sudo`, `curl ... | sh`, `git reset --hard`, `git clean`, `git push`, `chmod -R` and similar. With no one to answer (for example `--json` with closed stdin) they are denied.
+3. `/undo` in an interactive session reverts the last file write or edit. It does not undo commands.
+4. If the model's reply is cut off at the token limit, the half-written tool call is not run. The model is told to do less per call.
+
+Every write, edit and command is appended to `<root>/.agent/audit.jsonl`. kidwit creates `.agent/.gitignore` so the log stays out of git.
+
+## Limits you should know about
+
+The path jail covers the file tools only. `read_file`, `write_file` and `edit_file` refuse paths outside the project root, including `../` and symlinks that lead out, and `list_files` and `search` skip such symlinks. `run` executes real shell commands, and those can reach anything your user account can reach. The protections in v1 are the approval prompt and `--read-only`. There is no OS-level sandbox, so review each command before you approve it, and avoid `--yes` on a machine or directory you care about.
+
+Long sessions are kept within the model's context in two steps. First, old tool output is dropped. If the history is still too large (over about 200,000 characters), kidwit asks the model to summarize the older half of the conversation and continues from that summary.
+
+Other deliberate omissions: no streaming of model responses (command output does stream), no sub-agents, no MCP, no saved sessions, and no UI beyond plain terminal text.
+
+## Event protocol
+
+With `--json`, kidwit writes one JSON object per line on stdout. Tasks and approval answers are read from stdin, one line each. End of input during an approval counts as "no". A UI can drive the engine through this protocol.
+
+| `type` | Fields |
+|---|---|
+| `info` | `message` |
+| `text` | `text` |
+| `tool` | `id`, `name`, `input` |
+| `output` | `line` (live command output) |
+| `approval` | `action` (`write`, `edit` or `run`), `target`, `detail` (diff or command) |
+| `result` | `id`, `output`, `is_error` |
+| `done` | `turns`, `input_tokens`, `output_tokens` |
+| `error` | `message` |
+
+## Tests
+
+The tests use only `unittest` and need no API key. A fake local HTTP server stands in for both APIs.
+
+```
+scripts/check.sh                              # everything below in one go
+python3 -m unittest discover -s tests -t .    # just the tests
+```
+
+`scripts/check.sh` checks the Python version, compiles `src/kidwit.py`, confirms it imports only the standard library, runs the tests, and checks the size (`grep -vcE '^\s*(#.*)?$' src/kidwit.py`). Over 500 lines prints a warning, and over 700 fails.
+
+## Evals
+
+`evals/` holds 8 small tasks, each a tiny repo with a failing check: an off-by-one, a name error, implementing a function, a root cause in a second file, a rename across files, a CRLF file, a missing config default, and a new CLI flag. They show whether a change to kidwit makes it better or worse.
+
+```
+python3 evals/run_evals.py --selftest     # checks the tasks themselves, no model needed
+python3 evals/run_evals.py                # runs every task against your configured model
+python3 evals/run_evals.py --task crlf-file -- --provider openai --model deepseek-chat
+```
+
+The runner prints pass or fail, turns, tokens and time per task, and saves the results to `evals/results/`. Be aware that it runs the agent with `--yes`, so the model's commands run as you with no sandbox. Dangerous commands still need approval, and because the runner closes stdin they are denied. Run it only where that is acceptable.
+
+## License
+
+MIT.

kidwit-0.1.0rc1/README.md

@@ -0,0 +1,144 @@
+# kidwit
+
+A complete coding-agent engine in one file of standard-library Python, around 500 lines of code. It works inside one directory and asks before it changes anything.
+
+kidwit reads your code, edits files, runs commands and reports back, using native tool calling with Anthropic or any OpenAI-compatible API. It is a from-scratch build and a case study in how little a coding agent needs. It is not a replacement for mature agents.
+
+## Install
+
+You need Python 3.9 or newer. kidwit has no dependencies. From a clone of this repository:
+
+```
+pipx install .            # or: pip install .
+export ANTHROPIC_API_KEY=sk-ant-...
+kidwit "fix the failing test"
+```
+
+The whole engine is one file, so you can also run it with no install at all:
+
+```
+python3 src/kidwit.py "fix the failing test"
+```
+
+Without a task, kidwit starts an interactive session and, on a color terminal, shows a small banner. The banner never appears with `--json`, with a task on the command line, or when output is piped. `/clear` resets the conversation, `/verbose` toggles detailed output, and `/exit` (or `/quit`, or end of input) leaves.
+
+## Status
+
+kidwit is early software. This is what has and has not been tested:
+
+1. **Tested live:** the OpenAI-compatible adapter with DeepSeek (`deepseek-chat`) on macOS with Python 3.14. The 8 tasks in `evals/` all pass end to end.
+2. **Not yet run against the real service:** the Anthropic adapter. It is covered by the offline tests, but nobody has run it against the live Anthropic API yet.
+3. **Untested:** Windows. The command runner uses Python's default shell there (`cmd.exe`), and killing a timed-out command's child processes is only verified on macOS and Linux-style systems. Linux and Python 3.9 to 3.13 are also untested for now.
+
+If you try it somewhere on this list, please open an issue with what happened.
+
+## Configuration
+
+Environment variables:
+
+| Variable | Meaning |
+|---|---|
+| `ANTHROPIC_API_KEY` | Key for the Anthropic Messages API. If set, Anthropic is the default provider. |
+| `OPENAI_API_KEY` | Key for an OpenAI-compatible API. Optional for local servers. |
+| `AGENT_PROVIDER` | `anthropic` or `openai`. |
+| `AGENT_MODEL` | Model name. Defaults to `claude-sonnet-5-5` for Anthropic. Required for `openai`. |
+| `AGENT_BASE_URL` | API base URL, for OpenRouter, Ollama, LM Studio, vLLM or a proxy. |
+
+Provider choice follows this order: `--provider`, then `AGENT_PROVIDER`, then Anthropic if `ANTHROPIC_API_KEY` is set, then OpenAI-compatible.
+
+Local model example:
+
+```
+AGENT_BASE_URL=http://localhost:11434/v1 AGENT_MODEL=qwen2.5-coder kidwit "add a docstring to main()"
+```
+
+Flags:
+
+| Flag | Meaning |
+|---|---|
+| `--root DIR` | Project directory. Default: the current directory. |
+| `--provider anthropic\|openai` | Choose the API. |
+| `--model M`, `--base-url URL` | Override `AGENT_MODEL` and `AGENT_BASE_URL`. |
+| `--yes` | Never ask for approval. |
+| `--read-only` | Offer only the read tools. Writes and commands are not available. |
+| `--json` | Print JSON-lines events instead of text. |
+| `--verbose` | Show live command output and full tool arguments. By default kidwit prints one short dim line per tool call and then the answer. |
+| `--no-log` | Do not write `.agent/audit.jsonl`. |
+| `--version` | Print the version. |
+
+A task given on the command line runs once. The exit code is 0 on success and 1 on error, so it works in scripts and benchmarks.
+
+## Reading the terminal
+
+On a color terminal each kind of text looks different. Your prompt is a bold green `>>`. The model's replies have a magenta `▌` gutter. Tool calls are one dim line with the tool name in cyan. Permission requests start with a yellow `┌ permission needed` line, followed by the diff or command and a yellow `└ allow?` prompt. Errors are red. With `NO_COLOR`, piped output or `--json`, none of this styling is printed.
+
+## Tools
+
+The model gets six tools: `list_files`, `search`, `read_file`, `write_file`, `edit_file` and `run`. Inside a git repository, `list_files` and `search` honour `.gitignore`, so build output and similar files stay out of the way. Edits are exact-string replacement: a match that differs only in trailing whitespace is accepted, and when nothing matches the error names the closest line so the model can correct itself. `old_text` must match exactly once unless `replace_all` is set, and files with Windows line endings keep them.
+
+If the project root has an `AGENTS.md` (or else a `CLAUDE.md`), its first 20,000 characters are added to the system prompt.
+
+## Approval modes
+
+1. `ask` (default). Before every `write_file`, `edit_file` and `run`, kidwit shows the diff or command and waits for `y`, `a`, `n`, or any other text. Other text is sent to the model as the reason for the denial. `a` (always) approves that kind of action for the rest of the session: file changes, or commands, but not both.
+2. `auto` (`--yes`). No prompts, except for dangerous commands.
+3. `read-only` (`--read-only`). The write tools are not offered to the model, and are refused if called anyway.
+
+Some things are handled for you in every mode:
+
+1. In `ask` mode, harmless commands run without a prompt: `ls`, `pwd`, `git status`, `git diff`, `git log`, `git show`, `git branch`, and test runners such as `python3 -m unittest`, `pytest`, `npm test`, `go test` and `cargo test`. They must use plain relative arguments with no shell operators. `cat`, `grep` and `head` are not on the list on purpose: the path jail does not cover `run`, so they could read files outside the project.
+2. Dangerous commands always ask, even with `--yes` or after `a`: `rm -rf`, `sudo`, `curl ... | sh`, `git reset --hard`, `git clean`, `git push`, `chmod -R` and similar. With no one to answer (for example `--json` with closed stdin) they are denied.
+3. `/undo` in an interactive session reverts the last file write or edit. It does not undo commands.
+4. If the model's reply is cut off at the token limit, the half-written tool call is not run. The model is told to do less per call.
+
+Every write, edit and command is appended to `<root>/.agent/audit.jsonl`. kidwit creates `.agent/.gitignore` so the log stays out of git.
+
+## Limits you should know about
+
+The path jail covers the file tools only. `read_file`, `write_file` and `edit_file` refuse paths outside the project root, including `../` and symlinks that lead out, and `list_files` and `search` skip such symlinks. `run` executes real shell commands, and those can reach anything your user account can reach. The protections in v1 are the approval prompt and `--read-only`. There is no OS-level sandbox, so review each command before you approve it, and avoid `--yes` on a machine or directory you care about.
+
+Long sessions are kept within the model's context in two steps. First, old tool output is dropped. If the history is still too large (over about 200,000 characters), kidwit asks the model to summarize the older half of the conversation and continues from that summary.
+
+Other deliberate omissions: no streaming of model responses (command output does stream), no sub-agents, no MCP, no saved sessions, and no UI beyond plain terminal text.
+
+## Event protocol
+
+With `--json`, kidwit writes one JSON object per line on stdout. Tasks and approval answers are read from stdin, one line each. End of input during an approval counts as "no". A UI can drive the engine through this protocol.
+
+| `type` | Fields |
+|---|---|
+| `info` | `message` |
+| `text` | `text` |
+| `tool` | `id`, `name`, `input` |
+| `output` | `line` (live command output) |
+| `approval` | `action` (`write`, `edit` or `run`), `target`, `detail` (diff or command) |
+| `result` | `id`, `output`, `is_error` |
+| `done` | `turns`, `input_tokens`, `output_tokens` |
+| `error` | `message` |
+
+## Tests
+
+The tests use only `unittest` and need no API key. A fake local HTTP server stands in for both APIs.
+
+```
+scripts/check.sh                              # everything below in one go
+python3 -m unittest discover -s tests -t .    # just the tests
+```
+
+`scripts/check.sh` checks the Python version, compiles `src/kidwit.py`, confirms it imports only the standard library, runs the tests, and checks the size (`grep -vcE '^\s*(#.*)?$' src/kidwit.py`). Over 500 lines prints a warning, and over 700 fails.
+
+## Evals
+
+`evals/` holds 8 small tasks, each a tiny repo with a failing check: an off-by-one, a name error, implementing a function, a root cause in a second file, a rename across files, a CRLF file, a missing config default, and a new CLI flag. They show whether a change to kidwit makes it better or worse.
+
+```
+python3 evals/run_evals.py --selftest     # checks the tasks themselves, no model needed
+python3 evals/run_evals.py                # runs every task against your configured model
+python3 evals/run_evals.py --task crlf-file -- --provider openai --model deepseek-chat
+```
+
+The runner prints pass or fail, turns, tokens and time per task, and saves the results to `evals/results/`. Be aware that it runs the agent with `--yes`, so the model's commands run as you with no sandbox. Dangerous commands still need approval, and because the runner closes stdin they are denied. Run it only where that is acceptable.
+
+## License
+
+MIT.

kidwit-0.1.0rc1/evals/run_evals.py

@@ -0,0 +1,146 @@
+#!/usr/bin/env python3
+"""Score kidwit on small coding tasks (standard library only).
+
+  python3 evals/run_evals.py --selftest                    check the tasks themselves, no model needed
+  python3 evals/run_evals.py                               run every task against the configured model
+  python3 evals/run_evals.py --task off-by-one --task crlf-file
+  python3 evals/run_evals.py -- --provider openai --model deepseek-chat --base-url URL
+
+Arguments after `--` go to kidwit. The agent runs with --yes inside a throwaway directory,
+but its commands are not sandboxed: they run as you. Dangerous commands still need approval,
+and stdin is closed, so they are denied. Run this only on a machine where that is acceptable.
+Results are printed and saved to evals/results/<timestamp>.json.
+"""
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE.parent))
+from evals.tasks import TASKS  # noqa: E402
+
+AGENT = HERE.parent / "src" / "kidwit.py"
+
+
+def write_files(root, files):
+    for name, content in files.items():
+        (root / name).parent.mkdir(parents=True, exist_ok=True)
+        (root / name).write_bytes(content.encode("utf-8"))  # bytes keep CRLF exactly
+
+
+def verify(root, task):
+    env = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}  # a stale .pyc must not hide a fix
+    proc = subprocess.run(task["verify"], shell=True, cwd=root, capture_output=True, text=True, timeout=120, env=env)
+    return proc.returncode == 0, (proc.stdout + proc.stderr).strip()[-400:]
+
+
+def selftest(tasks):
+    """Every task must fail as shipped and pass once its reference solution is applied."""
+    bad = 0
+    for task in tasks:
+        root = Path(tempfile.mkdtemp(prefix="kidwit-selftest-"))
+        try:
+            write_files(root, task["files"])
+            before, _ = verify(root, task)
+            write_files(root, task["solution"])
+            after, detail = verify(root, task)
+            ok = (not before) and after
+            bad += not ok
+            print(f"{'ok  ' if ok else 'FAIL'} {task['id']:<24} fails before: {not before}  passes after: {after}")
+            if not ok and detail:
+                print("     " + detail.replace("\n", "\n     "))
+        finally:
+            shutil.rmtree(root, ignore_errors=True)
+    return 1 if bad else 0
+
+
+def run_task(task, agent_args, timeout, keep):
+    root = Path(tempfile.mkdtemp(prefix=f"kidwit-eval-{task['id']}-"))
+    write_files(root, task["files"])
+    cmd = [sys.executable, str(AGENT), "--root", str(root), "--yes", "--no-log", "--json",
+           *agent_args, task["prompt"]]
+    result = {"id": task["id"], "passed": False, "turns": 0, "input_tokens": 0, "output_tokens": 0,
+              "tool_calls": 0, "tool_errors": 0, "error": None, "seconds": 0.0,
+              "calls": [], "tool_error_messages": [], "approvals_asked": []}
+    start = time.time()
+    try:
+        proc = subprocess.run(cmd, stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=timeout)
+        for line in proc.stdout.splitlines():
+            try:
+                event = json.loads(line)
+            except ValueError:
+                continue
+            if event["type"] == "tool":
+                result["tool_calls"] += 1
+                result["calls"].append(event["name"])
+            elif event["type"] == "approval":  # in --yes mode only dangerous commands ask, and stdin is closed
+                result["approvals_asked"].append(f"{event['action']}: {event['detail'][:150]}")
+            elif event["type"] == "result" and event["is_error"]:
+                result["tool_errors"] += 1
+                result["tool_error_messages"].append(event["output"][:200])
+            elif event["type"] == "done":
+                result.update(turns=event["turns"], input_tokens=event["input_tokens"],
+                              output_tokens=event["output_tokens"])
+            elif event["type"] == "error":
+                result["error"] = event["message"][:300]
+        if proc.returncode != 0 and not result["error"]:
+            result["error"] = (proc.stderr or "agent exited with an error")[-300:]
+    except subprocess.TimeoutExpired:
+        result["error"] = f"timed out after {timeout}s"
+    result["seconds"] = round(time.time() - start, 1)
+    result["passed"] = verify(root, task)[0]
+    if keep:
+        result["kept"] = str(root)
+    else:
+        shutil.rmtree(root, ignore_errors=True)
+    return result
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(description="Run kidwit on small coding tasks.")
+    ap.add_argument("--task", action="append", help="task id to run (repeatable); default: all")
+    ap.add_argument("--selftest", action="store_true", help="check the tasks themselves; no model is called")
+    ap.add_argument("--timeout", type=int, default=600, help="seconds allowed per task")
+    ap.add_argument("--keep", action="store_true", help="keep the working directories")
+    ap.add_argument("--out", help="results file (default: evals/results/<timestamp>.json)")
+    args, agent_args = ap.parse_known_args(argv)
+    agent_args = [a for a in agent_args if a != "--"]
+    tasks = [t for t in TASKS if not args.task or t["id"] in args.task]
+    unknown = set(args.task or []) - {t["id"] for t in TASKS}
+    if unknown:
+        print("unknown task: " + ", ".join(sorted(unknown)), file=sys.stderr)
+        return 2
+    if args.selftest:
+        return selftest(tasks)
+    results = []
+    for task in tasks:
+        r = run_task(task, agent_args, args.timeout, args.keep)
+        results.append(r)
+        note = f"  {r['error']}" if r["error"] else ""
+        print(f"{'PASS' if r['passed'] else 'FAIL'} {r['id']:<24} {r['turns']:>2} turns  "
+              f"{r['input_tokens']:>7} in {r['output_tokens']:>6} out  {r['seconds']:>6.1f}s{note}", flush=True)
+        print(f"     calls: {' > '.join(r['calls']) or '(none)'}", flush=True)
+        for asked in r["approvals_asked"]:
+            print(f"     asked for approval: {asked}", flush=True)
+        for message in r["tool_error_messages"]:
+            print(f"     tool error: {message}", flush=True)
+    passed = sum(r["passed"] for r in results)
+    total = {k: sum(r[k] for r in results) for k in ("turns", "input_tokens", "output_tokens", "tool_errors")}
+    print(f"\n{passed}/{len(results)} passed | {total['turns']} turns | {total['input_tokens']} in / "
+          f"{total['output_tokens']} out tokens | {total['tool_errors']} tool errors")
+    out = Path(args.out) if args.out else HERE / "results" / (time.strftime("%Y%m%d-%H%M%S") + ".json")
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps({"agent_args": agent_args, "passed": passed, "total": len(results),
+                               "results": results}, indent=2) + "\n")
+    print(f"saved {out}")
+    return 0 if passed == len(results) else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

kidwit-0.1.0rc1/evals/tasks.py

@@ -0,0 +1,126 @@
+"""Eval tasks for kidwit. Each task is a tiny repo with a failing check.
+
+A task has: id, prompt, files (the starting repo), verify (a shell command that must exit 0
+when the task is solved) and solution (files that replace the starting ones, used only to
+check that the task itself is sound: `python3 evals/run_evals.py --selftest`).
+"""
+
+UNITTEST = "python3 -m unittest -q"
+
+TASKS = [
+    {
+        "id": "off-by-one",
+        "prompt": "The tests fail. Find the bug and fix it.",
+        "files": {
+            "calc.py": "def add(a, b):\n    return a - b\n",
+            "test_calc.py": ("import unittest\nfrom calc import add\n\n\nclass T(unittest.TestCase):\n"
+                             "    def test_add(self):\n        self.assertEqual(add(2, 3), 5)\n"),
+        },
+        "verify": f"{UNITTEST} test_calc",
+        "solution": {"calc.py": "def add(a, b):\n    return a + b\n"},
+    },
+    {
+        "id": "name-error",
+        "prompt": "test_stats crashes. Fix the code, not the test.",
+        "files": {
+            "stats.py": "def mean(values):\n    total = sum(values)\n    return totl / len(values)\n",
+            "test_stats.py": ("import unittest\nfrom stats import mean\n\n\nclass T(unittest.TestCase):\n"
+                              "    def test_mean(self):\n        self.assertEqual(mean([2, 4, 6]), 4)\n"),
+        },
+        "verify": f"{UNITTEST} test_stats",
+        "solution": {"stats.py": "def mean(values):\n    total = sum(values)\n    return total / len(values)\n"},
+    },
+    {
+        "id": "implement-slugify",
+        "prompt": "Implement slugify() in slug.py so the tests in test_slug.py pass.",
+        "files": {
+            "slug.py": "def slugify(text):\n    raise NotImplementedError\n",
+            "test_slug.py": ("import unittest\nfrom slug import slugify\n\n\nclass T(unittest.TestCase):\n"
+                             "    def test_basic(self):\n        self.assertEqual(slugify('Hello World'), 'hello-world')\n"
+                             "    def test_punctuation(self):\n"
+                             "        self.assertEqual(slugify('  Hello, World!  '), 'hello-world')\n"
+                             "    def test_runs(self):\n        self.assertEqual(slugify('a  --  b'), 'a-b')\n"),
+        },
+        "verify": f"{UNITTEST} test_slug",
+        "solution": {"slug.py": ("import re\n\n\ndef slugify(text):\n"
+                                 "    return re.sub(r'[^a-z0-9]+', '-', text.lower()).strip('-')\n")},
+    },
+    {
+        "id": "root-cause-two-files",
+        "prompt": "test_cart fails. Find the root cause and fix it.",
+        "files": {
+            "pricing.py": "def apply_discount(price, percent):\n    return price * percent / 100\n",
+            "cart.py": ("from pricing import apply_discount\n\n\ndef total(prices, percent=0):\n"
+                        "    return sum(apply_discount(p, percent) if percent else p for p in prices)\n"),
+            "test_cart.py": ("import unittest\nfrom cart import total\n\n\nclass T(unittest.TestCase):\n"
+                             "    def test_plain(self):\n        self.assertEqual(total([10, 20]), 30)\n"
+                             "    def test_discount(self):\n        self.assertEqual(total([100, 50], 10), 135)\n"),
+        },
+        "verify": f"{UNITTEST} test_cart",
+        "solution": {"pricing.py": "def apply_discount(price, percent):\n    return price * (100 - percent) / 100\n"},
+    },
+    {
+        "id": "rename-everywhere",
+        "prompt": "Rename the function fetch to load, and update every place that uses it.",
+        "files": {
+            "utils.py": "def fetch(key):\n    return {'a': 1, 'b': 2}[key]\n",
+            "one.py": "from utils import fetch\n\n\ndef first():\n    return fetch('a')\n",
+            "two.py": "import utils\n\n\ndef second():\n    return utils.fetch('b')\n",
+            "test_rename.py": ("import unittest\nimport one\nimport two\nfrom utils import load\n\n\n"
+                               "class T(unittest.TestCase):\n    def test_all(self):\n"
+                               "        self.assertEqual((one.first(), two.second(), load('a')), (1, 2, 1))\n"),
+        },
+        "verify": f"{UNITTEST} test_rename && ! grep -rw fetch --include=*.py utils.py one.py two.py",
+        "solution": {
+            "utils.py": "def load(key):\n    return {'a': 1, 'b': 2}[key]\n",
+            "one.py": "from utils import load\n\n\ndef first():\n    return load('a')\n",
+            "two.py": "import utils\n\n\ndef second():\n    return utils.load('b')\n",
+        },
+    },
+    {
+        "id": "crlf-file",
+        "prompt": "test_greet fails. Fix greet.py and keep its Windows line endings.",
+        "files": {
+            "greet.py": "def greet(name):\r\n    return 'Hello ' + nme\r\n",
+            "test_greet.py": ("import unittest\nfrom greet import greet\n\n\nclass T(unittest.TestCase):\n"
+                              "    def test_greet(self):\n        self.assertEqual(greet('Ana'), 'Hello Ana')\n"),
+        },
+        "verify": (f"{UNITTEST} test_greet && python3 -c \"d = open('greet.py', 'rb').read(); "
+                   "assert b'\\r\\n' in d and d.count(b'\\n') == d.count(b'\\r\\n')\""),
+        "solution": {"greet.py": "def greet(name):\r\n    return 'Hello ' + name\r\n"},
+    },
+    {
+        "id": "default-config",
+        "prompt": "load_retries() crashes when config.json has no 'retries'. It should return 3 in that case.",
+        "files": {
+            "config.json": "{\"timeout\": 30}\n",
+            "app.py": ("import json\n\n\ndef load_retries(path='config.json'):\n"
+                       "    with open(path) as f:\n        return json.load(f)['retries']\n"),
+            "test_app.py": ("import unittest\nfrom app import load_retries\n\n\nclass T(unittest.TestCase):\n"
+                            "    def test_default(self):\n        self.assertEqual(load_retries(), 3)\n"),
+        },
+        "verify": f"{UNITTEST} test_app",
+        "solution": {"app.py": ("import json\n\n\ndef load_retries(path='config.json'):\n"
+                                "    with open(path) as f:\n        return json.load(f).get('retries', 3)\n")},
+    },
+    {
+        "id": "add-cli-flag",
+        "prompt": "Add a --shout flag to greet_cli.py that prints the greeting in upper case. Keep the default unchanged.",
+        "files": {
+            "greet_cli.py": ("import sys\n\nname = sys.argv[1] if len(sys.argv) > 1 else 'world'\n"
+                             "print(f'Hello, {name}!')\n"),
+            "test_cli.py": ("import subprocess\nimport sys\nimport unittest\n\n\n"
+                            "def run(*args):\n    return subprocess.run([sys.executable, 'greet_cli.py', *args],\n"
+                            "                          capture_output=True, text=True).stdout.strip()\n\n\n"
+                            "class T(unittest.TestCase):\n    def test_default(self):\n"
+                            "        self.assertEqual(run('Ana'), 'Hello, Ana!')\n"
+                            "    def test_shout(self):\n        self.assertEqual(run('Ana', '--shout'), 'HELLO, ANA!')\n"
+                            "        self.assertEqual(run('--shout', 'Ana'), 'HELLO, ANA!')\n"),
+        },
+        "verify": f"{UNITTEST} test_cli",
+        "solution": {"greet_cli.py": ("import sys\n\nargs = [a for a in sys.argv[1:] if a != '--shout']\n"
+                                      "name = args[0] if args else 'world'\n"
+                                      "text = f'Hello, {name}!'\n"
+                                      "print(text.upper() if '--shout' in sys.argv else text)\n")},
+    },
+]