otto-cli-agent 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- otto_cli_agent-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md +21 -0
- otto_cli_agent-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md +17 -0
- otto_cli_agent-0.1.0/.github/PULL_REQUEST_TEMPLATE.md +11 -0
- otto_cli_agent-0.1.0/.github/workflows/publish.yml +49 -0
- otto_cli_agent-0.1.0/.github/workflows/tests.yml +59 -0
- otto_cli_agent-0.1.0/.gitignore +26 -0
- otto_cli_agent-0.1.0/.python-version +1 -0
- otto_cli_agent-0.1.0/CODE_OF_CONDUCT.md +16 -0
- otto_cli_agent-0.1.0/CONTRIBUTING.md +42 -0
- otto_cli_agent-0.1.0/LICENSE +21 -0
- otto_cli_agent-0.1.0/PKG-INFO +115 -0
- otto_cli_agent-0.1.0/README.md +256 -0
- otto_cli_agent-0.1.0/SECURITY.md +17 -0
- otto_cli_agent-0.1.0/agent/README.md +22 -0
- otto_cli_agent-0.1.0/agent/__init__.py +0 -0
- otto_cli_agent-0.1.0/agent/cli/README.md +77 -0
- otto_cli_agent-0.1.0/agent/cli/__init__.py +0 -0
- otto_cli_agent-0.1.0/agent/cli/art.py +371 -0
- otto_cli_agent-0.1.0/agent/cli/chat.py +309 -0
- otto_cli_agent-0.1.0/agent/cli/clipboard.py +106 -0
- otto_cli_agent-0.1.0/agent/cli/context.py +56 -0
- otto_cli_agent-0.1.0/agent/cli/doctor.py +60 -0
- otto_cli_agent-0.1.0/agent/cli/errors.py +58 -0
- otto_cli_agent-0.1.0/agent/cli/eval.py +130 -0
- otto_cli_agent-0.1.0/agent/cli/eval_claw.py +414 -0
- otto_cli_agent-0.1.0/agent/cli/eval_compaction.py +106 -0
- otto_cli_agent-0.1.0/agent/cli/eval_hle.py +92 -0
- otto_cli_agent-0.1.0/agent/cli/eval_memory.py +253 -0
- otto_cli_agent-0.1.0/agent/cli/eval_swe.py +172 -0
- otto_cli_agent-0.1.0/agent/cli/lessons.py +97 -0
- otto_cli_agent-0.1.0/agent/cli/main.py +67 -0
- otto_cli_agent-0.1.0/agent/cli/modals.py +570 -0
- otto_cli_agent-0.1.0/agent/cli/models.py +64 -0
- otto_cli_agent-0.1.0/agent/cli/output.py +54 -0
- otto_cli_agent-0.1.0/agent/cli/route.py +78 -0
- otto_cli_agent-0.1.0/agent/cli/sessions.py +125 -0
- otto_cli_agent-0.1.0/agent/cli/setup_screen.py +562 -0
- otto_cli_agent-0.1.0/agent/cli/shell.py +548 -0
- otto_cli_agent-0.1.0/agent/cli/tui.py +1807 -0
- otto_cli_agent-0.1.0/agent/cli/ui.py +14 -0
- otto_cli_agent-0.1.0/agent/cli/usage_panel.py +159 -0
- otto_cli_agent-0.1.0/agent/config/README.md +7 -0
- otto_cli_agent-0.1.0/agent/config/__init__.py +0 -0
- otto_cli_agent-0.1.0/agent/config/envfile.py +76 -0
- otto_cli_agent-0.1.0/agent/eval/README.md +76 -0
- otto_cli_agent-0.1.0/agent/eval/__init__.py +0 -0
- otto_cli_agent-0.1.0/agent/eval/claw_bench.py +1031 -0
- otto_cli_agent-0.1.0/agent/eval/compaction_bench.py +229 -0
- otto_cli_agent-0.1.0/agent/eval/data/README.md +10 -0
- otto_cli_agent-0.1.0/agent/eval/data/claw/README.md +108 -0
- otto_cli_agent-0.1.0/agent/eval/data/claw/llm_judge-gemini.patch +57 -0
- otto_cli_agent-0.1.0/agent/eval/data/claw/otto.yaml +35 -0
- otto_cli_agent-0.1.0/agent/eval/failures.py +276 -0
- otto_cli_agent-0.1.0/agent/eval/golden/README.md +33 -0
- otto_cli_agent-0.1.0/agent/eval/golden/code_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/code_02.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/code_03.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/code_04.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/code_05.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/code_06.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/math_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/math_02.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/math_03.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/math_04.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/math_05.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/math_06.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_gcp_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_ksp_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_math_binpacking_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_math_clique_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_math_setcover_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_math_subsetsum_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_tsp_01.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/golden/nphard_tsp_02.json +6 -0
- otto_cli_agent-0.1.0/agent/eval/hle_bench.py +273 -0
- otto_cli_agent-0.1.0/agent/eval/langfuse_sync.py +172 -0
- otto_cli_agent-0.1.0/agent/eval/memory_bench.py +538 -0
- otto_cli_agent-0.1.0/agent/eval/runner.py +174 -0
- otto_cli_agent-0.1.0/agent/eval/single_agent.py +120 -0
- otto_cli_agent-0.1.0/agent/eval/swe_bench.py +604 -0
- otto_cli_agent-0.1.0/agent/eval/terminal_bench.py +345 -0
- otto_cli_agent-0.1.0/agent/memory/README.md +102 -0
- otto_cli_agent-0.1.0/agent/memory/__init__.py +42 -0
- otto_cli_agent-0.1.0/agent/memory/embeddings.py +302 -0
- otto_cli_agent-0.1.0/agent/memory/hashing.py +15 -0
- otto_cli_agent-0.1.0/agent/memory/lessons.py +483 -0
- otto_cli_agent-0.1.0/agent/memory/queue.py +531 -0
- otto_cli_agent-0.1.0/agent/memory/retrieval.py +493 -0
- otto_cli_agent-0.1.0/agent/memory/session.py +60 -0
- otto_cli_agent-0.1.0/agent/memory/sessions.py +436 -0
- otto_cli_agent-0.1.0/agent/memory/store.py +429 -0
- otto_cli_agent-0.1.0/agent/memory/tokens.py +60 -0
- otto_cli_agent-0.1.0/agent/memory/wiring.py +146 -0
- otto_cli_agent-0.1.0/agent/pipeline/README.md +135 -0
- otto_cli_agent-0.1.0/agent/pipeline/__init__.py +0 -0
- otto_cli_agent-0.1.0/agent/pipeline/browsing.py +609 -0
- otto_cli_agent-0.1.0/agent/pipeline/budget.py +403 -0
- otto_cli_agent-0.1.0/agent/pipeline/codemap.py +254 -0
- otto_cli_agent-0.1.0/agent/pipeline/evidence.py +325 -0
- otto_cli_agent-0.1.0/agent/pipeline/execution.py +67 -0
- otto_cli_agent-0.1.0/agent/pipeline/modes.py +137 -0
- otto_cli_agent-0.1.0/agent/pipeline/native.py +1137 -0
- otto_cli_agent-0.1.0/agent/pipeline/nodes.py +3644 -0
- otto_cli_agent-0.1.0/agent/pipeline/pricing.py +209 -0
- otto_cli_agent-0.1.0/agent/pipeline/progress.py +139 -0
- otto_cli_agent-0.1.0/agent/pipeline/rag.py +139 -0
- otto_cli_agent-0.1.0/agent/pipeline/research.py +1325 -0
- otto_cli_agent-0.1.0/agent/pipeline/run.py +528 -0
- otto_cli_agent-0.1.0/agent/pipeline/screen.py +77 -0
- otto_cli_agent-0.1.0/agent/pipeline/state.py +220 -0
- otto_cli_agent-0.1.0/agent/pipeline/toolkit.py +328 -0
- otto_cli_agent-0.1.0/agent/pipeline/tools.py +1990 -0
- otto_cli_agent-0.1.0/agent/pipeline/tracing.py +147 -0
- otto_cli_agent-0.1.0/agent/pipeline/usage.py +251 -0
- otto_cli_agent-0.1.0/agent/pipeline/vision.py +84 -0
- otto_cli_agent-0.1.0/agent/pipeline/walkthrough.py +735 -0
- otto_cli_agent-0.1.0/agent/pipeline/workspace.py +229 -0
- otto_cli_agent-0.1.0/agent/router/README.md +60 -0
- otto_cli_agent-0.1.0/agent/router/__init__.py +0 -0
- otto_cli_agent-0.1.0/agent/router/automap.py +114 -0
- otto_cli_agent-0.1.0/agent/router/health.py +229 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/README.md +38 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/__init__.py +202 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/anthropic_provider.py +128 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/base.py +507 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/custom.py +152 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/gemini_provider.py +122 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/inception_provider.py +687 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/openai_provider.py +151 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/retired.py +145 -0
- otto_cli_agent-0.1.0/agent/router/llm_provider/temperature.py +371 -0
- otto_cli_agent-0.1.0/agent/router/mapping.py +579 -0
- otto_cli_agent-0.1.0/agent/router/outcomes.py +363 -0
- otto_cli_agent-0.1.0/agent/router/overrides.py +389 -0
- otto_cli_agent-0.1.0/agent/router/reload.py +28 -0
- otto_cli_agent-0.1.0/agent/router/router.py +413 -0
- otto_cli_agent-0.1.0/agent/router/setup.py +123 -0
- otto_cli_agent-0.1.0/containers/README.md +14 -0
- otto_cli_agent-0.1.0/containers/otto-desktop/Dockerfile +31 -0
- otto_cli_agent-0.1.0/containers/otto-desktop/README.md +20 -0
- otto_cli_agent-0.1.0/containers/otto-desktop/start-desktop.sh +12 -0
- otto_cli_agent-0.1.0/debug_pipeline.py +43 -0
- otto_cli_agent-0.1.0/docs/HISTORY.md +197 -0
- otto_cli_agent-0.1.0/docs/README.md +7 -0
- otto_cli_agent-0.1.0/docs/RESEARCH.md +43 -0
- otto_cli_agent-0.1.0/docs/_config.yml +6 -0
- otto_cli_agent-0.1.0/docs/design/README.md +7 -0
- otto_cli_agent-0.1.0/docs/design/tiered-memory.md +383 -0
- otto_cli_agent-0.1.0/docs/index.md +66 -0
- otto_cli_agent-0.1.0/docs/media/codebase-code-map.png +0 -0
- otto_cli_agent-0.1.0/docs/media/exercise-todo-cli.png +0 -0
- otto_cli_agent-0.1.0/docs/media/otto-demo.gif +0 -0
- otto_cli_agent-0.1.0/docs/media/palette-doctor-route.png +0 -0
- otto_cli_agent-0.1.0/docs/media/run-trace-and-judge.png +0 -0
- otto_cli_agent-0.1.0/docs/media/session-resumed-thinking.png +0 -0
- otto_cli_agent-0.1.0/docs/media/sessions-picker.png +0 -0
- otto_cli_agent-0.1.0/docs/media/setup-screen.png +0 -0
- otto_cli_agent-0.1.0/docs/media/social-preview.png +0 -0
- otto_cli_agent-0.1.0/docs/pypi.md +55 -0
- otto_cli_agent-0.1.0/pyproject.toml +66 -0
- otto_cli_agent-0.1.0/tests/README.md +57 -0
- otto_cli_agent-0.1.0/tests/conftest.py +209 -0
- otto_cli_agent-0.1.0/tests/test_agent_loop.py +1422 -0
- otto_cli_agent-0.1.0/tests/test_art.py +161 -0
- otto_cli_agent-0.1.0/tests/test_ask_user_node.py +784 -0
- otto_cli_agent-0.1.0/tests/test_automap.py +84 -0
- otto_cli_agent-0.1.0/tests/test_browsing.py +589 -0
- otto_cli_agent-0.1.0/tests/test_budget.py +445 -0
- otto_cli_agent-0.1.0/tests/test_call_budget.py +354 -0
- otto_cli_agent-0.1.0/tests/test_chat_fast_path.py +192 -0
- otto_cli_agent-0.1.0/tests/test_claw_bench.py +385 -0
- otto_cli_agent-0.1.0/tests/test_clipboard.py +86 -0
- otto_cli_agent-0.1.0/tests/test_codemap.py +191 -0
- otto_cli_agent-0.1.0/tests/test_compaction_bench.py +114 -0
- otto_cli_agent-0.1.0/tests/test_custom_endpoints.py +161 -0
- otto_cli_agent-0.1.0/tests/test_diffusion_retry.py +218 -0
- otto_cli_agent-0.1.0/tests/test_embedding_backends.py +314 -0
- otto_cli_agent-0.1.0/tests/test_envfile.py +100 -0
- otto_cli_agent-0.1.0/tests/test_eval_honesty.py +133 -0
- otto_cli_agent-0.1.0/tests/test_eval_runner.py +304 -0
- otto_cli_agent-0.1.0/tests/test_evaluator_node.py +615 -0
- otto_cli_agent-0.1.0/tests/test_evicted_context.py +221 -0
- otto_cli_agent-0.1.0/tests/test_evidence.py +485 -0
- otto_cli_agent-0.1.0/tests/test_failures.py +265 -0
- otto_cli_agent-0.1.0/tests/test_health.py +195 -0
- otto_cli_agent-0.1.0/tests/test_hle_bench.py +122 -0
- otto_cli_agent-0.1.0/tests/test_inception_provider.py +17 -0
- otto_cli_agent-0.1.0/tests/test_inception_stream.py +212 -0
- otto_cli_agent-0.1.0/tests/test_langfuse_sync.py +153 -0
- otto_cli_agent-0.1.0/tests/test_lessons.py +376 -0
- otto_cli_agent-0.1.0/tests/test_lessons_cli.py +20 -0
- otto_cli_agent-0.1.0/tests/test_mapping.py +400 -0
- otto_cli_agent-0.1.0/tests/test_memory_bench.py +310 -0
- otto_cli_agent-0.1.0/tests/test_memory_embeddings.py +124 -0
- otto_cli_agent-0.1.0/tests/test_memory_hashing.py +21 -0
- otto_cli_agent-0.1.0/tests/test_memory_queue.py +398 -0
- otto_cli_agent-0.1.0/tests/test_memory_retrieval.py +346 -0
- otto_cli_agent-0.1.0/tests/test_memory_session.py +55 -0
- otto_cli_agent-0.1.0/tests/test_memory_store.py +105 -0
- otto_cli_agent-0.1.0/tests/test_memory_tokens.py +82 -0
- otto_cli_agent-0.1.0/tests/test_memory_wiring.py +152 -0
- otto_cli_agent-0.1.0/tests/test_model_policies.py +261 -0
- otto_cli_agent-0.1.0/tests/test_modes.py +164 -0
- otto_cli_agent-0.1.0/tests/test_native.py +352 -0
- otto_cli_agent-0.1.0/tests/test_no_import_shadowing.py +59 -0
- otto_cli_agent-0.1.0/tests/test_output.py +52 -0
- otto_cli_agent-0.1.0/tests/test_overrides.py +258 -0
- otto_cli_agent-0.1.0/tests/test_pipeline_nodes.py +0 -0
- otto_cli_agent-0.1.0/tests/test_pipeline_run.py +276 -0
- otto_cli_agent-0.1.0/tests/test_progress.py +123 -0
- otto_cli_agent-0.1.0/tests/test_prompt_tool_sync.py +260 -0
- otto_cli_agent-0.1.0/tests/test_reload.py +33 -0
- otto_cli_agent-0.1.0/tests/test_research_router.py +232 -0
- otto_cli_agent-0.1.0/tests/test_research_workflow.py +475 -0
- otto_cli_agent-0.1.0/tests/test_retrieval_method.py +255 -0
- otto_cli_agent-0.1.0/tests/test_router.py +652 -0
- otto_cli_agent-0.1.0/tests/test_run_scoped_toolkit.py +458 -0
- otto_cli_agent-0.1.0/tests/test_screen.py +239 -0
- otto_cli_agent-0.1.0/tests/test_seat_outcomes.py +318 -0
- otto_cli_agent-0.1.0/tests/test_sessions.py +586 -0
- otto_cli_agent-0.1.0/tests/test_setup.py +60 -0
- otto_cli_agent-0.1.0/tests/test_swe_bench.py +318 -0
- otto_cli_agent-0.1.0/tests/test_temperature_learning.py +179 -0
- otto_cli_agent-0.1.0/tests/test_terminal_bench_adapter.py +76 -0
- otto_cli_agent-0.1.0/tests/test_tool_loop.py +548 -0
- otto_cli_agent-0.1.0/tests/test_tools_stubs.py +318 -0
- otto_cli_agent-0.1.0/tests/test_tracing.py +217 -0
- otto_cli_agent-0.1.0/tests/test_tui.py +1357 -0
- otto_cli_agent-0.1.0/tests/test_tui_progress.py +221 -0
- otto_cli_agent-0.1.0/tests/test_tui_setup.py +453 -0
- otto_cli_agent-0.1.0/tests/test_usage.py +415 -0
- otto_cli_agent-0.1.0/tests/test_vision_tool.py +329 -0
- otto_cli_agent-0.1.0/tests/test_walkthrough.py +420 -0
- otto_cli_agent-0.1.0/tests/test_workspace_session.py +270 -0
- otto_cli_agent-0.1.0/tests/test_workspace_tools.py +597 -0
- otto_cli_agent-0.1.0/uv.lock +3284 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Bug report
|
|
3
|
+
about: Something ran and did the wrong thing
|
|
4
|
+
labels: bug
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## What you ran
|
|
8
|
+
|
|
9
|
+
The command, the front end (`otto tui`, `otto chat`, an eval), and the
|
|
10
|
+
workspace or container it ran against.
|
|
11
|
+
|
|
12
|
+
## What happened
|
|
13
|
+
|
|
14
|
+
The exact output or error text. For a model problem, the seat and vendor
|
|
15
|
+
(`otto route <task>` prints it) and the model calls the turn spent.
|
|
16
|
+
|
|
17
|
+
## What you expected
|
|
18
|
+
|
|
19
|
+
## Version
|
|
20
|
+
|
|
21
|
+
`otto --version` or the commit, your OS, and Python version.
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Feature request
|
|
3
|
+
about: Something Otto should do, with the case for it
|
|
4
|
+
labels: enhancement
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## What should happen
|
|
8
|
+
|
|
9
|
+
## Why
|
|
10
|
+
|
|
11
|
+
The task it would help with, and if you have one, a measurement or a
|
|
12
|
+
reference that says it works. Design decisions in this project carry their
|
|
13
|
+
numbers; a proposal that arrives with one is easier to accept.
|
|
14
|
+
|
|
15
|
+
## What it would touch
|
|
16
|
+
|
|
17
|
+
The folder or module, if you know it. Every folder has a README.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Builds the wheel and sdist and publishes them to PyPI.
|
|
2
|
+
#
|
|
3
|
+
# Runs when a GitHub release is published, or by hand from the Actions tab.
|
|
4
|
+
# Publishing uses PyPI's trusted publishing (OpenID Connect): PyPI trusts
|
|
5
|
+
# this repository's workflow directly, so no API token is stored anywhere.
|
|
6
|
+
# The one-time setup on pypi.org is in the README under "Publishing".
|
|
7
|
+
|
|
8
|
+
name: publish
|
|
9
|
+
|
|
10
|
+
on:
|
|
11
|
+
release:
|
|
12
|
+
types: [published]
|
|
13
|
+
workflow_dispatch:
|
|
14
|
+
|
|
15
|
+
permissions:
|
|
16
|
+
contents: read
|
|
17
|
+
|
|
18
|
+
jobs:
|
|
19
|
+
build:
|
|
20
|
+
runs-on: ubuntu-latest
|
|
21
|
+
steps:
|
|
22
|
+
- uses: actions/checkout@v4
|
|
23
|
+
- uses: astral-sh/setup-uv@v6
|
|
24
|
+
with:
|
|
25
|
+
enable-cache: true
|
|
26
|
+
cache-dependency-glob: uv.lock
|
|
27
|
+
- run: uv python install
|
|
28
|
+
- run: uv sync --locked
|
|
29
|
+
- run: uv run pytest -q
|
|
30
|
+
- run: uv build
|
|
31
|
+
- uses: actions/upload-artifact@v4
|
|
32
|
+
with:
|
|
33
|
+
name: dist
|
|
34
|
+
path: dist/
|
|
35
|
+
|
|
36
|
+
publish:
|
|
37
|
+
needs: build
|
|
38
|
+
runs-on: ubuntu-latest
|
|
39
|
+
environment:
|
|
40
|
+
name: pypi
|
|
41
|
+
url: https://pypi.org/p/otto-cli-agent
|
|
42
|
+
permissions:
|
|
43
|
+
id-token: write
|
|
44
|
+
steps:
|
|
45
|
+
- uses: actions/download-artifact@v4
|
|
46
|
+
with:
|
|
47
|
+
name: dist
|
|
48
|
+
path: dist/
|
|
49
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Runs the test suite on every push to main and every pull request.
|
|
2
|
+
#
|
|
3
|
+
# No secrets are configured on purpose: the suite is designed to run with no
|
|
4
|
+
# API keys, so there is nothing here to leak and nothing for a fork PR to be
|
|
5
|
+
# denied access to.
|
|
6
|
+
|
|
7
|
+
name: tests
|
|
8
|
+
|
|
9
|
+
on:
|
|
10
|
+
push:
|
|
11
|
+
branches: [main]
|
|
12
|
+
pull_request:
|
|
13
|
+
|
|
14
|
+
concurrency:
|
|
15
|
+
group: ${{ github.workflow }}-${{ github.ref }}
|
|
16
|
+
cancel-in-progress: true
|
|
17
|
+
|
|
18
|
+
# Read the code, and nothing else. Without this block the job gets whatever
|
|
19
|
+
# the repository's default token scope happens to be, which on many repos is
|
|
20
|
+
# write access to contents, issues, packages and more -- handed to a workflow
|
|
21
|
+
# whose entire job is to run pytest.
|
|
22
|
+
#
|
|
23
|
+
# It matters most on exactly the event this workflow accepts: `pull_request`
|
|
24
|
+
# runs a checkout of somebody else's branch, so any step that executes code
|
|
25
|
+
# from the PR is executing it with whatever the token can do. Nothing here
|
|
26
|
+
# needs to write anything, so nothing here is allowed to.
|
|
27
|
+
permissions:
|
|
28
|
+
contents: read
|
|
29
|
+
|
|
30
|
+
jobs:
|
|
31
|
+
pytest:
|
|
32
|
+
strategy:
|
|
33
|
+
fail-fast: false
|
|
34
|
+
matrix:
|
|
35
|
+
os: [ubuntu-latest, windows-latest, macos-latest]
|
|
36
|
+
|
|
37
|
+
runs-on: ${{ matrix.os }}
|
|
38
|
+
|
|
39
|
+
defaults:
|
|
40
|
+
run:
|
|
41
|
+
shell: bash
|
|
42
|
+
|
|
43
|
+
steps:
|
|
44
|
+
- uses: actions/checkout@v4
|
|
45
|
+
|
|
46
|
+
- name: Install uv
|
|
47
|
+
uses: astral-sh/setup-uv@v6
|
|
48
|
+
with:
|
|
49
|
+
enable-cache: true
|
|
50
|
+
cache-dependency-glob: uv.lock
|
|
51
|
+
|
|
52
|
+
- name: Install Python
|
|
53
|
+
run: uv python install
|
|
54
|
+
|
|
55
|
+
- name: Install dependencies
|
|
56
|
+
run: uv sync --locked
|
|
57
|
+
|
|
58
|
+
- name: Run tests
|
|
59
|
+
run: uv run pytest -q
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
.env
|
|
2
|
+
.venv
|
|
3
|
+
__pycache__
|
|
4
|
+
.DS_Store
|
|
5
|
+
langfuse
|
|
6
|
+
# per-run final-answer files (13.5) -- not something to commit
|
|
7
|
+
otto_output/
|
|
8
|
+
# Downloaded benchmark data -- LoCoMo, HLE and Claw-Eval's own parquet and
|
|
9
|
+
# fixture archives, 2.9GB of it, fetched on demand rather than committed.
|
|
10
|
+
#
|
|
11
|
+
# The rule used to be the whole of agent/eval/data/, which also silently
|
|
12
|
+
# swallowed the hand-authored files that live alongside the downloads: the
|
|
13
|
+
# Claw-Eval config, its judge patch, and the README telling you to apply it.
|
|
14
|
+
# Three files the docs point at and git had never heard of. Ignore the
|
|
15
|
+
# downloads by what they are instead.
|
|
16
|
+
agent/eval/data/*.json
|
|
17
|
+
agent/eval/data/*.parquet
|
|
18
|
+
agent/eval/data/**/*.parquet
|
|
19
|
+
agent/eval/data/**/*.tar.gz
|
|
20
|
+
|
|
21
|
+
# local caches and tooling state
|
|
22
|
+
.pytest_cache/
|
|
23
|
+
.claude/
|
|
24
|
+
# per-run research documents (agent/pipeline/research.py)
|
|
25
|
+
otto_research/
|
|
26
|
+
research_papers
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.12
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Code of Conduct
|
|
2
|
+
|
|
3
|
+
This project follows the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
|
|
4
|
+
|
|
5
|
+
In short: be respectful and constructive, assume good faith, keep
|
|
6
|
+
disagreements about the work and not the person, and welcome people who are
|
|
7
|
+
new to the project or to the field.
|
|
8
|
+
|
|
9
|
+
Unacceptable behaviour includes harassment, personal attacks, publishing
|
|
10
|
+
others' private information, and any conduct that would be inappropriate in
|
|
11
|
+
a professional setting.
|
|
12
|
+
|
|
13
|
+
To report a problem, contact the maintainer privately at the email address on
|
|
14
|
+
the [GitHub profile](https://github.com/siddharth23P). Reports are handled
|
|
15
|
+
confidentially. The maintainer may remove, edit or reject contributions and
|
|
16
|
+
comments that do not follow this code, and may ban repeat offenders.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Thanks for looking. The bar for a change here is the same one the project was
|
|
4
|
+
built to: say what you measured, or what you would need to measure.
|
|
5
|
+
|
|
6
|
+
## Running it
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
uv sync
|
|
10
|
+
uv run pytest -q # 1,552 tests, no API key, no network, about two minutes
|
|
11
|
+
uv run otto doctor # once you have keys in .env
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
The suite is designed to pass on a machine with no keys at all. If a test
|
|
15
|
+
you add needs a real vendor, mark it with the `live_*` markers in
|
|
16
|
+
`tests/conftest.py` so it skips without one.
|
|
17
|
+
|
|
18
|
+
## Where things are
|
|
19
|
+
|
|
20
|
+
Every folder has a README describing what is in it and why. Start at the
|
|
21
|
+
[root README](README.md), then the folder you are changing. The design
|
|
22
|
+
decisions carry their measurements in the module docstrings; if you change a
|
|
23
|
+
behaviour that has a number behind it, the pull request should carry the new
|
|
24
|
+
number.
|
|
25
|
+
|
|
26
|
+
## What a good pull request looks like
|
|
27
|
+
|
|
28
|
+
- One change, with the reason in the description: what was wrong, how you
|
|
29
|
+
know, what changed, what you measured.
|
|
30
|
+
- A test that fails without the change. For a bug, the exact input that
|
|
31
|
+
broke a real run is the best test there is.
|
|
32
|
+
- No new dependency without a sentence on why the standard library or an
|
|
33
|
+
existing one would not do.
|
|
34
|
+
- Prompt changes are measured, not argued: this codebase records a case
|
|
35
|
+
where adding one more instruction erased the effect of the four before it.
|
|
36
|
+
- Commits under your own name.
|
|
37
|
+
|
|
38
|
+
## Reporting a bug
|
|
39
|
+
|
|
40
|
+
Open an issue with the command you ran, what you expected, and what
|
|
41
|
+
happened, including the exact error text. If it involved a model, say which
|
|
42
|
+
seat and vendor (`otto route <task>` shows it).
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Siddharth Priyadarshi
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: otto-cli-agent
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Otto: a terminal AI agent built around measured results -- one agent loop, a rubric-first evaluator, tiered memory, multi-vendor routing, and six benchmark harnesses.
|
|
5
|
+
Project-URL: Repository, https://github.com/siddharth23P/otto_agent
|
|
6
|
+
Project-URL: Issues, https://github.com/siddharth23P/otto_agent/issues
|
|
7
|
+
Author-email: Siddharth Priyadarshi <sid23p@gmail.com>
|
|
8
|
+
License: MIT License
|
|
9
|
+
|
|
10
|
+
Copyright (c) 2026 Siddharth Priyadarshi
|
|
11
|
+
|
|
12
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
13
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
14
|
+
in the Software without restriction, including without limitation the rights
|
|
15
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
16
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
17
|
+
furnished to do so, subject to the following conditions:
|
|
18
|
+
|
|
19
|
+
The above copyright notice and this permission notice shall be included in all
|
|
20
|
+
copies or substantial portions of the Software.
|
|
21
|
+
|
|
22
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
23
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
24
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
25
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
26
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
27
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
28
|
+
SOFTWARE.
|
|
29
|
+
License-File: LICENSE
|
|
30
|
+
Keywords: agent,claw-eval,evaluation,langgraph,llm,memory,swe-bench,tui
|
|
31
|
+
Classifier: Development Status :: 4 - Beta
|
|
32
|
+
Classifier: Environment :: Console
|
|
33
|
+
Classifier: Intended Audience :: Developers
|
|
34
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
35
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
36
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
37
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
38
|
+
Classifier: Topic :: Software Development
|
|
39
|
+
Requires-Python: >=3.12
|
|
40
|
+
Requires-Dist: emoji>=2.15.0
|
|
41
|
+
Requires-Dist: fastembed>=0.8.0
|
|
42
|
+
Requires-Dist: inceptionai>=0.1.3
|
|
43
|
+
Requires-Dist: langchain-anthropic>=1.7.1
|
|
44
|
+
Requires-Dist: langchain-google-genai>=4.4.0
|
|
45
|
+
Requires-Dist: langchain-openai>=1.6.0
|
|
46
|
+
Requires-Dist: langchain>=1.4.0
|
|
47
|
+
Requires-Dist: langfuse>=4.15.1
|
|
48
|
+
Requires-Dist: langgraph-cli[inmem]>=0.4.31
|
|
49
|
+
Requires-Dist: langgraph>=1.2.11
|
|
50
|
+
Requires-Dist: numpy>=2.5.3
|
|
51
|
+
Requires-Dist: openpyxl>=3.1.5
|
|
52
|
+
Requires-Dist: prompt-toolkit>=3.0.53
|
|
53
|
+
Requires-Dist: python-docx>=1.1.2
|
|
54
|
+
Requires-Dist: python-dotenv>=1.2.3
|
|
55
|
+
Requires-Dist: reportlab>=4.2
|
|
56
|
+
Requires-Dist: textual>=8.2.8
|
|
57
|
+
Requires-Dist: tiktoken>=0.14.0
|
|
58
|
+
Requires-Dist: typer>=0.27.2
|
|
59
|
+
Description-Content-Type: text/markdown
|
|
60
|
+
|
|
61
|
+
# Otto
|
|
62
|
+
|
|
63
|
+
A terminal AI agent that works on a codebase, a container, a browser or a
|
|
64
|
+
desktop, uses what it built, and judges its own work against criteria it
|
|
65
|
+
wrote before it started. One agent loop with modes over four model vendors,
|
|
66
|
+
a rubric-first evaluator, tiered memory, six benchmark harnesses, and 1,552
|
|
67
|
+
tests that need no API key.
|
|
68
|
+
|
|
69
|
+

|
|
70
|
+
|
|
71
|
+
## Install
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
pip install otto-cli-agent
|
|
75
|
+
otto doctor
|
|
76
|
+
otto tui
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Put keys in a `.env` in the directory you run it from. `INCEPTION_API_KEY`
|
|
80
|
+
is required (it alone serves the fill-in-the-middle and edit endpoints);
|
|
81
|
+
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY` and `GEMINI_API_KEY` are optional and
|
|
82
|
+
each unlocks the seats routed to that vendor. `otto tui` opens its setup
|
|
83
|
+
screen on first start when nothing is configured. Python 3.12 or newer.
|
|
84
|
+
|
|
85
|
+
## What it does
|
|
86
|
+
|
|
87
|
+
- One agent loop with four modes that swap the model underneath without
|
|
88
|
+
losing the conversation; routing that learns from outcomes, per-model
|
|
89
|
+
cooldowns and a per-provider circuit breaker.
|
|
90
|
+
- A rubric written from the task before any attempt, shared by the loop and
|
|
91
|
+
the judge, so the judge never grades against the actor's own output.
|
|
92
|
+
- Once-only holds before an irreversible action, before finishing on unrun
|
|
93
|
+
code, and before finishing without using what was built.
|
|
94
|
+
- 18 tools: shell and Python, files, `code_map`, a browser, `exercise` (walk
|
|
95
|
+
through a page, a served app, a CLI, an API, a terminal program or a device
|
|
96
|
+
app and report each step as the machine saw it), a desktop, web search,
|
|
97
|
+
workspace search and memory recall.
|
|
98
|
+
- Tiered memory with type-aware compaction and two-stage semantic recall
|
|
99
|
+
(96% on LoCoMo), a lesson bank, and sessions that survive the process.
|
|
100
|
+
- A Textual TUI with a setup screen, live progress, a token and dollar
|
|
101
|
+
ledger; a REPL with the same pipeline.
|
|
102
|
+
- Six benchmark harnesses: a golden set, SWE-bench Verified, Claw-Eval,
|
|
103
|
+
LoCoMo, a compaction bench and Humanity's Last Exam.
|
|
104
|
+
|
|
105
|
+
## Links
|
|
106
|
+
|
|
107
|
+
- Source, folder-by-folder READMEs and screenshots:
|
|
108
|
+
https://github.com/siddharth23P/otto_agent
|
|
109
|
+
- Website: https://siddharth23p.github.io/otto_agent/
|
|
110
|
+
- How it was built, commit by commit, with the measurement behind each
|
|
111
|
+
change: https://github.com/siddharth23P/otto_agent/blob/main/docs/HISTORY.md
|
|
112
|
+
- The research each design decision draws on:
|
|
113
|
+
https://github.com/siddharth23P/otto_agent/blob/main/docs/RESEARCH.md
|
|
114
|
+
|
|
115
|
+
MIT licensed.
|
|
@@ -0,0 +1,256 @@
|
|
|
1
|
+
# Otto
|
|
2
|
+
|
|
3
|
+
[](https://github.com/siddharth23P/otto_agent/actions/workflows/tests.yml)
|
|
4
|
+

|
|
5
|
+

|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
Otto is a terminal AI agent that works on a codebase, a container, a browser
|
|
9
|
+
or a desktop, uses what it built, and judges its own work against criteria it
|
|
10
|
+
wrote before it started. It runs as a full-screen TUI or a REPL, routes each
|
|
11
|
+
kind of work to the cheapest model that does it well across four vendors,
|
|
12
|
+
keeps a tiered memory that makes long sessions feel unbounded, and ships six
|
|
13
|
+
benchmark harnesses so every design decision in it is a measurement rather
|
|
14
|
+
than an opinion.
|
|
15
|
+
|
|
16
|
+
Every folder carries its own README describing what is in it and why. This
|
|
17
|
+
page is the map. The commit-by-commit record of how the design was arrived at,
|
|
18
|
+
with the measurement behind each change, is [docs/HISTORY.md](docs/HISTORY.md).
|
|
19
|
+
|
|
20
|
+
## What it does, measured
|
|
21
|
+
|
|
22
|
+
| property | measured |
|
|
23
|
+
| --- | --- |
|
|
24
|
+
| One agent loop with modes, no node boundaries | a one-tool task costs 3 model calls, a rejection and retry 4; overhead is flat in the length of the work |
|
|
25
|
+
| Criteria written from the task before any attempt, shared by the loop and the judge | a write-and-run task is judged in 7 calls and 31 s, approved first time, on five checkable criteria |
|
|
26
|
+
| A mutation gate held once before an irreversible action | Claw-Eval T026 (three contacts match, one must be asked about) scores 0.955 with safety 1.0 |
|
|
27
|
+
| Type-aware compaction: what the person said is never summarised | 8/8 planted constraints kept at 120 and 400 turns through 23 compaction rounds |
|
|
28
|
+
| Two-stage semantic recall over a content-addressed store | LoCoMo recall 96% at about 2,600 tokens per query, store coverage 100% |
|
|
29
|
+
| A greeting has no criteria, so it skips the loop | "hi otto!" costs 2 model calls |
|
|
30
|
+
| A document is a sectioned workflow, not a loop | a ten-section document of 42,000 words in 54 calls, every section over the asked length |
|
|
31
|
+
| `exercise`: the agent uses what it built and the judge reads a code-written report | five runs in a row (a CLI, an API, a curses app among them) walk through what they built before finishing |
|
|
32
|
+
| Golden set of 20 code, math and NP-hard tasks with real checkers | 20/20 |
|
|
33
|
+
| SWE-bench Verified, graded by each repository's own tests | 2 resolved of 3 soundly graded, with a control run on an untouched repository as part of the harness |
|
|
34
|
+
|
|
35
|
+
## See it run
|
|
36
|
+
|
|
37
|
+

|
|
38
|
+
|
|
39
|
+
One minute of `otto tui`, unedited. Full quality:
|
|
40
|
+
[otto-demo.mp4](https://github.com/siddharth23P/otto_agent/releases/download/v0.1.0/otto-demo.mp4)
|
|
41
|
+
(2.2 MB, from the [v0.1.0 release](https://github.com/siddharth23P/otto_agent/releases/tag/v0.1.0)).
|
|
42
|
+
|
|
43
|
+
Every screenshot below is a real run, taken while the thinking block was
|
|
44
|
+
open, so the tool trace, the holds and the judge are visible.
|
|
45
|
+
|
|
46
|
+
**The run, mid-judgment.** The agent wrote `fib.py`, was held once for
|
|
47
|
+
running it without checking anything, exercised it against expected
|
|
48
|
+
output, and the judge is scoring the answer against the two criteria it
|
|
49
|
+
wrote before the attempt existed.
|
|
50
|
+
|
|
51
|
+

|
|
52
|
+
|
|
53
|
+
**Using what it built.** Asked for a small CLI, the agent wrote `todo.py`,
|
|
54
|
+
walked through it with `exercise` (three steps, all passed), checked the
|
|
55
|
+
files, and the judge approved on 3 of 3 criteria.
|
|
56
|
+
|
|
57
|
+

|
|
58
|
+
|
|
59
|
+
**Working on a codebase.** Asked where `TieredQueue` is defined and what
|
|
60
|
+
uses it, with this repository as the workspace; the answer lists file paths
|
|
61
|
+
and line numbers from `code_map` while the judge reads it.
|
|
62
|
+
|
|
63
|
+

|
|
64
|
+
|
|
65
|
+
**Sessions.** Every turn is saved as it finishes. The picker lists what was
|
|
66
|
+
saved, with its title, turns and workspace, and picking one brings its
|
|
67
|
+
history and workspace back.
|
|
68
|
+
|
|
69
|
+

|
|
70
|
+
|
|
71
|
+
**A resumed session, mid-turn.** Three earlier turns replayed, then a new
|
|
72
|
+
request: read, edit, exercise with an expected output, the hold, and the
|
|
73
|
+
judge checking, in one open trace.
|
|
74
|
+
|
|
75
|
+

|
|
76
|
+
|
|
77
|
+
**Setup.** Providers with masked keys and a live probe, then models, then a
|
|
78
|
+
per-seat mapping with pins.
|
|
79
|
+
|
|
80
|
+

|
|
81
|
+
|
|
82
|
+
**The palette.** "Check providers" and "Route a task" run inside the
|
|
83
|
+
transcript without spending a turn.
|
|
84
|
+
|
|
85
|
+

|
|
86
|
+
|
|
87
|
+
## Install
|
|
88
|
+
|
|
89
|
+
The wheel and sdist are attached to every release:
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
pip install https://github.com/siddharth23P/otto_agent/releases/download/v0.1.0/otto_cli_agent-0.1.0-py3-none-any.whl
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
## Quick start
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
git clone https://github.com/siddharth23P/otto_agent.git
|
|
99
|
+
cd otto_agent
|
|
100
|
+
uv sync
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Put keys in `.env` at the repository root. `INCEPTION_API_KEY` is the one
|
|
104
|
+
Otto cannot run without (it alone serves the fill-in-the-middle and edit
|
|
105
|
+
endpoints); `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` and `GEMINI_API_KEY` are
|
|
106
|
+
optional and each unlocks the seats routed to that vendor. `otto tui` opens
|
|
107
|
+
its setup screen on first start when nothing is configured.
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
uv run otto doctor # which providers and seats resolve, and why not
|
|
111
|
+
uv run otto tui # full-screen front end, opens on the current directory
|
|
112
|
+
uv run otto chat # the same pipeline at a prompt
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
| command | what it does |
|
|
116
|
+
| --- | --- |
|
|
117
|
+
| `otto tui` / `otto chat` | interactive sessions; `--workspace PATH`, `--no-workspace`, `--resume <id\|prefix\|last>` |
|
|
118
|
+
| `otto sessions` | list, `--delete`, `--rename`, `--export`, `--import`, `--prune` |
|
|
119
|
+
| `otto doctor` | provider and route health, exit 2 on a missing required key |
|
|
120
|
+
| `otto models` | every model each configured vendor lists, with detected capabilities |
|
|
121
|
+
| `otto route <task>` | the fallback chain for a seat, pins starred, observed outcomes shown |
|
|
122
|
+
| `otto lessons` | print, clear, `--export`, `--import` the lesson bank |
|
|
123
|
+
| `otto eval` | the golden set |
|
|
124
|
+
| `otto eval-swe` | SWE-bench Verified |
|
|
125
|
+
| `otto eval-claw` | Claw-Eval (needs a checkout; see [agent/eval/data/claw/README.md](agent/eval/data/claw/README.md)) |
|
|
126
|
+
| `otto eval-memory` | LoCoMo recall |
|
|
127
|
+
| `otto eval-compaction` | what each compaction policy loses |
|
|
128
|
+
| `otto eval-hle` | Humanity's Last Exam, raw model vs. the agent |
|
|
129
|
+
|
|
130
|
+
Environment variables: `OTTO_MAX_MODEL_CALLS` (per-turn ceiling, default 120),
|
|
131
|
+
`OTTO_COMMAND_TIMEOUT` (120 s with a workspace, 10 s without),
|
|
132
|
+
`OTTO_EMBEDDING_MODEL` (`provider:model`; local BGE is the floor),
|
|
133
|
+
`OTTO_MODEL_PRICES` (a JSON file that overrides the price table),
|
|
134
|
+
`OTTO_IGNORE_ROUTES=1` (use the shipped routing table untouched; evals do),
|
|
135
|
+
`OTTO_BROWSER_PYTHON` (an interpreter with Playwright and `pyte`, which
|
|
136
|
+
enables the browser and terminal tools), `OTTO_NO_ANIMATION=1`, `OTTO_THEME`.
|
|
137
|
+
|
|
138
|
+
Otto ships no browser. To let it load the pages and terminal programs it
|
|
139
|
+
writes, install Playwright and `pyte` into any Python once and point
|
|
140
|
+
`OTTO_BROWSER_PYTHON` at it:
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
python3 -m venv ~/.otto/browser && ~/.otto/browser/bin/pip install playwright pyte && ~/.otto/browser/bin/playwright install chromium-headless-shell
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
The file tools cannot touch anything outside the workspace root, symlinks and
|
|
147
|
+
`..` included, and that is enforced and tested. `execute_bash` cannot be
|
|
148
|
+
confined the same way, so run Otto against a repository you have committed.
|
|
149
|
+
|
|
150
|
+
## How it works
|
|
151
|
+
|
|
152
|
+
One agent, one evaluator. The agent works the task end to end in a single
|
|
153
|
+
conversation and changes mode when the kind of work changes: a mode is a model
|
|
154
|
+
and a way of thinking, not a separate node, so switching swaps the model
|
|
155
|
+
underneath while the conversation, the tools and everything learned so far
|
|
156
|
+
carry over.
|
|
157
|
+
|
|
158
|
+
```mermaid
|
|
159
|
+
flowchart TD
|
|
160
|
+
start([request]) --> rubric[write the criteria<br/>from the task alone]
|
|
161
|
+
rubric -->|no task in it| chat[answer it<br/>one cheap call] --> done
|
|
162
|
+
rubric -->|a document| research[outline, then one section<br/>at a time with a continuity ledger]
|
|
163
|
+
research -->|report on the file| evaluator
|
|
164
|
+
rubric -->|criteria| agent
|
|
165
|
+
|
|
166
|
+
agent{{agent}} -->|ACTION| tools
|
|
167
|
+
tools -->|result| agent
|
|
168
|
+
agent -->|switch_mode| agent
|
|
169
|
+
agent -->|delegate| child[bounded sub-agent<br/>contract down, report up]
|
|
170
|
+
child -->|report| agent
|
|
171
|
+
|
|
172
|
+
agent -->|FINAL| gate{code changed<br/>with nothing run?}
|
|
173
|
+
gate -->|yes, once| agent
|
|
174
|
+
gate -->|no| evaluator
|
|
175
|
+
|
|
176
|
+
evaluator{{evaluator}} -->|rejected + why| agent
|
|
177
|
+
evaluator -->|approved| learn[distil at most<br/>three lessons]
|
|
178
|
+
learn --> done([final answer])
|
|
179
|
+
|
|
180
|
+
agent -->|ask_user| pause([paused for a question])
|
|
181
|
+
|
|
182
|
+
subgraph tools [18 tools]
|
|
183
|
+
direction LR
|
|
184
|
+
shell_and_python
|
|
185
|
+
files
|
|
186
|
+
browser
|
|
187
|
+
exercise
|
|
188
|
+
screen
|
|
189
|
+
code_map
|
|
190
|
+
recall_memory
|
|
191
|
+
web_search
|
|
192
|
+
end
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
The pieces, each documented in its own folder:
|
|
196
|
+
|
|
197
|
+
| folder | what lives there |
|
|
198
|
+
| --- | --- |
|
|
199
|
+
| [agent/pipeline](agent/pipeline/README.md) | the agent loop, the evaluator, the 18 tools, modes, the gates, the workspace and container seams, the document workflow |
|
|
200
|
+
| [agent/memory](agent/memory/README.md) | the tiered queue, the content-addressed store, recall, embeddings, the lesson bank, sessions |
|
|
201
|
+
| [agent/router](agent/router/README.md) | task seats, the routing table, provider adapters, learned ordering, health, pins, temperature policy |
|
|
202
|
+
| [agent/cli](agent/cli/README.md) | the TUI, the REPL, sessions, setup, and every `otto` command |
|
|
203
|
+
| [agent/eval](agent/eval/README.md) | the six benchmark harnesses, the failure taxonomy, the single-agent control |
|
|
204
|
+
| [agent/config](agent/config/README.md) | the one `.env` file Otto reads and writes |
|
|
205
|
+
| [containers](containers/README.md) | the throwaway desktop image the screen tools drive |
|
|
206
|
+
| [tests](tests/README.md) | 1,552 tests that need no key and no network |
|
|
207
|
+
| [docs](docs/README.md) | the development log, the research sources, the memory design |
|
|
208
|
+
|
|
209
|
+
## Evaluation
|
|
210
|
+
|
|
211
|
+
Six harnesses, each grading by something outside the model: a checker, a
|
|
212
|
+
repository's own tests, a benchmark's own graders, or a planted constraint.
|
|
213
|
+
Every report carries the number of model calls beside the score and a
|
|
214
|
+
fingerprint of the grading path; a run that measured nothing refuses to print
|
|
215
|
+
a number, and a single Claw-Eval run is labelled as not evidence because
|
|
216
|
+
identical runs swing by 0.36. Details, results and the rules the harnesses
|
|
217
|
+
enforce: [agent/eval/README.md](agent/eval/README.md).
|
|
218
|
+
|
|
219
|
+
## Testing
|
|
220
|
+
|
|
221
|
+
1,552 tests pass and 12 skip on macOS, Linux and Windows in under two minutes,
|
|
222
|
+
with no API keys and no network. Tests assert on the messages handed to the
|
|
223
|
+
model, on the exact inputs that broke real runs, on call counts against the
|
|
224
|
+
real compiled graph, and directly on the library behaviours the code relies
|
|
225
|
+
on. What is covered and how: [tests/README.md](tests/README.md).
|
|
226
|
+
|
|
227
|
+
## Research
|
|
228
|
+
|
|
229
|
+
Each design decision traces to a published finding, listed with the number
|
|
230
|
+
that motivated it and where it landed in the code:
|
|
231
|
+
[docs/RESEARCH.md](docs/RESEARCH.md).
|
|
232
|
+
|
|
233
|
+
## Limitations
|
|
234
|
+
|
|
235
|
+
- The shell is not sandboxed on the host; only the file tools are confined.
|
|
236
|
+
- `code_map` covers Python only, by reading `ast`, and refuses other languages by name rather than answering partially.
|
|
237
|
+
- `browse` and `browse_act` each drive a fresh page and restore cookies and storage from disk; in-page state that never touches storage does not survive between calls. `exercise` exists for sequences that need it.
|
|
238
|
+
- The Android, iOS, Linux and Windows `exercise` drivers are covered by faked commands only; macOS was driven for real (issues #5 to #8).
|
|
239
|
+
- Screen grounding is a description plus coordinates; expect look, act, look again.
|
|
240
|
+
- Benchmark results are mostly single runs on small samples, and the harnesses say so.
|
|
241
|
+
- Inception's Mercury models are absent from the price table on purpose, so a turn on them shows an unpriced marker rather than a guess.
|
|
242
|
+
|
|
243
|
+
## Publishing
|
|
244
|
+
|
|
245
|
+
Releases are built and published to PyPI by `.github/workflows/publish.yml`
|
|
246
|
+
through PyPI's trusted publishing, so no token is stored anywhere. It runs
|
|
247
|
+
when a GitHub release is published, or by hand from the Actions tab. The
|
|
248
|
+
one-time setup on pypi.org is a pending trusted publisher for the project
|
|
249
|
+
`otto-cli-agent`: owner `siddharth23P`, repository `otto_agent`, workflow
|
|
250
|
+
`publish.yml`, environment `pypi`. PyPI's project page uses
|
|
251
|
+
[docs/pypi.md](docs/pypi.md) as its description, since PyPI cannot render
|
|
252
|
+
this page's relative images.
|
|
253
|
+
|
|
254
|
+
## License
|
|
255
|
+
|
|
256
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# Security
|
|
2
|
+
|
|
3
|
+
Otto runs shell commands, edits files and drives a browser on the machine it
|
|
4
|
+
is installed on. The workspace boundary confines the file tools; the shell is
|
|
5
|
+
not confined, and the README says so. Please read that section before
|
|
6
|
+
running it against anything you have not committed.
|
|
7
|
+
|
|
8
|
+
## Reporting a vulnerability
|
|
9
|
+
|
|
10
|
+
Report privately through
|
|
11
|
+
[GitHub's security advisory form](https://github.com/siddharth23P/otto_agent/security/advisories/new)
|
|
12
|
+
rather than a public issue. Include what you ran, what happened, and why it
|
|
13
|
+
matters. You will get an acknowledgement within a few days.
|
|
14
|
+
|
|
15
|
+
Issues of this kind are in scope: a tool reaching outside the workspace root,
|
|
16
|
+
a URL check that can be bypassed, a prompt that lets a page steer the next
|
|
17
|
+
request, secrets written anywhere but `.env`.
|