autoforge-agent 0.4.0__tar.gz

autoforge_agent-0.4.0/PKG-INFO

@@ -0,0 +1,477 @@
+Metadata-Version: 2.4
+Name: autoforge-agent
+Version: 0.4.0
+Summary: Agent framework with verifiable self-made tools (the forge pipeline).
+Author: Zhang Yangyi
+License: MIT
+Keywords: agents,llm,tools,self-improvement,verification
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.28
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-xdist; extra == "dev"
+
+# autoforge
+
+An agent framework that **builds its own tools** — and makes them earn their keep.
+
+Most agent frameworks stop at "the model can call tools." A few let the model
+*write* tools. Almost none ask the question that actually decides whether
+self-made tools help or hurt:
+
+> **Does the tool fire when it should, and stay quiet when it shouldn't?
+> And does it keep working after the tenth call?**
+
+`autoforge` is built around that question. Tool creation is free; tool *trust*
+is earned through a three-stage closed loop, and kept honest by a live ledger.
+
+```
+FORGE ──────> VERIFY ──────> SEAL      (creation)
+generate      5-check gate    register in lifecycle
+
+OBSERVE ────> JUDGE ───────> ACT       (maintenance)
+ledger        degraded?       quarantine / rehab / retire
+```
+
+---
+
+## Quick start
+
+One command, any machine, nothing to clone:
+
+```bash
+pip install autoforge-agent
+auto setup                          # one wizard: provider, key, model
+auto                                # the REPL
+```
+
+`autoforge-agent` is the distribution name; the import is `autoforge` and the
+command is `auto` (or `autoforge`).
+
+Working from a checkout instead:
+
+```bash
+git clone https://github.com/<you>/autoforge && cd autoforge
+./auto                              # macOS / Linux / WSL / Git Bash
+auto                                # Windows cmd     (PowerShell: .\auto)
+```
+
+The launcher checks your Python (3.10+), fetches `requests` once if it is
+missing, and drops you into the REPL. Nothing else to set up. `auto.cmd` ships
+alongside it so Windows works without touching your PATH.
+
+Prefer `auto` on your PATH everywhere? Install the console script instead:
+
+```bash
+pip install -e .
+auto                                # now a real entry point, any directory
+```
+
+### Configure once, then forget about it
+
+```bash
+auto setup
+```
+
+One wizard: pick a provider, paste a key, confirm the model. It writes
+`~/.autoforge/config.json`, tests the endpoint with a single short request, and
+from then on `auto` just runs — in this terminal and in every new one.
+
+```
+  provider:
+   * 1) aiping.cn gateway (hosted, needs an API key)
+     2) Ollama on this machine (local, no key)
+     3) Something else (any OpenAI-compatible endpoint)
+```
+
+Every prompt shows the current value and takes it on a bare Enter, so re-running
+the wizard is how you change one field without retyping the rest. To see what is
+actually in effect, and which layer supplied each value:
+
+```bash
+auto config
+```
+
+```
+  base_url   https://aiping.cn/api/v1  config C:\Users\you\.autoforge\config.json
+  model      DeepSeek-V4.1-Flash       config C:\Users\you\.autoforge\config.json
+  api_key    QC-5...e262 (len 68)      config C:\Users\you\.autoforge\config.json
+  max_tokens 3000                      config C:\Users\you\.autoforge\config.json
+  proxy      True                      config C:\Users\you\.autoforge\config.json
+```
+
+Resolution order is **flag > environment > config file > default**, so a one-off
+override never means re-running the wizard:
+
+```bash
+auto --model other-model                    # this invocation only
+AUTOFORGE_MODEL=other-model auto            # this shell only
+auto --base-url http://127.0.0.1:11434/v1 --model qwen2.5:7b --no-proxy
+auto --policy supervised                    # keep the harness, drop the latitude
+AUTOFORGE_POLICY=supervised auto            # same, this shell only
+```
+
+Why bother with a file when environment variables exist: a variable exported
+*after* a terminal was opened is invisible to that terminal — Windows and POSIX
+alike inherit the environment at process start. That turns configuration into
+"it worked a minute ago, in the other window." A file has no such lag, which is
+why `auto setup` is the supported path and the env vars are the escape hatch.
+
+`auto setup` is also safe to run with no terminal: without a tty it takes
+whatever the flags and environment already say, saves them, and exits instead of
+blocking on a prompt.
+
+`auto` on its own drops you into a REPL: type a recurring need in plain language
+and the agent decides whether to forge, verify and keep a tool for it. Forged
+tools persist for the rest of the session.
+
+```bash
+auto --help                         # all flags
+auto forge "<need>" --out t.json    # forge one tool, one shot, then exit
+auto list                           # inspect artifacts written by --out
+auto list autoforge_tools/t.json    # dump one artifact
+```
+
+Point it at a local model (no key, no network) — either through `auto setup`
+above, or per-invocation:
+
+```bash
+# local, CPU-friendly: drops the LLM-driven checks
+AUTOFORGE_BASE_URL=http://127.0.0.1:11434/v1 AUTOFORGE_MODEL=qwen2.5:7b \
+  AUTOFORGE_FAST=1 auto
+```
+
+Local endpoints never use the socks proxy and never need a key, so those two
+questions are skipped automatically.
+
+Inside the REPL: `/help`, `/tools` (library + health), `/report` (policy and
+self-amendments), `/trace` (the decision log), `/reset`, `/quit`.
+
+### The keyboard stays yours
+
+A run is not a modal dialog. In `chat` the input line is live the whole time —
+the run narrates itself *above* it, so you can keep typing while a tool is
+running:
+
+```
+  [14:02:11] turn 1 +0.4s  asking the model…
+  [14:02:19] +8.6s  -> bash
+      … waiting on model (12s)
+  you> also handle the empty file case
+```
+
+Type a sentence mid-run and it reaches the agent at its next step, labelled as
+a correction rather than a new task. `/status` asks where the run is, `/stop`
+ends it after the current step, and a leading space sends something that starts
+with `/` as text. Paste a block and it collapses to a one-line
+`[Pasted text #1: 40 lines → …]` placeholder — the text is kept on disk and
+expanded again before it reaches the model.
+
+The behaviour degrades honestly: a pipe, a redirect, or a test gets the plain
+cooked-mode reader and no heartbeat, because there is no terminal to own.
+
+Offline demo, zero API keys:
+
+```bash
+python examples/demo_offline.py
+python -m pytest tests/ -o addopts= -q
+```
+
+Use the library directly:
+
+```python
+import os
+from autoforge import ForgeAgent, OpenAICompatClient
+
+llm = OpenAICompatClient(
+    model="deepseek-v3",
+    base_url="https://aiping.cn/api/v1",
+    api_key=os.environ["AIPING_API_KEY"],
+)
+agent = ForgeAgent(llm)
+print(agent.run("Reverse this string: hello").content)
+```
+
+---
+
+## The problem this solves
+
+Four failure modes kill self-made-tool agents in practice. Each has a paper or
+a production report behind it, and each maps to one mechanism here.
+
+| Failure | What happens | autoforge's answer |
+|:---|:---|:---|
+| **Constraint Tax** | A tool is correct but the agent never calls it — the trigger is unreachable | `ToolVerifier.check_trigger` — positive probes run against a live agent |
+| **Over-triggering** | A tool fires on everything, stealing calls from better tools | `check_negative` — negative probes assert it stays quiet |
+| **Misevolution** | A tool works at birth, then silently rots | `ToolStats` ledger + auto-quarantine below threshold |
+| **Semantic routing** | Retrieval picks the tool that *reads* right, not the one that *runs* right | `BehaviourRouter` — scores on observed success, not just text |
+
+Sources: the `Constraint Tax` effect (schema constraints starving tool-call
+tokens), ICLR'26 "Misevolution" (self-evolving agents introducing regressions
+via tool creation/reuse), Memento-Skills (behaviour-aligned skill routing).
+
+---
+
+## Architecture
+
+```
+autoforge/
+├── core/
+│   ├── message.py     # provider-neutral Message / ToolCall
+│   ├── llm.py         # LLMClient protocol; OpenAI-compat + Mock impls
+│   └── agent.py       # the tool-calling loop (deliberately thin)
+├── tools/
+│   ├── spec.py        # ToolSpec + ToolState lifecycle + ToolStats ledger
+│   ├── registry.py    # hot-swap registry, auto-quarantine, reporting
+│   └── composition.py # DAG composition — tools built out of tools
+├── forge/
+│   ├── sandbox.py     # out-of-process execution + measured reach report
+│   ├── generator.py   # LLMToolGenerator + offline TemplateGenerator
+│   ├── verifier.py    # execution + robustness + adversarial + trigger + negative
+│   ├── pipeline.py    # forge→verify→seal, plus judge/rehab
+│   ├── fuzzer.py      # 30+ edge probes per tool (robustness)
+│   ├── invariance.py  # metamorphic oracles — probes mean nothing without them
+│   ├── adversary.py   # an LLM attacker that tries to break each tool
+│   ├── evolution.py   # population competition — mutants race, best survives
+│   ├── validity.py    # independent gate + frozen baseline (anti-misevolution)
+│   └── metacog.py     # proactive gap discovery + pre-forging
+├── route/
+│   └── router.py      # behaviour-aligned tool routing
+├── gpu/
+│   ├── bench.py       # benchmarking that cannot lie about its units
+│   └── units.py       # ms-vs-seconds audits, the do_bench lesson as code
+├── cpu/
+│   ├── probe.py       # what this machine actually is, measured not assumed
+│   ├── ops.py         # the problem set: a described problem, not an instance
+│   ├── kernel.py      # compile → cache → load, refusing targets it cannot run
+│   ├── safety.py      # preflight: the checks that run before anything executes
+│   └── tune.py        # legal search over flags/source, verified then raced
+├── autonomy/
+│   ├── policy.py      # AutonomyPolicy — freedom is the default, all True
+│   ├── selfmod.py     # every self-modification, with rationale + audit log
+│   ├── spawn.py       # derive child agents (shared or isolated registry)
+│   └── topology.py    # Topology + TopologyDesigner — agent designs its own team
+├── store.py           # SQLite persistence (tools, versions, deps, baselines, events)
+└── agent.py           # ForgeAgent — everything wired together
+```
+
+### Tool lifecycle
+
+```
+DRAFT ──verify──> PROBATION ──earn──> ACTIVE ──decay──> QUARANTINED
+                     ^                    │                  │
+                     └──────rehab─────────┴──────────────────┘
+                                        └──retire──> RETIRED
+```
+
+Only `PROBATION` and `ACTIVE` are injected into the model's context. Creation
+is unrestricted — you can forge anything — but **context budget and trust are
+earned**, which is what keeps a growing tool library from drowning the prompt.
+
+### What makes it different
+
+| Framework | Its gap | autoforge |
+|:---|:---|:---|
+| Hermes Agent | Skills are *documents*, not executable; tool changes need a session reset | Tools are code, hot-swapped, verified |
+| Claude Code | Fixed toolset; permissions govern *calling*, not *creating* | Governs the tool lifecycle itself |
+| OpenHands / CodeAct | Action-as-code, but no quality governance | Same expressiveness, plus a ledger |
+| Voyager | Has a skill library + self-verification, but retrieves by text | Retrieves by behaviour |
+| Tea Agent / ATLASS | Can forge tools; trigger problem unsolved | Trigger + negative verification |
+
+---
+
+## Design decisions
+
+**Tools are data.** A `ToolSpec` is a claim with a contract: implementation,
+schema, trigger probes, effect signature, provenance, verification record, and
+a live reliability ledger. Nothing about a tool is a boolean.
+
+**Sandbox bounds blast radius, not capability.** Process isolation + timeout +
+env scrubbing — so an infinite loop or an `os._exit()` can't take down the
+agent. Builtins are *not* crippled by default: an agent that can't use the
+standard library can't forge useful tools. `restrict_builtins=True` narrows the
+surface when you want it, and `Sandbox(runner=...)` is the hook for real OS-level
+containment.
+
+**Verification asks five orthogonal questions.** "Does it run?" "Does it survive
+garbage input?" "Can an adversary talk it into misbehaving?" "Does it fire when
+it should?" and "Does it stay quiet when it shouldn't?" Most frameworks only
+test the first. All five are checked, and the negative probe guards the failure
+mode that's worse than silence.
+
+**Retries feed the failure back.** A failed verification becomes the next
+generation prompt's context, so the model repairs rather than re-rolls.
+
+**Quarantine is a trust signal, not a wall.** A degraded tool leaves the
+context but `force=True` still runs it. `rehab()` puts it back on trial.
+
+**Routing weights are inspectable policy**, not an opaque vector index:
+
+```
+score = w_text·similarity + w_success·success_rate + w_trust·state_trust
+        − w_cost·cost_penalty − w_over·over_trigger_penalty
+```
+
+**Every freedom is declared, and each declaration is classified.** Two presets
+ship — `full` (the default: nothing denied) and `supervised` — selected with
+`--policy` or `AUTOFORGE_POLICY`, and `autoforge config` prints which is live.
+Each field in the policy is labelled *enforced* (a gate you can watch close),
+*partial*, *confirm* (off means "not without a yes": the tool stops and asks
+before it runs), or *declared-only* (a promise no code path keeps yet — an
+empty class today, kept so a future unclassified field shows up loudly). The
+report is not decoration: `my_capabilities` hands the same classification to the
+agent, `set_autonomy` tells you whether switching a freedom off closes a door,
+narrows one, or turns it into a question, and `describe()` prints
+`[asks before running: ...]` — so "switched off" can never quietly mean
+"still on".
+
+The four execution freedoms (`may_read_filesystem`, `may_write_filesystem`,
+`may_access_network`, `may_install_packages`) are the *confirm* ones. Switching
+one off makes `ToolRegistry.call` ask, once per run, about any tool whose own
+declared scope needs it — a tool that declares nothing is treated as capable of
+everything and is therefore asked about too. Nobody to ask (a headless run, the
+web harness's worker threads) means it does not run; an unanswered prompt is
+never a yes. On a terminal the question is asked by `cli._TerminalConfirmer`,
+which defaults to No and prints the tool, the switch and the arguments. Tools
+declare their scope in `agent.BUILTIN_SCOPES`; a test fails if the table and the
+tool specs ever disagree.
+
+---
+
+## Proof it works
+
+`examples/demo_offline.py` runs three parts with no network:
+
+**Part 1** — forges `word_stats` from a one-line need, verifies it
+(execution + 2 triggers + negative), promotes it to ACTIVE, then calls the
+forged code out-of-process and confirms it's injected into context.
+
+**Part 2** — a tool's upstream breaks. The ledger catches the rot on the third
+failure, auto-quarantines it, removes it from context, blocks further calls —
+then `rehab()` puts it back on trial.
+
+**Part 3** — two near-identically-described JSON parsers. The one with **higher
+text similarity** has a 0% success rate; the router correctly picks the other
+one. This is the whole argument for behaviour-aligned routing in one output.
+
+`examples/demo_misevolution.py` reproduces three ways a self-evolving tool
+population goes wrong, then asserts each is closed:
+
+| Exploit | Old behaviour | Now |
+|:---|:---|:---|
+| Delete your own guardrail | 0.800 → **1.000** for identical behaviour | Neutralised — a missing check class is a failed check class |
+| Shrink the exam | 6-probe tool tied a 50-probe tool | Evidence mass decides, and saturates so it can't be farmed |
+| Silent scope creep | Undetected | Boolean veto, audited, before fitness is ever computed |
+
+A fourth exploit lived one layer down, in the robustness check itself — and it
+was the worst of the set, because it sat on the **forge** path, where every tool
+ever created has to pass through it:
+
+| Exploit | Old behaviour | Now |
+|:---|:---|:---|
+| Ignore `"ISBN "` prefix | 19/19 probes survived, **passed** | Fails: output changes under a transform that must not change it |
+| Return a constant `"nope"` | 19/19 probes survived, **passed** | Fails: degenerate — validates nothing |
+| Return constant `True` | 19/19 probes survived, **passed** | Fails: degenerate — accepts everything |
+
+The probes were never the problem — the `ISBN ` probe was already being
+generated. The problem was the oracle: a probe was scored `survived` iff the
+tool did not raise (`ok = out is not None`), so *wrong-but-total* functions
+scored 100%. A probe means nothing without a verdict behind it.
+
+`forge/invariance.py` supplies verdicts of two kinds, neither authored by the
+tool being scored:
+
+- **computed** — true of any honest implementation, so the verifier derives
+  them: `defined`, `deterministic`, and `non_degenerate` (a tool must not emit
+  one constant across well-formed *and* garbage input).
+- **declared** — semantic obligations needing domain knowledge the verifier
+  lacks (is `"ISBN "` part of the value or noise around it?). The generator
+  asserts these at birth; `FrozenBaseline` then keeps them, so a later mutant
+  cannot quietly drop one.
+
+They are metamorphic, not exact: nothing labels the correct output for a novel
+input, but you can still assert how outputs must *relate*. Scope is
+deliberately narrow — free-text parameters like `title` get no relations at
+all, because there the whitespace and casing are the content.
+
+The fix has three parts, and none of them is a bigger penalty term:
+
+1. **The denominator is fixed.** `_compute_fitness` scores against
+   `REQUIRED_CHECK_CLASSES`, not against however many checks the mutant chose to
+   declare. Omitting a probe scores exactly as if it failed.
+2. **The gate is independent and boolean.** `ValidityGate` runs *before*
+   verification and returns admissible / not. A veto is not a number a mutant
+   can out-earn by being good at the task — there is no trade to make.
+3. **The baseline only grows.** `FrozenBaseline` pins the obligations a tool
+   had when it was trusted, persists them, and `extended_with` lets it absorb
+   *new* probes permanently. Without the ratchet, each generation freezes its
+   own predecessor and guardrails erode one step at a time.
+
+---
+
+## Extending
+
+Add a generator:
+
+```python
+class MyGenerator:
+    def generate(self, need: str, context: str = "") -> GeneratedTool: ...
+```
+
+Add a sandbox backend (Docker, nsjail, a cloud runner):
+
+```python
+Sandbox(runner=lambda code, entry, args: SandboxResult(...))
+```
+
+Swap the model — anything speaking `/chat/completions`:
+
+```python
+OpenAICompatClient(model=..., base_url=..., api_key=..., proxies={...})
+```
+
+---
+
+## Status
+
+v0.4.0 — the self-growth layer is complete: forge → verify (execution,
+robustness, adversarial, trigger, negative) → seal, plus evolution, proactive
+gap-filling, tool composition, self-modification with an audit trail, agent
+spawning, and self-designed multi-agent topology. On top of it an
+anti-misevolution layer: an independent validity gate, a fitness function the
+mutant cannot author, a frozen baseline that only ratchets forward, Pareto
+selection so safety cannot be paid for with capability, and metamorphic
+oracles so the robustness layer actually has a verdict. Underneath it a native
+layer: the machine is measured rather than assumed, a target ISA the host cannot
+run is refused before the compile rather than after the crash, a benchmark's
+units are audited so a `ms` label cannot sit over a 1000x value, and kernel
+search reports its own noise floor instead of promoting the fastest noisy run.
+1142 tests passing. MIT.
+
+**Breaking since v0.3.0:** the robustness check now has an oracle. Tools that
+previously passed it by not raising will fail if they are degenerate or break a
+declared normalisation relation.
+
+Known limits: the default sandbox is process isolation, not a security
+boundary against adversarial code (`restrict_builtins` narrows it; use a
+`runner` for real containment). A *confirm* freedom has no approval flow in the
+browser: the web harness runs agents in worker threads with no terminal, so it
+answers "nobody to ask" and refuses rather than hanging a request on input it
+cannot show you. Running `supervised` over the web UI therefore refuses the
+gated tools instead of prompting for them — the CLI is the surface where the
+question can actually be put to a person. It also does **not** separate the agent from the
+host: forged code runs as a subprocess of the agent process on the same machine,
+with the whole host filesystem and outbound network. `Sandbox.reach(probe=True)`
+measures that with a real round-trip rather than asserting it, and
+`my_capabilities` reports the measurement — because an agent that answers "can
+you reach my machine?" from its tool list gets the answer wrong. The same rule
+covers memory: the tool ledger is sqlite on disk (`ToolStore.report()`), forged
+tools are persisted so they survive a restart, and `my_history` reads the ledger
+and the self-modification log back. A measured self-report is appended to the
+system prompt on every request, so the agent's description of itself is
+recomputed from the machine instead of drifting as prose. Routing uses lexical
+similarity rather than embeddings — swap `text_similarity` for a vector index
+when the library is big enough to need it. `RoleType.FORGE` is declarative only:
+a topology names a forge role, it does not yet change which tools that child can
+reach.