archforge-optimizer 0.3.0__tar.gz → 0.4.0__tar.gz

archforge_optimizer-0.3.0/README.md → archforge_optimizer-0.4.0/PKG-INFO

@@ -1,3 +1,36 @@
+Metadata-Version: 2.5
+Name: archforge-optimizer
+Version: 0.4.0
+Summary: ArchForge: a self-improving meta-layer over multi-agent systems
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: litellm
+Requires-Dist: pydantic>=2.7
+Requires-Dist: python-dotenv>=1.0
+Provides-Extra: deepeval
+Requires-Dist: deepeval; extra == 'deepeval'
+Provides-Extra: dev
+Requires-Dist: hypothesis>=6; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: providers
+Requires-Dist: anthropic>=0.40; extra == 'providers'
+Requires-Dist: google-genai>=1.0; extra == 'providers'
+Requires-Dist: groq>=0.11; extra == 'providers'
+Requires-Dist: openai>=1.40; extra == 'providers'
+Provides-Extra: providers-anthropic
+Requires-Dist: anthropic>=0.40; extra == 'providers-anthropic'
+Provides-Extra: providers-gemini
+Requires-Dist: google-genai>=1.0; extra == 'providers-gemini'
+Provides-Extra: providers-groq
+Requires-Dist: groq>=0.11; extra == 'providers-groq'
+Provides-Extra: providers-openai
+Requires-Dist: openai>=1.40; extra == 'providers-openai'
+Description-Content-Type: text/markdown
+
 # ArchForge
 
 <p align="center">
@@ -215,8 +248,17 @@ archforge-optimizer approve --all   # move PENDING_HUMAN structural wins into ac
 
 The `--provider` flag selects the LLM backing the Architect + Judge (`anthropic` / `openai` / `groq` / `gemini` for real runs). Every real provider goes through **one LiteLLM client**: the provider just prefixes the model id (`openai/gpt-4o`, `gemini/gemini-3.6-flash`, …). The host MAS is wired via `--adapter my_pkg.my_host:MyAdapter`. After `init`, `evolve` already defaults it to `archforge_optimizer.host:AppAdapter`, so you only pass the flag for a custom adapter.
 
+### Evaluation backends
+
+The Judge is pluggable behind one `JudgeProtocol` seam (`score` / `score_suite`), so the optimizer never knows which evaluator produced a score. `--evaluator` picks the backend (default `DEFAULT_EVALUATOR="native"` in `.archforge/archforge.py`):
+
+- **`native`** (default): the built-in LLM-as-judge. One structured call to your `--provider` model scores every rubric dimension plus a per-step breakdown (used for credit assignment).
+- **`deepeval`**: the external [DeepEval](https://deepeval.com) backend (optional: `pip install "archforge-optimizer[deepeval]"`). Each run is projected into a DeepEval `LLMTestCase` and scored by standalone metrics (`--deepeval-metric answer_relevancy --deepeval-metric faithfulness`, or the `DEFAULT_DEEPEVAL_METRICS` tunable). Metric scores land in `RunScore.rubric_scores`; the aggregate is their mean, comparable to the native judge's [0,1] aggregate. DeepEval is run-level, not per-step, so `step_scores` is empty and credit assignment degrades gracefully (the Architect falls back to conservative, blame-free proposals). The judge model is configurable: it reuses the `--judge-model` / `DEFAULT_JUDGE_MODELS` seam, prefixed with your `--provider` (e.g. `gemini/gemini-3.6-flash`) and routed through LiteLLM, so it scores with the same vendor and env keys as the rest of the run, never DeepEval's OpenAI default.
+
 ---
 
+
+
 ## The optimization loop (P-E-C)
 
 One cycle, end-to-end:
@@ -296,6 +338,8 @@ archforge-optimizer <command> [flags]
 | `--seed <path>` | bootstrap the root incumbent from a Spec JSON (first run); defaults to `archforge_optimizer/spec.json` when present |
 | `--adapter <dotted.path[:Class]>` | your `HostMAS` adapter; defaults to `archforge_optimizer.host:AppAdapter` when the scaffold is present (not on the `--provider scripted` fake path) |
 | `--provider {scripted\|anthropic\|openai\|groq\|gemini}` | LLM backing the Architect + Judge |
+| `--evaluator {native\|deepeval}` | evaluation backend (default: `DEFAULT_EVALUATOR`); `deepeval` needs the `[deepeval]` extra |
+| `--deepeval-metric {answer_relevancy\|faithfulness}` | DeepEval metric to score with (repeatable; default: `DEFAULT_DEEPEVAL_METRICS`) |
 | `--suite <path>` | evaluation suite JSON (default: `.archforge/suite.json`) |
 | `--tau <float>` | promotion margin τ |
 | `--delta <float>` | regression floor δ (≥ τ) |

archforge_optimizer-0.3.0/PKG-INFO → archforge_optimizer-0.4.0/README.md

@@ -1,34 +1,3 @@
-Metadata-Version: 2.5
-Name: archforge-optimizer
-Version: 0.3.0
-Summary: ArchForge: a self-improving meta-layer over multi-agent systems
-License-Expression: MIT
-License-File: LICENSE
-Requires-Python: >=3.11
-Requires-Dist: litellm
-Requires-Dist: pydantic>=2.7
-Requires-Dist: python-dotenv>=1.0
-Provides-Extra: dev
-Requires-Dist: hypothesis>=6; extra == 'dev'
-Requires-Dist: mypy>=1.11; extra == 'dev'
-Requires-Dist: pytest-cov>=5; extra == 'dev'
-Requires-Dist: pytest>=8; extra == 'dev'
-Requires-Dist: ruff>=0.5; extra == 'dev'
-Provides-Extra: providers
-Requires-Dist: anthropic>=0.40; extra == 'providers'
-Requires-Dist: google-genai>=1.0; extra == 'providers'
-Requires-Dist: groq>=0.11; extra == 'providers'
-Requires-Dist: openai>=1.40; extra == 'providers'
-Provides-Extra: providers-anthropic
-Requires-Dist: anthropic>=0.40; extra == 'providers-anthropic'
-Provides-Extra: providers-gemini
-Requires-Dist: google-genai>=1.0; extra == 'providers-gemini'
-Provides-Extra: providers-groq
-Requires-Dist: groq>=0.11; extra == 'providers-groq'
-Provides-Extra: providers-openai
-Requires-Dist: openai>=1.40; extra == 'providers-openai'
-Description-Content-Type: text/markdown
-
 # ArchForge
 
 <p align="center">
@@ -246,8 +215,17 @@ archforge-optimizer approve --all   # move PENDING_HUMAN structural wins into ac
 
 The `--provider` flag selects the LLM backing the Architect + Judge (`anthropic` / `openai` / `groq` / `gemini` for real runs). Every real provider goes through **one LiteLLM client**: the provider just prefixes the model id (`openai/gpt-4o`, `gemini/gemini-3.6-flash`, …). The host MAS is wired via `--adapter my_pkg.my_host:MyAdapter`. After `init`, `evolve` already defaults it to `archforge_optimizer.host:AppAdapter`, so you only pass the flag for a custom adapter.
 
+### Evaluation backends
+
+The Judge is pluggable behind one `JudgeProtocol` seam (`score` / `score_suite`), so the optimizer never knows which evaluator produced a score. `--evaluator` picks the backend (default `DEFAULT_EVALUATOR="native"` in `.archforge/archforge.py`):
+
+- **`native`** (default): the built-in LLM-as-judge. One structured call to your `--provider` model scores every rubric dimension plus a per-step breakdown (used for credit assignment).
+- **`deepeval`**: the external [DeepEval](https://deepeval.com) backend (optional: `pip install "archforge-optimizer[deepeval]"`). Each run is projected into a DeepEval `LLMTestCase` and scored by standalone metrics (`--deepeval-metric answer_relevancy --deepeval-metric faithfulness`, or the `DEFAULT_DEEPEVAL_METRICS` tunable). Metric scores land in `RunScore.rubric_scores`; the aggregate is their mean, comparable to the native judge's [0,1] aggregate. DeepEval is run-level, not per-step, so `step_scores` is empty and credit assignment degrades gracefully (the Architect falls back to conservative, blame-free proposals). The judge model is configurable: it reuses the `--judge-model` / `DEFAULT_JUDGE_MODELS` seam, prefixed with your `--provider` (e.g. `gemini/gemini-3.6-flash`) and routed through LiteLLM, so it scores with the same vendor and env keys as the rest of the run, never DeepEval's OpenAI default.
+
 ---
 
+
+
 ## The optimization loop (P-E-C)
 
 One cycle, end-to-end:
@@ -327,6 +305,8 @@ archforge-optimizer <command> [flags]
 | `--seed <path>` | bootstrap the root incumbent from a Spec JSON (first run); defaults to `archforge_optimizer/spec.json` when present |
 | `--adapter <dotted.path[:Class]>` | your `HostMAS` adapter; defaults to `archforge_optimizer.host:AppAdapter` when the scaffold is present (not on the `--provider scripted` fake path) |
 | `--provider {scripted\|anthropic\|openai\|groq\|gemini}` | LLM backing the Architect + Judge |
+| `--evaluator {native\|deepeval}` | evaluation backend (default: `DEFAULT_EVALUATOR`); `deepeval` needs the `[deepeval]` extra |
+| `--deepeval-metric {answer_relevancy\|faithfulness}` | DeepEval metric to score with (repeatable; default: `DEFAULT_DEEPEVAL_METRICS`) |
 | `--suite <path>` | evaluation suite JSON (default: `.archforge/suite.json`) |
 | `--tau <float>` | promotion margin τ |
 | `--delta <float>` | regression floor δ (≥ τ) |

{archforge_optimizer-0.3.0 → archforge_optimizer-0.4.0}/archforge/cli.py

@@ -67,10 +67,15 @@ from archforge import userconfig as ucfg
 from archforge.config import (
     ALL_PROVIDERS as _PROVIDERS,
     DEFAULT_SUITE_ID, DEFAULT_TASK_ID, DEFAULT_TASK_INPUT,
+    EVALUATORS as _EVALUATORS,
     PROG, load_env,
 )
 from archforge.userconfig import ConfigNotInitialized
 
+# The DeepEval metrics the `--deepeval-metric` flag accepts (its `choices`). Kept in
+# sync with the factory registry in archforge/judge/deepeval.py.
+_DEEPEVAL_METRICS = ("answer_relevancy", "faithfulness")
+
 # Fixed run-state / config-discovery dir (a system path, independent of the
 # tunable DEFAULT_ROOT_DIR which the embedder API reads via archforge.userconfig).
 _DEFAULT_ROOT = ".archforge"
@@ -105,12 +110,42 @@ def _import_adapter(dotted: str) -> HostMAS:
     its own ``__init__`` carries whatever its MAS needs (Lumina loads its base
     prompts; a framework adapter wraps its graph). No PR into core to adapt a
     new MAS: ``--adapter mypkg:MyAdapter`` wires it; ``--provider`` keeps the
-    Architect/Judge organs, ``--seed`` the bootstrap Spec."""
+    Architect/Judge organs, ``--seed`` the bootstrap Spec.
+
+    Surfaces clear, actionable errors for the common failure modes users hit when
+    editing the scaffold's ``app.py``:
+
+      * ``ImportError``/``ModuleNotFoundError`` — a MAS dependency listed in
+        ``app.py``'s imports is not installed (e.g. ``from mymas.graph import
+        build_graph`` when ``mymas`` isn't on ``sys.path``).
+      * ``KeyError`` — ``node_ids(build_graph())`` can't find a node name that
+        ``_GID[...]`` references (a rename in ``build_graph`` that wasn't
+        reflected in ``_GID``).
+    """
     if ":" in dotted:
         modpath, cls = dotted.split(":", 1)
     else:
         modpath, cls = dotted, ""
-    module = importlib.import_module(modpath)
+    try:
+        module = importlib.import_module(modpath)
+    except ModuleNotFoundError as exc:
+        # Surface the missing package name explicitly so the user knows what to
+        # install, and point at `app.py` as the likely source of the bad import.
+        missing = exc.name or str(exc)
+        raise SystemExit(
+            f"! --adapter: cannot import {modpath!r} — missing module {missing!r}.\n"
+            f"  Check the imports at the top of archforge_optimizer/app.py: make sure\n"
+            f"  every package your MAS needs (e.g. `langgraph`, your MAS package) is\n"
+            f"  installed in the current Python environment.\n"
+            f"  Original error: {exc}"
+        ) from exc
+    except ImportError as exc:
+        raise SystemExit(
+            f"! --adapter: failed to import {modpath!r}.\n"
+            f"  Check the imports at the top of archforge_optimizer/app.py and make\n"
+            f"  sure all your MAS dependencies are installed.\n"
+            f"  Original error: {exc}"
+        ) from exc
     if not cls:
         # Bare module: expect it to expose a ``HostMAS``-protocol attr named
         # ``HostMAS`` or the last path segment; else error loudly.
@@ -123,7 +158,33 @@ def _import_adapter(dotted: str) -> HostMAS:
             f"Pass it as `module:ClassName`."
         ) from exc
     if isinstance(obj, type):
-        return obj()                 # a HostMAS/BaseHostAdapter subclass → instance
+        try:
+            return obj()             # a HostMAS/BaseHostAdapter subclass → instance
+        except KeyError as exc:
+            # Usually `_GID = node_ids(build_graph())` in app.py referencing a
+            # node name the graph doesn't define — but a KeyError can come from
+            # ANY dict/env lookup in the adapter's __init__, so name the likely
+            # cause without asserting it.
+            app_py = Path.cwd() / "archforge_optimizer" / "app.py"
+            raise SystemExit(
+                f"! --adapter: {dotted!r} raised KeyError {exc} at startup.\n"
+                f"  Most common cause: the `_GID` dict in {app_py} references a\n"
+                f"  node name that `build_graph()` never defines (check the\n"
+                f"  `add_node(..., name=...)` calls in your graph builder). If\n"
+                f"  `_GID` matches, look for another dict/env lookup in the\n"
+                f"  adapter's __init__ (e.g. os.environ[...])."
+            ) from exc
+        except AssertionError:
+            # Re-raise so `make-spec`'s own handler (which formats build_spec()
+            # lint failures as clean rc=1 output) can catch it.
+            raise
+        except Exception as exc:        # noqa: BLE001
+            app_py = Path.cwd() / "archforge_optimizer" / "app.py"
+            raise SystemExit(
+                f"! --adapter: failed to instantiate {dotted!r}.\n"
+                f"  Check {app_py} — the adapter raised an unexpected error at\n"
+                f"  startup: {type(exc).__name__}: {exc}"
+            ) from exc
     if isinstance(obj, HostMAS):
         return obj                   # already an instance
     raise SystemExit(f"--adapter: {dotted!r} resolved to a {type(obj).__name__}, "
@@ -163,6 +224,14 @@ def _add_evolve_args(p: argparse.ArgumentParser, *, loop: bool) -> None:
                    help="model id for the Architect (else the provider default)")
     p.add_argument("--judge-model", default=None,
                    help="model id for the Judge (else the provider default)")
+    p.add_argument("--evaluator", choices=_EVALUATORS, default=None,
+                   help="evaluation backend (default from archforge.py): native is the "
+                        "built-in LLM-as-judge; deepeval is the external DeepEval backend "
+                        "(needs the [deepeval] extra)")
+    p.add_argument("--deepeval-metric", choices=_DEEPEVAL_METRICS, action="append",
+                   default=None, dest="deepeval_metrics",
+                   help="DeepEval metric to score with (repeatable; else the "
+                        "DEFAULT_DEEPEVAL_METRICS tunable)")
     p.add_argument("--suite", metavar="PATH", default=None,
                    help="path to a suite.json (overrides the DEFAULT_SUITE_FILE tunable; "
                         "the file's tasks define what you optimize against)")
@@ -331,8 +400,34 @@ def _default_components(args: argparse.Namespace) -> Components:
     arch_models = ucfg.get("DEFAULT_ARCHITECT_MODELS")
     judge_models = ucfg.get("DEFAULT_JUDGE_MODELS")
     arch = Architect(llm, model=args.architect_model or arch_models[provider])
-    judge = Judge(llm, model=args.judge_model or judge_models[provider],
-                  rubric=default_rubric())
+
+    # Evaluation backend seam (issue #2): `native` keeps the built-in Judge over
+    # the provider's LLM; `deepeval` swaps in the external DeepEval backend, which
+    # scores independently of the Architect's LLM (it carries its own model/keys).
+    # `default=` on both: a config written by an OLDER init (before issue #2)
+    # lacks these names and would KeyError otherwise — the defaults keep the
+    # pre-existing native behavior for those projects.
+    evaluator = getattr(args, "evaluator", None) or ucfg.get(
+        "DEFAULT_EVALUATOR", default="native")
+    if evaluator == "deepeval":
+        from archforge.judge import make_evaluator
+
+        try:
+            metrics = getattr(args, "deepeval_metrics", None) or ucfg.get(
+                "DEFAULT_DEEPEVAL_METRICS", default=["answer_relevancy"])
+            # DeepEval scores through LiteLLM too: reuse the Judge model seam
+            # (--judge-model / DEFAULT_JUDGE_MODELS) and prefix it with the
+            # provider so LiteLLM routes to the same vendor (never DeepEval's
+            # OpenAI default).
+            from archforge.llm.litellm import prefix_model
+            de_model = prefix_model(provider, args.judge_model or judge_models[provider])
+            judge = make_evaluator("deepeval", model=de_model, metrics=metrics)
+        except LLMError as exc:
+            print(f"[evaluator] {exc}", file=sys.stderr)
+            raise
+    else:
+        judge = Judge(llm, model=args.judge_model or judge_models[provider],
+                      rubric=default_rubric())
     return Components(host=FakeHostMAS(), judge=judge, architect=arch, suite=suite)
 
 
@@ -549,6 +644,31 @@ def _scaffolded_spec_available() -> bool:
     return (Path.cwd() / "archforge_optimizer" / "spec.json").is_file()
 
 
+# Unique token written by `init` into the placeholder `build_graph` — present ONLY
+# when the user hasn't replaced it yet (spec E-guard: refuse to evolve template code).
+_TEMPLATE_MARKER = "EDIT app.py: import your MAS"
+
+
+def _scaffolded_adapter_is_template() -> bool:
+    """True when the scaffolded `app.py` still contains the unedited placeholder
+    `build_graph` (the ``NotImplementedError`` with the "EDIT app.py" marker).
+
+    This is the signal that the user ran `init` but hasn't wired their real MAS yet.
+    Returning True causes `evolve`/`evolve-loop` to refuse with a clear message
+    instead of silently running the Judge against the fake placeholder spec.
+    Only the scaffolded adapter path is checked — an explicit ``--adapter`` flag
+    skips this (the user owns that adapter).
+    """
+    app_py = Path.cwd() / "archforge_optimizer" / "app.py"
+    if not app_py.is_file():
+        return False
+    try:
+        text = app_py.read_text(encoding="utf-8")
+    except OSError:
+        return False
+    return _TEMPLATE_MARKER in text
+
+
 def _cmd_evolve(args: argparse.Namespace, *, components: Components | None,
                 loop: bool) -> int:
     # Injected `components` (the test/embedding path) always win — they ARE the
@@ -597,6 +717,26 @@ def _cmd_evolve(args: argparse.Namespace, *, components: Components | None,
     # Skipped on the scripted fake path (FakeHostMAS is the zero-cost host) and
     # when `components` were injected (test/embedding path owns the host).
     adapter_path = getattr(args, "adapter", None)
+    # Guard: refuse to run against the unedited scaffold template. When the user
+    # ran `init` but hasn't replaced `build_graph` in `app.py` yet, the placeholder
+    # Spec is valid (it lints) but the MAS is a stub — running `evolve` would silently
+    # score fake outputs through the Judge with no real pipeline. Only checked when
+    # the scaffold is auto-detected (no explicit --adapter flag) so user-supplied
+    # adapters are never blocked. The scripted path is also exempt: it keeps
+    # FakeHostMAS and never touches the scaffold, so a template app.py is inert
+    # there (and blocking it would break the zero-cost demo in a scaffolded dir).
+    if (adapter_path is None and not components_injected
+            and (args.provider or ucfg.get("PROVIDER")) != "scripted"
+            and _scaffolded_adapter_available()
+            and _scaffolded_adapter_is_template()):
+        print(
+            f"! archforge_optimizer/app.py still contains the placeholder `build_graph` "
+            f"(the scaffold written by `{PROG} init` has not been edited yet).\n"
+            f"  Edit the # EDIT: markers in archforge_optimizer/app.py to wire your real "
+            f"MAS, then re-run `{PROG} make-spec` to rebuild the Spec before evolving.",
+            file=sys.stderr,
+        )
+        return 1
     if (adapter_path is None and not components_injected
             and _scaffolded_adapter_available()
             and (args.provider or ucfg.get("PROVIDER")) != "scripted"):

{archforge_optimizer-0.3.0 → archforge_optimizer-0.4.0}/archforge/config.py

@@ -29,7 +29,7 @@ from pathlib import Path
 # =========================================================================== #
 # The package version. Read by hatchling for the built distribution and
 # re-exported as `archforge.__version__` (archforge/__init__.py). One place.
-VERSION: str = "0.3.0"
+VERSION: str = "0.4.0"
 
 
 # =========================================================================== #
@@ -46,6 +46,20 @@ REAL_PROVIDERS: tuple[str, ...] = ("anthropic", "openai", "groq", "gemini")
 ALL_PROVIDERS: tuple[str, ...] = (SCRIPTED_PROVIDER,) + REAL_PROVIDERS
 
 
+# =========================================================================== #
+# Evaluation backends — the ROSTER only (NOT the default choice; that's a tunable)
+# =========================================================================== #
+# The evaluators the CLI's `--evaluator` flag ACCEPTS (its `choices`). `native` is
+# the built-in LLM-as-judge (`Judge` over an `LLMClient`); `deepeval` is the
+# external DeepEval backend (`DeepEvalEvaluator`, an optional extra). WHICH is the
+# default is a tunable → it lives in .archforge/archforge.py (resolved by
+# archforge.userconfig), NOT here. `EVALUATORS` is re-exported from archforge.judge;
+# `make_evaluator(evaluator)` dispatches on these.
+NATIVE_EVALUATOR: str = "native"
+DEEPEVAL_EVALUATOR: str = "deepeval"
+EVALUATORS: tuple[str, ...] = (NATIVE_EVALUATOR, DEEPEVAL_EVALUATOR)
+
+
 # =========================================================================== #
 # Project environment (`.env`) — how secrets reach the provider SDKs
 # =========================================================================== #
@@ -128,6 +142,8 @@ __all__ = [
     "VERSION",
     # llm provider roster
     "SCRIPTED_PROVIDER", "REAL_PROVIDERS", "ALL_PROVIDERS",
+    # evaluation backend roster
+    "NATIVE_EVALUATOR", "DEEPEVAL_EVALUATOR", "EVALUATORS",
     # project environment (.env loader for provider API keys)
     "load_env",
     # storage layout

{archforge_optimizer-0.3.0 → archforge_optimizer-0.4.0}/archforge/config_init.py

@@ -57,6 +57,13 @@ _FIELDS: tuple[tuple[str, str, str], ...] = (
     # --- LLM provider
     ("PROVIDER", '"gemini"',
      "which LLM to use (scripted|anthropic|openai|groq|gemini); scripted needs no API key"),
+    # --- evaluation backend
+    ("DEFAULT_EVALUATOR", '"native"',
+     "which evaluator scores runs (native|deepeval); native = built-in LLM-as-judge, "
+     "deepeval = external DeepEval backend (needs the [deepeval] extra)"),
+    ("DEFAULT_DEEPEVAL_METRICS", '["answer_relevancy"]',
+     "DeepEval metrics to run per run (answer_relevancy|faithfulness); "
+     "each becomes a rubric_scores dimension and the aggregate is their mean"),
     ("DEFAULT_ARCHITECT_MODELS", _DEFAULT_ARCHITECT_MODELS,
      "default Architect (proposer) model per provider; a bare LLMClient call falls "
      "back here too — edit the dict to change it"),

archforge_optimizer-0.4.0/archforge/judge/__init__.py

@@ -0,0 +1,56 @@
+"""The Judge — LLM-as-judge scoring of runs (spec §3, §4).
+
+`Judge.score(trace, task, rubric_id) -> RunScore` turns a run's trace into a
+scored verdict: an aggregate + named rubric dimensions + confidence, plus a
+*per-step breakdown* (StepScore[]) that names which agent lost which points. The
+per-step breakdown is the raw material the Architect uses to credit-assign a
+fault to a node/route.
+
+`score_suite(...)` aggregates over R repeats: mean (down-weighted by
+confidence) and a stable aggregate used by the Gatekeeper. `rubric_id` is
+stamped on every score so cross-rubric comparisons never masquerade as
+improvement (invariant I5, spec E2).
+"""
+
+from __future__ import annotations
+
+from typing import Sequence
+
+from archforge.config import EVALUATORS
+from archforge.judge.base import (
+    Judge, JudgeProtocol, SuiteAggregate, default_rubric,
+)
+from archforge.judge.scripted import ScriptedJudge
+from archforge.llm.base import LLMError
+
+# `EVALUATORS` is re-exported (in __all__) straight from archforge.config — the
+# single source of truth the CLI also imports for its `--evaluator` choices.
+
+
+def make_evaluator(
+    evaluator: str, *, model: str | None = None, metrics: Sequence[str] | None = None
+) -> JudgeProtocol:
+    """Build an external evaluation backend by name.
+
+    Resolves `DeepEvalEvaluator` lazily (so importing DeepEval is deferred to here,
+    and the optional extra is only required at construction). Raises `LLMError` for
+    an unknown evaluator so the CLI surfaces one clear message across the backend
+    seam. The `native` evaluator is NOT built here: it is the built-in `Judge`,
+    constructed inline by the CLI from the provider's `LLMClient`.
+    """
+
+    if evaluator == "native":
+        raise LLMError(
+            "'native' is the built-in Judge; construct it with Judge(llm, model=...)"
+        )
+    if evaluator == "deepeval":
+        from archforge.judge.deepeval import DeepEvalEvaluator
+
+        return DeepEvalEvaluator(model=model, metrics=metrics)  # type: ignore[return-value]
+    raise LLMError(f"unknown evaluator {evaluator!r}; expected one of {EVALUATORS}")
+
+
+__all__ = [
+    "Judge", "JudgeProtocol", "ScriptedJudge", "SuiteAggregate", "default_rubric",
+    "make_evaluator", "EVALUATORS",
+]

archforge_optimizer-0.4.0/archforge/judge/deepeval.py

@@ -0,0 +1,236 @@
+"""The DeepEval evaluation backend — an external `JudgeProtocol` (issue #2).
+
+`DeepEvalEvaluator` implements the SAME protocol the built-in `Judge` does
+(`JudgeProtocol` in archforge/judge/base.py), so the SuiteRunner and optimizer
+treat it exactly like the native backend: `score(trace, task, rubric_id) ->
+RunScore` and `score_suite(...) -> SuiteAggregate`. Nothing downstream knows or
+cares which backend produced a score — that is the pluggability the issue asks
+for (criterion 5: the optimizer stays independent of the evaluator).
+
+Unlike the native `Judge` (one LLM call returning a whole JSON rubric verdict),
+DeepEval scores a run with one or more standalone LLM-judge METRICS
+(AnswerRelevancy, Faithfulness, ...), each yielding a score in [0,1]. We project
+an ArchForge trace into a DeepEval `LLMTestCase` and convert each metric's score
+into a `RunScore`:
+
+  * `rubric_scores` = {metric slug: metric.score}  (the named dimensions)
+  * `aggregate`      = mean of the metric scores  (comparable to the native Judge's
+                       [0,1] aggregate; invariant I5 still holds: rubric_id is
+                       stamped on every score)
+  * `step_scores`    = []  (top-level only — DeepEval is run-level, not per-step;
+                       credit assignment degrades gracefully to `blame=None`)
+
+DeepEval is imported LAZILY inside `__init__` (it is an optional extra and is
+heavy — it pulls an LLM provider SDK + more). A missing install surfaces as a
+clear `LLMError` at construction, never a bare ImportError at import time, so
+`import archforge` stays deepeval-free.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Sequence
+
+from archforge.llm.base import LLMError
+from archforge.judge.base import SuiteAggregate, aggregate_scores
+from archforge.host.base import Task
+import archforge.models as m
+
+
+# A metric name -> factory. Kept as a mapping (not `if` chains) so a configured
+# name resolves to a metric object AND gives us the stable rubric_scores slug.
+_METRIC_FACTORIES: dict[str, Any] = {}
+
+
+def _metric_slug(name: str) -> str:
+    """Canonical rubric_scores key for a DeepEval metric name (lower snake)."""
+    return name.strip().lower().replace("-", "_").replace(" ", "_")
+
+
+class DeepEvalEvaluator:
+    """Scores runs via DeepEval metrics, satisfying `JudgeProtocol`.
+
+    `metrics` are metric NAMES (`"answer_relevancy"`, `"faithfulness"`) or
+    already-constructed DeepEval metric objects. Names resolve to metric objects
+    lazily at construction (DeepEval stays unimported until then). `model` is a
+    LiteLLM-style model string (e.g. `"gemini/gemini-3.6-flash"`; prefix with the
+    provider) or an already-built DeepEval model object; a string is wrapped in
+    DeepEval's `LiteLLMModel` so scoring routes through LiteLLM with the
+    provider's env key, never DeepEval's OpenAI default. When None, DeepEval
+    falls back to its own default model.
+    """
+
+    def __init__(
+        self,
+        *,
+        metrics: Sequence[str | Any] | None = None,
+        model: str | Any | None = None,
+        threshold: float = 0.5,
+    ) -> None:
+        self._deepeval = _import_deepeval()  # LLMError if the extra is missing
+        # A string model is wrapped in LiteLLMModel (below) so the provider
+        # prefix routes correctly; `_judge_label` is only the provenance stamp
+        # on JudgeMeta.
+        self._model = self._wrap_model(model)
+        if isinstance(model, str):
+            self._judge_label = model
+        elif model is not None and hasattr(model, "get_model_name"):
+            self._judge_label = model.get_model_name()  # DeepEvalBaseLLM API
+        else:
+            self._judge_label = "deepeval"
+        self._threshold = threshold
+        self._metrics = self._build_metrics(metrics)  # list[(slug, metric)]
+
+    def _wrap_model(self, model: str | Any | None) -> Any:
+        """Wrap a string model id in DeepEval's `LiteLLMModel` (lazy import).
+
+        A bare string handed to a DeepEval metric resolves to DeepEval's OpenAI
+        default regardless of any provider prefix, so we wrap it explicitly:
+        `LiteLLMModel` is a `DeepEvalBaseLLM`, which `initialize_model` passes
+        through untouched, and LiteLLM routes by the model's provider prefix
+        using the provider's env key (same as ArchForge's own LiteLLMClient).
+        Non-string models (already-built DeepEval models) pass through.
+        """
+        if model is None or not isinstance(model, str):
+            return model
+        try:
+            from deepeval.models import LiteLLMModel
+
+            return LiteLLMModel(model=model)
+        except LLMError:
+            raise
+        except Exception as exc:  # noqa: BLE001 — deepeval/litellm construction errors
+            raise LLMError(
+                f"could not build the DeepEval judge model {model!r}: {exc}"
+            ) from exc
+
+    # --------------------------------------------------------------- protocol
+    def score(self, trace: m.Trace, task: Task, rubric_id: str) -> m.RunScore:
+        test_case = self._llm_test_case(trace, task)
+        rubric_scores: dict[str, float] = {}
+        for slug, metric in self._metrics:
+            # `measure()` is SYNCHRONOUS in deepeval (the async entry point is
+            # `a_measure`), matching the sync SuiteRunner path directly — do NOT
+            # wrap it in asyncio.run (that raises TypeError on a non-coroutine).
+            # A metric LLM outage raises deepeval's own error; convert to
+            # LLMError so the SuiteRunner's bounded retry treats it exactly like
+            # a native Judge failure (E9), never a fabricated 0.0.
+            try:
+                metric.measure(test_case)
+            except Exception as exc:  # noqa: BLE001 — deepeval's error type
+                raise LLMError(f"DeepEval metric {slug!r} failed: {exc}") from exc
+            rubric_scores[slug] = float(metric.score or 0.0)
+
+        aggregate = _mean(rubric_scores.values()) if rubric_scores else 0.0
+        return m.RunScore(
+            run_id=trace.run_id,
+            spec_id=trace.spec_id,
+            task_id=trace.task_id,
+            rubric_scores=rubric_scores,
+            aggregate=aggregate,
+            confidence=1.0,
+            judge_meta=m.JudgeMeta(model=self._judge_label, rubric_id=rubric_id),
+            step_scores=[],  # top-level only (DeepEval is run-level, not per-step)
+        )
+
+    def score_suite(
+        self, scores: Sequence[m.RunScore], *, suite_id: str, rubric_id: str
+    ) -> SuiteAggregate:
+        # Reuse the pure shared aggregation (same as the native Judge) so the two
+        # backends produce structurally identical, comparable SuiteAggregates.
+        return aggregate_scores(scores, suite_id=suite_id, rubric_id=rubric_id)
+
+    # --------------------------------------------------------------- internals
+    def _build_metrics(
+        self, metrics: Sequence[str | Any] | None
+    ) -> list[tuple[str, Any]]:
+        built: list[tuple[str, Any]] = []
+        for item in (metrics or ["answer_relevancy"]):
+            if isinstance(item, str):
+                slug = _metric_slug(item)
+                factory = _METRIC_FACTORIES.get(slug)
+                if factory is None:
+                    raise LLMError(
+                        f"unknown DeepEval metric {item!r}; expected one of "
+                        f"{sorted(_METRIC_FACTORIES)}"
+                    )
+                # A metric validates its judge model at CONSTRUCTION (deepeval
+                # raises its own DeepEvalError if no API key is configured). Catch
+                # ANY construction failure and re-raise as LLMError so the CLI /
+                # SuiteRunner see the same error type as the native Judge (E9).
+                try:
+                    metric = factory(model=self._model, threshold=self._threshold)
+                except Exception as exc:  # noqa: BLE001 — deepeval's DeepEvalError
+                    raise LLMError(
+                        f"could not build DeepEval metric {item!r}: {exc}"
+                    ) from exc
+                built.append((slug, metric))
+            else:
+                built.append((_metric_slug(type(item).__name__), item))
+        return built
+
+    def _llm_test_case(self, trace: m.Trace, task: Task):
+        """Project an ArchForge trace into a DeepEval `LLMTestCase`.
+
+        `actual_output` is the run's final answer (the traced final_output, else
+        the last step's response). Grounding comes from each step's prompt_in
+        and is set on BOTH `context` and `retrieval_context`: Faithfulness
+        REQUIRES `retrieval_context` (its `_required_params`) and errors without
+        it, while RAG-style metrics read `context`. `expected_output` is carried
+        through when the task JSON declares one (Task allows extra fields).
+        """
+        actual_output = trace.final_output or (
+            trace.steps[-1].response_out if trace.steps else ""
+        )
+        expected_output = getattr(task, "expected_output", None)
+        grounding = [s.prompt_in for s in trace.steps]
+        kwargs: dict[str, Any] = {
+            "input": task.input,
+            "actual_output": actual_output,
+            "context": grounding,
+            "retrieval_context": grounding,  # Faithfulness requires this param
+        }
+        if expected_output:
+            kwargs["expected_output"] = expected_output
+        return self._deepeval.test_case.LLMTestCase(**kwargs)
+
+
+def _import_deepeval():
+    """Lazily import the `deepeval` package, or raise a clear LLMError."""
+    try:
+        import deepeval
+    except ImportError as exc:  # pragma: no cover - exercised via monkeypatch
+        raise LLMError(
+            "DeepEval is not installed; install it with "
+            "`pip install archforge-optimizer[deepeval]` to use --evaluator deepeval"
+        ) from exc
+    return deepeval
+
+
+def _mean(values: Sequence[float]) -> float:
+    return sum(values) / len(values) if values else 0.0
+
+
+# Register the metrics ArchForge exposes by name. Factories construct the metric
+# objects lazily (deepeval classes are imported inside each factory, not here, so
+# this module stays importable before the extra is installed).
+def _answer_relevancy_factory(*, model, threshold):
+    from deepeval.metrics import AnswerRelevancyMetric
+
+    return AnswerRelevancyMetric(threshold=threshold, model=model)
+
+
+def _faithfulness_factory(*, model, threshold):
+    from deepeval.metrics import FaithfulnessMetric
+
+    return FaithfulnessMetric(threshold=threshold, model=model)
+
+
+_METRIC_FACTORIES.update(
+    {
+        "answer_relevancy": _answer_relevancy_factory,
+        "faithfulness": _faithfulness_factory,
+    }
+)
+
+
+__all__ = ["DeepEvalEvaluator"]

{archforge_optimizer-0.3.0 → archforge_optimizer-0.4.0}/archforge/llm/litellm.py

@@ -48,6 +48,22 @@ _PROVIDER_PREFIX: dict[str, str] = {
 }
 
 
+def prefix_model(provider: str, model: str) -> str:
+    """Prefix a model id with the provider (``openai/gpt-4o``).
+
+    A model that ALREADY carries THIS provider's prefix (``openai/gpt-4o``
+    under provider=openai) is passed through untouched, so fully-qualified
+    overrides still work. But a ``/`` alone does NOT mean "already routed" —
+    e.g. Groq's model ``openai/gpt-oss-120b`` has a slash in its native id,
+    so it must still get the provider prefix (``groq/openai/gpt-oss-120b``)
+    or LiteLLM would route it to OpenAI. Prefix unless it already matches.
+    """
+    prefix = _PROVIDER_PREFIX.get(provider, provider)
+    if model.startswith(f"{prefix}/"):
+        return model
+    return f"{prefix}/{model}"
+
+
 class LiteLLMClient:
     """An `LLMClient` backed by `litellm.completion` for one provider.
 
@@ -65,19 +81,8 @@ class LiteLLMClient:
         self._base_url = base_url
 
     def _prefixed(self, model: str) -> str:
-        """Prefix a model id with the provider (``openai/gpt-4o``).
-
-        A model that ALREADY carries THIS provider's prefix (``openai/gpt-4o``
-        under provider=openai) is passed through untouched, so fully-qualified
-        overrides still work. But a ``/`` alone does NOT mean "already routed" —
-        e.g. Groq's model ``openai/gpt-oss-120b`` has a slash in its native id,
-        so it must still get the provider prefix (``groq/openai/gpt-oss-120b``)
-        or LiteLLM would route it to OpenAI. Prefix unless it already matches.
-        """
-        prefix = _PROVIDER_PREFIX[self._provider]
-        if model.startswith(f"{prefix}/"):
-            return model
-        return f"{prefix}/{model}"
+        """Prefix a model id with this client's provider (see `prefix_model`)."""
+        return prefix_model(self._provider, model)
 
     def _default_model(self) -> str:
         """The provider's default model id for a bare complete() call, resolved
@@ -138,4 +143,4 @@ class LiteLLMClient:
         )
 
 
-__all__ = ["LiteLLMClient"]
+__all__ = ["LiteLLMClient", "prefix_model"]

{archforge_optimizer-0.3.0 → archforge_optimizer-0.4.0}/archforge/userconfig.py

@@ -39,8 +39,7 @@ _DISCOVERY_DIR: Path = Path(".archforge")
 _DISCOVERY_FILE: Path = _DISCOVERY_DIR / "archforge.py"
 
 _MAIN_MSG = (
-    "ArchForge is not initialized. Run `archforge-optimizer init` to create "
-    ".archforge/archforge.py (the project's tunable config), then the CLI will work."
+    "ArchForge is not initialized. Run `archforge-optimizer init`."
 )
 
 

{archforge_optimizer-0.3.0 → archforge_optimizer-0.4.0}/pyproject.toml

@@ -39,6 +39,8 @@ providers = [
     "google-genai>=1.0",
 ]
 
+deepeval = ["deepeval"]
+
 [build-system]
 requires = ["hatchling"]
 build-backend = "hatchling.build"

archforge_optimizer-0.3.0/archforge/judge/__init__.py

@@ -1,20 +0,0 @@
-"""The Judge — LLM-as-judge scoring of runs (spec §3, §4).
-
-`Judge.score(trace, task, rubric_id) -> RunScore` turns a run's trace into a
-scored verdict: an aggregate + named rubric dimensions + confidence, plus a
-*per-step breakdown* (StepScore[]) that names which agent lost which points. The
-per-step breakdown is the raw material the Architect uses to credit-assign a
-fault to a node/route.
-
-`score_suite(...)` aggregates over R repeats: mean (down-weighted by
-confidence) and a stable aggregate used by the Gatekeeper. `rubric_id` is
-stamped on every score so cross-rubric comparisons never masquerade as
-improvement (invariant I5, spec E2).
-"""
-
-from __future__ import annotations
-
-from archforge.judge.base import Judge, SuiteAggregate, default_rubric
-from archforge.judge.scripted import ScriptedJudge
-
-__all__ = ["Judge", "ScriptedJudge", "SuiteAggregate", "default_rubric"]