vibesys-evaluator 0.3.0rc2__tar.gz

vibesys_evaluator-0.3.0rc2/PKG-INFO

@@ -0,0 +1,39 @@
+Metadata-Version: 2.4
+Name: vibesys-evaluator
+Version: 0.3.0rc2
+Summary: Build VibeSys benchmark evaluators for services: workloads, phases, validation
+License-Expression: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: protobuf-py==0.6.0
+Requires-Dist: protovalidate==2.0.0
+Requires-Dist: pyyaml==6.0.3
+
+# vibesys-evaluator
+
+Build VibeSys benchmark evaluators for services: workloads, phases and validation.
+
+```
+pip install vibesys-evaluator
+```
+
+The Python package is a thin wrapper over a Go core. The wheels for Linux and macOS (x86_64 and
+arm64) bundle the Go binaries (`vibesys-evaluator`, `loadgen` and `otelcapture`), so one
+`pip install` is all a task needs: no Go toolchain, no `go install`. The Linux wheels are static
+and carry both the manylinux and the musllinux tags.
+
+The package looks for each binary in the wheel first, then in an environment variable
+(`VIBESYS_EVALUATOR_BIN`, `VIBESYS_EVALUATOR_LOADGEN`, `VIBESYS_EVALUATOR_OTELCAPTURE`), then on
+`PATH`. A source distribution, or a platform without a wheel, has no bundled binary: build the
+binaries with Go and name them in those variables.
+
+```
+go install github.com/uw-syfi/vibesys/sdk/vibesys-evaluator/go/cmd/vibesys-evaluator@<tag>
+go install github.com/uw-syfi/vibesys/sdk/vibesys-evaluator/go/cmd/loadgen@<tag>
+export VIBESYS_EVALUATOR_BIN="$(command -v vibesys-evaluator)"
+```
+
+Go authors import the module `github.com/uw-syfi/vibesys/sdk/vibesys-evaluator/go` (tags
+`sdk/vibesys-evaluator/go/v<semver>`, cut with each release) and need no wheel.
+
+The design and the public API are in the repository: `sdk/vibesys-evaluator/docs/spec.md`.

vibesys_evaluator-0.3.0rc2/README.md

@@ -0,0 +1,28 @@
+# vibesys-evaluator
+
+Build VibeSys benchmark evaluators for services: workloads, phases and validation.
+
+```
+pip install vibesys-evaluator
+```
+
+The Python package is a thin wrapper over a Go core. The wheels for Linux and macOS (x86_64 and
+arm64) bundle the Go binaries (`vibesys-evaluator`, `loadgen` and `otelcapture`), so one
+`pip install` is all a task needs: no Go toolchain, no `go install`. The Linux wheels are static
+and carry both the manylinux and the musllinux tags.
+
+The package looks for each binary in the wheel first, then in an environment variable
+(`VIBESYS_EVALUATOR_BIN`, `VIBESYS_EVALUATOR_LOADGEN`, `VIBESYS_EVALUATOR_OTELCAPTURE`), then on
+`PATH`. A source distribution, or a platform without a wheel, has no bundled binary: build the
+binaries with Go and name them in those variables.
+
+```
+go install github.com/uw-syfi/vibesys/sdk/vibesys-evaluator/go/cmd/vibesys-evaluator@<tag>
+go install github.com/uw-syfi/vibesys/sdk/vibesys-evaluator/go/cmd/loadgen@<tag>
+export VIBESYS_EVALUATOR_BIN="$(command -v vibesys-evaluator)"
+```
+
+Go authors import the module `github.com/uw-syfi/vibesys/sdk/vibesys-evaluator/go` (tags
+`sdk/vibesys-evaluator/go/v<semver>`, cut with each release) and need no wheel.
+
+The design and the public API are in the repository: `sdk/vibesys-evaluator/docs/spec.md`.

vibesys_evaluator-0.3.0rc2/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=69"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "vibesys-evaluator"
+version = "0.3.0rc2"
+description = "Build VibeSys benchmark evaluators for services: workloads, phases, validation"
+requires-python = ">=3.12"
+readme = "README.md"
+license = "MIT"
+# The schema's generated Python types (protoc-gen-py) and their validation, and a YAML
+# parser for the tasks' Kubernetes manifests (microservices.k8s). Nothing else: the package runs in
+# every task image, and the task's own code brings its own clients.
+dependencies = [
+    "protobuf-py==0.6.0",
+    "protovalidate==2.0.0",
+    "pyyaml==6.0.3",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["vibesys_evaluator*"]
+
+[tool.setuptools.package-data]
+# _bin/ holds the Go binaries of a platform wheel (packaging/evaluator_wheels.py); a development
+# tree has none.
+vibesys_evaluator = ["py.typed", "_bin/*"]

vibesys_evaluator-0.3.0rc2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

vibesys_evaluator-0.3.0rc2/src/vibesys_evaluator/__init__.py

@@ -0,0 +1,50 @@
+"""vibesys-evaluator: the sample lifecycle every VibeSys task evaluator shares.
+
+The core knows no domain. A task's ``bench.py`` calls ``main`` with its
+metrics and its ``check``, ``benchmark`` and ``profile`` functions; each gets a
+``Run`` (its phases and its report). Domains add what their tasks need on top:
+``vibesys_evaluator.microservices`` (workloads over HTTP, load phases, deploys)
+and ``vibesys_evaluator.llm_serving`` (the candidate server, AgentX replay,
+golden accuracy checks). See the package's ``docs/spec.md`` and ``examples/``.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from typing import TYPE_CHECKING, NoReturn
+
+from vibesys_evaluator._private import cli
+from vibesys_evaluator._private.rand import Rand
+from vibesys_evaluator._private.results import Metric, Validity
+from vibesys_evaluator._private.run import Run
+
+if TYPE_CHECKING:
+    from collections.abc import Callable, Mapping
+
+__all__ = ["Metric", "Rand", "Run", "Validity", "main"]
+
+
+def main(
+    *,
+    metrics: Mapping[str, Metric],
+    benchmark: Callable[[Run], None],
+    check: Callable[[Run], str | None] | None = None,
+    profile: Callable[[Run], None] | None = None,
+) -> NoReturn:
+    """Run the task script's command (``check``, ``benchmark`` or ``profile``) and exit.
+
+    ``benchmark --vs-output FILE`` writes result-protocol-2 records: ``hello``
+    with ``metrics``, then the result ``benchmark`` reports with
+    ``run.report`` or an error naming what failed. ``check`` exits 0 when the
+    function returns None and 1 with the reason it returns. ``profile`` runs
+    the load a profiler records; it is never scored.
+
+    Args:
+        metrics: The metrics ``run.report`` reports, by name.
+        benchmark: The scoring definition: runs phases and calls ``run.report``.
+        check: The correctness check: None when the system is correct, else what is wrong.
+        profile: The standard load to profile the system under.
+    """
+    task = cli.Task(metrics=metrics, benchmark=benchmark, check=check, profile=profile)
+    sys.exit(cli.run(task, sys.argv[1:], env=os.environ))

vibesys_evaluator-0.3.0rc2/src/vibesys_evaluator/_private/__init__.py

@@ -0,0 +1 @@
+"""Implementation of ``vibesys_evaluator``; nothing here is public API."""

vibesys_evaluator-0.3.0rc2/src/vibesys_evaluator/_private/cli.py

@@ -0,0 +1,98 @@
+"""The task script's command line when the task brings no system of its own (``main``).
+
+    bench.py check                              exit 0 when the check passes
+    bench.py benchmark --vs-output FILE [--seed N]   one sample, result-protocol-2 records
+    bench.py profile [--seed N]                 the standard profile load, never scored
+
+The system under test is already running (vibesys-cli starts the candidate);
+domains reach it through their own helpers. A mode the task did not define
+exits 2 naming it.
+"""
+
+from __future__ import annotations
+
+import argparse
+from dataclasses import dataclass
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from vibesys_evaluator._private.run import new_run
+from vibesys_evaluator._private.sample import (
+    Mode,
+    benchmark_sample,
+    log,
+    unscored_sample,
+    with_signals,
+)
+
+if TYPE_CHECKING:
+    from collections.abc import Callable, Mapping, Sequence
+
+    from vibesys_evaluator._private.results import Metric
+    from vibesys_evaluator._private.run import Run
+    from vibesys_evaluator._private.sample import Sample
+
+#: Exit status of a usage error (argparse's), also for a mode the task does not define.
+USAGE = 2
+
+
+@dataclass(frozen=True, kw_only=True)
+class Task:
+    """Everything the task script passes to ``main``."""
+
+    metrics: Mapping[str, Metric]
+    benchmark: Callable[[Run], None]
+    check: Callable[[Run], str | None] | None
+    profile: Callable[[Run], None] | None
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog="bench.py", description="A vibesys-evaluator task.")
+    commands = parser.add_subparsers(dest="command", required=True)
+    commands.add_parser(Mode.CHECK, help="check the system's correctness (exit 0 = pass)")
+    bench = commands.add_parser(Mode.BENCHMARK, help="measure one sample")
+    bench.add_argument("--vs-output", type=Path, required=True)
+    bench.add_argument("--seed", type=int, default=None, help="replay the sample with this seed")
+    profile = commands.add_parser(Mode.PROFILE, help="run the profile load (never scored)")
+    profile.add_argument("--seed", type=int, default=None)
+    return parser
+
+
+def run(task: Task, argv: Sequence[str], *, env: Mapping[str, str]) -> int:
+    """Run one command of the task script; returns the exit status."""
+    args = _parser().parse_args(list(argv))
+    mode = Mode(args.command)
+    if mode is Mode.BENCHMARK:
+        return with_signals(
+            lambda: benchmark_sample(
+                metrics=task.metrics,
+                output=args.vs_output,
+                seed_flag=args.seed,
+                env=env,
+                body=lambda sample: task.benchmark(new_run(sample)),
+            )
+        )
+    seed = getattr(args, "seed", None)
+    body = _unscored_body(task, mode)
+    if body is None:
+        log(f"this task defines no {mode}: pass {mode}= to main(...)")
+        return USAGE
+    return with_signals(lambda: unscored_sample(mode, seed_flag=seed, env=env, body=body))
+
+
+def _unscored_body(task: Task, mode: Mode) -> Callable[[Sample], str | None] | None:
+    """What a check or profile sample runs, or None when the task does not define the mode."""
+    check, profile = task.check, task.profile
+    if mode is Mode.CHECK and check is not None:
+
+        def checked(sample: Sample) -> str | None:
+            return check(new_run(sample))
+
+        return checked
+    if mode is Mode.PROFILE and profile is not None:
+
+        def profiled(sample: Sample) -> None:
+            profile(new_run(sample))
+
+        return profiled
+    return None

vibesys_evaluator-0.3.0rc2/src/vibesys_evaluator/_private/core.py

@@ -0,0 +1,240 @@
+"""The Go core's command line (``docs/spec.md``, "Wrapper calling convention") behind one helper.
+
+``call(group, verb, request)`` runs ``vibesys-evaluator <group> <verb>``, writes the request as
+one JSON message on its stdin and decodes the one JSON message it writes to stdout;
+``stream(group, verb, request, on_event)`` does the same for a long verb, which writes
+``{"event": ...}`` lines before its response. Its stderr
+(progress and logs) passes through to the user. Exit 0 is the response; exit 1 is a domain error
+(``CoreDomainError``, from the response's ``error_kind`` and ``message``); any other exit, a
+reply that is not JSON, or a missing binary is a ``CoreError`` of the core itself.
+
+The binary is found in the wheel (``vibesys_evaluator/_bin``, where the platform wheels bundle the
+Go binaries), then ``$VIBESYS_EVALUATOR_BIN``, then ``PATH``. ``loadgen`` and ``otelcapture`` are
+found the same way (``find_binary(name)``), with ``$VIBESYS_EVALUATOR_LOADGEN`` and
+``$VIBESYS_EVALUATOR_OTELCAPTURE``.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import json
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+    from collections.abc import Callable, Mapping
+
+#: Names the core binary; consulted after the wheel's bundled one.
+BIN_ENV = "VIBESYS_EVALUATOR_BIN"
+BIN_NAME = "vibesys-evaluator"
+#: The environment variable that names each of the wheel's binaries.
+ENV_VARS = {
+    BIN_NAME: BIN_ENV,
+    "loadgen": "VIBESYS_EVALUATOR_LOADGEN",
+    "otelcapture": "VIBESYS_EVALUATOR_OTELCAPTURE",
+}
+_BUNDLED_DIR = Path(__file__).resolve().parent.parent / "_bin"
+_DOMAIN_EXIT = 1
+
+
+class CoreError(Exception):
+    """The core itself failed: no binary, a crash, or a reply that breaks the convention."""
+
+
+class CoreNotFoundError(CoreError):
+    """No ``vibesys-evaluator`` binary was found."""
+
+
+class CoreCrashError(CoreError):
+    """The core exited with a code other than 0 and 1, or wrote a reply that is not JSON."""
+
+    def __init__(self, message: str, *, code: int | None = None) -> None:
+        """Keep the exit code (None when the reply, not the exit, broke the convention)."""
+        super().__init__(message)
+        self.code = code
+
+
+class CoreDomainError(CoreError):
+    """The operation failed on its own terms (exit 1): ``kind`` says how, ``response`` is the reply."""
+
+    def __init__(self, kind: str, message: str, response: dict[str, Any]) -> None:
+        """Keep the ``error_kind``, the message and the whole reply."""
+        super().__init__(f"{kind}: {message}" if kind else message)
+        self.kind = kind
+        self.message = message
+        self.response = response
+
+
+def find_binary(name: str = BIN_NAME) -> str:
+    """The binary ``name``: the wheel's own, then its environment variable, then ``PATH``.
+
+    ``name`` is one of the wheel's binaries (``vibesys-evaluator``, ``loadgen``, ``otelcapture``);
+    each has its own variable (``ENV_VARS``).
+    """
+    suffix = ".exe" if sys.platform == "win32" else ""
+    bundled = _BUNDLED_DIR / (name + suffix)
+    if bundled.is_file():
+        return str(bundled)
+    env = ENV_VARS[name]
+    named = os.environ.get(env)
+    if named:
+        return named
+    found = shutil.which(name)
+    if found:
+        return found
+    msg = (
+        f"the {name} binary is not bundled in this installation of vibesys-evaluator, ${env} is "
+        f"unset and it is not on PATH. The platform wheels bundle it (pip install vibesys-evaluator "
+        f"on Linux or macOS, x86_64 or arm64); a source distribution or another platform does not. "
+        f"Set ${env} to the binary, or build it: "
+        f"`go install github.com/uw-syfi/vibesys/sdk/vibesys-evaluator/go/cmd/{name}@<tag>`"
+    )
+    raise CoreNotFoundError(msg)
+
+
+def call(
+    group: str,
+    verb: str,
+    request: Mapping[str, Any] | None = None,
+    *,
+    timeout_secs: float | None = None,
+    terminate_grace_secs: float = 0.0,
+) -> dict[str, Any]:
+    """Run ``vibesys-evaluator GROUP VERB`` with ``request`` and return its decoded response.
+
+    When the wait is cut short (a signal handler raising, ``KeyboardInterrupt``), the core is killed
+    at once; with ``terminate_grace_secs`` it is sent SIGTERM first and given that long to clean up
+    (a verb that owns cluster state deletes it on SIGTERM), then killed.
+    """
+    binary = find_binary()
+    body = json.dumps({} if request is None else request)
+    try:
+        process = subprocess.Popen(
+            [binary, group, verb],
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            text=True,
+        )
+    except FileNotFoundError as error:
+        msg = f"cannot run {BIN_NAME} at {binary!r}: {error.strerror}"
+        raise CoreNotFoundError(msg) from error
+    try:
+        stdout, _ = process.communicate(body, timeout=timeout_secs)
+    except subprocess.TimeoutExpired as error:
+        _stop(process, 0.0)
+        msg = f"{BIN_NAME} {group} {verb} ran past {timeout_secs}s"
+        raise CoreCrashError(msg) from error
+    except BaseException:
+        _stop(process, terminate_grace_secs)
+        raise
+    return _decode(group, verb, process.returncode, stdout)
+
+
+def _stop(process: subprocess.Popen[str], grace_secs: float) -> None:
+    """End the child: SIGTERM and a grace period when asked, then SIGKILL."""
+    if grace_secs > 0 and process.poll() is None:
+        process.terminate()
+        with contextlib.suppress(subprocess.TimeoutExpired):
+            process.wait(timeout=grace_secs)
+    if process.poll() is None:
+        process.kill()
+    process.wait()
+
+
+def _decode(group: str, verb: str, code: int, stdout: str) -> dict[str, Any]:
+    if code not in {0, _DOMAIN_EXIT}:
+        msg = f"{BIN_NAME} {group} {verb} exited {code}"
+        raise CoreCrashError(msg, code=code)
+    try:
+        response = json.loads(stdout)
+    except ValueError as error:
+        msg = f"{BIN_NAME} {group} {verb} wrote a reply that is not JSON: {stdout[:200]!r}"
+        raise CoreCrashError(msg, code=code) from error
+    if not isinstance(response, dict):
+        msg = f"{BIN_NAME} {group} {verb} wrote a reply that is not a JSON object"
+        raise CoreCrashError(msg, code=code)
+    if code == _DOMAIN_EXIT:
+        kind = response.get("error_kind", response.get("errorKind", ""))
+        raise CoreDomainError(str(kind), str(response.get("message", "")), response)
+    return response
+
+
+def stream(
+    group: str,
+    verb: str,
+    request: Mapping[str, Any] | None = None,
+    *,
+    on_event: Callable[[dict[str, Any]], None],
+) -> dict[str, Any]:
+    """Run a long verb: each ``{"event": E}`` line it writes goes to ``on_event(E)`` as it comes.
+
+    The last line is the response, decoded as ``call`` decodes it. The core is stopped (SIGTERM)
+    when this returns early: an exception of ``on_event``, or a signal that raised here.
+    """
+    binary = find_binary()
+    try:
+        process = subprocess.Popen(
+            [binary, group, verb], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True
+        )
+    except FileNotFoundError as error:
+        msg = f"cannot run {BIN_NAME} at {binary!r}: {error.strerror}"
+        raise CoreNotFoundError(msg) from error
+    try:
+        last = _events(process, f"{BIN_NAME} {group} {verb}", request, on_event)
+        code = process.wait()
+    finally:
+        if process.poll() is None:
+            process.terminate()
+            process.wait()
+    return _reply(f"{BIN_NAME} {group} {verb}", code, last)
+
+
+def _events(
+    process: subprocess.Popen[str],
+    what: str,
+    request: Mapping[str, Any] | None,
+    on_event: Callable[[dict[str, Any]], None],
+) -> str:
+    """Write the request, pass each event on, and return the last line that is not an event."""
+    stdin, stdout = process.stdin, process.stdout
+    if stdin is None or stdout is None:
+        msg = f"{what}: no pipes to the core"
+        raise CoreCrashError(msg)
+    stdin.write(json.dumps({} if request is None else request))
+    stdin.close()
+    last = ""
+    for line in stdout:
+        text = line.strip()
+        if not text.startswith('{"event":'):
+            last = text or last
+            continue
+        try:
+            event = json.loads(text)["event"]
+        except (ValueError, KeyError, TypeError) as error:
+            msg = f"{what} wrote an event that is not JSON: {text[:200]!r}"
+            raise CoreCrashError(msg) from error
+        on_event(event)
+    return last
+
+
+def _reply(what: str, code: int, last: str) -> dict[str, Any]:
+    """The response line of a streamed verb, or the error it carries."""
+    if code not in {0, _DOMAIN_EXIT}:
+        msg = f"{what} exited {code}"
+        raise CoreCrashError(msg, code=code)
+    try:
+        response = json.loads(last)
+    except ValueError as error:
+        msg = f"{what} wrote a reply that is not JSON: {last[:200]!r}"
+        raise CoreCrashError(msg, code=code) from error
+    if not isinstance(response, dict):
+        msg = f"{what} wrote a reply that is not a JSON object"
+        raise CoreCrashError(msg, code=code)
+    if code == _DOMAIN_EXIT:
+        kind = response.get("error_kind", "")
+        raise CoreDomainError(str(kind), str(response.get("message", "")), response)
+    return response