rtpbench 0.3.0__tar.gz → 0.4.0__tar.gz

{rtpbench-0.3.0 → rtpbench-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rtpbench
-Version: 0.3.0
+Version: 0.4.0
 Summary: Wire-level latency and conformance tests for voice agents over SIP/RTP
 Author-email: Juanjo Aroztegi <juanjo@aroztegi.com>
 License-Expression: Apache-2.0
@@ -33,11 +33,12 @@ without a carrier in front, on a mock engine that holds the AI engine's share of
 | | [LiveKit](https://github.com/livekit/sip) | [AI Voice Connector CE](https://github.com/OpenSIPS/opensips-ai-voice-connector-ce) at b95f5af | Asterisk + [AVA](https://github.com/hkjarral/AVA-AI-Voice-Agent-for-Asterisk) |
 | --- | --- | --- | --- |
 | cases failed of 24 | 5, and 2 unsupported | 5, 3 unsupported and 1 inconclusive | 4, and 3 inconclusive |
-| turn latency, clean, p50 | 1833 ms | 1511 ms | 1588 ms |
+| turn latency, clean, p50 | 1833 ms | 1511 ms | 1589 ms |
 | its own media path per turn, p50 | 335 ms | 9 ms | 88 ms |
 | stops for an interruption, clean, p50 | 306 ms | 24 ms | never |
 
-The counts take every profile, direct and through the carrier. Run direct only, as
+The counts take every profile, direct and through the carrier, and AVA as it ships, over
+AudioSocket. Run direct only, as
 `target NAME --mock` does by default, LiveKit fails 4: `late-offer` fails only through the
 carrier. All three pass the 3 cases for lost SIP messages, run direct on a clean network.
 Unsupported is an optional feature declined as its RFC allows, with the call unharmed:
@@ -47,7 +48,8 @@ the digit never reaches the engine: a failure.
 
 - LiveKit keeps sending audio to a held caller in every kind of hold, AI Voice Connector CE when
   held with `c=0.0.0.0`, and both answer a re-INVITE that changes codec with the old codec. AVA
-  sends RTP only while it speaks, so whether its media resumes after a hold cannot be judged.
+  sends RTP only while it speaks, so whether its media resumes after a hold cannot be judged,
+  except over Asterisk's WebSocket channel, where Asterisk keeps sending to a held caller.
 - LiveKit's extra latency is its own media path: 326 ms per turn more than AI Voice Connector
   CE's, against 322 ms between their turn latencies.
 - LiveKit's RTP falls 20 ms behind its own timestamps 4 s into every call, so a receiver needs a
@@ -55,10 +57,12 @@ the digit never reaches the engine: a failure.
 - AI Voice Connector CE delivers the engine's speech sample for sample. LiveKit re-encodes it on
   the way, through its WebRTC room, and on a clean network stretches or cuts a pause by 20-40 ms
   in 6% of answers (10% in a second wave).
-- AVA, as its golden configuration ships, cuts the last 0.8 s of every answer, and it never
-  stops for an interruption: it keeps the caller's audio from the engine while it speaks. Over
-  AudioSocket under packet loss, the engine sometimes got silence instead of the caller, so 5 more
-  cases fail now and then.
+- AVA, as its golden configuration ships, cuts the last 0.6 s of speech from every answer, over
+  each of its three transports, and over Asterisk's WebSocket channel its pacer also breaks every
+  answer with 7-10 short silences. It never stops for an interruption: it keeps the caller's
+  audio from the engine while it speaks. Over AudioSocket under packet loss, the engine sometimes
+  got silence instead of the caller (a squelch AVA has fixed since), so 5 more cases flip now
+  and then.
 - AI Voice Connector CE's main branch fixed 4 of the 10 cases it failed in the baseline (a failed
   transfer no longer hangs up on the caller; compact headers, a bare `a=ssrc` and SIP over TCP
   are taken), and now declines a DTMF INFO with 501 where it left it unanswered. It still
@@ -101,7 +105,7 @@ defaults and valid names.
 
 - [Calibration](docs/calibration.md): error bars, clock steps, network profiles and their sources
 - [Conformance cases](docs/cases.md): what each case checks
-- [Targets, your own agent, the mock engine and the carrier](docs/targets.md)
+- [Targets (and adding one), your own agent, the mock engine and the carrier](docs/targets.md)
 - [CI](docs/ci.md): pass or fail your agent's run against a policy
 
 ## Development

{rtpbench-0.3.0 → rtpbench-0.4.0}/README.md

@@ -18,11 +18,12 @@ without a carrier in front, on a mock engine that holds the AI engine's share of
 | | [LiveKit](https://github.com/livekit/sip) | [AI Voice Connector CE](https://github.com/OpenSIPS/opensips-ai-voice-connector-ce) at b95f5af | Asterisk + [AVA](https://github.com/hkjarral/AVA-AI-Voice-Agent-for-Asterisk) |
 | --- | --- | --- | --- |
 | cases failed of 24 | 5, and 2 unsupported | 5, 3 unsupported and 1 inconclusive | 4, and 3 inconclusive |
-| turn latency, clean, p50 | 1833 ms | 1511 ms | 1588 ms |
+| turn latency, clean, p50 | 1833 ms | 1511 ms | 1589 ms |
 | its own media path per turn, p50 | 335 ms | 9 ms | 88 ms |
 | stops for an interruption, clean, p50 | 306 ms | 24 ms | never |
 
-The counts take every profile, direct and through the carrier. Run direct only, as
+The counts take every profile, direct and through the carrier, and AVA as it ships, over
+AudioSocket. Run direct only, as
 `target NAME --mock` does by default, LiveKit fails 4: `late-offer` fails only through the
 carrier. All three pass the 3 cases for lost SIP messages, run direct on a clean network.
 Unsupported is an optional feature declined as its RFC allows, with the call unharmed:
@@ -32,7 +33,8 @@ the digit never reaches the engine: a failure.
 
 - LiveKit keeps sending audio to a held caller in every kind of hold, AI Voice Connector CE when
   held with `c=0.0.0.0`, and both answer a re-INVITE that changes codec with the old codec. AVA
-  sends RTP only while it speaks, so whether its media resumes after a hold cannot be judged.
+  sends RTP only while it speaks, so whether its media resumes after a hold cannot be judged,
+  except over Asterisk's WebSocket channel, where Asterisk keeps sending to a held caller.
 - LiveKit's extra latency is its own media path: 326 ms per turn more than AI Voice Connector
   CE's, against 322 ms between their turn latencies.
 - LiveKit's RTP falls 20 ms behind its own timestamps 4 s into every call, so a receiver needs a
@@ -40,10 +42,12 @@ the digit never reaches the engine: a failure.
 - AI Voice Connector CE delivers the engine's speech sample for sample. LiveKit re-encodes it on
   the way, through its WebRTC room, and on a clean network stretches or cuts a pause by 20-40 ms
   in 6% of answers (10% in a second wave).
-- AVA, as its golden configuration ships, cuts the last 0.8 s of every answer, and it never
-  stops for an interruption: it keeps the caller's audio from the engine while it speaks. Over
-  AudioSocket under packet loss, the engine sometimes got silence instead of the caller, so 5 more
-  cases fail now and then.
+- AVA, as its golden configuration ships, cuts the last 0.6 s of speech from every answer, over
+  each of its three transports, and over Asterisk's WebSocket channel its pacer also breaks every
+  answer with 7-10 short silences. It never stops for an interruption: it keeps the caller's
+  audio from the engine while it speaks. Over AudioSocket under packet loss, the engine sometimes
+  got silence instead of the caller (a squelch AVA has fixed since), so 5 more cases flip now
+  and then.
 - AI Voice Connector CE's main branch fixed 4 of the 10 cases it failed in the baseline (a failed
   transfer no longer hangs up on the caller; compact headers, a bare `a=ssrc` and SIP over TCP
   are taken), and now declines a DTMF INFO with 501 where it left it unanswered. It still
@@ -86,7 +90,7 @@ defaults and valid names.
 
 - [Calibration](docs/calibration.md): error bars, clock steps, network profiles and their sources
 - [Conformance cases](docs/cases.md): what each case checks
-- [Targets, your own agent, the mock engine and the carrier](docs/targets.md)
+- [Targets (and adding one), your own agent, the mock engine and the carrier](docs/targets.md)
 - [CI](docs/ci.md): pass or fail your agent's run against a policy
 
 ## Development

{rtpbench-0.3.0 → rtpbench-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "rtpbench"
-version = "0.3.0"
+version = "0.4.0"
 description = "Wire-level latency and conformance tests for voice agents over SIP/RTP"
 readme = "README.md"
 license = "Apache-2.0"

{rtpbench-0.3.0 → rtpbench-0.4.0}/src/rtpbench/bench.py

@@ -9,6 +9,7 @@ import secrets
 import shlex
 import shutil
 import subprocess
+import tempfile
 import textwrap
 import time
 from collections.abc import Callable, Iterator, Sequence
@@ -196,6 +197,57 @@ def _image_tag() -> str:
     return "rtpbench:" + h.hexdigest()[:12]
 
 
+def _git(path: Path, *args: str) -> str:
+    res = subprocess.run(["git", "-C", str(path), *args], capture_output=True, text=True)
+    if res.returncode:
+        raise ValueError(f"not a git checkout: {res.stderr.strip()}")
+    return res.stdout
+
+
+def _aside(path: Path, out: Path | None) -> Callable[[str], bool]:
+    """Whether a path in the checkout `path`, as git gives it, is in the results directory
+    `out`: a run writes there, so it is no part of the target."""
+    try:
+        rel = out.resolve().relative_to(path.resolve()).as_posix() + "/" if out else None
+    except ValueError:  # not inside the checkout
+        rel = None
+    return lambda name: rel is not None and (name + "/").startswith(rel)
+
+
+def checkout(path: Path, out: Path | None = None) -> dict[str, Any]:
+    """The commit a git checkout is at, and whether it has changes on top (the results
+    directory `out` aside): what a target built from it (`--source`) records."""
+    commit = _git(path, "rev-parse", "HEAD").strip()
+    aside = _aside(path, out)
+    fields = iter(_git(path, "status", "--porcelain", "-z", "--untracked-files=all").split("\0"))
+    changed = []
+    for e in fields:
+        if e[3:]:
+            changed.append(e[3:])
+        if set(e[:2]) & {"R", "C"} and (origin := next(fields, "")):  # a field of its own
+            changed.append(origin)
+    return {"commit": commit, "dirty": any(not aside(name) for name in changed)}
+
+
+def snapshot(path: Path, out: Path | None, dest: Path) -> None:
+    """Copy into `dest` what a target builds from the checkout `path`: the files git tracks and
+    those it would (untracked, not ignored), as they are now; not .git, not the results
+    directory `out`, not __pycache__ (which _build_hash leaves out too)."""
+    aside = _aside(path, out)
+    names = _git(path, "ls-files", "-z", "--cached", "--others", "--exclude-standard")
+    for name in dict.fromkeys(names.split("\0")):
+        src = path / name
+        if not name or aside(name) or "__pycache__" in name.split("/"):
+            continue
+        if not (src.exists() or src.is_symlink()):  # deleted
+            continue
+        (dest / name).parent.mkdir(parents=True, exist_ok=True)
+        if src.is_dir() and not src.is_symlink():  # a submodule
+            shutil.copytree(src, dest / name, symlinks=True, ignore=shutil.ignore_patterns(".git"))
+        else:
+            shutil.copy2(src, dest / name, follow_symlinks=False)
+
+
 def _build_hash(build: dict[str, Any]) -> str:
     """A hash of a compose `build` section as `config` resolves it, with each local context's
     files (all of them: the build may copy any) in place of its path, so the same from any
@@ -223,7 +275,9 @@ class Stack:
     `network`, and with `ports` (first-last) the host publishes that range on the router. With
     `tls`, out/tls holds a test certificate for the stack's TLS servers, and ca.pem, what the
     caller trusts: `ca`, or with no `uri` the stack's test CA; else the caller trusts the
-    system's certificates. With `srtp`, the caller offers SRTP with these suites."""
+    system's certificates. With `srtp`, the caller offers SRTP with these suites. With
+    `source`, a git checkout, the target builds from a copy of its files that `down` removes
+    (targets/<name>/source.yaml)."""
 
     def __init__(
         self,
@@ -237,6 +291,7 @@ class Stack:
         tls: bool = False,
         ca: Path | None = None,
         srtp: list[str] | None = None,
+        source: Path | None = None,
     ) -> None:
         self.project = f"rtpbench-{secrets.token_hex(3)}"
         self.env = os.environ | {
@@ -259,6 +314,15 @@ class Stack:
         if target and mock:
             self.files.append(ROOT / "mock" / "compose.yaml")
             self.env |= MOCK[target]
+        self.source: Path | None = None  # the copy built from
+        self.checkout: dict[str, Any] = {}  # what it was copied from, as it was then
+        if target and source:
+            self.files.append(ROOT / "targets" / target / "source.yaml")
+            self.checkout = checkout(source, out)
+            self.source = Path(tempfile.mkdtemp(prefix="rtpbench-source-"))
+            snapshot(source, out, self.source)
+            self.env["RTPBENCH_SOURCE"] = str(self.source)
+            self.env["RTPBENCH_SOURCE_COMMIT"] = self.checkout["commit"][:7]
         if carrier:
             self.files.append(ROOT / "carrier" / "compose.yaml")
             self.env["RTPBENCH_CARRIER_NEXT"] = carrier
@@ -385,6 +449,8 @@ class Stack:
 
     def down(self) -> None:
         self.compose("down", "-t", "0", "-v", "--remove-orphans", check=False)
+        if self.source:
+            shutil.rmtree(self.source, ignore_errors=True)
         # the test certificate's key is no use to judging; the CAs stay with the results
         (Path(self.env["RTPBENCH_OUT"]) / "tls" / "server.key").unlink(missing_ok=True)
 
@@ -809,6 +875,7 @@ def target(
     ca: Path | None = None,
     srtp: list[str] | None = None,
     tls: bool = False,
+    source: Path | None = None,
     *,
     progress: Progress,
 ) -> str:
@@ -823,7 +890,8 @@ def target(
     next few to it), else with `network` its router's address there; over TLS (a sips: URI or
     proxy, or transport=tls) it trusts the certificates in `ca`, else the system's; with `srtp`
     the caller offers SRTP with these suites. A compose target takes `tls` and `srtp` with
-    `carrier` only, which takes them on the caller's leg."""
+    `carrier` only, which takes them on the caller's leg, and builds from the git checkout
+    `source` if given."""
     from rtpbench import cases as cs
     from rtpbench import realtime
     from rtpbench.caller import ADVERTISE_PORTS, parse_uri
@@ -875,7 +943,8 @@ def target(
                 stack.env["RTPBENCH_CARRIER_TRANSPORT"] = transport(name).lower()
         else:
             proxy_tls = False
-            stack = Stack(out, name, "target" if carrier else None, mock, tls=tls, srtp=srtp)
+            next_hop = "target" if carrier else None
+            stack = Stack(out, name, next_hop, mock, tls=tls, srtp=srtp, source=source)
         logs = None
         try:
             stack.up(progress.phase, profiles)
@@ -912,6 +981,8 @@ def target(
                 router_ip = stack.sh("target", "getent hosts router").split()[0]
                 stack.sh("target", f"ip route replace {caller_ip} via {router_ip}")
                 conds["engine"] = realtime.conditions() if mock else _engine(settings)
+                if source:
+                    conds["source"] = stack.checkout
                 if mock and (own := {k: v for k, v in settings.items() if k not in MOCK[name]}):
                     conds["settings"] = _engine(own)  # the target's own, such as AVA's pacer
                 uri, proxy = f"sip:{user}@{target_ip}", stack.entry("operator")

{rtpbench-0.3.0 → rtpbench-0.4.0}/src/rtpbench/cli.py

@@ -8,8 +8,8 @@ import textwrap
 from datetime import datetime
 from pathlib import Path
 
-from rtpbench import analyze, cases, pcap
-from rtpbench.bench import PROFILES, TARGETS, docker, is_uri, network_problem
+from rtpbench import ROOT, analyze, cases, pcap
+from rtpbench.bench import PROFILES, TARGETS, checkout, docker, is_uri, network_problem
 from rtpbench.caller import GAP_SECONDS, LEAD_SECONDS
 from rtpbench.srtp import SUITES
 
@@ -62,9 +62,16 @@ def _target_args(args: argparse.Namespace) -> str | None:
             return "--tls, --srtp: with a URI, or through the carrier (--carrier)"
         if args.lines and args.mock:
             return "--lines: not with --mock, which recognises only the bundled lines"
+        if args.source:
+            if not (ROOT / "targets" / args.name / "source.yaml").exists():
+                return f"--source: {args.name} does not build from a checkout"
+            try:
+                checkout(args.source)
+            except ValueError as e:
+                return f"--source {args.source}: {e}"
         return None
-    if args.mock or args.tls:
-        return "--mock, --tls: with a bundled target only (a URI gives TLS as sips:)"
+    if args.mock or args.tls or args.source:
+        return "--mock, --tls, --source: with a bundled target only (a URI gives TLS as sips:)"
     if args.carrier:
         from rtpbench.sip import transport
 
@@ -236,6 +243,13 @@ examples:
     p.add_argument("--seed", type=int, default=1, help=seed)
     p.add_argument("--carrier", action="store_true", help="call through the carrier (Docker)")
     p.add_argument("--mock", action="store_true", help="the mock engine instead of the AI engine")
+    p.add_argument(
+        "--source",
+        type=Path,
+        metavar="DIR",
+        help="a bundled target built from this git checkout instead of its pinned version: of"
+        " AI Voice Connector CE (aivc), AVA (ava) or livekit/sip (livekit)",
+    )
     p.add_argument(
         "--lines",
         type=Path,
@@ -507,6 +521,7 @@ example:
                 args.ca,
                 _suites(args.srtp),
                 args.tls,
+                args.source,
                 progress=progress,
             )
         print(summary)

{rtpbench-0.3.0 → rtpbench-0.4.0}/src/rtpbench/realtime.py

@@ -935,7 +935,8 @@ def uplink(out: Path) -> str:
             continue
         st = analyze.stream(up, w.log["pt"])
         for turn in w.log["turns"]:
-            for spans in (turn.get("prompt_speech"), turn["speech"]):
+            # a barge-in turn whose prompt got no reply has no interjection
+            for spans in (turn.get("prompt_speech"), turn.get("speech")):
                 if not spans:
                     continue
                 a, b = st.index(w.log["seq0"], spans[0][0]), st.index(w.log["seq0"], spans[-1][1])

rtpbench-0.4.0/targets/aivc/source.yaml

@@ -0,0 +1,9 @@
+# `rtpbench target aivc --source DIR`: the connector built from the checkout DIR with upstream's
+# own Dockerfile, as the project builds it, in place of the pinned commit. Merged over
+# compose.yaml; the tag follows the checkout's commit and, as every build's, a hash of the build.
+services:
+  engine:
+    build: !override
+      context: ${RTPBENCH_SOURCE:?}
+      dockerfile: docker/Dockerfile
+    image: ${RTPBENCH_REGISTRY:?}/rtpbench-aivc:${RTPBENCH_SOURCE_COMMIT:?}${RTPBENCH_BUILD_ENGINE:-}

rtpbench-0.4.0/targets/ava/source.yaml

@@ -0,0 +1,8 @@
+# `rtpbench target ava --source DIR`: AVA built from the checkout DIR with upstream's own
+# Dockerfile, as the project builds it, in place of the pinned commit. Merged over compose.yaml;
+# the tag follows the checkout's commit and, as every build's, a hash of the build.
+services:
+  engine:
+    build: !override
+      context: ${RTPBENCH_SOURCE:?}
+    image: ${RTPBENCH_REGISTRY:?}/rtpbench-ava:${RTPBENCH_SOURCE_COMMIT:?}${RTPBENCH_BUILD_ENGINE:-}

rtpbench-0.4.0/targets/livekit/source.yaml

@@ -0,0 +1,10 @@
+# `rtpbench target livekit --source DIR`: the SIP bridge built from a checkout DIR of
+# livekit/sip with upstream's own Dockerfile, in place of its pinned image; the server and the
+# Agents worker stay pinned. Merged over compose.yaml; the tag follows the checkout's commit and,
+# as every build's, a hash of the build.
+services:
+  sip:
+    build:
+      context: ${RTPBENCH_SOURCE:?}
+      dockerfile: build/sip/Dockerfile
+    image: ${RTPBENCH_REGISTRY:?}/rtpbench-livekit-sip:${RTPBENCH_SOURCE_COMMIT:?}${RTPBENCH_BUILD_SIP:-}

rtpbench-0.4.0/tests/test_images.py

@@ -0,0 +1,124 @@
+import json
+import re
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+from rtpbench import ROOT, bench
+from rtpbench.bench import TARGETS, Stack
+
+
+@pytest.fixture
+def root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+    """A copy of the checkout's stack files and package, which bench reads instead."""
+    root = tmp_path / "root"
+    root.mkdir()
+    for name in ("compose.yaml", "Dockerfile"):
+        shutil.copyfile(ROOT / name, root / name)
+    for name in ("corpus", "uri", "mock", "carrier", "targets", "src"):
+        shutil.copytree(ROOT / name, root / name)
+    monkeypatch.setattr(bench, "ROOT", root)
+    monkeypatch.setattr(bench, "PACKAGE", root / "src" / "rtpbench")
+    for k in ("OPENAI_API_KEY", "AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_API_KEY"):
+        monkeypatch.setenv(k, "-")
+    return root
+
+
+def builds(root: Path, target: str | None) -> dict[str, str]:
+    return Stack(root / "out", target, carrier="agent").builds()
+
+
+@pytest.mark.parametrize("target", [None, *TARGETS])
+def test_compose_tags_each_built_image_with_the_hash_of_its_build(
+    root: Path, target: str | None
+) -> None:
+    stack = Stack(root / "out", target, carrier="agent")
+    built = stack.builds()
+    services = json.loads(stack.compose("config", "--format", "json"))["services"]
+    assert built == {name: s["image"] for name, s in services.items() if "build" in s}
+    assert built["router"] == f"{bench.REGISTRY}/{bench._image_tag()}"
+    for name, image in built.items():
+        if name != "router":  # its own content hash
+            assert re.fullmatch(r"[^:]+:[^:]+-[0-9a-f]{12}", image), image
+
+
+def test_editing_a_build_changes_its_image_s_tag_and_an_unrelated_file_none(root: Path) -> None:
+    before = builds(root, "ava")
+    agent = root / "targets" / "livekit" / "agent.py"  # not in this stack
+    agent.write_text(agent.read_text() + "# edited\n")
+    assert builds(root, "ava") == before
+
+    compose = root / "targets" / "ava" / "compose.yaml"
+    old = "ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1"  # the engine's dockerfile_inline
+    assert old in compose.read_text()
+    compose.write_text(compose.read_text().replace(old, "ENV PYTHONUNBUFFERED=1"))
+    after = builds(root, "ava")
+    assert after["engine"] != before["engine"]
+    assert after["engine"].startswith(before["engine"].rpartition("-")[0] + "-")  # pin kept
+    for name in ("router", "carrier-media", "carrier-sip"):
+        assert after[name] == before[name]
+
+    constraints = root / "targets" / "ava" / "constraints.txt"  # the engine's pins context
+    constraints.write_text(constraints.read_text() + "\n")
+    assert builds(root, "ava")["engine"] != after["engine"]
+
+
+@pytest.mark.parametrize(
+    ("target", "service"), [("aivc", "engine"), ("ava", "engine"), ("livekit", "sip")]
+)
+def test_a_target_built_from_a_checkout_is_tagged_by_its_commit_and_files(
+    root: Path, tmp_path: Path, target: str, service: str
+) -> None:
+    src = tmp_path / "checkout"
+    src.mkdir()
+    (src / "main.py").write_text("print('hi')\n")
+    (src / ".gitignore").write_text(".venv/\n")
+    git = ["git", "-C", str(src), "-c", "user.name=t", "-c", "user.email=t@t"]
+    for cmd in (["init", "-q"], ["add", "."], ["commit", "-qm", "x"]):
+        subprocess.run([*git, *cmd], check=True)
+    out = src / "results" / "run"  # a run's results inside the checkout: no part of it
+    out.mkdir(parents=True)
+
+    def files(stack: Stack) -> list[Path]:
+        return [f for f in (stack.source or Path()).rglob("*") if f.is_file()]
+
+    def tag() -> str:
+        stack = Stack(out, target, carrier="agent", source=src)
+        try:
+            copied = {f.relative_to(stack.source or "").as_posix() for f in files(stack)}
+            assert copied == {".gitignore", "main.py"}  # not .git, .venv or the results
+            return stack.builds()[service]
+        finally:
+            shutil.rmtree(stack.source or "")
+
+    pinned = builds(root, target).get(service, "")
+    clean = tag()
+    commit = bench.checkout(src, out)
+    assert re.fullmatch(rf".+:{commit['commit'][:7]}-[0-9a-f]{{12}}", clean), clean
+    assert clean != pinned
+    (out / "caller.pcap").write_bytes(b"x")
+    (src / ".venv").mkdir()
+    (src / ".venv" / "lib.py").write_text("ignored\n")
+    assert tag() == clean
+    assert bench.checkout(src, out)["dirty"] is False
+    assert bench.checkout(src)["dirty"] is True  # the results are untracked files to git
+
+    (src / "__pycache__").mkdir()
+    (src / "__pycache__" / "main.pyc").write_bytes(b"x")
+    subprocess.run([*git, "add", "-f", "__pycache__"], check=True)  # tracked: still left out
+    assert tag() == clean
+    subprocess.run([*git, "rm", "-rq", "--cached", "__pycache__"], check=True)
+    shutil.rmtree(src / "__pycache__")
+    (out / "a.txt").write_text("a\n")  # a rename within the results: two fields, both aside
+    subprocess.run([*git, "add", "-f", str(out / "a.txt")], check=True)
+    subprocess.run([*git, "commit", "-qm", "results"], check=True)
+    subprocess.run([*git, "mv", str(out / "a.txt"), str(out / "b.txt")], check=True)
+    assert bench.checkout(src, out)["dirty"] is False
+
+    (src / "main.py").write_text("print('changed')\n")  # uncommitted: same commit, new files
+    assert bench.checkout(src, out)["dirty"] is True
+    assert tag() != clean
+    with pytest.raises(ValueError, match="not a git checkout"):
+        bench.checkout(tmp_path)

rtpbench-0.3.0/tests/test_images.py

@@ -1,64 +0,0 @@
-import json
-import re
-import shutil
-from pathlib import Path
-
-import pytest
-
-from rtpbench import ROOT, bench
-from rtpbench.bench import TARGETS, Stack
-
-
-@pytest.fixture
-def root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
-    """A copy of the checkout's stack files and package, which bench reads instead."""
-    root = tmp_path / "root"
-    root.mkdir()
-    for name in ("compose.yaml", "Dockerfile"):
-        shutil.copyfile(ROOT / name, root / name)
-    for name in ("corpus", "uri", "mock", "carrier", "targets", "src"):
-        shutil.copytree(ROOT / name, root / name)
-    monkeypatch.setattr(bench, "ROOT", root)
-    monkeypatch.setattr(bench, "PACKAGE", root / "src" / "rtpbench")
-    for k in ("OPENAI_API_KEY", "AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_API_KEY"):
-        monkeypatch.setenv(k, "-")
-    return root
-
-
-def builds(root: Path, target: str | None) -> dict[str, str]:
-    return Stack(root / "out", target, carrier="agent").builds()
-
-
-@pytest.mark.parametrize("target", [None, *TARGETS])
-def test_compose_tags_each_built_image_with_the_hash_of_its_build(
-    root: Path, target: str | None
-) -> None:
-    stack = Stack(root / "out", target, carrier="agent")
-    built = stack.builds()
-    services = json.loads(stack.compose("config", "--format", "json"))["services"]
-    assert built == {name: s["image"] for name, s in services.items() if "build" in s}
-    assert built["router"] == f"{bench.REGISTRY}/{bench._image_tag()}"
-    for name, image in built.items():
-        if name != "router":  # its own content hash
-            assert re.fullmatch(r"[^:]+:[^:]+-[0-9a-f]{12}", image), image
-
-
-def test_editing_a_build_changes_its_image_s_tag_and_an_unrelated_file_none(root: Path) -> None:
-    before = builds(root, "ava")
-    agent = root / "targets" / "livekit" / "agent.py"  # not in this stack
-    agent.write_text(agent.read_text() + "# edited\n")
-    assert builds(root, "ava") == before
-
-    compose = root / "targets" / "ava" / "compose.yaml"
-    old = "ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1"  # the engine's dockerfile_inline
-    assert old in compose.read_text()
-    compose.write_text(compose.read_text().replace(old, "ENV PYTHONUNBUFFERED=1"))
-    after = builds(root, "ava")
-    assert after["engine"] != before["engine"]
-    assert after["engine"].startswith(before["engine"].rpartition("-")[0] + "-")  # pin kept
-    for name in ("router", "carrier-media", "carrier-sip"):
-        assert after[name] == before[name]
-
-    constraints = root / "targets" / "ava" / "constraints.txt"  # the engine's pins context
-    constraints.write_text(constraints.read_text() + "\n")
-    assert builds(root, "ava")["engine"] != after["engine"]