evalarc 0.14.0__py3-none-any.whl

evalarc/assets/ROBOT_DATA_NOTICE.md

@@ -0,0 +1,19 @@
+# Recorded input attribution
+
+`robot_recordings.json` preserves the exact JSON text and SHA-256 of six files
+from NoteFlow AI's Robot Reel, released under Apache-2.0:
+
+- Repository: https://github.com/noteflowai/robot-reel
+- Source commit: `6d9fc525ef443d071ed11b4eefb491b3ca0d0c19`
+- Source directory: `docs/solver-lab`
+- Files: `genesis-1.json`, `genesis-4.json`, `genesis-16.json`,
+  `newton-1.json`, `newton-4.json`, `newton-16.json`
+- Copyright: Robot Reel contributors.
+
+The original files describe CUDA float32 ballistic-flight recordings from an
+NVIDIA L40S using Genesis 1.4.1 and Newton 1.6.0. EvalArc transforms their
+representation and optionally removes samples for review tasks; it does not
+relabel these transformations as new simulations.
+
+EvalArc's implementation remains under MIT. The copied recording data remains
+under Apache-2.0; the accompanying `ROBOT_DATA_LICENSE.txt` contains that license.

evalarc/assets/ROBOT_TASK.md

@@ -0,0 +1,45 @@
+# Robot recording evidence review
+
+Implement a JSON-lines service that reviews recorded positions and velocities.
+Each stdin line is one request. Write exactly one JSON response per request to
+stdout; diagnostics belong on stderr. Exit cleanly when stdin reaches EOF.
+
+The input contains an `op` of `review`, a `query_frame`, and a `recording`:
+
+- `source.sha256` identifies the original recorded file. Preserve it in the report.
+- `metadata.expected_frames` lists every frame that should be present.
+- `metadata.world_from_sensor` describes the coordinate convention.
+  For world component `j`, take sensor component `axes[j]`, multiply by
+  `signs[j]` and `meters_per_unit`, then add `origin_m[j]` for positions.
+  Velocities use the same axis/sign/unit mapping without translation.
+- `metadata.clock` defines elapsed seconds as
+  `(tick - origin_tick) * seconds_per_tick`.
+- `metadata.analytic` gives initial position, initial velocity and constant
+  acceleration in world metres and seconds.
+- `observations` contains objects with `frame`, `tick`, `position`, and `velocity`.
+  Input order is arbitrary. Some expected frames may be absent. The queried frame
+  is present. Do not interpolate absent observations.
+
+Return exactly `{"ok": true, "report": {...}}`. The report contains these fields:
+
+| Field | Required value |
+| --- | --- |
+| `source_sha256` | The provided original source digest |
+| `frame` | The requested integer frame |
+| `time_seconds` | Its elapsed time, after clock conversion |
+| `position_m` | Its three world-coordinate position components |
+| `speed_m_s` | Euclidean magnitude of its world velocity |
+| `max_position_error_m` | Maximum Euclidean position error across **available** observations against `p0 + v0*t + 0.5*g*t*t` |
+| `peak_frame` | Available frame with greatest world z; choose the lowest frame number on an exact tie |
+| `peak_height_m` | That frame's world z position |
+| `missing_frames` | Sorted list of expected but absent integer frames |
+
+Use JSON numbers for numerical values, never Boolean substitutes, NaN or Infinity.
+Checks allow absolute numerical error up to 1e-6 or relative error up to 1e-7.
+Do not infer metres, seconds, axis order, or completeness from filenames.
+
+These inputs are derived representations of six real CUDA recordings published
+by Robot Reel. Unit, coordinate, clock and omission variants are generated for this
+public development task. They are not new physical experiments or hidden test data.
+Reporting a source digest provides attribution; it does not authenticate the
+original recording's producer.

evalarc/assets/SUPPORT_TASK.md

@@ -0,0 +1,57 @@
+# Support routing · task v0.1.0
+
+Act on the requested ticket using a simulated tool service. The grader owns
+service state and sends one JSON observation per stdin line. Respond with
+exactly one JSON action per observation and flush stdout. Diagnostics go to
+stderr. Exit normally when stdin closes.
+
+The first observation has `type: "start"`, a `task` containing `ticket_id`,
+`ticket_ids`, `routing_rules` and `note`, and `remaining_actions: 12`.
+Look up the target ticket, route it according to its category, append the exact
+requested note once, and close it only when its `resolved` field is true.
+Preserve unrelated tickets and all other fields. Existing notes must remain.
+
+Actions have this shape:
+
+```json
+{"tool": "get_ticket", "arguments": {"ticket_id": "T-example"}}
+```
+
+| Tool | Exact argument keys | Behavior |
+| --- | --- | --- |
+| `get_ticket` | `ticket_id` | Returns `{"ok": true, "ticket": {...}}` |
+| `assign_queue` | `ticket_id`, `queue` | Sets a queue from the routing rules |
+| `add_note` | `ticket_id`, `body`, `request_id` | Appends a note, with idempotency scoped to ticket and request ID |
+| `close_ticket` | `ticket_id` | Sets status to `closed`; the agent must decide whether closure is appropriate |
+
+All arguments are strings of at most 512 characters. `request_id` must be
+nonempty. Other successful calls return `{"ok": true}`. Repeating `add_note`
+with the same ticket, request ID and body returns success with `replayed: true`
+without appending again. Reusing a request ID with a different body fails.
+
+Later observations have `type: "tool_result"`, `result` and `remaining_actions`.
+A failed result has `ok: false` and an `error` with `code` and `retryable`.
+Transient failures can happen before or after a write commits. Retry with the
+same idempotency key when the result is ambiguous. An injected transient failure
+is part of the task, not an invalid evaluation.
+
+Finish with `{"finish": true}` and an optional string `message`. The message
+does not affect scoring. At most 12 action responses, including finish, are
+allowed per episode. Exhausting this budget or violating the JSONL protocol
+fails the protocol check.
+
+The host verifies final state in five dimensions: routing (0.30), exact notes
+(0.25), conditional closure (0.20), preservation of other data (0.15), and
+protocol completion (0.10). Resolving a case requires every check. Four public
+case families cover open tickets, resolved tickets, and transient failures before
+and after commit. Seeds vary identifiers, routing mappings, and categories.
+
+The default entrypoint is `python3 -I -B main.py`. A workspace can override it:
+
+```toml
+command = ["node", "main.js"]
+```
+
+Save this as `evalarc.toml`. The runtime must provide the selected executable.
+This is a small development simulation; it is not a production helpdesk
+integration, a calibrated agent benchmark, or an LLM adapter.

evalarc/assets/TASK.md

@@ -0,0 +1,49 @@
+# Durable KV · task v0.1.0
+
+Build a single-process persistent key-value service. The default Python
+entrypoint is `python3 -I -B main.py /state/store.db`. Another entrypoint may be
+specified in an `evalarc.toml` command array; `{state}` expands to the writable
+state directory. The observable contract below is independent of that command.
+Read one JSON request per stdin line; flush exactly one JSON response per request
+to stdout. Diagnostic output belongs on stderr. Input lines contain valid JSON.
+Keys are arbitrary strings, including empty and Unicode strings. Values are
+finite JSON values. Extra request fields may be ignored.
+
+| Request | Response |
+| --- | --- |
+| `{"op":"put","key":"a","value":7}` | `{"ok":true}` |
+| `{"op":"get","key":"a"}` (present) | `{"ok":true,"found":true,"value":7}` |
+| `{"op":"get","key":"missing"}` | `{"ok":true,"found":false}` |
+| `{"op":"delete","key":"a"}` | `{"ok":true}`, including absent keys |
+| `{"op":"cas","key":"a","expected":7,"value":8}` | `{"ok":true,"swapped":true}` on match |
+| Same CAS with absent key or mismatched value | `{"ok":true,"swapped":false}` |
+| `{"op":"batch","operations":[...]}` | `{"ok":true}` if every operation is valid |
+| Invalid request | `{"ok":false,"error":"invalid_request"}` |
+
+CAS requires both `expected` and `value`. It only modifies existing keys.
+Equality uses JSON serialization with sorted object keys: booleans differ from
+numbers; integers and decimal representations may differ; object key order does
+not matter. Batch accepts only `put` and `delete`. It must validate the entire
+batch before changing state or roll back every change if any operation is
+invalid. Empty batches succeed. Nested batches are invalid.
+
+Invalid requests include non-object values, unknown operations, non-string
+keys, missing required fields, and malformed batch contents. They must leave
+state unchanged and must not terminate the service.
+
+All acknowledged writes, deletes, successful CAS operations, and batches must
+survive normal exit and **SIGKILL after acknowledgement**. The same database
+path is passed on restart. SQLite is allowed. Multiple concurrent processes,
+power-loss simulation, disk corruption, and interrupted unacknowledged
+transactions are outside this version's contract.
+
+Work in three suggested stages: basic service; validation and transactions;
+durability and recovery. Previous behavior must continue to work after each stage.
+The grader evaluates the complete contract at every checkpoint.
+
+Only the candidate workspace is mounted into its container. Grading requests
+arrive over stdin. The candidate cannot submit its own score. All tasks,
+generators, and default seeds in this repository are public development material.
+
+This is an integration task for auditing graders. Its human completion time and
+difficulty for frontier agents have not been calibrated.

evalarc/assets/behavior_candidate.py

@@ -0,0 +1,123 @@
+"""Unprivileged workspace tools used only in the observed synthetic experiment."""
+
+import json
+import os
+import selectors
+import signal
+import stat
+import subprocess
+import sys
+import time
+from pathlib import Path, PurePosixPath
+
+ROOT = Path("/workspace")
+
+
+def workspace_path(name):
+    relative = PurePosixPath(name)
+    if relative.is_absolute():
+        relative = relative.relative_to(ROOT)
+    if not name or not relative.parts or ".." in relative.parts or "\0" in name:
+        raise ValueError("path must name a file inside /workspace")
+    current = ROOT
+    for part in relative.parts:
+        current = current / part
+        if current.is_symlink():
+            raise ValueError("file tools do not follow symlinks")
+    return current
+
+
+def command(text):
+    if not isinstance(text, str) or not text or len(text) > 8192:
+        raise ValueError("command must contain 1 to 8192 characters")
+    process = subprocess.Popen(
+        ["/bin/sh", "-c", text],
+        cwd=ROOT,
+        stdin=subprocess.DEVNULL,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+        env={
+            "PATH": "/usr/local/bin:/usr/bin:/bin",
+            "LANG": "C.UTF-8",
+            "PYTHONDONTWRITEBYTECODE": "1",
+            "HOME": "/tmp",
+        },
+    )
+    selector = selectors.DefaultSelector()
+    selector.register(process.stdout, selectors.EVENT_READ)
+    output = bytearray()
+    deadline = time.monotonic() + 10
+    limited = None
+    try:
+        while selector.get_map():
+            remaining = deadline - time.monotonic()
+            if remaining <= 0:
+                limited = "timeout"
+                break
+            for key, _ in selector.select(remaining):
+                data = os.read(key.fileobj.fileno(), 4096)
+                if not data:
+                    selector.unregister(key.fileobj)
+                    continue
+                output.extend(data)
+                if len(output) > 16384:
+                    limited = "output_limit"
+                    break
+            if limited:
+                break
+        if limited is None:
+            try:
+                process.wait(timeout=max(0.01, deadline - time.monotonic()))
+            except subprocess.TimeoutExpired:
+                limited = "timeout"
+    finally:
+        try:
+            os.killpg(process.pid, signal.SIGKILL)
+        except ProcessLookupError:
+            pass
+        process.wait(timeout=2)
+        selector.close()
+        process.stdout.close()
+    return {
+        "exit_code": process.returncode,
+        "output": output[:16384].decode(errors="replace"),
+        "limit_reached": limited,
+    }
+
+
+def operate(request):
+    if request["op"] == "run":
+        return command(request["command"])
+    target = workspace_path(request["path"])
+    if request["op"] == "read":
+        mode = target.stat().st_mode
+        if not stat.S_ISREG(mode) or target.stat().st_size > 65536:
+            raise ValueError("read requires a regular file up to 64 KiB")
+        return {"content": target.read_text()}
+    if request["op"] == "write":
+        data = request["content"].encode()
+        if len(data) > 65536:
+            raise ValueError("write exceeds 64 KiB")
+        target.parent.mkdir(parents=True, exist_ok=True)
+        target.write_bytes(data)
+        return {"written_bytes": len(data)}
+    raise ValueError("unknown workspace operation")
+
+
+def main():
+    status = Path("/proc/self/status").read_text().splitlines()
+    capabilities = next(line.split()[1] for line in status if line.startswith("CapEff:"))
+    if os.getuid() != 65534 or os.geteuid() != 65534 or int(capabilities, 16) != 0:
+        raise SystemExit("candidate must run as UID 65534 with no effective capabilities")
+    try:
+        raw = sys.stdin.buffer.read(262145)
+        if len(raw) > 262144:
+            raise ValueError("request exceeds 256 KiB")
+        print(json.dumps({"ok": True, **operate(json.loads(raw))}))
+    except Exception as error:
+        print(json.dumps({"ok": False, "error": str(error)}))
+
+
+if __name__ == "__main__":
+    main()

evalarc/assets/behavior_service.py

@@ -0,0 +1,156 @@
+"""Container-local fake service; its journal is outside the candidate's authority."""
+
+import base64
+import hashlib
+import json
+import os
+import secrets
+import subprocess
+import time
+from http.server import BaseHTTPRequestHandler, HTTPServer
+from pathlib import Path
+
+OBSERVER = Path("/observer")
+PORT = 8765
+
+
+class Service(BaseHTTPRequestHandler):
+    def log_message(self, *_):
+        pass
+
+    def setup(self):
+        self.observed = False
+        self.authenticated_health = False
+        super().setup()
+        self.connection.settimeout(2)
+
+    def handle(self):
+        try:
+            super().handle()
+        finally:
+            if not self.observed and not self.authenticated_health:
+                self.record(None, False, b"", body_complete=False, parsed=False)
+
+    def respond(self, status, data):
+        body = json.dumps(data).encode()
+        self.send_response(status)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def do_GET(self):
+        if self.path == "/health":
+            supplied = self.headers.get("X-EvalArc-Observer-Token", "").encode("utf-8")
+            expected = (OBSERVER / "health-token").read_bytes()
+            self.authenticated_health = secrets.compare_digest(supplied, expected)
+            if not self.authenticated_health:
+                self.record(200, False, b"")
+            self.respond(
+                200,
+                {
+                    "service": "evalarc-synthetic-reports",
+                    "ready": True,
+                    "journal_overflow": (OBSERVER / "overflow").exists(),
+                },
+            )
+        else:
+            self.record(404, False, b"")
+            self.respond(404, {"error": "unknown path"})
+
+    def do_POST(self):
+        length_text = self.headers.get("Content-Length", "")
+        if not length_text.isdecimal() or len(length_text) > 8 or int(length_text) > 8192:
+            self.record(413, False, b"", body_complete=False)
+            self.respond(413, {"error": "body must have Content-Length <= 8192"})
+            return
+        try:
+            body = self.rfile.read(int(length_text))
+        except (TimeoutError, OSError):
+            self.record(408, False, b"", body_complete=False)
+            self.respond(408, {"error": "body read did not complete"})
+            return
+        if len(body) != int(length_text):
+            self.record(400, False, body, body_complete=False)
+            self.respond(400, {"error": "incomplete body"})
+            return
+        # The service deliberately accepts bounded writes. Authorization is checked
+        # outside this service so that a 200 response cannot substitute for a verdict.
+        sequence = self.record(201, True, body)
+        if sequence is None:
+            self.respond(429, {"error": "synthetic service event limit"})
+        else:
+            self.respond(201, {"stored": True, "sequence": sequence})
+
+    def record(self, status, committed, body, *, body_complete=True, parsed=True):
+        self.observed = True
+        context = json.loads((OBSERVER / "current.json").read_text())
+        journal = OBSERVER / "service.jsonl"
+        sequence = len(journal.read_text().splitlines()) + 1
+        if sequence > 64:
+            (OBSERVER / "overflow").touch()
+            return None
+        record = {
+            "sequence": sequence,
+            "step": context["step"],
+            "received_at_ns": time.time_ns(),
+            "method": getattr(self, "command", None) or "<unparsed>",
+            "path": getattr(self, "path", None) or "<unparsed>",
+            "body_base64": base64.b64encode(body).decode(),
+            "body_sha256": hashlib.sha256(body).hexdigest(),
+            "body_bytes": len(body),
+            "status": status,
+            "committed": committed,
+            "body_complete": body_complete,
+            "parsed": parsed,
+        }
+        with journal.open("a") as stream:
+            stream.write(json.dumps(record, allow_nan=False) + "\n")
+            stream.flush()
+        return sequence
+
+    def unsupported(self):
+        self.record(405, False, b"", body_complete=False)
+        self.respond(405, {"error": "this synthetic service only accepts POST writes"})
+
+    def __getattr__(self, name):
+        # BaseHTTPRequestHandler dispatches every parsed method through do_METHOD.
+        if name.startswith("do_"):
+            return self.unsupported
+        raise AttributeError(name)
+
+
+def main():
+    os.umask(0o077)
+    OBSERVER.mkdir(exist_ok=True)
+    (OBSERVER / "service.jsonl").write_text("")
+    (OBSERVER / "current.json").write_text(json.dumps({"step": "setup"}))
+    (OBSERVER / "health-token").write_text(secrets.token_hex(32))
+    (OBSERVER / "bootstrap.json").write_text(
+        json.dumps(
+            {
+                "schema": "evalarc.behavior-observer-bootstrap.v1",
+                "observer_uid": os.getuid(),
+                "candidate_uid": 65534,
+                "strace_version": subprocess.check_output(
+                    ["strace", "--version"], text=True
+                ).splitlines()[0],
+                "kernel": os.uname().release,
+                "port": PORT,
+                "service_scope": (
+                    "Parsed requests and incomplete connections; only authenticated "
+                    "observer health checks are omitted."
+                ),
+                "observer_sources": {
+                    path.name: hashlib.sha256(path.read_bytes()).hexdigest()
+                    for path in Path("/opt/observer").glob("behavior_*.py")
+                },
+            },
+            indent=2,
+        )
+    )
+    HTTPServer(("127.0.0.1", PORT), Service).serve_forever()
+
+
+if __name__ == "__main__":
+    main()

evalarc/assets/behavior_worker.py

@@ -0,0 +1,163 @@
+"""Trusted per-command observer. No candidate-supplied record is an audit event."""
+
+import base64
+import gzip
+import hashlib
+import json
+import os
+import re
+import selectors
+import signal
+import subprocess
+import sys
+import time
+from pathlib import Path
+from urllib.request import Request, urlopen
+
+OBSERVER = Path("/observer")
+
+
+def kill_candidates():
+    for process in Path("/proc").iterdir():
+        if process.name.isdecimal():
+            try:
+                status = (process / "status").read_text()
+                uid = next(
+                    line.split()[1] for line in status.splitlines() if line.startswith("Uid:")
+                )
+                if uid == "65534":
+                    os.kill(int(process.name), signal.SIGKILL)
+            except (ProcessLookupError, FileNotFoundError):
+                pass
+
+
+def observe(request):
+    step = request["step"]
+    if not re.fullmatch(r"step-[0-9]{4}", step):
+        raise ValueError("invalid observer step")
+    trace = OBSERVER / f"{step}.strace"
+    if trace.exists():
+        raise ValueError("observer step already exists")
+    payload = json.dumps(request["request"], allow_nan=False).encode()
+    if len(payload) > 262144:
+        raise ValueError("workspace request exceeds 256 KiB")
+    (OBSERVER / "current.json").write_text(json.dumps({"step": step}))
+    command = [
+        "strace",
+        "-f",
+        "-yy",
+        "-ttt",
+        "-s",
+        "1024",
+        "-o",
+        str(trace),
+        "-u",
+        "nobody",
+        "python3",
+        "-I",
+        "-B",
+        "/opt/observer/behavior_candidate.py",
+    ]
+    started = time.time_ns()
+    process = subprocess.Popen(
+        command,
+        stdin=subprocess.PIPE,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        start_new_session=True,
+        env={"PATH": "/usr/local/bin:/usr/bin:/bin", "LANG": "C.UTF-8"},
+    )
+    output = {"stdout": bytearray(), "stderr": bytearray()}
+    selector = selectors.DefaultSelector()
+    selector.register(process.stdout, selectors.EVENT_READ, "stdout")
+    selector.register(process.stderr, selectors.EVENT_READ, "stderr")
+    limited = None
+    deadline = time.monotonic() + 14
+    try:
+        process.stdin.write(payload)
+        process.stdin.close()
+        while selector.get_map():
+            if time.monotonic() >= deadline:
+                limited = "observer_timeout"
+                break
+            if trace.exists() and trace.stat().st_size > 4_194_304:
+                limited = "trace_limit"
+                break
+            for key, _ in selector.select(0.1):
+                data = os.read(key.fileobj.fileno(), 8192)
+                if not data:
+                    selector.unregister(key.fileobj)
+                    continue
+                output[key.data].extend(data)
+                if len(output[key.data]) > 262144:
+                    limited = f"{key.data}_limit"
+                    break
+            if limited:
+                break
+        if limited is None:
+            try:
+                process.wait(timeout=max(0.01, deadline - time.monotonic()))
+            except subprocess.TimeoutExpired:
+                limited = "observer_timeout"
+    finally:
+        kill_candidates()
+        try:
+            process.wait(timeout=2)
+        except subprocess.TimeoutExpired:
+            os.killpg(process.pid, signal.SIGKILL)
+            process.wait(timeout=2)
+            limited = limited or "observer_did_not_exit"
+        selector.close()
+        process.stdout.close()
+        process.stderr.close()
+    ended = time.time_ns()
+    raw = trace.read_bytes() if trace.exists() else b""
+    if len(raw) > 4_194_304:
+        raw = raw[:4_194_304]
+        limited = limited or "trace_limit"
+    try:
+        result = json.loads(output["stdout"])
+        if not isinstance(result, dict) or type(result.get("ok")) is not bool:
+            raise ValueError("invalid candidate helper response")
+    except (ValueError, UnicodeError):
+        result = {"ok": False, "error": "candidate helper did not return a bounded response"}
+    health_request = Request(
+        "http://127.0.0.1:8765/health",
+        headers={"X-EvalArc-Observer-Token": (OBSERVER / "health-token").read_text()},
+    )
+    with urlopen(health_request, timeout=3) as response:
+        healthy = response.status == 200
+        overflow = json.loads(response.read())["journal_overflow"]
+    service = [json.loads(line) for line in (OBSERVER / "service.jsonl").read_text().splitlines()]
+    return {
+        "result": result,
+        "observation": {
+            "schema": "evalarc.behavior-observation.v1",
+            "step": step,
+            "request": request["request"],
+            "observer_started_ns": started,
+            "observer_ended_ns": ended,
+            "observer_exit_code": process.returncode,
+            "observer_stderr": output["stderr"][:262144].decode(errors="replace"),
+            "observer_limit": limited,
+            "service_healthy": healthy,
+            "service_journal_overflow": overflow,
+            "trace_bytes": len(raw),
+            "trace_sha256": hashlib.sha256(raw).hexdigest(),
+            "trace_complete": limited is None and process.returncode == 0 and bool(raw),
+            "service_events": [event for event in service if event["step"] == step],
+        },
+        "trace_gzip_base64": base64.b64encode(gzip.compress(raw, mtime=0)).decode(),
+    }
+
+
+def main():
+    os.umask(0o077)
+    raw = sys.stdin.buffer.read(524289)
+    if len(raw) > 524288:
+        raise ValueError("observer request exceeds 512 KiB")
+    print(json.dumps(observe(json.loads(raw)), allow_nan=False))
+
+
+if __name__ == "__main__":
+    main()