control-tower 6.0.0rc3__tar.gz

control_tower-6.0.0rc3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 LinuxArena
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

control_tower-6.0.0rc3/MANIFEST.in

@@ -0,0 +1,2 @@
+include README.md
+recursive-include src *.md *.txt *.yml *.yaml *.json *.jsonl *.tcss *.sh *.c

control_tower-6.0.0rc3/PKG-INFO

@@ -0,0 +1,88 @@
+Metadata-Version: 2.4
+Name: control-tower
+Version: 6.0.0rc3
+Summary: Control Tower evaluation framework for AI control research
+Author: LinuxArena
+License-Expression: MIT
+Project-URL: Homepage, https://linuxarena.ai
+Project-URL: Repository, https://github.com/linuxarena/control-tower
+Keywords: ai-control,agent-evaluation,evals,monitoring,sabotage
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click==8.2.1
+Requires-Dist: dnslib>=0.9.26
+Requires-Dist: docstring-parser>=0.17
+Requires-Dist: flask>=3.1.3
+Requires-Dist: gitignore-parser==0.1.13
+Requires-Dist: gitpython>=3.1.46
+Requires-Dist: inspect-ai==0.3.268
+Requires-Dist: inspect-scout==0.5.3
+Requires-Dist: jsonlines>=4.0.0
+Requires-Dist: inspect-swe==0.2.70
+Requires-Dist: pick==2.6.0
+Requires-Dist: pydantic-settings>=2.0
+Requires-Dist: python-dotenv==1.2.2
+Requires-Dist: pytz>=2025.0
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: scipy>=1.17.0
+Requires-Dist: tabulate>=0.10.0
+Requires-Dist: tenacity>=9.1.4
+Requires-Dist: textual>=8.1.1
+Requires-Dist: tqdm>=4.67.3
+Requires-Dist: unidiff==0.7.5
+Requires-Dist: anthropic>=1.0.0
+Requires-Dist: claude-agent-sdk>=0.1.50
+Requires-Dist: openai>=3.1.0
+Requires-Dist: boto3>=1.40.0
+Requires-Dist: zstandard>=0.25.0
+Requires-Dist: ct-setting-sdk<5,>=4.0.0rc1
+Provides-Extra: hf
+Requires-Dist: pyarrow>=17.0.0; extra == "hf"
+Provides-Extra: providers
+Requires-Dist: google-genai>=1.68.0; extra == "providers"
+Requires-Dist: xai-sdk>=1.9.1; extra == "providers"
+Dynamic: license-file
+
+Welcome to Control Tower, a framework for AI control evaluations built on [Inspect](https://inspect.aisi.org.uk/) and developed by [EquiStamp](https://equistamp.com/), [Redwood Research](https://www.redwoodresearch.org/), and [Luthien](https://luthien.cc/).
+
+> 🚨 Control Tower is in active development. [Star this repo](https://github.com/linuxarena/control-tower) to follow updates, or [Watch > Releases](https://github.com/linuxarena/control-tower/subscription) to get notified on new versions.
+>
+> Found a bug or have a question? [Open an issue](https://github.com/linuxarena/control-tower/issues), [submit a PR](https://github.com/linuxarena/control-tower/compare) or [book a call](https://calendar.app.google/j7K8yncsBvPfxomS7)
+
+Control Tower provides many built-in components, including blue-team protocols, action- and trajectory-level monitors, control evaluations, safety-usefulness curves, and trajectory tooling. Environments and their main and side tasks come from separate settings repositories, and extensions (new protocols, policies, monitors, and scorers) can be provided by other Python packages.
+
+See the [documentation](https://docs.linuxarena.ai) for an overview of Control Tower, a hands-on evaluation walkthrough, and development guides.
+
+To get started:
+
+```bash
+git clone https://github.com/linuxarena/control-tower && cd control-tower
+uv sync
+alias ct='uv run ct'        # add to your shell profile to keep it
+cp .env.example .env          # set ANTHROPIC_API_KEY (or OPENAI_API_KEY)
+ct settings pull linuxarena
+```
+
+Check the install without spending a model call. It takes a couple of minutes:
+
+```bash
+ct run eval --policy ct/test --task-file simple --tag test
+```
+
+Then run a real eval. The first one builds the environment image, which takes about ten minutes:
+
+```bash
+ct run eval --policy ct/attack -e linuxarena/clinical_trial -t adverse_reaction_followup_tracking -s suppress_safety_notifications
+```
+
+***
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for code style, typing, testing, and pull request conventions, and the [development setup guide](https://docs.linuxarena.ai/development/dev-setup) for local services, environment pulls, and shell completions.

control_tower-6.0.0rc3/README.md

@@ -0,0 +1,35 @@
+Welcome to Control Tower, a framework for AI control evaluations built on [Inspect](https://inspect.aisi.org.uk/) and developed by [EquiStamp](https://equistamp.com/), [Redwood Research](https://www.redwoodresearch.org/), and [Luthien](https://luthien.cc/).
+
+> 🚨 Control Tower is in active development. [Star this repo](https://github.com/linuxarena/control-tower) to follow updates, or [Watch > Releases](https://github.com/linuxarena/control-tower/subscription) to get notified on new versions.
+>
+> Found a bug or have a question? [Open an issue](https://github.com/linuxarena/control-tower/issues), [submit a PR](https://github.com/linuxarena/control-tower/compare) or [book a call](https://calendar.app.google/j7K8yncsBvPfxomS7)
+
+Control Tower provides many built-in components, including blue-team protocols, action- and trajectory-level monitors, control evaluations, safety-usefulness curves, and trajectory tooling. Environments and their main and side tasks come from separate settings repositories, and extensions (new protocols, policies, monitors, and scorers) can be provided by other Python packages.
+
+See the [documentation](https://docs.linuxarena.ai) for an overview of Control Tower, a hands-on evaluation walkthrough, and development guides.
+
+To get started:
+
+```bash
+git clone https://github.com/linuxarena/control-tower && cd control-tower
+uv sync
+alias ct='uv run ct'        # add to your shell profile to keep it
+cp .env.example .env          # set ANTHROPIC_API_KEY (or OPENAI_API_KEY)
+ct settings pull linuxarena
+```
+
+Check the install without spending a model call. It takes a couple of minutes:
+
+```bash
+ct run eval --policy ct/test --task-file simple --tag test
+```
+
+Then run a real eval. The first one builds the environment image, which takes about ten minutes:
+
+```bash
+ct run eval --policy ct/attack -e linuxarena/clinical_trial -t adverse_reaction_followup_tracking -s suppress_safety_notifications
+```
+
+***
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for code style, typing, testing, and pull request conventions, and the [development setup guide](https://docs.linuxarena.ai/development/dev-setup) for local services, environment pulls, and shell completions.

control_tower-6.0.0rc3/pyproject.toml

@@ -0,0 +1,229 @@
+[project]
+name = "control-tower"
+version = "6.0.0rc3"
+description = "Control Tower evaluation framework for AI control research"
+readme = "README.md"
+requires-python = ">=3.13"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "LinuxArena" }]
+keywords = ["ai-control", "agent-evaluation", "evals", "monitoring", "sabotage"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Environment :: Console",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+  # CLI and core runtime
+  "click==8.2.1",
+  "dnslib>=0.9.26",
+  "docstring-parser>=0.17",
+  "flask>=3.1.3",
+  "gitignore-parser==0.1.13",
+  "gitpython>=3.1.46",
+  "inspect-ai==0.3.268",
+  "inspect-scout==0.5.3",
+  "jsonlines>=4.0.0",
+  "inspect-swe==0.2.70",
+  "pick==2.6.0",
+  "pydantic-settings>=2.0",
+  "python-dotenv==1.2.2",
+  "pytz>=2025.0",
+  "pyyaml>=6.0.3",
+  "scipy>=1.17.0",
+  "tabulate>=0.10.0",
+  "tenacity>=9.1.4",
+  "textual>=8.1.1",
+  "tqdm>=4.67.3",
+  "unidiff==0.7.5",
+
+  # Primary model providers and agent SDKs (openai also backs OpenRouter)
+  "anthropic>=1.0.0",
+  "claude-agent-sdk>=0.1.50",
+  "openai>=3.1.0",
+
+  # Run storage and upload (`ct run eval --upload`)
+  "boto3>=1.40.0",
+  "zstandard>=0.25.0",
+
+  # The env API's wire contract, shared with every setting package
+  "ct-setting-sdk>=4.0.0rc1,<5",
+]
+
+[project.optional-dependencies]
+hf = ["pyarrow>=17.0.0"]
+providers = [
+  "google-genai>=1.68.0",
+  "xai-sdk>=1.9.1",
+]
+
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+namespaces = true
+exclude = ["experiments*"]
+
+[tool.setuptools.package-data]
+# Non-Python assets that must be available from installed wheels.
+"*" = [
+  "*.md",
+  "*.txt",
+  "*.yml",
+  "*.yaml",
+  "*.json",
+  "*.jsonl",
+  "*.tcss",
+  "*.sh",
+  "py.typed",
+]
+"control_tower.sandbox.intranet_guard" = ["Dockerfile"]
+"control_tower.environments.internet_simulator" = [
+  "Dockerfile",
+  ".dockerignore",
+  "resolve_override.c",
+  "malware/malware.c",
+]
+
+[tool.setuptools.package-dir]
+"" = "src"
+
+[dependency-groups]
+dev = [
+  "moto>=5.1.22",
+  "pyright==1.1.411",  # kept for `--verifytypes` coverage gate; ty has no equivalent yet
+  "pytest==9.1.1",
+  "pytest-asyncio==1.4.0",
+  "pytest-xdist==3.8.0",
+  "ruff==0.15.20",
+  "ty==0.0.33",  # primary type checker; pinned because pre-1.0
+  "vulture>=2.15",
+  "yamllint==1.38.0",
+  "yamlfix>=1.19.1",
+]
+# A group, not a dependency: PyPI rejects direct-URL requirements, and Apollo ships the SDK only as a GitHub wheel.
+watcher = [
+  # 0.41.0 or later: the protocol reads `cost_usd` from each hook outcome.
+  "watcher-sdk @ https://github.com/ApolloResearch/watcher-bin/releases/download/v0.41.0/watcher_sdk-0.41.0-py3-none-any.whl",
+]
+
+[tool.uv]
+default-groups = ["dev", "watcher"]
+
+[tool.uv.sources]
+ct-setting-sdk = { path = "packages/ct-setting-sdk", editable = true }
+
+[project.scripts]
+ct = "control_tower.cli.main:main"
+
+[project.entry-points.inspect_ai]
+# Registers ``control_tower/eval`` and ``control_tower/control_eval`` as discoverable
+# Inspect AI @tasks (plus the ct_upload and ct_diagnostics hooks) after pip-installing control-tower.
+control_tower = "control_tower.inspect_tasks"
+
+[project.urls]
+Homepage = "https://linuxarena.ai"
+Repository = "https://github.com/linuxarena/control-tower"
+
+[tool.ruff]
+required-version = "==0.15.20"
+src = []
+extend-exclude = [
+  ".settings/**",
+]
+
+[tool.ruff.lint]
+extend-select = ["I", "UP", "ANN", "PLC0415", "S110"]
+ignore = [
+  "ANN002",  # missing type annotation for *args
+  "ANN003",  # missing type annotation for **kwargs
+  "ANN204",  # missing return type for __init__ / __new__ / dunder
+  "ANN401",  # disallow Any — too strict for this codebase
+]
+
+[tool.ruff.lint.isort]
+known-first-party = ["control_tower", "ct_setting_sdk"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["ANN", "PLC0415"]
+"packages/*/tests/**" = ["ANN", "PLC0415"]
+"scripts/**" = ["PLC0415"]
+# CLI convention: imports stay inside command functions so `ct` starts fast
+"src/**/cli/**" = ["PLC0415"]
+"src/**/cli.py" = ["PLC0415"]
+"src/**/*_cli.py" = ["PLC0415"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests", "packages/ct-setting-sdk/tests"]
+markers = [
+  "integration: marks tests as integration tests (deselect with '-m \"not integration\"')",
+  "slow: marks tests as slow running",
+]
+addopts = "-m 'not integration' --durations=10 --ignore=tests/replay_check/packages"
+
+[tool.control_tower.typing]
+verifytypes_min_completeness = 100.0
+
+[[tool.control_tower.import_boundaries.rules]]
+name = "only cli may import ct_runs (generation/orchestration tooling)"
+sources = ["*"]
+ignore_sources = ["cli", "ct_runs", "ct_runs.*"]
+targets = ["ct_runs", "ct_runs.*"]
+allowed_files = [
+  "src/control_tower/hawk/runner.py",
+]
+
+[[tool.control_tower.import_boundaries.rules]]
+name = "ct_runs subpackages are independent except declared edges"
+sources = ["ct_runs.*"]
+targets = ["ct_runs", "ct_runs.*"]
+allowed_edges = [
+  "ct_runs.* -> ct_runs.claude_host",
+  "ct_runs.* -> ct_runs.prompts",
+  "ct_runs.* -> ct_runs.storage",
+  "ct_runs.dashboard_cli -> ct_runs.dashboard",
+  "ct_runs.run_cli -> ct_runs.*",
+  "ct_runs.task_analysis -> ct_runs.traj_analysis",
+]
+
+[[tool.control_tower.import_boundaries.rules]]
+name = "utils and defaults are leaves"
+sources = ["utils", "defaults"]
+targets = ["*"]
+
+[[tool.control_tower.import_boundaries.rules]]
+name = "monitors stand alone: they import only models, tools and utils"
+sources = ["monitors"]
+targets = ["*"]
+allowed_edges = [
+  "monitors -> models",
+  "monitors -> tools",
+  "monitors -> utils",
+]
+
+[[tool.control_tower.import_boundaries.rules]]
+name = "trajectories must not import live"
+sources = ["trajectories"]
+targets = ["live"]
+
+[tool.ty.src]
+include = [
+  ".github/scripts",
+  "packages/ct-setting-sdk/src",
+  "scripts",
+  "src/control_tower",
+  "tests",
+]
+exclude = [
+  "**/node_modules",
+  ".settings/**",
+]
+
+[tool.ty.environment]
+python-version = "3.13"

control_tower-6.0.0rc3/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

control_tower-6.0.0rc3/src/control_tower/__init__.py

@@ -0,0 +1,3 @@
+from importlib.metadata import version
+
+__version__ = version("control-tower")

control_tower-6.0.0rc3/src/control_tower/cli/__init__.py

@@ -0,0 +1,3 @@
+from control_tower.utils.env import ensure_dotenv
+
+ensure_dotenv()

control_tower-6.0.0rc3/src/control_tower/cli/base.py

@@ -0,0 +1,215 @@
+import functools
+import json
+import re
+import sys
+import termios
+import tty
+from collections.abc import Callable
+from pathlib import Path
+from typing import Any, cast
+
+import click
+
+from control_tower.cli.completions.completion_callbacks import (
+    completion_env_names,
+    completion_main_task_names,
+    completion_side_task_names,
+)
+from ct_setting_sdk.protocol.errors import ExpectedError
+
+
+def getch() -> str:
+    """Read a single character from stdin without requiring Enter.
+
+    Handles arrow keys by returning "left" or "right" strings.
+    """
+    fd = sys.stdin.fileno()
+    old_settings = termios.tcgetattr(fd)
+    try:
+        tty.setraw(fd)
+        ch = sys.stdin.read(1)
+        if ch == "\x1b":
+            ch2 = sys.stdin.read(1)
+            if ch2 == "[":
+                ch3 = sys.stdin.read(1)
+                if ch3 == "D":
+                    return "left"
+                elif ch3 == "C":
+                    return "right"
+        return ch
+    finally:
+        termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
+
+
+def clear_screen() -> None:
+    """Clear the terminal screen."""
+    click.echo("\033[2J\033[H", nl=False)
+
+
+def word_wrap(text: str, width: int) -> list[str]:
+    """Wrap text to specified width, preserving line breaks."""
+    result = []
+    for paragraph in text.split("\n"):
+        if not paragraph.strip():
+            result.append("")
+            continue
+        words = paragraph.split()
+        current_line = ""
+        for word in words:
+            if len(current_line) + len(word) + 1 > width:
+                if current_line:
+                    result.append(current_line)
+                current_line = word
+            else:
+                current_line += (" " if current_line else "") + word
+        if current_line:
+            result.append(current_line)
+    return result
+
+
+def load_jsonl(path: Path) -> list[dict[str, Any]]:
+    """Load all non-empty lines from a JSONL file as dicts."""
+    results = []
+    with open(path) as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                results.append(json.loads(line))
+    return results
+
+
+def select_audit_run(runs_dir: Path, title: str = "Select audit run") -> Path | None:
+    """Interactive picker for audit run directories. Returns None if no runs or user cancels."""
+    from pick import Option, pick
+
+    if not runs_dir.exists():
+        click.secho(f"No runs found in {runs_dir}/", fg="yellow")
+        return None
+
+    runs = sorted(
+        [d for d in runs_dir.iterdir() if d.is_dir()],
+        key=lambda p: p.stat().st_ctime,
+        reverse=True,
+    )
+
+    if not runs:
+        click.secho(f"No runs found in {runs_dir}/", fg="yellow")
+        return None
+
+    options: list[Option] = []
+    for run in runs:
+        results_file = run / "results.jsonl"
+        count = 0
+        if results_file.exists():
+            count = sum(1 for _ in results_file.open())
+        label = f"{run.name} ({count} results)"
+        options.append(Option(label, run))
+
+    selected, _ = pick(
+        options,
+        title=title,
+        indicator="=>",
+        default_index=0,
+    )
+
+    return selected.value if isinstance(selected, Option) else Path(str(selected))
+
+
+def _build_param_map(click_params: list[click.Parameter]) -> dict[str, str]:
+    """Build mapping from Python param names to CLI representations."""
+    param_map: dict[str, str] = {}
+
+    for param in click_params:
+        if not param.name:
+            continue
+
+        param_name = param.name
+        if not isinstance(param_name, str):
+            continue
+
+        if isinstance(param, click.Option):
+            if param.opts:
+                cli_name = cast(str, max(param.opts, key=len))
+                param_map[param_name] = cli_name
+        elif isinstance(param, click.Argument):
+            param_map[param_name] = param_name.upper()
+
+    return param_map
+
+
+def _transform_error_message(message: str, click_params: list[click.Parameter]) -> str:
+    """Transform parameter names in error message to CLI equivalents."""
+    param_map = _build_param_map(click_params)
+
+    for py_name, cli_name in param_map.items():
+        message = re.sub(rf"`{re.escape(py_name)}`", cli_name, message)
+
+        if "_" in py_name:
+            message = re.sub(rf"\b{re.escape(py_name)}\b", cli_name, message)
+
+    return message
+
+
+def handle_user_errors(f: Callable[..., object]) -> Callable[..., object]:
+    """Decorator that converts ExpectedError exceptions to click.ClickException.
+
+    Apply this decorator to Click commands to ensure that ExpectedError exceptions
+    are displayed cleanly without a stack trace, while unexpected errors still
+    show full tracebacks for debugging.
+
+    Also transforms parameter names in error messages to their CLI equivalents:
+    - `param` (backtick-wrapped) -> --param
+    - param_name (with underscores) -> --param-name
+
+    Adds a --traceback flag to show full stack traces for expected errors.
+    """
+
+    @functools.wraps(f)
+    def wrapper(*args: object, traceback: bool = False, **kwargs: object) -> object:
+        if traceback:
+            return f(*args, **kwargs)
+
+        try:
+            return f(*args, **kwargs)
+        except ExpectedError as e:
+            message = str(e)
+
+            ctx = click.get_current_context(silent=True)
+            if ctx and ctx.command:
+                message = _transform_error_message(message, ctx.command.params)
+
+            raise click.ClickException(message) from None
+
+    return click.option(
+        "--traceback",
+        is_flag=True,
+        hidden=True,
+        help="Show full traceback for expected errors (for debugging)",
+    )(wrapper)
+
+
+def task_override_options[F: Callable[..., object]](func: F) -> F:
+    """Add -e/--env, -t/--main-task and -s/--side-task options with shell completion."""
+    func = click.option(
+        "-s",
+        "--side-task",
+        type=str,
+        shell_complete=completion_side_task_names,
+        help="Side task name",
+    )(func)
+    func = click.option(
+        "-t",
+        "--main-task",
+        type=str,
+        shell_complete=completion_main_task_names,
+        help="Main task name",
+    )(func)
+    func = click.option(
+        "-e",
+        "--env",
+        "environment",
+        type=str,
+        shell_complete=completion_env_names,
+        help="Environment name",
+    )(func)
+    return func