horus-agent 0.1.2__tar.gz

horus_agent-0.1.2/.gitignore

@@ -0,0 +1,54 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+*.ipynb
+
+# Virtual environments
+venv/
+.venv/
+ENV/
+env/
+
+# IDEs
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+
+# OS
+.DS_Store
+Thumbs.db
+
+*.xml
+
+# i18n
+*.mo
+
+# logs
+logs/
+
+no-commit/

horus_agent-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Temple Compute
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

horus_agent-0.1.2/PKG-INFO

@@ -0,0 +1,130 @@
+Metadata-Version: 2.5
+Name: horus_agent
+Version: 0.1.2
+Summary: Horus target for machines that only allow outbound connections: a small agent polls a broker for target operations and runs them locally.
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.13
+Requires-Dist: horus-runtime>=0.8.4
+Requires-Dist: httpx>=0.28
+Description-Content-Type: text/markdown
+
+# horus-agent
+
+A Horus target for machines the orchestrator cannot connect to, for example an HPC login node or a lab server behind a VPN.
+
+The machine runs `horus-agent`. It connects **out** to a broker over HTTPS, polls it for work and runs it locally. Nothing listens on the machine, so no inbound port or VPN access is needed.
+
+The target, kind `agent`, works like the SSH target. Each channel primitive (`launch`, `poll`, `put_file`, `get_file`, `mkdir`, ...) becomes an op that the broker queues for the agent, and the agent runs it with Horus's own `LocalTarget`. These work on agent targets exactly as they do on SSH:
+
+- executors (shell, containers, Python environments);
+- Slurm with `inner: {kind: agent, ...}`;
+- the resource monitor;
+- transfers.
+
+## Install and register
+
+The machine needs Python 3.13 or newer. `uv` downloads it if it is missing.
+
+```sh
+uv tool install horus-agent      # or: pip install horus-agent
+horus-agent register --url https://api.example.com/api/v1 --token <one-time token>
+horus-agent run
+```
+
+`register` stores the agent's credential in `~/.config/horus-agent/agent.json` (mode 0600). To use another location, set `HORUS_AGENT_CONFIG`. The agent's name comes from the token when the admin set one in the UI, else from `--name`, else the hostname.
+
+`--work-dir PATH` sets where tasks run on this machine; the default is `~/horus-agent`. It is created if missing, stored in the config, and reported to the broker, so workflows do not set it. `horus-agent run --work-dir PATH` overrides it for one run.
+
+### Keep it running
+
+As a systemd user service (see `examples/horus-agent.service`):
+
+```sh
+cp examples/horus-agent.service ~/.config/systemd/user/
+systemctl --user enable --now horus-agent
+loginctl enable-linger "$USER"   # keep it running after logout
+```
+
+Without systemd (for example on a login node), run it in `tmux` or with `nohup horus-agent run &`.
+
+Jobs run detached, in their own session, with their output in log files. Stopping, restarting or upgrading the agent does not stop them; the orchestrator picks them up again once the agent is back.
+
+Proxies: the agent honours `HTTPS_PROXY`, `NO_PROXY` and `SSL_CERT_FILE`.
+
+### Logs
+
+By default the agent logs each op it receives at info level: the command it runs, or the path it reads or writes, with the run it belongs to. It also logs exit codes, the pids of launched jobs, and when each job exits. The status checks it repeats every few seconds for running jobs (`poll`, `read_output`) appear only in debug logs.
+
+### Debug logs
+
+`horus-agent run --debug` (or `HORUS_AGENT_DEBUG=1`) logs everything the agent does:
+- every TCP and TLS connection and HTTP request;
+- each poll, with how long it waited and how many ops it got;
+- each op with its arguments, its duration and its result;
+- every command it runs, with a preview of the output;
+- file transfers, job pids, cancels, and results delivered.
+
+Secrets stay out of the log, so it can be shared. Environment variables appear by name only, because they carry task secrets. Presigned URLs lose their query strings, which hold the signatures. Tokens are never logged.
+
+## Use it in a workflow
+
+```yaml
+target:
+  kind: agent
+  agent: lab-server          # the agent's name
+```
+
+Slurm on the agent's machine:
+
+```yaml
+target:
+  kind: slurm
+  inner:
+    kind: agent
+    agent: hpc-login
+```
+
+| Field | Default | Meaning |
+|---|---|---|
+| `agent` | required | Agent name |
+| `working_directory` | the agent's `--work-dir` | Usually unset. Absolute: used instead of the agent's directory. Relative: taken under it |
+| `offline_timeout` | `900` | Seconds without a poll before the agent's tasks count as lost |
+
+## Orchestrator configuration
+
+The broker is set on the orchestrator, not in workflows:
+
+| Variable | Meaning |
+|---|---|
+| `HORUS_AGENT_BROKER_URL` | Broker base URL |
+| `HORUS_AGENT_BROKER_TOKEN` | Token for the broker |
+| `HORUS_AGENT_BROKER_TOKEN_CONTEXT_KEY` | Instead of a fixed token: the `HorusContext.data` key holding a per-run token |
+
+The broker HTTP API is described in `src/horus_agent/protocol.py`. Temple Compute OS implements it.
+
+## Limits
+
+- Files over 4 MiB, in either direction, go through presigned object-storage URLs that the broker issues.
+- Output from `run_command(detach=False)` is buffered and capped at 4 MiB. Long-running commands go through `launch`, the default for this target.
+- The `python_function_external` executor is refused: it would load bytes written on the agent's machine as code in the orchestrator.
+- Task commands run as the agent's Unix user and can read its config file. Run the agent as a dedicated user, the same advice as for GitHub self-hosted runners.
+
+## Development
+
+```sh
+uv sync --group dev
+uv run make lint
+uv run make test
+```
+
+`tests/integration/test_slurm.py` runs Slurm through an agent against a fake Slurm CLI (`tests/fake_slurm`) and is part of `make test`. To check it against a real cluster, start horus-slurm's docker cluster and run the opt-in test, which installs the agent in the login container:
+
+```sh
+# from a horus-slurm checkout
+ITEST_DIR=/tmp/horus-agent-slurm-itest-$(id -u) \
+ITEST_PROJECT=horus-agent-slurm-itest tests/integration/cluster/up.sh
+# from this repo
+HORUS_AGENT_SLURM_ITEST=1 uv run pytest --no-cov tests/integration/test_slurm_cluster.py
+```

horus_agent-0.1.2/README.md

@@ -0,0 +1,118 @@
+# horus-agent
+
+A Horus target for machines the orchestrator cannot connect to, for example an HPC login node or a lab server behind a VPN.
+
+The machine runs `horus-agent`. It connects **out** to a broker over HTTPS, polls it for work and runs it locally. Nothing listens on the machine, so no inbound port or VPN access is needed.
+
+The target, kind `agent`, works like the SSH target. Each channel primitive (`launch`, `poll`, `put_file`, `get_file`, `mkdir`, ...) becomes an op that the broker queues for the agent, and the agent runs it with Horus's own `LocalTarget`. These work on agent targets exactly as they do on SSH:
+
+- executors (shell, containers, Python environments);
+- Slurm with `inner: {kind: agent, ...}`;
+- the resource monitor;
+- transfers.
+
+## Install and register
+
+The machine needs Python 3.13 or newer. `uv` downloads it if it is missing.
+
+```sh
+uv tool install horus-agent      # or: pip install horus-agent
+horus-agent register --url https://api.example.com/api/v1 --token <one-time token>
+horus-agent run
+```
+
+`register` stores the agent's credential in `~/.config/horus-agent/agent.json` (mode 0600). To use another location, set `HORUS_AGENT_CONFIG`. The agent's name comes from the token when the admin set one in the UI, else from `--name`, else the hostname.
+
+`--work-dir PATH` sets where tasks run on this machine; the default is `~/horus-agent`. It is created if missing, stored in the config, and reported to the broker, so workflows do not set it. `horus-agent run --work-dir PATH` overrides it for one run.
+
+### Keep it running
+
+As a systemd user service (see `examples/horus-agent.service`):
+
+```sh
+cp examples/horus-agent.service ~/.config/systemd/user/
+systemctl --user enable --now horus-agent
+loginctl enable-linger "$USER"   # keep it running after logout
+```
+
+Without systemd (for example on a login node), run it in `tmux` or with `nohup horus-agent run &`.
+
+Jobs run detached, in their own session, with their output in log files. Stopping, restarting or upgrading the agent does not stop them; the orchestrator picks them up again once the agent is back.
+
+Proxies: the agent honours `HTTPS_PROXY`, `NO_PROXY` and `SSL_CERT_FILE`.
+
+### Logs
+
+By default the agent logs each op it receives at info level: the command it runs, or the path it reads or writes, with the run it belongs to. It also logs exit codes, the pids of launched jobs, and when each job exits. The status checks it repeats every few seconds for running jobs (`poll`, `read_output`) appear only in debug logs.
+
+### Debug logs
+
+`horus-agent run --debug` (or `HORUS_AGENT_DEBUG=1`) logs everything the agent does:
+- every TCP and TLS connection and HTTP request;
+- each poll, with how long it waited and how many ops it got;
+- each op with its arguments, its duration and its result;
+- every command it runs, with a preview of the output;
+- file transfers, job pids, cancels, and results delivered.
+
+Secrets stay out of the log, so it can be shared. Environment variables appear by name only, because they carry task secrets. Presigned URLs lose their query strings, which hold the signatures. Tokens are never logged.
+
+## Use it in a workflow
+
+```yaml
+target:
+  kind: agent
+  agent: lab-server          # the agent's name
+```
+
+Slurm on the agent's machine:
+
+```yaml
+target:
+  kind: slurm
+  inner:
+    kind: agent
+    agent: hpc-login
+```
+
+| Field | Default | Meaning |
+|---|---|---|
+| `agent` | required | Agent name |
+| `working_directory` | the agent's `--work-dir` | Usually unset. Absolute: used instead of the agent's directory. Relative: taken under it |
+| `offline_timeout` | `900` | Seconds without a poll before the agent's tasks count as lost |
+
+## Orchestrator configuration
+
+The broker is set on the orchestrator, not in workflows:
+
+| Variable | Meaning |
+|---|---|
+| `HORUS_AGENT_BROKER_URL` | Broker base URL |
+| `HORUS_AGENT_BROKER_TOKEN` | Token for the broker |
+| `HORUS_AGENT_BROKER_TOKEN_CONTEXT_KEY` | Instead of a fixed token: the `HorusContext.data` key holding a per-run token |
+
+The broker HTTP API is described in `src/horus_agent/protocol.py`. Temple Compute OS implements it.
+
+## Limits
+
+- Files over 4 MiB, in either direction, go through presigned object-storage URLs that the broker issues.
+- Output from `run_command(detach=False)` is buffered and capped at 4 MiB. Long-running commands go through `launch`, the default for this target.
+- The `python_function_external` executor is refused: it would load bytes written on the agent's machine as code in the orchestrator.
+- Task commands run as the agent's Unix user and can read its config file. Run the agent as a dedicated user, the same advice as for GitHub self-hosted runners.
+
+## Development
+
+```sh
+uv sync --group dev
+uv run make lint
+uv run make test
+```
+
+`tests/integration/test_slurm.py` runs Slurm through an agent against a fake Slurm CLI (`tests/fake_slurm`) and is part of `make test`. To check it against a real cluster, start horus-slurm's docker cluster and run the opt-in test, which installs the agent in the login container:
+
+```sh
+# from a horus-slurm checkout
+ITEST_DIR=/tmp/horus-agent-slurm-itest-$(id -u) \
+ITEST_PROJECT=horus-agent-slurm-itest tests/integration/cluster/up.sh
+# from this repo
+HORUS_AGENT_SLURM_ITEST=1 uv run pytest --no-cov tests/integration/test_slurm_cluster.py
+```

horus_agent-0.1.2/pyproject.toml

@@ -0,0 +1,124 @@
+[project]
+name = "horus_agent"
+description = "Horus target for machines that only allow outbound connections: a small agent polls a broker for target operations and runs them locally."
+readme = "README.md"
+requires-python = ">=3.13"
+classifiers = [
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+]
+dependencies = ["horus-runtime>=0.8.4", "httpx>=0.28"]
+dynamic = ["version"]
+
+[project.scripts]
+horus-agent = "horus_agent.cli:main"
+
+# ================
+# Plugin entry-points
+# ================
+[project.entry-points."horus.target"]
+agent = "horus_agent.target"
+
+# ================
+
+[dependency-groups]
+dev = [
+  "licenseheaders~=0.8",
+  "mypy~=1.19",
+  "pre_commit~=4.0",
+  "pytest~=9.0",
+  "pytest-asyncio~=1.3.0",
+  "pytest-cov~=7.0",
+  "ruff~=0.15",
+  "horus-slurm==0.6.2",
+]
+
+[build-system]
+requires = ["hatchling>=1.29,<2.0", "uv-dynamic-versioning>=0.14.0,<1.0"]
+build-backend = "hatchling.build"
+
+[tool.coverage.run]
+source = ["horus_agent"]
+
+[tool.coverage.report]
+omit = ["tests/*"]
+
+# Other tool configurations
+[tool.hatch.version]
+source = "uv-dynamic-versioning"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/horus_agent"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src/horus_agent", "tests"]
+
+[tool.mypy]
+python_version = "3.13"
+strict = true
+exclude = ["build", "dist"]
+plugins = ["pydantic.mypy"]
+
+[tool.pydantic-mypy]
+init_typed = true
+init_forbid_extra = true
+warn_required_dynamic_aliases = true
+
+[tool.pytest.ini_options]
+addopts = """
+    --verbose
+    --cov=src/horus_agent
+    --cov-report=html
+    --cov-report=term-missing
+    --cov-fail-under=90
+"""
+minversion = "7.0"
+python_classes = ["Test*"]
+python_files = ["test_*.py", "*_test.py"]
+python_functions = ["test_*"]
+testpaths = ["tests"]
+asyncio_mode = "auto"
+
+[tool.ruff]
+line-length = 79
+target-version = "py313"
+exclude = [".git", ".venv", "build", "dist", "__pycache__"]
+
+[tool.ruff.lint]
+select = [
+  "D",  # pydocstyle
+  "E",  # pycodestyle errors
+  "W",  # pycodestyle warnings
+  "F",  # pyflakes
+  "I",  # isort
+  "N",  # pep8-naming
+  "UP",  # pyupgrade
+  "B",  # flake8-bugbear
+  "PL",  # pylint
+  "RUF",  # ruff-specific
+  "ARG",  # ruff-arglint
+  "SLF",  # ruff-slf
+]
+extend-select = ["T201"]  # Prevent use of print()
+
+ignore = [
+  "E203",  # Whitespace before ':', conflicts with formatter
+  "D104",  # Missing docstring in __init__ (often redundant)
+  "D205",  # 1 blank line required between summary line and description
+  "D212",  # Multi-line docstring summary should start at the first line
+  "D200",  # One-line docstring should fit on one line
+  "PLR2004",  # Magic numbers are often more readable than constants for things like HTTP status codes
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**/*.py" = ["SLF001", "PLC0415", "ARG002", "D"]
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+
+[tool.uv-dynamic-versioning]
+fallback-version = "0.0.1"

horus_agent-0.1.2/src/horus_agent/__init__.py

@@ -0,0 +1,6 @@
+#
+# horus_agent
+# Copyright (c) 2026 Temple Compute
+#
+# MIT License
+#

horus_agent-0.1.2/src/horus_agent/__main__.py

@@ -0,0 +1,13 @@
+#
+# horus_agent
+# Copyright (c) 2026 Temple Compute
+#
+# MIT License
+#
+"""``python -m horus_agent``."""
+
+import sys
+
+from horus_agent.cli import main
+
+sys.exit(main())

horus_agent-0.1.2/src/horus_agent/blobs.py

@@ -0,0 +1,89 @@
+#
+# horus_agent
+# Copyright (c) 2026 Temple Compute
+#
+# MIT License
+#
+"""
+Streamed transfers to and from presigned staging URLs.
+
+Used on both sides: the orchestrator and the agent each move large files
+through object storage instead of through the broker.
+"""
+
+import time
+from collections.abc import AsyncIterator
+from pathlib import Path
+
+import httpx
+from horus_runtime.logging import horus_logger
+
+from horus_agent.logs import safe_url
+
+_CHUNK = 1024 * 1024
+MAX_SINGLE_PUT = 5 * 1024**3
+"""S3's limit for one PUT."""
+
+
+async def _read_chunks(path: Path) -> AsyncIterator[bytes]:
+    with path.open("rb") as fh:
+        while chunk := fh.read(_CHUNK):
+            yield chunk
+
+
+async def upload(url: str, source: Path | bytes) -> None:
+    """
+    PUT *source* to the presigned *url*, streaming a file from disk.
+
+    ``Content-Length`` is always sent: S3 refuses a chunked presigned PUT.
+    """
+    if isinstance(source, Path):
+        size = source.stat().st_size
+        if size > MAX_SINGLE_PUT:
+            # shortcut: one presigned PUT, S3's limit; add multipart if
+            # agents need to move larger single files.
+            raise ValueError(
+                f"{source} is {size} bytes; files over 5 GiB cannot be "
+                "moved to or from an agent yet"
+            )
+        content: bytes | AsyncIterator[bytes] = _read_chunks(source)
+    else:
+        size = len(source)
+        content = source
+    log = horus_logger.log
+    log.debug(f"Upload {size} bytes to {safe_url(url)}")
+    started = time.monotonic()
+    async with httpx.AsyncClient(timeout=None) as client:
+        response = await client.put(
+            url, content=content, headers={"Content-Length": str(size)}
+        )
+        response.raise_for_status()
+    log.debug(f"Uploaded {size} bytes in {time.monotonic() - started:.2f}s")
+
+
+async def download_to(url: str, dest: Path) -> None:
+    """GET the presigned *url* into *dest*, streamed, creating parents."""
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    log = horus_logger.log
+    log.debug(f"Download {safe_url(url)} to {dest}")
+    started = time.monotonic()
+    async with (
+        httpx.AsyncClient(timeout=None) as client,
+        client.stream("GET", url) as response,
+    ):
+        response.raise_for_status()
+        with dest.open("wb") as fh:
+            async for chunk in response.aiter_bytes(_CHUNK):
+                fh.write(chunk)
+    log.debug(
+        f"Downloaded {dest.stat().st_size} bytes to {dest} in "
+        f"{time.monotonic() - started:.2f}s"
+    )
+
+
+async def download(url: str) -> bytes:
+    """GET the presigned *url* into memory."""
+    async with httpx.AsyncClient(timeout=None) as client:
+        response = await client.get(url)
+        response.raise_for_status()
+        return response.content