optio-host 0.1.0__tar.gz

optio_host-0.1.0/PKG-INFO

@@ -0,0 +1,79 @@
+Metadata-Version: 2.4
+Name: optio-host
+Version: 0.1.0
+Summary: Local-or-remote host abstraction + log/deliverables coordination protocol for optio task types.
+Author-email: Kristof Csillag <kristof.csillag@deai-labs.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/deai-network/optio
+Project-URL: Repository, https://github.com/deai-network/optio
+Project-URL: Issues, https://github.com/deai-network/optio/issues
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Topic :: System :: Systems Administration
+Classifier: Framework :: AsyncIO
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: optio-core<0.2,>=0.1
+Requires-Dist: asyncssh>=2.14
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+
+# optio-host
+
+Local-or-remote host abstraction plus the log/deliverables coordination protocol used by optio task types.
+
+`optio-host` lets a task author run shell commands, manage workdirs, and stream files **without caring whether the work happens locally or on a remote host over SSH**. It also provides a small line-based protocol that long-running worker processes can use to report progress and produce file deliverables.
+
+## What's in the box
+
+- **`Host` Protocol + `LocalHost` / `RemoteHost` / `make_host()`** — uniform interface for running commands, opening port forwards, transferring files, and tearing down workdirs. SSH details (auth, multiplexing, channel cleanup) are hidden behind `asyncssh`.
+- **`HookContext`** — small carrier passed into task hooks so they can run additional host commands, request file fetches, and report progress without touching `optio-core` internals.
+- **`optio_host.protocol`** — a line-oriented session driver. A long-running process on the host writes lines prefixed `STATUS:`, `DELIVERABLE:`, `DONE`, or `ERROR`. The driver tails the log, dispatches progress events, fetches deliverable files, and resolves the session on `DONE` / `ERROR`.
+- **`create_download_task(...)`** — a ready-made optio task that downloads a file from a remote host with progress reporting and integrity checks.
+
+## When to use it
+
+You're building an [optio](https://github.com/deai-network/optio) task type that needs to run work on a host — local or remote — and you want:
+
+- one abstraction that works in both modes,
+- a structured way for the running process to talk back to optio (progress + deliverables),
+- SSH transport handled for you.
+
+If you're writing the end-user task type directly (not consuming this library from another optio task package), you probably want `optio-core` instead.
+
+## Installation
+
+```bash
+pip install optio-host
+```
+
+`optio-host` depends on `optio-core` and `asyncssh`. Python 3.11+.
+
+## Minimal example
+
+```python
+from optio_host import make_host, SSHConfig
+
+# Local
+async with make_host(ssh=None) as host:
+    result = await host.run(["uname", "-a"])
+    print(result.stdout)
+
+# Remote
+ssh = SSHConfig(host="worker-1", user="optio", key_path="~/.ssh/id_optio")
+async with make_host(ssh=ssh) as host:
+    result = await host.run(["uname", "-a"])
+    print(result.stdout)
+```
+
+## License
+
+Apache-2.0.

optio_host-0.1.0/README.md

@@ -0,0 +1,51 @@
+# optio-host
+
+Local-or-remote host abstraction plus the log/deliverables coordination protocol used by optio task types.
+
+`optio-host` lets a task author run shell commands, manage workdirs, and stream files **without caring whether the work happens locally or on a remote host over SSH**. It also provides a small line-based protocol that long-running worker processes can use to report progress and produce file deliverables.
+
+## What's in the box
+
+- **`Host` Protocol + `LocalHost` / `RemoteHost` / `make_host()`** — uniform interface for running commands, opening port forwards, transferring files, and tearing down workdirs. SSH details (auth, multiplexing, channel cleanup) are hidden behind `asyncssh`.
+- **`HookContext`** — small carrier passed into task hooks so they can run additional host commands, request file fetches, and report progress without touching `optio-core` internals.
+- **`optio_host.protocol`** — a line-oriented session driver. A long-running process on the host writes lines prefixed `STATUS:`, `DELIVERABLE:`, `DONE`, or `ERROR`. The driver tails the log, dispatches progress events, fetches deliverable files, and resolves the session on `DONE` / `ERROR`.
+- **`create_download_task(...)`** — a ready-made optio task that downloads a file from a remote host with progress reporting and integrity checks.
+
+## When to use it
+
+You're building an [optio](https://github.com/deai-network/optio) task type that needs to run work on a host — local or remote — and you want:
+
+- one abstraction that works in both modes,
+- a structured way for the running process to talk back to optio (progress + deliverables),
+- SSH transport handled for you.
+
+If you're writing the end-user task type directly (not consuming this library from another optio task package), you probably want `optio-core` instead.
+
+## Installation
+
+```bash
+pip install optio-host
+```
+
+`optio-host` depends on `optio-core` and `asyncssh`. Python 3.11+.
+
+## Minimal example
+
+```python
+from optio_host import make_host, SSHConfig
+
+# Local
+async with make_host(ssh=None) as host:
+    result = await host.run(["uname", "-a"])
+    print(result.stdout)
+
+# Remote
+ssh = SSHConfig(host="worker-1", user="optio", key_path="~/.ssh/id_optio")
+async with make_host(ssh=ssh) as host:
+    result = await host.run(["uname", "-a"])
+    print(result.stdout)
+```
+
+## License
+
+Apache-2.0.

optio_host-0.1.0/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "optio-host"
+version = "0.1.0"
+description = "Local-or-remote host abstraction + log/deliverables coordination protocol for optio task types."
+readme = "README.md"
+license = "Apache-2.0"
+requires-python = ">=3.11"
+authors = [
+    { name = "Kristof Csillag", email = "kristof.csillag@deai-labs.com" },
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Topic :: System :: Distributed Computing",
+    "Topic :: System :: Systems Administration",
+    "Framework :: AsyncIO",
+]
+dependencies = [
+    "optio-core>=0.1,<0.2",
+    "asyncssh>=2.14",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.23",
+]
+
+[project.urls]
+Homepage = "https://github.com/deai-network/optio"
+Repository = "https://github.com/deai-network/optio"
+Issues = "https://github.com/deai-network/optio/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

optio_host-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

optio_host-0.1.0/src/optio_host/__init__.py

@@ -0,0 +1,37 @@
+"""optio-host — local-or-remote host abstraction + log/deliverables protocol.
+
+Top-level public API. See ``optio_host.host`` for primitives,
+``optio_host.context`` for HookContext, and ``optio_host.protocol``
+for the log/deliverables coordination protocol.
+"""
+
+from optio_host.context import (
+    HookContext,
+    HookContextProtocol,
+    HostCommandError,
+    RunResult,
+)
+from optio_host.download import DownloadFailed, create_download_task
+from optio_host.host import (
+    Host,
+    LocalHost,
+    ProcessHandle,
+    RemoteHost,
+    make_host,
+)
+from optio_host.types import SSHConfig
+
+__all__ = [
+    "Host",
+    "LocalHost",
+    "RemoteHost",
+    "ProcessHandle",
+    "make_host",
+    "HookContext",
+    "HookContextProtocol",
+    "HostCommandError",
+    "RunResult",
+    "SSHConfig",
+    "DownloadFailed",
+    "create_download_task",
+]

optio_host-0.1.0/src/optio_host/archive.py

@@ -0,0 +1,114 @@
+"""Tar+gzip helpers for persisting and restoring a workdir.
+
+`yield_workdir_archive` is an async generator that yields gzipped tar
+chunks of a directory's contents. `consume_workdir_archive` consumes such
+a stream, wiping the destination first and extracting in. Both run their
+synchronous tarfile work in a thread executor so the event loop stays
+responsive.
+
+These helpers back `LocalHost.archive_workdir` and `LocalHost.restore_workdir`.
+`RemoteHost` does not use them — it shells out to `tar` over SSH instead.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import fnmatch
+import io
+import os
+import shutil
+import tarfile
+from typing import AsyncIterator, Iterable
+
+
+DEFAULT_WORKDIR_EXCLUDES: list[str] = [
+    ".git",
+    "node_modules",
+    "__pycache__",
+    ".venv",
+    "*.pyc",
+    ".DS_Store",
+]
+
+_CHUNK_SIZE = 1 << 20  # 1 MiB
+
+
+def _excluded(relpath: str, patterns: list[str]) -> bool:
+    """Return True if `relpath` matches any pattern in `patterns`.
+
+    Matching is applied at each path segment (so `.git` in `patterns`
+    excludes `./a/b/.git/…`) as well as against the full relative path
+    (so `*.pyc` matches `mod.pyc` but also `a/b/mod.pyc`).
+    """
+    parts = relpath.split(os.sep)
+    for pat in patterns:
+        if fnmatch.fnmatch(relpath, pat):
+            return True
+        for part in parts:
+            if fnmatch.fnmatch(part, pat):
+                return True
+    return False
+
+
+def _build_archive_bytes(root: str, patterns: list[str]) -> bytes:
+    """Build the entire tar.gz in memory and return it as bytes."""
+    buf = io.BytesIO()
+    with tarfile.open(fileobj=buf, mode="w:gz") as tar:
+        for dirpath, dirnames, filenames in os.walk(root):
+            rel_dir = os.path.relpath(dirpath, root)
+            dirnames[:] = [
+                d for d in dirnames
+                if not _excluded(os.path.join(rel_dir, d) if rel_dir != "." else d, patterns)
+            ]
+            for name in filenames:
+                rel = os.path.join(rel_dir, name) if rel_dir != "." else name
+                if _excluded(rel, patterns):
+                    continue
+                tar.add(os.path.join(dirpath, name), arcname=rel, recursive=False)
+    return buf.getvalue()
+
+
+async def yield_workdir_archive(
+    root: str,
+    exclude: Iterable[str] | None = None,
+) -> AsyncIterator[bytes]:
+    """Yield 1 MiB chunks of the gzipped tar of `root`."""
+    patterns = list(DEFAULT_WORKDIR_EXCLUDES) if exclude is None else list(exclude)
+    loop = asyncio.get_event_loop()
+    blob = await loop.run_in_executor(None, _build_archive_bytes, root, patterns)
+    for offset in range(0, len(blob), _CHUNK_SIZE):
+        yield blob[offset : offset + _CHUNK_SIZE]
+
+
+def _empty_dir(path: str) -> None:
+    if not os.path.isdir(path):
+        os.makedirs(path, exist_ok=True)
+        return
+    for entry in os.listdir(path):
+        full = os.path.join(path, entry)
+        if os.path.isdir(full) and not os.path.islink(full):
+            shutil.rmtree(full, ignore_errors=True)
+        else:
+            try:
+                os.unlink(full)
+            except OSError:
+                pass
+
+
+def _extract_sync(blob: bytes, dest: str) -> None:
+    _empty_dir(dest)
+    with tarfile.open(fileobj=io.BytesIO(blob), mode="r:gz") as tar:
+        tar.extractall(dest)
+
+
+async def consume_workdir_archive(
+    stream: AsyncIterator[bytes],
+    dest: str,
+) -> None:
+    """Empty `dest`, then untar the chunked gzipped stream into it."""
+    chunks = []
+    async for chunk in stream:
+        chunks.append(chunk)
+    blob = b"".join(chunks)
+    loop = asyncio.get_event_loop()
+    await loop.run_in_executor(None, _extract_sync, blob, dest)

optio_host-0.1.0/src/optio_host/context.py

@@ -0,0 +1,271 @@
+"""HookContext: ProcessContext + host primitives for task-body hooks."""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+
+@dataclass
+class RunResult:
+    stdout: str
+    stderr: str
+    exit_code: int
+
+
+class HostCommandError(Exception):
+    def __init__(
+        self,
+        command: str,
+        exit_code: int,
+        stdout: str,
+        stderr: str,
+    ) -> None:
+        self.command = command
+        self.exit_code = exit_code
+        self.stdout = stdout
+        self.stderr = stderr
+        super().__init__(
+            f"command failed (exit {exit_code}): {command!r}\n"
+            f"stderr: {stderr[:200]}"
+        )
+
+
+from typing import Any, AsyncIterator, Awaitable, Callable, Protocol
+
+# Imported at module top so tests can monkeypatch this name on the module.
+from optio_host.download import create_download_task
+
+
+class HookContext:
+    """ProcessContext + host primitives, passed to before/after_execute hooks
+    and to on_deliverable callbacks.
+
+    Attributes not defined on this class fall through to the wrapped
+    ProcessContext via __getattr__, so consumers can call e.g.
+    ``hook_ctx.report_progress(...)`` directly.
+    """
+
+    def __init__(self, ctx, host) -> None:
+        # Use object.__setattr__ to avoid __getattr__ recursion in __init__.
+        object.__setattr__(self, "_ctx", ctx)
+        object.__setattr__(self, "_host", host)
+
+    def __getattr__(self, name: str) -> Any:
+        # Only called if the attribute isn't found on the instance / class.
+        return getattr(self._ctx, name)
+
+    async def run_on_host(
+        self,
+        command: str,
+        *,
+        check: bool = True,
+        capture_stderr: bool = False,
+        cwd: str | None = None,
+    ):
+        result = await self._host.run_command(command, cwd=cwd)
+        if check:
+            if result.exit_code != 0:
+                raise HostCommandError(
+                    command=command,
+                    exit_code=result.exit_code,
+                    stdout=result.stdout,
+                    stderr=result.stderr,
+                )
+            return result.stdout + (result.stderr if capture_stderr else "")
+        return result
+
+    async def copy_file(
+        self,
+        source,
+        target: str,
+        *,
+        skip_if_unchanged: bool = False,
+    ) -> None:
+        from bson import ObjectId  # local import to avoid a hard top-level dep
+
+        host_home = await self._host.resolve_host_home()
+        abs_target = _resolve_target_path(target, self._host.workdir, host_home)
+        basename = os.path.basename(abs_target) or abs_target
+        ctx = self._ctx
+        skipped = False
+
+        def _progress_cb(percent, message):
+            nonlocal skipped
+            if message == "already up to date":
+                skipped = True
+                ctx.report_progress(None, f"Already up to date: {basename}")
+            elif percent is not None:
+                ctx.report_progress(percent, None)
+
+        # Resolve blob sources to (iterator, expected_sha) pair.
+        expected_sha: str | None = None
+        if isinstance(source, ObjectId):
+            payload = await self._read_blob_bytes(source)
+            if skip_if_unchanged:
+                import hashlib
+                expected_sha = hashlib.sha256(payload).hexdigest()
+
+            async def _gen():
+                # Yield the payload as one (or a few) chunks. For very large
+                # blobs we could stream via load_blob, but reading-then-iterating
+                # is simpler and correct.
+                yield payload
+
+            source = _gen()
+
+        if skip_if_unchanged:
+            ctx.report_progress(None, f"Verifying {basename}...")
+        else:
+            ctx.report_progress(None, f"Copying {basename}...")
+
+        await self._host.put_file_to_host(
+            source,
+            abs_target,
+            expected_sha256=expected_sha,
+            skip_if_unchanged=skip_if_unchanged,
+            progress_cb=_progress_cb,
+        )
+
+        if skip_if_unchanged and not skipped:
+            ctx.report_progress(None, f"Copying {basename}...")
+
+    async def _read_blob_bytes(self, file_id) -> bytes:
+        async with self._ctx.load_blob(file_id) as reader:
+            return await reader.read()
+
+    async def read_from_host(self, path: str, *, silent: bool = False) -> bytes:
+        host_home = await self._host.resolve_host_home()
+        abs_path = _resolve_target_path(path, self._host.workdir, host_home)
+        if not silent:
+            basename = os.path.basename(abs_path) or abs_path
+            self._ctx.report_progress(None, f"Reading {basename}...")
+
+        def _progress_cb(percent, message):
+            if percent is not None:
+                self._ctx.report_progress(percent, None)
+
+        return await self._host.fetch_bytes_from_host(
+            abs_path, progress_cb=_progress_cb,
+        )
+
+    async def read_text_from_host(self, path: str, *, silent: bool = False) -> str:
+        data = await self.read_from_host(path, silent=silent)
+        return data.decode("utf-8")
+
+    async def download_file(
+        self,
+        url: str,
+        target: str,
+        *,
+        description: str | None = None,
+        cleanup_on_fail: bool = True,
+    ) -> None:
+        """Download ``url`` to ``target`` on the host as a child task.
+
+        Spawns a child process under the current task that runs curl on the
+        same host the parent runs on. Reports a single "Downloading
+        <basename>" message followed by numeric progress percent updates on
+        the child's ProcessContext.
+
+        ``target`` is resolved by the same rules as ``copy_file``: absolute
+        path | ``~`` / ``~/...`` home-relative | workdir-relative. On a
+        workdir-escape attempt this raises ``ValueError`` without spawning
+        a child.
+
+        Returns None on success. Raises
+        ``optio_core.exceptions.ChildProcessFailed`` if the child fails;
+        the original ``DownloadFailed`` is preserved via ``__cause__``
+        (and on the child's ``ChildResult.original_exception`` when caller
+        uses ``parallel_group``). Parent-task cancellation propagates to
+        the child automatically.
+        """
+        host_home = await self._host.resolve_host_home()
+        abs_target = _resolve_target_path(target, self._host.workdir, host_home)
+        basename = os.path.basename(abs_target) or abs_target
+
+        n = self._ctx._child_counter.get("next", 0)
+        child_process_id = f"{self._ctx.process_id}.download-{n}"
+        child_name = f"download {basename}"
+
+        task = create_download_task(
+            process_id=child_process_id,
+            name=child_name,
+            url=url,
+            target=abs_target,
+            host=self._host,
+            description=description,
+            cleanup_on_fail=cleanup_on_fail,
+        )
+        await self._ctx.run_child_task(task)
+
+
+class HookContextProtocol(Protocol):
+    """Type-hint surface for hook authors who want IDE discoverability.
+
+    Subset of ProcessContext + the four new methods.
+    """
+
+    process_id: str
+
+    @property
+    def params(self) -> dict: ...
+
+    @property
+    def metadata(self) -> dict: ...
+
+    def report_progress(self, percent: float | None, message: str | None = None) -> None: ...
+    def should_continue(self) -> bool: ...
+    async def copy_file(
+        self,
+        source,
+        target: str,
+        *,
+        skip_if_unchanged: bool = False,
+    ) -> None: ...
+    async def run_on_host(
+        self,
+        command: str,
+        *,
+        check: bool = True,
+        capture_stderr: bool = False,
+        cwd: str | None = None,
+    ) -> "str | RunResult": ...
+    async def read_from_host(self, path: str, *, silent: bool = False) -> bytes: ...
+    async def read_text_from_host(self, path: str, *, silent: bool = False) -> str: ...
+    async def download_file(
+        self,
+        url: str,
+        target: str,
+        *,
+        description: str | None = None,
+        cleanup_on_fail: bool = True,
+    ) -> None: ...
+
+
+def _resolve_target_path(path: str, workdir: str, host_home: str) -> str:
+    """Resolve a user-supplied path to an absolute host path.
+
+    Three forms:
+      - starts with `/` → absolute, used as-is
+      - `~` or starts with `~/` → home-relative, expand once
+      - otherwise → workdir-relative; reject `..` and any escape
+
+    workdir and host_home must both be absolute paths.
+    """
+    if not path:
+        raise ValueError("path must not be empty")
+    if path.startswith("/"):
+        return path
+    if path == "~":
+        return host_home
+    if path.startswith("~/"):
+        return host_home + "/" + path[2:]
+    # workdir-relative
+    if ".." in path.split("/"):
+        raise ValueError(f"workdir-relative path may not contain '..': {path!r}")
+    resolved = os.path.normpath(os.path.join(workdir, path))
+    workdir_norm = os.path.normpath(workdir).rstrip("/")
+    if resolved != workdir_norm and not resolved.startswith(workdir_norm + "/"):
+        raise ValueError(f"workdir-relative path escapes workdir: {path!r}")
+    return resolved