mantatech-sdk 0.5b0__py3-none-any.whl

examples/mnist_cnn/README.md

@@ -0,0 +1,107 @@
+# MNIST CNN Example
+
+Trains a CNN to classify handwritten digits from the MNIST dataset on a Manta node cluster.
+
+## Prerequisites
+
+- A Manta account
+- Python 3.10+ and the manta SDK + examples bundle installed: `pip install --pre mantatech-sdk[examples]` (~700 MB CPU / ~2 GB CUDA — most of the weight is `torch` + `torchvision`)
+- **Docker** running locally — the example uses the `ghcr.io/mantatech/manta-light:pytorch` image to execute the worker task
+- Outbound network: gRPC to the manager (`:443`), MQTT-TLS to the broker (`:8883`), HTTPS to `ghcr.io` (image pull, first run only) and the MNIST mirror (dataset download, first run only)
+
+> **`--pre` scope:** `pip install --pre` allows pre-release versions across the **whole** dependency tree, not just `mantatech-sdk`. To constrain pre-release resolution to this package only (recommended for reproducibility), use uv: `uv pip install --prerelease=only mantatech-sdk[examples]`.
+
+By default the example **bootstraps a cluster + an in-process node for you** — no separate setup needed. If you already have a running cluster with attached nodes (e.g. via [manta-front](https://front.manta-tech.io)'s node wizard or `manta-deploy staging connect`), the example will use that instead.
+
+## Files
+
+| File | Where it runs | Purpose |
+|------|--------------|---------|
+| `worker.py` | Inside the Docker container on your node | Defines and trains the CNN |
+| `swarm.py` | Driver (your machine) | Declares the task graph and hyperparameters |
+| `driver.py` | Driver (your machine) | Orchestrates the full run via the Manta API |
+| `notebook.py` | Driver (your machine) | Interactive Marimo notebook — same flow with a live UI |
+| `_bootstrap.py` | Driver (your machine) | Helper: pick an existing RUNNING cluster, or bootstrap a fresh one with an embedded local node (example-internal, not a public SDK API) |
+| `__init__.py` | — | Marks `examples.mnist_cnn` as a Python package so it ships with the wheel |
+
+## Configuration
+
+Set these environment variables before running:
+
+```bash
+export MANTA_HOST=api.manta-tech.io   # or your self-hosted manager host
+export MANTA_PORT=443                  # 443 for cloud, 50051 for local
+
+# Either a JWT token (preferred — skips sign-in):
+export MANTA_TOKEN=<your-jwt-token>
+
+# Or email + password sign-in:
+export MANTA_EMAIL=you@example.com
+export MANTA_PASSWORD=yourpassword
+
+# Optional — cluster selection:
+export MANTA_CLUSTER_ID=<cluster-id>   # pin to a specific RUNNING cluster
+export MANTA_BOOTSTRAP=0               # disable the auto-bootstrap fallback
+```
+
+The notebook UI surfaces credentials as fields. `MANTA_TOKEN` takes precedence; if it's set, email/password are ignored.
+
+**Cluster resolution order**: a specific `MANTA_CLUSTER_ID` (if set) → the first RUNNING cluster you own with at least one node attached → a freshly bootstrapped cluster with an embedded local node (unless `MANTA_BOOTSTRAP=0`).
+
+## Running
+
+The example is bundled inside the SDK install (`mantatech-sdk[examples]`), so it works from any directory once installed.
+
+**Script (terminal):**
+
+```bash
+python -m examples.mnist_cnn.driver
+```
+
+The driver signs in, acquires a cluster (existing or freshly bootstrapped), deploys the swarm, polls until training finishes, and prints per-node accuracy and loss. A cluster the example bootstrapped is cleaned up on exit; one you already owned is left alone.
+
+Expected output (3 epochs, 5 000 samples, seed 42):
+
+```
+Connecting to api.manta-tech.io:443 ...
+Using cluster <id>
+Swarm deployed: <id>
+  [  10s] status: ACTIVE
+  ...
+  [ 30s] status: ACTIVE
+
+Results:
+  iteration=0  node=<node_id>  accuracy=0.9646  loss=0.1107
+```
+
+**Interactive notebook (Marimo):**
+
+```bash
+# Marimo loads notebooks by path; resolve the installed location:
+NOTEBOOK=$(python -c "import examples.mnist_cnn; print(examples.mnist_cnn.__path__[0])")/notebook.py
+
+marimo run "$NOTEBOOK"   # browser UI, read-only
+marimo edit "$NOTEBOOK"  # browser UI, editable cells
+```
+
+The notebook provides credential inputs, hyperparameter sliders, a run button, a live status log, and a results table — no terminal required.
+
+## Hyperparameters
+
+Edit `swarm.py` to change training behaviour:
+
+```python
+self.set_global("hyperparameters", {
+    "epochs": 3,
+    "batch_size": 64,
+    "lr": 0.001,
+    "seed": 42,        # controls dataset subset + weight init
+    "num_samples": 5000,
+})
+```
+
+The `seed` value makes results reproducible across runs.
+
+## Reproducibility
+
+Fixed seeds (`seed=42`, fixed `num_samples`, seeded DataLoader generator) make runs reproducible on the same node hardware. The example is not currently wired into CI; future integration tests can build on these seeds to assert a known accuracy value.

examples/mnist_cnn/_bootstrap.py

@@ -0,0 +1,282 @@
+"""Embed a manta-node in the driver process so the example is self-contained.
+
+Not a public SDK API — lives next to the example because a fresh Beta user
+has no cluster + no nodes, so the canonical pattern (find a RUNNING cluster
+via stream_clusters) errors fast with no recovery path. This helper creates
+a cluster, embeds a Node in the current event loop, and yields once the
+node has registered with the manager.
+
+Requires Docker on the host machine — the embedded node still spawns a
+container per task (manta-light:pytorch image). See the example README.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import shutil
+import socket
+import ssl
+import tempfile
+import time
+from contextlib import asynccontextmanager
+from pathlib import Path
+from typing import AsyncIterator, Tuple
+
+from manta.apis import AsyncUserAPI
+from manta_node import Node
+from manta_node.infrastructure.config import NodeConfigManager
+
+_logger = logging.getLogger(__name__)
+
+_DEFAULT_TIMEOUT_S = 60
+_STOP_TIMEOUT_S = 10
+_TLS_READY_TIMEOUT_S = 120.0
+_TLS_RETRY_S = 5.0
+
+
+def _wait_for_tls_ready_sync(host: str, port: int, timeout: float) -> None:
+    """Block until a TLS handshake to host:port succeeds with the system CA bundle.
+
+    cert-manager provisions a real TLS cert lazily for fresh per-cluster
+    subdomains — there's a window where the ingress serves a self-signed
+    fallback. Polling here avoids the Node startup race that would otherwise
+    surface as a misleading "JWT_SECRET mismatch" error from
+    `node_orchestrator.check_manager_availability`.
+    """
+    ctx = ssl.create_default_context()
+    deadline = time.monotonic() + timeout
+    attempt = 0
+    last_exc: BaseException = Exception("no attempt yet")
+    while True:
+        attempt += 1
+        remaining = deadline - time.monotonic()
+        if remaining <= 0:
+            raise TimeoutError(
+                f"TLS readiness probe timed out after {timeout:.0f}s waiting for "
+                f"{host}:{port} to serve a trusted certificate. Last error: {last_exc!r}"
+            )
+        sock_timeout = min(10.0, remaining)
+        try:
+            with socket.create_connection((host, port), timeout=sock_timeout) as raw:
+                with ctx.wrap_socket(raw, server_hostname=host):
+                    return
+        except ssl.SSLCertVerificationError as exc:
+            last_exc = exc
+        except OSError as exc:
+            last_exc = exc
+        print(f"  waiting for TLS readiness: {host}:{port} (attempt {attempt})...")
+        time.sleep(min(_TLS_RETRY_S, max(0.0, deadline - time.monotonic())))
+
+
+def _render_node_toml(
+    *, alias: str, manager_host: str, manager_port: int, jwt_token: str
+) -> str:
+    return f"""# Auto-generated by mnist_cnn bootstrap — do not edit.
+
+[identity]
+alias = {json.dumps(alias)}
+secured_token = {json.dumps(jwt_token)}
+random_id = false
+
+[metadata]
+name = {json.dumps(alias)}
+
+[network]
+manager_host = {json.dumps(manager_host)}
+manager_port = {manager_port}
+
+[logging]
+level = "INFO"
+log_to_file = true
+log_to_console = false
+"""
+
+
+_DEFAULT_CLUSTER_NAME = "manta-sdk-mnist-cnn-example"
+
+
+async def _sweep_prior_clusters(api: AsyncUserAPI, name: str) -> int:
+    """Stop + delete any existing clusters with this exact name.
+
+    Idempotency over orphans: the example's create-or-reset model assumes
+    any prior cluster with our reserved name is stale (the embedded node
+    that was attached to it died with the previous process). Sweep them
+    before creating a fresh one to avoid the manager surfacing them as
+    healthy-but-actually-dead.
+    """
+    swept = 0
+    async for cluster in api.stream_clusters():
+        if cluster.name != name:
+            continue
+        try:
+            await api.stop_cluster(cluster.cluster_id)
+        except Exception as exc:
+            _logger.debug("stop_cluster(%s) failed during sweep: %s", cluster.cluster_id, exc)
+        try:
+            await api.delete_cluster(cluster.cluster_id)
+            swept += 1
+            print(f"  swept prior example cluster {cluster.cluster_id}")
+        except Exception as exc:
+            _logger.warning(
+                "delete_cluster(%s) failed during sweep: %s — continuing",
+                cluster.cluster_id,
+                exc,
+            )
+    return swept
+
+
+@asynccontextmanager
+async def bootstrap_local_node(
+    api: AsyncUserAPI,
+    *,
+    cluster_name: str = _DEFAULT_CLUSTER_NAME,
+    node_alias: str = "local-node",
+    wait_timeout_s: int = _DEFAULT_TIMEOUT_S,
+) -> AsyncIterator[Tuple[str, str]]:
+    """Create a cluster, embed a Node in the current loop, yield (cluster_id, node_id).
+
+    Idempotent across runs: any prior cluster with this exact name is
+    swept before creating a new one (the previous embedded node is dead
+    by the time we run again). On exit: stops the node, deletes the
+    cluster, removes the temp config.
+    """
+    work_dir = Path(tempfile.mkdtemp(prefix="manta-mnist-bootstrap-"))
+    config_path = work_dir / "node.toml"
+    cluster_id: str | None = None
+    node: Node | None = None
+    node_task: asyncio.Task | None = None
+
+    try:
+        await _sweep_prior_clusters(api, cluster_name)
+        cluster_id = await api.create_cluster(name=cluster_name)
+        start = await api.start_cluster(cluster_id)
+        manager_host = getattr(start, "node_host", None)
+        manager_port = getattr(start, "node_port", None)
+        jwt_token = getattr(start, "jwt_token", None)
+        if not (manager_host and manager_port and jwt_token):
+            raise RuntimeError(
+                "start_cluster did not return node_host/node_port/jwt_token"
+            )
+
+        config_path.write_text(
+            _render_node_toml(
+                alias=node_alias,
+                manager_host=manager_host,
+                manager_port=int(manager_port),
+                jwt_token=jwt_token,
+            )
+        )
+        config_path.chmod(0o600)
+
+        # Wait for cert-manager to provision a trusted TLS cert for the
+        # per-cluster watchdog subdomain before starting the node. Without
+        # this, Node.check_manager_availability() races the cert issuance
+        # and dies with a misleading "JWT_SECRET mismatch" error.
+        await asyncio.to_thread(
+            _wait_for_tls_ready_sync,
+            manager_host,
+            int(manager_port),
+            _TLS_READY_TIMEOUT_S,
+        )
+
+        mgr = NodeConfigManager(config_path)
+        node = Node(mgr)
+
+        # Node.start() runs forever — drive it as a task, use node registration
+        # at the manager as the readiness signal.
+        node_task = asyncio.create_task(node.start(), name="manta-node-embedded")
+
+        deadline = time.monotonic() + wait_timeout_s
+        node_id: str | None = None
+        while True:
+            if node_task.done():
+                exc = node_task.exception()
+                if exc:
+                    raise exc
+                raise RuntimeError("Node.start() returned before registration")
+            node_ids = await api.list_node_ids(cluster_id)
+            if node_ids:
+                node_id = node_ids[0]
+                break
+            if time.monotonic() > deadline:
+                raise TimeoutError(
+                    f"Embedded node did not register within {wait_timeout_s}s"
+                )
+            await asyncio.sleep(1)
+
+        yield cluster_id, node_id
+
+    finally:
+        if node is not None:
+            try:
+                await asyncio.wait_for(node.stop(), timeout=_STOP_TIMEOUT_S)
+            except (asyncio.TimeoutError, Exception) as exc:
+                _logger.warning("node.stop() failed: %s", exc)
+        if node_task is not None and not node_task.done():
+            node_task.cancel()
+            try:
+                await node_task
+            except (asyncio.CancelledError, Exception):
+                pass
+        if cluster_id:
+            try:
+                await api.stop_cluster(cluster_id)
+                await api.delete_cluster(cluster_id)
+            except Exception as exc:
+                _logger.warning("cluster cleanup failed: %s", exc)
+        shutil.rmtree(work_dir, ignore_errors=True)
+
+
+@asynccontextmanager
+async def acquire_cluster(
+    api: AsyncUserAPI,
+    *,
+    requested_cluster_id: str | None = None,
+    allow_bootstrap: bool = True,
+) -> AsyncIterator[str]:
+    """Pick a usable cluster.
+
+    Resolution order:
+    1. If `requested_cluster_id` is set, use that cluster (must be RUNNING
+       with at least one node attached). No bootstrap, no cleanup on exit.
+    2. Else, if `allow_bootstrap`, sweep any prior auto-bootstrapped cluster
+       and bootstrap a fresh one with an embedded local node. Cleaned up
+       on exit.
+
+    The "auto-pick an existing RUNNING cluster" fallback was deliberately
+    removed: it produced confusing behaviour when a previous run's embedded
+    node had died but its cluster + stale node entry still existed
+    server-side, causing the example to silently deploy onto a dead node.
+    Power users with their own running cluster should set MANTA_CLUSTER_ID.
+    """
+    if requested_cluster_id:
+        async for cluster in api.stream_clusters():
+            if cluster.cluster_id != requested_cluster_id:
+                continue
+            if str(cluster.status) != "RUNNING":
+                raise RuntimeError(
+                    f"Cluster {requested_cluster_id} status={cluster.status!s} — must be RUNNING."
+                )
+            node_ids = await api.list_node_ids(cluster.cluster_id)
+            if not node_ids:
+                raise RuntimeError(
+                    f"Cluster {cluster.cluster_id} has 0 nodes attached. "
+                    "Attach a node before running the example."
+                )
+            yield cluster.cluster_id
+            return
+        raise RuntimeError(
+            f"No cluster found matching MANTA_CLUSTER_ID={requested_cluster_id}."
+        )
+
+    if not allow_bootstrap:
+        raise RuntimeError(
+            "MANTA_BOOTSTRAP=0 is set and no MANTA_CLUSTER_ID provided. "
+            "Set MANTA_CLUSTER_ID to a RUNNING cluster you've already prepared, "
+            "or re-enable bootstrap."
+        )
+
+    async with bootstrap_local_node(api) as (cluster_id, _node_id):
+        yield cluster_id

examples/mnist_cnn/driver.py

@@ -0,0 +1,98 @@
+import asyncio
+import os
+import sys
+
+from manta.apis import AsyncUserAPI
+from manta_common.conversions import ID, bytes_to_dict
+
+from examples.mnist_cnn._bootstrap import acquire_cluster
+from examples.mnist_cnn.swarm import MnistCnnSwarm
+
+POLL_INTERVAL_S = 10
+TIMEOUT_S = 600
+
+
+async def main():
+    host = os.environ.get("MANTA_HOST", "localhost")
+    port = int(os.environ.get("MANTA_PORT", "50051"))
+    token = os.environ.get("MANTA_TOKEN")
+    email = os.environ.get("MANTA_EMAIL")
+    password = os.environ.get("MANTA_PASSWORD")
+
+    if not token and not (email and password):
+        print(
+            "Error: set MANTA_TOKEN (preferred) or MANTA_EMAIL + MANTA_PASSWORD."
+        )
+        sys.exit(1)
+
+    print(f"Connecting to {host}:{port} ...")
+
+    requested_cluster_id = os.environ.get("MANTA_CLUSTER_ID")
+    allow_bootstrap = os.environ.get("MANTA_BOOTSTRAP", "1") != "0"
+
+    if token:
+        api = AsyncUserAPI(token=token, host=host, port=port)
+    else:
+        api = await AsyncUserAPI.sign_in(email, password, host=host, port=port)
+
+    async with api:
+        async with acquire_cluster(
+            api,
+            requested_cluster_id=requested_cluster_id,
+            allow_bootstrap=allow_bootstrap,
+        ) as cluster_id:
+            print(f"Using cluster {cluster_id}")
+
+            swarm_proto = await api.deploy_swarm(cluster_id, MnistCnnSwarm())
+            swarm_id = ID(swarm_proto.swarm_id).xid
+            print(f"Swarm deployed: {swarm_id}")
+
+            # Single-shot training: poll for results to appear, with the swarm
+            # status as a backstop in case training fails. We don't wait for
+            # the swarm itself to transition to STOPPED — SwarmStatusEnum has
+            # no terminal "completed" state for one-off tasks.
+            elapsed = 0
+            result_obj = None
+            while elapsed < TIMEOUT_S:
+                await asyncio.sleep(POLL_INTERVAL_S)
+                elapsed += POLL_INTERVAL_S
+                results = await api.select_results(swarm_id, "metrics")
+                result_obj = results.get(swarm_id)
+                if result_obj and result_obj.values:
+                    break
+                current = await api.get_swarm(swarm_id)
+                status = str(current.status)
+                print(f"  [{elapsed:>4}s] status: {status}")
+                if status == "KILLED":
+                    print(f"Swarm ended in KILLED state — training failed.")
+                    return
+
+            if not (result_obj and result_obj.values):
+                print(f"Timed out after {elapsed}s without receiving any results.")
+                return
+
+            print("\nResults:")
+            for i, iter_data in enumerate(result_obj.values):
+                for node_id, row in zip(result_obj.schema.rows, iter_data):
+                    column_data = dict(zip(result_obj.schema.columns, row))
+                    # The "metrics" column holds the serialized worker payload
+                    # (the dict the worker stored via self.world.results.add).
+                    payload = column_data.get("metrics")
+                    if isinstance(payload, (bytes, bytearray)):
+                        metrics = bytes_to_dict(payload)
+                    elif isinstance(payload, dict):
+                        metrics = payload
+                    else:
+                        metrics = {}
+                    acc = metrics.get("accuracy")
+                    loss = metrics.get("loss")
+                    acc_s = f"{acc:.4f}" if isinstance(acc, (int, float)) else "?"
+                    loss_s = f"{loss:.4f}" if isinstance(loss, (int, float)) else "?"
+                    print(
+                        f"  iteration={i}  node={node_id}  "
+                        f"accuracy={acc_s}  loss={loss_s}"
+                    )
+
+
+if __name__ == "__main__":
+    asyncio.run(main())