ellf-cli 21.0.9__tar.gz → 22.0.1__tar.gz

{ellf_cli-21.0.9/ellf_cli.egg-info → ellf_cli-22.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ellf-cli
-Version: 21.0.9
+Version: 22.0.1
 Summary: Ellf Command Line Interface
 Home-page: https://prodi.gy
 Author: ExplosionAI GmbH

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/README.md

@@ -329,23 +329,6 @@ Delete an asset registered with Ellf
 | `--cluster` | `str` | Name, slug, or ID of the cluster to search for asset name (or the last cluster if not set) | `None` |
 | `--json` | `bool` | Output the result as JSON | `False` |
 
-### `ellf auth`
-
-Manage authentication artifacts (import/push device-flow tokens)
-
-#### `ellf auth import`
-
-Read a serialised FileSecrets blob from stdin and write it to the local secrets file. Counterpart to ``ellf auth push``. Existing local auth state is overwritten; the file is written 0600 by FileSecrets.save.
-
-#### `ellf auth push`
-
-Push local auth state to a remote machine via SSH. Reads the local FileSecrets, JSON-serialises it, and pipes it to ``<remote_ellf> auth import`` over an SSH connection to *host*. Authentication is delegated to ssh — anything in your ssh config (host aliases, ProxyJump, key-based auth) just works. The pushed blob includes id_token, api_token, and any cached broker_tokens. id_token expiry is dictated by the issuer (typically ~1h), so this is a "give the remote a usable session" operation, not a permanent provisioning. Re-run when the token expires.
-
-| Argument | Type | Description | Default |
-| --- | --- | --- | --- |
-| `host` | `str` | SSH host to push the token to |  |
-| `--remote-ellf` | `str` | Command to run on the remote (default: 'ellf' from PATH). Pass an absolute path or shell-expanded form when ellf isn't on the remote login shell's PATH, e.g. '$HOME/ellf-test/venv/bin/ellf'. | `None` |
-
 ### `ellf clusters`
 
 Interact with clusters
@@ -676,12 +659,12 @@ Deploy and manage cluster infrastructure (Kubernetes, TLS, Terraform)
 
 #### `ellf infra register`
 
-Register a new cluster with PAM and write credentials to cluster-creds.json. Pass --domain to use your own DNS, or omit it to be allocated a managed subdomain (with base + wildcard A records) under an Explosion-controlled zone.
+Register a new cluster with PAM and write credentials to cluster-creds.json. Pass --domain to use your own DNS, or omit it to be allocated a managed subdomain (with base + wildcard A records) under an Explosion-controlled zone. An org on the free Solo plan has no managed subdomain: omitting --domain registers it at http://localhost.
 
 | Argument | Type | Description | Default |
 | --- | --- | --- | --- |
 | `--name` | `str` | Human-readable name for the cluster |  |
-| `--domain` | `str` | Public address the cluster is reached on, host or host:port (e.g. 'cluster.example.com' for a hosted cluster on port 443, or 'localhost:8443' for a workstation cluster behind an SSH tunnel). Stored verbatim as the cluster address; the chart derives the bare hostname for K8s ingress / cert SAN where a port isn't allowed. Omit to get a managed subdomain under an Explosion-controlled zone. | `None` |
+| `--domain` | `str` | Public address the cluster is reached on, host or host:port (e.g. 'cluster.example.com' for a hosted cluster on port 443, or 'http://localhost' for a cluster on this machine, which needs no certificate and is what the free Solo plan uses). Stored verbatim as the cluster address; the chart derives the bare hostname for K8s ingress / cert SAN where a port isn't allowed. Omit to get a managed subdomain under an Explosion-controlled zone. | `None` |
 | `--subdomain` | `str` | Requested label for a managed subdomain (e.g. 'acme' for acme.<zone>). Only valid without --domain; defaults to a slug derived from your org/cluster name. | `None` |
 | `--public-ip` | `str` | Cluster ingress IP to point the managed subdomain at. Omit to reserve the name now and re-run with --public-ip once the load balancer IP is known. | `None` |
 | `--cloud-provider` | `str` | Cloud provider: gcp, k3s, aws, azure |  |
@@ -727,7 +710,7 @@ Apply terraform to create or update cluster infrastructure. Returns True when an
 
 #### `ellf infra destroy`
 
-Destroy the cluster infrastructure created by 'infra terraform', including every annotation, file and dataset stored on it. The database, data bucket and shared volume are deletion-protected, so this refuses to run without --delete-data. To stop paying for compute while keeping the data, use 'ellf infra pause' instead. With --delete-data, a first apply lifts the protection, then the destroy runs. On AWS the database still leaves a final snapshot behind. Note this never touches the terraform state bucket: the bucket is managed outside the terraform (see 'infra init --state-bucket') and outlives the cluster so the destroy itself has state to work against.
+Destroy the cluster infrastructure created by 'infra terraform', including every annotation, file and dataset stored on it. The database, data bucket and shared volume are deletion-protected, so this refuses to run without --delete-data. To stop paying for compute while keeping the data, use 'ellf infra pause' instead. With --delete-data, a first apply lifts the protection, then the destroy runs, once the broker, its jobs and traefik are removed from the cluster. On AWS the database still leaves a final snapshot behind. Note this never touches the terraform state bucket: the bucket is managed outside the terraform (see 'infra init --state-bucket') and outlives the cluster so the destroy itself has state to work against.
 
 | Argument | Type | Description | Default |
 | --- | --- | --- | --- |
@@ -893,20 +876,6 @@ Start a cluster on infrastructure configured under your own cloud account.
 | `--config-key` | `str` | Top-level key for the cluster config. | `None` |
 | `--offline` | `Path` | Path to a credentials JSON file. Deploys without connecting to PAM. | `None` |
 
-#### `ellf infra tls`
-
-Manage TLS certificates for local cluster access. Use --self-signed when you have direct kubectl access to the cluster. Use --setup HOST to do everything from your laptop over SSH. Use --trust HOST to fetch and install an existing CA from a remote cluster.
-
-| Argument | Type | Description | Default |
-| --- | --- | --- | --- |
-| `--self-signed` | `bool` | Create a self-signed CA and certificate | `False` |
-| `--trust` | `str` | SSH host to fetch CA from and install into local trust store | `None` |
-| `--setup` | `str` | SSH host: create certs remotely, then fetch and trust the CA locally | `None` |
-| `--status` | `bool` | Show current TLS configuration | `False` |
-| `--fqdn` | `str` | Certificate FQDN (default: localhost) | `'localhost'` |
-| `--namespace` | `str` | Kubernetes namespace | `'ellf'` |
-| `--output` | `str` | Output format for trust commands: human, json, shell | `'human'` |
-
 ### `ellf jobs`
 
 Run jobs directly on the cluster

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/__main__.py

@@ -6,11 +6,10 @@ import truststore
 
 # Use the OS native trust store (macOS Keychain, /etc/ssl/certs on Linux,
 # Windows certificate store) for every httpx/urllib/requests call this
-# CLI makes — instead of httpx's default certifi bundle. This collapses
-# three trust stores into one: a `sudo security add-trusted-cert
-# ~/.prodigy-teams/ca.crt` (or the Linux equivalent) makes the browser,
-# curl, AND the CLI all trust the cluster's self-signed CA from one
-# install. No SSL_CERT_FILE, no combined ca.pem, no per-cluster bundle.
+# CLI makes — instead of httpx's default certifi bundle — so a CA the
+# machine already trusts (a corporate proxy's, or a dev stack's) is
+# trusted by the browser, curl and the CLI alike, with no SSL_CERT_FILE
+# or separate bundle to maintain.
 # Must run before any code constructs an ssl.SSLContext, so it sits at
 # module top.
 truststore.inject_into_ssl()

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/auth.py

@@ -560,9 +560,8 @@ class AuthStateImpl:
             # lookup is brittle: the registered cluster.address (e.g.
             # 'cluster.example.com' or 'localhost') often doesn't textually
             # match broker_host (e.g. 'https://localhost:8443'), and even
-            # canonicalized forms differ when the user includes a port to
-            # reach the cluster via tunnel that the cluster.address doesn't
-            # carry. cluster_id is unambiguous.
+            # canonicalized forms differ when broker_host carries a port the
+            # cluster.address doesn't. cluster_id is unambiguous.
             body: dict[str, str] = {}
             if self._cluster_id is not None:
                 body["id"] = str(self._cluster_id)

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/cli.py

@@ -106,7 +106,6 @@ cli.placeholder("actions", description=Messages.doc_actions)
 cli.placeholder("todo", description=Messages.doc_todos)
 cli.placeholder("agents", description=Messages.doc_agents)
 cli.placeholder("assets", description=Messages.doc_assets)
-cli.placeholder("auth", description=Messages.doc_auth)
 cli.placeholder("clusters", description=Messages.doc_clusters)
 cli.placeholder("config", description=Messages.doc_config)
 cli.placeholder("datasets", description=Messages.doc_datasets)

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/__init__.py

@@ -2,7 +2,6 @@ from . import (
     actions,
     agents,
     assets,
-    auth,
     clusters,
     config,
     datasets,
@@ -39,7 +38,6 @@ __all__ = [
     "todos",
     "agents",
     "assets",
-    "auth",
     "clusters",
     "config",
     "datasets",

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/__init__.py

@@ -10,5 +10,4 @@ from . import (  # noqa: F401
     setup,
     start,
     terraform,
-    tls,
 )

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/_helpers.py

@@ -5,7 +5,7 @@ import json
 import subprocess
 import tempfile
 from pathlib import Path
-from typing import Annotated
+from typing import TYPE_CHECKING, Annotated
 from uuid import UUID
 
 from pydantic import AliasChoices, BaseModel, Field, StringConstraints
@@ -14,6 +14,9 @@ from wasabi import msg
 from ellf_pam_sdk import Client
 from ellf_pam_sdk.models import ClusterCredentials, ClusterDetail
 
+if TYPE_CHECKING:
+    from ...cluster_config import ClusterConfig
+
 # Matches pam/ellf_pam/config.py's NonEmptyStr convention.
 NonEmptyStr = Annotated[str, StringConstraints(min_length=1)]
 
@@ -146,6 +149,32 @@ def get_cluster_credentials(
         )
 
 
+def use_cluster_credentials(config: "ClusterConfig") -> None:
+    """Point kubectl at the cluster *config* describes."""
+    cluster_name = config.deployment.cluster_name
+    cloud_provider = config.deployment.cloud_provider
+    if cloud_provider == "aws":
+        assert config.deployment.aws
+        get_cluster_credentials(
+            "aws", cluster_name, region=config.deployment.aws.region
+        )
+    elif cloud_provider == "azure":
+        assert config.deployment.azure
+        get_cluster_credentials(
+            "azure",
+            cluster_name,
+            resource_group=config.deployment.azure.resource_group,
+        )
+    else:
+        assert config.deployment.gcp
+        get_cluster_credentials(
+            "gcp",
+            cluster_name,
+            project=config.deployment.gcp.project,
+            zone=config.deployment.gcp.zone,
+        )
+
+
 def ensure_cluster_keypair(infra_secret_name: str, namespace: str = "ellf") -> str:
     """Ensure the infra K8s secret has a cluster RSA keypair, return the public PEM.
 

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/deploy.py

@@ -100,6 +100,12 @@ def deploy(
         )
     overrides.image.broker_version = creds.image.broker_version
     overrides.image.cpl_version = creds.image.cpl_version
+    # The registry key is PAM's too: each org has its own, which PAM may
+    # replace (a leaked key is deleted and the org given a fresh one). The
+    # copy init-values saved in values.yaml goes stale when that happens,
+    # so always deploy the key PAM hands out now.
+    if overrides.secrets is not None:
+        overrides.secrets.container_sa_key = creds.secrets.container_sa_key
     # 1. Create namespace
     msg.info(f"Ensuring namespace {namespace}...")
     subprocess.run(

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/init.py

@@ -169,12 +169,23 @@ def init(
         # Stage 1 covers the cloud path; the k3s deployment isn't
         # representable in ClusterConfig yet, so hand over to the documented
         # local flow after registration.
+        # A cluster on http://localhost needs no certificate; one on a
+        # public domain gets Let's Encrypt through cert-manager.
+        if address.startswith("http://"):
+            steps = [
+                "ellf infra init-values --creds cluster-creds.json "
+                f"--cloud-provider k3s --domain {address}",
+            ]
+        else:
+            steps = [
+                "ellf infra setup --cert-manager",
+                "ellf infra init-values --creds cluster-creds.json "
+                f"--cloud-provider k3s --domain {address} --acme-email <email>",
+            ]
+        steps.append("ellf infra deploy --values values.yaml --wait")
         msg.info(
             "Local (k3s) cluster registered. Continue with the local flow:\n"
-            "  ellf infra setup --cert-manager\n"
-            "  ellf infra init-values --creds cluster-creds.json "
-            "--cloud-provider k3s --domain <domain>\n"
-            "  ellf infra deploy --values values.yaml --wait"
+            + "\n".join(f"  {step}" for step in steps)
         )
         return
 

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/init_values.py

@@ -139,10 +139,10 @@ def init_values(
         # release registry — but that's a cross-project IAM grant only
         # Explosion can make, so it can't be the general mechanism for a
         # customer-run (BYOC) cluster in their own GCP project. Use the
-        # same shared-credential pull secret k3s/aws/azure already rely on
-        # instead: container_sa_key below is PAM's own service account,
-        # which already carries a standing reader grant on the release
-        # registry (mirrors the pattern in pam/ellf_pam/api/pypi.py).
+        # same pull secret k3s/aws/azure already rely on instead:
+        # container_sa_key below is the org's own registry key from PAM,
+        # whose service account carries a reader grant on the release
+        # registry (see pam/ellf_pam/registry_keys.py).
         values["imagePullSecrets"] = [{"name": IMAGE_PULL_SECRET_NAME}]
         values["secrets"] = {
             "infraSecretName": "ellf-infra",

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/register.py

@@ -23,6 +23,11 @@ CLOUD_PROVIDERS = ("gcp", "k3s", "aws", "azure")
 PENDING_ADDRESS = "pending.invalid"
 
 
+def _org_is_solo(client, org_id) -> bool:
+    """Whether the org is on the free Solo plan, as PAM judges it."""
+    return client.org.read(id=org_id).billing_state == "solo"
+
+
 class Registration(NamedTuple):
     """Outcome of ensure_registered, consumed by register and infra init."""
 
@@ -43,7 +48,8 @@ class Registration(NamedTuple):
         help=(
             "Public address the cluster is reached on, host or host:port "
             "(e.g. 'cluster.example.com' for a hosted cluster on port 443, "
-            "or 'localhost:8443' for a workstation cluster behind an SSH tunnel). "
+            "or 'http://localhost' for a cluster on this machine, which needs "
+            "no certificate and is what the free Solo plan uses). "
             "Stored verbatim as the cluster address; the chart derives the "
             "bare hostname for K8s ingress / cert SAN where a port isn't allowed. "
             "Omit to get a managed subdomain under an Explosion-controlled zone."
@@ -101,7 +107,8 @@ def register(
 
     Pass --domain to use your own DNS, or omit it to be allocated a managed
     subdomain (with base + wildcard A records) under an Explosion-controlled
-    zone.
+    zone. An org on the free Solo plan has no managed subdomain: omitting
+    --domain registers it at http://localhost.
     """
     from .._state import get_auth_state
 
@@ -175,6 +182,12 @@ def ensure_registered(
         ),
         None,
     )
+    # Solo's one cluster is on localhost, and PAM refuses it anything else.
+    if domain is None and _org_is_solo(client, org_id):
+        msg.info(Messages.T074)
+        # Plain http: browsers treat localhost as a secure context, so
+        # the cluster needs no certificate.
+        domain = "http://localhost"
     # With no --domain, register under a placeholder first; the managed
     # subdomain allocation below replaces it with the real FQDN. On
     # re-registration keep whatever address the cluster already has so an
@@ -255,8 +268,8 @@ def ensure_registered(
     srsly.write_json(creds_path, creds_data)
     # Persist the cluster id locally so subsequent `ellf` commands resolve the
     # cluster identity from saved-defaults instead of looking it up by address
-    # (which is brittle when the user later sets `broker_host` to a tunnel URL
-    # like `https://localhost:8443` that doesn't match the registered domain).
+    # (which is brittle when `broker_host` is spelled differently from the
+    # registered domain, e.g. `https://localhost:8443` against `localhost`).
     settings = get_saved_settings()
     prior_cluster_id = settings.cluster_id
     settings.cluster_id = cluster.id

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/start.py

@@ -24,10 +24,10 @@ from ._helpers import (
     ensure_cluster_record,
     ensure_traefik,
     get_cluster_auth,
-    get_cluster_credentials,
     get_traefik_lb_address,
     helm_registry_login,
     point_managed_dns,
+    use_cluster_credentials,
 )
 
 DEFAULT_NAMESPACE = "ellf"
@@ -62,28 +62,8 @@ def start(
         src = Path(src)
     config = load_cluster_config(src, config_path, config_key)
     check_deploy_ready(config)
-    cluster_name = config.deployment.cluster_name
     cloud_provider = config.deployment.cloud_provider
-    if cloud_provider == "aws":
-        assert config.deployment.aws
-        get_cluster_credentials(
-            "aws", cluster_name, region=config.deployment.aws.region
-        )
-    elif cloud_provider == "azure":
-        assert config.deployment.azure
-        get_cluster_credentials(
-            "azure",
-            cluster_name,
-            resource_group=config.deployment.azure.resource_group,
-        )
-    else:
-        assert config.deployment.gcp
-        get_cluster_credentials(
-            "gcp",
-            cluster_name,
-            project=config.deployment.gcp.project,
-            zone=config.deployment.gcp.zone,
-        )
+    use_cluster_credentials(config)
     public_key = ensure_cluster_keypair(config.deployment.infra_secret_name)
     cluster_detail = None
     dns_client = None

{ellf_cli-21.0.9 → ellf_cli-22.0.1}/ellf_cli/commands/infra/terraform.py

@@ -7,7 +7,7 @@ import subprocess
 import sys
 import tarfile
 from pathlib import Path
-from typing import Optional, Union
+from typing import Callable, Optional, Union
 
 from radicli import Arg
 from wasabi import msg
@@ -495,7 +495,8 @@ def destroy_terraform(
     The database, data bucket and shared volume are deletion-protected, so
     this refuses to run without --delete-data. To stop paying for compute
     while keeping the data, use 'ellf infra pause' instead. With
-    --delete-data, a first apply lifts the protection, then the destroy runs.
+    --delete-data, a first apply lifts the protection, then the destroy runs,
+    once the broker, its jobs and traefik are removed from the cluster.
     On AWS the database still leaves a final snapshot behind.
 
     Note this never touches the terraform state bucket: the bucket is managed
@@ -537,6 +538,7 @@ def destroy_terraform(
         destroy=True,
         prompt="Destroy all listed resources? 'yes' to accept\n",
         auto_accept=yes_tf_auto_accept,
+        before_apply=lambda: _clear_cluster_workloads(config),
     ):
         msg.warn(
             "Destroy declined, but the deletion protection is lifted. Re-run "
@@ -555,6 +557,7 @@ def _plan_and_apply(
     destroy: bool,
     prompt: str,
     auto_accept: bool,
+    before_apply: Optional[Callable[[], None]] = None,
 ) -> bool:
     """Plan, confirm, apply. Returns False when the plan was declined."""
     cmd = ["terraform", "plan"]
@@ -574,11 +577,59 @@ def _plan_and_apply(
             )
         if input(prompt).strip() != "yes":
             return False
+    if before_apply is not None:
+        before_apply()
     cmd = ["terraform", "apply", "-auto-approve", plan_file]
     _ = subprocess.run(cmd, check=True, text=True, cwd=tf_dir)
     return True
 
 
+def _clear_cluster_workloads(config: ClusterConfig) -> None:
+    """Remove what 'infra start' and the broker put in the cluster, which
+    terraform doesn't manage but its destroy depends on.
+
+    Every pod in the ellf namespace -- the broker's, and the jobs it runs
+    -- mounts the shared volume, and Kubernetes won't delete a PVC that a
+    pod still uses, so terraform's destroy of it hangs until it times out.
+    Traefik's LoadBalancer Service, deleted along with the cluster, can
+    leave its cloud load balancer behind, still billing.
+    """
+    from ._helpers import use_cluster_credentials
+    from .hibernate import _release_load_balancer
+    from .pause import _run
+
+    if config.deployment.hibernated:
+        return
+    try:
+        use_cluster_credentials(config)
+    except subprocess.CalledProcessError as err:
+        msg.warn(f"Couldn't reach the cluster ({err}); destroying it as it is.")
+        return
+    if _helm_release_installed("ellf", "ellf"):
+        msg.info("Uninstalling the ellf release")
+        _run(["helm", "uninstall", "ellf", "-n", "ellf", "--wait"])
+    # Jobs the broker started aren't part of the release, and helm's --wait
+    # returns before the pods of the workloads it deleted are gone.
+    _run(
+        ["kubectl", "delete", "deployments,statefulsets,jobs,pods", "--all"]
+        + ["-n", "ellf", "--ignore-not-found", "--wait", "--timeout=10m"]
+    )
+    if config.deployment.cloud_provider == "gcp":
+        _release_load_balancer(config)
+    elif _helm_release_installed("traefik", "traefik"):
+        msg.info("Uninstalling traefik to release the load balancer")
+        _run(["helm", "uninstall", "traefik", "-n", "traefik", "--wait"])
+
+
+def _helm_release_installed(release: str, namespace: str) -> bool:
+    status = subprocess.run(
+        ["helm", "status", release, "-n", namespace],
+        capture_output=True,
+        text=True,
+    )
+    return status.returncode == 0
+
+
 def _find_csi_filestore(config: ClusterConfig) -> "tuple[str, str] | None":
     """The (zone, instance) of the GKE shared volume's CSI-provisioned
     Filestore, or None when the cluster has none.