nomadctl 0.2.0__tar.gz → 0.3.0__tar.gz

{nomadctl-0.2.0 → nomadctl-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nomadctl
-Version: 0.2.0
+Version: 0.3.0
 Summary: A friendly command-line tool for managing a HashiCorp Nomad homelab cluster.
 Keywords: cli,devops,hashicorp,homelab,nomad,orchestration
 Author: Nathaniel Landau
@@ -60,9 +60,9 @@ your easy to remember job and volume names and the cli does the rest.
 
 - Python 3.13 or 3.14.
 - A reachable Nomad cluster.
-- The `nomad` binary on your `PATH`. The `plan`, `run`, `exec`, and `logs` commands
-  shell out to it, because the HTTP API cannot parse HCL2 job files and does not own
-  the interactive exec protocol. The other commands use the API only.
+- The `nomad` binary on your `PATH`. The `plan`, `run`, `update`, `exec`, and `logs`
+  commands shell out to it, because the HTTP API cannot parse HCL2 job files and does
+  not own the interactive exec protocol. The other commands use the API only.
 
 ## Installation
 
@@ -171,6 +171,7 @@ Run `nd --help`, or `nd <command> --help`, for the full option list at any time.
 | `nd list`                   | List discovered job files and whether each is running, dead, or not deployed.     |
 | `nd plan [JOB]`             | Preview the changes one or more job files would apply, including to running jobs. |
 | `nd run [JOB]`              | Deploy not-yet-running job files and watch the rollout.                           |
+| `nd update [JOB]`           | Recreate a running job from its local file and watch the rollout.                 |
 | `nd stop [JOB]`             | Stop, and optionally purge, running jobs and watch them drain.                    |
 | `nd logs [JOB]`             | Stream, tail, or export a task's logs.                                            |
 | `nd exec [JOB]`             | Open an interactive shell inside a running task.                                  |
@@ -197,12 +198,13 @@ touching the cluster:
 
 ```bash
 nd run --dry-run
+nd update web --dry-run
 nd stop web --dry-run
 nd volume register --dry-run
 ```
 
-For `nd run`, a dry run still validates each job file locally, so it catches a
-broken spec without registering anything.
+For `nd run` and `nd update`, a dry run still validates each job file locally, so it
+catches a broken spec without registering anything.
 
 ### Deploying jobs
 
@@ -216,6 +218,30 @@ nd run web            # deploy the job whose name starts with "web"
 nd run web --detach   # register and return immediately
 ```
 
+### Updating jobs
+
+`nd update` recreates a job that is already running. Reach for it to roll out an
+edited job file, or to pull a fresh version when the file is unchanged, such as a
+container that tracks a moving tag. It only offers jobs that are both running and
+have a local file.
+
+Each selected job is stopped, drained, purged, then re-registered from its local
+file and watched until the new rollout settles. The job is fully recreated, so
+expect brief downtime. `nd update` confirms before it acts unless you pass `--force`
+(`-f`), and purges by default (unlike `nd stop`, which keeps the job unless you pass
+`--purge`); pass `--no-purge` to keep the job's version history.
+
+```bash
+nd update                 # choose from every running job that has a local file
+nd update web             # recreate the job whose name starts with "web"
+nd update web --no-purge  # recreate but keep the version history
+nd update web --force     # skip the confirmation prompt
+```
+
+Whether a new container image is actually pulled depends on the job's Docker driver
+config, such as `force_pull` or a pinned digest, not on `nd`. The recreate
+guarantees fresh allocations; the image policy stays with your job spec.
+
 ### Working with logs
 
 `nd logs` streams both stdout and stderr live until you press Ctrl-C. Narrow or

{nomadctl-0.2.0 → nomadctl-0.3.0}/README.md

@@ -28,9 +28,9 @@ your easy to remember job and volume names and the cli does the rest.
 
 - Python 3.13 or 3.14.
 - A reachable Nomad cluster.
-- The `nomad` binary on your `PATH`. The `plan`, `run`, `exec`, and `logs` commands
-  shell out to it, because the HTTP API cannot parse HCL2 job files and does not own
-  the interactive exec protocol. The other commands use the API only.
+- The `nomad` binary on your `PATH`. The `plan`, `run`, `update`, `exec`, and `logs`
+  commands shell out to it, because the HTTP API cannot parse HCL2 job files and does
+  not own the interactive exec protocol. The other commands use the API only.
 
 ## Installation
 
@@ -139,6 +139,7 @@ Run `nd --help`, or `nd <command> --help`, for the full option list at any time.
 | `nd list`                   | List discovered job files and whether each is running, dead, or not deployed.     |
 | `nd plan [JOB]`             | Preview the changes one or more job files would apply, including to running jobs. |
 | `nd run [JOB]`              | Deploy not-yet-running job files and watch the rollout.                           |
+| `nd update [JOB]`           | Recreate a running job from its local file and watch the rollout.                 |
 | `nd stop [JOB]`             | Stop, and optionally purge, running jobs and watch them drain.                    |
 | `nd logs [JOB]`             | Stream, tail, or export a task's logs.                                            |
 | `nd exec [JOB]`             | Open an interactive shell inside a running task.                                  |
@@ -165,12 +166,13 @@ touching the cluster:
 
 ```bash
 nd run --dry-run
+nd update web --dry-run
 nd stop web --dry-run
 nd volume register --dry-run
 ```
 
-For `nd run`, a dry run still validates each job file locally, so it catches a
-broken spec without registering anything.
+For `nd run` and `nd update`, a dry run still validates each job file locally, so it
+catches a broken spec without registering anything.
 
 ### Deploying jobs
 
@@ -184,6 +186,30 @@ nd run web            # deploy the job whose name starts with "web"
 nd run web --detach   # register and return immediately
 ```
 
+### Updating jobs
+
+`nd update` recreates a job that is already running. Reach for it to roll out an
+edited job file, or to pull a fresh version when the file is unchanged, such as a
+container that tracks a moving tag. It only offers jobs that are both running and
+have a local file.
+
+Each selected job is stopped, drained, purged, then re-registered from its local
+file and watched until the new rollout settles. The job is fully recreated, so
+expect brief downtime. `nd update` confirms before it acts unless you pass `--force`
+(`-f`), and purges by default (unlike `nd stop`, which keeps the job unless you pass
+`--purge`); pass `--no-purge` to keep the job's version history.
+
+```bash
+nd update                 # choose from every running job that has a local file
+nd update web             # recreate the job whose name starts with "web"
+nd update web --no-purge  # recreate but keep the version history
+nd update web --force     # skip the confirmation prompt
+```
+
+Whether a new container image is actually pulled depends on the job's Docker driver
+config, such as `force_pull` or a pinned digest, not on `nd`. The recreate
+guarantees fresh allocations; the image policy stays with your job spec.
+
 ### Working with logs
 
 `nd logs` streams both stdout and stderr live until you press Ctrl-C. Narrow or

{nomadctl-0.2.0 → nomadctl-0.3.0}/pyproject.toml

@@ -28,7 +28,7 @@
     name = "nomadctl"
     readme = "README.md"
     requires-python = ">=3.13, <3.15"
-    version = "0.2.0"
+    version = "0.3.0"
 
     [project.urls]
         Homepage   = "https://github.com/natelandau/nd"

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/__init__.py

@@ -1,6 +1,6 @@
 """The nd package."""
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 from nd.cli import main
 

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/cli.py

@@ -9,7 +9,7 @@ import typer
 from nclutils import pp
 
 from nd import __version__
-from nd.commands import clean, exec, logs, plan, run, status, stop, volume  # noqa: A004
+from nd.commands import clean, exec, logs, plan, run, status, stop, update, volume  # noqa: A004
 from nd.commands import list as list_cmd
 from nd.nomad import (
     NomadAuthError,
@@ -28,6 +28,7 @@ app.add_typer(clean.app, name="clean")
 app.add_typer(list_cmd.app, name="list")
 app.add_typer(plan.app, name="plan")
 app.add_typer(run.app, name="run")
+app.add_typer(update.app, name="update")
 app.add_typer(logs.app, name="logs")
 app.add_typer(exec.app, name="exec")
 app.add_typer(volume.app, name="volume")

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/commands/run.py

@@ -17,7 +17,7 @@ from nd.commands._common import VerboseOption, configure_verbosity
 from nd.constants import DEPLOY_TIMEOUT_SECONDS, HEALTHY_ALLOC_STATUSES, POLL_INTERVAL_SECONDS
 from nd.jobfiles import candidates_for, discover_job_files, load_job_directories
 from nd.nomad import NomadClient, NomadConfig
-from nd.nomad.errors import NomadError
+from nd.nomad.errors import NomadDecodeError, NomadError
 from nd.targets import resolve_targets, select_candidates
 from nd.ui.alloc_rows import alloc_children
 from nd.ui.duration import summary_title
@@ -106,10 +106,12 @@ def _task_role(lifecycle: dict[str, object] | None, index: int) -> tuple[int, st
     return (1_000 + index, "main")
 
 
+# Outcome labels carry the same color as their glyph so the status word reads as
+# success/failure at a glance, not just the leading mark.
 _OUTCOME_ROW: dict[DeployStatus, tuple[str, str]] = {
-    DeployStatus.DEPLOYED: (OUTCOME_GLYPH["ok"], "deployed"),
-    DeployStatus.FAILED: (OUTCOME_GLYPH["fail"], "failed"),
-    DeployStatus.TIMEOUT: (OUTCOME_GLYPH["warn"], "still deploying"),
+    DeployStatus.DEPLOYED: (OUTCOME_GLYPH["ok"], "[green]deployed[/]"),
+    DeployStatus.FAILED: (OUTCOME_GLYPH["fail"], "[red]failed[/]"),
+    DeployStatus.TIMEOUT: (OUTCOME_GLYPH["warn"], "[yellow]still deploying[/]"),
 }
 
 
@@ -316,8 +318,13 @@ async def _deploy_one(
         lifecycle = task_lifecycle(body)
         update("registering")
         resp = await client.jobs.register(body)
-        outcome = await _watch(
-            client, candidate.name, node_names=node_names, lifecycle=lifecycle, update=update
+        outcome = await watch_deploy(
+            client,
+            candidate.name,
+            node_names=node_names,
+            lifecycle=lifecycle,
+            update=update,
+            since_index=resp.job_modify_index,
         )
         # Attach any register warnings so the caller can surface them after the panel closes.
         return replace(outcome, warnings=resp.warnings)
@@ -325,13 +332,14 @@ async def _deploy_one(
         return DeployOutcome(candidate.name, DeployStatus.FAILED, str(exc))
 
 
-async def _watch(
+async def watch_deploy(
     client: NomadClient,
     job_id: str,
     *,
     node_names: dict[str, str],
     lifecycle: TaskLifecycle,
     update: PanelUpdate,
+    since_index: int = 0,
 ) -> DeployOutcome:
     """Poll a registered job until its deployment (or allocations) settle or time out.
 
@@ -346,27 +354,47 @@ async def _watch(
         node_names: Map of node ID to node name for the per-allocation detail rows.
         lifecycle: Task ordering and labels from the compiled job spec.
         update: Callback to update the live panel phase text and detail rows.
+        since_index: The ``JobModifyIndex`` from this registration. Deployments
+            created before it belong to a previous run and are ignored so a re-run
+            of a dead job is not reported as instantly deployed off a stale record.
 
     Returns:
         The terminal deploy outcome for this job.
     """
     deadline = time.monotonic() + DEPLOY_TIMEOUT_SECONDS
     while True:
-        allocs = await client.jobs.allocations(job_id)
-        children = alloc_children(allocs, node_names, lifecycle)
-        deployments = await client.jobs.deployments(job_id)
-        if deployments:  # service job: follow the most-recent deployment
-            dep = await client.deployments.read(deployments[0].id)
-            if dep.status == _DEPLOY_SUCCESS:
-                return DeployOutcome(job_id, DeployStatus.DEPLOYED)
-            if dep.status in _DEPLOY_FAILURE:
-                return DeployOutcome(job_id, DeployStatus.FAILED, dep.status_description)
-            update(deploy_phase(dep), children)
-        else:  # batch/system job: follow allocations
-            running = sum(1 for a in allocs if a.client_status in HEALTHY_ALLOC_STATUSES)
-            if allocs and running == len(allocs):
-                return DeployOutcome(job_id, DeployStatus.DEPLOYED)
-            update(f"placing {running}/{len(allocs) or '?'} allocs", children)
+        try:
+            allocs = await client.jobs.allocations(job_id)
+            deployments = await client.jobs.deployments(job_id)
+            # The plural endpoint's ordering is undocumented, so pick this run's
+            # deployment by index rather than trusting position. A job that has ever
+            # run keeps its prior deployments listed; ignoring those created before
+            # this registration is what stops a stale "successful" record from
+            # ending the watch the instant a dead job is re-run.
+            mine = [d for d in deployments if d.create_index >= since_index]
+            latest = max(mine, key=lambda d: d.create_index) if mine else None
+            dep = await client.deployments.read(latest.id) if latest else None
+        except NomadDecodeError as exc:
+            # A freshly-placed allocation can momentarily serialize in a shape we
+            # cannot decode (e.g. TaskStates: null before its tasks start). Skip
+            # this tick and retry rather than failing an otherwise-healthy deploy;
+            # the deadline below is the backstop if it never recovers.
+            pp.debug(f"{job_id}: skipping poll after transient decode error: {exc}")
+        else:
+            children = alloc_children(allocs, node_names, lifecycle)
+            if dep is not None:  # service job: follow this run's deployment
+                if dep.status == _DEPLOY_SUCCESS:
+                    return DeployOutcome(job_id, DeployStatus.DEPLOYED)
+                if dep.status in _DEPLOY_FAILURE:
+                    return DeployOutcome(job_id, DeployStatus.FAILED, dep.status_description)
+                update(deploy_phase(dep), children)
+            elif deployments:  # service job whose new deployment has not appeared yet
+                update("registering", children)
+            else:  # batch/system job: follow allocations
+                running = sum(1 for a in allocs if a.client_status in HEALTHY_ALLOC_STATUSES)
+                if allocs and running == len(allocs):
+                    return DeployOutcome(job_id, DeployStatus.DEPLOYED)
+                update(f"placing {running}/{len(allocs) or '?'} allocs", children)
         if time.monotonic() >= deadline:
             return DeployOutcome(job_id, DeployStatus.TIMEOUT, "deploy still in progress")
         await asyncio.sleep(POLL_INTERVAL_SECONDS)

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/commands/status/render.py

@@ -69,29 +69,67 @@ def _banner_title(report: StatusReport) -> str:
     return title
 
 
+def _fraction(part: int, whole: int, *, suffix: str = "") -> str:
+    """Render an ``n/total`` fraction, flagging a shortfall in yellow.
+
+    Values stay at full brightness so the data rail reads louder than the dim label rail.
+    """
+    value = f"{part}/{whole}"
+    colored = value if part >= whole else f"[yellow]{value}[/]"
+    return f"{colored} {suffix}" if suffix else colored
+
+
+def _count(value: int, label: str, *, style: str) -> str:
+    """Render a ``value label`` pair, applying ``style`` only when the count is non-zero."""
+    shown = f"[{style}]{value}[/]" if value else str(value)
+    return f"{shown} {label}"
+
+
+def _allocs_value(report: StatusReport) -> str:
+    """Render the running/failed/pending alloc breakdown with severity-coded counts."""
+    return "  ".join(
+        (
+            _count(report.allocs_running, "running", style="green"),
+            _count(report.allocs_failed, "failed", style="red"),
+            _count(report.allocs_pending, "pending", style="yellow"),
+        )
+    )
+
+
 def _banner(report: StatusReport) -> Panel:
-    """Build the top summary banner."""
+    """Build the top summary banner as an aligned two-column key/value grid."""
     style = _HEALTH_STYLE[report.health]
-    grid = Table.grid(padding=(0, 0))
-    grid.add_row(f"[{style}]●[/] [bold {style}]{report.health.value}[/]")
-    grid.add_row(
-        f"Servers {report.servers_alive}/{report.servers_total} · "
-        f"leader {report.leader_name or 'none'}   "
-        f"Nodes {report.nodes_ready}/{report.nodes_total} ready"
-    )
-    grid.add_row(
-        f"Jobs {report.jobs_running}/{report.jobs_total} running   "
-        f"Allocs {report.allocs_running} running · "
-        f"{report.allocs_failed} failed · {report.allocs_pending} pending"
-    )
-    grid.add_row(
-        f"Deployments {len(report.deployments_active)} active   "
-        f"Evals {len(report.evals_problem)} blocked   "
-        f"Volumes {report.volumes_total}"
-    )
-    return Panel(
-        grid, title=_banner_title(report), title_align="left", border_style=style, expand=False
-    )
+
+    # Left group is infrastructure, right group is workload; uppercase labels echo the
+    # table column headers below so the banner and panels read as one type system.
+    left: list[tuple[str, str]] = [
+        ("SERVERS", _fraction(report.servers_alive, report.servers_total, suffix="alive")),
+        ("LEADER", report.leader_name or "[red]none[/]"),
+        ("NODES", _fraction(report.nodes_ready, report.nodes_total, suffix="ready")),
+        ("VOLUMES", str(report.volumes_total)),
+    ]
+    right: list[tuple[str, str]] = [
+        ("JOBS", _fraction(report.jobs_running, report.jobs_total, suffix="running")),
+        ("ALLOCS", _allocs_value(report)),
+        ("DEPLOYS", _count(len(report.deployments_active), "active", style="cyan")),
+        ("EVALS", _count(len(report.evals_problem), "blocked", style="yellow")),
+    ]
+
+    # Left-aligned label and value columns form four vertical rails the eye can follow; the
+    # dim divider stacks into a continuous hairline that splits infrastructure from workload.
+    grid = Table.grid(padding=(0, 2))
+    grid.add_column(justify="left", style="dim")  # left label
+    grid.add_column(justify="left")  # left value
+    grid.add_column(justify="left", style="dim")  # divider
+    grid.add_column(justify="left", style="dim")  # right label
+    grid.add_column(justify="left")  # right value
+    for (l_label, l_value), (r_label, r_value) in zip(left, right, strict=True):
+        grid.add_row(l_label, l_value, "│", r_label, r_value)
+
+    # The verdict lives in the title, right after the host, so the body is pure data.
+    verdict = f"[{style}]●[/] [bold {style}]{report.health.value}[/]"
+    title = f"{_banner_title(report)}  ·  {verdict}"
+    return Panel(grid, title=title, title_align="left", border_style=style, expand=False)
 
 
 def _nodes_panel(report: StatusReport) -> Panel:

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/commands/status/report.py

@@ -25,6 +25,9 @@ if TYPE_CHECKING:
 
 # Allocation client statuses that represent live work (counted in the per-node/per-job columns).
 _ACTIVE_ALLOC_STATUSES = frozenset({"running", "pending"})
+# Desired statuses Nomad assigns to allocations it has retired (rescheduled, drained, superseded).
+# Their client_status is historical, so they must not count toward current cluster health.
+_RETIRED_DESIRED_STATUSES = frozenset({"stop", "evict"})
 # Deployment statuses that represent an in-progress (notable) rollout.
 _ACTIVE_DEPLOYMENT_STATUSES = frozenset({"running", "pending", "blocked", "paused", "unblocking"})
 # Evaluation statuses that indicate the scheduler is stuck.
@@ -137,9 +140,12 @@ def build_report(  # noqa: PLR0913
         key=lambda e: e.job_id,
     )
 
-    allocs_failed = sum(1 for a in allocs if a.client_status == "failed")
-    allocs_pending = sum(1 for a in allocs if a.client_status == "pending")
-    allocs_unhealthy = any(a.client_status not in HEALTHY_ALLOC_STATUSES for a in allocs)
+    # Only allocs Nomad still wants running reflect current health; a "failed" corpse that has
+    # already been rescheduled carries desired_status "stop"/"evict" and must be ignored.
+    live_allocs = [a for a in allocs if a.desired_status not in _RETIRED_DESIRED_STATUSES]
+    allocs_failed = sum(1 for a in live_allocs if a.client_status == "failed")
+    allocs_pending = sum(1 for a in live_allocs if a.client_status == "pending")
+    allocs_unhealthy = any(a.client_status not in HEALTHY_ALLOC_STATUSES for a in live_allocs)
     node_alloc_counts = Counter(
         a.node_id for a in allocs if a.client_status in _ACTIVE_ALLOC_STATUSES
     )

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/commands/stop.py

@@ -19,7 +19,7 @@ from nd.constants import (
     TERMINAL_ALLOC_STATUSES,
 )
 from nd.nomad import NomadClient, NomadConfig
-from nd.nomad.errors import NomadError
+from nd.nomad.errors import NomadDecodeError, NomadError
 from nd.targets import resolve_targets, select_candidates
 from nd.ui.alloc_rows import alloc_children
 from nd.ui.duration import summary_title
@@ -53,14 +53,15 @@ class StopOutcome:
     detail: str = ""
 
 
-# Maps each terminal stop status to its outcome glyph and row label.
+# Maps each terminal stop status to its outcome glyph and row label. Each label
+# carries its glyph's color so the status word reads as success/failure on its own.
 _OUTCOME_ROW: dict[StopStatus, tuple[str, str]] = {
-    StopStatus.STOPPED: (OUTCOME_GLYPH["ok"], "stopped"),
-    StopStatus.TIMEOUT: (OUTCOME_GLYPH["warn"], "still draining"),
-    StopStatus.FAILED: (OUTCOME_GLYPH["fail"], "failed"),
+    StopStatus.STOPPED: (OUTCOME_GLYPH["ok"], "[green]stopped[/]"),
+    StopStatus.TIMEOUT: (OUTCOME_GLYPH["warn"], "[yellow]still draining[/]"),
+    StopStatus.FAILED: (OUTCOME_GLYPH["fail"], "[red]failed[/]"),
     # The workload did stop; only the follow-up garbage-collection failed, so this
     # warns rather than reading as a hard "failed to stop".
-    StopStatus.PURGE_FAILED: (OUTCOME_GLYPH["warn"], "stopped, purge failed"),
+    StopStatus.PURGE_FAILED: (OUTCOME_GLYPH["warn"], "[yellow]stopped, purge failed[/]"),
 }
 
 
@@ -153,21 +154,29 @@ async def stop_and_wait(
 
         deadline = time.monotonic() + STOP_TIMEOUT_SECONDS
         while True:
-            start = time.perf_counter()
-            allocs = await client.jobs.allocations(job.id)
-            pending = sum(1 for a in allocs if a.client_status not in TERMINAL_ALLOC_STATUSES)
-            elapsed_ms = (time.perf_counter() - start) * 1000
-            pp.trace(
-                f"GET /v1/job/{job.id}/allocations -> {len(allocs)} allocs "
-                f"({pending} not terminal), {elapsed_ms:.0f}ms"
-            )
-            if all_allocs_terminal(allocs):
-                if purge:
-                    return await _purge_dead_job(client, job, update=update)
-                return StopOutcome(job, StopStatus.STOPPED)
+            try:
+                start = time.perf_counter()
+                allocs = await client.jobs.allocations(job.id)
+            except NomadDecodeError as exc:
+                # A post-stop/cleanup task that just (re)started can momentarily
+                # serialize in a shape we cannot decode; skip this tick and retry
+                # rather than reporting the stop as failed. The deadline below is
+                # the backstop if it never recovers.
+                pp.debug(f"{job.id}: skipping drain poll after transient decode error: {exc}")
+            else:
+                pending = sum(1 for a in allocs if a.client_status not in TERMINAL_ALLOC_STATUSES)
+                elapsed_ms = (time.perf_counter() - start) * 1000
+                pp.trace(
+                    f"GET /v1/job/{job.id}/allocations -> {len(allocs)} allocs "
+                    f"({pending} not terminal), {elapsed_ms:.0f}ms"
+                )
+                if all_allocs_terminal(allocs):
+                    if purge:
+                        return await _purge_dead_job(client, job, update=update)
+                    return StopOutcome(job, StopStatus.STOPPED)
+                update(phase_text(allocs), alloc_children(allocs, node_names, None))
             if time.monotonic() >= deadline:
                 return StopOutcome(job, StopStatus.TIMEOUT, "stop requested, still draining")
-            update(phase_text(allocs), alloc_children(allocs, node_names, None))
             await asyncio.sleep(POLL_INTERVAL_SECONDS)
     except NomadError as exc:
         return StopOutcome(job, StopStatus.FAILED, str(exc))

nomadctl-0.3.0/src/nd/commands/update.py

@@ -0,0 +1,345 @@
+"""The ``nd update`` command: recreate a running job from its local job file."""
+
+from __future__ import annotations
+
+import asyncio
+import enum
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Annotated
+
+import typer
+from nclutils import pp
+
+from nd.binary import NomadBinary, NomadBinaryError
+from nd.commands._common import VerboseOption, configure_verbosity
+from nd.commands.run import DeployStatus, task_lifecycle, watch_deploy
+from nd.commands.stop import StopStatus, stop_and_wait
+from nd.jobfiles import candidates_for, discover_job_files, load_job_directories
+from nd.nomad import NomadClient, NomadConfig
+from nd.nomad.errors import NomadError
+from nd.targets import resolve_targets, select_candidates
+from nd.ui.duration import summary_title
+from nd.ui.live_panel import PanelUpdate, run_rows
+from nd.ui.prompts import select_one
+from nd.ui.styles import OUTCOME_GLYPH
+
+if TYPE_CHECKING:
+    from nd.jobfiles import JobFile
+    from nd.nomad.models.job import JobListStub
+
+
+@dataclass(frozen=True)
+class UpdateTarget:
+    """A running job paired with the local file that declares it.
+
+    Carries both halves the recreate needs: the ``file`` to compile and register,
+    and the running ``job`` stub to stop and drain.
+    """
+
+    name: str
+    file: JobFile
+    job: JobListStub
+
+
+def build_update_targets(files: list[JobFile], running: list[JobListStub]) -> list[UpdateTarget]:
+    """Pair each declared job that is currently running with its local file.
+
+    A job running in the cluster with no local file cannot be updated (there is no
+    spec to re-register), and a local job that is not running is a ``run``, not an
+    update, so both are omitted.
+
+    Args:
+        files: Discovered local job files and the names each declares.
+        running: The jobs currently running in the cluster.
+
+    Returns:
+        One target per declared job name that is also running.
+    """
+    by_name = {job.name: job for job in running}
+    targets: list[UpdateTarget] = []
+    for candidate in candidates_for(files):
+        job = by_name.get(candidate.name)
+        if job is not None:
+            targets.append(UpdateTarget(name=candidate.name, file=candidate.file, job=job))
+    return targets
+
+
+class UpdateStatus(enum.StrEnum):
+    """The terminal outcome of recreating one job."""
+
+    UPDATED = "updated"
+    FAILED = "failed"
+    TIMEOUT = "timeout"
+
+
+@dataclass(frozen=True)
+class UpdateOutcome:
+    """The result of recreating one job, ready for summary rendering."""
+
+    name: str
+    status: UpdateStatus
+    detail: str = ""
+    warnings: str = ""
+
+
+# Outcome labels carry their glyph's color so the status word reads on its own.
+_OUTCOME_ROW: dict[UpdateStatus, tuple[str, str]] = {
+    UpdateStatus.UPDATED: (OUTCOME_GLYPH["ok"], "[green]updated[/]"),
+    UpdateStatus.FAILED: (OUTCOME_GLYPH["fail"], "[red]failed[/]"),
+    UpdateStatus.TIMEOUT: (OUTCOME_GLYPH["warn"], "[yellow]still deploying[/]"),
+}
+
+# Map the deploy watch's terminal status onto the update outcome.
+_DEPLOY_TO_UPDATE: dict[DeployStatus, UpdateStatus] = {
+    DeployStatus.DEPLOYED: UpdateStatus.UPDATED,
+    DeployStatus.FAILED: UpdateStatus.FAILED,
+    DeployStatus.TIMEOUT: UpdateStatus.TIMEOUT,
+}
+
+# Stop outcomes that mean the workload reached a terminal state, so the recreate
+# may safely re-register. A drain that timed out or errored must not be re-run.
+_STOP_PROCEED = frozenset({StopStatus.STOPPED, StopStatus.PURGE_FAILED})
+
+
+async def _update_one(
+    client: NomadClient,
+    target: UpdateTarget,
+    *,
+    node_names: dict[str, str],
+    update: PanelUpdate,
+    nomad: NomadBinary,
+    purge: bool,
+) -> UpdateOutcome:
+    """Recreate one running job: compile, stop, re-register, and watch the rollout.
+
+    The new spec is compiled (and thereby parsed) before the running job is stopped,
+    so a bad file never tears down a healthy job. A drain that does not reach a
+    terminal state aborts the recreate without re-registering; a re-register that
+    fails after a clean stop is reported as the job being left down. Never raises:
+    Nomad/binary failures become a terminal outcome so a sibling job is unaffected.
+
+    Args:
+        client: Authenticated Nomad client.
+        target: The running job paired with its local file.
+        node_names: Map of node ID to node name for the per-allocation detail rows.
+        update: Callback to update the live panel phase text and detail rows.
+        nomad: Configured ``nomad`` binary handle for the compile step.
+        purge: Whether to garbage-collect the job after it drains.
+
+    Returns:
+        The terminal outcome for this target.
+    """
+    try:
+        update("compiling")
+        # compile_to_json shells out to the nomad binary (blocking); run it off the
+        # event loop so sibling recreates keep making progress concurrently.
+        body = await asyncio.to_thread(nomad.compile_to_json, target.file.path)
+        lifecycle = task_lifecycle(body)
+    except NomadBinaryError as exc:
+        return UpdateOutcome(target.name, UpdateStatus.FAILED, f"compile failed: {exc}")
+
+    # Stop and watch the drain (and optional purge) before touching the new version.
+    stop_outcome = await stop_and_wait(
+        client, target.job, purge=purge, node_names=node_names, update=update
+    )
+    if stop_outcome.status not in _STOP_PROCEED:
+        # The job is still draining or the stop errored; leaving it untouched is safer
+        # than re-registering on top of a job that has not come down.
+        status = (
+            UpdateStatus.TIMEOUT
+            if stop_outcome.status is StopStatus.TIMEOUT
+            else UpdateStatus.FAILED
+        )
+        return UpdateOutcome(
+            target.name,
+            status,
+            f"not re-deployed (stop {stop_outcome.status}): {stop_outcome.detail}",
+        )
+
+    try:
+        update("registering")
+        resp = await client.jobs.register(body)
+    except NomadError as exc:
+        # The old version is already gone here, so call out that the job is down.
+        return UpdateOutcome(
+            target.name, UpdateStatus.FAILED, f"stopped but re-deploy failed: {exc}"
+        )
+
+    try:
+        outcome = await watch_deploy(
+            client,
+            target.name,
+            node_names=node_names,
+            lifecycle=lifecycle,
+            update=update,
+            since_index=resp.job_modify_index,
+        )
+    except NomadError as exc:
+        # watch_deploy polls the cluster, so a transient error here would otherwise
+        # escape and crash the concurrent panel, killing every sibling job's watch.
+        # The job is already registered, so report the watch failure rather than raise.
+        return UpdateOutcome(
+            target.name, UpdateStatus.FAILED, f"re-registered but watch failed: {exc}"
+        )
+    return UpdateOutcome(
+        target.name, _DEPLOY_TO_UPDATE[outcome.status], outcome.detail, warnings=resp.warnings
+    )
+
+
+# allow_interspersed_args lets options follow the positional JOB argument; Typer
+# groups disable that by default, which would parse flags as subcommands.
+app = typer.Typer(context_settings={"allow_interspersed_args": True})
+
+
+@app.callback(invoke_without_command=True)
+def update(
+    ctx: typer.Context,
+    job: Annotated[
+        str | None,
+        typer.Argument(
+            help="Running job to update; matches any running job whose name starts "
+            "with this and has a local file. Omit to pick from a list."
+        ),
+    ] = None,
+    no_purge: Annotated[  # noqa: FBT002
+        bool,
+        typer.Option(
+            "--no-purge",
+            help="Stop without garbage-collecting the job (keeps its version history).",
+        ),
+    ] = False,
+    force: Annotated[  # noqa: FBT002
+        bool,
+        typer.Option("--force", "-f", help="Skip the confirmation prompt."),
+    ] = False,
+    dry_run: Annotated[  # noqa: FBT002
+        bool,
+        typer.Option("--dry-run", "-n", help="Resolve and validate without recreating."),
+    ] = False,
+    verbose: VerboseOption = 0,
+) -> None:
+    """Recreate one or more running jobs from their local job files.
+
+    Each selected job is compiled and validated, then stopped, drained, purged (unless
+    --no-purge), re-registered, and watched to a terminal deploy state. Use this to
+    roll out a changed job file or to force a fresh version (e.g. re-pull a docker
+    image); whether an image is actually re-pulled depends on the job's docker driver
+    config (force_pull), not on nd.
+    """
+    configure_verbosity(ctx, verbose)
+    exit_code = asyncio.run(_run(job_arg=job, no_purge=no_purge, force=force, dry_run=dry_run))
+    if exit_code != 0:
+        raise typer.Exit(exit_code)
+
+
+async def _run(*, job_arg: str | None, no_purge: bool, force: bool, dry_run: bool) -> int:  # noqa: PLR0911
+    """Resolve running targets with local files, confirm, then recreate them.
+
+    Returns the exit code: 0 on clean success, 1 on any failure. The new spec for
+    every target is validated up front, before any job is stopped.
+    """
+    files = discover_job_files(load_job_directories())
+    config = NomadConfig.resolve()
+    async with NomadClient.from_config(config) as client:
+        jobs = await client.jobs.list()
+        running = [j for j in jobs if j.status == "running"]
+        targets_all = build_update_targets(files, running)
+        if not targets_all:
+            pp.info("No running jobs have a local job file to update.")
+            return 0
+
+        resolution = resolve_targets(targets_all, job_arg, name_of=lambda t: t.name)
+        targets = await select_candidates(
+            resolution,
+            "Select jobs to update",
+            label_of=lambda t: f"{t.name}  [{t.file.path.name}]",
+        )
+        if targets is None:
+            return 0
+        if not targets:
+            pp.error(f"No running job with a local file matching '{job_arg}'")
+            return 1
+
+        purge = not no_purge
+        if not force and not await _confirm(targets, purge=purge):
+            pp.info("Aborted")
+            return 0
+
+        try:
+            nomad = NomadBinary.create(config)
+            # dict.fromkeys dedups so a multi-job file is validated once.
+            for path in dict.fromkeys(t.file.path for t in targets):
+                nomad.validate(path)
+        except NomadBinaryError as exc:
+            pp.error(str(exc))
+            return 1
+
+        if dry_run:
+            for t in targets:
+                pp.dryrun(f"would recreate {t.name} ({t.file.path})")
+            return 0
+
+        outcomes = await _update_all(client, targets, nomad, purge=purge)
+
+    return 0 if all(o.status is UpdateStatus.UPDATED for o in outcomes) else 1
+
+
+async def _confirm(targets: list[UpdateTarget], *, purge: bool) -> bool:
+    """Ask the user to confirm recreating the resolved jobs."""
+    names = ", ".join(t.name for t in targets)
+    verb = "Stop, PURGE and re-deploy" if purge else "Stop and re-deploy"
+    answer = await select_one(
+        [("Yes", True), ("No", False)],
+        f"{verb} {len(targets)} job(s): {names}?",
+    )
+    return bool(answer)
+
+
+async def _update_all(
+    client: NomadClient, targets: list[UpdateTarget], nomad: NomadBinary, *, purge: bool
+) -> list[UpdateOutcome]:
+    """Recreate every target concurrently under one live panel.
+
+    Args:
+        client: Authenticated Nomad client.
+        targets: The running jobs to recreate.
+        nomad: Configured ``nomad`` binary handle for the compile step.
+        purge: Whether to garbage-collect each job after it drains.
+
+    Returns:
+        Ordered list of outcomes, one per target.
+    """
+    # Resolve node IDs to names once so every job's detail rows can show placement.
+    node_names = {node.id: node.name for node in await client.nodes.list()}
+
+    async def do_work(target: UpdateTarget, update: PanelUpdate) -> UpdateOutcome:
+        return await _update_one(
+            client, target, node_names=node_names, update=update, nomad=nomad, purge=purge
+        )
+
+    ordered = await run_rows(
+        targets,
+        do_work,
+        label_of=lambda t: t.name,
+        initial_phase="compiling",
+        finish_of=lambda o: _OUTCOME_ROW[o.status],
+        running_title=f"Updating {len(targets)} job(s)",
+        final_title=_final_title,
+    )
+
+    # The live panel is transient on a pipe/CI; emit a durable line for anything that
+    # did not update cleanly so timeouts and a left-down job are never silent.
+    for o in ordered:
+        if o.status is UpdateStatus.TIMEOUT:
+            pp.warning(f"{o.name}: {o.detail or 'still deploying'}")
+        elif o.status is UpdateStatus.FAILED:
+            pp.error(f"{o.name} failed to update", details=[o.detail] if o.detail else None)
+        if o.warnings:
+            pp.warning(f"{o.name}: {o.warnings}")
+    return ordered
+
+
+def _final_title(outcomes: list[UpdateOutcome], elapsed_seconds: float) -> str:
+    """Build the final panel title with updated totals and elapsed seconds."""
+    ok = sum(1 for o in outcomes if o.status is UpdateStatus.UPDATED)
+    return summary_title("Updated", ok, len(outcomes), elapsed_seconds)

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/commands/volume/render.py

@@ -5,16 +5,36 @@ from __future__ import annotations
 from typing import TYPE_CHECKING
 
 from nclutils import pp
+from rich.console import Group
 from rich.tree import Tree
 
 from nd.commands.volume.report import ALREADY_REGISTERED_REASON
 from nd.ui.panels import status_table, titled_panel
 
 if TYPE_CHECKING:
+    from rich.console import RenderableType
+
     from nd.commands.volume.report import Registration, VolumeRow
     from nd.nomad.models.volume import HostVolumeListStub
 
 
+def _print_volume_groups(title: str, trees: list[Tree]) -> None:
+    """Print per-volume trees stacked in one titled panel, blank-line separated.
+
+    Mirrors the framed result panels of ``nd run``/``nd stop`` so every command
+    speaks the same visual language, and the blank line between groups gives each
+    volume room to breathe instead of crowding the next one.
+    """
+    if not trees:
+        return
+    body: list[RenderableType] = []
+    for index, tree in enumerate(trees):
+        if index:
+            body.append("")  # blank line so adjacent volume groups stay distinct
+        body.append(tree)
+    pp.console().print(titled_panel(Group(*body), title))
+
+
 def render_list(rows: list[VolumeRow]) -> None:
     """Print the host-volume table inside a titled panel with plain node names.
 
@@ -77,11 +97,17 @@ def render_registration_results(
     for reg, outcome in results:
         groups.setdefault(reg.spec.name, []).append((reg, outcome))
 
+    trees: list[Tree] = []
     for name, items in groups.items():
         tree = Tree(f"[bold]{name}[/]")
         for reg, outcome in items:
             tree.add(_registration_leaf(reg, outcome))
-        pp.console().print(tree)
+        trees.append(tree)
+
+    # A dry-run never actually registers, so the title must not claim it did.
+    verb = "Would register" if any(outcome == "dryrun" for _, outcome in results) else "Registered"
+    count = len(groups)
+    _print_volume_groups(f"{verb} {count} volume{'s' if count != 1 else ''}", trees)
 
 
 def _deletion_leaf(vol: HostVolumeListStub, outcome: str, node_names: dict[str, str]) -> str:
@@ -125,8 +151,14 @@ def render_deletion_results(
     for vol, outcome in results:
         groups.setdefault(vol.name, []).append((vol, outcome))
 
+    trees: list[Tree] = []
     for name, items in groups.items():
         tree = Tree(f"[bold]{name}[/]")
         for vol, outcome in items:
             tree.add(_deletion_leaf(vol, outcome, node_names))
-        pp.console().print(tree)
+        trees.append(tree)
+
+    # A dry-run never actually deletes, so the title must not claim it did.
+    verb = "Would delete" if any(outcome == "would-delete" for _, outcome in results) else "Deleted"
+    count = len(groups)
+    _print_volume_groups(f"{verb} {count} volume{'s' if count != 1 else ''}", trees)

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/nomad/models/allocation.py

@@ -24,10 +24,17 @@ class AllocListStub(msgspec.Struct, rename="pascal", frozen=True, kw_only=True):
     task_group: str
     client_status: str
     desired_status: str
-    task_states: dict[str, TaskState] = msgspec.field(name="TaskStates", default_factory=dict)
+    # Nomad sends TaskStates: null (not an empty object) for a freshly-placed
+    # allocation whose tasks have not started yet, so this must tolerate null.
+    task_states_raw: dict[str, TaskState] | None = msgspec.field(name="TaskStates", default=None)
     create_index: int
     modify_index: int
 
+    @property
+    def task_states(self) -> dict[str, TaskState]:
+        """Per-task run state, with Nomad's null (tasks not yet started) read as empty."""
+        return self.task_states_raw or {}
+
 
 class Allocation(AllocListStub, frozen=True, kw_only=True):
     """An allocation as returned by ``GET /v1/allocation/:id``.

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/nomad/resources/jobs.py

@@ -64,7 +64,7 @@ class JobsResource(BaseResource):
     async def deployments(self, job_id: str) -> builtins.list[DeploymentListStub]:
         """List a job's deployments (``GET /v1/job/:id/deployments``), following pagination.
 
-        Nomad returns the most recent deployment first, used to watch a freshly
-        registered service job roll out.
+        The endpoint's ordering is undocumented, so callers that need the most
+        recent deployment must select it by ``CreateIndex`` rather than position.
         """
         return await self._paginate_list(f"/job/{job_id}/deployments", DeploymentListStub)

{nomadctl-0.2.0 → nomadctl-0.3.0}/src/nd/ui/live_panel.py

@@ -13,12 +13,14 @@ from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Protocol
 
 from nclutils import pp
+from rich import box
 from rich.live import Live
 from rich.spinner import Spinner
 from rich.table import Table
 
 from nd.ui.duration import fmt_elapsed
 from nd.ui.panels import titled_panel
+from nd.ui.styles import OUTCOME_GLYPH
 
 if TYPE_CHECKING:
     from collections.abc import Awaitable, Callable, Sequence
@@ -68,30 +70,70 @@ def finish_row(
     row.ended_at = clock()
 
 
+def _last_siblings(children: list[LiveChild]) -> list[bool]:
+    """Flag whether each child is the last among its siblings at its own depth.
+
+    Lets the tree close a branch with ``└`` and continue it with ``├`` so the
+    indentation reads as a real tree instead of every row looking like a leaf.
+    A child is the last sibling when no later row shares its depth before the
+    indentation drops back out of its branch.
+    """
+    result = [True] * len(children)
+    for i, child in enumerate(children):
+        for nxt in children[i + 1 :]:
+            if nxt.depth < child.depth:
+                break  # left this branch; later rows belong to an outer level
+            if nxt.depth == child.depth:
+                result[i] = False
+                break
+    return result
+
+
 def _build_panel(rows: list[LiveRow], *, title: str, now: float) -> Panel:
     """Render the panel: a spinner for in-flight rows, a glyph for finished ones.
 
     Parent rows are bold and carry the spinner/glyph; child rows nest under them
     with a tree marker indented inside the label column so the text follows the
-    tree rather than staying flush left.
+    tree rather than staying flush left. A dim hairline separates each parent
+    group, and a parent's detail rows dim once it finishes healthy so the eye
+    stays on the rows still in flight.
     """
-    table = Table.grid(padding=(0, 2))
+    # A real table (not Table.grid) so add_section() can draw a dim hairline
+    # between groups; show_edge=False keeps the rule internal, collapse_padding
+    # reproduces the grid's tight column spacing.
+    table = Table(
+        box=box.HORIZONTALS,
+        show_header=False,
+        show_edge=False,
+        pad_edge=False,
+        collapse_padding=True,
+        padding=(0, 2),
+        border_style="dim",
+    )
     table.add_column()  # spinner / glyph (parent rows only)
     # no_wrap keeps the tree marker attached to its label on a narrow terminal
     # (the label truncates with an ellipsis instead of splitting across lines).
     table.add_column(no_wrap=True)  # label, tree-indented for children
     table.add_column()  # phase / role
     table.add_column(justify="right")  # elapsed / status
-    for row in rows:
+    for index, row in enumerate(rows):
+        if index:
+            table.add_section()  # dim hairline giving each group room to breathe
         glyph = Spinner("dots") if row.glyph is None else row.glyph
         ended = row.ended_at if row.ended_at is not None else now
         table.add_row(
             glyph, f"[bold]{row.label}[/]", row.phase, fmt_elapsed(ended - row.started_at)
         )
-        for child in row.children:
+        # A healthy finish lets the detail rows recede; failures stay bright so
+        # the tasks remain readable while debugging.
+        recede = row.glyph == OUTCOME_GLYPH["ok"]
+        for child, is_last in zip(row.children, _last_siblings(row.children), strict=True):
             cells = [*child.cells, "", "", ""][:3]  # pad/truncate to the 3 detail columns
-            label = f"[dim]{'  ' * child.depth}└[/] {cells[0]}"
-            table.add_row("", label, cells[1], cells[2])
+            connector = "└" if is_last else "├"
+            detail = (f"[dim]{'  ' * child.depth}{connector}[/] {cells[0]}", cells[1], cells[2])
+            if recede:
+                detail = tuple(f"[dim]{cell}[/]" for cell in detail)
+            table.add_row("", *detail)
     return titled_panel(table, title)