dockhand-cli 0.3.1__tar.gz → 0.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. dockhand_cli-0.4.0/.github/workflows/publish.yml +16 -0
  2. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/AGENTS.md +33 -18
  3. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/PKG-INFO +22 -6
  4. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/README.md +21 -5
  5. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/__init__.py +1 -1
  6. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/pyproject.toml +1 -1
  7. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/uv.lock +2 -2
  8. dockhand_cli-0.3.1/.claude/settings.local.json +0 -9
  9. dockhand_cli-0.3.1/CLAUDE.md +0 -309
  10. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/.dockhand.json +0 -0
  11. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/.gitignore +0 -0
  12. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/LICENSE +0 -0
  13. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/build.py +0 -0
  14. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/client/__init__.py +0 -0
  15. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/client/base.py +0 -0
  16. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/client/local.py +0 -0
  17. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/client/ssh.py +0 -0
  18. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/config.py +0 -0
  19. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/constants.py +0 -0
  20. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/download.py +0 -0
  21. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/error.py +0 -0
  22. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/history.py +0 -0
  23. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/manage.py +0 -0
  24. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/queue.py +0 -0
  25. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/resubmit.py +0 -0
  26. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/submit.py +0 -0
  27. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/sync.py +0 -0
  28. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/tagging.py +0 -0
  29. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/transport.py +0 -0
  30. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/tunnel.py +0 -0
  31. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/dockhand/volumes.py +0 -0
  32. {dockhand_cli-0.3.1 → dockhand_cli-0.4.0}/media/Gemini_Generated_Image_i700fvi700fvi700.png +0 -0
@@ -0,0 +1,16 @@
1
+ name: Publish to PyPI
2
+
3
+ on:
4
+ release:
5
+ types: [published]
6
+
7
+ jobs:
8
+ publish:
9
+ runs-on: ubuntu-latest
10
+ permissions:
11
+ id-token: write # required for PyPI trusted publishing (OIDC) - no token needed
12
+ steps:
13
+ - uses: actions/checkout@v4
14
+ - uses: astral-sh/setup-uv@v3
15
+ - run: uv build
16
+ - run: uv publish
@@ -42,6 +42,12 @@ uv run ruff format .
42
42
  uv run ruff check --fix .
43
43
  ```
44
44
 
45
+ **Build / release:**
46
+ The PyPI distribution is named `dockhand-cli` (the `dockhand` name is taken); the installed command and import package are still `dockhand`. Version lives in `pyproject.toml`. Publishing a GitHub release triggers `.github/workflows/publish.yml`, which runs `uv build && uv publish` via PyPI trusted publishing (OIDC, no API token). Pushing to `main` alone publishes nothing.
47
+ ```bash
48
+ uv build # local sanity check; artifacts land in dist/
49
+ ```
50
+
45
51
  ## Architecture
46
52
 
47
53
  ### Entry Point
@@ -52,6 +58,7 @@ uv run ruff check --fix .
52
58
  - `dockhand run` — queue/run from an already-built image, no sync
53
59
  - `dockhand install` — build the image only
54
60
  - `dockhand logs`, `stop`, `remove`, `jobs`, `urgent`, `prune` — job/queue lifecycle (`manage.py` + `queue.py`)
61
+ - `jobs` shows this project's last 30 active jobs; `--all` lifts the cap and includes finished/failed/stopped; `--queue` shows the host's whole task-spooler queue, including other projects' jobs
55
62
  - `dockhand history` — show past runs
56
63
  - `dockhand volumes`, `download` — inspect and pull files from mounted volumes
57
64
  - `dockhand resubmit` — rerun a past job with optional overrides
@@ -76,10 +83,10 @@ Config file: `.dockhand.json` in the project root (must also contain a `.git`
76
83
  ### Key Modules
77
84
  - `submit.py` — Builds the `docker run` command (code mount vs. data volumes vs. GPU/port flags), resolves code delivery, and hands off to a transport to start the job.
78
85
  - `build.py` — Runs `docker build` on the client (optionally syncing first).
79
- - `manage.py` — Job lifecycle: `logs`, `stop`, `remove`, `jobs` (`execute_stats`), and `prune` (removes baked images no longer referenced by an active job).
86
+ - `manage.py` — Job lifecycle: `logs`, `stop`, `remove`, `jobs` (`execute_stats`), `jobs --queue` (`execute_queue`), and `prune` (removes baked images no longer referenced by an active job). Also owns job timing (`_observe_job_time`, `_docker_started_at`, `_host_now`).
80
87
  - `resubmit.py` — Looks up a history entry and re-invokes `execute_submit` with overrides; pins to the original baked image when applicable.
81
- - `queue.py` — Task spooler (`tsp`) integration: submit/list/promote/remove/kill, plus `ts -l` output parsing.
82
- - `transport.py` — Abstracts "how a job runs": `TaskSpoolerTransport` (queue enabled) vs. `DockerTransport` (direct `docker run -d`). Both expose the same interface (`submit`, `list_jobs`, `logs`, `stop`, `remove`) so job-management commands don't care which backend created a job; the transport used is recorded per job in history.
88
+ - `queue.py` — Task spooler (`tsp`) integration: submit/list/promote/remove/kill, `tsp -l` output parsing, max slot count (`ts_max_slots`), and start times from `tsp -i` (`ts_start_times`).
89
+ - `transport.py` — Abstracts "how a job runs": `TaskSpoolerTransport` (queue enabled) vs. `DockerTransport` (direct `docker run -d`). Both expose the same interface (`run_flags`, `submit`, `list_jobs`, `container_name`, `logs`, `stop`, `remove`) so job-management commands don't care which backend created a job; the transport used is recorded per job in history. Containers are named `dockhand-<local_id>` in both modes (queued: `--rm --name`; direct: `-d --name`, no `--rm` so logs survive exit).
83
90
  - `tagging.py` — Resolves the image tag/ref for baked code delivery (content-addressed from git commit + dirty-state hash; unique per queued submit, reused for direct runs).
84
91
  - `history.py` — Reads/writes `.dockhand_history.json`, reserves/looks up local job IDs.
85
92
  - `volumes.py` — Lists the container filesystem as a tree (code mount + data volumes) and resolves a workdir-relative path back to its host path.
@@ -111,28 +118,36 @@ Stores container runs in `.dockhand_history.json` as JSON. Each entry contains:
111
118
  "handle": 12, // tsp job id, or container name for direct runs
112
119
  "ts_job_id": 12, // task_spooler transport only
113
120
  "host": "remote.example.com", // or "localhost"
114
- "started_at": 1234567891.0, // optional, set the first time `jobs` observes it running
115
- "ended_at": 1234567895.0 // optional, set the first time `jobs` observes it finished/failed/stopped
121
+ "started_at": 1234567891.0, // optional, set when `jobs`/`logs` first sees it running
122
+ "ended_at": 1234567895.0, // optional, set when `jobs`/`logs` first sees it finished/failed/stopped
123
+ "stopped": true // optional, set by `dockhand stop`; shows finished/failed as "stopped"
116
124
  }
117
125
  ```
118
126
 
119
127
  Used by `resubmit` (look up a previous run and re-run with overrides, pinning the original baked image when unchanged), `logs`, `stop`, `remove`, `jobs`, `urgent`, `prune` (default to latest if no ID given). Job-management commands dispatch to the transport recorded on the entry (`transport_for_entry`), so `logs`/`stop`/`remove` work the same regardless of whether a job went through the queue or ran directly.
120
128
 
121
- `started_at`/`ended_at` aren't queried from tsp/docker (neither exposes exact start/end timestamps cheaply for both transports) — `jobs` and `logs` each stamp them lazily the first time they happen to observe a job in the running/terminal state, so a job never checked on while running will show no start time once it finishes. `dockhand jobs` displays these as Started/Ended (absolute, `%Y-%m-%d %H:%M:%S`) and Duration (elapsed while running, total once finished) columns; `dockhand logs` prints a one-line "running for Xm" / "finished in Xm" header before the log output.
129
+ `started_at`/`ended_at` are written the first time `jobs` or `logs` observes a job in the running/terminal state (`_observe_job_time`), but the values come from docker/ts, not the observation time:
130
+ - `started_at` — the container's real `docker inspect .State.StartedAt` (found via the `dockhand-<local_id>` name). Falls back to "now" only if the container can't be inspected (pre-naming jobs, already removed).
131
+ - `ended_at` — `started_at + duration_seconds` from ts's own elapsed-time field when available, else "now".
132
+ - Running durations are measured against the host clock (`_host_now` runs `date +%s` on the host) so local/host clock skew doesn't distort them.
133
+
134
+ `dockhand jobs` shows Started/Ended (absolute, `%Y-%m-%d %H:%M:%S`) and Duration columns, grouped running → queued → finished/failed/stopped, newest ID first within each group. `dockhand logs` prints a one-line "running for Xm" / "finished in Xm" header before the log output.
135
+
136
+ `dockhand jobs --queue` bypasses history entirely: it prints the raw host-wide `tsp -l` queue (ts is one shared queue per host), with each job's dockhand ID parsed from its `dockhand-<n>` container name, a project column from the image name, start times from `tsp -i`, and a "N running / M slots, K queued" footer.
122
137
 
123
138
  ### Design Patterns
124
139
 
125
140
  **DockerDefault factory:**
126
- In `__init__.py`, `DockerDefault` is a callable factory that pulls defaults from `cli_config.docker` at call time. Enables Typer's `default_factory` to respect active profiles.
141
+ In `__init__.py`, `DockerDefault` is a callable factory that pulls defaults from `cli_config.docker` at call time (after the `--profile` callback has run), for use with Typer's `default_factory`. `SyncDefault` does the same for `cli_config.sync`.
127
142
 
128
143
  **Transport abstraction:**
129
- `get_transport()` in `transport.py` picks `TaskSpoolerTransport` or `DockerTransport` based on `cli_config.queue.enabled` at submit time. Every job-management command re-derives the correct transport per-job from the history entry (`transport_for_entry`) rather than from current config, so switching `queue.enabled` doesn't strand in-flight jobs from the other mode.
144
+ `get_transport()` in `transport.py` picks `TaskSpoolerTransport` or `DockerTransport` based on `cli_config.queue.enabled` at submit time. Per-job commands (`logs`, `stop`, `remove`, `urgent`) re-derive the transport from the history entry (`transport_for_entry`; entries without a `transport` field are treated as task spooler), and `logs`/`stop`/`remove` also connect to the entry's recorded `host`. The exception is `jobs`, which lists via the *current* config's transport and host.
130
145
 
131
146
  **Code delivery resolution:**
132
147
  `DockerConfig.resolve_code_delivery(queue_enabled)` picks `mount` or `bake` when `code_delivery` isn't explicitly set: `bake` when the queue is enabled (avoids code drift while a job waits in queue), `mount` otherwise (zero-rebuild iteration). See README's [Code delivery](README.md#code-delivery-mount-vs-bake) section for the full rationale.
133
148
 
134
149
  **Profile support:**
135
- `cli_config.load_profile(name)` loads overrides from `.dockhand.json` profiles section (currently merges `history_path`, `remote_path`, `ssh`). All config classes support `validate()` for safe merging.
150
+ `cli_config.load_profile(name)` loads overrides from `.dockhand.json` profiles section. It only merges `history_path`, `remote_path` and `ssh` — `docker`, `queue` and `sync` keys in a profile are silently ignored. `SSHConfig.validate()` is used for safe merging of partial ssh overrides.
136
151
 
137
152
  **Command structure:**
138
153
  Each command:
@@ -155,7 +170,7 @@ Each command:
155
170
  | `__init__.py` | CLI app definition, command routing, config defaults |
156
171
  | `submit.py` | Builds the `docker run` command, resolves code delivery, submits via transport |
157
172
  | `build.py` | `docker build` execution |
158
- | `manage.py` | `logs`, `stop`, `remove`, `jobs`, `prune` |
173
+ | `manage.py` | `logs`, `stop`, `remove`, `jobs`, `jobs --queue`, `prune`, job timing |
159
174
  | `resubmit.py` | Rerun a past job with overrides |
160
175
  | `queue.py` | Task spooler (`tsp`) integration |
161
176
  | `transport.py` | Task-spooler vs. direct-docker execution backends |
@@ -172,6 +187,7 @@ Each command:
172
187
  | `sync.py` | rsync-based code synchronization |
173
188
  | `error.py` | Error reporting |
174
189
  | `constants.py` | Config/history file names |
190
+ | `.github/workflows/publish.yml` | Publish `dockhand-cli` to PyPI on GitHub release |
175
191
 
176
192
  ## Configuration Format
177
193
 
@@ -207,11 +223,9 @@ Each command:
207
223
  },
208
224
  "remote_path": "~/my-project",
209
225
  "profiles": {
210
- "dev": {
211
- "docker": {
212
- "gpus": "1",
213
- "dockerfile": "Dockerfile.dev"
214
- }
226
+ "gpu2": {
227
+ "ssh": { "hostname": "gpu2.example.com" },
228
+ "remote_path": "~/my-project-gpu2"
215
229
  }
216
230
  }
217
231
  }
@@ -242,13 +256,14 @@ uv run dockhand run 'bash' # Run interactively
242
256
 
243
257
  **Using profiles:**
244
258
  ```bash
245
- uv run dockhand --profile dev submit 'python train.py' # Use dev profile
246
- uv run dockhand --profile prod install # Build prod image
259
+ uv run dockhand --profile gpu2 install # Build on the gpu2 host
260
+ uv run dockhand --profile gpu2 submit 'python train.py' # Run there
247
261
  ```
248
262
 
249
263
  **Queue management:**
250
264
  ```bash
251
265
  uv run dockhand jobs # List active jobs
266
+ uv run dockhand jobs --queue # Whole host queue, all projects, with slot usage
252
267
  uv run dockhand urgent 5 # Promote job #5 to front of queue
253
268
  uv run dockhand prune --dry-run # Preview unused baked images before removing
254
269
  ```
@@ -269,7 +284,7 @@ uv run ruff format --check . # Format check
269
284
  2. **SSH by default** — Assumes docker host is remote. Auto-detects local by resolving the configured hostname to a loopback address.
270
285
  3. **History in JSON** — Simple, human-readable, easily editable.
271
286
  4. **Profile support** — Allows per-project config variants.
272
- 5. **Minimal dependencies** — typer, fabric, paramiko, gitpython, rich.
287
+ 5. **Minimal dependencies** — typer, fabric, paramiko, gitpython (rich comes in via typer).
273
288
  6. **Transport abstraction over queue state** — job-management commands read the transport from each job's own history entry rather than current `queue.enabled`, so toggling the queue mid-flight doesn't orphan existing jobs.
274
289
  7. **Code-delivery default follows queue mode** — `bake` when queued (avoids code drift while waiting), `mount` when not (zero-rebuild iteration); explicit `code_delivery` overrides either way.
275
290
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: dockhand-cli
3
- Version: 0.3.1
3
+ Version: 0.4.0
4
4
  Summary: CLI for managing Docker containers on remote machines
5
5
  Project-URL: Repository, https://github.com/NicholasPHansen/dockhand
6
6
  Author: Nicholas P. Hansen
@@ -108,7 +108,7 @@ All commands work with the same `.dockhand.json` configuration file. See [Config
108
108
  | `submit` | Build the image and queue a container run |
109
109
  | `run` | Queue a container run from an already-built image |
110
110
  | `install` | Build the Docker image without running it |
111
- | `jobs` | List active (running/queued) jobs, with start/end/duration — use `--all` for finished jobs too |
111
+ | `jobs` | List active (running/queued) jobs, with start/end/duration — use `--all` for finished jobs too, or `--queue` for the host's whole task-spooler queue across all projects |
112
112
  | `logs` | Show logs from a job, preceded by a running/finished duration line — use `--follow`/`-f` to stream live |
113
113
  | `stop` | Stop a **running** job |
114
114
  | `remove` | Remove a **queued** job before it starts |
@@ -164,6 +164,9 @@ dockhand jobs
164
164
  # Check all jobs including finished
165
165
  dockhand jobs --all
166
166
 
167
+ # Show the host's whole task-spooler queue, including other projects' jobs
168
+ dockhand jobs --queue
169
+
167
170
  # Stream logs from the last job
168
171
  dockhand logs --follow
169
172
 
@@ -198,10 +201,23 @@ dockhand resubmit --gpus 2
198
201
  **Note:** If you've installed dockhand globally, you can omit `uv run`.
199
202
 
200
203
  `jobs` shows Started/Ended/Duration for each job, and `logs` prints a one-line
201
- `running for 5m30s` / `finished in 12m45s` header before the log output. Neither the
202
- queue nor docker exposes exact start/end timestamps cheaply, so these are recorded the
203
- first time `jobs` or `logs` happens to observe a job as running or finished — a job
204
- never checked on while running will show no duration once it's done.
204
+ `running for 5m30s` / `finished in 12m45s` header before the log output. Start times come
205
+ from the container's `docker inspect` and finished durations from task spooler itself, so
206
+ they're exact no matter when you check (older jobs from before container naming fall back
207
+ to the time they were first observed). A running job's elapsed time is measured against the
208
+ host's clock, so clock skew between your machine and the host doesn't distort it (keep the
209
+ host's clock NTP-synced, though — Started/Ended are shown in the host's time).
210
+
211
+ `jobs --queue` lists every job in the host's task-spooler queue, not just this project's:
212
+
213
+ | Column | Meaning |
214
+ |--------|---------|
215
+ | `ID` | dockhand job ID in the submitting project — what `logs`/`stop`/`urgent` take there (`-` for jobs submitted before dockhand named its containers) |
216
+ | `Project` | Image the job runs (the project's `imagename`) |
217
+ | `Status` / `Started` / `Duration` | From task spooler (`tsp -l`, `tsp -i`), measured on the host's clock |
218
+ | `TS ID` | Task spooler's own job ID, for use with `tsp` directly |
219
+
220
+ Running and queued jobs come first (queued in queue order), then the rest newest-started first.
205
221
 
206
222
  Resubmitting a job that ran in [`bake`](#code-delivery-mount-vs-bake) mode reruns the exact
207
223
  image it originally built — recorded per job — rather than rebuilding from current code, so a
@@ -91,7 +91,7 @@ All commands work with the same `.dockhand.json` configuration file. See [Config
91
91
  | `submit` | Build the image and queue a container run |
92
92
  | `run` | Queue a container run from an already-built image |
93
93
  | `install` | Build the Docker image without running it |
94
- | `jobs` | List active (running/queued) jobs, with start/end/duration — use `--all` for finished jobs too |
94
+ | `jobs` | List active (running/queued) jobs, with start/end/duration — use `--all` for finished jobs too, or `--queue` for the host's whole task-spooler queue across all projects |
95
95
  | `logs` | Show logs from a job, preceded by a running/finished duration line — use `--follow`/`-f` to stream live |
96
96
  | `stop` | Stop a **running** job |
97
97
  | `remove` | Remove a **queued** job before it starts |
@@ -147,6 +147,9 @@ dockhand jobs
147
147
  # Check all jobs including finished
148
148
  dockhand jobs --all
149
149
 
150
+ # Show the host's whole task-spooler queue, including other projects' jobs
151
+ dockhand jobs --queue
152
+
150
153
  # Stream logs from the last job
151
154
  dockhand logs --follow
152
155
 
@@ -181,10 +184,23 @@ dockhand resubmit --gpus 2
181
184
  **Note:** If you've installed dockhand globally, you can omit `uv run`.
182
185
 
183
186
  `jobs` shows Started/Ended/Duration for each job, and `logs` prints a one-line
184
- `running for 5m30s` / `finished in 12m45s` header before the log output. Neither the
185
- queue nor docker exposes exact start/end timestamps cheaply, so these are recorded the
186
- first time `jobs` or `logs` happens to observe a job as running or finished — a job
187
- never checked on while running will show no duration once it's done.
187
+ `running for 5m30s` / `finished in 12m45s` header before the log output. Start times come
188
+ from the container's `docker inspect` and finished durations from task spooler itself, so
189
+ they're exact no matter when you check (older jobs from before container naming fall back
190
+ to the time they were first observed). A running job's elapsed time is measured against the
191
+ host's clock, so clock skew between your machine and the host doesn't distort it (keep the
192
+ host's clock NTP-synced, though — Started/Ended are shown in the host's time).
193
+
194
+ `jobs --queue` lists every job in the host's task-spooler queue, not just this project's:
195
+
196
+ | Column | Meaning |
197
+ |--------|---------|
198
+ | `ID` | dockhand job ID in the submitting project — what `logs`/`stop`/`urgent` take there (`-` for jobs submitted before dockhand named its containers) |
199
+ | `Project` | Image the job runs (the project's `imagename`) |
200
+ | `Status` / `Started` / `Duration` | From task spooler (`tsp -l`, `tsp -i`), measured on the host's clock |
201
+ | `TS ID` | Task spooler's own job ID, for use with `tsp` directly |
202
+
203
+ Running and queued jobs come first (queued in queue order), then the rest newest-started first.
188
204
 
189
205
  Resubmitting a job that ran in [`bake`](#code-delivery-mount-vs-bake) mode reruns the exact
190
206
  image it originally built — recorded per job — rather than rebuilding from current code, so a
@@ -15,7 +15,7 @@ from dockhand.submit import execute_submit
15
15
  from dockhand.tunnel import execute_tunnel
16
16
  from dockhand.volumes import execute_volumes
17
17
 
18
- __version__ = "0.2.0"
18
+ __version__ = "0.4.0"
19
19
 
20
20
  cli = typer.Typer(pretty_exceptions_show_locals=False)
21
21
 
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "dockhand-cli"
7
- version = "0.3.1"
7
+ version = "0.4.0"
8
8
  description = "CLI for managing Docker containers on remote machines"
9
9
  readme = "README.md"
10
10
  license = { text = "MIT" }
@@ -266,8 +266,8 @@ wheels = [
266
266
  ]
267
267
 
268
268
  [[package]]
269
- name = "dockhand"
270
- version = "0.3.1"
269
+ name = "dockhand-cli"
270
+ version = "0.4.0"
271
271
  source = { editable = "." }
272
272
  dependencies = [
273
273
  { name = "fabric" },
@@ -1,9 +0,0 @@
1
- {
2
- "permissions": {
3
- "allow": [
4
- "Bash(uv run:*)",
5
- "Bash(uv sync:*)",
6
- "WebFetch(domain:github.com)"
7
- ]
8
- }
9
- }
@@ -1,309 +0,0 @@
1
- # CLAUDE.md — dockhand Development Guide
2
-
3
- ## Project Overview
4
-
5
- **dockhand** is a standalone CLI tool, originally extracted from DTU-HPC-CLI, for managing Docker containers on remote machines (or locally). It provides a unified interface to build, run, queue, manage, and monitor Docker containers via SSH or locally.
6
-
7
- **Key philosophy:** Simple, flat command structure with sensible SSH defaults. One `.dockhand.json` config file works everywhere.
8
-
9
- Since extraction, dockhand has grown features DTU-HPC-CLI never had: a task-spooler-based job queue, a mount-vs-bake code delivery mode, a direct (non-queued) run transport, and baked-image pruning. See [Relationship to DTU-HPC-CLI](#relationship-to-dtu-hpc-cli).
10
-
11
- ## Commands
12
-
13
- **Prerequisites:**
14
- - Install [uv](https://docs.astral.sh/uv/): `curl -LsSf https://astral.sh/uv/install.sh | sh`
15
-
16
- **Installation & Setup:**
17
- ```bash
18
- # Install dependencies and create lock file
19
- uv sync
20
-
21
- # Update lock file with latest versions (if needed)
22
- uv lock --upgrade
23
- ```
24
-
25
- **Run the CLI locally:**
26
- ```bash
27
- uv run dockhand --help
28
- ```
29
-
30
- **Lint:**
31
- ```bash
32
- uv run ruff check .
33
- ```
34
-
35
- **Format:**
36
- ```bash
37
- uv run ruff format .
38
- ```
39
-
40
- **Auto-fix lint issues:**
41
- ```bash
42
- uv run ruff check --fix .
43
- ```
44
-
45
- ## Architecture
46
-
47
- ### Entry Point
48
- `dockhand/__init__.py` — Defines the Typer CLI app. All commands are at the top level (flat structure, not nested under a `docker` sub-group). Each command validates config with `cli_config.check_docker()` and delegates to an `execute_*` function in the relevant module below.
49
-
50
- **Commands at top level:**
51
- - `dockhand submit` — sync code, then queue/run a container
52
- - `dockhand run` — queue/run from an already-built image, no sync
53
- - `dockhand install` — build the image only
54
- - `dockhand logs`, `stop`, `remove`, `jobs`, `urgent`, `prune` — job/queue lifecycle (`manage.py` + `queue.py`)
55
- - `dockhand history` — show past runs
56
- - `dockhand volumes`, `download` — inspect and pull files from mounted volumes
57
- - `dockhand resubmit` — rerun a past job with optional overrides
58
- - `dockhand tunnel` — SSH-forward container ports to localhost
59
-
60
- ### Configuration System
61
- `dockhand/config.py` — The `CLIConfig` class is loaded at module import time (`cli_config = CLIConfig.load()`) by walking up the directory tree to find `.dockhand.json`. Contains:
62
- - `SSHConfig` — hostname, user, identity file for SSH connections
63
- - `DockerConfig` — dockerfile, imagename, volumes, ports, gpus, containerworkdir, preserve_paths, code_delivery
64
- - `QueueConfig` — enabled, tool, slots (task-spooler queue settings)
65
- - `CLIConfig` — top-level config holder (ssh, docker, queue, sync, remote_path, profiles)
66
-
67
- Config file: `.dockhand.json` in the project root (must also contain a `.git` — `CLIConfig.load()` errors if no git repo is found). History file: `.dockhand_history.json` (same location by default, overridable via `history_path`).
68
-
69
- ### Client Abstraction
70
- `dockhand/client/` — `Client` (abstract base) has two implementations:
71
- - `SSHClient` — connects via Fabric/Paramiko over SSH
72
- - `LocalClient` — runs commands locally
73
-
74
- `get_client()` picks `LocalClient` when `cli_config.ssh` is unset or the configured hostname resolves to loopback, otherwise `SSHClient`. `get_client_for_host(hostname)` does the same but for an explicit hostname (used by `logs`/`stop`/`remove`, which read the host from the job's history entry rather than current config).
75
-
76
- ### Key Modules
77
- - `submit.py` — Builds the `docker run` command (code mount vs. data volumes vs. GPU/port flags), resolves code delivery, and hands off to a transport to start the job.
78
- - `build.py` — Runs `docker build` on the client (optionally syncing first).
79
- - `manage.py` — Job lifecycle: `logs`, `stop`, `remove`, `jobs` (`execute_stats`), and `prune` (removes baked images no longer referenced by an active job).
80
- - `resubmit.py` — Looks up a history entry and re-invokes `execute_submit` with overrides; pins to the original baked image when applicable.
81
- - `queue.py` — Task spooler (`tsp`) integration: submit/list/promote/remove/kill, plus `ts -l` output parsing.
82
- - `transport.py` — Abstracts "how a job runs": `TaskSpoolerTransport` (queue enabled) vs. `DockerTransport` (direct `docker run -d`). Both expose the same interface (`submit`, `list_jobs`, `logs`, `stop`, `remove`) so job-management commands don't care which backend created a job; the transport used is recorded per job in history.
83
- - `tagging.py` — Resolves the image tag/ref for baked code delivery (content-addressed from git commit + dirty-state hash; unique per queued submit, reused for direct runs).
84
- - `history.py` — Reads/writes `.dockhand_history.json`, reserves/looks up local job IDs.
85
- - `volumes.py` — Lists the container filesystem as a tree (code mount + data volumes) and resolves a workdir-relative path back to its host path.
86
- - `download.py` — Uses `volumes._resolve_to_host` + rsync to pull a file/directory from a volume.
87
- - `tunnel.py` — SSH local port forwarding to container ports (via Fabric).
88
- - `sync.py` — Uses `rsync` over SSH to copy local files to `remote_path`, respecting `.gitignore`; prompts to confirm when the worktree is dirty.
89
- - `config.py` — Configuration loading and validation.
90
- - `error.py` — Centralized error reporting with rich panels.
91
- - `constants.py` — Config/history filenames.
92
-
93
- > **Known inconsistency:** `tunnel.py` still looks up history entries by a `container_id` field (`entry["container_id"]`), but `history.py` no longer writes that field — entries are keyed by `local_id` with a transport `handle`. Passing an explicit `container_id` to `dockhand tunnel` will not resolve; only the no-argument (last job) path works reliably.
94
-
95
- ### Docker History
96
- Stores container runs in `.dockhand_history.json` as JSON. Each entry contains:
97
- ```json
98
- {
99
- "local_id": 7,
100
- "timestamp": 1234567890.123,
101
- "config": {
102
- "gpus": "all",
103
- "volumes": [...],
104
- "imagename": "my-image",
105
- "commands": ["python", "train.py"],
106
- "ports": ["6006:6006"],
107
- "image_ref": "my-image:abc123def456", // present when built (baked delivery)
108
- "branch": "main" // optional, detected from git
109
- },
110
- "transport": "task_spooler", // or "docker"
111
- "handle": 12, // tsp job id, or container name for direct runs
112
- "ts_job_id": 12, // task_spooler transport only
113
- "host": "remote.example.com", // or "localhost"
114
- "started_at": 1234567891.0, // optional, set the first time `jobs` observes it running
115
- "ended_at": 1234567895.0 // optional, set the first time `jobs` observes it finished/failed/stopped
116
- }
117
- ```
118
-
119
- Used by `resubmit` (look up a previous run and re-run with overrides, pinning the original baked image when unchanged), `logs`, `stop`, `remove`, `jobs`, `urgent`, `prune` (default to latest if no ID given). Job-management commands dispatch to the transport recorded on the entry (`transport_for_entry`), so `logs`/`stop`/`remove` work the same regardless of whether a job went through the queue or ran directly.
120
-
121
- `started_at`/`ended_at` aren't queried from tsp/docker (neither exposes exact start/end timestamps cheaply for both transports) — `jobs` and `logs` each stamp them lazily the first time they happen to observe a job in the running/terminal state, so a job never checked on while running will show no start time once it finishes. `dockhand jobs` displays these as Started/Ended (absolute, `%Y-%m-%d %H:%M:%S`) and Duration (elapsed while running, total once finished) columns; `dockhand logs` prints a one-line "running for Xm" / "finished in Xm" header before the log output.
122
-
123
- ### Design Patterns
124
-
125
- **DockerDefault factory:**
126
- In `__init__.py`, `DockerDefault` is a callable factory that pulls defaults from `cli_config.docker` at call time. Enables Typer's `default_factory` to respect active profiles.
127
-
128
- **Transport abstraction:**
129
- `get_transport()` in `transport.py` picks `TaskSpoolerTransport` or `DockerTransport` based on `cli_config.queue.enabled` at submit time. Every job-management command re-derives the correct transport per-job from the history entry (`transport_for_entry`) rather than from current config, so switching `queue.enabled` doesn't strand in-flight jobs from the other mode.
130
-
131
- **Code delivery resolution:**
132
- `DockerConfig.resolve_code_delivery(queue_enabled)` picks `mount` or `bake` when `code_delivery` isn't explicitly set: `bake` when the queue is enabled (avoids code drift while a job waits in queue), `mount` otherwise (zero-rebuild iteration). See README's [Code delivery](README.md#code-delivery-mount-vs-bake) section for the full rationale.
133
-
134
- **Profile support:**
135
- `cli_config.load_profile(name)` loads overrides from `.dockhand.json` profiles section (currently merges `history_path`, `remote_path`, `ssh`). All config classes support `validate()` for safe merging.
136
-
137
- **Command structure:**
138
- Each command:
139
- 1. Calls `cli_config.check_docker()` to validate config
140
- 2. Calls an `execute_*` function from the relevant module
141
- 3. Passes `cli_config.docker` (and often `cli_config.queue`) as config arguments
142
-
143
- ## Relationship to DTU-HPC-CLI
144
-
145
- - **Flat commands** — no `docker` sub-group. Commands are top-level.
146
- - **No HPC support** — SubmitConfig, InstallConfig, and all LSF/job submission code removed
147
- - **Docker-focused** — all UI/UX optimized for docker workflows
148
- - **Config-compatible in spirit, not in filename** — dockhand originally reused DTU-HPC-CLI's `.dtu_hpc.json`/`.dtu_docker_history.json` names; both have since been renamed to `.dockhand.json`/`.dockhand_history.json` (see `constants.py`). Old DTU-HPC-CLI config files need renaming (or a `history_path` override) to work with dockhand.
149
- - **Diverged, not just trimmed** — dockhand has added its own features DTU-HPC-CLI doesn't have: a task-spooler queue (`queue.py`/`transport.py`), slot reservations, `mount`/`bake` code delivery (`tagging.py`), a direct-run transport for when the queue is off, `urgent`/`prune`/`tunnel` commands. Maintained independently; version/release cadence is separate from DTU-HPC-CLI.
150
-
151
- ## Key Files and Their Responsibilities
152
-
153
- | File | Purpose |
154
- |------|---------|
155
- | `__init__.py` | CLI app definition, command routing, config defaults |
156
- | `submit.py` | Builds the `docker run` command, resolves code delivery, submits via transport |
157
- | `build.py` | `docker build` execution |
158
- | `manage.py` | `logs`, `stop`, `remove`, `jobs`, `prune` |
159
- | `resubmit.py` | Rerun a past job with overrides |
160
- | `queue.py` | Task spooler (`tsp`) integration |
161
- | `transport.py` | Task-spooler vs. direct-docker execution backends |
162
- | `tagging.py` | Image tag resolution for baked code delivery |
163
- | `history.py` | Read/write `.dockhand_history.json`, job ID allocation |
164
- | `volumes.py` | Container filesystem tree, path resolution |
165
- | `download.py` | rsync-based file download from volumes |
166
- | `tunnel.py` | SSH port forwarding to container ports |
167
- | `config.py` | Config loading, validation, profile support |
168
- | `client/__init__.py` | Auto-detect and return appropriate client |
169
- | `client/base.py` | Abstract Client interface |
170
- | `client/local.py` | Local command execution |
171
- | `client/ssh.py` | Remote execution via SSH (Fabric/Paramiko) |
172
- | `sync.py` | rsync-based code synchronization |
173
- | `error.py` | Error reporting |
174
- | `constants.py` | Config/history file names |
175
-
176
- ## Configuration Format
177
-
178
- `.dockhand.json` in project root (project root is also where `CLIConfig.load()` expects to find a `.git` directory):
179
-
180
- ```json
181
- {
182
- "sync": true,
183
- "ssh": {
184
- "user": "your_username",
185
- "identityfile": "~/.ssh/id_rsa",
186
- "hostname": "remote.example.com"
187
- },
188
- "queue": {
189
- "enabled": true,
190
- "slots": 1
191
- },
192
- "docker": {
193
- "dockerfile": "Dockerfile",
194
- "imagename": "my-image",
195
- "volumes": [
196
- {
197
- "hostpath": "/local/data",
198
- "containerpath": "/data",
199
- "permissions": "rw"
200
- }
201
- ],
202
- "ports": ["8080:80"],
203
- "gpus": "all",
204
- "containerworkdir": "/",
205
- "preserve_paths": [".venv"],
206
- "code_delivery": null
207
- },
208
- "remote_path": "~/my-project",
209
- "profiles": {
210
- "dev": {
211
- "docker": {
212
- "gpus": "1",
213
- "dockerfile": "Dockerfile.dev"
214
- }
215
- }
216
- }
217
- }
218
- ```
219
-
220
- All options are optional except `dockerfile`, `imagename`, and `volumes` within docker config. `queue.slots` may also be set as `docker.slots` for back-compat (deprecation warning printed). Full option tables and defaults are documented in README.md's [Configuration](README.md#configuration) section — keep both in sync when config shape changes.
221
-
222
- ## Common Workflows
223
-
224
- **First time setting up:**
225
- 1. Create `.dockhand.json` with `docker` and `ssh` sections
226
- 2. Run `dockhand install` to build the image
227
- 3. Run `dockhand submit 'command'` to run a container
228
-
229
- **Development iteration:**
230
- ```bash
231
- uv run dockhand submit --gpus 1 'python train.py' # Sync + queue/run with 1 GPU
232
- uv run dockhand logs --n 50 # Check last 50 log lines
233
- uv run dockhand resubmit --gpus 2 # Re-run with 2 GPUs
234
- uv run dockhand download results/model.pth # Get results back
235
- ```
236
-
237
- **Quick rebuild:**
238
- ```bash
239
- uv run dockhand install --dockerfile Dockerfile.dev # Rebuild only
240
- uv run dockhand run 'bash' # Run interactively
241
- ```
242
-
243
- **Using profiles:**
244
- ```bash
245
- uv run dockhand --profile dev submit 'python train.py' # Use dev profile
246
- uv run dockhand --profile prod install # Build prod image
247
- ```
248
-
249
- **Queue management:**
250
- ```bash
251
- uv run dockhand jobs # List active jobs
252
- uv run dockhand urgent 5 # Promote job #5 to front of queue
253
- uv run dockhand prune --dry-run # Preview unused baked images before removing
254
- ```
255
-
256
- ## Testing & Validation
257
-
258
- No automated tests yet. Manual validation:
259
- ```bash
260
- uv run dockhand --help # Check CLI structure
261
- uv run dockhand submit --help # Check submit options
262
- uv run ruff check . # Lint
263
- uv run ruff format --check . # Format check
264
- ```
265
-
266
- ## Important Design Decisions
267
-
268
- 1. **Flat commands** — No `docker` sub-group. Simpler for a docker-only tool.
269
- 2. **SSH by default** — Assumes docker host is remote. Auto-detects local by resolving the configured hostname to a loopback address.
270
- 3. **History in JSON** — Simple, human-readable, easily editable.
271
- 4. **Profile support** — Allows per-project config variants.
272
- 5. **Minimal dependencies** — typer, fabric, paramiko, gitpython, rich.
273
- 6. **Transport abstraction over queue state** — job-management commands read the transport from each job's own history entry rather than current `queue.enabled`, so toggling the queue mid-flight doesn't orphan existing jobs.
274
- 7. **Code-delivery default follows queue mode** — `bake` when queued (avoids code drift while waiting), `mount` when not (zero-rebuild iteration); explicit `code_delivery` overrides either way.
275
-
276
- ## Extraction History
277
-
278
- - **Source:** DTU-HPC-CLI (github.com/ChrisFugl/DTU-HPC-CLI)
279
- - **Extracted:** 2026-04-09
280
- - **Initial changes:**
281
- - Removed: SubmitConfig, InstallConfig, types.py, all HPC modules
282
- - Modified: config.py (trimmed), __init__.py (flat commands)
283
- - Unchanged at the time: docker.py, sync.py, error.py, client/*, constants.py
284
- - **Since extraction:** the original monolithic `docker.py` was split into `build.py`, `submit.py`, `manage.py`, `resubmit.py`, `history.py`, `download.py`, `volumes.py`, `tunnel.py`; `queue.py`, `transport.py`, and `tagging.py` were added for the task-spooler queue, transport abstraction, and baked code delivery. Config/history filenames moved from `.dtu_hpc.json`/`.dtu_docker_history.json` to `.dockhand.json`/`.dockhand_history.json`.
285
-
286
- ## Future Enhancements
287
-
288
- Potential improvements specific to dockhand (volume inspection is already implemented — see `volumes.py`):
289
- - Docker Compose support (`dockhand compose up`)
290
- - Kubernetes pod management
291
- - Local-only mode (drop SSH dependency for pure local docker)
292
- - Config auto-generation wizard
293
- - Container registry integration (Docker Hub, ECR, GCR)
294
- - Multi-container orchestration helpers
295
- - Fix `tunnel.py`'s stale `container_id` history lookup (see [Known inconsistency](#key-modules) above)
296
-
297
- ## Contributing
298
-
299
- When working on dockhand:
300
- 1. Keep the flat command structure — don't re-introduce `docker` sub-groups
301
- 2. Maintain `.dockhand.json` compatibility
302
- 3. Test manually with `uv run dockhand <command> --help` and actual docker operations
303
- 4. Follow ruff lint/format rules (use `uv run ruff check . && uv run ruff format .`)
304
- 5. Document new features in README.md and CLAUDE.md
305
- 6. Update this file's Architecture section when modules are added, split, or renamed — it has drifted from actual code structure before
306
-
307
- ## Maintenance
308
-
309
- dockhand is maintained independently from DTU-HPC-CLI. Version bumps and releases are separate. If you need HPC support, use the original DTU-HPC-CLI package.
File without changes
File without changes