jobd 0.1.0__tar.gz

jobd-0.1.0/.env.example

@@ -0,0 +1,14 @@
+# jobd broker configuration — copy to `.env` and fill in.
+#   cp .env.example .env
+# docker-compose.yml reads this file; JOBD_API_TOKEN is required.
+
+# Shared secret for broker<->worker<->client auth. Generate a strong one:
+#   openssl rand -hex 32
+# Every worker systemd unit and every CLI/MCP wrapper must use the SAME value.
+JOBD_API_TOKEN=
+
+# Address uvicorn binds to. Loopback by default. Set to a Tailscale CGNAT IP
+# (100.64.0.0/10) to expose the broker to other tailnet machines. NEVER bind to
+# a public/LAN address — the access-control model assumes a trusted tailnet.
+# See docs/security.md.
+JOBD_HOST=127.0.0.1

jobd-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,41 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Install dependencies (from lockfile)
+        run: uv sync --frozen --python ${{ matrix.python-version }} --extra dev --extra worker --extra mcp
+
+      - name: Lint (ruff)
+        run: uv run ruff check .
+
+      - name: Tests (excluding live broker)
+        run: uv run pytest -m "not live" -q
+
+      # Informational: the codebase is mid-typing-adoption, so mypy is reported
+      # but does not gate the build. Tighten to a hard gate once it's clean.
+      - name: Types (mypy, non-blocking)
+        run: uv run mypy src/jobd src/job_cli
+        continue-on-error: true

jobd-0.1.0/.gitignore

@@ -0,0 +1,21 @@
+__pycache__/
+*.pyc
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+*.egg-info/
+dist/
+build/
+.venv/
+venv/
+logs/
+jobd.db
+jobd.db-journal
+jobd.db-wal
+jobd.db-shm
+.coverage
+coverage.xml
+htmlcov/
+.env
+.env.*
+!.env.example

jobd-0.1.0/CHANGELOG.md

@@ -0,0 +1,77 @@
+# Changelog
+
+All notable changes to jobd. Format roughly follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## [0.1.0] — 2026-05-31
+
+Initial public release. The bullets below summarize the capabilities built during
+pre-release development; parenthetical short hashes reference that internal history
+and are not present in this repository's squashed tree.
+
+### Added — Auto-preempt protocol
+
+- **#8 path A — auto-preempt on queue-age** (`6db2712`): broker sweeper auto-preempts running jobs when their host has a higher-priority candidate waiting and queue age crosses threshold.
+- **#28/#29 manual preempt-blockers + warnings-only filter** (`b06e78d`): `--no-preempt` submit flag and warnings-only `job list --warnings` filter.
+- **#32 `events.jsonl` auto-preempt event** (`e719842`): broker emits a structured event each time a sweep fires, for observability.
+- **#34 `/preempt` accepts ASSIGNED-but-not-started state** (`22a3791`): pinned by test.
+- **#35 `depends_on` cascade on PREEMPTED parent** (`26e56ac`): pinned by test.
+- **#8 path B — checkpoint protocol** (`e7593b5`): SIGTERM grace window with `JOBD_CHECKPOINT_GRACE_S` knob; terminal `preempted` state with `RESUME_FROM` operator-driven resume contract.
+- **`JOBD_CHECKPOINT_DIR` env var** (`f0cb754`): exposed to every job, pointing at a per-job directory `${JOBD_WORKER_CHECKPOINT_ROOT:-$XDG_DATA_HOME/jobd/checkpoints}/<job_id>/` (default `~/.local/share/jobd/checkpoints/<job_id>/`). Workloads write durable preempt-time checkpoints here; broker does not sweep contents.
+- **`JOBD_WORKER_CHECKPOINT_ROOT` operator override** (`71ae23b`, fix-forward `bf195bc`): worker-level env var redirects checkpoint root (e.g., to a faster filesystem); `os.path.expanduser` applied.
+
+### Added — Resource-aware admission & GPU matching
+
+- **#42 heartbeat-aware GPU matcher** (`a3509c9`): matcher consults live worker heartbeats (`unregistered_vram_gb`) and drops saturated hosts; 2 GB VRAM floor on `--gpu`.
+- **#42 multi-tenant per-host co-routing** (`f9e6152`): `JOBD_WORKER_MAX_CONCURRENT_JOBS` enables threaded per-host dispatch with per-job thread tracking.
+- **`unregistered_vram_gb` in `/workers`** (`93795b0`): exposed for matcher + observability.
+- **#41 resource-aware admission at dispatch** (`53ffc09`): NVML probe → admission gate → `/jobs/{id}/refuse-admission` re-queue path; tier-tag inference; bypass marker.
+- **#41d `vram_gb` on `JobSubmit`** (`ed154f5`): CLI `--vram-required` flag now persists end-to-end.
+- **#43 unsatisfiable-placement preflight** (`bf15ffa`): submit-time check rejects jobs whose `needs:` tags or VRAM ask cannot be satisfied by any registered worker.
+- **CUDA VRAM tier tags** (`860fdf6`): `cuda-32gb`, `cuda-12gb`, etc. so `needs:cuda-32gb` routes only to matching hosts.
+- **`cuda-8gb` tier** (`a8ed96d`): added so RTX 2080-class GPUs advertise a discoverable tier.
+
+### Added — Scheduling, watchdogs, MCP
+
+- **Per-job idle-output watchdog + max-wall timeout** (`5f074c0`): broker-side wall enforcement; idle-output watchdog terminates silent-hung jobs.
+- **`--max-wall` and `--idle-timeout` flags on `job submit`** (`f105724`).
+- **Scheduler-awareness warnings** (`0ef4497`): single-slot stall + queue-age-blocked-by-load surface as `warnings:` on `/jobs/{id}`.
+- **Per-job time estimation v1** (`1642d5d`): `eta_*` fields on `Job` and `--eta` flag on `job list`.
+- **Default-on ETA banner on `job submit`**: `--eta/--no-eta` defaults on; prints `Estimated wall p50 X, p90 Y (n=N prior runs)` or `ColdStart` line to stderr after submit. Closes BACKLOG "Auto-surface ETA on submit" Part 1.
+- **ETA Part 2 — ctest-aware sub-job parsing**: opt-in via `JOBD_CTEST_PARSE=1`. New module `src/jobd/ctest_eta.py` parses `<cwd>/build*/Testing/Temporary/CTestCostData.txt`, filters tests by the `ctest -R <regex>` arg, and sums avg-cost values. Broker reports `eta_basis="ctest-cost-K=<n>"` ahead of history-based prediction; CLI banner renders `Estimated wall ~Xs (ctest cost-data, k=K tests)`. Falls through to history when env unset, regex misses, or cost file absent. Closes BACKLOG "Auto-surface ETA on submit" Part 2.
+- **First-byte smoke watchdog — pieces 2+3**: worker-side, in `worker/job_worker.py`. Piece 3 (pre-dispatch launcher-existence check) verifies `cmd[0]` exists and is executable when it looks like a path (`/`, `./`, `../`); on miss, POSTs `/complete` with `termination_reason="launcher_missing"` instead of exec-ing into a silent exit-127. Piece 2 (first-output watchdog) adds env var `JOBD_WORKER_FIRST_OUTPUT_TIMEOUT_S`: fires once if no stdout byte lands within N seconds of job start, disarms permanently on first byte. Both surface as `final_state="failed"` for depends_on cascades. Closes BACKLOG "First-byte smoke" pieces 2+3; piece 1 (push-on-terminal-failure) deferred.
+- **Per-project defaults block in `projects.yaml`** (`34c2a30`): keys at submit time inherit from project block when omitted.
+- **`fast_path` field honored in `JobSubmit`** (`557a0e8`).
+- **#51 `submitted_via` marker** (`636969c`): `JobSubmit.submitted_via: Literal["cli", "mcp"]` round-trips through translation layer; structural test pinned.
+
+### Added — Worker management & deploy
+
+- **DELETE `/workers/{host}` for purging stale registrations** (`ea04d6a`).
+- **`job delete-worker` + `jobd_worker_delete` MCP tool** (`ccdf35c`): exposes the DELETE endpoint via CLI and MCP.
+- **`jobd-broker.service` systemd unit** (`5334a5c`, #52): with tailscale IP wait.
+- **`job-worker.laptop.service` variant** (`c4591b0`): for full-repo-checkout hosts (vs. the server standalone install).
+- **Nightly live integration test cron wrapper** (`ef24309`): runs `tests/mcp/test_live.py` against the live broker.
+- **Audit instrumentation pass** (`07aa2cf`): coverage inventory + gap report against the existing observability surface.
+
+### Fixed
+
+- **`jobd --help` / `--version` no longer crash with SQLite OperationalError**: entry point now parses argv before `build_app()`, so `--help`/`--version` short-circuit cleanly without touching the database. Help text documents `JOBD_CONFIG_DIR` / `JOBD_DB_URL` / `JOBD_PORT` / `JOBD_LOGS_DIR`.
+- **`#51` install-worker no longer writes static `tags:` to `worker.yaml`** (`5c01882`): tags now come from runtime probe.
+- **`#73eaa46` cancel via `systemctl kill` on named scope** rather than `Popen.pid`: ensures the entire scope tree dies, not just the tracked PID.
+- **`59561aa` ASSIGNED → RUNNING transition** so cancel reports SIGTERM correctly.
+- **`969abc3` MCP `log_tail` and `depends_on` field-name alignment** (mcp-v1 field-test fallout).
+- **`353a4fb`** clear pyright noise in `_print_resolved`/`_row`.
+
+### Documentation
+
+- **Checkpoint directory contract** in `docs/preemption.md` (`8b5e339`, fix-forward `7ebffca`, final-review polish `2eec63b`): canonical surface for workload authors; covers env vars, default root, mode-0700 + cross-user-resume caveat.
+- **Auto-preempt default-flip design spec** (`37e2dc4`) + amendment (`31e843d`).
+- **Auto-preempt jobd-side implementation plan** (`915970f`).
+- **Backlog reconciliation** (`ddca65e`, `d34f3b0`, `6d0fbaf`, `b257cca`, `354e043`, `ab8f7f7`, `7eb1203`, `394269c`, `b8de199`): Open vs Done sweep after the 2026-04-27..04-29 ship train; field-test backlog items filed.
+- **Projects.yaml defaults — implementation blueprint** (`c98cf26`).
+- **CHANGELOG.md created** (`86b4880`); this entry backfills the gap from `mcp-v1` to current tip.
+
+---
+
+## [mcp-v1] — 2026-04-26
+
+Translation-layer/MCP shipping point. Earlier history is not catalogued here; see `git log mcp-v1` for detail.

jobd-0.1.0/Dockerfile

@@ -0,0 +1,32 @@
+# Pinned to a specific Debian release for reproducibility. For stricter supply-
+# chain guarantees, pin by digest instead: FROM python:3.11-slim-bookworm@sha256:<digest>
+FROM python:3.11-slim-bookworm
+
+WORKDIR /app
+ENV PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    JOBD_CONFIG_DIR=/app/config \
+    JOBD_DB_URL=sqlite:////app/data/jobd.db \
+    JOBD_LOGS_DIR=/app/logs \
+    JOBD_PORT=8765
+
+COPY pyproject.toml ./
+COPY src ./src
+RUN pip install -U pip && pip install .
+
+# Run as an unprivileged user; the broker never needs root. Data/logs must be
+# writable by that user (the SQLite DB lives under /app/data).
+RUN useradd --create-home --uid 10001 jobd \
+    && mkdir -p /app/data /app/logs \
+    && chown -R jobd:jobd /app
+USER jobd
+
+EXPOSE 8765
+
+# /health is behind the bearer-token auth dependency, and the slim image has no
+# curl, so the healthcheck is a plain TCP connect: a listening socket means the
+# broker is up.
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
+    CMD python -c "import os,socket; socket.create_connection(('127.0.0.1', int(os.environ.get('JOBD_PORT','8765'))), timeout=3).close()"
+
+CMD ["jobd"]

jobd-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jaret Arnold
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

jobd-0.1.0/PKG-INFO

@@ -0,0 +1,225 @@
+Metadata-Version: 2.4
+Name: jobd
+Version: 0.1.0
+Summary: Self-hostable GPU-aware job broker for your own machines, with native MCP/agent integration
+Project-URL: Homepage, https://github.com/musharna/jobd
+Project-URL: Repository, https://github.com/musharna/jobd
+Project-URL: Issues, https://github.com/musharna/jobd/issues
+Project-URL: Changelog, https://github.com/musharna/jobd/blob/main/CHANGELOG.md
+Author: Jaret Arnold
+License: MIT
+License-File: LICENSE
+Keywords: gpu,homelab,job-queue,mcp,scheduler,self-hosted
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.11
+Requires-Dist: fastapi>=0.110
+Requires-Dist: httpx>=0.26
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: sqlalchemy>=2.0
+Requires-Dist: sse-starlette>=2.0
+Requires-Dist: starlette>=1.0.1
+Requires-Dist: typer>=0.9
+Requires-Dist: uvicorn[standard]>=0.27
+Provides-Extra: dev
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: respx>=0.20; extra == 'dev'
+Requires-Dist: ruff>=0.3; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: httpx>=0.27; extra == 'mcp'
+Requires-Dist: mcp>=1.0; extra == 'mcp'
+Provides-Extra: worker
+Requires-Dist: httpx>=0.26; extra == 'worker'
+Requires-Dist: nvidia-ml-py>=12; extra == 'worker'
+Requires-Dist: psutil>=5.9; extra == 'worker'
+Requires-Dist: pyyaml>=6.0; extra == 'worker'
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+# jobd
+
+[![CI](https://github.com/musharna/jobd/actions/workflows/ci.yml/badge.svg)](https://github.com/musharna/jobd/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/jobd)](https://pypi.org/project/jobd/)
+![Python](https://img.shields.io/pypi/pyversions/jobd)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**A self-hostable, GPU-aware job broker for your own machines — with native MCP/agent integration.**
+
+> Like [task-spooler](https://manpages.ubuntu.com/manpages/noble/man1/tsp.1.html), but across more than one machine — and VRAM-aware.
+
+</div>
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/musharna/jobd/main/docs/assets/demo.svg" alt="jobd in action: submit a GPU job, watch it route to a worker with free VRAM and stream back, then inspect the full lifecycle" width="100%">
+</p>
+
+You have a couple of boxes with GPUs — a workstation, a server, maybe a laptop — wired together over [Tailscale](https://tailscale.com/) or a LAN. You want to fire off training runs, data pipelines, and long batch jobs from anywhere, have them land on whichever machine actually has the VRAM free, survive across sessions, and get preempted cleanly when something more important shows up. You don't have a cloud, a Kubernetes cluster, or a Slurm install, and you don't want one.
+
+jobd is that missing piece: a small broker that turns a handful of personal machines into a single queue. Think _SkyPilot / Modal, for people without a cloud_ — except the fleet is the hardware you already own, and an LLM agent can drive it directly.
+
+```bash
+# from any machine on your tailnet:
+job submit --project myproj --gpu --vram-required 16 --wait -- python train.py
+# → routed to whichever worker has ≥16 GB VRAM free, streamed back to your terminal
+```
+
+## Why it exists
+
+Most schedulers assume a datacenter. The lightweight ones that don't (a bare `nohup`, a tmux session, an ssh-and-pray script) give you nothing: no queue, no VRAM-aware routing, no preemption, no record of what ran where. jobd fills the gap between "ssh in and run it" and "stand up Slurm":
+
+- **VRAM-fit routing.** The broker matches each job against live worker capacity (free VRAM / RAM / CPUs, capability tags, arch/OS) and dispatches to a worker that actually fits — instead of you guessing which box is free.
+- **Preempt + checkpoint.** A higher-priority job can preempt a running one: the worker sends `SIGTERM`, the workload gets a grace window to checkpoint, then `SIGKILL`. A preempted job reaches a terminal `preempted` state with a durable checkpoint to resume from — it isn't silently re-run. (See [docs/preemption.md](docs/preemption.md).)
+- **Survives sessions.** Submit, close your laptop, check back tomorrow. Jobs live in the broker, not your shell.
+- **Agent-native.** Ships a first-class [MCP](https://modelcontextprotocol.io/) server so an LLM agent (Claude Code, etc.) can submit, monitor, and babysit jobs as tool calls — the thing most schedulers bolt on as an afterthought, if at all.
+- **Yours.** One broker process you run on a machine you own. No accounts, no egress, no per-GPU-hour billing. Tailnet-bound by default.
+
+## Why not just use…?
+
+| Tool                                                                           | What it gives you                                           | Why jobd instead                                                                                               |
+| ------------------------------------------------------------------------------ | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
+| **`nohup` / `tmux` / ssh-and-pray**                                            | Runs a command on one box                                   | No queue, no VRAM-aware routing, no preemption, no record of what ran where                                    |
+| **[task-spooler](https://manpages.ubuntu.com/manpages/noble/man1/tsp.1.html)** | A real job queue — on a single machine                      | jobd queues across _all_ your machines and routes by live VRAM/CPU fit                                         |
+| **Slurm**                                                                      | Datacenter-grade scheduling                                 | Heavy to stand up and operate for 2–3 personal boxes; jobd is one process + a poller per host                  |
+| **SkyPilot / Modal / dstack**                                                  | Provision and run on clouds (SkyPilot also on-prem via SSH) | jobd targets hardware you _already own_, with no cloud/K8s assumptions and a much smaller footprint            |
+| **Ray**                                                                        | A distributed-compute framework                             | jobd is a job _queue_, not a programming model — submit any command, no code changes, GPU-fit routing built in |
+
+Closest in spirit are task-spooler (single-node) and on-prem SkyPilot (heavier, cloud-shaped). jobd's niche is the 2–3-GPU homelab: multi-machine VRAM-fit routing + preempt/checkpoint + a native agent interface, with nothing to stand up.
+
+## Architecture
+
+```mermaid
+flowchart TD
+    CLI["job CLI"]:::client --> B
+    MCP["jobd-mcp<br/>MCP tools"]:::client --> B
+    API["HTTP · SSE"]:::client --> B
+    B["<b>jobd broker</b> — FastAPI<br/>queue · matcher · priorities · SQLite"]:::broker
+    B <-->|poll · dispatch| WA["worker A<br/>24 GB GPU"]:::worker
+    B <-->|poll · dispatch| WB["worker B<br/>8 GB GPU"]:::worker
+    B <-->|poll · dispatch| WC["worker C<br/>CPU-only"]:::worker
+    classDef client fill:#1f2937,stroke:#4b5563,color:#e5e7eb;
+    classDef broker fill:#0e7490,stroke:#155e75,color:#ecfeff;
+    classDef worker fill:#14532d,stroke:#166534,color:#dcfce7;
+```
+
+Workers **poll** the broker (pull model — no inbound connection to a worker); the broker matches each job against live capacity and hands it back on the poll. One broker process, one poller per host.
+
+- **Broker** — a FastAPI + SQLite service. Holds the queue, runs the matcher, resolves per-project priorities and defaults, exposes a small HTTP API and an SSE stream. Single source of truth.
+- **Workers** — lightweight polling agents, one per host. Each advertises live capacity via heartbeat, claims jobs it can run, executes them (`shell=False`, no shell-injection surface), streams logs back, and honors preemption signals.
+- **Clients** — the `job` CLI, the `jobd-mcp` MCP server, or anything that speaks the HTTP API.
+
+## Install
+
+```bash
+pip install jobd            # broker + CLI
+pip install "jobd[mcp]"     # adds the MCP server
+```
+
+Requires Python ≥ 3.11. The `pip` package is the **broker + CLI + MCP server**; the worker is intentionally not bundled (it has host-specific system deps). Workers run from a clone of this repo (`worker/job_worker.py`) — `scripts/install-worker.sh` sets one up under `~/jobd-worker` with its own venv. (Packaging the worker as `jobd-worker` is planned for a later release.)
+
+## Quickstart (single host)
+
+```bash
+# 1. start the broker (binds 127.0.0.1:8765 by default)
+JOBD_ALLOW_NO_AUTH=1 jobd          # no-auth is fine for a loopback-only broker
+
+# 2. in another shell (from a clone of this repo), start a worker pointed at it
+pip install httpx psutil pyyaml pynvml   # worker deps (pynvml only needed for GPU hosts)
+JOBD_URL=http://127.0.0.1:8765 JOBD_WORKER_HOST=local \
+  python worker/job_worker.py
+
+# 3. submit a job and wait for it
+job submit --project demo --wait -- echo hello
+job list
+job logs <id>
+```
+
+For a real multi-host deployment (Docker broker + systemd workers, Tailscale binding, shared auth token), see **[docs/security.md](docs/security.md)** and the templates in `docker-compose.yml`, `scripts/`, and `worker/`. Day-2 operations (health, draining a worker, upgrades, token rotation, backups) are in **[docs/runbook.md](docs/runbook.md)**.
+
+## Supported platforms
+
+Python 3.11+ everywhere.
+
+| Component                              | Linux   | macOS       | Windows              |
+| -------------------------------------- | ------- | ----------- | -------------------- |
+| **Broker** (`jobd`)                    | ✅      | ✅          | ✅ (WSL recommended) |
+| **CLI** (`job`) / **MCP** (`jobd-mcp`) | ✅      | ✅          | ✅                   |
+| **Worker** (`job_worker.py`)           | ✅ full | ⚠️ degraded | ⚠️ degraded          |
+
+The **worker** runs its best on Linux with a systemd user instance: memory caps, process reaping, and preemption use `systemd-run --user` scopes and cgroups. On non-systemd hosts the worker still executes jobs, but silently drops those guarantees — fine for a single trusted box, not for hard resource isolation. GPU features need NVIDIA + `nvidia-ml-py`. The broker, CLI, and MCP server are pure-Python and portable.
+
+## CLI
+
+```
+job submit -p PROJ [--gpu] [--vram-required N] [--needs TAG]... [--wait] -- CMD...
+job list [--state STATE] [--project P]      # queue + recent jobs
+job status ID [--watch]                     # one job, optionally live
+job logs ID [-n BYTES]                      # tail captured output
+job wait ID                                 # block until terminal
+job cancel ID  /  job preempt ID            # stop a job
+job workers                                 # fleet snapshot + health
+job projects list | set NAME PRI | nudge NAME DELTA
+job audit [--project P] [--since 24h]       # event history
+```
+
+`job submit --explain` dry-runs the resolution (priority, profile, project defaults, host pin) and prints the effective config without enqueuing anything.
+
+## MCP / agent integration
+
+jobd ships an MCP server (`jobd-mcp`) exposing the queue as nine tools — `jobd_submit`, `jobd_status`, `jobd_logs`, `jobd_list`, `jobd_cancel`, `jobd_preempt`, `jobd_workers`, `jobd_job_get`, `jobd_worker_delete`. Point your MCP client at it:
+
+```json
+{
+  "mcpServers": {
+    "jobd": {
+      "command": "jobd-mcp",
+      "env": {
+        "JOBD_URL": "http://127.0.0.1:8765",
+        "JOBD_API_TOKEN": "<your-token>"
+      }
+    }
+  }
+}
+```
+
+`JOBD_API_TOKEN` must match the broker's token, or every call returns 401. Omit it only when the broker runs with `JOBD_ALLOW_NO_AUTH=1`.
+
+Now an agent can "run this overnight," check on it next session, and route GPU work through the broker instead of colliding on a shared card. The `examples/claude-code-hooks/` directory has optional [Claude Code](https://docs.claude.com/en/docs/claude-code) hooks that _nudge_ (or hard-block) an agent toward submitting heavy commands through jobd — including a VRAM-aware GPU guard with `# NO_GPU` / `# CONCURRENT_OK` / `# VRAM=NGB` override markers.
+
+## Configuration
+
+Three optional YAML files under `JOBD_CONFIG_DIR` (defaults shipped in `config/`):
+
+- **`projects.yaml`** — per-project base priority and submit defaults (preemptibility, wall/idle timeouts, host pins, capability requirements). See [docs/plans/projects-yaml.md](docs/plans/projects-yaml.md) for the full resolution model.
+- **`profiles.yaml`** — named resource bundles (`--profile gpu-train-large`) the matcher uses to size a job.
+- **`classifier.yaml`** — rules that auto-suggest a profile from the command string.
+
+All three are optional; with none present, every job runs at the global default priority.
+
+## Security
+
+The broker has **no TCP-layer auth beyond a shared bearer token**, so it is meant to run on a trusted network (loopback or a Tailscale tailnet), never on a public interface. Two stacked controls:
+
+1. **Interface binding** — `JOBD_HOST` must be `127.0.0.1` or a Tailscale CGNAT address (`100.64.0.0/10`), never `0.0.0.0`. A CI lint (`tests/test_deploy_lint.py`) enforces this on the Docker deployment.
+2. **Bearer token** — set `JOBD_API_TOKEN` (≥32 random bytes) on every broker/worker/CLI/MCP host. The broker refuses to start without it unless you explicitly set `JOBD_ALLOW_NO_AUTH=1`. **`JOBD_ALLOW_NO_AUTH=1` is for a loopback-only broker (`JOBD_HOST=127.0.0.1`) — for local dev/tests.** Combined with a non-loopback `JOBD_HOST` it exposes an unauthenticated RCE endpoint to your whole tailnet; the broker logs a startup warning if you do this. Don't.
+
+Full threat model, env-var reference, and token rotation: **[docs/security.md](docs/security.md)**.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

jobd-0.1.0/README.md

@@ -0,0 +1,173 @@
+<div align="center">
+
+# jobd
+
+[![CI](https://github.com/musharna/jobd/actions/workflows/ci.yml/badge.svg)](https://github.com/musharna/jobd/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/jobd)](https://pypi.org/project/jobd/)
+![Python](https://img.shields.io/pypi/pyversions/jobd)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**A self-hostable, GPU-aware job broker for your own machines — with native MCP/agent integration.**
+
+> Like [task-spooler](https://manpages.ubuntu.com/manpages/noble/man1/tsp.1.html), but across more than one machine — and VRAM-aware.
+
+</div>
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/musharna/jobd/main/docs/assets/demo.svg" alt="jobd in action: submit a GPU job, watch it route to a worker with free VRAM and stream back, then inspect the full lifecycle" width="100%">
+</p>
+
+You have a couple of boxes with GPUs — a workstation, a server, maybe a laptop — wired together over [Tailscale](https://tailscale.com/) or a LAN. You want to fire off training runs, data pipelines, and long batch jobs from anywhere, have them land on whichever machine actually has the VRAM free, survive across sessions, and get preempted cleanly when something more important shows up. You don't have a cloud, a Kubernetes cluster, or a Slurm install, and you don't want one.
+
+jobd is that missing piece: a small broker that turns a handful of personal machines into a single queue. Think _SkyPilot / Modal, for people without a cloud_ — except the fleet is the hardware you already own, and an LLM agent can drive it directly.
+
+```bash
+# from any machine on your tailnet:
+job submit --project myproj --gpu --vram-required 16 --wait -- python train.py
+# → routed to whichever worker has ≥16 GB VRAM free, streamed back to your terminal
+```
+
+## Why it exists
+
+Most schedulers assume a datacenter. The lightweight ones that don't (a bare `nohup`, a tmux session, an ssh-and-pray script) give you nothing: no queue, no VRAM-aware routing, no preemption, no record of what ran where. jobd fills the gap between "ssh in and run it" and "stand up Slurm":
+
+- **VRAM-fit routing.** The broker matches each job against live worker capacity (free VRAM / RAM / CPUs, capability tags, arch/OS) and dispatches to a worker that actually fits — instead of you guessing which box is free.
+- **Preempt + checkpoint.** A higher-priority job can preempt a running one: the worker sends `SIGTERM`, the workload gets a grace window to checkpoint, then `SIGKILL`. A preempted job reaches a terminal `preempted` state with a durable checkpoint to resume from — it isn't silently re-run. (See [docs/preemption.md](docs/preemption.md).)
+- **Survives sessions.** Submit, close your laptop, check back tomorrow. Jobs live in the broker, not your shell.
+- **Agent-native.** Ships a first-class [MCP](https://modelcontextprotocol.io/) server so an LLM agent (Claude Code, etc.) can submit, monitor, and babysit jobs as tool calls — the thing most schedulers bolt on as an afterthought, if at all.
+- **Yours.** One broker process you run on a machine you own. No accounts, no egress, no per-GPU-hour billing. Tailnet-bound by default.
+
+## Why not just use…?
+
+| Tool                                                                           | What it gives you                                           | Why jobd instead                                                                                               |
+| ------------------------------------------------------------------------------ | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
+| **`nohup` / `tmux` / ssh-and-pray**                                            | Runs a command on one box                                   | No queue, no VRAM-aware routing, no preemption, no record of what ran where                                    |
+| **[task-spooler](https://manpages.ubuntu.com/manpages/noble/man1/tsp.1.html)** | A real job queue — on a single machine                      | jobd queues across _all_ your machines and routes by live VRAM/CPU fit                                         |
+| **Slurm**                                                                      | Datacenter-grade scheduling                                 | Heavy to stand up and operate for 2–3 personal boxes; jobd is one process + a poller per host                  |
+| **SkyPilot / Modal / dstack**                                                  | Provision and run on clouds (SkyPilot also on-prem via SSH) | jobd targets hardware you _already own_, with no cloud/K8s assumptions and a much smaller footprint            |
+| **Ray**                                                                        | A distributed-compute framework                             | jobd is a job _queue_, not a programming model — submit any command, no code changes, GPU-fit routing built in |
+
+Closest in spirit are task-spooler (single-node) and on-prem SkyPilot (heavier, cloud-shaped). jobd's niche is the 2–3-GPU homelab: multi-machine VRAM-fit routing + preempt/checkpoint + a native agent interface, with nothing to stand up.
+
+## Architecture
+
+```mermaid
+flowchart TD
+    CLI["job CLI"]:::client --> B
+    MCP["jobd-mcp<br/>MCP tools"]:::client --> B
+    API["HTTP · SSE"]:::client --> B
+    B["<b>jobd broker</b> — FastAPI<br/>queue · matcher · priorities · SQLite"]:::broker
+    B <-->|poll · dispatch| WA["worker A<br/>24 GB GPU"]:::worker
+    B <-->|poll · dispatch| WB["worker B<br/>8 GB GPU"]:::worker
+    B <-->|poll · dispatch| WC["worker C<br/>CPU-only"]:::worker
+    classDef client fill:#1f2937,stroke:#4b5563,color:#e5e7eb;
+    classDef broker fill:#0e7490,stroke:#155e75,color:#ecfeff;
+    classDef worker fill:#14532d,stroke:#166534,color:#dcfce7;
+```
+
+Workers **poll** the broker (pull model — no inbound connection to a worker); the broker matches each job against live capacity and hands it back on the poll. One broker process, one poller per host.
+
+- **Broker** — a FastAPI + SQLite service. Holds the queue, runs the matcher, resolves per-project priorities and defaults, exposes a small HTTP API and an SSE stream. Single source of truth.
+- **Workers** — lightweight polling agents, one per host. Each advertises live capacity via heartbeat, claims jobs it can run, executes them (`shell=False`, no shell-injection surface), streams logs back, and honors preemption signals.
+- **Clients** — the `job` CLI, the `jobd-mcp` MCP server, or anything that speaks the HTTP API.
+
+## Install
+
+```bash
+pip install jobd            # broker + CLI
+pip install "jobd[mcp]"     # adds the MCP server
+```
+
+Requires Python ≥ 3.11. The `pip` package is the **broker + CLI + MCP server**; the worker is intentionally not bundled (it has host-specific system deps). Workers run from a clone of this repo (`worker/job_worker.py`) — `scripts/install-worker.sh` sets one up under `~/jobd-worker` with its own venv. (Packaging the worker as `jobd-worker` is planned for a later release.)
+
+## Quickstart (single host)
+
+```bash
+# 1. start the broker (binds 127.0.0.1:8765 by default)
+JOBD_ALLOW_NO_AUTH=1 jobd          # no-auth is fine for a loopback-only broker
+
+# 2. in another shell (from a clone of this repo), start a worker pointed at it
+pip install httpx psutil pyyaml pynvml   # worker deps (pynvml only needed for GPU hosts)
+JOBD_URL=http://127.0.0.1:8765 JOBD_WORKER_HOST=local \
+  python worker/job_worker.py
+
+# 3. submit a job and wait for it
+job submit --project demo --wait -- echo hello
+job list
+job logs <id>
+```
+
+For a real multi-host deployment (Docker broker + systemd workers, Tailscale binding, shared auth token), see **[docs/security.md](docs/security.md)** and the templates in `docker-compose.yml`, `scripts/`, and `worker/`. Day-2 operations (health, draining a worker, upgrades, token rotation, backups) are in **[docs/runbook.md](docs/runbook.md)**.
+
+## Supported platforms
+
+Python 3.11+ everywhere.
+
+| Component                              | Linux   | macOS       | Windows              |
+| -------------------------------------- | ------- | ----------- | -------------------- |
+| **Broker** (`jobd`)                    | ✅      | ✅          | ✅ (WSL recommended) |
+| **CLI** (`job`) / **MCP** (`jobd-mcp`) | ✅      | ✅          | ✅                   |
+| **Worker** (`job_worker.py`)           | ✅ full | ⚠️ degraded | ⚠️ degraded          |
+
+The **worker** runs its best on Linux with a systemd user instance: memory caps, process reaping, and preemption use `systemd-run --user` scopes and cgroups. On non-systemd hosts the worker still executes jobs, but silently drops those guarantees — fine for a single trusted box, not for hard resource isolation. GPU features need NVIDIA + `nvidia-ml-py`. The broker, CLI, and MCP server are pure-Python and portable.
+
+## CLI
+
+```
+job submit -p PROJ [--gpu] [--vram-required N] [--needs TAG]... [--wait] -- CMD...
+job list [--state STATE] [--project P]      # queue + recent jobs
+job status ID [--watch]                     # one job, optionally live
+job logs ID [-n BYTES]                      # tail captured output
+job wait ID                                 # block until terminal
+job cancel ID  /  job preempt ID            # stop a job
+job workers                                 # fleet snapshot + health
+job projects list | set NAME PRI | nudge NAME DELTA
+job audit [--project P] [--since 24h]       # event history
+```
+
+`job submit --explain` dry-runs the resolution (priority, profile, project defaults, host pin) and prints the effective config without enqueuing anything.
+
+## MCP / agent integration
+
+jobd ships an MCP server (`jobd-mcp`) exposing the queue as nine tools — `jobd_submit`, `jobd_status`, `jobd_logs`, `jobd_list`, `jobd_cancel`, `jobd_preempt`, `jobd_workers`, `jobd_job_get`, `jobd_worker_delete`. Point your MCP client at it:
+
+```json
+{
+  "mcpServers": {
+    "jobd": {
+      "command": "jobd-mcp",
+      "env": {
+        "JOBD_URL": "http://127.0.0.1:8765",
+        "JOBD_API_TOKEN": "<your-token>"
+      }
+    }
+  }
+}
+```
+
+`JOBD_API_TOKEN` must match the broker's token, or every call returns 401. Omit it only when the broker runs with `JOBD_ALLOW_NO_AUTH=1`.
+
+Now an agent can "run this overnight," check on it next session, and route GPU work through the broker instead of colliding on a shared card. The `examples/claude-code-hooks/` directory has optional [Claude Code](https://docs.claude.com/en/docs/claude-code) hooks that _nudge_ (or hard-block) an agent toward submitting heavy commands through jobd — including a VRAM-aware GPU guard with `# NO_GPU` / `# CONCURRENT_OK` / `# VRAM=NGB` override markers.
+
+## Configuration
+
+Three optional YAML files under `JOBD_CONFIG_DIR` (defaults shipped in `config/`):
+
+- **`projects.yaml`** — per-project base priority and submit defaults (preemptibility, wall/idle timeouts, host pins, capability requirements). See [docs/plans/projects-yaml.md](docs/plans/projects-yaml.md) for the full resolution model.
+- **`profiles.yaml`** — named resource bundles (`--profile gpu-train-large`) the matcher uses to size a job.
+- **`classifier.yaml`** — rules that auto-suggest a profile from the command string.
+
+All three are optional; with none present, every job runs at the global default priority.
+
+## Security
+
+The broker has **no TCP-layer auth beyond a shared bearer token**, so it is meant to run on a trusted network (loopback or a Tailscale tailnet), never on a public interface. Two stacked controls:
+
+1. **Interface binding** — `JOBD_HOST` must be `127.0.0.1` or a Tailscale CGNAT address (`100.64.0.0/10`), never `0.0.0.0`. A CI lint (`tests/test_deploy_lint.py`) enforces this on the Docker deployment.
+2. **Bearer token** — set `JOBD_API_TOKEN` (≥32 random bytes) on every broker/worker/CLI/MCP host. The broker refuses to start without it unless you explicitly set `JOBD_ALLOW_NO_AUTH=1`. **`JOBD_ALLOW_NO_AUTH=1` is for a loopback-only broker (`JOBD_HOST=127.0.0.1`) — for local dev/tests.** Combined with a non-loopback `JOBD_HOST` it exposes an unauthenticated RCE endpoint to your whole tailnet; the broker logs a startup warning if you do this. Don't.
+
+Full threat model, env-var reference, and token rotation: **[docs/security.md](docs/security.md)**.
+
+## License
+
+MIT — see [LICENSE](LICENSE).