jambu-gpu 0.1.0__tar.gz → 0.3.0__tar.gz

{jambu_gpu-0.1.0 → jambu_gpu-0.3.0}/.gitignore

@@ -5,6 +5,7 @@ venv/
 dist/
 build/
 *.egg-info/
+.gpu/
 .jambu/
 .env
 .env.*
@@ -13,3 +14,4 @@ build/
 .coverage
 node_modules/
 *.tsbuildinfo
+.claude/

{jambu_gpu-0.1.0 → jambu_gpu-0.3.0}/CONTRIBUTING.md

@@ -62,9 +62,9 @@ job queues, a web dashboard. See [docs/ACCEPTANCE.md](docs/ACCEPTANCE.md).
 
 ## Secrets
 
-Never commit credentials, `.env`, `.jambu/state.json`, or provider instance
+Never commit credentials, `.env`, `.gpu/state.json`, or provider instance
 ids from a real run. Keys are read from the environment, a gitignored `.env`,
-or `~/.jambu/credentials`. `.env.example` ships with empty values only.
+or `~/.gpu/credentials`. `.env.example` ships with empty values only.
 
 ## Pull requests
 

{jambu_gpu-0.1.0 → jambu_gpu-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: jambu-gpu
-Version: 0.1.0
+Version: 0.3.0
 Summary: Provider-agnostic CLI for provisioning and managing temporary GPU runtimes
 Project-URL: Homepage, https://github.com/jambuai/jambu-gpu
 Project-URL: Repository, https://github.com/jambuai/jambu-gpu
@@ -45,7 +45,11 @@ gpu run python experiment.py
 
 When you are finished, `gpu destroy`. If you walk away, the idle timeout does it.
 
-The API key is read from the environment, a project `.env`, or `~/.jambu/credentials`. It never goes in the manifest.
+From an application in any language: `gpu --json setup` returns the OpenAI-compatible `base_url`, `api_key` and `model`, and the app uses any OpenAI SDK. See [Integrating gpu into an application](docs/INTEGRATION.md), with a ready TypeScript client.
+
+As a last line of defense, `gpu ls` shows every jambu instance in the account and `gpu sweep --install-timer` sets up a systemd user timer that guards every project on the machine by its own rules and cleans up the orphans no project tracks. See [Keeping the account clean](docs/USAGE.md#keeping-the-account-clean).
+
+The API key is read from the environment, a project `.env`, or `~/.gpu/credentials`. It never goes in the manifest.
 
 ## Declare
 
@@ -64,6 +68,7 @@ The file is `jambu.yaml`. This tool reads only the `gpu_runtime:` key, so other
 
 ## Docs
 
+- [Integrating into an application (any language)](docs/INTEGRATION.md)
 - [Commands, workloads, and agents](docs/USAGE.md)
 - [Every config field](docs/CONFIG.md)
 - [Choosing a model](docs/CHOOSING_A_MODEL.md)

{jambu_gpu-0.1.0 → jambu_gpu-0.3.0}/README.md

@@ -22,7 +22,11 @@ gpu run python experiment.py
 
 When you are finished, `gpu destroy`. If you walk away, the idle timeout does it.
 
-The API key is read from the environment, a project `.env`, or `~/.jambu/credentials`. It never goes in the manifest.
+From an application in any language: `gpu --json setup` returns the OpenAI-compatible `base_url`, `api_key` and `model`, and the app uses any OpenAI SDK. See [Integrating gpu into an application](docs/INTEGRATION.md), with a ready TypeScript client.
+
+As a last line of defense, `gpu ls` shows every jambu instance in the account and `gpu sweep --install-timer` sets up a systemd user timer that guards every project on the machine by its own rules and cleans up the orphans no project tracks. See [Keeping the account clean](docs/USAGE.md#keeping-the-account-clean).
+
+The API key is read from the environment, a project `.env`, or `~/.gpu/credentials`. It never goes in the manifest.
 
 ## Declare
 
@@ -41,6 +45,7 @@ The file is `jambu.yaml`. This tool reads only the `gpu_runtime:` key, so other
 
 ## Docs
 
+- [Integrating into an application (any language)](docs/INTEGRATION.md)
 - [Commands, workloads, and agents](docs/USAGE.md)
 - [Every config field](docs/CONFIG.md)
 - [Choosing a model](docs/CHOOSING_A_MODEL.md)

jambu_gpu-0.3.0/docs/ACCEPTANCE.md

@@ -0,0 +1,69 @@
+# Acceptance criteria — where each one is implemented and tested
+
+Eighteen guarantees, mapped to the code and the test that holds each one.
+
+| # | Guarantee | Implementation | Test |
+| --- | --- | --- | --- |
+| 1 | Repeated `setup` does not duplicate infrastructure | `core/engine.py::Engine.setup` reconciles first, compares `config.fingerprint()`, and holds an exclusive `.gpu/runtime.lock` | `test_engine.py::test_setup_twice_does_not_create_two_instances`, `::test_setup_refuses_to_reuse_an_incompatible_instance` |
+| 2 | Provider APIs do not leak into core orchestration | `providers/base.py` contract; all Vast wire format confined to `providers/vast/{client,mapper}.py` | `test_engine.py::test_the_core_never_branches_on_a_provider_name` (greps `core/`) |
+| 3 | Missing credentials fail before provisioning | `VastProvider.validate` returns before any API call; `CredentialResolver` never reads jambu.yaml | `test_vast_provider.py::test_validate_fails_before_provisioning_without_a_key` (asserts zero HTTP calls) |
+| 4 | Failed provisioning cleans up | `Engine.setup` persists state *before* the long wait, then destroys on failure when `cleanup_on_setup_failure` | `test_engine.py::test_failed_setup_cleans_up_partially_created_resources` |
+| 5 | Active workloads are never stopped by the idle guard | `core/policy.py::evaluate` returns `workload_active` before any idle check | `test_policy.py::test_active_workload_is_never_stopped_by_the_idle_guard`, `test_watchdog.py::test_a_heartbeating_workload_is_never_stopped` |
+| 6 | An idle unlocked instance stops within `idle_timeout + watchdog_interval` | remote watchdog loop, `watchdog/agent_body.py::guard_loop` | `test_watchdog.py::test_idle_instance_is_stopped_with_a_recorded_reason` (real subprocess) |
+| 7 | `max_lifetime` bounds forgotten instances | `policy.evaluate` checks age before activity | `test_policy.py::test_max_lifetime_stops_even_when_busy`, `test_watchdog.py::test_max_lifetime_stops_a_busy_instance` |
+| 8 | Locks prevent shutdown until expiry | `policy.Snapshot.lock_active` / `lock_expired`; TTL required unless `allow_indefinite_lock` | `test_policy.py::test_expired_lock_stops_protecting`, `test_watchdog.py::test_a_lock_prevents_shutdown_until_it_expires` |
+| 9 | A crashed CLI does not disable protection | the guard runs *on the instance*; heartbeats go stale after `heartbeat_grace` | `test_watchdog.py::test_a_vanished_heartbeat_falls_back_to_idle`, `::test_startup_grace_expires_when_the_cli_never_confirms` |
+| 10 | Remote state is reconciled | `Engine._reconcile` merges provider truth + watchdog truth and repairs local state | `test_engine.py::test_status_repairs_state_when_the_instance_vanished` |
+| 11 | A new provider needs only the contract | `providers/registry.py`; the engine is driven end to end by `tests/fakes.py::FakeProvider` | `test_engine.py::test_a_new_provider_needs_only_the_contract` |
+| 12 | Every automatic shutdown records its reason | `lifecycle_guard_triggered` + `instance_stopped` events carry `reason`, locally and on the instance | `test_engine.py::test_guard_stops_an_idle_unlocked_instance_and_records_the_reason`, `test_watchdog.py::test_idle_instance_is_stopped_with_a_recorded_reason` |
+| 13 | A restarted instance stays guarded | `Engine._reconcile` takes running/stopped from the provider; the watchdog's `terminated` flag only counts when the provider cannot tell. `RuntimeState.session_started_at` restarts on every start, and `max_lifetime` and the session budget are measured from it | `test_engine_sessions.py::test_reconcile_trusts_the_provider_over_a_stale_terminated_flag`, `::test_a_stale_terminated_flag_does_not_disable_the_guard`, `::test_restarting_a_stopped_instance_starts_a_new_session`, `::test_guard_budget_resets_with_a_new_session`; watchdog-side session tests in `test_policy.py` / `test_watchdog.py` |
+| 14 | Traffic on the model endpoint is activity | `VllmRuntime.activity_probe` hands the watchdog vLLM's `/metrics` gauges and counters; `lifecycle.traffic_counts_as_activity` turns it off | `test_engine_sessions.py::test_the_watchdog_is_told_how_to_read_vllm_traffic`, `::test_traffic_can_be_excluded_from_activity`, `test_runtime_vllm.py::test_activity_probe_reads_vllm_metrics_on_loopback`; the watchdog's metrics-reading tests in `test_watchdog.py` |
+| 15 | A stopped instance does not bill storage forever | `lifecycle.stopped_ttl` -> `PolicyConfig.stopped_ttl_s`; `RuntimeState.stopped_since`; `Engine.guard_tick` destroys with reason `stopped_ttl`; `validate` warns when nothing would remove it | `test_engine_sessions.py::test_guard_destroys_an_instance_stopped_longer_than_stopped_ttl`, `::test_validate_warns_when_nothing_removes_a_billing_stopped_instance`; the stopped-TTL policy tests in `test_policy.py` |
+| 16 | Orphans are found account-wide, each project keeps its own rules, and nothing else is touched | `GPUProvider.list_instances` (capability `list_instances`) returns only `jambu-` labels; `gpu setup` registers the project in `~/.gpu/projects.json`; `core/sweep.py::guard_projects` runs each registered project's own `Engine.guard_tick`; only untracked instances are orphans: running past `--orphan-max-age` is stopped (never destroyed), stopped past `--stopped-ttl` (from the sweep's own first sighting in `~/.gpu/sweep.json`) is destroyed; `gpu sweep --install-timer` schedules one working-directory-independent unit and refuses a key that is only in the shell | `test_sweep.py::test_vast_lists_only_jambu_labelled_instances`, `::test_non_jambu_instances_are_never_touched`, `::test_another_projects_restarted_and_locked_instance_is_not_destroyed`, `::test_each_project_is_judged_by_its_own_max_lifetime`, `::test_a_running_orphan_past_orphan_max_age_is_stopped_never_destroyed`, `::test_a_stopped_orphan_is_destroyed_once_seen_stopped_past_stopped_ttl`, `::test_install_timer_writes_one_unit_independent_of_the_project`, `::test_install_timer_refuses_a_key_only_in_the_shell_environment` |
+| 17 | The CLI guard never stops a serving endpoint it cannot see | `Engine._idle_blind`: with the watchdog enabled and installed but unreachable, `guard_tick` evaluates without `idle_timeout` and records `idle_skipped`; `max_lifetime`, budget and `stopped_ttl` still apply | `test_engine_sessions.py::test_guard_leaves_idle_to_an_unreachable_watchdog`, `::test_max_lifetime_still_applies_when_the_watchdog_is_unreachable` |
+| 18 | A fresh stop or a short API outage never costs the disk | `RuntimeState.mark_stopped` restarts `stopped_since` on evidence the instance ran since; the watchdog escalates a failing stop to destroy only after `escalate_after_s` (30m) of continuous failure and resets on renewed use | `test_engine_sessions.py::test_a_stale_stopped_since_does_not_destroy_a_fresh_stop`, `test_watchdog.py::test_an_api_blip_does_not_destroy_the_disk`, `::test_failures_reset_when_the_instance_is_back_in_use` |
+
+## Validated on real hardware
+
+On 2026-10-24 the lifecycle ran end to end on Vast.ai (RTX 3090, Qwen2.5-0.5B, `idle_timeout: 4m`, `stopped_ttl: 5m`), for $0.35 of Vast credit in total across every attempt:
+
+| What was checked | What happened |
+| --- | --- |
+| Traffic is activity | 10 requests sent straight to the endpoint over 7.5 minutes, none through `gpu run`. The machine stayed up past the 4-minute idle timeout. |
+| Idle stops | 269 s after the last request the watchdog stopped its own machine (`idle_timeout`). Vast showed it `exited`. |
+| A restart is guarded | `gpu run` restarted it. The watchdog logged `session_started` with `terminated: false` and measured `max_lifetime` from the restart. After the workload it stopped the machine again on its own. |
+| Stopped storage is reclaimed | 334 s after the stop, `gpu guard` destroyed it (`stopped_ttl`). |
+| Failed setups leave nothing | An image built for a newer CUDA, a model server that died at import, a host whose network dropped the weights download, a 30-minute image pull, and a Ctrl-C each ended with the instance destroyed. |
+| Bad hosts are skipped | Hosts that could not pull the image were destroyed, excluded and replaced, with never two instances alive at once. |
+
+The run also found what unit tests could not, and each of these is fixed and covered by a test:
+- the boot script was over Vast's 16 KB onstart limit;
+- `latest` had moved to CUDA 13;
+- the `-cu129` image variant could not load torchcodec;
+- a 15-minute boot timeout was too short for a 12 GB pull;
+- Ctrl-C did not clean up;
+- a fresh contract was reported `READY`.
+
+### Application integration (0.2.0)
+
+Driven end to end by [`examples/typescript-client`](../examples/typescript-client/) through the OpenAI SDK, on an RTX 3090 with Qwen2.5-0.5B:
+
+| Check | Result |
+| --- | --- |
+| `ensureGpu()` on an empty account | Two bad hosts (weights download cut by the host's network; image pull stalled 30 minutes) were destroyed and replaced; the third served. 37 minutes in all, one instance alive at a time. |
+| The endpoint requires its key | No key: `401`. Wrong key: `401`. The key from `gpu --json setup`: `200`. `/health` without a key: `200`. |
+| Chat through the SDK | Answered. |
+| Healing after a stop | `stopGpu()`, then a call on the stale connection: it failed to connect, `gpu setup` restarted the same instance, and the retry answered 41 s later with the same key. |
+
+It found two more bugs, both fixed and tested: Vast reports a stopping instance as `running` for a few seconds, which made a setup right after a stop skip the restart; and a Vast rate limit (`429`) failed setup instead of being waited out.
+
+## The MVP command surface
+
+`jambu.yaml` schema, CLI, provider contract, provider registry, Vast.ai adapter,
+vLLM runtime, `setup`, `run`, `status`, `stop`, `destroy`, `lock`/`unlock`, idle
+timeout, max lifetime, remote watchdog, structured logs — all present. On top of
+it, `ls` and `sweep` for account-wide orphan control.
+
+Deliberately not implemented: multi-instance scheduling, multi-GPU
+orchestration, Kubernetes, distributed inference, provider optimization, automatic
+provider selection, job queues, web dashboard.

{jambu_gpu-0.1.0 → jambu_gpu-0.3.0}/docs/ADDING_A_PROVIDER.md

@@ -52,7 +52,12 @@ providers/runpod/
 
 The core never branches on `provider.name` — only on `provider.capabilities`. Set
 `stop_eliminates_billing=False` if a stopped instance still costs money; the policy
-then knows it may need to escalate to destroy.
+then knows it may need to escalate to destroy, and `lifecycle.stopped_ttl` applies.
+
+Set `list_instances=True` and implement `list_instances()` if the API can enumerate
+the account. Return only instances whose label starts with `JAMBU_LABEL_PREFIX`
+(`core/models.py`); `gpu ls` and `gpu sweep` are built on it and stay unavailable
+otherwise.
 
 ## 3. Normalize state and errors
 
@@ -63,6 +68,12 @@ and raise only the normalized errors from `core/errors.py`:
 `ProviderTimeoutError`, `UnsupportedCapabilityError`. CLI behaviour must be
 identical across providers.
 
+Report a machine that will not boot as `failed` rather than leaving it in
+`starting`: `setup` then destroys it at once and rents another machine
+(`compute.host_attempts`). Set `Offer.host_id` and `Instance.host_id` to the
+physical machine when the API exposes it, and leave out offers whose `host_id` is
+in `ComputeSpec.exclude_hosts`, or `setup` may rent the same bad machine again.
+
 ## 4. Supply the watchdog actions
 
 The lifecycle *policy* is the core's; the adapter contributes only the two calls

{jambu_gpu-0.1.0 → jambu_gpu-0.3.0}/docs/CONFIG.md

@@ -64,8 +64,8 @@ for why this exists and `gpu profiles` / `gpu inspect-model
 | `runtype` | `ssh_direc ssh_proxy` | Vast container run mode. |
 | `verified_only` | `true` | Only rent verified machines. |
 | `min_reliability` | `0.95` | Minimum machine reliability score. |
-| `min_inet_down_mbps` | `100` | Minimum download bandwidth (model pulls are big). |
-| `min_cuda` | `12.1` | Minimum `cuda_max_good`. |
+| `min_inet_down_mbps` | `500` | Minimum download bandwidth. The vLLM image alone is ~10 GB and a slow pull is the commonest boot failure; most offers clear 500 at the same price. |
+| `min_cuda` | `12.1` | Floor for the host's `cuda_max_good` (the newest CUDA its driver runs). The search uses the higher of this and `compute.gpu.cuda_min`, and offers are re-checked client-side. |
 | `search_limit` | `24` | How many offers to consider. |
 | `order_by` / `order_dir` | `dph_total` / `asc` | Offer ranking — cheapest first. |
 | `image_login` | – | `-u user -p pass docker.io` for a private image. |
@@ -85,11 +85,12 @@ each number comes from.
 | `gpu.min_vram_gb` | `24` | Minimum VRAM **per GPU**. |
 | `gpu.count` | `1` | Number of GPUs on one machine. |
 | `gpu.name_filter` | – | Substring match, e.g. `A100`, `RTX 4090`. |
-| `gpu.cuda_min` | – | Minimum CUDA version. |
+| `gpu.cuda_min` | – | Minimum host CUDA version. `gpu` raises it to what the runtime image needs, read from its tag (`-cu129` means 12.9, vLLM's bare `v0.32+` and `latest` mean 13.0), so an image never lands on a driver too old to run it. |
 | `disk_gb` | `80` | Disk. Must fit the model weights plus the container. |
 | `region` | – | Comma-separated geolocations. |
 | `spot` | `false` | Use interruptible/bid capacity where supported. |
-| `image` | – | Override the runtime's container image. |
+| `host_attempts` | `3` | How many machines `setup` tries. A host that fails to boot (the provider reports it failed, it vanishes, or it is not running within `health.boot_timeout`) is destroyed, replaced from fresh offers (each held to the budget), and remembered in `$GPU_HOME/bad_hosts.json` so no setup rents it again for 24h. A model server that dies once its container runs fails at once, since the same config would fail on any machine, unless its log shows the host's network failed it (a dropped connection or timeout while downloading weights): that counts as a host failure too. Needs `lifecycle.cleanup_on_setup_failure`; without it the first failed host is kept for inspection. |
+| `image` | – | Override the runtime's container image. vLLM's default is `vllm/vllm-openai:v0.32.0`. Pin a tag: `validate` warns on `latest` or an untagged image, because the CUDA it needs can change under you. On Vast, `vastai/vllm:v0.32.0-cuda-13.0` (the same vLLM, packaged by Vast) is often cached on the host and boots in minutes where a 12 GB Docker Hub pull can stall. |
 
 ---
 
@@ -117,7 +118,7 @@ How the model runs. Independent from where it runs.
 | `dtype` | `auto` | vLLM dtype. |
 | `gpu_memory_utilization` | `0.90` | Fraction of VRAM vLLM may claim. |
 | `tensor_parallel_size` | `compute.gpu.count` | Shards across GPUs. |
-| `api_key` | – | Require this key on the model endpoint. |
+| `api_key` | generated | The key the model endpoint requires. Unset, `setup` generates a random one per instance; `gpu env` prints it. |
 | `vllm_args` | `{}` | Structured escape hatch for any vLLM flag this schema doesn't model: `{flag: value}` → `--flag value` (`true`/`false` → a bare/negated flag, a dict or list → JSON-encoded). Applied before `extra_args`. See [`docs/CHOOSING_A_MODEL.md`](CHOOSING_A_MODEL.md#dynamic--un-modeled-vllm-flags). |
 | `extra_args` | `[]` | Raw argv appended verbatim, last — for anything `vllm_args` can't express. |
 | `tool_calling` | `auto` | `auto` guesses `--tool-call-parser` from `model.id` and turns tool calling on if recognized; `on` requires a resolvable parser (fails `validate` otherwise); `off` never passes tool-calling flags. |
@@ -162,6 +163,7 @@ The cost protection. See [the safety invariant](#safety-invariant).
 | --- | --- | --- |
 | `max_lifetime` | `6h` | Hard ceiling on instance age. `null` = unbounded. |
 | `idle_timeout` | `15m` | Shut down after this long with no activity. `null` = unbounded. |
+| `stopped_ttl` | `24h` | Destroy an instance that has been `STOPPED` this long. A stopped Vast instance no longer bills GPU time but still bills storage. `null` keeps stopped instances forever. |
 | `auto_stop` | `true` | The guard may stop the instance. |
 | `auto_destroy` | `false` | The guard destroys instead of stopping. |
 | `cleanup_on_setup_failure` | `true` | Destroy partially created resources when setup fails. |
@@ -169,6 +171,7 @@ The cost protection. See [the safety invariant](#safety-invariant).
 | `heartbeat_interval` | `30s` | How often a running workload reports in. |
 | `heartbeat_grace` | `5m` | Missing heartbeats tolerated before a workload is stale. |
 | `ssh_counts_as_activity` | `false` | An SSH session alone is not useful work. |
+| `traffic_counts_as_activity` | `true` | Requests served by the model runtime count as activity. The watchdog reads vLLM's `/metrics` on the instance (requests running or waiting, completed requests, prompt tokens), so an endpoint in use by something other than `gpu run` is not idled out. `false` = only workloads, startup and locks count. |
 | `lock.enabled` | `false` | Whether locks are offered in this project. |
 | `lock.default_ttl` | `2h` | TTL used by `gpu lock` with no `--for`. |
 | `watchdog.enabled` | `true` | Install the independent remote guard. |
@@ -180,6 +183,19 @@ otherwise.
 
 `idle_timeout` may not exceed `max_lifetime`.
 
+`max_lifetime` and `budget.max_session_cost_usd` are measured per **session**: from
+creation, and again from every restart of a stopped instance. A restart gets a fresh
+bound instead of being killed for the hours it spent stopped.
+
+`stopped_ttl` counts from when the instance was first observed stopped. On a
+provider where a stopped instance still bills, `validate` warns when `stopped_ttl`
+is `null` and `auto_destroy` is `false`: nothing would ever remove the instance.
+
+`gpu sweep` guards every project registered by `gpu setup` with that project's own
+`max_lifetime`, lock and `stopped_ttl`. Instances no project tracks are orphans,
+judged by the sweep's own `--orphan-max-age` and `--stopped-ttl` flags. See
+[Keeping the account clean](USAGE.md#keeping-the-account-clean).
+
 ---
 
 ## `health`
@@ -187,7 +203,8 @@ otherwise.
 | Field | Default | Meaning |
 | --- | --- | --- |
 | `interval` | `60s` | Probe interval while waiting for the runtime. |
-| `startup_timeout` | `10m` | Give up if the model server never answers. Raise it for very large models. |
+| `boot_timeout` | `30m` | How long the machine may take to reach running, mostly the image pull. Past it the host counts as failed and `setup` tries another (`compute.host_attempts`). |
+| `startup_timeout` | `10m` | Once the machine runs, how long the model server may take to answer. Raise it for very large models. |
 | `path` | `/health` | Health path on the model endpoint. |
 
 A passing health check is **not** activity. A healthy but unused server still goes idle.
@@ -199,7 +216,7 @@ A passing health check is **not** activity. A healthy but unused server still go
 | Field | Default | Meaning |
 | --- | --- | --- |
 | `max_hourly_cost_usd` | – | Refuse to provision above this hourly price. |
-| `max_session_cost_usd` | – | The remote guard shuts down once accrued cost passes this. |
+| `max_session_cost_usd` | – | The remote guard shuts down once the current session's cost (session age x hourly price) passes this. |
 
 Override once with `gpu setup --allow-cost-override`.
 

jambu_gpu-0.3.0/docs/INTEGRATION.md

@@ -0,0 +1,232 @@
+# Integrating gpu into an application
+
+How any application, in any language, uses a GPU that `gpu` rents, serves and shuts down. No SDK is needed: the app talks to two things it already knows how to use.
+
+| Plane | What | How the app uses it |
+| --- | --- | --- |
+| Control | the `gpu` command | runs it as a subprocess with `--json` and reads stdout and the exit code |
+| Data | vLLM's OpenAI-compatible HTTP API | any OpenAI SDK, or plain HTTP, with the base URL, key and model `gpu` printed |
+
+The app never has to stop the GPU. The watchdog on the machine stops it once nothing has used it for `lifecycle.idle_timeout`, and every inference request counts as use. The app's only duty is to call `gpu setup` again when the machine it was using is gone.
+
+## 1. Prepare the host the app runs on
+
+```bash
+pipx install jambu-gpu            # or: uv tool install jambu-gpu / pip install jambu-gpu
+cd /srv/my-app                    # the directory the app runs gpu from
+gpu init --model Qwen/Qwen2.5-7B-Instruct
+echo "VAST_API_KEY=..." >> .env   # or export it, or ~/.gpu/credentials
+gpu validate                      # fails here, not in production, on a bad config
+gpu sweep --install-timer         # optional safety net, see USAGE.md
+```
+
+Requirements:
+
+- `jambu-gpu` 0.3.0 or later, on Python 3.10+. The app can be in any language.
+- The directory holding `jambu.yaml` must be **persistent**. `gpu` keeps what it rented in `.gpu/state.json` next to it. In a container, mount it as a volume: a container that loses `.gpu/` forgets its instance, rents a second one, and leaves the first to the watchdog and the sweep.
+- Before 0.3.0 these directories were `.jambu/` and `~/.jambu`. The first command after an upgrade renames them, so an instance already rented is kept, not rented again.
+- `.gpu/state.json` holds the endpoint key and the watchdog token. It is written `0600`. Add `.gpu/` and `.env` to `.gitignore` and keep them out of images.
+
+Tune the lifecycle in `jambu.yaml` for the app. For a service with bursty traffic, a longer idle timeout avoids paying a restart on every burst:
+
+```yaml
+gpu_runtime:
+  lifecycle:
+    idle_timeout: 20m     # stop after 20 minutes without a request
+    max_lifetime: 12h     # hard cap per session, even under traffic
+    stopped_ttl: 24h      # destroy a machine that stayed stopped this long
+  budget:
+    max_hourly_cost_usd: 0.60
+```
+
+All fields are in [CONFIG.md](CONFIG.md).
+
+## 2. The contract
+
+### Commands an app calls
+
+Always pass `--json` (before the command name). Run them from the project directory, or pass `--config /path/to/jambu.yaml`.
+
+| Command | Use it to | Blocks | Starts billing |
+| --- | --- | --- | --- |
+| `gpu --json setup` | get a working endpoint: rents a machine if there is none, restarts it if it was stopped, waits until the model answers | until ready | yes |
+| `gpu --json env` | read the endpoint without starting anything | no | no |
+| `gpu --json status` | full state, cost so far, and what the guard would do now | no | no |
+| `gpu --json stop` | stop it now instead of waiting for the idle timeout; disk and model cache are kept | until stopped | stops it |
+| `gpu --json destroy --yes` | delete it; nothing is billed afterwards | until gone | stops it |
+| `gpu --json lock --for 2h` | keep it up through a quiet period (a long job, a demo) | no | |
+| `gpu --json unlock` | return to the normal rules | no | |
+
+`setup` is idempotent and serialized by a file lock: ten processes calling it at once rent one machine, and the other nine wait and get the same answer.
+
+### What `setup` and `env` print
+
+```json
+{
+  "instance_id": "27581234",
+  "state": "ready",
+  "endpoint": "http://203.0.113.7:41022",
+  "base_url": "http://203.0.113.7:41022/v1",
+  "api_key": "k6V0...43 characters",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "hourly_cost_usd": 0.196
+}
+```
+
+| Field | Meaning |
+| --- | --- |
+| `base_url` | Pass as the OpenAI SDK's `baseURL` / `base_url` |
+| `api_key` | Pass as the SDK's `apiKey`. Requests without it get `401` |
+| `model` | Pass as `model` in every request (`model.served_name` when set, otherwise `model.id`) |
+| `state` | `ready` after `setup`. `env` reports whatever the last command saw, without asking the provider |
+| `instance_id` | Changes when a new machine is rented |
+
+`setup` also prints a `watchdog` object (URL, port, last contact; never its token). `env` exits `3` with `"state": "none"` when there is no endpoint.
+
+`gpu env` without `--json` prints shell exports instead, for scripts:
+
+```bash
+eval "$(gpu env)"     # OPENAI_BASE_URL, OPENAI_API_KEY, JAMBU_GPU_ENDPOINT, JAMBU_MODEL_ID
+gpu env --format dotenv > .env.gpu
+```
+
+### Output rules
+
+- With `--json`, stdout carries exactly one JSON document and nothing else. Progress lines are suppressed.
+- Errors go to stderr as one line, `error: <message>`, with a non-zero exit code.
+
+### Exit codes
+
+| Code | Meaning | What the app should do |
+| --- | --- | --- |
+| 0 | success | |
+| 1 | unexpected error | log stderr, retry later |
+| 2 | config error (`jambu.yaml` invalid, or changed since setup) | fix the config; `setup --recreate` replaces a machine built from the old one |
+| 3 | no instance / no endpoint (from `env`, `endpoint`) | call `setup` |
+| 4 | provider API error | retry with backoff |
+| 5 | provider authentication failed | fix `VAST_API_KEY` |
+| 6 | no capacity matches the config | retry later, or relax `compute.gpu` / the budget |
+| 7 | provisioning failed | retry; the failed machine was already destroyed |
+| 8 | provider timed out | retry with backoff |
+| 9 | the provider cannot do this | change the config |
+| 10 | over `budget.max_hourly_cost_usd` | raise the budget or wait for cheaper offers |
+| 11 | the model server did not start (setup failed, or every host tried failed to boot) | read stderr: it carries the server's log tail; the machine was destroyed |
+
+### Lifecycle the app has to expect
+
+| Event | What the app sees | What it does |
+| --- | --- | --- |
+| first `setup` | blocks while a machine is rented, the image pulled and the model downloaded: usually 5–15 minutes. A bad host (stalled image pull, broken network) is destroyed and replaced automatically, each costing up to `health.boot_timeout` (30 min), at most `compute.host_attempts` (3) times | wait; do not put a short timeout on `setup`, it bounds itself |
+| idle for `idle_timeout` | the watchdog stops the machine; requests fail with connection refused or a timeout | call `setup` again and retry the request |
+| `setup` on a stopped machine | it is restarted: no new rent, no model download, under a minute for a small model (41 s measured) | wait, then use the **new** `base_url`: the provider may assign a new IP and port on restart |
+| stopped for `stopped_ttl` | the machine is destroyed to stop the storage bill | the next `setup` rents a new one (and a new key) |
+| `max_lifetime` reached | stopped even under traffic | same as idle |
+
+The key is per machine: it survives stops and restarts, and changes only when a new machine is rented. Always take `base_url`, `api_key` and `model` from the latest `setup` output instead of caching them across restarts.
+
+## 3. Patterns
+
+### A short job: wrap it
+
+The job gets the connection as environment variables and its heartbeats keep the machine up while it runs:
+
+```bash
+gpu run node dist/batch.js
+gpu run python score.py --input data.jsonl
+```
+
+`gpu run` sets `OPENAI_BASE_URL`, `OPENAI_API_KEY`, `JAMBU_GPU_ENDPOINT` and `JAMBU_MODEL_ID` ([USAGE.md](USAGE.md#running-a-workload)). Any OpenAI SDK reads the first two by itself.
+
+### A long-running service: ensure, then heal
+
+1. On the first request (or at boot), call `gpu --json setup` and keep the result.
+2. Send requests with any OpenAI client, with a client timeout that is not too long (1–2 minutes), so a stopped machine fails fast.
+3. When a request fails with a **connection** error (refused, reset, timeout), call `gpu --json setup` again and retry once with the new result. Do not do this for HTTP errors like `400` or `401`: those come from a live server.
+4. Share one in-flight `setup` between concurrent requests.
+
+The TypeScript helper below does exactly this.
+
+### A batch that must not be interrupted
+
+`gpu lock --for 3h` before the batch and `gpu unlock` after it. Traffic already counts as activity, so a lock only matters if the batch has quiet stretches longer than `idle_timeout`.
+
+## 4. Recipes
+
+### TypeScript / Node
+
+[`examples/typescript-client/src/jambuGpu.ts`](../examples/typescript-client/src/jambuGpu.ts) is a single file with no dependencies beyond Node 18+. Copy it into the app.
+
+```ts
+import OpenAI from "openai";
+import { Gpu } from "./jambuGpu.ts";
+
+const gpu = new Gpu({ cwd: "/srv/my-app" });   // where jambu.yaml lives
+
+const reply = await gpu.call((c) =>
+  new OpenAI({ baseURL: c.baseURL, apiKey: c.apiKey, maxRetries: 0, timeout: 120_000 })
+    .chat.completions.create({ model: c.model, messages: [{ role: "user", content: "hi" }] }),
+);
+```
+
+`gpu.call` starts the GPU on first use, shares one `setup` between concurrent calls, and restarts the GPU and retries once when the call fails because the machine is gone. The file also exports `ensureGpu`, `currentGpu`, `stopGpu`, `destroyGpu`, `gpuStatus`, `GpuError` (with `exitCode`) and the `EXIT` codes. [`examples/typescript-client/`](../examples/typescript-client/) runs it end to end:
+
+```bash
+npm --prefix examples/typescript-client install
+node --experimental-strip-types examples/typescript-client/src/index.ts "Say hi"
+```
+
+### Python
+
+```python
+import json, subprocess
+from openai import OpenAI, APIConnectionError
+
+def ensure_gpu(cwd="/srv/my-app"):
+    out = subprocess.run(["gpu", "--json", "setup"], cwd=cwd, check=True,
+                         capture_output=True, text=True).stdout
+    return json.loads(out)
+
+conn = ensure_gpu()
+def ask(prompt):
+    global conn
+    for attempt in (1, 2):
+        client = OpenAI(base_url=conn["base_url"], api_key=conn["api_key"], max_retries=0, timeout=120)
+        try:
+            return client.chat.completions.create(
+                model=conn["model"], messages=[{"role": "user", "content": prompt}]
+            ).choices[0].message.content
+        except APIConnectionError:
+            if attempt == 2:
+                raise
+            conn = ensure_gpu()   # stopped while idle: restart it, then retry
+```
+
+### Shell / curl
+
+```bash
+eval "$(gpu --json setup >/dev/null && gpu env)"
+curl -s "$OPENAI_BASE_URL/chat/completions" \
+  -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \
+  -d "{\"model\": \"$JAMBU_MODEL_ID\", \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}]}"
+```
+
+### Any other language
+
+Run `gpu --json setup` as a subprocess, parse the JSON, and point the language's OpenAI client (or HTTP client) at `base_url` with `Authorization: Bearer <api_key>`. On a connection error, run it again and retry once.
+
+## 5. Security
+
+- **The endpoint requires a key.** Unless `runtime.api_key` is set, `setup` generates one per machine (43 random URL-safe characters) and starts vLLM with it. Requests without it get `401`. `/health` and `/metrics` stay open: they reveal no data and the watchdog reads `/metrics`. Instances provisioned before this existed run without a key until they are recreated.
+- **The endpoint is plain HTTP** on a public IP and port. The key and the prompts travel unencrypted. For sensitive data, put the app and the GPU on a private network or a tunnel, or front the endpoint with a TLS proxy.
+- **The watchdog holds the provider key** on the instance so it can stop its own machine. A restricted key is the safer choice; see [LIFECYCLE.md](LIFECYCLE.md).
+- `jambu.yaml` never holds credentials. `.env` and `.gpu/` do not belong in version control.
+
+## 6. Troubleshooting
+
+| Symptom | Look at |
+| --- | --- |
+| `setup` fails with exit 11 | stderr has the model server's last log lines; `gpu logs --source runtime` |
+| `setup` slow on the first call | `gpu logs --source onstart`; on Vast, `compute.image: vastai/vllm:v0.32.0-cuda-13.0` pulls faster than Docker Hub's `vllm/vllm-openai` |
+| `401` from the endpoint | the key changed (a new machine was rented): take it from a fresh `setup` |
+| the machine stopped while the app was busy | `gpu events`; check `idle_timeout` against the longest silence between requests, or `gpu lock` |
+| unsure what is still billing | `gpu ls` lists every jambu instance in the account |

jambu_gpu-0.3.0/docs/LIFECYCLE.md

@@ -0,0 +1,128 @@
+# How shutdown works
+
+The CLI process cannot be responsible for idle shutdown. Close the laptop and a local watchdog dies while the GPU keeps billing. `setup` installs a watchdog on the instance itself:
+
+```
+CLI ──provisions──> instance
+                      │
+                      ├── supervisor (bash respawn loop)
+                      │     └── watchdog (HTTP :8777, authenticated)
+                      │           ├── holds the activity and heartbeat state
+                      │           ├── reads vLLM /metrics for inference traffic
+                      │           ├── evaluates the lifecycle policy every 60s
+                      │           └── calls the provider API to stop or destroy itself
+                      └── vLLM :8000
+```
+
+The watchdog script is generated at provision time. The lifecycle policy source is inlined from `jambu_gpu/core/policy.py`, so the remote guard and `gpu status` apply the same rule. The provider adapter injects only the two calls that stop or destroy the machine.
+
+An instance stays billable only while:
+
+1. a workload is running (heartbeat alive), or
+2. a workload is queued, or the model is still starting, or
+3. the model server is serving requests, or
+4. an explicit lock is active.
+
+Anything else becomes `STOPPED` or `DESTROYED`, with the reason recorded.
+
+Activity is a running workload, a queued workload, model startup, inference traffic, or an active lock. A healthy vLLM server, a health check, a metrics scrape, and an open SSH session are not activity. If they were, an idle server would stay up forever.
+
+A crashed CLI, a dropped SSH session, or a sleeping laptop stops the heartbeat. After `lifecycle.heartbeat_grace` the workload is stale, the instance goes idle, and the normal idle rules apply.
+
+## Inference traffic is activity
+
+An agent that calls the vLLM endpoint directly, without `gpu run`, is still using the machine. The watchdog reads vLLM's Prometheus endpoint (`http://127.0.0.1:8000/metrics`) every `min(interval, 15s)` and marks activity when:
+
+- a gauge is above zero: `vllm:num_requests_running`, `vllm:num_requests_waiting`, or
+- a counter grew since the last successful read: `vllm:request_success_total`, `vllm:prompt_tokens_total`.
+
+The first read only sets the baseline, so totals left over from earlier traffic do not keep the machine up. A counter that goes down means vLLM restarted; it rebaselines and does not count. A probe error is never activity, and it is logged as `activity_probe_error` at most once every 10 minutes. `GET /state` shows the last read under `last_probe` (`ok`, `values`, `last_traffic_at`).
+
+Traffic only feeds the idle timer. It does not override `max_lifetime`, the session budget, or locks.
+
+## Sessions and restarts
+
+A Vast.ai stop keeps the container disk, and the onstart script runs again on every container start. The watchdog state in `/var/lib/jambu/activity.json` therefore survives a stop/start, including the flag that says the instance was already stopped.
+
+The onstart script sets `JAMBU_SESSION_STARTED_AT=$(date +%s)` once per container start and passes it to the watchdog. On boot the watchdog compares it with the stored `session_started_at`:
+
+- **Different** (the instance was restarted): new session. The guard is re-armed, startup grace starts over, `last_activity` is now, workloads left running or queued are marked failed, and retry counters reset. Logged as `session_started`.
+- **Equal** (the supervisor respawned a crashed watchdog in the same container start): state is kept.
+
+`max_lifetime`, the session budget (`hourly_cost × session_age`) and startup grace are measured from the session start, not from creation. A restarted instance gets a fresh lease and is not killed the moment it boots. `GET /state` reports both `age_seconds` (since creation) and `session_age_seconds`.
+
+## Retries and escalation
+
+A failed provider call never disarms the guard. A transient API error is logged as `provider_error`, then `guard_retry` with the attempt count, and the guard tries again on its next tick.
+
+When stops have failed continuously for `escalate_after_s` (watchdog config, default 30 minutes), the watchdog destroys the instance instead, if the provider supports destroy (reason `stop_failed_escalation`). A machine that bills forever costs more than a lost disk, but a few minutes of provider API outage is not worth the disk. The failure count and clock reset after a successful call and whenever the decision stops being a shutdown (the instance is in use again), so failures from separate idle spells never add up. A completed manual shutdown clears its pending request.
+
+A successful call does not end the loop either. If the provider accepted the call but the machine is still running after `max(300s, 5 × interval)`, the watchdog issues it again (`guard_reissue`). A failed manual `POST /shutdown` is retried the same way.
+
+## The supervisor
+
+The watchdog runs inside a bash respawn loop started with `nohup`. If the watchdog crashes, the loop logs its exit code to `/var/log/jambu/watchdog.log` and starts it again 5 seconds later. `/var/run/jambu/watchdog.pid` holds the loop's pid; sending it `TERM` stops the loop and the watchdog together. The watchdog still starts before the model server.
+
+## Stopped instances
+
+A stopped Vast.ai instance still bills for storage, and its watchdog is not running. Set `lifecycle.stopped_ttl` and the policy destroys an instance that has been stopped longer than that (reason `stopped_ttl`). This needs destroy support and no active lock. `gpu guard` enforces it from outside, since nothing runs on a stopped machine.
+
+`stopped_since` keeps the first time the CLI saw the instance stopped, but only while nothing says it ran since. A session start, local activity, watchdog contact or a provider start time newer than `stopped_since` means the instance was restarted and stopped again while the CLI was not looking (from the web console, say), so the clock restarts at the current observation instead of destroying a fresh stop.
+
+## A second path
+
+Run the same rule from anywhere that has API access:
+
+```bash
+*/5 * * * * cd /path/to/experiment && gpu guard
+```
+
+`guard` evaluates the policy against reconciled state and enforces it. `gpu sweep --install-timer` runs it for every registered project; see [Keeping the account clean](USAGE.md#keeping-the-account-clean).
+
+Inference traffic is counted on the instance only. When the watchdog is enabled and installed but did not answer during this reconcile, `guard` therefore does not apply `idle_timeout`: its local idle clock would stop an endpoint that is busy serving requests. `max_lifetime`, the session budget and `stopped_ttl` still apply, and the decision detail records `idle_skipped: watchdog_unreachable`. With `lifecycle.watchdog.enabled: false` the CLI guard owns the idle rule as before.
+
+## A model server that dies at boot
+
+The runtime start command writes the model server's pid to `/var/run/jambu/vllm.pid`. `GET /state` reports `runtime_alive`: `true` or `false` from that pid, `null` while there is no pid file yet (or only one older than the current container start). If the process is gone while the instance is still starting, the watchdog logs `runtime_exited` once and ends startup, so the idle rules apply instead of a startup grace (`boot_timeout` + `startup_timeout` + 5 minutes) kept open by a dead server.
+
+`gpu setup` asks the watchdog on every readiness poll. When `runtime_alive` is `false` it fails at once with the last 30 lines of the model server's log in the error (a CUDA driver mismatch, an out-of-memory, a bad flag), and `cleanup_on_setup_failure` destroys the instance.
+
+`GET /logs?name=runtime|onstart|watchdog&lines=N` (authenticated, default 200, at most 2000) serves the tail of those files without SSH. `gpu logs` reads it.
+
+## The watchdog credential
+
+The watchdog stops its own machine, so it needs provider API access. `VAST_API_KEY` is written on the instance at `/etc/jambu/watchdog.json` (mode 600).
+
+Use a dedicated, restricted Vast.ai key for this, not a personal key. Set `lifecycle.watchdog.enabled: false` to opt out. Shutdown then depends on `gpu guard` or on the CLI staying alive, and `validate` warns you.
+
+## Architecture
+
+```
+jambu.yaml (gpu_runtime:)
+    │
+    ▼
+gpu
+    │
+    ├── Lifecycle    core/policy.py     the only shutdown rule
+    ├── Runtime      runtime/vllm.py    how the model runs
+    └── State        core/state.py      .gpu/state.json
+                │
+        Provider contract   providers/base.py
+                │
+        ┌───────┴────────┐
+        ▼                ▼
+    Vast adapter    future adapter
+```
+
+Four concerns stay independent: model, inference runtime, lifecycle policy, infrastructure provider. The core never branches on a provider name. It branches on `provider.capabilities`.
+
+```
+jambu_gpu/
+├── cli/          setup, run, status, lifecycle, providers, validate
+├── core/         config, models, policy, state, events, engine, execution, health
+├── providers/    base, registry, vast/
+├── runtime/      base, vllm
+└── watchdog/     builder, agent_body
+```
+
+Adding a provider: [ADDING_A_PROVIDER.md](ADDING_A_PROVIDER.md). The guarantees and the tests that hold them: [ACCEPTANCE.md](ACCEPTANCE.md).