gipper 0.3.0__tar.gz

gipper-0.3.0/.env.example

@@ -0,0 +1,5 @@
+# Optional: copy to ~/.config/gipper/.env and chmod 600 that file.
+# Set only the providers you intend to use. Do not commit real credentials.
+RUNPOD_API_KEY=
+VAST_API_KEY=
+SHADEFORM_API_KEY=

gipper-0.3.0/AGENTS.md

@@ -0,0 +1,89 @@
+# gipper: notes for agents
+
+gipper rents GPUs (Runpod, Vast.ai; Shadeform wired in but unused), runs a job,
+brings results home, gives the machine back, and records everything in an
+append-only ledger. Read [README.md](README.md) for what it does and
+[docs/placement.md](docs/placement.md) for how it chooses. This file covers
+working *on* gipper.
+
+## Before you change anything
+
+- **Protect running clients.** Editable installs import directly from their
+  checkout. Check for active runs before editing a shared installation:
+  `pgrep -af "gipper pod run|cloud.py|runner.run"`. Use a git worktree on a
+  branch for non-trivial changes, and integrate when no clients are running.
+- **Keep the client surface stable** ([docs/clients.md](docs/clients.md)):
+  `run.run`, `run.plan`, `run.estimate`, `run.preflight`, `run.settings`,
+  `run.BadHost`, `job.load`, `providers.load(...)` with its methods, `ledger`,
+  `ssh.Remote`, and the `[placement]` fields client jobs use. Clients can wrap
+  `run.preflight`, `run.plan`, and `run.settings`; `run.py` must keep looking
+  these up by module-global name at call time.
+- **Run the tests:** `uv run --extra test pytest -q` (no network or rentals).
+  Use `uv run` to select the project's Python environment.
+
+## Invariants
+
+1. **A rented machine is always given back, and only counted given back when
+   the provider stops listing it** (`lifecycle.terminate`). Teardown is in
+   `finally`. Never rent a replacement while a previous machine's closure is
+   unconfirmed.
+2. **Keys never leave a request header.** Not in argv, logs, receipts or errors;
+   `config.redact()` scrubs every provider key, and `http.request` applies it to
+   error bodies. Jobs' `[env]` refuses names ending in KEY, TOKEN or SECRET.
+3. **The ledger is append-only and is the memory.** Never rewrite receipts. New
+   facts are new receipt kinds or new payload fields, and readers must tolerate
+   old receipts without them: receipts before 2026-10-26 have no `provider` and
+   are Runpod's; before 0.2 there's no `machine` on `pod.create`, no `compute_s`
+   on `run.end` and no `kind` on `host.reject`.
+4. **Memory must not blame a host for what isn't its fault.** Job-specific
+   rejects raise `run.Unsuitable`; only broken hosts and lost connections count
+   as bad; compute time comes only from successful runs; link speed comes only
+   from real transfers of 20 MB or more; the billing factor only from rentals of
+   10 minutes or more. A wrong memory steers every later run.
+5. **Never touch a machine not named `gipper-*`**, and no provider resource
+   gipper didn't create.
+6. **One provider failing never stops the others** (shopping, reaping, reconciling).
+7. **Stdlib only.** No runtime dependencies.
+
+## Map
+
+| File | Job |
+|---|---|
+| `job.py`, `placement.py` | the job file and its `[placement]` hints |
+| `market.py` | `Offer`, GPU-name matching, filters (`admit`), order (`rank`), `shop` |
+| `costs.py` | the whole-run estimate and score for one offer |
+| `history.py` | memory, read from the ledger: failure rates, ready times, link speeds, compute times, billing factors |
+| `run.py` | the run: plan, rent, preflight, upload, detached job, log, checkpoints, collect, teardown |
+| `lifecycle.py` | verified teardown |
+| `reap.py` | the 15-minute sweep for forgotten machines |
+| `billing.py` | providers' bills → `spend.billed`; spend for the daily cap |
+| `cuda_probe.py`, `network_probe.py` | host checks before upload |
+| `providers/{runpod,vast,shadeform}.py` | one module per market behind the interface in `providers/__init__.py` |
+| `runpod/` | import paths kept for pre-0.2 scripts only |
+| `http.py`, `ssh.py`, `ledger.py`, `config.py` | plumbing |
+
+## Adding a provider
+
+Implement the interface documented in `providers/__init__.py`. Map locations to
+ISO country codes, memory to decimal GB, and advertised link speeds (Mb/s) and
+fees ($/GB) into `Offer.metrics` under the names `costs.py` reads. Set `udp`
+honestly. Implement `billed()` if the provider has a billing API. Add it to
+`providers.NAMES` and `config.DEFAULTS["providers"]`. Test the parsing with
+response shapes captured from the real API. Then do a real smoke run
+(`examples/smoke.toml --provider <name>`) and, if it claims UDP,
+`examples/udp-echo.toml` plus a packet sent from here, and write what you learn
+in `docs/lessons.md`.
+
+## Tests
+
+Test decisions and parsing, not plumbing: what gets chosen, what counts as a bad
+host, what gets billed, what a job file accepts, what the reaper ends. Build
+histories from receipts (`tests/conftest.py: receipt`), and providers' answers
+from real response shapes. Don't build tests that mock `run.run`'s internals
+end to end: an explicitly authorized live smoke run qualifies that path separately.
+
+## Spending money
+
+A real rental needs the user's go-ahead unless the task already asked for one.
+Use `--dry-run` and `gipper pod estimate` first, keep `[budget]` tight, and check
+`gipper pod ls` is empty afterwards.

gipper-0.3.0/CHANGELOG.md

@@ -0,0 +1,107 @@
+# Changelog
+
+## 0.3.0 (2026-11-04)
+
+### Added
+
+- Public documentation site, GitHub Pages deployment through Artoo, correctness
+  lint checks, and a trusted-publishing workflow for PyPI.
+
+- MIT license, public installation/contribution/security guidance, release checklist,
+  package metadata, and Linux CI for Python 3.12–3.14 with distribution checks.
+- User-config dotenv support beside `config.toml` for non-editable installs and
+  cleanup services; environment and explicit `GIPPER_ENV` keep precedence, and
+  the checkout `.env` remains a fallback.
+
+### Changed
+
+- Present gipper as a standalone GPU job runner, with workload-neutral examples
+  and operational guidance based on each installation's own history.
+- Replace internal planning with a public roadmap and full third-party research
+  captures with source references. Document billing and host-trust limitations.
+
+### Fixed
+
+- Cost reports include spend receipts without a job ID instead of crashing and
+  hiding later rows.
+
+## 0.2.2 (2026-10-28)
+
+Round trips are measured and remembered; geography only steers work that needs it.
+
+- **Interactive jobs ping candidate hosts before renting** (`probe_rtt`, on by
+  default for `profile = "interactive"`): the 48 cheapest offers' public IPs,
+  from this machine, in about 10 seconds. About 60% of Vast hosts answer.
+- **Round-trip order of trust:** the job's own measurement from `latency_target`,
+  then a ping from here, then memory of that machine, then the median for its
+  country, then `preferred_countries`. Pings from here pass `max_rtt_ms` only
+  with `latency_target = "here"`; otherwise they rank but don't qualify.
+- **Every rental records `host.rtt`** (fastest of 5 TCP connects to its ssh
+  port), and `pod.create` records the offer's `country`. Shops for interactive
+  jobs record `market.rtt`.
+- **Link-speed memory is per country** before per provider, so far hosts' slower
+  transfers are priced from experience.
+- **Batch jobs ignore geography:** a cheap host on the far side of the world wins
+  whenever the whole run costs less.
+
+## 0.2.1 (2026-10-28)
+
+Everything from the 0.1 placement work (the `codex/workload-placement-20261027`
+branch) is available again, on top of 0.2's model:
+
+- **`profile = "interactive"` with `objective = "latency"`:** offers rank by
+  measured round trip (`measured_rtt_ms_by_machine`, `latency_target`), then
+  `preferred_countries`, then distance (`near_latitude`/`near_longitude`), then
+  the usual score. `max_rtt_ms` and `max_distance_km` fail closed when nothing
+  was measured.
+- **Restored fields:** `min_dlperf` (Vast's DLPerf as a filter),
+  `client_upload_mbps` and `client_download_mbps` (caps on this end's link),
+  `fallback_mbps`.
+- **0.1 names in every estimate:** `estimated_total_usd`,
+  `estimated_bandwidth_usd`, `known_bandwidth_usd`, `bandwidth_fees_known`,
+  `cost_is_lower_bound`, `setup_download_s`, `upload_s`, `download_s`,
+  `compute_s`, `dlperf`, `assumptions` and the rest, so reports that read
+  estimate receipts keep their fields. `offer.placement_estimate`,
+  `placement.estimate_offer` and `placement.rank_for_job` work again.
+- **Vast source references** behind the placement docs are in `docs/sources/vast/`.
+
+## 0.2.0 (2026-10-28)
+
+Offers are judged on the whole run, and gipper learns from its own rentals.
+
+- **Whole-run estimate and score for every job** (`costs.py`): ready, setup,
+  upload, the job's own downloads, compute and output download; rental plus
+  Vast's bandwidth fees, times the provider's billed-to-computed factor; plus the
+  chance of a bad host and the value of waiting. Short jobs favour reliable,
+  fast hosts and long ones favour price. Replaces the opt-in `[placement]`
+  profiles from 0.1; the fields client jobs used still load.
+- **Memory from the ledger** (`history.py`, `gipper pod hosts`): host failure
+  rates per provider and per machine, ready times, measured link speeds, compute
+  time per workload and GPU, billing factors.
+- **Billed spend** (`billing.py`, `gipper pod reconcile`, run by the reap timer):
+  Vast's itemised charges and Runpod's per-pod billing become `spend.billed`;
+  the daily cap and `gipper pod cost` use them, attributed to the machine's own day.
+- **New receipts:** `transfer` (timed real uploads and downloads),
+  `host.requirements`; `pod.create` gains `machine`, `run.end` gains
+  `compute_s`, and `host.reject` gains `kind`.
+- **`run.Unsuitable`** (a `BadHost`): rejects a host that works but doesn't suit
+  the job, without counting it against the host in memory.
+- **Runpod:** GPU types its REST API won't create are skipped instead of ending
+  the run; `[placement] min_cuda` becomes `allowedCudaVersions`.
+- **Vast:** advertised speeds, fees, CUDA and compute capability on every offer;
+  memory in decimal GB, so nominal 48 GB cards qualify; more machines per GPU
+  model are considered.
+- **Kept from client contributions in 0.1:** verified teardown, the `cuInit` check
+  with its 804 retry, and the optional transfer probe.
+- **Removed:** the default 2× hourly ceiling (the score prices the run; an
+  explicit `[gpu] max_usd_per_h` still caps it), and `docs/workload-placement.md`
+  (now `docs/placement.md`). Placement's latency and geography fields were
+  removed here and restored in 0.2.1.
+- **Docs:** README, `docs/placement.md`, `docs/clients.md`, `docs/lessons.md`,
+  `AGENTS.md`.
+
+## 0.1.0 (2026-10-20 to 2026-10-27)
+
+Rented GPUs from the CLI: Runpod, then Vast.ai and Shadeform behind one
+interface; regions, UDP ports, preflight and host replacement, caps, reaper,
+receipts.

gipper-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing
+
+Bug reports and focused pull requests are welcome. Include the Python version,
+provider, command, expected behavior, and a redacted error. Never attach API
+keys, SSH keys, raw ledgers, or private job data. See [SECURITY.md](SECURITY.md)
+for vulnerabilities.
+
+## Development
+
+On Linux with Python 3.12+ and uv:
+
+```sh
+git clone https://github.com/lavallee/gipper.git
+cd gipper
+uv sync --locked --extra test
+uv run --extra test pytest -q
+```
+
+The tests use fake providers and a temporary ledger: no network or rental is
+needed. Keep the runtime stdlib-only. Preserve the public interfaces in
+[docs/clients.md](docs/clients.md), including module-global hooks used by clients.
+Test provider parsing and placement/cleanup decisions with realistic fixtures.
+
+If clients use an editable install of your checkout, check for active runs
+before editing it and work on a branch in a separate worktree. See [AGENTS.md](AGENTS.md).
+
+A real smoke run spends money and needs explicit operator authorization. Review
+an estimate and dry run first, use tight budgets, and verify `gipper pod ls`
+and the provider console after cleanup. Never touch resources outside the
+`gipper-*` namespace. Qualify provider changes with live runs separately from
+unit tests; record the result and any untested behavior in `docs/lessons.md`.
+
+## Before a pull request
+
+- Run `uv run ruff check gipper tests scripts` and `uv run --extra test pytest -q`.
+- Run `uv build` and `uv run --no-project scripts/check_dist.py`.
+- Update documentation and add an Unreleased changelog entry for user changes.
+- Run a redacted secret scan; never paste credential values into an issue.
+
+CI checks Python 3.12, 3.13, and 3.14 on Linux, builds both distributions, and
+smoke-tests the installed wheel. Ruff checks correctness and import errors; avoid unrelated formatting churn.
+
+Contributions are made under the repository's MIT license. Only submit material
+you have the right to license; link third-party research sources instead of
+copying complete pages or repositories.
+
+## Documentation site
+
+Edit `site/index.html`, `site/style.css`, or the Markdown guides in `docs/`.
+`uv run --group site python scripts/build_site.py` deterministically renders
+`docs/index.html` and `docs/guides/`. The root Artoo artifact owns publishing;
+run `artoo build .`, `artoo verify .`, and `artoo deploy . --dry-run` before
+`artoo deploy .`. GitHub Pages serves `main` / `docs`.

gipper-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Marc Lavallee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gipper-0.3.0/MANIFEST.in

@@ -0,0 +1,10 @@
+include .env.example uv.lock AGENTS.md LICENSE README.md CHANGELOG.md CONTRIBUTING.md SECURITY.md RELEASING.md ROADMAP.md
+recursive-include docs *.md
+recursive-include examples *.toml *.txt
+recursive-include deploy *.service *.timer
+recursive-include tests *.py
+recursive-include scripts *.py
+prune plans
+exclude CLAUDE.md
+include artifact.toml
+recursive-include site *.html *.css *.svg

gipper-0.3.0/PKG-INFO

@@ -0,0 +1,275 @@
+Metadata-Version: 2.4
+Name: gipper
+Version: 0.3.0
+Summary: Run GPU jobs across cloud markets with cost estimates, verified teardown, and an append-only ledger
+Author: Marc Lavallee
+License-Expression: MIT
+Project-URL: Homepage, https://lavallee.github.io/gipper/
+Project-URL: Documentation, https://lavallee.github.io/gipper/guides/quickstart.html
+Project-URL: Repository, https://github.com/lavallee/gipper
+Project-URL: Issues, https://github.com/lavallee/gipper/issues
+Project-URL: Changelog, https://github.com/lavallee/gipper/blob/main/CHANGELOG.md
+Keywords: gpu,runpod,vast,cloud,batch
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# gipper
+
+[Website](https://lavallee.github.io/gipper/) · [Quickstart](https://lavallee.github.io/gipper/guides/quickstart.html) · [Python API](https://lavallee.github.io/gipper/guides/clients.html)
+
+gipper runs jobs on rented NVIDIA GPUs. Describe the hardware, command, input
+files, outputs, and budget in a TOML file; gipper compares cloud offers, rents a
+machine, checks it, runs the job, collects results, and verifies teardown.
+
+Use it for model training and fine-tuning, batch inference, rendering, or other
+GPU workloads that run from a Linux shell. Your job supplies its own software
+and dependencies. The same CLI and Python API work across Runpod and Vast.ai.
+
+Placement considers the whole job: startup, setup, transfers, compute, bandwidth
+fees, and host failures. An append-only local ledger records runs and costs;
+future estimates learn from **your own** workloads and provider history. A fresh
+installation starts with stated defaults, not someone else's rental history.
+
+**Status: 0.3.0, alpha.** Linux client, CLI and Python API, no daemon. Runpod and
+Vast have live-run validation; Shadeform is experimental and has not had a live
+smoke test. See the [roadmap](ROADMAP.md) for planned features and
+[SECURITY.md](SECURITY.md) for trust and billing limits. Licensed under [MIT](LICENSE).
+
+## Install
+
+Requires Linux, Python 3.12+, OpenSSH (`ssh`, `ssh-keygen`), and `tar` with
+gzip support. `ping` is optional for interactive placement. The Python runtime
+has no third-party dependencies. Remote machines need a Linux shell, `tar`,
+`timeout`, `curl`, and NVIDIA tools; the bootstrap installs `uv` if needed.
+
+Install from source (no PyPI release is assumed):
+
+```sh
+git clone https://github.com/lavallee/gipper.git
+cd gipper
+uv tool install .
+gipper --help
+```
+
+Use `pip install .` in a virtual environment if you do not use `uv`.
+For development, use `uv sync --extra test` (see [CONTRIBUTING.md](CONTRIBUTING.md)).
+
+Configure only the providers you intend to use:
+
+```sh
+mkdir -p ~/.config/gipper
+(umask 077; cp -n .env.example ~/.config/gipper/.env)
+chmod 600 ~/.config/gipper/.env
+# Edit ~/.config/gipper/.env to add your provider keys.
+```
+
+Keys can also come from environment variables: `RUNPOD_API_KEY`, `VAST_API_KEY`,
+`SHADEFORM_API_KEY`. Environment variables take precedence. `GIPPER_ENV` selects
+an exclusive dotenv file; otherwise the user-config dotenv overrides the legacy
+checkout `.env`. A provider without a key is skipped when renting. Keys travel
+in API request headers and are scrubbed from receipts and API error bodies.
+Machines get gipper's own SSH key (`~/.config/gipper/id_ed25519`).
+
+Check estimates before paying for a smoke run:
+
+```sh
+gipper pod estimate examples/smoke.toml --provider vast
+gipper pod run examples/smoke.toml --provider vast --dry-run
+# The following command spends money:
+gipper pod run examples/smoke.toml --provider vast
+gipper pod ls
+```
+
+A dry run contacts provider APIs but does not rent. Set provider-side spending
+alerts too: estimates and client-side caps cannot guarantee a final bill.
+
+Install the cleanup timer on the machine that runs jobs (recommended):
+
+```sh
+mkdir -p ~/.config/systemd/user
+cp deploy/gipper-reap.* ~/.config/systemd/user/
+systemctl --user daemon-reload
+systemctl --user enable --now gipper-reap.timer
+systemctl --user list-timers gipper-reap.timer
+```
+
+The service expects `~/.local/bin/gipper`; adjust `ExecStart` if installed
+elsewhere. A service cannot see keys exported only in an interactive shell; use
+the user-config dotenv above. User timers require a running user manager. If
+jobs must outlive logout, configure user lingering or an equivalent scheduler.
+See [SECURITY.md](SECURITY.md) for host trust and cleanup limitations.
+
+### Local configuration and data
+
+| Path / override | Purpose |
+|---|---|
+| `~/.config/gipper/config.toml` / `GIPPER_CONFIG` | Settings such as `daily_cap_usd = 5.0`; see [`config.DEFAULTS`](gipper/config.py) |
+| `.env` beside the config file / `GIPPER_ENV` | Provider keys (simple `NAME=value`, no shell expansion) |
+| `~/.config/gipper/` / `GIPPER_KEYS` | SSH key directory |
+| `~/.local/share/gipper/` / `GIPPER_DATA` | Append-only receipts used for history and spend |
+
+Keep the ledger on durable storage and restrict access to it. Separate installs
+sharing a provider account should coordinate: the reaper owns the `gipper-*`
+namespace, not just the machines in one local ledger.
+
+## Use
+
+```
+gipper pod gpus --min-mem 24 --region US,CA [--udp]   # what's on offer, scored for a 30-minute job
+gipper pod estimate job.toml                          # how long and how much, offer by offer
+gipper pod run job.toml [--dry-run] [--provider vast] [--gpu "RTX 4090"]
+gipper pod hosts                                      # what gipper has learned about providers and machines
+gipper pod cost [--since 2026-10-27]                  # per job: computed and billed
+gipper pod ls | reap [--dry-run] | reconcile | account
+gipper ledger [--since ...] [--job ID]                # raw receipts
+```
+
+A job ([`examples/smoke.toml`](examples/smoke.toml); every field in
+[`gipper/job.py`](gipper/job.py) and [`gipper/placement.py`](gipper/placement.py)):
+
+```toml
+name = "my-gpu-job"
+workload = "my-workload"          # jobs with the same workload share a memory of compute times
+
+[gpu]
+min_mem_gb = 24
+prefer = ["RTX 4090", "RTX A6000"]    # in order; names match across providers
+regions = ["US", "CA"]                # ISO country codes; default anywhere
+providers = ["vast", "runpod"]        # default: every provider with a key
+
+[pod]
+ports = ["5004/udp"]                  # published; UDP rules out Runpod
+
+[[inputs]]
+src = "~/gpu-project"
+dst = "project"
+exclude = [".git", "runs"]
+
+[[outputs]]
+src = "out"
+dst = "~/gpu-project/runs/x/out"
+
+[run]
+command = "python project/main.py"
+checkpoint = { src = "out", every_s = 300 }
+
+[budget]
+max_usd = 1.0
+max_hours = 2.0
+
+[placement]                           # optional, and better than memory's guess
+compute_s = 900
+setup_download_gb = 5                 # weights and wheels the job fetches itself
+```
+
+## How gipper chooses
+
+Every offer is judged on the whole run, not its hourly price
+([docs/placement.md](docs/placement.md) has the details):
+
+- **Time:** machine ready + setup + upload + the job's own downloads + compute +
+  output download. Each figure comes from the job's `[placement]` hints, from
+  memory, from what the provider advertises (Vast publishes host link speeds),
+  or from a stated default, and `estimate` says which.
+- **Dollars:** rental for that time, plus Vast's bandwidth fees, times what this
+  provider has billed compared with rate × lifetime in your ledger. Without
+  qualifying history, the billing factor is 1.0; storage and rounding can make
+  the final bill differ.
+- **Risk:** the chance the host turns out bad, from memory: the machine's own
+  record, pulled towards its provider's, pulled towards 10% for a stranger.
+  A failure costs `failure_cost_usd` ($1) plus the time and rent it wastes.
+- **Score** = dollars + risk + waiting, with an hour of waiting worth
+  `hour_value_usd` ($3).
+
+For short jobs, startup time and the cost of a failed host can outweigh a lower
+hourly price. For long jobs, the hourly rate has more influence. Provider names
+do not determine the winner: the ranking follows the job and available history.
+`prefer` comes first, and an offer whose expected cost exceeds `max_usd` is
+left out. `[placement] objective = "cost"` removes the value of waiting while
+still pricing failures; `"time"` values waiting at $50/h. For latency-sensitive
+services, `profile = "interactive"` probes candidate hosts and ranks by round
+trip first. Batch jobs account for transfer time without preferring nearby hosts.
+
+Memory is the ledger itself. A rental is *good* once its job started, and *bad*
+if the host was broken (busy GPU, CUDA that won't initialise, ssh that never came
+up, a failed probe) or the connection was lost. A host that works but doesn't
+suit the job (`run.Unsuitable`), a job that fails on a working host, and an
+error on this side count against nobody.
+
+## Run lifecycle and safeguards
+
+- **The machine is given back, and gipper checks.** Teardown runs in `finally`
+  after success, failure, Ctrl-C or a cap, and counts as done only when the
+  provider stops listing the machine. Until then it stays
+  `pod.termination_pending`, gipper rents no replacement, and the reaper (every
+  15 minutes) retries it first. The reaper also ends any `gipper-*` machine the
+  ledger forgot or that passed its deadline, on every provider. The job runs
+  detached under `timeout max_hours`. This limits the command, not the rental;
+  the reaper must still terminate a machine if the client dies.
+- **Hosts are checked before anything is uploaded:** the GPU is idle,
+  `cuInit` succeeds (with one retry without CUDA forward-compat libraries on
+  error 804), the compute capability meets the job's floor, and optionally a
+  measured transfer probe passes. A bad host is replaced, up to 3 times.
+- **Caps:** per job (`max_usd`, `max_hours`, `[gpu] max_usd_per_h`) and per day
+  (`daily_cap_usd` in `~/.config/gipper/config.toml`, default $5). Daily spend
+  uses providers' bills where reconciled.
+- **Outputs** sync down every `checkpoint.every_s` and are salvaged on failure.
+- **Environment:** each provider's default image unless `[pod] image` says
+  otherwise. Each job sources a bootstrap providing `uv` and
+  `GIPPER_TORCH_INDEX`, the PyTorch wheel index for the host's driver. Published
+  ports arrive as `GIPPER_PUBLIC_IP` and `GIPPER_PORT_<n>_<PROTO>` (the outside
+  port, which on Vast differs from the inside one).
+
+## Markets
+
+| | Runpod | Vast.ai | Shadeform (unused) |
+|---|---|---|---|
+| You rent | container, root | container on someone's machine, root | VM on one of 30+ clouds, user `shadeform` |
+| Good for | secure cloud, network volumes | consumer GPUs everywhere, cheap, UDP | datacenter cards with one bill |
+| Ports | TCP/HTTP, **no UDP** | TCP and UDP, random outside port | TCP and UDP (UFW; the cloud may filter) |
+| Bills | per pod per day | itemised: GPU, disk, bandwidth | no billing API |
+
+An RTC job that must stay on Runpod can relay media through Cloudflare's TURN
+over TCP/TLS on 443, at some latency cost.
+
+## Python API
+
+The CLI is the simplest surface. From Python (gipper is stdlib-only):
+
+```python
+from gipper import run, ledger
+from gipper.job import load
+from gipper.providers import load as provider
+
+job = load("job.toml")
+offers, ceiling, notes = run.plan(job, provider="vast")   # ranked; each has .estimate
+run_id, outcome = run.run(job, provider="vast", log_dir="runs/x")
+provider("vast").list_ours()                              # for your own guards
+```
+
+`run.preflight(remote)` can be wrapped to add host checks of your own: raise
+`run.Unsuitable` when the host works but doesn't suit the job, and `run.BadHost`
+when it's broken. Both replace the host, but only `BadHost` counts against it in
+memory. To move a job between providers, check [docs/clients.md](docs/clients.md).
+
+## More
+
+- [docs/placement.md](docs/placement.md): the cost model, memory and every `[placement]` field
+- [docs/clients.md](docs/clients.md): writing jobs that run on any provider; the Python surface
+- [docs/lessons.md](docs/lessons.md): operational behavior and troubleshooting
+- [CONTRIBUTING.md](CONTRIBUTING.md): development and tests
+- [SECURITY.md](SECURITY.md): trust model and vulnerability reports
+- [RELEASING.md](RELEASING.md): release checklist
+- [CHANGELOG.md](CHANGELOG.md)