gpuhedge 0.1.0__tar.gz

gpuhedge-0.1.0/LICENSE

@@ -0,0 +1,179 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative
+          Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work, excluding
+          those notices that do not pertain to any part of the Derivative
+          Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2026 Miroslav Lzicar
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

gpuhedge-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+include src/gpuhedge/config/*.yaml
+include LICENSE
+include README.md

gpuhedge-0.1.0/PKG-INFO

@@ -0,0 +1,275 @@
+Metadata-Version: 2.4
+Name: gpuhedge
+Version: 0.1.0
+Summary: Speculative execution across serverless GPU clouds: race providers, return the first valid result, cancel the loser.
+Author-email: Miroslav Lzicar <miroslav.lzicar@deepmedchem.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/mireklzicar/gpuhedge
+Project-URL: Documentation, https://github.com/mireklzicar/gpuhedge#readme
+Project-URL: Repository, https://github.com/mireklzicar/gpuhedge
+Project-URL: Issues, https://github.com/mireklzicar/gpuhedge/issues
+Keywords: serverless-gpu,hedged-requests,speculative-execution,modal,runpod,cerebrium,cold-start,tts
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: requests>=2.31
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: soundfile>=0.13
+Requires-Dist: rich>=13.7
+Provides-Extra: modal
+Requires-Dist: modal>=1.5; extra == "modal"
+Provides-Extra: runpod
+Requires-Dist: runpod>=1.10; extra == "runpod"
+Requires-Dist: runpod-flash>=1.18; extra == "runpod"
+Provides-Extra: cerebrium
+Requires-Dist: cerebrium>=2.5; extra == "cerebrium"
+Provides-Extra: providers
+Requires-Dist: modal>=1.5; extra == "providers"
+Requires-Dist: runpod>=1.10; extra == "providers"
+Requires-Dist: runpod-flash>=1.18; extra == "providers"
+Requires-Dist: cerebrium>=2.5; extra == "providers"
+Provides-Extra: analysis
+Requires-Dist: matplotlib>=3.8; extra == "analysis"
+Requires-Dist: numpy>=1.26; extra == "analysis"
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: bump2version>=1.0; extra == "dev"
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# GPUHedge
+
+**State-aware speculative execution for serverless GPUs.**
+
+> First valid result wins. Loser cancellation is audited — and the receipt
+> says when the cloud didn't stop.
+
+Serverless GPU cold starts are wildly bimodal: the same 17 GB TTS model
+returns complete audio in ~6 s on a warm-path hit but 89–122 s on a miss —
+with nothing in between. In 36 evaluation rounds across three real providers,
+a delayed hedge cut p95 cold-start latency from **116.6 s to 29.4 s**, cut
+60-second deadline misses from **11/36 to 0/36**, and launched a second job on
+only **11/36** requests.
+
+The sharper discovery: the primary provider's **queue state at 2.5 seconds**
+predicted every 90–122 s tail with zero overlap in 54 rounds — early enough
+to cancel the queued job *before its worker starts* and reroute. A
+pre-registered live validation found the cutover reduced median latency from
+29.8 s to 22.6 s and cut RunPod-side billed cost from $0.01475 to
+$0.00532/request — but one 104.2 s *hedge-provider* tail made its p95
+advantage estimator-sensitive at n=20 (28.2 s vs 31.0 s under nearest-rank,
+32.0 s vs 31.0 s under linear interpolation), so we don't claim that
+hypothesis. The outlier motivated the **cascaded policy** below. Full
+results are under
+[`benchmarks/2026-07-queue-cutover/`](benchmarks/2026-07-queue-cutover/).
+
+![queue delay predicts the tail](benchmarks/2026-07-moss/figures/fig1_queue_delay_hero.png)
+
+## What it does
+
+GPUHedge races your *own* deployments (containers, weights, custom code)
+across serverless GPU clouds under an SLO policy:
+
+1. submit the request to the cheap/fast **primary**;
+2. watch the primary's **lifecycle state** (queue vs running), not just a timer;
+3. launch a **hedge** only when the primary is entering its tail — and
+   **escalate to a third provider** when the hedge enters its own
+   (`CascadePolicy`, at most two live jobs);
+4. return the **first result that passes validation** (a fast HTTP 200 with
+   malformed output does not win);
+5. **cancel every loser** through its provider-native job API and record a
+   structured `CancellationReceipt` with an explicit **evidence level**
+   (`confirmed_terminal` > `provider_ack` > `request_channel_closed` >
+   `no_evidence`), cancellation scope, ack latency, estimated wasted GPU-$,
+   and a leak flag that defaults to *leaked* until proof arrives.
+
+## Try it in 60 seconds — no accounts, no spend
+
+```bash
+pip install -e .
+gpuhedge demo                              # simulated providers, real policy engines
+gpuhedge replay traces/moss_rounds.jsonl   # reproduce the benchmark tables
+```
+
+The demo races a bimodal primary against a steady hedge through the exact
+code paths used against real clouds — including a malformed-result round the
+validator catches and a queued-cancel cutover.
+
+## The public API
+
+```python
+from gpuhedge import Router
+from gpuhedge.policies import StateAwarePolicy
+
+router = Router(
+    primary="runpod", hedge="cerebrium",
+    policy=StateAwarePolicy(queue_cutover_ms=2_500, safety_hedge_ms=8_500),
+)
+outcome = await router.run()
+outcome.winner          # "runpod" | "cerebrium"
+outcome.total_ms        # end-to-end latency from request start
+outcome.cancellation    # the loser's receipt (or None if no hedge launched)
+```
+
+Policies: `SingleProvider()`, `FixedHedgePolicy(hedge_after_ms)`,
+`StateAwarePolicy(queue_cutover_ms, safety_hedge_ms)`, and
+`CascadePolicy(queue_cutover_ms, safety_hedge_ms, escalate_after_ms)` (pass
+`fallback="modal"` to the Router). Policies are dispatched through one
+protocol method — implement `async def execute(self, ctx)` plus
+`min_providers` and your own policy runs exactly like the built-ins.
+Validators are pluggable (`wav`, `json`, `nonempty`, or
+`register_validator("mine")(...)`).
+
+Providers are YAML: built-in adapters for **RunPod** (Flash queue SDK +
+REST), **Modal** (`FunctionCall.spawn`/`cancel`), **Cerebrium** (sync-first
+REST), a **simulator** (`adapter: sim`), and a **generic HTTP adapter**
+(`adapter: http`) that turns any submit/status/result/cancel service into a
+raceable backend without writing Python — see
+[`docs/adding-a-provider.md`](docs/adding-a-provider.md).
+
+## The benchmark (2026-07, MOSS-TTS-v1.5, three providers)
+
+Same 17 GB model, same commit, same request, pre-seeded weights, identical
+WAV validation on every arm. 54 paired cold-start rounds + warm companions,
+18 live hedged requests, ~$7 of GPU spend, every trace committed under
+[`traces/`](traces/) (sanitized: job ids remapped, account balances withheld,
+deployment identifiers replaced — every latency, receipt, and cost delta is
+untouched, so all tables and figures reproduce identically). Full method and
+results: [`benchmarks/2026-07-moss/`](benchmarks/2026-07-moss/). Regenerate
+the figures with `pip install -e ".[analysis]" && python
+benchmarks/2026-07-moss/analysis.py` (CI does this on every push).
+
+| Policy (36 evaluation rounds) | p50 | p95 | miss>60 s | active $/req | hedge rate |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| single: primary (RTX 4090) | 6.0 s | 116.6 s | 11/36 | $0.0114 | — |
+| fixed hedge @10 s | 6.0 s | 29.4 s | 0/36 | $0.0083 | 11/36 |
+| queue cutover @2.5 s (replay*) | 6.0 s | 21.9 s | 0/36 | $0.0056 | 11/36 |
+| cascade →modal @25 s (replay*) | 6.0 s | 21.9 s | 0/36 | $0.0056 | 11/36 |
+
+\* the cutover and cascade rows are post-hoc replays on data that informed
+them; the cutover's own pre-registered, randomized 60-request live validation
+(with account-level billing deltas and forced loser-cancels on all three
+providers) is in
+[`benchmarks/2026-07-queue-cutover/`](benchmarks/2026-07-queue-cutover/). In
+this replay the hedge never stalled, so the cascade's escalation never fired
+— identical numbers, tail protection for free.
+
+| Pre-registered live validation (20/arm) | p50 | p90 | max | miss>60 s | RunPod billed $/req |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| single RunPod | 6.5 s | 175.6 s | 200.8 s | 7/20 | $0.01475 |
+| fixed hedge @10 s | 29.8 s | 30.8 s | 31.2 s | 0/20 | $0.00649 |
+| **queue cutover @2.5 s** | **22.6 s** | **25.8 s** | 104.2 s | **1/20** | **$0.00532** |
+
+The cutover passed the RunPod-billing and queued-cancel-unbilled hypotheses
+and clearly improved typical latency, but its registered p95 hypothesis is
+**inconclusive** — at n=20 the verdict flips with the quantile estimator
+(28.2 s vs 31.0 s nearest-rank, 32.0 s vs 31.0 s linear) — and it failed the
+zero-miss and active-cost-order hypotheses: one correctly-switched request
+spent 104.2 s on the hedge provider's own tail. See the
+[validation results](benchmarks/2026-07-queue-cutover/results.md).
+
+**That 104.2 s outlier is why the cascade exists**: the primary can have a
+tail, but so can the hedge. `CascadePolicy` escalates to a third provider
+when the sole surviving attempt has produced nothing by `escalate_after_ms`
+(never more than two live jobs). Its live validation is pre-registered — with
+the quantile estimator fixed and request-level interleaving this time — in
+[`benchmarks/2026-08-cascade/`](benchmarks/2026-08-cascade/preregistration.yaml)
+and has not run yet; `gpuhedge demo` shows the mechanism on simulated
+providers today.
+
+**Cost honesty**: "active $" is modeled execution-seconds × rate. Real bills
+also include idle windows (one provider's balance delta ran ~3× its summed
+per-job execution time at a 60 s idle timeout) — both models are reported,
+and the validation measures actual balance deltas per policy arm. See
+[`docs/cost-accounting.md`](docs/cost-accounting.md).
+
+**Cancellation honesty**: every receipt carries an explicit evidence level —
+RunPod cancels are polled to a `confirmed_terminal` state with billing
+reconciled; Modal cancels are `provider_ack` only (input-granular: the
+container idles to its scaledown window and keeps billing); Cerebrium's
+strongest proof is `request_channel_closed`. Anything less is recorded as a
+leak, never assumed stopped. The capability matrix in
+[`docs/provider-capabilities.md`](docs/provider-capabilities.md) states
+exactly which cancel paths have been exercised live on which provider —
+including the forced audit that caught our own Modal adapter silently
+failing before the fix.
+
+## Run it against real clouds
+
+```bash
+pip install -e ".[providers]"     # or a subset: .[runpod], .[modal], .[cerebrium]
+mkdir -p config
+cp src/gpuhedge/config/benchmark.example.yaml config/benchmark.yaml
+# fill in your endpoint ids / app names, set deployed: true
+gpuhedge login-check              # verifies auth, spends nothing
+gpuhedge plan                     # what the config encodes, incl. budget gates
+gpuhedge bench --dry-run          # inspect before spending
+gpuhedge bench --go               # Stage 2 paired cold-start rounds
+gpuhedge cutover --go             # one live state-aware request
+```
+
+Nothing submits a GPU job without `--go`. A projected-cost ledger enforces
+hard budget gates (`BudgetExceeded` halts submission), and `gpuhedge costs`
+reconciles projections against provider-reported billing.
+
+Deployment recipes for the benchmark model on all three providers are under
+[`deploy/`](deploy/) — self-contained, with the provider gotchas we hit
+documented in [`docs/provider-capabilities.md`](docs/provider-capabilities.md).
+
+## Repository map
+
+```
+src/gpuhedge/
+  router.py          Router — the public API
+  policies/          SingleProvider / FixedHedgePolicy / StateAwarePolicy /
+                     CascadePolicy — plus the execute() protocol for custom ones
+  backends/          runpod / modal / cerebrium / sim / generic http adapters
+  validators/        wav / json / custom validator registry
+  benchmark/         paired rounds, live hedging, state-aware engine,
+                     replay (offline policy evaluation), reports, demo
+  telemetry/         JSONL traces, projected-cost ledger with budget gates,
+                     provider billing monitors
+  config/            benchmark.yaml (plan-as-data), demo.yaml
+benchmarks/          2026-07-moss (method, results, figures, analysis.py),
+                     2026-07-queue-cutover (preregistration + validation),
+                     2026-08-cascade (preregistration; not yet run)
+deploy/              MOSS-TTS deployment recipes per provider
+traces/              committed benchmark traces (JSONL, sanitized)
+docs/                architecture, policies, capabilities, cancellation,
+                     cost accounting, adding a provider, security
+```
+
+## Status & caveats
+
+- Benchmark data is one workload (TTS, ~5–7 s of audio), one region per
+  provider, collected across two days in 2026-07. Provider rankings move;
+  that is the argument *for* state-aware routing, not against it — treat any
+  single number as a snapshot.
+- The Cerebrium adapter discovers run ids from the runs list (its async API
+  does not return usable handles): **benchmark-safe, not concurrency-safe**
+  — see [`docs/provider-capabilities.md`](docs/provider-capabilities.md).
+- No p99s are quoted anywhere: n is tens, not thousands. Miss rates carry
+  Wilson 95% intervals.
+
+## Contributing
+
+`make dev && make lint && make test`. CI runs Python 3.10–3.12 and a wheel
+smoke test. Adapter contributions welcome — the contract is one class with
+`submit / result / status / cancel` returning honest lifecycle events, plus
+mock tests; see [`docs/adding-a-provider.md`](docs/adding-a-provider.md) and
+[`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## License
+
+Apache-2.0.