vram-lend 0.1.0__tar.gz

vram_lend-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+docs/drafts/
+__pycache__/
+.venv/
+dist/
+*.egg-info/
+.pytest_cache/

vram_lend-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 TacoTakumi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vram_lend-0.1.0/PKG-INFO

@@ -0,0 +1,187 @@
+Metadata-Version: 2.5
+Name: vram-lend
+Version: 0.1.0
+Summary: Lend yourself GPU memory from a serving inference engine, run a job, give it back
+Project-URL: Homepage, https://github.com/TacoTakumi/vram-lend
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,gpu,llama-swap,llama.cpp,skill,vram
+Classifier: Environment :: Console
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Requires-Dist: agentsquire>=0.5.0
+Description-Content-Type: text/markdown
+
+# VRAM Lend
+
+VRAM Lend lets an inference engine lend part of its GPU memory to another
+program on the same machine while it keeps serving requests, and take the
+memory back after.
+
+A large MoE model on consumer GPUs fills every GPU, mostly with its expert
+cache: copies of experts that also live in RAM or on disk. An image
+generator, a speech service or a small helper model needs a few GiB on one
+GPU for a short time. With VRAM Lend the engine empties that many MiB of its
+expert cache on the GPU asked for and releases them to the driver at once.
+The borrower allocates the memory, uses it and frees it, and a return gives
+it back to the engine. The engine never stops serving.
+
+```
+POST /vram/lend   {"gpu": "0000:09:00.0", "mib": 8192}  ->  {"id": "7", "gpu": "0000:09:00.0", "mib": 8208}
+POST /vram/return {"id": "7"}                           ->  {"id": "7", "gpu": "0000:09:00.0", "mib": 8208}
+GET  /vram/state                                        ->  {"gpus": [...], "leases": [...]}
+```
+
+## Status
+
+| Component | Status | Code |
+|---|---|---|
+| Project Maya (GLM-5.3-Flash and DeepSeek-V4.1 engines) | Done: lend and return on any GPU while requests run | Fork branch: <https://github.com/TacoTakumi/project-maya/tree/vram-lend>. Upstream PR: not opened yet |
+| Strata | Lend and return on both GPUs of a 2x RTX 3090 layer split while requests run | Fork branch: <https://github.com/TacoTakumi/Strata/tree/vram-lend>, on [Niko1221/Strata](https://github.com/Niko1221/Strata) v0.2.7.1 with upstream PR [#2522](https://github.com/Niko1221/Strata/pull/2522), which now carries the four fixes from this work as its own commits. Upstream issue: [#4168](https://github.com/Niko1221/Strata/issues/4168) |
+| llama.cpp | Done: lend and return of the MoE expert cache on any GPU that has one, while requests run. Served on this host through llama-swap as its own entry | Fork branch: <https://github.com/TacoTakumi/llama.cpp/tree/vram-lend>, on master 961c376a2 with the per-device `--moe-cache-mib` of [#30205](https://github.com/ggml-org/llama.cpp/pull/30205). Upstream PR: not opened yet |
+| llama-swap (one address for every engine) | Done as a general change: a request matching `upstream.ignorePaths` on a loaded model is served outside the scheduler (no start, no concurrency slot, no in-flight count, no ttl reset), so `/upstream/<model>/vram/lend`, `/vram/return` and `/vram/state` with `^/vram/` in `ignorePaths` reach a serving engine and never load one. Ten lines of logic, tests, docs. See [docs/llama-swap.md](docs/llama-swap.md) | Fork branch: <https://github.com/TacoTakumi/llama-swap/tree/untracked-paths>, on [mostlygeek/llama-swap](https://github.com/mostlygeek/llama-swap) 6c855da (v267). Upstream issue: [#1135](https://github.com/mostlygeek/llama-swap/issues/1135), existing. Upstream PR: not opened yet. Replaces the earlier `vram-route` branch (a `/vram` route on llama-swap's own port, routed by card) |
+| `vram-lend` command and agent skill (borrower side) | Done: one command that finds the engine, directly or behind llama-swap, lends on the card named, runs the job and returns the lease when the job exits; a skill that teaches an agent when and how to use it. See [Agent instructions](#agent-instructions) | This repository: `src/vram_lend/`, the skill in `src/vram_lend/skills/vram-lend/`. Package `vram-lend` on PyPI |
+| Berth (broker side) | Done on one GPU, as an experimental feature of the v0.4.0 pre-releases (rc.3 tested on this host: a 6.5 GiB voice borrower on an RTX 4060 Ti held by Maya, lend, grant and take back while Maya streams). Several GPUs under one lease are planned | [v0.4.0-rc.3](https://github.com/T-Crypt/berth/releases/tag/v0.4.0-rc.3), [contrib/LEND-PROTOCOL.md](https://github.com/T-Crypt/berth/blob/next/contrib/LEND-PROTOCOL.md). Design and tests: <https://github.com/T-Crypt/berth/issues/189> |
+
+## Results
+
+Maya fork, final builds before and after the rebase onto upstream v1.1.27
+(the results pages give the build of each row), on 2x RTX 3090, RTX 5060 Ti
+and 2x RTX 4060 Ti. An 8 GiB lend on an RTX 4060 Ti while a request streams:
+
+| Check | GLM-5.3-Flash | DeepSeek-V4.1 |
+|---|---|---|
+| Lend answer / driver shows the memory free | 0.36-0.41 s / 0.01-0.04 s after | 0.16-0.22 s / 0.01-0.05 s after |
+| Largest token gap, lend / return | 0.40-0.45 s / 0.16-0.23 s | 0.23-0.28 s / 0.12-0.21 s |
+| Return answer | 0.05-0.07 s | 0.08-0.13 s |
+| 2 GiB on each GPU, other GPUs | all pass, 0 MiB change | all pass, 0 MiB change |
+| Prompt rate at full lendable | 91% of the rate with nothing lent | 85-98% |
+| Lendable before the lend = after the return | 10320 = 10320 MiB | 10478 = 10478 MiB |
+| Decode with 8 GiB lent | within noise (52.5 / 50.8 / 52.2 and 47.7 / 48.8 / 49.3 t/s before / lent / after) | no penalty visible |
+
+Over lendable (409), two leases on one GPU, a return while the memory is in
+use (409, retry), a lend during a long prompt read, a late lease given back by
+the server, and an engine restart all pass.
+
+Strata fork, branch `vram-lend`, Swift 1.5 IQ4_XS on a layer split over the
+2x RTX 3090 (0000:01:00.0 on a 24.4 GB/s PCIe link, 0000:02:00.0 on a
+6.1 GB/s link). An 8 GiB lend on each RTX 3090 while a request streams:
+
+| Check | 0000:01:00.0 | 0000:02:00.0 |
+|---|---|---|
+| Lend answer / driver shows the memory free | 0.04-0.05 s / 0.01-0.04 s after | 0.04-0.05 s / 0.01-0.02 s after |
+| Largest token gap, lend / return | 0.046-0.056 s / 0.029-0.039 s | 0.053-0.055 s / 0.067-0.074 s |
+| Return answer | 0.01-0.02 s | 0.02-0.03 s |
+| 2 GiB on each GPU, other GPUs | all pass, 0 MiB change | all pass, 0 MiB change |
+| Prompt rate at full lendable | 98.5-99.7% of the rate with nothing lent | 69-73%: Failed against the 90% target, accepted as the slow link's cost |
+| Lendable before the lend = after the return | 12800 = 12800 MiB | 12288 = 12288 MiB |
+| Decode with 8 GiB lent | -5.6 / -6.5% (hit rate 99.6 to 98.3%) | -11.5 / -9.5% (hit rate 99.6 to 97.2%) |
+
+A lend during a long prompt read is answered at the next prompt chunk. Over
+lendable, two leases on one GPU, a return while the memory is in use, an
+engine restart, 50 lend/return cycles during streaming requests and the
+quality suite with 8 GiB lent on each GPU all pass. A late lend (503) was
+not reachable, because the engine answers inside the server's wait; the
+server's unit tests cover that path. HIP and SYCL builds were not run.
+
+llama.cpp fork, branch `vram-lend`, the same Swift 1.5 IQ4_XS with its MTP
+draft head and vision encoder on an explicit split over all five GPUs. The
+experts of 14 layers stay in host RAM behind an 8192 MiB MoE expert cache on
+the RTX 5060 Ti (0000:05:00.0) and on one RTX 4060 Ti (0000:09:00.0); those
+two cards lend 8112 MiB each. An 8 GiB lend on 0000:05:00.0 while a request
+streams:
+
+| Check | 0000:05:00.0 |
+|---|---|
+| Lend answer / driver shows the memory free | 0.04-0.05 s / at or before the answer |
+| Largest token gap, lend / return | 0.11 s / 0.11-0.12 s |
+| Return answer | 0.03-0.10 s |
+| 2 GiB on each lending GPU, other GPUs | all pass, 0 MiB change |
+| Prompt rate at full lendable | 101% of the rate with nothing lent (the prompt path reads host experts directly) |
+| Lendable before the lend = after the return | 8112 = 8112 MiB, 20 cycles |
+| Decode with 8 GiB lent | 25.0 t/s against 58.3 t/s (the host-expert rate: a full lend leaves each expert group its floor only); back within 5% on the first 300-token run after the return |
+
+Over lendable, two leases on one GPU, a return while the memory is in use
+(409, retry), a lend during a 63k-token prompt read (answered at the next
+batch, 1.2-1.4 s), a late lend (503 with a 1 ms wait), an engine restart,
+and the quality suite (coding 16/16, math 12/12, needles 6/6 at 32k and 128k
+with both cards fully lent) all pass. Three lend cycles under the CUDA memory
+checker: 0 errors. The lend-ready layout itself costs 36% of the prompt rate
+against the plain split (18 GiB of experts read from host RAM) and nothing
+on decode. HIP and SYCL builds were not run. The full tables are in
+[bench/results/](bench/results/).
+
+## Agent instructions
+
+An agent that needs GPU memory for a short job on a machine where a model is
+served - an Android emulator with `-gpu host`, a speech or image service, a
+small test model, a bench - can lend it from the engine with the `vram-lend`
+command instead of stopping the model. The engine keeps serving, a little
+slower, until the lease is returned.
+
+Install the command and the skill:
+
+```
+pipx install vram-lend          # or: pip install vram-lend
+vram-lend skills install        # the skill, into every agent harness on the machine
+```
+
+`skills install` finds Claude Code, pi, Hermes and opencode and copies the
+skill into each one's user skills directory (`--scope project` for the
+current repository only, `--harness NAME` for one harness). `skills status`
+and `skills update` keep it current. The skill tells the agent to find the
+card, read `vram-lend state`, run the job under `vram-lend run`, and what
+each refusal means. It also tells it what it must never do: start, swap or
+restart a model, guess ports, or use memory after a refusal.
+
+The command itself is one Python file with no dependencies outside the
+standard library, `src/vram_lend/lend.py`. Without pip, copy it onto PATH as
+`vram-lend` and copy `src/vram_lend/skills/vram-lend/` into the harness's
+skills directory by hand (`/home/you/.claude/skills/` for Claude Code,
+`/home/you/.pi/agent/skills/` for pi).
+
+One run, with the default server `http://127.0.0.1:8080` (llama-swap) and
+the job on the first RTX 4060 Ti:
+
+```
+vram-lend state
+vram-lend run --gpu 0000:09:00.0 --mib 2048 emulator -avd Pixel_8 -gpu host
+```
+
+`run` lends 2048 MiB on that card from whichever serving engine lists it,
+starts the emulator with the lease in its environment, and returns the lease
+when the emulator exits, also on a signal. For a job that is stopped later,
+`vram-lend lend` prints the lease and `vram-lend return --id <id> --endpoint
+<endpoint>` gives it back. `--server URL` or `VRAM_LEND_SERVER` names an
+engine port or another llama-swap; `VRAM_LEND_API_KEY` carries llama-swap's
+key. The verbs mirror
+the HTTP API in [docs/api.md](docs/api.md), and the skill itself is at
+[src/vram_lend/skills/vram-lend/SKILL.md](src/vram_lend/skills/vram-lend/SKILL.md).
+
+## Documentation
+
+- [docs/api.md](docs/api.md): the HTTP API (`/vram/lend`, `/vram/return`,
+  `/vram/state`), status codes and lease rules.
+- [docs/llama-swap.md](docs/llama-swap.md): the same API through
+  llama-swap's `/upstream/<model>/` passthrough for every engine it runs:
+  the `ignorePaths` entry, what a call does on a loaded and an unloaded
+  model, the fork branch that keeps a call outside the scheduler, the Berth
+  `lend_url`, status codes and common failures.
+- [docs/engine-protocol.md](docs/engine-protocol.md): the control lines
+  between the server and the engine (`LEND`, `RETURN`, `LENDABLE`), when the
+  engine answers, answer order and late leases.
+- [docs/implementing.md](docs/implementing.md): how to add VRAM Lend to an
+  engine: the VMM pool, the order before an unmap, the lendable rule, and the
+  lessons from the defects found.
+- [docs/research/findings.md](docs/research/findings.md): prior art, where
+  the memory is, what a lend costs, and why the memory must go back to the
+  driver. Also [glossary.md](docs/research/glossary.md) and
+  [questions.md](docs/research/questions.md).
+- [bench/README.md](bench/README.md): the scripts behind the results.
+
+## Licence
+
+MIT, see [LICENSE](LICENSE). The Maya fork branch keeps the licence of
+Project Maya, the Strata and llama.cpp fork branches the licences of those
+projects.

vram_lend-0.1.0/README.md

@@ -0,0 +1,172 @@
+# VRAM Lend
+
+VRAM Lend lets an inference engine lend part of its GPU memory to another
+program on the same machine while it keeps serving requests, and take the
+memory back after.
+
+A large MoE model on consumer GPUs fills every GPU, mostly with its expert
+cache: copies of experts that also live in RAM or on disk. An image
+generator, a speech service or a small helper model needs a few GiB on one
+GPU for a short time. With VRAM Lend the engine empties that many MiB of its
+expert cache on the GPU asked for and releases them to the driver at once.
+The borrower allocates the memory, uses it and frees it, and a return gives
+it back to the engine. The engine never stops serving.
+
+```
+POST /vram/lend   {"gpu": "0000:09:00.0", "mib": 8192}  ->  {"id": "7", "gpu": "0000:09:00.0", "mib": 8208}
+POST /vram/return {"id": "7"}                           ->  {"id": "7", "gpu": "0000:09:00.0", "mib": 8208}
+GET  /vram/state                                        ->  {"gpus": [...], "leases": [...]}
+```
+
+## Status
+
+| Component | Status | Code |
+|---|---|---|
+| Project Maya (GLM-5.3-Flash and DeepSeek-V4.1 engines) | Done: lend and return on any GPU while requests run | Fork branch: <https://github.com/TacoTakumi/project-maya/tree/vram-lend>. Upstream PR: not opened yet |
+| Strata | Lend and return on both GPUs of a 2x RTX 3090 layer split while requests run | Fork branch: <https://github.com/TacoTakumi/Strata/tree/vram-lend>, on [Niko1221/Strata](https://github.com/Niko1221/Strata) v0.2.7.1 with upstream PR [#2522](https://github.com/Niko1221/Strata/pull/2522), which now carries the four fixes from this work as its own commits. Upstream issue: [#4168](https://github.com/Niko1221/Strata/issues/4168) |
+| llama.cpp | Done: lend and return of the MoE expert cache on any GPU that has one, while requests run. Served on this host through llama-swap as its own entry | Fork branch: <https://github.com/TacoTakumi/llama.cpp/tree/vram-lend>, on master 961c376a2 with the per-device `--moe-cache-mib` of [#30205](https://github.com/ggml-org/llama.cpp/pull/30205). Upstream PR: not opened yet |
+| llama-swap (one address for every engine) | Done as a general change: a request matching `upstream.ignorePaths` on a loaded model is served outside the scheduler (no start, no concurrency slot, no in-flight count, no ttl reset), so `/upstream/<model>/vram/lend`, `/vram/return` and `/vram/state` with `^/vram/` in `ignorePaths` reach a serving engine and never load one. Ten lines of logic, tests, docs. See [docs/llama-swap.md](docs/llama-swap.md) | Fork branch: <https://github.com/TacoTakumi/llama-swap/tree/untracked-paths>, on [mostlygeek/llama-swap](https://github.com/mostlygeek/llama-swap) 6c855da (v267). Upstream issue: [#1135](https://github.com/mostlygeek/llama-swap/issues/1135), existing. Upstream PR: not opened yet. Replaces the earlier `vram-route` branch (a `/vram` route on llama-swap's own port, routed by card) |
+| `vram-lend` command and agent skill (borrower side) | Done: one command that finds the engine, directly or behind llama-swap, lends on the card named, runs the job and returns the lease when the job exits; a skill that teaches an agent when and how to use it. See [Agent instructions](#agent-instructions) | This repository: `src/vram_lend/`, the skill in `src/vram_lend/skills/vram-lend/`. Package `vram-lend` on PyPI |
+| Berth (broker side) | Done on one GPU, as an experimental feature of the v0.4.0 pre-releases (rc.3 tested on this host: a 6.5 GiB voice borrower on an RTX 4060 Ti held by Maya, lend, grant and take back while Maya streams). Several GPUs under one lease are planned | [v0.4.0-rc.3](https://github.com/T-Crypt/berth/releases/tag/v0.4.0-rc.3), [contrib/LEND-PROTOCOL.md](https://github.com/T-Crypt/berth/blob/next/contrib/LEND-PROTOCOL.md). Design and tests: <https://github.com/T-Crypt/berth/issues/189> |
+
+## Results
+
+Maya fork, final builds before and after the rebase onto upstream v1.1.27
+(the results pages give the build of each row), on 2x RTX 3090, RTX 5060 Ti
+and 2x RTX 4060 Ti. An 8 GiB lend on an RTX 4060 Ti while a request streams:
+
+| Check | GLM-5.3-Flash | DeepSeek-V4.1 |
+|---|---|---|
+| Lend answer / driver shows the memory free | 0.36-0.41 s / 0.01-0.04 s after | 0.16-0.22 s / 0.01-0.05 s after |
+| Largest token gap, lend / return | 0.40-0.45 s / 0.16-0.23 s | 0.23-0.28 s / 0.12-0.21 s |
+| Return answer | 0.05-0.07 s | 0.08-0.13 s |
+| 2 GiB on each GPU, other GPUs | all pass, 0 MiB change | all pass, 0 MiB change |
+| Prompt rate at full lendable | 91% of the rate with nothing lent | 85-98% |
+| Lendable before the lend = after the return | 10320 = 10320 MiB | 10478 = 10478 MiB |
+| Decode with 8 GiB lent | within noise (52.5 / 50.8 / 52.2 and 47.7 / 48.8 / 49.3 t/s before / lent / after) | no penalty visible |
+
+Over lendable (409), two leases on one GPU, a return while the memory is in
+use (409, retry), a lend during a long prompt read, a late lease given back by
+the server, and an engine restart all pass.
+
+Strata fork, branch `vram-lend`, Swift 1.5 IQ4_XS on a layer split over the
+2x RTX 3090 (0000:01:00.0 on a 24.4 GB/s PCIe link, 0000:02:00.0 on a
+6.1 GB/s link). An 8 GiB lend on each RTX 3090 while a request streams:
+
+| Check | 0000:01:00.0 | 0000:02:00.0 |
+|---|---|---|
+| Lend answer / driver shows the memory free | 0.04-0.05 s / 0.01-0.04 s after | 0.04-0.05 s / 0.01-0.02 s after |
+| Largest token gap, lend / return | 0.046-0.056 s / 0.029-0.039 s | 0.053-0.055 s / 0.067-0.074 s |
+| Return answer | 0.01-0.02 s | 0.02-0.03 s |
+| 2 GiB on each GPU, other GPUs | all pass, 0 MiB change | all pass, 0 MiB change |
+| Prompt rate at full lendable | 98.5-99.7% of the rate with nothing lent | 69-73%: Failed against the 90% target, accepted as the slow link's cost |
+| Lendable before the lend = after the return | 12800 = 12800 MiB | 12288 = 12288 MiB |
+| Decode with 8 GiB lent | -5.6 / -6.5% (hit rate 99.6 to 98.3%) | -11.5 / -9.5% (hit rate 99.6 to 97.2%) |
+
+A lend during a long prompt read is answered at the next prompt chunk. Over
+lendable, two leases on one GPU, a return while the memory is in use, an
+engine restart, 50 lend/return cycles during streaming requests and the
+quality suite with 8 GiB lent on each GPU all pass. A late lend (503) was
+not reachable, because the engine answers inside the server's wait; the
+server's unit tests cover that path. HIP and SYCL builds were not run.
+
+llama.cpp fork, branch `vram-lend`, the same Swift 1.5 IQ4_XS with its MTP
+draft head and vision encoder on an explicit split over all five GPUs. The
+experts of 14 layers stay in host RAM behind an 8192 MiB MoE expert cache on
+the RTX 5060 Ti (0000:05:00.0) and on one RTX 4060 Ti (0000:09:00.0); those
+two cards lend 8112 MiB each. An 8 GiB lend on 0000:05:00.0 while a request
+streams:
+
+| Check | 0000:05:00.0 |
+|---|---|
+| Lend answer / driver shows the memory free | 0.04-0.05 s / at or before the answer |
+| Largest token gap, lend / return | 0.11 s / 0.11-0.12 s |
+| Return answer | 0.03-0.10 s |
+| 2 GiB on each lending GPU, other GPUs | all pass, 0 MiB change |
+| Prompt rate at full lendable | 101% of the rate with nothing lent (the prompt path reads host experts directly) |
+| Lendable before the lend = after the return | 8112 = 8112 MiB, 20 cycles |
+| Decode with 8 GiB lent | 25.0 t/s against 58.3 t/s (the host-expert rate: a full lend leaves each expert group its floor only); back within 5% on the first 300-token run after the return |
+
+Over lendable, two leases on one GPU, a return while the memory is in use
+(409, retry), a lend during a 63k-token prompt read (answered at the next
+batch, 1.2-1.4 s), a late lend (503 with a 1 ms wait), an engine restart,
+and the quality suite (coding 16/16, math 12/12, needles 6/6 at 32k and 128k
+with both cards fully lent) all pass. Three lend cycles under the CUDA memory
+checker: 0 errors. The lend-ready layout itself costs 36% of the prompt rate
+against the plain split (18 GiB of experts read from host RAM) and nothing
+on decode. HIP and SYCL builds were not run. The full tables are in
+[bench/results/](bench/results/).
+
+## Agent instructions
+
+An agent that needs GPU memory for a short job on a machine where a model is
+served - an Android emulator with `-gpu host`, a speech or image service, a
+small test model, a bench - can lend it from the engine with the `vram-lend`
+command instead of stopping the model. The engine keeps serving, a little
+slower, until the lease is returned.
+
+Install the command and the skill:
+
+```
+pipx install vram-lend          # or: pip install vram-lend
+vram-lend skills install        # the skill, into every agent harness on the machine
+```
+
+`skills install` finds Claude Code, pi, Hermes and opencode and copies the
+skill into each one's user skills directory (`--scope project` for the
+current repository only, `--harness NAME` for one harness). `skills status`
+and `skills update` keep it current. The skill tells the agent to find the
+card, read `vram-lend state`, run the job under `vram-lend run`, and what
+each refusal means. It also tells it what it must never do: start, swap or
+restart a model, guess ports, or use memory after a refusal.
+
+The command itself is one Python file with no dependencies outside the
+standard library, `src/vram_lend/lend.py`. Without pip, copy it onto PATH as
+`vram-lend` and copy `src/vram_lend/skills/vram-lend/` into the harness's
+skills directory by hand (`/home/you/.claude/skills/` for Claude Code,
+`/home/you/.pi/agent/skills/` for pi).
+
+One run, with the default server `http://127.0.0.1:8080` (llama-swap) and
+the job on the first RTX 4060 Ti:
+
+```
+vram-lend state
+vram-lend run --gpu 0000:09:00.0 --mib 2048 emulator -avd Pixel_8 -gpu host
+```
+
+`run` lends 2048 MiB on that card from whichever serving engine lists it,
+starts the emulator with the lease in its environment, and returns the lease
+when the emulator exits, also on a signal. For a job that is stopped later,
+`vram-lend lend` prints the lease and `vram-lend return --id <id> --endpoint
+<endpoint>` gives it back. `--server URL` or `VRAM_LEND_SERVER` names an
+engine port or another llama-swap; `VRAM_LEND_API_KEY` carries llama-swap's
+key. The verbs mirror
+the HTTP API in [docs/api.md](docs/api.md), and the skill itself is at
+[src/vram_lend/skills/vram-lend/SKILL.md](src/vram_lend/skills/vram-lend/SKILL.md).
+
+## Documentation
+
+- [docs/api.md](docs/api.md): the HTTP API (`/vram/lend`, `/vram/return`,
+  `/vram/state`), status codes and lease rules.
+- [docs/llama-swap.md](docs/llama-swap.md): the same API through
+  llama-swap's `/upstream/<model>/` passthrough for every engine it runs:
+  the `ignorePaths` entry, what a call does on a loaded and an unloaded
+  model, the fork branch that keeps a call outside the scheduler, the Berth
+  `lend_url`, status codes and common failures.
+- [docs/engine-protocol.md](docs/engine-protocol.md): the control lines
+  between the server and the engine (`LEND`, `RETURN`, `LENDABLE`), when the
+  engine answers, answer order and late leases.
+- [docs/implementing.md](docs/implementing.md): how to add VRAM Lend to an
+  engine: the VMM pool, the order before an unmap, the lendable rule, and the
+  lessons from the defects found.
+- [docs/research/findings.md](docs/research/findings.md): prior art, where
+  the memory is, what a lend costs, and why the memory must go back to the
+  driver. Also [glossary.md](docs/research/glossary.md) and
+  [questions.md](docs/research/questions.md).
+- [bench/README.md](bench/README.md): the scripts behind the results.
+
+## Licence
+
+MIT, see [LICENSE](LICENSE). The Maya fork branch keeps the licence of
+Project Maya, the Strata and llama.cpp fork branches the licences of those
+projects.

vram_lend-0.1.0/pyproject.toml

@@ -0,0 +1,46 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "vram-lend"
+version = "0.1.0"
+description = "Lend yourself GPU memory from a serving inference engine, run a job, give it back"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+dependencies = [
+    "agentsquire>=0.5.0",
+]
+keywords = ["gpu", "vram", "llama.cpp", "llama-swap", "agent", "skill"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: POSIX :: Linux",
+    "Environment :: Console",
+]
+
+[project.urls]
+Homepage = "https://github.com/TacoTakumi/vram-lend"
+
+[project.scripts]
+vram-lend = "vram_lend.cli:main"
+
+# Mark vram_lend as a skill-carrying package (AgentSquire marker group). The key
+# and value are the importable package name. Nothing reads this today.
+[project.entry-points.'agentsquire.skills']
+vram_lend = "vram_lend"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/vram_lend"]
+
+[tool.hatch.build.targets.sdist]
+only-include = ["src", "tests", "README.md", "LICENSE", "pyproject.toml"]
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

vram_lend-0.1.0/src/vram_lend/__init__.py

@@ -0,0 +1,3 @@
+"""vram-lend: lend yourself GPU memory from a serving inference engine."""
+
+__version__ = "0.1.0"

vram_lend-0.1.0/src/vram_lend/cli.py

@@ -0,0 +1,41 @@
+"""Console-script entry for vram-lend.
+
+The lend verbs live in lend.py, which imports only the standard library so the
+file also runs on its own. This module adds the one verb that needs a
+dependency: ``skills``, AgentSquire's install/status/update/uninstall group.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from . import lend
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = list(sys.argv[1:] if argv is None else argv)
+    _notice_stale_skill()
+    if args and args[0] == "skills":
+        from agentsquire.cli import skills_command_group
+
+        group = skills_command_group("vram_lend", default_scope="user")
+        return group.main(args=args[1:], prog_name="vram-lend skills",
+                          standalone_mode=False) or 0
+    return lend.main(args)
+
+
+def _notice_stale_skill() -> None:
+    """One advisory line on stderr when an installed copy of the skill is
+    older than the shipped one. Never prompts, never updates, never changes
+    the exit code; CI or AGENTSQUIRE_NO_UPDATE_CHECK silence it."""
+    try:
+        from agentsquire import BundledPackageDataSource, check_stale
+
+        check_stale(BundledPackageDataSource("vram_lend"), prog_name="vram-lend",
+                    update_command="vram-lend skills update")
+    except Exception:
+        pass
+
+
+if __name__ == "__main__":
+    sys.exit(main())