hugpy-fleet 0.2.2__tar.gz

hugpy_fleet-0.2.2/LICENSE

@@ -0,0 +1,41 @@
+hugpy — Source-Available License
+
+Copyright (c) 2026 putkoff (hugpy.ai). All rights reserved.
+
+Permission is granted, free of charge, to use this software ("hugpy") for
+personal and non-commercial purposes, and for time-limited commercial
+evaluation, subject to the following conditions:
+
+1. Non-commercial use means use by an individual for personal purposes, or
+   use by a non-profit or educational institution for its own internal
+   purposes. Any use by, for, or on behalf of a for-profit business or in
+   connection with revenue-generating activity is commercial use — including
+   internal business use, use in producing goods or services, and use on
+   paid engagements.
+
+2. Commercial use requires a commercial license from the copyright holder.
+   Exception: a business may evaluate the software internally for up to
+   thirty (30) days free of charge; continued use after that requires a
+   commercial license.
+
+3. Redistribution of this software, in source or binary form, modified or
+   unmodified, is not permitted without prior written permission from the
+   copyright holder. Downloading the software from an official distribution
+   channel (PyPI, npm, hugpy.ai) is not redistribution.
+
+4. Modification for personal use or internal evaluation is permitted;
+   distribution of modified versions is not.
+
+5. This notice must be retained in all copies or substantial portions of
+   the software.
+
+6. Any use outside these terms automatically terminates this license.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+IN THE SOFTWARE.
+
+For commercial licensing or redistribution permission: https://hugpy.ai

hugpy_fleet-0.2.2/PKG-INFO

@@ -0,0 +1,43 @@
+Metadata-Version: 2.4
+Name: hugpy-fleet
+Version: 0.2.2
+Summary: Hugpy fleet: central worker registry, enrollment, heartbeats, evictions, doctrine and placement, plus the full, GGUF and phone worker agents
+Author-email: putkoff <support@hugpy.ai>
+License-Expression: LicenseRef-Proprietary
+Project-URL: Homepage, https://hugpy.ai
+Project-URL: Documentation, https://github.com/hugpy/hugpy/blob/main/py/fleet/hugpy_fleet/README.md
+Project-URL: Repository, https://github.com/hugpy/hugpy
+Project-URL: Source, https://github.com/hugpy/hugpy/tree/main/py/fleet/hugpy_fleet
+Project-URL: Issues, https://github.com/hugpy/hugpy/issues
+Project-URL: Changelog, https://github.com/hugpy/hugpy/releases
+Project-URL: Architecture, https://github.com/hugpy/hugpy/blob/main/PARTITION.md
+Keywords: hugpy,llm,self-hosted,gpu,fleet,workers,distributed-inference,rpc-sharding
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: hugpy-platform
+Requires-Dist: hugpy-control
+Requires-Dist: hugpy-storage
+Requires-Dist: hugpy-engine
+Requires-Dist: flask>=3
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Provides-Extra: media
+Requires-Dist: hugpy-media; extra == "media"
+Provides-Extra: video
+Requires-Dist: hugpy-video; extra == "video"
+Dynamic: license-file
+
+# hugpy-fleet
+
+`hugpy_fleet` — extracted from `abstract_hugpy_dev` as part of the Hugpy
+partition. Ownership and allowed dependencies are declared in
+`py/partition.toml`; see `PARTITION.md` at the workspace root.
+
+Allowed Python dependencies inside the ecosystem: hugpy_platform, hugpy_control, hugpy_storage, hugpy_engine.

hugpy_fleet-0.2.2/README.md

@@ -0,0 +1,7 @@
+# hugpy-fleet
+
+`hugpy_fleet` — extracted from `abstract_hugpy_dev` as part of the Hugpy
+partition. Ownership and allowed dependencies are declared in
+`py/partition.toml`; see `PARTITION.md` at the workspace root.
+
+Allowed Python dependencies inside the ecosystem: hugpy_platform, hugpy_control, hugpy_storage, hugpy_engine.

hugpy_fleet-0.2.2/docs/COMFY-LEDGER.md

@@ -0,0 +1,87 @@
+# The ComfyUI resident ledger
+
+**Since:** 2026-09-22. **Code:** `hugpy_fleet/worker/comfy_ledger.py` (pure),
+bound in `hugpy_fleet/worker/agent.py` (`_comfy_ledger`, `_comfy_need_detail`,
+`_comfy_headroom_target`, `_comfy_busy_reason`). **Tests:**
+`tests/test_comfy_ledger.py`, `tests/test_vram_evict_to_fit.py`,
+`tests/test_comfy_headroom.py`.
+
+## The problem it closes
+
+ComfyUI is an external process. nvidia-smi shows one VRAM lump under its PID
+and nothing says which checkpoints are inside it, so the worker's VRAM budget
+never had a per-occupant figure for comfy — which meant three loops with three
+different targets and no shared notion of *need*:
+
+| Loop | What it did | Blind spot |
+|---|---|---|
+| eviction planner (`_partition_residents`) | ranks measured residents LRU and evicts the minimum set for a load | comfy rows were **always protected** (0.1.137 "out of allocations") |
+| Fix B headroom (`_worker_ensure_comfy_headroom`) | evicts managed models before a comfy gen | cleared a **constant** 7 GiB for every gen — 2 GiB SD1.5 and 12 GiB checkpoints alike |
+| idle watchdog (`comfy_watchdog`) | frees an idle comfy after a TTL, or under contention | a side channel the planner could not see or rank |
+
+The one thing a budget needs — how much *this* checkpoint takes — was never
+computed, although it is knowable before the load: `ComfyRunner` names the
+checkpoint file in the graph and, for the fp16 safetensors comfy serves,
+weights on disk ≈ weights in VRAM.
+
+## What the ledger is
+
+`ComfyLedger` records what hugpy has asked comfy to load since comfy's last
+known `/free`: `{model_key, filename, bytes, loaded_at, last_used}`, oldest
+first. Writes happen at the two places the truth changes:
+
+* `note_dispatch` — inside the headroom hook, right before the runner POSTs
+  the graph; the file is sized by `checkpoint_size_bytes`.
+* `note_freed` — inside `_comfy_free_models` on a 200, which every `/free`
+  goes through (watchdog, evict verb, planner).
+
+Reads:
+
+* `_vram_residents` unions the ledger's rows in as `host_mode="comfy"` with
+  per-row bytes **capped at comfy's measured process VRAM**, and only when a
+  comfy process actually holds VRAM (a ledger with no process behind it names
+  nothing). The registry still contributes the active call's attribution;
+  the ledger adds what comfy still caches.
+* `_comfy_status()["resident"]` carries the same rows in the heartbeat, so the
+  console's comfy card can show checkpoints by name instead of one lump.
+
+## The three behaviour changes
+
+1. **Per-checkpoint need.** `_comfy_need_detail` prices a gen as
+   `checkpoint file size + gen cushion`, or the cushion alone when comfy
+   already holds the checkpoint *and* its measured VRAM covers the weights (a
+   ledger claim the device does not back is dropped, never trusted).
+   `_comfy_headroom_target` uses that need; when the file cannot be sized the
+   legacy constant applies unchanged, and an **explicitly set**
+   `HUGPY_COMFY_TARGET_FREE_GIB` is a floor the need never undercuts.
+2. **Comfy is an evictable resident under contention.** `_partition_residents`
+   asks `_comfy_busy_reason` once per partition — the watchdog's own predicate
+   (a registered comfy call, a non-empty `/queue`, or anything unreadable = busy)
+   — and comfy rows are candidates when idle, protected with
+   `why="comfy busy: …"` otherwise. `_evict_model`'s comfy branch honours the
+   same predicate (unless `force`), because `/free` drops every checkpoint at
+   once. The **idle sweep** still never touches comfy; TTL-based reclaim stays
+   the watchdog's job.
+3. **Named residents in telemetry** (above).
+
+## Knobs
+
+| Env | Default | Meaning |
+|---|---|---|
+| `HUGPY_COMFY_GEN_CUSHION_GIB` | `2.0` | working room on top of the weights (activations, VAE, CLIP, fragmentation); ae recon: SD1.5 at 512² grew the comfy process ~1 GiB during a gen |
+| `HUGPY_COMFY_TARGET_FREE_GIB` | `7.0` (fallback only) | the legacy constant: used when the checkpoint cannot be sized; a floor when set explicitly |
+| `COMFY_CHECKPOINTS_DIR` | `~/ComfyUI/models/checkpoints` | where hugpy symlinks checkpoints for comfy (`hugpy_storage.provision`); first place a name is resolved |
+| `COMFY_CHECKPOINT_DIRS` | unset | `os.pathsep`-separated extra roots — set it to the roots comfy's own `extra_model_paths.yaml` scans on this box so a name resolves to the file comfy will really load |
+
+Resolution order for a `ckpt_name`: as given under each root, then by basename
+anywhere below each root (comfy's scan is recursive). Symlinks are followed;
+a dangling link is "not found" (unknown need → legacy target), never 0.
+
+## Doctrine kept
+
+* Degrade-not-guess: unmeasurable free VRAM, an unsizable file, a failing
+  catalog read — each falls back to exactly the previous behaviour.
+* A render is never killed for an LLM load: the busy predicate is the
+  watchdog's, and unprovable idleness reads as busy.
+* Comfy's weights are never counted as headroom for another model's admission
+  (`_subject_resident_vram_bytes` is unchanged).

hugpy_fleet-0.2.2/docs/README.md

@@ -0,0 +1,14 @@
+# hugpy-fleet docs
+
+Canonical copies moved from the monolith's `deploy/` folder (partition 2026-09-22).
+
+| Document | Subject | Owner |
+|---|---|---|
+| `STATE-MODEL.md` | Model state nomenclature: residency ladder, ETG cost model, policy gates, divergence register | fleet (Layers 2–3, registry/heartbeat truth); Layer-1 residency vocabulary is shared with `hugpy-engine`'s eviction planner |
+| `WORKER-WIREGUARD.md` | One-step WireGuard join for a remote worker: the wg-peer helper + sudoers, `hugpy-fleet join-code`, the worker join script, and revocation | fleet |
+| `WORKER-BOOT-PREWARM.md` | The per-worker boot-load star (`boot_prewarm`) | fleet |
+| `WORKER-WILDCARD.md` | The per-worker wildcard routing opt-in | fleet |
+| `COMFY-LEDGER.md` | What ComfyUI holds, by name and size: per-checkpoint headroom need, comfy as an evictable resident under contention, the knobs | fleet |
+
+Code paths in these documents refer to the extracted packages
+(`hugpy_fleet/...`, `hugpy_engine/...`).

hugpy_fleet-0.2.2/docs/STATE-MODEL.md

@@ -0,0 +1,270 @@
+# Model State Nomenclature — Canonical Reference
+
+**Status:** canonical vocabulary (target state). Some producers/consumers do not yet
+conform; every known divergence is listed in the **Divergence Register** below with a
+`file:line` and a resolution. New code MUST use these terms; drift against this doc is a
+bug in the code, not in the doc.
+
+**Why this exists:** the same words (`cold`, `hot`, `warm`, `serving`, `resident`,
+`loaded`, `in_flight`) were used for different things in different subsystems, which
+produced real incidents — e.g. an already-on-disk model refused as if it needed a
+multi‑GB download (the cold‑hold `cold_load_capacity` refusal). This doc fixes the
+meanings.
+
+---
+
+## The one rule
+
+**A term names exactly one thing on exactly one layer.**
+
+- **Layer 1 — Residency state**: *where the weights physically are* (a thermal ladder).
+- **Layer 2 — Cost model (ETG)**: *how long until this candidate produces output* (times).
+- **Layer 3 — Policy / eligibility**: *may we, and is it even resolvable* (boolean gates).
+
+State words are nouns for a location. Cost words always carry a time unit (`t_*`, seconds).
+Policy words are adjectives that filter candidates and never enter a cost sum.
+
+---
+
+## Layer 1 — Residency state (the thermal ladder)
+
+"Temperature" = **proximity to GPU compute**. The colder a model is, the more work stands
+between it and producing a token. Per `(worker, model, variant)` the model occupies exactly
+one state:
+
+| State | Meaning | Physical location | Authoritative signal |
+|---|---|---|---|
+| `UNAVAILABLE` | Not resolvable anywhere — not on the fleet source-of-truth nor a remote origin (HF). Cannot be provisioned. **Off the ladder: no worker delegation.** | nowhere fetchable | `serveable=False` + `unserveable_reason` |
+| `COLD` | Resolvable on the **source-of-truth** array / origin, but **not on this worker's hot drive**. Must be pulled before it can load. | `MODELS_HOME` (`/mnt/llm_storage/models`) / HF only | resolvable **and** NOT in worker `models_local` |
+| `HOT` | Present on this worker's **hot drive** — the box-local NVMe it loads into VRAM from. Downloaded, ready to load. | `HUGPY_HOT_CACHE_ROOT` (box-local NVMe hot-cache) | in worker `models_local` (disk-truth) |
+| `LOADED` | Weights **measured resident** in VRAM, idle (not answering a request right now). | GPU VRAM | heartbeat allocation row `healthy && materialized`, not `serving`/`busy` |
+| `SERVING` | Resident in VRAM **and actively answering** a request (within the active window). | GPU VRAM | allocation row `serving` (busy or last_used within `_SERVING_WINDOW_S`) |
+
+### Transitions (each is the unit of work Layer 2 prices)
+
+| Transition | From → To | Name | In-flight state | Signal |
+|---|---|---|---|---|
+| Download | COLD → HOT | **pull** (fetch) | `PULLING` | `provisioning` / `provision_progress` |
+| Load | HOT → LOADED | **load** | `LOADING` (heating) | `loading` |
+| Serve | LOADED → SERVING | — (a request arrives) | — | `serving` flips true |
+| VRAM evict | LOADED/SERVING → HOT | **evict** | `EVICTING` | eviction plan; weights remain HOT on the drive |
+| Hot-drive reap | HOT → COLD | **reap** | — | hot-cache FIFO by time-since-called (`hot_cache.py`) |
+
+Notes:
+- **evict ≠ reap.** Eviction frees **VRAM** (the model falls back to `HOT`, still on the
+  drive — cheap to re-load). Reaping frees **hot-drive space** (the model falls back to
+  `COLD`, must be re-pulled). Conflating these is how "cold" came to mean two things.
+- `RESERVED` (a grant/allocation held but **not** measured-resident, `materialized=false`)
+  is not a residency state — it is a Layer-3 attribution. See Layer 3.
+
+---
+
+## Layer 2 — Cost model (Estimated Time to Generation)
+
+Placement and eviction are **decisions**, and a residency state cannot make a decision — a
+cost can. For each candidate `(worker, model, variant)`:
+
+```
+ETG = t_pull      # >0 only if COLD          bytes / download_bw(worker)
+    + t_evict     # >0 only if it won't fit  cost to evict victim(s) for `need` bytes
+    + t_load      # >0 unless LOADED/SERVING  upload_time_s EMA  (learned per worker-card)
+    + t_queue     # wait behind requests already in flight on that worker/slot   [MISSING]
+    + t_generate  # avg_output_tokens / tok_per_s EMA            (learned per worker-card)
+```
+
+Choose the candidate with **min ETG**. Eviction is the dual: a victim's cost is not its
+bytes, it is its **regret** — `P(needed soon) × (its future t_pull + t_load)`.
+
+**Wiring status (as of this writing): NOT WIRED.**
+- `estimate_total_time` computes `t_pull + t_evict + t_load + t_generate`
+  (`hugpy_fleet/central/model_metrics.py:265-292`) but has **zero callers** — the estimator is orphaned.
+- `t_queue` does not exist yet.
+- Placement (`_pick_worker` → injected provider, `hugpy_engine/resolvers/remote.py:394`) ranks
+  by **residency** (`_resident_on`, `hugpy_fleet/central/workers.py:2646`), not ETG.
+- Eviction (`hugpy_engine/eviction.py`) ranks victims by **recency + bytes**, not regret.
+
+This doc defines the vocabulary the wiring will use; the wiring itself is a separate change.
+
+---
+
+## Layer 3 — Policy / eligibility (boolean gates)
+
+These **filter** the candidate set. They never enter the ETG sum.
+
+| Flag | Meaning | Signal |
+|---|---|---|
+| `serveable` / `unserveable_reason` | Is the model resolvable & offerable at all (the `UNAVAILABLE` gate; "no delegation") | `v1_routes.py:154-156`; `hugpy_engine/resolvers/model_resolver.py` |
+| `assigned` | **Attribution only** — central designated this box. NOT disk presence, NOT eviction protection. | `assigned_models` (`hugpy_fleet/worker/agent.py:3594`); ruling `hugpy_fleet/worker/agent.py:3263-3272` |
+| `materialized` | Honesty bit: were weights actually **measured** in VRAM (vs merely claimed). `false` ⇒ `RESERVED`, not `LOADED`. | `hugpy_fleet/worker/agent.py:1201-1205` |
+| `pinned` | Never evict from VRAM. | residency/pin config |
+| `protected` (+ `why`) | Transient VRAM-evict shield: `static / loaded / loading / provisioning / store-gate`. | `hugpy_fleet/worker/agent.py:3229-3298` |
+| `blocked` | Operator block — refused from the serving pool. | `hugpy_fleet/central/blocklist.py` |
+| `serves_locally` | Worker **policy**: does this box serve at all (`HUGPY_NO_LOCAL_SERVING`). Unrelated to any model's state. | `remote.py:2449` |
+| `evictable` | Derived: `resident && !pinned && !protected && !mid_generation`. | eviction planner |
+
+---
+
+## Canonical term dictionary (quick lookup)
+
+| Term | The one meaning | Layer |
+|---|---|---|
+| **unavailable** | not resolvable anywhere; cannot be delegated | 1 (floor) |
+| **cold** | on source-of-truth/origin, **not** on this worker's hot drive | 1 |
+| **hot** | on this worker's hot drive (box-local NVMe), not in VRAM | 1 |
+| **loaded** | measured-resident in VRAM, idle | 1 |
+| **serving** | resident in VRAM and actively answering | 1 |
+| **pull / fetch** | download source-of-truth → hot drive | 1 (transition) |
+| **load** | hot drive → VRAM | 1 (transition) |
+| **evict** | drop from VRAM (→ hot) | 1 (transition) |
+| **reap** | delete from hot drive (→ cold), hot-cache FIFO | 1 (transition) |
+| **hot drive** | the box-local NVMe a worker loads from (`HUGPY_HOT_CACHE_ROOT`) | storage tier |
+| **source of truth** | the shared model array (`MODELS_HOME`), never reaped | storage tier |
+| **t_pull / t_load / t_evict / t_queue / t_generate** | ETG cost terms (seconds) | 2 |
+| **ETG** | estimated time to generation = sum of `t_*` | 2 |
+| **materialized** | measured-in-VRAM honesty bit | 3 |
+| **assigned** | central attribution only | 3 |
+| **serves_locally** | worker serving policy | 3 |
+
+---
+
+## Divergence Register (shore up here; do not repeat)
+
+Every known place code contradicts this doc. Resolution = what the code should become.
+
+1. **`cold` = "not in VRAM" (should be "not on hot drive").** ✅ RESOLVED (2026-08-29)
+   `hugpy_engine/resolvers/remote.py` (`_admit_cold_hold`, cold-hold gate) treated any
+   not-`healthy` model as cold, refusing it under the **download** cap even when it was
+   already `HOT` (on the worker's drive). The load-state contract had no on-disk field.
+   **Fix applied:** `load_state_for_model` (`hugpy_fleet/central/workers.py`) now emits
+   `on_disk` from `models_local` (disk-truth), and `_admit_cold_hold` admits a HOT model
+   UNCOUNTED — a `HOT` model's admission is a `t_load`, never a `t_pull`, so it no longer
+   consumes a download permit. Landed 2026-08-29; patch archived (fully applied, verified
+   2026-09-14) to `archive/deploy-patches-20260914/div1-cold-hold-on_disk.patch`.
+
+2. **Console labels invert `cold`.** ✅ RESOLVED-by-verification (2026-08-30)
+   The LIVE console SPA is already canon-correct: `WorkerRow.jsx deriveModelState()` maps
+   `onWorkerDisk?'hot':centralHas?'cold':'missing'` with a `🌡 hot` pill and a correct
+   sort-rank ladder. The inversion survives ONLY in a dead backup monolith
+   (`backs/WorkersPanel/WorkersPanel.jsx`, not imported/bundled). The backend no longer
+   encodes the triple (it ships raw signals; the stale `dispatch.py:86` ref is gone). No
+   change needed in live code.
+
+3. **`hot`/`cold` = VRAM-at-pick latency bucket.** ✅ RESOLVED (2026-08-30)
+   `TEMPERATURES=("hot","cold")` used `hot` for in-VRAM-at-pick. **Fix applied:** buckets
+   are now `("loaded","unloaded")` keyed on LOADED-at-pick; `estimate_total_time` default
+   and the record path updated; an idempotent SQLite migration rebuckets legacy hot→loaded
+   / cold→unloaded so history is preserved. Landed 2026-08-30; patch archived (fully
+   applied, verified 2026-09-14) to `archive/deploy-patches-20260914/div3-6-loaded_at_pick.patch`.
+
+4. **Video picker `· cold` = "not answered in 180s".** ✅ RESOLVED (2026-08-30)
+   **Fix applied:** `video_prompt_assist_models` now computes the hottest per-key state
+   across online workers (serving>loaded>hot>cold, folding in `models_local`+`allocations`)
+   and emits a canonical `state` field; the picker (`PromptAssistButtons.tsx`,
+   `GenerateStation.tsx`) renders serving/loaded=ready, `· hot (loads)`, `· cold (downloads)`
+   — `cold` reserved for not-on-hot-drive. Landed 2026-08-30; patch archived (fully applied
+   backend+frontend, verified 2026-09-14) to
+   `archive/deploy-patches-20260914/div4-video-picker-canonical-state.patch`.
+
+5. **`serving` is three things.** ✅ RESOLVED-by-verification (2026-08-30)
+   Audit shows the three concepts are ALREADY distinct at the field level: the alloc-row
+   `serving` is computed solely from *answering within `_SERVING_WINDOW_S` (180s)*
+   (`_slot_serving` / `last_used<window`, `hugpy_fleet/worker/agent.py`) = canonical SERVING; readiness is the
+   separate `healthy` field (load-state seam); policy is `serves_locally` /
+   `HUGPY_NO_LOCAL_SERVING`. No rename needed. Residual `"serving"` appeared only as (a) loose
+   log prose and (b) a LEGACY INPUT alias for the on-demand residency default (originally
+   retained for backward-compat, like `warm` in #6). **Update 2026-08-30 (same day, later):**
+   the compat-shim cleanup removed the `"serving"`/`"warm"` legacy input aliases entirely
+   (operator ruling: no back-compat) — neither string is accepted by `hugpy_fleet/worker/agent.py`'s residency
+   handler or `worker_routes.py:_normalize_residency` any more (verified 2026-09-14). Patch
+   archived to `archive/deploy-patches-20260914/div-cleanup-strip-compat-shims.patch`. New
+   code must not use "serving" to mean "ready".
+
+6. **`warm` is overloaded.** ✅ RESOLVED (2026-08-30)
+   **Fix applied:** the `warm_at_pick` metric → `loaded_at_pick` (producer + both readers;
+   the telemetry `model.warm` field → `model.loaded_at_pick`). The `"warm"` *input* alias
+   (originally retained to map an old client string to the on-demand default) was itself
+   removed by the later same-day compat-shim cleanup (operator ruling: no back-compat) —
+   `"warm"` no longer appears anywhere in `hugpy_fleet/worker/agent.py` (verified 2026-09-14; cleanup patch
+   archived to `archive/deploy-patches-20260914/div-cleanup-strip-compat-shims.patch`).
+   Landed 2026-08-30; patch archived to
+   `archive/deploy-patches-20260914/div3-6-loaded_at_pick.patch`.
+
+7. **`resident` = VRAM in two places, disk in a third.** ✅ RESOLVED (2026-08-30)
+   VRAM predicate (canonical, unchanged), eviction unit, and disk `resident_bytes`.
+   **#7A:** eviction unit class `Resident` → `EvictUnit` (4 constructors + hints; compat
+   alias `Resident = EvictUnit` originally kept). The alias was itself removed by the later
+   same-day compat-shim cleanup (no back-compat) — `Resident` no longer appears anywhere in
+   `hugpy_engine/eviction.py` (verified 2026-09-14). **#7B:** on-disk footprint
+   `resident_bytes`/`resident_model_bytes`/`resident_source` → `hot_bytes`/`hot_model_bytes`/
+   `hot_source`, `gauge_basis:"resident"`→`"hot"` (workers.py, worker_routes.py) — this was a
+   WIRE key read by the console SPA (`FleetResidency.jsx`, `WorkerStorageBar.jsx`,
+   `ResourceStrip.jsx`, `WorkerRow.jsx`); console consumer `WorkerStorageBar.jsx` reads
+   `hot_bytes`. No back-compat. Both landed 2026-08-30; patches archived to
+   `archive/deploy-patches-20260914/div7a-Resident-to-EvictUnit.patch`,
+   `archive/deploy-patches-20260914/div7b-resident_bytes-to-hot_bytes.patch`, and
+   (the alias-removal) `archive/deploy-patches-20260914/div-cleanup-strip-compat-shims.patch`.
+
+8. **`loaded_models` has three scopes under one name.** ✅ RESOLVED-by-verification (2026-08-30)
+   The authoritative shape ALREADY EXISTS: `_allocations()` (heartbeat allocation rows,
+   `hugpy_fleet/worker/agent.py`) reports one row per slot-seated model ∪ per in-process resident — explicitly
+   "a NEW field parallel to loaded_models/slots so old central/UI keep working."
+   `loaded_model_keys()` deliberately subtracts `slot_backed_model_keys()` (stops the
+   loaded/serving flap). `loaded_models` is INTENTIONALLY retained as backward-compat; the
+   canonical residency truth is `allocations`. No change made — the operator's compat design
+   is deliberate.
+
+9. **`/health` is not residency-truthful.** ✅ RESOLVED-by-verification (2026-08-30)
+   Verified the DECISION path already conforms: the central residency predicate
+   `_resident_on()` reads the heartbeat `allocations` rows (honoring a slot row only when
+   `healthy`/`serving`/`busy`) plus `loaded_models` — never `/health`. The only `/health`
+   residency read is an ADDITIVE overlay in `load_state_for_model` that can turn "no
+   movement" into movement but never the reverse, so it cannot produce a false
+   "nothing loaded". No decision is misled; no change needed.
+
+10. **`in_flight` overloaded.** ✅ RESOLVED (2026-08-30)
+    Request-concurrency cap vs eviction mid-generation lock shared the word. **Fix
+    applied:** the eviction lock `Resident.in_flight` → `mid_generation` (internal, the
+    misleading one — removes the collision); the central `WorkerBusyError` concurrency
+    field → `requests_in_flight` (the `"in_flight"` error-payload wire key originally kept
+    for client compat). That wire-key alias was itself dropped by the later same-day
+    compat-shim cleanup (no back-compat) — `remote.py`'s payload now emits only
+    `requests_in_flight` (verified 2026-09-14). gen_gate concurrency counters keep
+    `in_flight` (unambiguous — different subsystem). Landed 2026-08-30; patch archived to
+    `archive/deploy-patches-20260914/div10-in_flight-disambiguation.patch`.
+
+11. **`in_progress` fuses pull + load.** ✅ RESOLVED (2026-08-30)
+    Load-state field documented as "weights loading OR still downloading now" only fed
+    error wording. **Fix applied:** `load_state_for_model` now emits `pulling` (COLD→HOT
+    provisioning) and `loading` (HOT→VRAM) as distinct flags, keeping `in_progress` as
+    their union for forward-progress callers; the `cold_load_capacity` refusal now names
+    the actual transition ("still downloading onto" / "still loading into VRAM on" / "not
+    loaded yet"). Landed 2026-08-30; patch archived to
+    `archive/deploy-patches-20260914/div11-in_progress-split.patch`.
+
+12. **`assigned` was conflated with disk presence.** ✅ RESOLVED (by prior ruling) — stated here:
+    `assigned_models` is attribution only (`hugpy_fleet/worker/agent.py:3263-3272`, `hugpy_fleet/central/workers.py:2245-2253`);
+    the on-disk signal is `models_local` (`hugpy_fleet/worker/agent.py:5129`). Do not reintroduce the conflation.
+
+---
+
+## Retired words (do not use in new code)
+
+| Retired | Use instead |
+|---|---|
+| `warm` (any sense) | `LOADED` / `SERVING` (state) or `residency_mode="on_demand"` (config) |
+| `hot` for "in VRAM" | `LOADED` / `SERVING`; `hot` means **on the hot drive** only |
+| `cold` for "not in VRAM" | `HOT` (on drive, not loaded) — `cold` means **not on the hot drive** |
+| `serving` for "ready/healthy" | `LOADED` (resident-idle) |
+| `serving` for the policy | `serves_locally` |
+| `resident` for on-disk bytes | `hot_bytes` |
+| `in_flight` for evict lock | `mid_generation` |
+
+---
+
+## Glossary anchors (source of truth for the reconciliation)
+
+- Hot-drive doctrine (verbatim operator ruling): `hugpy_engine/serve/hot_cache.py` header.
+- Two-stage lifecycle (`provision.*` vs `load.*`) and residency tiers: `hugpy_fleet/central/evictions.py:110-183`.
+- ETG estimator (orphaned): `hugpy_fleet/central/model_metrics.py:8-9,265-292`.
+- Disk-truth signal: `hugpy_fleet/worker/agent.py:5129-5171` (`models_local`).
+- Residency truth (heartbeat allocation rows): `hugpy_fleet/worker/agent.py:1830-2010`.

hugpy_fleet-0.2.2/docs/WORKER-BOOT-PREWARM.md

@@ -0,0 +1,135 @@
+# Per-worker BOOT-LOAD STAR (`boot_prewarm`)
+
+Operator RULING **2026-07-23 (post-incident)** — verbatim:
+
+- "the star is only supposed to indicate **load that model on boot**."
+- "it **shouldn't effect anything but priority for ambiguous model calls**."
+
+A worker carries a single ⭐ **star** model. The star does **exactly two things
+and nothing else**:
+
+1. **Load on boot** — once per worker process lifetime.
+2. **Ranking priority** — a tie-break for ambiguous / no-warm model calls:
+   central prefers the worker whose star == the requested model.
+
+It is **NOT keep-warm**, it does **NOT re-warm after eviction**, and it has **no
+eviction interaction**. A starred model that gets evicted under pressure **stays
+cold until the worker process restarts**. The identifier stays
+`boot_prewarm`/`boot-prewarm` (rename churn isn't worth it) and once again means
+exactly **boot-once**.
+
+> **Why the revert (incident 2026-07-23).** The prior release (0.1.201) made the
+> star **reconcile-kept-warm** — central re-warmed it every beat and the worker
+> reloaded it whenever it was absent. On **ae** that re-warm fired against
+> **coder-next while active inference was in flight** → the star's slot child
+> stalled → a zombie seat → the agent froze. Re-warm-after-eviction is only safe
+> once a **co-fit gate** (reload an evicted model only when it co-fits its
+> evictor) exists — that is **future work (Slice D), not yet built** — so the
+> star is strictly boot-once until then.
+
+## The three levers (locked semantics)
+
+| lever | keeps warm? | boot-loads? | eviction | notes |
+|---|---|---|---|---|
+| ⭐ **star** (`boot_prewarm`) | **No** | **Yes — once, on boot** | **Evictable; once evicted STAYS cold until restart** | Also a **ranking tie-break** for ambiguous calls. NOT eviction-protected, NOT re-warmed. |
+| 🔒 **static** | **Yes** — the keep-warm tier | yes (eager) | **Protected** — never evicted by the LLM plane | "start here AND stay here". |
+| 📌 **pin** | **No** | no | n/a | Routing persistence only — never warms. |
+
+🔒 **static is the keep-warm tier** — the one lever that keeps a model resident
+across evictions. If you want "start here **and stay here**", promote the model
+to static; the ⭐ star will not hold it.
+
+And for contrast, the /media star:
+
+| | media_default (the /media star) | ⭐ boot_prewarm (this per-worker star) |
+|---|---|---|
+| Scope | one global model | one model **per worker** |
+| Effect | "first in the list + default-selected" — a **UI/routing preference** | "**boot-load** this model + rank it first for ambiguous calls" |
+| Loads anything? | **No** | **Yes — once, on the worker's boot** |
+
+**Nothing warms until starred (boot-load) or static.** The fleet `TASK_DEFAULTS`
+(sd-turbo et al.) are a **routing fallback** — a request naming only a task still
+resolves to its default model (`model_resolver.TASK_DEFAULTS`) — but they are
+**not kept warm** (the task-defaults floor is dead and is not coming back). 📌
+pins never warm. Everything but 🔒static lazy-loads on first real request (the
+star additionally boot-loads once).
+
+### How the star works
+
+- **Worker side — boot-load once.** `_adopt_boot_prewarm` runs on every
+  register/heartbeat reply, but a **process-lifetime done-latch**
+  (`_BOOT_PREWARM_DONE`) makes it fire the load **exactly once**: on the **first**
+  reply carrying a star. It loads through the normal on-demand path (no residency
+  write — the model stays FIFO-evictable). Every later beat is a **no-op**,
+  **including after an eviction** — the star is **not** reloaded. A genuine retry
+  only comes with a worker **restart**. Missing model → logged, never crashes.
+- **Central side — ranking priority only.** `_reconcile_warm_set(worker)` keeps
+  warm **`🔒static ∩ models_local − blocked`** — the **star is not in it**, so
+  central never re-probes the star warm. The star's only central effect is a
+  **ranking tie-break**: in `pick_for_model` / `candidates_for_model` the sort key
+  is `(home, warm, star, gpu, last_picked, id)` — home beats everything, a warm
+  box beats a starred box, and the star breaks the tie when **nothing is warm**
+  (prefer the box that boot-loads the model anyway). Alias-tolerant: a star
+  recorded under a `~`-qualified key matches a bare-key request and vice versa.
+
+**Future work (Slice D, not built): co-fit-gated re-entry** — an evicted star
+would reload only when it **co-fits** with whatever evicted it (no thrash by
+construction). Until that exists, an evicted star stays cold until restart.
+
+## API
+
+All paths are relative to central (e.g. `https://dev.hugpy.ai/api`). Worker ids
+come from `GET /llm/workers` (the `id` field).
+
+- **Set / replace a worker's star** (operator-gated):
+  `POST /llm/workers/<worker_id>/boot-prewarm`
+  Body: `{"model_key": "<key>", "enabled": true}` — makes `model_key` the star,
+  replacing any previous one. `{"enabled": false}` (optionally with a matching
+  `model_key`, or none) clears it.
+
+- **Read the full star map** (open, read tier): `GET /llm/workers/boot-prewarm`
+  → `{"<worker_id>": "<model_key>", ...}`
+
+- **Per-worker surfacing**: every row of `GET /llm/workers` (and
+  `GET /llm/workers/<id>`) carries `"boot_prewarm": "<model_key>"|null`, so the
+  console can render the star.
+
+The star rides the register/heartbeat reply to the worker as
+`boot_prewarm: "<model_key>"` — additive and **omit-when-unset**, so a released
+worker that predates the feature simply ignores it (the `extra=forbid` relay
+schema is never broken).
+
+## Seed the operator's two defaults
+
+These are **data, not code** — run them once against central with the operator
+token. Replace `<computron-id>` / `<ae-id>` with the ids from `GET /llm/workers`.
+
+```bash
+# computron (RTX-4060) -> a present 7B. Use whichever 7B key the fleet actually
+# holds (check `GET /llm/workers/<computron-id>` models / `GET /models`):
+#   Qwen~Qwen2-7B-Instruct-GGUF   (or Qwen~Qwen2.5-7B-Instruct-GGUF)
+curl -fsS -X POST https://dev.hugpy.ai/api/llm/workers/<computron-id>/boot-prewarm \
+  -H "X-Operator-Token: $HUGPY_OPERATOR_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"model_key": "Qwen~Qwen2-7B-Instruct-GGUF", "enabled": true}'
+
+# ae (RTX-3090) -> the agent brain, coder-next (DEFAULT_AGENT_BRAIN):
+curl -fsS -X POST https://dev.hugpy.ai/api/llm/workers/<ae-id>/boot-prewarm \
+  -H "X-Operator-Token: $HUGPY_OPERATOR_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"model_key": "Qwen~Qwen3-Coder-Next-GGUF", "enabled": true}'
+```
+
+Verify: `GET /llm/workers/boot-prewarm` returns both, and each worker
+**boot-loads** its star once on start (`boot star … loading once at boot …` in
+the agent log; then every later beat is a no-op — the star is **not** re-warmed).
+An evicted star stays cold until the worker restarts.
+
+### Note on the 7B key for computron
+
+The brief named `Qwen2-7B-Instruct-GGUF` / `Qwen2.5 7B`. Neither is a curated
+staple in `models_config.MODELS` (they arrive via discovery), so the exact
+`model_key` is whatever the fleet discovered — confirm against `GET /models`
+before seeding. The star store does **not** require the model to be present or
+allocated when you set it; if the model can't be loaded, the worker logs and
+continues (never crashes).