pixie-lab 0.3.2__tar.gz → 0.4.0__tar.gz

{pixie_lab-0.3.2 → pixie_lab-0.4.0}/CHANGELOG.md

@@ -11,6 +11,132 @@ operator-facing summary.
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-07-24
+
+### Changed
+
+**Machines list Image column reads like the new model.** The bound
+image now shows the catalog name linked to its Images page (short sha
+for an orphan blob whose entry was deleted), instead of a bare hash,
+and is blank for boot modes that don't consume an image (ipxe-exit /
+inventory / tui) rather than showing a stale sha.
+
+**Catalog table is leaner.** The sources listing drops the inline
+content sha + byte size (image facts that live on the linked Image
+page); the fetched/error/in-flight readiness state stays.
+
+**Default timestamp format drops the timezone suffix.** Operator-facing
+timestamps now default to `%Y-%m-%d %H:%M:%S` (24-hour, no ` %Z`); the
+display timezone is still an operator setting, just not repeated on
+every row. Set `PIXIE_DATETIME_FORMAT` or the Settings override to add
+it back.
+
+**Overlays are now globally-named single-writer volumes.** A persistent
+nbdboot overlay is no longer a per-`(machine, image, profile)` triple; it
+is a globally-unique named writable volume (`alias`) over ONE base image,
+and the base image is implied by the alias. At most one machine may hold
+an alias at a time: attaching one already held by a different machine is
+rejected in the app ("held by <mac>; detach first"), with qemu-nbd's
+qcow2 image-lock as the backstop. On a machine's detail page the picker
+now offers the aliases that are free (or already held here) plus a
+create-new flow; attaching an existing alias binds the machine to that
+alias's base image. The Overlays page keys on the alias, shows an
+**Attached to** column (a MAC or "free"), and classifies each row as
+serving / held / free / orphaned / missing; Prune still reclaims only the
+orphaned + file-missing rows and leaves a free alias (a deliberate keep)
+alone. On-disk overlays move from
+`overlays/<mac>/<image_sha>/<profile>.qcow2` to a flat
+`overlays/<alias>.qcow2`. Existing state.db rows migrate in place on
+first start: each old overlay becomes `alias = <profile>-<mac_slug>`
+holding its original MAC, its qcow2 path is kept as-is (no large-file
+move), and each machine's binding is rewritten to the same derived alias.
+
+**Dashboard speaks the new model.** The tiles now mirror the
+storage/lifecycle vocabulary + the nav: **Machines**, **Catalog**
+(sources you can fetch, with a not-yet-fetched count), **Images** (the
+materialised entities, with on-disk footprint + how many are
+reclaimable), **Overlays** (alias-keyed writable volumes, with disk-used
++ serving), and **Events** -- instead of the old catalog-images /
+NBD-exports framing. The Acknowledge button was removed from the
+dashboard (it's an action; the dashboard is output-only -- ack lives on
+the Events page).
+
+**Catalog is sources-only now.** With Images owning the materialised
+view, the Catalog page drops the NBD-serving column and the
+Stop-export / blob-delete actions (those live on Images). It keeps
+source management -- add, import, Fetch, delete-entry -- and a fetched
+disk-image row now links to its **Image** for footprint, live NBD /
+overlay usage, and GC.
+
+**Legacy `/ui/exports` lands on Images.** The kept-alive redirect for
+the old exports URL (and the Stop-export action) now points at the
+Images view, where NBD export usage is surfaced per content sha, rather
+than at Catalog.
+
+**Overlay picker shows aliases held elsewhere, disabled.** The
+machine-detail overlay dropdown used to hide any alias held by another
+machine, so an operator searching for it assumed it didn't exist and
+tried to create a duplicate (which the single-writer bind then
+rejected). Held-elsewhere aliases are now shown as a disabled option
+labelled with the holder MAC ("held by `<mac>`") -- visible, but not
+selectable.
+
+**Overlays distinguish "pending" from "file missing".** A reserved
+overlay alias whose qcow2 hasn't been lazy-created yet (never booted)
+previously read as *file missing* -- the same alarming state as a
+booted-then-lost overlay -- and Prune would reclaim it as junk. Such an
+overlay is now classified **pending** (benign, awaiting its first
+nbdboot) and is left alone by Prune; *missing* is reserved for an
+overlay whose qcow2 vanished after a prior boot (real data loss). The
+`last_boot_at` timestamp is the discriminator.
+
+### Added
+
+**First-contact machines emit a `machine.discovered` audit event.** The
+first time an unknown MAC hits `GET /pxe/<mac>` and is auto-registered,
+pixie now records one `machine.discovered` event (carrying the
+auto-registered boot mode + the client IP); repeat contacts do not
+re-emit. The kind was reserved but never fired before.
+
+**Images: the materialised content behind Catalog sources.** A new
+**Images** page (`/ui/images`) separates the *source* (a Catalog entry,
+a URL you can fetch) from the *entity* it produces once fetched (an
+image, identity = the disk content sha). Machines, NBD exports, and
+overlays all key off that sha, so each image rolls up its on-disk
+footprint (raw disk + rootfs + boot bundle + overlays) and every live
+usage -- machines bound, the ephemeral nbdkit export, and per-machine
+qemu-nbd overlays -- with counts that link into the per-usage admin
+surfaces. An **image detail hub** (`/ui/images/<sha>`) lays out the
+artifacts + every usage inline and offers a **guarded delete / blob GC**:
+allowed only when the usage count (the refcount) is zero, it removes the
+whole `blobs/<sha>/` dir (raw disk **and** the `rootfs.raw` the old
+per-entry delete leaked) and clears the sha off every entry that
+resolved to it. The list also surfaces **orphan blobs** -- sha dirs on
+disk with no catalog entry -- which is where the un-GC'd disk pressure
+actually hides; they delete straight through. Catalog is unchanged for
+now.
+
+### Fixed
+
+**Live-env artifacts are served after a post-startup fetch, no restart
+needed.** The `/boot/pixie-live-env/` mount was created at app startup
+only if the directory already existed, so on a fresh deploy -- where the
+operator stages the netboot-pc bake later via "Fetch live env" -- the
+plan pointed targets at `/boot/pixie-live-env/vmlinuz` and friends that
+then 404'd until pixie was restarted, silently breaking every
+`pixie-inventory` / `-tui` / `-flash` boot. The mount is now created
+unconditionally (dir ensured, `check_dir=False`), so a fetched live-env
+is served immediately.
+
+**`pixie-lab purge --all` no longer tracebacks on a root-owned parent.**
+When the deploy dir lives under a directory the operator can't write
+(the common `/opt/pixie` case), purge could empty the dir but not
+remove the dir itself, and surfaced the failure as a raw Python
+traceback. It now clears the state, prints a plain one-line caveat
+("emptied `<dir>`, but could not remove the directory itself ..."), and
+exits 0 -- the purge did its job; only the empty-dir removal was blocked
+by the parent's permissions.
+
 ## [0.3.2] - 2026-07-24
 
 ### Added

{pixie_lab-0.3.2 → pixie_lab-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pixie-lab
-Version: 0.3.2
+Version: 0.4.0
 Summary: Bare-metal netboot appliance: catalog + fetch + NBD + PXE + TFTP + operator TUI in one container
 Project-URL: Homepage, https://github.com/safl/pixie
 Project-URL: Repository, https://github.com/safl/pixie

{pixie_lab-0.3.2 → pixie_lab-0.4.0}/docs/src/boot-modes.md

@@ -54,29 +54,21 @@ overlay-on-tmpfs: writes go to RAM, nothing propagates back to the
 source blob. Multiple targets can nbdboot the same image
 simultaneously because they each get their own overlay.
 
-**Persistent overlays** flip a single target from ephemeral to dev
-mode without changing anything else about the bind. On the machine
-detail page, the `Overlay profile` field is blank by default
-(ephemeral, unchanged behaviour) or names a per-machine profile
-(e.g. `simon`, `karl`, `ci-with-nvme-tools`). A non-blank profile
-maps to a per-machine qcow2 file with the image's base blob as
-`backing_file`, served by `qemu-nbd` at a dedicated port. The target
-mounts the NBD device read-write; system-level changes (apt-installed
-packages, hardware-specific config, kernel modules) land on the qcow2
-and survive reboots.
-
-Overlays are keyed by `(mac, image_content_sha256, profile)`:
-different machines have fully independent files even under the same
-profile name, and rebinding a machine to a different image leaves the
-old image's overlays on disk (a rebind back resumes them). Storage is
-`data/overlays/<mac>/<image_sha>/<profile>.qcow2`. The **Reset**
-button on the machine detail page tears down `qemu-nbd`, unlinks the
-qcow2, and lets the next boot lazy-create a fresh overlay from the
-base.
-
-Concurrency is by construction: a MAC boots one target at a time, so
-two machines can never contend for the same qcow2. There is no
-holder-tracking or force-reclaim.
+**Persistent overlays** flip a target from ephemeral to dev mode
+without changing anything else about the bind. On the machine detail
+page, the `Overlay alias` field is blank by default (ephemeral,
+unchanged behaviour) or names a persistent overlay volume (e.g.
+`simon`, `karl`, `ci-with-nvme-tools`). A non-blank alias attaches a
+qcow2 volume with the image's base blob as `backing_file`, served by
+`qemu-nbd` at a dedicated port. The target mounts the NBD device
+read-write; system-level changes (apt-installed packages,
+hardware-specific config, kernel modules) land on the qcow2 and
+survive reboots.
+
+An overlay is a **globally-unique named volume over one base image**,
+not a per-machine file. See [](#overlays-are-volumes-not-per-machine-files)
+below for how the alias, its base image, and single-writer access
+work.
 
 **Kexec into a locally-installed kernel.** The netboot bundle owns
 the kernel and initrd pixie serves. Installing `linux-image-*` on the
@@ -93,6 +85,54 @@ workflow. Recovery from a bad kernel is a power-cycle back to
 pixie's kernel; nothing kexec's automatically, so a broken install
 never becomes a boot loop.
 
+## Overlays are volumes, not per-machine files
+
+An overlay is a **globally-unique named writable volume** layered over
+exactly one base image. The alias is the identity, not the machine:
+`data/overlays/<alias>.qcow2`, a single qcow2 whose `backing_file` is
+the image's base blob. There is no `(mac, image, profile)` key and no
+per-machine directory; two machines cannot mint independent files
+under the same name, because the name is the volume.
+
+**The alias implies its image.** Attaching an *existing* alias binds
+the machine to that overlay's base image; you don't re-pick the image
+on the bind form, the volume already knows it. Attaching a *new* alias
+takes the image you selected and lays a fresh qcow2 over it. Because
+the alias carries the image, moving a volume between machines is just
+re-binding the alias; the base image follows.
+
+**Single-writer, enforced in the app.** At most one machine holds an
+alias at a time. Binding an alias already attached to another machine
+is refused ("overlay `<alias>` held by `<mac>`; detach first") on both
+the operator form and the `PUT /machines/{mac}` API, before any qcow2
+is touched. The qemu qcow2 lock is the backstop; the app-level check
+is the operator-facing guard. Rebinding a machine to ephemeral, to a
+different alias, or to a non-`nbdboot` mode releases the hold it had.
+
+**Managing volumes.** The Overlays page (`/ui/overlays`) lists every
+volume with its state: *serving* (a live machine is bound to it and
+`qemu-nbd` is up), *held* (attached to a machine, nothing serving),
+*free* (unattached, kept for a future bind), *orphaned* (attached to a
+MAC with no machine row), or *missing* (the qcow2 is gone). **Reset**
+tears down `qemu-nbd` and unlinks one volume's qcow2 so the next boot
+lazy-creates it fresh from the base; **Prune** reclaims the junk
+states (orphaned + missing) in bulk and deliberately leaves free
+volumes alone.
+
+## Bindings
+
+A binding is what pixie serves a MAC next time it PXEs: a boot mode,
+plus (for the modes that need one) an image, plus (for `nbdboot`) an
+optional overlay alias. A fresh MAC auto-registers as `ipxe-exit` with
+no image on first contact; the machine detail page is where an
+operator promotes it. Saving the bind form persists all three fields
+together, and the plan renderer reads them per PXE request: it resolves
+the image's netboot bundle, resolves the alias to its overlay (and
+base image), ensures the NBD export is up, and renders the plan. A
+binding that can't be satisfied (no image bound, bundle not fetched,
+alias held elsewhere) degrades to an `unavailable` plan whose comment
+names the reason, rather than serving a broken boot.
+
 ## Prerequisites at a glance
 
 | Mode | Needs image | Needs inventory | Needs target disk |

{pixie_lab-0.3.2 → pixie_lab-0.4.0}/docs/src/deployment.md

@@ -69,7 +69,7 @@ unattended). Flags choose how far the wipe goes:
 
 - `--data` - also delete `data/`: `state.db` (catalog, machines,
   events, exports, overlays, settings), every fetched blob and unpacked
-  artifact, all per-machine overlays, and the staged `live-env/`.
+  artifact, all overlay volumes, and the staged `live-env/`.
   `data/` is a bind mount (`./data:/var/lib/pixie`), not a named
   volume, so purge unlinks it directly - `podman compose down -v`
   alone would not touch it.
@@ -137,8 +137,9 @@ workarounds; see [](hardware-quirks.md).
 
 **`PIXIE_DATA_DIR`** (default: `/var/lib/pixie`) - state root. Holds
 `state.db`, `blobs/` (fetched image bytes), `artifacts/` (extracted
-netboot bundles), `overlays/<mac>/<image_sha>/<profile>.qcow2`
-(per-machine writable overlays for nbdboot; see [](boot-modes.md)),
+netboot bundles), `overlays/<alias>.qcow2`
+(globally-named single-writer overlay volumes for nbdboot; see
+[](boot-modes.md)),
 `live-env/` (staged pixie live env).
 
 **`PIXIE_LIVE_ENV_DIR`** (default: `$PIXIE_DATA_DIR/live-env`) - the

pixie_lab-0.4.0/docs/src/quickstart.md

@@ -0,0 +1,97 @@
+# Quickstart
+
+This walks through a first pixie deploy: install, bring the container
+up, point DHCP at it, and boot a first target.
+
+## Run the deploy generator (no install)
+
+The `pixie-lab` commands (`init` / `deploy` / `purge`) are one-off
+admin tasks, so run them ephemerally with `uv` instead of installing
+anything:
+
+```
+uv tool run pixie-lab init
+```
+
+`uv tool run pixie-lab` fetches the distribution into a temporary
+isolated environment, runs the command, and leaves nothing behind: no
+system Python pollution, nothing to uninstall. Every command below is
+invoked the same way. (If you would rather keep it around, `uv tool
+install pixie-lab` installs the `pixie-lab` command persistently.)
+
+The distribution ships two console-scripts: `pixie-lab` (the deploy
+generator you run here) and `pixie` (the operator TUI baked into the
+pixie live env; targets run it after booting the live env, operators
+don't call it on their workstation).
+
+## Generate the deploy bundle
+
+```
+mkdir /opt/pixie && cd /opt/pixie
+uv tool run pixie-lab init
+${EDITOR:-vi} envvars     # set PIXIE_HOST_ADDR + PIXIE_ADMIN_PASSWORD
+```
+
+`uv tool run pixie-lab init` writes `compose.yml`, `envvars` (with placeholder
+values), and `envvars.example` into the current directory. Edit
+`envvars` to point pixie at your LAN address and set a real admin
+password.
+
+## Bring pixie up
+
+```
+podman compose --env-file envvars up -d
+curl http://<PIXIE_HOST_ADDR>:8080/healthz
+```
+
+The compose file runs pixie under `network_mode: host` so udp/69
+(TFTP) plus the NBD port range plus tcp/8080 (HTTP) all reach the
+LAN without publish gymnastics.
+
+## Point DHCP at pixie
+
+Set your DHCP server to chain PXE targets through pixie's TFTP and
+iPXE bootstrap. See [](deployment.md#dhcp-handoff) for BIOS + UEFI
+dnsmasq recipes.
+
+## Add a first catalog entry
+
+Open `http://<PIXIE_HOST_ADDR>:8080/ui/catalog` in a browser, log in
+with your admin password, and add a catalog entry with a source URL.
+Hit Fetch. Pixie pulls the bytes to disk (and for `img.gz` / `img.zst`
+inputs, decompresses on the way in). The row flips to `fetched` when
+the pipeline lands.
+
+## Bind a machine
+
+Once a target has PXEd at least once, it appears on `/ui/machines`.
+Click its MAC to open the machine detail page. Pick a boot mode from
+the card grid, pick an image, and Save. On the target's next PXE,
+pixie serves the plan you bound.
+
+See [](boot-modes.md) for what each mode does and how bindings pin a
+machine to an image and, for `nbdboot`, an overlay volume.
+
+## Tear it down
+
+`uv tool run pixie-lab purge` is the inverse of the deploy. Run it from
+the deploy directory (or pass the path):
+
+```
+uv tool run pixie-lab purge            # stop + remove the container only
+uv tool run pixie-lab purge --data     # also DELETE data/ (state.db, blobs, artifacts, overlays, live-env)
+uv tool run pixie-lab purge --images   # also remove the pixie container image
+uv tool run pixie-lab purge --all      # container + image + data/ + the deploy directory itself
+```
+
+Purge prints its plan and gates every variant behind a `y/N`
+confirmation; pass `--yes` to skip the prompt for scripted teardown.
+Even the plain stop is impactful: it drops every nbdkit / qemu-nbd
+export, so any target currently booted `nbdboot` off pixie loses its
+NBD root at that moment. The `--data` and `--all` deletions are
+irreversible, so the confirmation is deliberate.
+
+The single-shot `uv tool run pixie-lab deploy` is the mirror image of purge: it
+runs `init`, auto-fills `envvars`, brings the stack up, and waits on
+`/healthz` in one command, for when you don't need to hand-edit the
+generated files first.

{pixie_lab-0.3.2 → pixie_lab-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pixie-lab"
-version = "0.3.2"
+version = "0.4.0"
 description = "Bare-metal netboot appliance: catalog + fetch + NBD + PXE + TFTP + operator TUI in one container"
 readme = "README.md"
 requires-python = ">=3.11"

{pixie_lab-0.3.2 → pixie_lab-0.4.0}/src/pixie/deploy/_main.py

@@ -311,10 +311,32 @@ def _cmd_purge(args: argparse.Namespace) -> int:
         shutil.rmtree(data_dir)
         print(f"pixie-lab purge: removed {data_dir}", file=sys.stderr)
 
-    # 4. Delete the deploy directory itself.
+    # 4. Delete the deploy directory itself. ``dest`` often sits under a
+    #    root-owned parent (e.g. /opt), where an unprivileged operator can
+    #    empty the dir but can't rmdir it -- rmtree clears the contents,
+    #    then raises on the final rmdir. Report that plainly instead of
+    #    dumping a traceback; the state is gone either way, so it's a
+    #    caveat, not a failure.
     if remove_dir and dest.exists():
-        shutil.rmtree(dest)
-        print(f"pixie-lab purge: removed {dest}", file=sys.stderr)
+        try:
+            shutil.rmtree(dest)
+            print(f"pixie-lab purge: removed {dest}", file=sys.stderr)
+        except OSError as exc:
+            leftover = sorted(p.name for p in dest.iterdir()) if dest.is_dir() else []
+            if not leftover:
+                print(
+                    f"pixie-lab purge: emptied {dest}, but could not remove the "
+                    f"directory itself ({exc.strerror or exc}) -- its parent is not "
+                    f"writable (a root-owned /opt, say). The empty dir is harmless; "
+                    f"you can redeploy into it or 'sudo rmdir' it.",
+                    file=sys.stderr,
+                )
+            else:
+                print(
+                    f"pixie-lab purge: could not fully remove {dest} "
+                    f"({exc.strerror or exc}); leftover: {', '.join(leftover)}",
+                    file=sys.stderr,
+                )
 
     return 0
 

{pixie_lab-0.3.2 → pixie_lab-0.4.0}/src/pixie/events/_kinds.py

@@ -21,7 +21,9 @@ from __future__ import annotations
 # ---------- catalog + fetch pipeline ---------------------------------
 #
 # Every catalog-store mutation emits one of these. ``subject_kind`` is
-# always ``"entry"``; ``subject_id`` is the catalog entry name.
+# ``"entry"`` (``subject_id`` = the catalog entry name) for the per-row
+# kinds; the ``catalog.import.*`` pair is catalog-scoped instead
+# (``subject_kind = "catalog"``) since one import touches many entries.
 
 CATALOG_ENTRY_ADDED = "catalog.entry.added"
 """A new row landed via POST /catalog/entries, POST /ui/catalog/add,
@@ -95,36 +97,37 @@ EXPORT_NBDKIT_EXITED = "export.nbdkit.exited"
 (``_refresh_row`` noticed no live proc for a ``running`` row).
 ``details.previous_port`` + ``details.error`` explain."""
 
-# ---------- persistent overlays (per-machine qcow2) -----------------
+# ---------- persistent overlays (alias-keyed volumes) ---------------
 #
-# ``subject_kind`` is always ``"machine"``; ``subject_id`` is the
-# canonical MAC. ``details`` names the (image_sha, profile) triple.
+# ``subject_kind`` is always ``"overlay"``; ``subject_id`` is the
+# globally-unique ``alias`` (``""`` for a fleet-wide bulk action). An
+# overlay is a single-writer qcow2 volume over one base image, not a
+# per-machine file; ``details`` carries ``alias`` / ``image_sha`` /
+# ``qcow2_path`` as relevant.
 
 OVERLAY_CREATED = "overlay.created"
-"""A per-machine qcow2 overlay was materialised for the first time
-for a ``(mac, image_sha, profile)`` triple. ``details`` carries
-``image_sha``, ``profile``, and ``qcow2_path``."""
+"""An alias-keyed qcow2 overlay was materialised for the first time
+(lazy-created on the first nbdboot render, or after a Reset).
+``details`` carries ``alias``, ``image_sha``, and ``qcow2_path``."""
 
 OVERLAY_RESET = "overlay.reset"
-"""Operator hit Reset on the machine detail page: the qemu-nbd was
-terminated, the qcow2 was unlinked, the overlays row was deleted.
+"""Operator hit Reset (one alias) or Prune (bulk) on the Overlays
+page: the qemu-nbd was terminated, the qcow2 unlinked, the overlays
+row deleted. ``subject_id`` is the alias for a single Reset, ``""``
+for a bulk Prune (``details.pruned`` counts the reclaimed volumes).
 Next plan render lazily creates a fresh overlay from the base."""
 
-OVERLAY_BOOTED = "overlay.booted"
-"""Plan render for an already-existing overlay row (heartbeat).
-Emitted on every persistent nbdboot render so an operator can grep
-the events log for "which overlays am I actually using?"."""
-
 # ---------- machines + PXE ------------------------------------------
 #
 # ``subject_kind`` is always ``"machine"``; ``subject_id`` is the
 # canonical MAC address.
 
 MACHINE_DISCOVERED = "machine.discovered"
-"""First-ever GET /pxe/<mac> for a MAC pixie has not seen before.
-Emitted from ``machines_store.touch_seen`` when a new row is
-created; subsequent hits do NOT emit (they update ``last_seen_at``
-via the ``pxe.plan.rendered`` event instead)."""
+"""First-ever GET /pxe/<mac> for a MAC pixie has not seen before. The
+``/pxe/<mac>`` route detects the absent row before ``touch_seen``
+auto-registers it and emits this once; subsequent hits do NOT emit
+(they update ``last_seen_at`` via the ``pxe.plan.rendered`` event
+instead). ``details.boot_mode`` is the auto-registered default."""
 
 MACHINE_BOUND = "machine.bound"
 """First bind for a MAC: PUT /machines/<mac> or POST
@@ -232,7 +235,6 @@ KNOWN_EVENT_KINDS: frozenset[str] = frozenset(
         EXPORT_NBDKIT_EXITED,
         OVERLAY_CREATED,
         OVERLAY_RESET,
-        OVERLAY_BOOTED,
         MACHINE_DISCOVERED,
         MACHINE_BOUND,
         MACHINE_BINDING_CHANGED,