pixie-lab 0.3.1__tar.gz → 0.4.0__tar.gz

{pixie_lab-0.3.1 → pixie_lab-0.4.0}/.github/workflows/cicd.yaml

@@ -10,6 +10,11 @@ name: cicd
 #   * ``build-media``: build the live-env media (netboot-pc + usbboot-pc
 #     variants) via the cijoe pipelines under cijoe/tasks/. Requires
 #     live-build + passwordless sudo; the runner has both.
+#   * ``verify-usbboot``: structurally verify the usbboot-pc .iso
+#     (hybrid ISO + El Torito BIOS & UEFI entries + sha256), then boot
+#     it via a real Ventoy stick in QEMU and assert the pixie live env
+#     comes up. Gates the publish jobs so a broken / non-Ventoy .iso
+#     never ships.
 #   * ``build-wheel``: sdist + wheel for PyPI. Runs on every push +
 #     PR so the packaging story is exercised regardless of tag state.
 #   * ``publish-pypi``: trusted publishing of the wheel + sdist,
@@ -556,6 +561,66 @@ jobs:
           path: cijoe/_build/test-pxe
           if-no-files-found: warn
 
+  verify-usbboot:
+    name: Verify usbboot .iso (structural + Ventoy boot in QEMU)
+    # Downloads the usbboot-pc bake, structurally verifies the hybrid
+    # ISO (ISO 9660 + isohybrid MBR + El Torito BIOS & UEFI entries +
+    # sha256), then boots it via a real Ventoy stick in QEMU and asserts
+    # the pixie live env comes up: pixie-usb-grow skips (no PIXIE_IMGS
+    # under a loop-mounted ISO), pixie-images-discover bind-mounts the
+    # operator drop, pixie-on-tty1 renders the CLI wizard. Gates the
+    # release -- a broken / non-bootable / non-Ventoy-compatible ISO
+    # must not ship.
+    needs: build-media
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install host tools (qemu, ovmf, exfat, xorriso)
+        run: |
+          set -eu
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            qemu-system-x86 \
+            qemu-utils \
+            ovmf \
+            exfat-fuse \
+            exfatprogs \
+            xorriso \
+            file
+          sudo chmod a+rw /dev/kvm
+          ls -la /dev/kvm
+
+      - name: Install cijoe
+        run: pipx install --force cijoe
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v8.1.0
+
+      - name: Download usbboot-pc artifact
+        uses: actions/download-artifact@v7
+        with:
+          name: pixie-usbboot-pc-x86_64
+          path: ~/system_imaging/disk/
+
+      - name: Show downloaded artifact contents
+        run: |
+          set -eu
+          echo "--- ~/system_imaging/disk (recursive) ---"
+          find ~/system_imaging/disk -maxdepth 5 -print
+
+      - name: Verify + Ventoy-boot the usbboot .iso
+        run: make test-usb-ventoy
+
+      - name: Upload cijoe output
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: cijoe-output-test-usb-ventoy
+          path: cijoe/cijoe-output
+          if-no-files-found: warn
+
   build-wheel:
     name: Build wheel + sdist
     needs: test
@@ -591,6 +656,7 @@ jobs:
       - test-pxe-tui
       - test-pxe-flash
       - test-pxe-flash-always
+      - verify-usbboot
     runs-on: ubuntu-latest
     if: startsWith(github.ref, 'refs/tags/v')
     environment:
@@ -632,6 +698,7 @@ jobs:
       - test-pxe-tui
       - test-pxe-flash
       - test-pxe-flash-always
+      - verify-usbboot
     runs-on: ubuntu-latest
     if: startsWith(github.ref, 'refs/tags/v')
     permissions:
@@ -676,6 +743,7 @@ jobs:
       - test-pxe-tui
       - test-pxe-flash
       - test-pxe-flash-always
+      - verify-usbboot
     runs-on: ubuntu-latest
     if: startsWith(github.ref, 'refs/tags/v')
     permissions:

{pixie_lab-0.3.1 → pixie_lab-0.4.0}/CHANGELOG.md

@@ -11,6 +11,148 @@ operator-facing summary.
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-07-24
+
+### Changed
+
+**Machines list Image column reads like the new model.** The bound
+image now shows the catalog name linked to its Images page (short sha
+for an orphan blob whose entry was deleted), instead of a bare hash,
+and is blank for boot modes that don't consume an image (ipxe-exit /
+inventory / tui) rather than showing a stale sha.
+
+**Catalog table is leaner.** The sources listing drops the inline
+content sha + byte size (image facts that live on the linked Image
+page); the fetched/error/in-flight readiness state stays.
+
+**Default timestamp format drops the timezone suffix.** Operator-facing
+timestamps now default to `%Y-%m-%d %H:%M:%S` (24-hour, no ` %Z`); the
+display timezone is still an operator setting, just not repeated on
+every row. Set `PIXIE_DATETIME_FORMAT` or the Settings override to add
+it back.
+
+**Overlays are now globally-named single-writer volumes.** A persistent
+nbdboot overlay is no longer a per-`(machine, image, profile)` triple; it
+is a globally-unique named writable volume (`alias`) over ONE base image,
+and the base image is implied by the alias. At most one machine may hold
+an alias at a time: attaching one already held by a different machine is
+rejected in the app ("held by <mac>; detach first"), with qemu-nbd's
+qcow2 image-lock as the backstop. On a machine's detail page the picker
+now offers the aliases that are free (or already held here) plus a
+create-new flow; attaching an existing alias binds the machine to that
+alias's base image. The Overlays page keys on the alias, shows an
+**Attached to** column (a MAC or "free"), and classifies each row as
+serving / held / free / orphaned / missing; Prune still reclaims only the
+orphaned + file-missing rows and leaves a free alias (a deliberate keep)
+alone. On-disk overlays move from
+`overlays/<mac>/<image_sha>/<profile>.qcow2` to a flat
+`overlays/<alias>.qcow2`. Existing state.db rows migrate in place on
+first start: each old overlay becomes `alias = <profile>-<mac_slug>`
+holding its original MAC, its qcow2 path is kept as-is (no large-file
+move), and each machine's binding is rewritten to the same derived alias.
+
+**Dashboard speaks the new model.** The tiles now mirror the
+storage/lifecycle vocabulary + the nav: **Machines**, **Catalog**
+(sources you can fetch, with a not-yet-fetched count), **Images** (the
+materialised entities, with on-disk footprint + how many are
+reclaimable), **Overlays** (alias-keyed writable volumes, with disk-used
++ serving), and **Events** -- instead of the old catalog-images /
+NBD-exports framing. The Acknowledge button was removed from the
+dashboard (it's an action; the dashboard is output-only -- ack lives on
+the Events page).
+
+**Catalog is sources-only now.** With Images owning the materialised
+view, the Catalog page drops the NBD-serving column and the
+Stop-export / blob-delete actions (those live on Images). It keeps
+source management -- add, import, Fetch, delete-entry -- and a fetched
+disk-image row now links to its **Image** for footprint, live NBD /
+overlay usage, and GC.
+
+**Legacy `/ui/exports` lands on Images.** The kept-alive redirect for
+the old exports URL (and the Stop-export action) now points at the
+Images view, where NBD export usage is surfaced per content sha, rather
+than at Catalog.
+
+**Overlay picker shows aliases held elsewhere, disabled.** The
+machine-detail overlay dropdown used to hide any alias held by another
+machine, so an operator searching for it assumed it didn't exist and
+tried to create a duplicate (which the single-writer bind then
+rejected). Held-elsewhere aliases are now shown as a disabled option
+labelled with the holder MAC ("held by `<mac>`") -- visible, but not
+selectable.
+
+**Overlays distinguish "pending" from "file missing".** A reserved
+overlay alias whose qcow2 hasn't been lazy-created yet (never booted)
+previously read as *file missing* -- the same alarming state as a
+booted-then-lost overlay -- and Prune would reclaim it as junk. Such an
+overlay is now classified **pending** (benign, awaiting its first
+nbdboot) and is left alone by Prune; *missing* is reserved for an
+overlay whose qcow2 vanished after a prior boot (real data loss). The
+`last_boot_at` timestamp is the discriminator.
+
+### Added
+
+**First-contact machines emit a `machine.discovered` audit event.** The
+first time an unknown MAC hits `GET /pxe/<mac>` and is auto-registered,
+pixie now records one `machine.discovered` event (carrying the
+auto-registered boot mode + the client IP); repeat contacts do not
+re-emit. The kind was reserved but never fired before.
+
+**Images: the materialised content behind Catalog sources.** A new
+**Images** page (`/ui/images`) separates the *source* (a Catalog entry,
+a URL you can fetch) from the *entity* it produces once fetched (an
+image, identity = the disk content sha). Machines, NBD exports, and
+overlays all key off that sha, so each image rolls up its on-disk
+footprint (raw disk + rootfs + boot bundle + overlays) and every live
+usage -- machines bound, the ephemeral nbdkit export, and per-machine
+qemu-nbd overlays -- with counts that link into the per-usage admin
+surfaces. An **image detail hub** (`/ui/images/<sha>`) lays out the
+artifacts + every usage inline and offers a **guarded delete / blob GC**:
+allowed only when the usage count (the refcount) is zero, it removes the
+whole `blobs/<sha>/` dir (raw disk **and** the `rootfs.raw` the old
+per-entry delete leaked) and clears the sha off every entry that
+resolved to it. The list also surfaces **orphan blobs** -- sha dirs on
+disk with no catalog entry -- which is where the un-GC'd disk pressure
+actually hides; they delete straight through. Catalog is unchanged for
+now.
+
+### Fixed
+
+**Live-env artifacts are served after a post-startup fetch, no restart
+needed.** The `/boot/pixie-live-env/` mount was created at app startup
+only if the directory already existed, so on a fresh deploy -- where the
+operator stages the netboot-pc bake later via "Fetch live env" -- the
+plan pointed targets at `/boot/pixie-live-env/vmlinuz` and friends that
+then 404'd until pixie was restarted, silently breaking every
+`pixie-inventory` / `-tui` / `-flash` boot. The mount is now created
+unconditionally (dir ensured, `check_dir=False`), so a fetched live-env
+is served immediately.
+
+**`pixie-lab purge --all` no longer tracebacks on a root-owned parent.**
+When the deploy dir lives under a directory the operator can't write
+(the common `/opt/pixie` case), purge could empty the dir but not
+remove the dir itself, and surfaced the failure as a raw Python
+traceback. It now clears the state, prints a plain one-line caveat
+("emptied `<dir>`, but could not remove the directory itself ..."), and
+exits 0 -- the purge did its job; only the empty-dir removal was blocked
+by the parent's permissions.
+
+## [0.3.2] - 2026-07-24
+
+### Added
+
+**CI verifies the usbboot `.iso` boots via Ventoy.** A new
+`verify-usbboot` job structurally checks the bake (valid ISO 9660 +
+isohybrid MBR + El Torito BIOS *and* UEFI boot entries + sha256), then
+installs Ventoy onto a QEMU disk, drops the `.iso` + a sentinel image
+catalog onto it, and boots it -- asserting the pixie live env comes up
+correctly on the Ventoy loop-boot path (`pixie-usb-grow` skips with no
+`PIXIE_IMGS` partition, `pixie-images-discover` bind-mounts the
+operator drop at `/var/lib/pixie/images`, and `pixie-on-tty1` renders
+the CLI wizard). The job gates the publish jobs, so a broken,
+non-bootable, or non-Ventoy-compatible ISO can never ship. Ported from
+bty's `test-usb-ventoy`.
+
 ## [0.3.1] - 2026-07-24
 
 ### Fixed

{pixie_lab-0.3.1 → pixie_lab-0.4.0}/Makefile

@@ -23,7 +23,7 @@ endif
 
 .PHONY: help \
         deps test lint format format-check typecheck ci wheel \
-        media-deps build ipxe test-pxe \
+        media-deps build ipxe test-pxe test-usb-ventoy \
         clean
 
 help:
@@ -56,6 +56,8 @@ help:
 	@echo "                  (same deps as test-pxe-flash; asserts no /done flip)"
 	@echo "  test-pxe-tui  end-to-end PXE pixie-tui chain test"
 	@echo "                  (same deps as test-pxe-flash; asserts wizard entry)"
+	@echo "  test-usb-ventoy  structural + Ventoy-boot verify of the usbboot .iso"
+	@echo "                  (needs a prior VARIANT=usbboot-pc bake + qemu/KVM/OVMF + ventoy deps)"
 	@echo ""
 	@echo "Variant: $(VARIANT)  (override with VARIANT=netboot-pc, ...)"
 	@echo "  usbboot-pc    - bootable USB live ISO via live-build (.iso, x86_64)"
@@ -166,6 +168,13 @@ test-pxe-flash-always:
 test-pxe-tui:
 	cd cijoe && cijoe tasks/test-pxe-tui.yaml --monitor -c configs/test-pxe-tui.toml
 
+# Structural + Ventoy-boot verification of the usbboot .iso. Needs a
+# prior VARIANT=usbboot-pc bake staged under ~/system_imaging/disk/ (or
+# a downloaded pixie-usbboot-pc-x86_64 CI artifact) + qemu/KVM/OVMF +
+# ventoy tooling deps (losetup, exfat-fuse, passwordless sudo).
+test-usb-ventoy:
+	cd cijoe && cijoe tasks/test-usb-ventoy.yaml --monitor -c configs/test-usb-ventoy.toml
+
 # ---------- Cleanup ------------------------------------------------------
 
 clean:

{pixie_lab-0.3.1 → pixie_lab-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pixie-lab
-Version: 0.3.1
+Version: 0.4.0
 Summary: Bare-metal netboot appliance: catalog + fetch + NBD + PXE + TFTP + operator TUI in one container
 Project-URL: Homepage, https://github.com/safl/pixie
 Project-URL: Repository, https://github.com/safl/pixie

pixie_lab-0.4.0/cijoe/configs/test-usb-ventoy.toml

@@ -0,0 +1,48 @@
+# pixie USB Ventoy-boot test - cijoe config
+#
+# Verifies an operator's realistic Ventoy workflow:
+#   1. Install Ventoy onto a USB stick.
+#   2. Drop pixie-usbboot-pc-x86_64-v*.iso + image files + a catalog.toml
+#      onto the Ventoy data partition (under pixie-images/).
+#   3. Boot the stick; Ventoy menu auto-selects the only ISO + chains
+#      into the pixie live env.
+#   4. ``pixie-usb-grow.service`` skips correctly (no PIXIE_IMGS
+#      partition on the booted device -- the .iso is loop-mounted, not
+#      dd'd).
+#   5. ``pixie-images-discover.service`` finds the operator's drop on the
+#      Ventoy data partition and bind-mounts ``pixie-images/`` at
+#      ``/var/lib/pixie/images``.
+#   6. ``pixie-on-tty1.service`` renders the pixie CLI wizard on tty1.
+
+[cijoe.task]
+fail_fast = true
+
+[qemu]
+img_bin = "qemu-img"
+
+[qemu.systems.x86_64]
+bin = "qemu-system-x86_64"
+
+[qemu.guests.usb-ventoy]
+path = "{{ local.env.HOME }}/system_imaging/usb-ventoy"
+system_label = "x86_64"
+system_args.kwa = { cpu = "host", smp = 2, m = "1G", accel = "kvm" }
+# IDE-hd: Ventoy's bootloader (a chainloaded GRUB) reads via BIOS
+# INT13 -- virtio-blk is invisible to it. Present the Ventoy stick as
+# an IDE hard disk so the BIOS sees a bootable disk. ``-no-reboot`` so
+# a failed boot exits instead of looping.
+system_args.raw = "-drive file={{ local.env.HOME }}/system_imaging/usb-ventoy/disk.img,format=raw,if=ide -no-reboot"
+system_args.tcp_forward = { host = 12223, guest = 22 }
+
+[cijoe.transport.usb_ventoy_guest]
+username = "root"
+password = "pixie"
+hostname = "localhost"
+port = 12223
+
+[test.usb_ventoy]
+iso_dir = "{{ local.env.HOME }}/system_imaging/disk"
+# Pinned Ventoy version so the test is reproducible. Bump deliberately
+# when validating a newer release; this is the Ventoy2Disk.sh version
+# the test installs with, not what an operator must use in production.
+ventoy_version = "1.1.05"

pixie_lab-0.4.0/cijoe/scripts/usb_iso_verify.py

@@ -0,0 +1,92 @@
+"""
+Structurally verify the pixie usbboot .iso
+===========================================
+
+Fast, no-boot sanity checks on ``pixie-usbboot-pc-x86_64-v*.iso`` before
+the Ventoy boot test spends ~10 min proving it actually boots. Catches a
+broken / non-hybrid / single-firmware bake immediately:
+
+  1. ``file`` reports an ISO 9660 filesystem with a DOS/MBR boot sector
+     (the isohybrid MBR that makes ``dd`` to a USB stick BIOS-bootable).
+  2. The El Torito catalog carries BOTH a BIOS and a UEFI/EFI boot
+     image -- i.e. the ISO boots on legacy BIOS AND UEFI firmware.
+  3. The published ``.iso.sha256`` sidecar matches the image.
+
+Retargetable: False (host-side; needs ``file`` + ``xorriso`` +
+``sha256sum`` on the initiator, which the workflow installs).
+"""
+
+from __future__ import annotations
+
+import errno
+import logging as log
+from argparse import ArgumentParser
+from pathlib import Path
+
+ISO_BASENAME_GLOB = "pixie-usbboot-pc-x86_64-v*.iso"
+
+
+def add_args(parser: ArgumentParser):
+    del parser
+
+
+def _out(res) -> str:
+    return res.output() if hasattr(res, "output") else str(res)
+
+
+def main(args, cijoe):
+    del args
+    cfg = cijoe.getconf("test.usb_ventoy", {})
+    iso_dir = Path(cfg.get("iso_dir") or (Path.home() / "system_imaging" / "disk"))
+    candidates = sorted(iso_dir.glob(ISO_BASENAME_GLOB))
+    if not candidates:
+        log.error(f"no {ISO_BASENAME_GLOB} found in {iso_dir} (did usbboot-pc build run?)")
+        return errno.ENOENT
+    iso = candidates[-1]
+    log.info(f"verifying {iso.name} ({iso.stat().st_size} bytes)")
+
+    # 1. Valid ISO 9660 + isohybrid MBR (BIOS-bootable when dd'd).
+    err, res = cijoe.run_local(f"file -- {iso}")
+    if err:
+        log.error("`file` failed")
+        return errno.EIO
+    file_out = _out(res)
+    if "ISO 9660" not in file_out:
+        log.error(f"not an ISO 9660 image: {file_out}")
+        return errno.EINVAL
+    if "DOS/MBR boot sector" not in file_out and "bootable" not in file_out.lower():
+        log.error(f"no isohybrid MBR boot sector (not USB-dd bootable?): {file_out}")
+        return errno.EINVAL
+    log.info("OK: ISO 9660 + DOS/MBR boot sector")
+
+    # 2. El Torito catalog has both a BIOS and a UEFI/EFI boot image.
+    err, res = cijoe.run_local(f"xorriso -indev {iso} -report_el_torito plain 2>&1")
+    if err:
+        log.error("`xorriso -report_el_torito` failed")
+        return errno.EIO
+    et = _out(res)
+    log.info(f"El Torito report:\n{et}")
+    boot_lines = [ln for ln in et.splitlines() if "boot img" in ln.lower()]
+    has_bios = any("BIOS" in ln for ln in boot_lines)
+    has_uefi = any(("UEFI" in ln) or ("EFI" in ln) for ln in boot_lines)
+    if not has_bios:
+        log.error("El Torito catalog has no BIOS boot image")
+        return errno.EINVAL
+    if not has_uefi:
+        log.error("El Torito catalog has no UEFI/EFI boot image")
+        return errno.EINVAL
+    log.info("OK: El Torito has both BIOS and UEFI boot images")
+
+    # 3. sha256 sidecar matches.
+    sha = iso.with_name(iso.name + ".sha256")
+    if not sha.is_file():
+        log.error(f"missing sha256 sidecar: {sha}")
+        return errno.ENOENT
+    err, _ = cijoe.run_local(f"sh -c 'cd {iso_dir} && sha256sum -c {sha.name}'")
+    if err:
+        log.error("sha256 mismatch")
+        return errno.EIO
+    log.info("OK: sha256 sidecar matches")
+
+    log.info(f"{iso.name}: structural verification passed")
+    return 0

pixie_lab-0.4.0/cijoe/scripts/usb_ventoy_guest_start.py

@@ -0,0 +1,48 @@
+"""
+Start the Ventoy QEMU guest WITHOUT the is_up wait
+==================================================
+
+Thin wrapper around ``cijoe.qemu.wrapper.Guest.start()`` that returns
+immediately after the guest is daemonised, skipping the upstream
+``qemu.guest_start`` step's hardcoded 180-second ``is_up`` check
+(which waits for the literal string ``login:`` on the serial console).
+
+Why a local copy: Ventoy adds a chainload phase (menu -> grub ->
+.iso -> initrd -> squashfs) on top of the pixie live env's normal
+boot, which routinely pushes the time-to-login past the 180-second
+ceiling baked into the upstream script. The YAML's
+``core.wait_for_transport`` step already polls SSH readiness with
+its own (configurable) timeout, so this script intentionally splits
+"start the guest" from "wait for it to be reachable" -- the latter
+is the only concern that needs a tunable timeout.
+
+Retargetable: False (host-side; same constraints as the upstream
+script).
+"""
+
+from __future__ import annotations
+
+import logging as log
+from argparse import ArgumentParser
+
+from cijoe.qemu.wrapper import Guest
+
+
+def add_args(parser: ArgumentParser):
+    parser.add_argument("--guest_name", type=str, help="Name of the qemu guest.")
+
+
+def main(args, cijoe):
+    if "guest_name" not in args:
+        log.error("missing argument: guest_name")
+        return 1
+
+    guest = Guest(cijoe, cijoe.config, args.guest_name)
+
+    err = guest.start()
+    if err:
+        log.error(f"guest.start() : err({err})")
+        return err
+
+    log.info(f"guest '{args.guest_name}' started; readiness handled by wait_for_transport")
+    return 0