vagrant-xapi 0.0.1 → 1.0.0

data/docs/operations.md

@@ -0,0 +1,531 @@
+# Operations
+
+How to run vagrant-xapi against a pool, keep the pool clean, find out what
+went wrong, and work on the plugin itself. For the design behind it see
+[architecture.md](architecture.md); for the provider options see the
+[README](../README.md).
+
+## 1. What is needed
+
+On the local machine:
+
+| Tool | Needed for |
+| --- | --- |
+| Vagrant 2.4.x | everything |
+| `ssh` | `vagrant ssh`, port forwarding tunnels, `ssh_via_host` |
+| `rsync` | rsync synced folders |
+| `qemu-img` | qcow2 boxes on a host that cannot import qcow2, and qcow2 images with compressed clusters (most cloud images and many libvirt boxes), which XAPI cannot import directly (`brew install qemu`, `apt install qemu-utils`) |
+| `mkisofs`/`genisoimage` (Linux) or `hdiutil` (macOS) | cloud-init; Vagrant's `create_iso` host capability uses them |
+| a VNC viewer | `vagrant xapi console`; macOS Screen Sharing works (it gets the one-time password in the URL) |
+
+On the pool:
+
+- XenServer 8 or XCP-ng 8.3.
+- A default SR, or `sr` set in the Vagrantfile.
+- The template `Other install media` (present on every standard install);
+  needed to import disk-image boxes.
+- For `snapshot_memory` and `suspend`: a suspend SR and PV drivers or a guest
+  agent in the guest.
+- For `vtpm`: a version with the `VTPM` class (XAPI is checked at login).
+
+## 2. Connecting
+
+Put the credentials in the environment, not in a Vagrantfile in git:
+
+```sh
+export VAGRANT_XAPI_HOST=xcp-master.example.com
+export VAGRANT_XAPI_USERNAME=vagrant-svc
+export VAGRANT_XAPI_PASSWORD='...'
+# optional: the network of the first VIF when a Vagrantfile has no
+# public_network (name, UUID or bridge such as xenbr1)
+export VAGRANT_XAPI_DEFAULT_NETWORK=xenbr1
+# optional: TLS for self-signed certificates (the pins of all hosts,
+# separated by commas) or your own CA, instead of tls_fingerprint or
+# tls_ca_file in the Vagrantfile
+export VAGRANT_XAPI_TLS_FINGERPRINT=AB:CD:...,12:34:...
+export VAGRANT_XAPI_TLS_CA_FILE=~/certs/pool-ca.pem
+```
+
+Vagrantfiles written for vagrant-xenserver (`config.vm.provider
+"xenserver"` with `xs_*` options) keep working unchanged; see "Migrating
+from vagrant-xenserver" in the README and section 17 of architecture.md.
+Only the stand-in for the `xenapi` gem, for triggers that call XAPI
+themselves, is temporary.
+
+`host` may be any host of the pool; a pool member redirects to the master.
+With fingerprint pinning that needs the pins of all hosts (see below).
+
+### TLS
+
+The certificate is always verified. Three ways to make that work:
+
+1. **A certificate from a CA the local machine trusts**: nothing to configure.
+2. **Your own CA**: `xapi.tls_ca_file = "~/certs/pool-ca.pem"`.
+3. **Self-signed** (the default after installing XCP-ng/XenServer): pin the
+   SHA-256 fingerprint of the host certificate.
+
+```sh
+openssl s_client -connect xcp-master.example.com:443 </dev/null 2>/dev/null \
+  | openssl x509 -noout -fingerprint -sha256
+# sha256 Fingerprint=AB:CD:...
+```
+
+Without a pin, the error for a self-signed certificate shows the same
+fingerprint of the certificate it got. Before you pin it, check it on the
+host itself, where nothing can sit in between:
+
+```sh
+# on the host (dom0)
+openssl x509 -noout -fingerprint -sha256 -in /etc/xensource/xapi-ssl.pem
+```
+
+```ruby
+config.vm.provider :xapi do |xapi|
+  xapi.tls_fingerprint = "AB:CD:..."   # with or without colons
+end
+```
+
+Every pool member has its own certificate. One pin only matches one host, so
+either pin the master and use it as `host` (with a single pin on a member,
+the plugin stops and names the master), or pin the certificates of all
+hosts. Then any host can be `host`, and a change of master (HA, or
+`xe pool-designate-new-master`) does not break the configuration:
+
+```sh
+vagrant xapi hosts      # once connected to the master: every host's fingerprint
+```
+
+```ruby
+config.vm.provider :xapi do |xapi|
+  xapi.tls_fingerprint = ["AB:CD:...", "12:34:..."]   # or one string with commas
+end
+```
+
+ Uploads, downloads and consoles that run on another
+member use the certificate the master reports for that member
+(`host.get_server_certificate`), so they need no extra configuration. After
+a certificate renewal (e.g. `xe host-refresh-server-certificate`) the pin
+must be updated; the error says "certificate fingerprint ... does not
+match".
+
+`vagrant xapi hosts --pem` prints the certificates themselves, e.g. for a
+`tls_ca_file`. With a CA file the hostname is checked, so the address used
+must be in the certificate: the member's, and the master's as XAPI reports
+it on the redirect (its IP address). The self-signed certificates of
+XCP-ng and XenServer do not contain those IP addresses; pin them instead.
+
+### Which user and role
+
+With `pool-admin` everything works. The plugin needs to:
+
+- create, change, start, stop and destroy VMs, VDIs, VBDs, VIFs, snapshots;
+- import and export (`/import`, `/import_raw_vdi`, `/export`);
+- write `pool.other_config` (locks);
+- create and destroy networks (internal networks for `private_network`).
+
+`vm-admin` covers the VM part, but may not cover the last two. Then `vagrant
+up` stops with "The XAPI user may not write the pool's other_config ...".
+
+## 3. Boxes
+
+### Kinds
+
+| Box provider | Use |
+| --- | --- |
+| `xapi` | Our own format: `metadata.json` (`provider`, `format`, `architecture`) and `box.<format>` (xva, vhd, qcow2 or raw) |
+| `xenserver` | Boxes of vagrant-xenserver: `box.xva`, `box.vhd`, or a Vagrantfile with `xva_url` |
+| `libvirt` | qcow2 boxes from the libvirt ecosystem (e.g. `generic/*`); they need cloud-init and a static IP when they have no Xen guest agent |
+
+Vagrant chooses between providers of the same box version in this order:
+`xapi`, `xenserver`, `libvirt`.
+
+Always set `config.vm.box_architecture = "amd64"`: XenServer and XCP-ng only
+run amd64 guests, and on a Mac with Apple silicon Vagrant otherwise looks for
+an arm64 box. For the same reason, add a box *file* with
+`vagrant box add --architecture amd64 ...`: Vagrant stores a local box file
+under the local machine's architecture (it ignores `architecture` in its
+`metadata.json`), and a Vagrantfile asking for amd64 then does not find it
+and tries to download it instead ("could not be found in the remote
+catalog"). Boxes from a registry carry their architecture and need nothing
+extra. To remove a box added under the wrong architecture, use
+`vagrant box remove <name> --all-architectures --force` or delete
+`~/.vagrant.d/boxes/<name with / as -VAGRANTSLASH->/<version>/arm64`;
+`vagrant box remove --architecture` does not match any box (Vagrant
+compares the option as a string with symbols).
+
+### Building a box
+
+The recommended route is
+[packer-plugin-xenserver](https://github.com/vatesfr/packer-plugin-xenserver),
+producing an XVA with a guest agent (xe-guest-utilities or xen-guest-agent)
+and cloud-init installed. Then:
+
+```sh
+cat > metadata.json <<'EOF'
+{"provider": "xapi", "format": "xva", "architecture": "amd64"}
+EOF
+cat > Vagrantfile <<'EOF'
+Vagrant.configure("2") do |config|
+  config.vm.provider :xapi do |xapi|
+    xapi.firmware = :uefi      # whatever the image was built with
+  end
+end
+EOF
+cp output/debian-12.xva box.xva
+tar czf debian-12.box metadata.json Vagrantfile box.xva
+vagrant box add --architecture amd64 --name acme/debian-12 debian-12.box
+```
+
+Or make a box from a running machine with `vagrant package` (XVA by default,
+`package_format = :vhd` or `:qcow2` for only the first disk). Remove SSH host
+keys and `/etc/machine-id` in the guest first if the box will be shared.
+
+The box needs the Vagrant insecure key (or a configured
+`config.ssh.private_key_path`) for the `vagrant` user, as with any provider.
+
+## 4. Templates
+
+Each box version is imported once per SR as a template named
+`vagrant-xapi: <box> <version>`. Templates are shared by every user of the
+pool and take disk space on the SR.
+
+```sh
+vagrant xapi templates list                 # box, version, SR, size, clones
+vagrant xapi templates prune --keep 1       # per box: keep the newest version
+vagrant xapi templates remove "acme/debian-12 1.2.0"
+```
+
+`prune` and `remove` show which other users still have clones, and ask for
+confirmation (`--force` skips that). Removing a template does not break its
+clones: the storage layer keeps the shared base disk. The next `vagrant up`
+with that box version imports it again.
+
+The first `up` with a new box (or on a new SR) uploads and imports it; this
+can take a while for large boxes (`task_timeout`, default one hour). A second
+user starting the same box at the same time waits for the first import
+("Waiting for lock 'template....' held by ...") instead of importing again.
+
+## 5. Finding what the plugin created
+
+Everything the plugin creates is marked (full list in architecture.md,
+section 6). `vagrant xapi marked` lists it through XAPI, with the normal
+connection settings, from any directory whose Vagrantfile uses the xapi
+provider:
+
+```sh
+vagrant xapi marked                          # everything, of all users of the pool
+vagrant xapi marked --mine                   # only your VMs, with their snapshots, disks, networks
+vagrant xapi marked --project ~/src/shop     # VMs, snapshots, networks of one project
+vagrant xapi marked --vm <uuid>              # one VM, its snapshots and its disks
+vagrant xapi marked --box acme/debian-12     # the templates of one box
+vagrant xapi marked --plain | grep '^vdi'    # tab-separated, for scripts (or --json)
+```
+
+Filters can be repeated and combined; an object is listed when it matches
+any of them. The project path may be given as you see it (symlinks are
+resolved, e.g. `/var` → `/private/var` on macOS).
+
+The same with `xe` on dom0:
+
+```sh
+# All VMs of one user
+xe vm-list other-config:vagrant-xapi.user=alice params=uuid,name-label,power-state
+
+# All VMs of one project directory
+xe vm-list other-config:vagrant-xapi.project=/home/alice/src/shop
+
+# Templates made by the plugin
+xe template-list params=uuid,name-label | grep -B1 'vagrant-xapi:'
+
+# Disks of one VM (including extra disks, ISOs and snapshot disks)
+xe vdi-list other-config:vagrant-xapi.vm=<vm-uuid> params=uuid,name-label,virtual-size
+
+# Snapshot copies a `vagrant xapi diff` is using, or left after it was killed
+xe vdi-list name-label="vagrant-xapi diff of <vm-uuid>" params=uuid,other-config
+
+# Internal networks created for private_network
+xe network-list params=uuid,name-label,other-config | grep -B2 'vagrant-xapi.network'
+
+# Locks currently held
+xe pool-param-get uuid=$(xe pool-list --minimal) param-name=other-config | tr ';' '\n' | grep vagrant-xapi.lock
+```
+
+### Orphans
+
+When a project directory is removed without `vagrant destroy`, its VMs keep
+running. Find and remove them from any project with an xapi Vagrantfile:
+
+```sh
+vagrant xapi orphans             # list VMs of this Vagrant installation whose machine is gone
+vagrant xapi orphans --destroy   # remove them, their disks, snapshots and unused internal networks
+```
+
+Orphans are VMs of this Vagrant installation: VMs carry a random
+`vagrant-xapi.client_id`, stored in
+`~/.vagrant.d/data/vagrant-xapi/client-id`. (VMs from before that key match on
+login name and local hostname.) A VM is not an orphan while any Vagrant
+machine still points at it, also with a custom `VAGRANT_DOTFILE_PATH`. VMs of
+colleagues are never listed. Removing the `client-id` file makes earlier VMs
+look foreign; keep it with your Vagrant home. An administrator can find all
+marked VMs with the `xe` commands above and remove them with `xe
+vm-uninstall`.
+
+### Taking over an existing VM
+
+```sh
+vagrant xapi adopt web 1c2d...-uuid            # VM created by vagrant-xapi
+vagrant xapi adopt web 1c2d...-uuid --force    # any other VM
+```
+
+After `adopt`, `vagrant destroy` removes that VM and its disks.
+
+Machines of a vagrant-xenserver Vagrantfile are taken over without `adopt`:
+the first vagrant command finds the VM by the reference vagrant-xenserver
+stored, marks it and its own disks, and stores its UUID. A reference that no
+longer exists means the VM is gone (not created), as with vagrant-xenserver;
+the plugin never takes a VM by its name. A VM that vagrant-xapi meanwhile
+marked for another machine is left alone, with a warning.
+
+## 6. Locks
+
+Locks are keys `vagrant-xapi.lock.<name>` in the pool's `other_config`, with
+the owner and an expiry time (pool master's clock) in the value. While a lock
+is held, its owner refreshes `<name>.alive.0`/`.alive.1` every few minutes.
+Locks are released when the operation ends, also on errors and Ctrl-C. The
+lock of a crashed process stops being refreshed and is taken over once its
+expiry and heartbeats have passed: template locks after 15 minutes, network
+and placement locks after 10 minutes at most. Two takeovers never run at
+once: a takeover first sets `vagrant-xapi.lock.<name>.takeover` (gone again
+after a minute at most). To remove a stuck lock by hand (also its `.alive.*`
+keys):
+
+```sh
+POOL=$(xe pool-list --minimal)
+xe pool-param-get uuid=$POOL param-name=other-config param-key=vagrant-xapi.lock.<name>
+xe pool-param-remove uuid=$POOL param-name=other-config param-key=vagrant-xapi.lock.<name>
+```
+
+Only do that when the owner named in the value is really gone; otherwise two
+imports of the same box can run at the same time.
+
+## 7. Troubleshooting
+
+Every error is a short message without a stack trace. A stack trace is a
+bug: please report it with the output of `VAGRANT_LOG=debug`.
+
+| Message (start) | Cause | What to do |
+| --- | --- | --- |
+| `XAPI refused the login` | Wrong username or password | Check `VAGRANT_XAPI_USERNAME`/`PASSWORD` |
+| `Cannot connect to the XAPI host` | Host unreachable, wrong port, firewall | `curl -vk https://<host>/` from the local machine |
+| `provider :xenserver runs in vagrant-xenserver compatibility mode` | A Vagrantfile written for vagrant-xenserver | Nothing; it keeps working. For the newer options, move to `:xapi` and run `vagrant xapi migrate` (README, "Migrating from vagrant-xenserver") |
+| `a trigger uses XenApi::Client ... This stand-in is temporary` | A trigger calls XAPI through the `xenapi` gem of vagrant-xenserver | Replace the trigger with the provider's options (`cpus`, `config.vm.disk`) before the stand-in is removed |
+| `vagrant-xenserver is installed too, so vagrant-xapi leaves the :xenserver provider to it` | Both plugins installed | `vagrant plugin uninstall vagrant-xenserver` |
+| `Either a public network must be specified or the Host Internal Management Network parameter must be enabled` | A message of vagrant-xenserver itself: it is still installed and runs the `:xenserver` Vagrantfile | `vagrant plugin uninstall vagrant-xenserver` |
+| `VM ..., which vagrant-xenserver created for this machine, now belongs to machine ...` | The VM vagrant-xenserver stored for this machine is marked for another machine (e.g. a copied project); this machine is now not created | Nothing, or `vagrant xapi adopt <uuid> --force` if it really is this machine's VM |
+| `Machine '...' is in use by another Vagrant process` (`vagrant xapi migrate`) | Another vagrant command holds the machine | Run `vagrant xapi migrate` again when it has finished |
+| `Stale VM reference, looking the VM up by name instead` (from a trigger) | Trigger code of a vagrant-xenserver Vagrantfile uses `machine.id` as VM reference; vagrant-xapi stores the UUID | Nothing: the trigger finds the VM by name. Gone once the trigger is replaced by plugin options |
+| `The TLS certificate of ... could not be verified` | Self-signed certificate, wrong pin, renewed certificate; or `host` is a pool member and the master's certificate is not among the pins (the message names both) | The message shows the SHA-256 fingerprint of the certificate it got; check it on the host and set `tls_fingerprint` or `VAGRANT_XAPI_TLS_FINGERPRINT` (section 2). Once the master is pinned, `vagrant xapi hosts` shows every host's fingerprint |
+| `The XAPI connection is not configured` | Host, username or password missing | Set the environment variables or the provider options |
+| `The XAPI user lacks the permission` / `may not write the pool's other_config` | RBAC role too small | Use a role with more rights (section 2) |
+| `Warning: The guest closed the SSH connection right away ... Retrying...` | sshd refuses new connections for a while after failed logins (OpenSSH 9.8+ `PerSourcePenalties`), e.g. while cloud-init on the first boot is still creating the `vagrant` user | Nothing; the plugin keeps trying until `boot_timeout` and repeats the warning every minute. If it never ends: check that no other machine uses the IP (another VM with the same static IP answers, refuses the key and is then penalized), and look at the VM with `vagrant xapi console` |
+| A VNC viewer asks for a password (`vagrant xapi console`) | The console proxy asks for a one-time password | Use the password `vagrant xapi console` printed; it is valid while the command runs |
+| `console runs without a password` | Vagrant's Ruby has no 3DES (e.g. a FIPS-only OpenSSL) | Nothing needed; the console works without password, but other local users could connect while it is open |
+| The console (`vagrant xapi console`) shows GRUB's `error: out of memory` after `Loading initial ramdisk` | UEFI with too little memory: the firmware takes its share. Templates of disk-image boxes have 1 GiB; older ones, imported before that, still have the 256 MiB of XAPI's "Other install media" | Set `xapi.memory` (e.g. 2048), or remove the old template (`vagrant xapi templates remove`) so it is imported again |
+| `The guest did not report an IP within ... seconds` | No guest agent in the box, VM not booting (e.g. a BIOS box with `firmware = :uefi`, or no signed boot loader with `secure_boot`), or no IP on the first network | `vagrant xapi console` to look at the VM; install a guest agent, or set a static `ip:` on the first network and `ip_detection = [:static]` |
+| `The VM stopped ... while waiting for its IP` | The guest crashed or powered off during boot | `vagrant xapi console`, check firmware (`:bios`/`:uefi`) matches the image |
+| `Network '...' was not found in the pool` | Typo, or the network has another name on this pool | `xe network-list params=name-label,bridge` |
+| `There are N network objects named ...` | Duplicate names | Use the UUID |
+| `Storage repository '...' was not found` / `The pool has no default SR` | Wrong `sr`, or no default SR | `xe sr-list`; set `sr` |
+| `The storage repository is full` | Not enough space for the import or the clone | Free space, prune templates, or use another `sr` |
+| `The host does not have enough free memory` | Not enough RAM on the host | Lower `memory`, or set `affinity_host` to another host |
+| `This host cannot import qcow2 directly and qemu-img is not installed` | qcow2 box on a host whose XAPI cannot import qcow2 | Install `qemu-img` on the local machine |
+| `... is a qcow2 image with compressed clusters` / `uses qcow2 features that XAPI cannot import` / `XAPI could not import ... as qcow2` | XAPI's qcow2 import cannot read this image (compressed clusters, zstd, extended L2 entries) | Install `qemu-img` on the local machine; the plugin converts the image to VHD |
+| `Waiting for lock ... held by ...` (keeps waiting) | Another import of the same box runs, or a stale lock | Wait; or check the owner and remove the lock (section 6) |
+| `The VM cannot be suspended: ... no working PV drivers` | No guest agent or PV drivers | Install them in the box, or use `halt` |
+| `The VM cannot be suspended: the pool has no suspend SR` | No suspend SR | Set a suspend image SR on the pool |
+| `Disk '...' cannot shrink` | `size` smaller than the existing disk | Disks only grow; recreate the machine to shrink |
+| `Disk '...' can only grow while the VM is halted` | Resize on a running VM | Use `vagrant reload` |
+| `The SSH tunnel for port(s) ... did not start` | Host port in use, or SSH to the guest fails | See `.vagrant/machines/<name>/xapi/tunnels.log`; pick another host port or set `auto_correct: true` |
+| `This pool does not support vTPM` | XAPI without `VTPM` class | Remove `vtpm = true` |
+| `Box format ... is not supported` / `Boxes for provider ... cannot be used` | Wrong box | Use an xapi, xenserver or libvirt box |
+| `The box names ..., which is not a file inside the box directory` | The box's `metadata.json` or a symlink points outside the box | Do not use the box; ask its maker to fix it |
+| `The qcow2 image ... refers to a backing file` / `uses an external data file` | A qcow2 overlay, not a self-contained image | Flatten it: `qemu-img convert -O qcow2 in.qcow2 out.qcow2` |
+| `VM ... has no vagrant-xapi marking` | `destroy` of a VM the plugin did not create | Leave it, or `vagrant xapi adopt --force` it first |
+| `VM ... belongs to machine '...' of <dir>, not to this machine and project` | The project directory was copied or moved; the VM is marked for the other one | After a move: `vagrant xapi adopt [vm] <uuid>`. After a copy: leave it; the original project owns the VM |
+| `A network named '...' already exists and does not belong to this project` | A `private_network` name is used by another project | Choose another `name:`, or use `network: "<name>"` to attach to it on purpose |
+| `VM ... belongs to machine '...' of ... (user ...)` on `adopt` | Adopting someone else's VM | Check the UUID; add `--force` only if you really take it over |
+| `... is a member of a pool; its master is ...` | `host` is a pool member and `tls_fingerprint` is set | Use the master as `host` and pin its certificate |
+| `vagrant xapi diff reads the files through the guest` | `diff` of a halted or suspended machine | `vagrant up` first |
+| `Machine '...' uses the provider ...` on `diff --with` | The other machine is not a machine of vagrant-xapi | Compare with a machine of the `:xapi` (or `:xenserver`) provider |
+| `The guest uses crypto_LUKS` (or btrfs, ZFS, RAID) on `diff` | A copy of such disks cannot be attached next to the running system | Compare two running machines with `--with` |
+| `The guest lacks ...` on `diff` | Not a Linux guest with GNU findutils, coreutils and util-linux | Install them in the box |
+| `The file listing from the guest is incomplete or malformed` on `diff` | find in the guest could not write its whole listing, usually because the guest's `/tmp` is full | Free space in the guest's `/tmp` (often RAM) and run `diff` again |
+| `The snapshot disk contains crypto_LUKS` (or btrfs, ZFS, RAID) | The disk layout changed since the snapshot was taken | Compare with `--with`; if `lsblk` in the guest still shows volumes of the copy, reboot the guest |
+| `The guest lacks vgimportclone (lvm2)` | The guest (or the snapshot) uses LVM, but lvm2 is not complete | Install lvm2 in the box |
+| `A command in the guest failed (vgimportclone)` or `(vgchange)` | LVM could not import or activate the copy (e.g. a shared VG, or thin LVs without thin-provisioning-tools) | The message has LVM's text; `vagrant xapi diff` cleaned up |
+| `Could not deactivate the volume group vagrant_xapi_diff_...` | A logical volume of the copy was still open | Nothing; its devices go once they are closed. `sudo dmsetup ls` in the guest shows them |
+| `XAPI could not attach the snapshot disk to the running VM` | No PV drivers or guest agent, so XAPI refuses to hotplug a disk | Install xe-guest-utilities or xen-guest-agent in the box |
+| `Could not remove the snapshot copy <uuid> of the destroyed VM ...` | `vagrant destroy` could not remove a copy of an interrupted `diff`, also not after the VM was gone | Nothing removes it later: `vagrant xapi marked --vm <vm-uuid>` lists it, `xe vdi-destroy uuid=<uuid>` removes it |
+| `Could not remove the snapshot disk ...` | A `diff` could not unplug or destroy its copy (e.g. the guest kept it busy) | Run `vagrant xapi diff` again or `vagrant destroy`; both remove it. `vagrant reload` removes it too while it is still attached to the VM, not a copy whose VBD is already gone. Until then do not reboot from inside the guest: an attached copy has the same file system UUIDs |
+| `... snapshot copy (copies) of an interrupted vagrant xapi diff could not be removed from VM ...` on `up`, `reload` or `package` | A copy of an interrupted `diff` is still attached and XAPI refused to remove it (the warning before it says why) | Fix the cause and run the command again, or remove the copy by hand as the message says; until then the VM does not start, because the copy has the same file system UUIDs |
+| `Could not install the udev rule that keeps udev and systemd in the guest on its own disks ...` | `diff` could not write `/run/udev/rules.d/99-vagrant-xapi-diff.rules` or reload udev in the guest | The diff ran; check with `findmnt` that `/boot` and other fstab mounts by UUID are still mounted, and `sudo mount -a` if not |
+| `XAPI did not finish cloning the snapshot disk ...` | `VDI.clone` took longer than `task_timeout`, or the connection was lost | A clone that appears later has no marking, so no cleanup removes it: find it with the `xe vdi-list` command in the message and remove it with `xe vdi-destroy` |
+| `XAPI returned an error: <CODE>` | An XAPI error without its own text | Look the code up in the XAPI docs; the message names the call |
+
+SSH problems:
+
+- With `ssh_via_host`, the host key of dom0 must be in your `known_hosts`
+  (the ProxyCommand runs with `BatchMode=yes` and normal host key checking):
+  `ssh root@<host-address> true` once.
+- `vagrant ssh-config` shows exactly what Vagrant uses.
+
+### Debug logging
+
+```sh
+VAGRANT_LOG=debug vagrant up 2>debug.log
+grep -E '^ *[A-Z]+ xapi' debug.log
+```
+
+The plugin's lines start with `xapi:` (the client and services) or
+`xapi_<step>:` (the provider and each action step, e.g. `xapi_createvm:`).
+
+Every XAPI call is logged as `xapi -> <method> <params>` and
+`xapi <- <method> ok|error`. The password and the session id are replaced by
+`[REDACTED]`/`<session>`; check before sharing a log anyway (VM names and
+paths are in it).
+
+### Files in the machine directory
+
+`.vagrant/machines/<name>/xapi/`:
+
+| File | Contents |
+| --- | --- |
+| `id` | VM UUID (Vagrant) |
+| `forwarded_ports.json` | Running SSH tunnels: PID, addresses, the host port in use and the one in the Vagrantfile |
+| `tunnels.log` | stderr of the SSH tunnels |
+| `disk_meta` | Disks configured by `config.vm.disk` (Vagrant) |
+| `action_cloud_init` | Sentinel: cloud-init ran for this VM (Vagrant) |
+| `xapi_cloud_init_iso`, `vagrant-xapi-cidata-<sha>.iso` | Checksum and path of the cloud-init ISO |
+
+In the Vagrant home, `~/.vagrant.d/data/vagrant-xapi/`:
+
+| File | Contents |
+| --- | --- |
+| `digests.json` | Box digests, keyed on path, size and mtime; deleting it only costs one recomputation |
+| `client-id` | Random id of this installation, used to recognise its own VMs (orphans) |
+
+## 8. Development
+
+### Environment
+
+Ruby 3.0–3.4 (Vagrant does not support 3.5 or 4.x yet) with bundler.
+
+```sh
+bundle install
+bundle exec rake                       # unit tests
+bundle exec rspec spec/unit/api        # one directory
+bundle exec vagrant <cmd>              # Vagrant (pinned in the Gemfile) with this plugin loaded
+```
+
+The Gemfile has Vagrant (pinned to a release tag, currently v2.4.9) from
+git in group `:development` and the plugin itself in group `:plugins` (as `gem "vagrant-xapi", path:
+"."`), which `bundle exec vagrant` loads. Set `VAGRANT_CHECKPOINT_DISABLE=1`
+to stop Vagrant's version check. Running Vagrant outside its installer
+complains about a missing `bsdtar`; that only matters for `vagrant box add`
+of packed boxes and for `vagrant package` compression.
+
+If native gems fail to link (for example with a Homebrew Ruby whose
+`libruby` is built against Homebrew's glibc), build them with Homebrew's
+compiler by putting `gcc`/`cc`/`g++` shims that point to `gcc-NN`/`g++-NN`
+first in `PATH`.
+
+### Where things go
+
+- XAPI calls only in `lib/vagrant-xapi/api` and `services`; services never
+  require Vagrant.
+- Every user-facing text in `locales/en.yml` (`messages`, `errors`,
+  `config`, `commands`, `states`). Do not use `%{format}`, `%{scope}` or other
+  I18n reserved names as parameters.
+- New errors: raise `Services::Error.new(:key, **params)` in a service and add
+  `vagrant_xapi.errors.key`; `spec/unit/vagrant/errors_spec.rb` checks every
+  key renders.
+- New XAPI calls the fake does not know: add a `x_<Class>_<method>` method to
+  `spec/support/fake_xapi.rb` (generic getters, setters, `add_to_*`,
+  `remove_from_*`, `create`, `destroy`, `get_by_*` already work), and
+  remember that the fake only proves what we think XAPI does.
+- Anything the plugin creates in XAPI must carry a marking, and cleanup must
+  check it.
+
+### Tests
+
+- `spec/unit/api`: client and TLS (the TLS tests use a real local TLS
+  server, with WebMock disabled).
+- `spec/unit/services`: one spec per service, against the fake XAPI.
+- `spec/unit/vagrant`: config, errors, the real action stacks in a temporary
+  `Vagrant::Environment` (box placed straight in `VAGRANT_HOME`; SSH
+  communicator and guest stubbed), and the subcommands.
+
+### CI
+
+`.github/workflows/ci.yml` runs on every pull request and every push to
+`master`:
+
+- the unit tests on Ruby 3.1 to 3.4, the versions Vagrant's own CI tests
+  (Gemfile.lock is not checked in, so each run resolves the gems anew);
+- `shellcheck` on the acceptance script;
+- `gem build`, failing when the gem has a runtime dependency or lacks one of
+  the files listed under Releasing.
+
+The actions are pinned to a commit; bump them deliberately. The acceptance
+scenarios need a real pool and do not run in CI.
+
+### Acceptance tests
+
+Against a real pool; see [test/acceptance/README.md](../test/acceptance/README.md)
+for the environment variables.
+
+```sh
+test/acceptance/scenarios.sh          # A1-A19
+test/acceptance/scenarios.sh A3 A13   # some
+```
+
+Run them before a release. The scenarios check the pool with
+`vagrant xapi marked`, so they only need the normal connection settings, no
+access to dom0.
+
+## 9. Releasing
+
+1. Run the unit tests and the acceptance scenarios.
+2. Bump `VagrantPlugins::Xapi::VERSION` in `lib/vagrant-xapi/version.rb` and
+   add the changes to `CHANGELOG.md`, with the release date instead of
+   "unreleased".
+3. `gem build vagrant-xapi.gemspec` and check the contents:
+   `tar -xOf vagrant-xapi-*.gem data.tar.gz | tar -tz` must list `lib/`,
+   `locales/en.yml`, `docs/architecture.md`, `docs/operations.md`,
+   `README.md`, `LICENSE`, `CHANGELOG.md`, and the gem must have no runtime
+   dependencies.
+4. Try it in a clean Vagrant: `vagrant plugin install ./vagrant-xapi-X.Y.Z.gem`.
+5. Once that is merged into `master`, build the gem again from a clean
+   checkout of `master` and push it to rubygems.org, with an account that
+   owns `vagrant-xapi` there (MFA is required, so `gem push` asks for a
+   one-time code). The gem is public: `spec.files` takes every matching file
+   in the working tree, tracked or not, so `git status` must be clean.
+
+   ```sh
+   git status --short --ignored lib locales docs README.md LICENSE CHANGELOG.md   # must print nothing
+   gem build vagrant-xapi.gemspec
+   gem push vagrant-xapi-X.Y.Z.gem
+   ```
+
+   A pushed version cannot be replaced; a mistake needs a new version
+   (`gem yank` only hides it from new installs).
+6. Tag the release on GitHub, with the release's section of `CHANGELOG.md`
+   as its notes:
+
+   ```sh
+   gh release create vX.Y.Z -R exonet/vagrant-xapi \
+     --target master --title vX.Y.Z --notes-file notes.md
+   ```
+
+   The gem is not attached: rubygems.org is the only place it comes from
+   (`docs/architecture.md`, section 15).

data/lib/vagrant-xapi/action/base.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+require "log4r"
+
+require_relative "../errors"
+require_relative "../glue"
+
+module VagrantPlugins
+  module Xapi
+    module Action
+      # Shared helpers for our middleware.
+      class Base
+        def initialize(app, env)
+          @app = app
+          @logger = Log4r::Logger.new("vagrantplugins::xapi_#{self.class.name.split("::").last.downcase}")
+        end
+
+        protected
+
+        def ctx(env)
+          env[:xapi_ctx] ||= Glue.context(env[:machine], env, client: env[:xapi])
+        end
+
+        def vm(env)
+          ref = Glue.vm_ref(env[:machine], ctx(env))
+          raise Vagrant::Errors::VMNotCreatedError unless ref
+
+          ref
+        end
+
+        def wrap(&block)
+          Errors.wrap(&block)
+        end
+
+        def config(env)
+          env[:machine].provider_config
+        end
+      end
+
+      # First middleware of every stack: client and service errors become
+      # readable VagrantErrors, never stack traces.
+      class TranslateErrors
+        def initialize(app, _env)
+          @app = app
+        end
+
+        def call(env)
+          @app.call(env)
+        rescue Api::Error, Services::Error => e
+          raise Errors.translate(e)
+        end
+      end
+
+      # Puts the shared XAPI session in env[:xapi].
+      class ConnectXapi < Base
+        def call(env)
+          env[:xapi] = Glue.client(env[:machine])
+          env[:xapi_ctx] = Glue.context(env[:machine], env, client: env[:xapi])
+          previous = Thread.current[:vagrant_xapi_cancel]
+          Thread.current[:vagrant_xapi_cancel] = -> { env[:interrupted] ? true : false }
+          begin
+            @app.call(env)
+          ensure
+            Thread.current[:vagrant_xapi_cancel] = previous
+          end
+        end
+      end
+
+      # env[:result] = whether the VM exists.
+      class IsCreated < Base
+        def call(env)
+          env[:result] = env[:machine].state.id != :not_created
+          @app.call(env)
+        end
+      end
+
+      # Raises when the machine is not running (for ssh, ssh_run).
+      class CheckRunning < Base
+        def call(env)
+          state = env[:machine].state.id
+          raise Vagrant::Errors::VMNotCreatedError if state == :not_created
+          raise Vagrant::Errors::VMNotRunningError if state != :running
+
+          @app.call(env)
+        end
+      end
+
+      # Prints a message from our locale file.
+      class Message < Base
+        def initialize(app, env, key, level = :info)
+          super(app, env)
+          @key = key
+          @level = level
+        end
+
+        def call(env)
+          env[:ui].public_send(@level, I18n.t("vagrant_xapi.messages.#{@key}"))
+          @app.call(env)
+        end
+      end
+    end
+  end
+end