beaker-kubevirt 1.0.0 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 42851521ceebcac0549527587051965bce3698a15ea7995f3111db7b481b5ce9
4
- data.tar.gz: 4ed638d67f36721a8d656e57096f09ccd4e32c010c642625608ac830de41fea3
3
+ metadata.gz: 5b2973046f9f272d05b8397362b0cacd9fdb2a7aec80be0e9459d36bce83ee4c
4
+ data.tar.gz: a6b64fd54c8276ceb60a6904f636e9ff7a380849b652a3c7e51ab5f28848e4f4
5
5
  SHA512:
6
- metadata.gz: aaef810b582931893e1828c4fcf7cde1d42b86f30ab030ef0937bde204eb1b8cccc4e520e0b0bbbc854c55f774514c27dc5d5f1c00cb8ea41e1d39d632bd137d
7
- data.tar.gz: a56cb846fc5bb309755fca675017abd7c66f518bf266b092c26482ed64254f10e4cfae25580217b750dbb747cf2230c8bdd1cf8e8b947c7c5bf3c399bdc23187
6
+ metadata.gz: 75a0558dd8ad93cc7ef7c70ed129d05754a3998165768f0b94da4046cdd33e3e8d9a20617982c89095023225136afbcf54d1a09014d4957b91a992ca31d22f0e
7
+ data.tar.gz: b8716d9af99b6d13f86f2ab1df4bca2c6894392a6a2572863fbc5735578d426f1cb212ca9fcd1d81e29e1161ea58e6feeeda281a8b11bf8dc7b91c83b8a528c7
data/CHANGELOG.md CHANGED
@@ -2,6 +2,30 @@
2
2
 
3
3
  All notable changes to this project will be documented in this file.
4
4
 
5
+ ## [1.0.0](https://github.com/voxpupuli/beaker-kubevirt/tree/1.0.0) (2026-11-05)
6
+
7
+ [Full Changelog](https://github.com/voxpupuli/beaker-kubevirt/compare/1.0.0...1.0.0)
8
+
9
+ **Closed issues:**
10
+
11
+ - Allow the test-group identifier to be supplied via the environment [\#63](https://github.com/voxpupuli/beaker-kubevirt/issues/63)
12
+ - ImagePullBackOff treated as fatal aborts provisioning that would otherwise succeed [\#54](https://github.com/voxpupuli/beaker-kubevirt/issues/54)
13
+
14
+ **Merged pull requests:**
15
+
16
+ - Update voxpupuli-rubocop to ~\> 5.3.0 and fix new offenses [\#86](https://github.com/voxpupuli/beaker-kubevirt/pull/86) ([silug](https://github.com/silug))
17
+ - Add kubevirt\_labels and kubevirt\_annotations [\#85](https://github.com/voxpupuli/beaker-kubevirt/pull/85) ([jaevans](https://github.com/jaevans))
18
+ - Add kubevirt\_storage\_class for the root disk [\#84](https://github.com/voxpupuli/beaker-kubevirt/pull/84) ([jaevans](https://github.com/jaevans))
19
+ - Add kubevirt\_node\_selector and kubevirt\_tolerations [\#83](https://github.com/voxpupuli/beaker-kubevirt/pull/83) ([jaevans](https://github.com/jaevans))
20
+ - Honour BEAKER\_kubevirt\_disk\_cache [\#82](https://github.com/voxpupuli/beaker-kubevirt/pull/82) ([jaevans](https://github.com/jaevans))
21
+ - Add kubevirt\_disk\_cache option for the root disk cache mode [\#77](https://github.com/voxpupuli/beaker-kubevirt/pull/77) ([silug](https://github.com/silug))
22
+ - Update voxpupuli-rubocop requirement from ~\> 5.2.0 to ~\> 5.3.0 [\#72](https://github.com/voxpupuli/beaker-kubevirt/pull/72) ([dependabot[bot]](https://github.com/apps/dependabot))
23
+ - Add kubevirt\_cpu\_request and kubevirt\_cpu\_limit [\#66](https://github.com/voxpupuli/beaker-kubevirt/pull/66) ([jaevans](https://github.com/jaevans))
24
+ - Detect an unschedulable virt-launcher pod instead of waiting out the timeout [\#65](https://github.com/voxpupuli/beaker-kubevirt/pull/65) ([jaevans](https://github.com/jaevans))
25
+ - feat: allow the test-group identifier to be supplied via the environment [\#64](https://github.com/voxpupuli/beaker-kubevirt/pull/64) ([jaevans](https://github.com/jaevans))
26
+ - Remove CodeQL workflow in favor of default setup [\#62](https://github.com/voxpupuli/beaker-kubevirt/pull/62) ([silug](https://github.com/silug))
27
+ - Give image pull backoff a grace window before failing [\#55](https://github.com/voxpupuli/beaker-kubevirt/pull/55) ([jaevans](https://github.com/jaevans))
28
+
5
29
  ## [1.0.0](https://github.com/voxpupuli/beaker-kubevirt/tree/1.0.0) (2026-07-17)
6
30
 
7
31
  [Full Changelog](https://github.com/voxpupuli/beaker-kubevirt/compare/a388e545ac4aa46006e53f006ad5af1a243bdbbd...1.0.0)
data/README.md CHANGED
@@ -78,21 +78,33 @@ CONFIG:
78
78
  | `networks` | Custom network configuration | No | Auto-generated | HOSTS (per-host) |
79
79
  | `kubevirt_ssh_key` | SSH public key path or content | Yes | Auto-detect from `~/.ssh/` | HOSTS (per-host) |
80
80
  | `kubevirt_cpus` | CPU cores for VM | No | `1` | HOSTS (per-host) |
81
+ | `kubevirt_cpu_request` | CPU request for the VM pod, as a Kubernetes CPU quantity (`2`, `1500m`). Unset leaves KubeVirt to derive it from `kubevirt_cpus` divided by the cluster's `cpuAllocationRatio`. Set it for guests that must not be throttled while booting. | No | unset | HOSTS (per-host), CONFIG |
82
+ | `kubevirt_cpu_limit` | CPU limit for the VM pod, as a Kubernetes CPU quantity. Unset leaves the compute container burstable. | No | unset | HOSTS (per-host), CONFIG |
81
83
  | `kubevirt_memory` | Memory for VM | No | `2Gi` | HOSTS (per-host) |
82
84
  | `kubevirt_memory_overhead` | Extra memory added to the guest for the virt-launcher container memory limit. Raise this for Windows guests that OOMKill with the default. | No | `512Mi` | HOSTS (per-host), CONFIG |
83
85
  | `kubevirt_memory_request` | Memory request for the VM pod. | No | Same as `kubevirt_memory` | HOSTS (per-host), CONFIG |
84
86
  | `kubevirt_disk_size` | Size of the root disk | No | `10Gi` | HOSTS (per-host) |
87
+ | `kubevirt_storage_class` | Storage class for the root disk's DataVolume, used when `kubevirt_vm_image` is an `http(s)://` URL or a PVC to clone. Unset uses the cluster's default storage class. A `docker://`/`oci://` containerDisk has no DataVolume and ignores it. Cloning a PVC into a different class may make CDI copy the data instead of using a CSI clone. A non-empty `BEAKER_kubevirt_storage_class` environment variable overrides the host and CONFIG values. | No | cluster default | `BEAKER_kubevirt_storage_class`, HOSTS (per-host), CONFIG |
85
88
  | `kubevirt_cloud_init` | Custom cloud-init YAML | No | Auto-generated | HOSTS (per-host) |
86
89
  | `kubevirt_vm_ssh_port` | SSH port inside the VM | No | `22` | HOSTS (per-host) |
87
90
  | `kubevirt_readiness_probe_disabled` | Skip the VMI SSH readinessProbe. Default is `false` for pod-network modes and `true` for `multus` (where a probe from the virt-launcher netns typically can't reach a bridge-only guest). | No | mode-dependent | HOSTS (per-host), CONFIG |
88
91
  | `kubevirt_readiness_probe` | Probe tuning hash (snake_case keys): `initial_delay_seconds` (30), `period_seconds` (10), `timeout_seconds` (3), `failure_threshold` (60), `success_threshold` (1). The default failure budget (600s) accommodates slow Windows first-boots. When the probe is enabled, `timeout` is auto-raised to cover this budget. | No | see description | HOSTS (per-host), CONFIG |
92
+ | `kubevirt_unschedulable_grace` | Seconds the virt-launcher pod may stay unschedulable (`PodScheduled=False`, reason `Unschedulable` or `SchedulerError`) before provisioning gives up and reports the scheduler's own message. Cannot be immediate — an unschedulable pod is precisely what tells a cluster autoscaler to add a node. Bounded by the effective `timeout`, which this does not raise. | No | `300` | HOSTS (per-host), CONFIG |
93
+ | `kubevirt_image_pull_grace` | Seconds a containerDisk pull may sit in `ImagePullBackOff`/`ErrImagePull` before provisioning gives up. kubelet retries pulls, so a cold pull-through cache, a slow upstream registry or a dropped connection normally clears on a later attempt. Bounded by the effective `timeout` (which this does not raise), and skipped when the registry's error is definitive — `manifest unknown`, containerd's `failed to resolve reference …: not found` for a missing tag, `pull access denied`, `requested access to the resource is denied`, `unauthorized: authentication required` — so a typo'd tag or a refused credential fails on the first poll that shows the error. A blob 404 from a cold cache, or a `permission denied` from the network, gets the window. Set it to `0` to switch the window off and fail on the first backoff, as provisioning did before this option existed. The first backoff seen for each container is logged as a warning, so a run that reaches `timeout` before the window ends still records why. | No | `600` | HOSTS (per-host), CONFIG |
89
94
  | `kubevirt_disable_virtio` | Disable virtio devices (for Windows compatibility). If set to true the disk bus will be set to `sata` and the network adapter will be model `e1000` | No | `false` | HOSTS (per-host) |
95
+ | `kubevirt_disk_cache` | Cache mode for the root disk: `none`, `writethrough` or `writeback`. Unset leaves the choice to KubeVirt (`none` when the backing store supports O_DIRECT). `writeback` lets the virt-launcher's page cache absorb guest writes until the guest flushes; use it for throwaway test VMs on slow or shared storage, where a lost VM only costs a re-run. A non-empty `BEAKER_kubevirt_disk_cache` environment variable overrides both. | No | unset | `BEAKER_kubevirt_disk_cache`, HOSTS (per-host), CONFIG |
96
+ | `kubevirt_node_selector` | Hash of node labels the VM's pod must match, e.g. `{ 'node-pool.example.com/ci' => 'true' }`. Select a group of nodes by a label they share, not a single node by name. Quote values that look like decimals (`'1.10'`): an unquoted float is refused, since YAML would read `1.10` as `1.1`. `BEAKER_kubevirt_node_selector`, a comma-separated list of `key=value` pairs, replaces the host and CONFIG values when non-empty. | No | unset | `BEAKER_kubevirt_node_selector`, HOSTS (per-host), CONFIG |
97
+ | `kubevirt_tolerations` | List of taints the VM's pod tolerates. Each entry is a hash with `key`, `effect` (`NoSchedule`, `PreferNoSchedule` or `NoExecute`) and an optional `value`; an entry without a value tolerates the key whatever its value. Other fields (`operator`, `tolerationSeconds`) are refused rather than ignored. Tolerating a taint only allows the VM onto those nodes; pair it with `kubevirt_node_selector` to keep it on them. `BEAKER_kubevirt_tolerations`, a comma-separated list of `key=value:Effect` or `key:Effect` entries, replaces the host and CONFIG values when non-empty. | No | unset | `BEAKER_kubevirt_tolerations`, HOSTS (per-host), CONFIG |
98
+ | `kubevirt_labels` | Hash of extra labels for every resource beaker labels: the VM, its template (and so the VMI and virt-launcher pod), the root DataVolume and the cloud-init Secret. `beaker/test-group`, `beaker/host` and `kubevirt.io/vm` are beaker's own and raise an error if set here, as does a key or value that is not valid Kubernetes label syntax (values are at most 63 characters of letters, digits, `-`, `_` and `.`). `BEAKER_kubevirt_labels`, a comma-separated list of `key=value` pairs, replaces the host and CONFIG values when non-empty. A host's hash replaces the CONFIG hash rather than merging with it. | No | unset | `BEAKER_kubevirt_labels`, HOSTS (per-host), CONFIG |
99
+ | `kubevirt_annotations` | Hash of annotations for the VM template only (and so the VMI and virt-launcher pod), e.g. `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"`. Keys must be valid Kubernetes keys; values are free text. `BEAKER_kubevirt_annotations`, a comma-separated list of `key=value` pairs, replaces the host and CONFIG values when non-empty. A host's hash replaces the CONFIG hash rather than merging with it; values given that way cannot contain commas, so set complex values in the hosts file. | No | unset | `BEAKER_kubevirt_annotations`, HOSTS (per-host), CONFIG |
90
100
 
91
101
  **Important**: The `namespace`, `kubeconfig`, and `kubecontext` options must be specified in the global `CONFIG` section, not per-host. All VMs will be created in the same Kubernetes namespace.
92
102
 
93
103
  Notes:
94
104
  - Several per-host options support global fallbacks via `CONFIG` (e.g., `kubevirt_cpus`, `kubevirt_memory`, `kubevirt_vm_ssh_port`).
95
105
  - The `networks` key for Multus is intentionally unprefixed (use `networks`, not `kubevirt_networks`).
106
+ - `kubevirt_cpus` sets the guest's core count, which is not the same thing as the pod's CPU request. Left to itself, KubeVirt divides the core count by the cluster's `cpuAllocationRatio` — 10 by default — so a 2-core guest lands in a burstable pod requesting `200m`. On a busy node that guest is throttled to roughly a tenth of what it advertises, and a slow first boot can outlast the readiness probe's failure budget while nothing anywhere reports an error. Set `kubevirt_cpu_request` when boot time matters; Windows guests are the usual case.
107
+ - A virt-launcher pod the scheduler cannot place used to be invisible: its phase is `Pending`, not `Failed`, and it carries no container statuses at all, so provisioning waited out the full `timeout` and reported `Timeout waiting for VM ... to be ready` — indistinguishable from a slow guest boot. It now reports the scheduler's own message (`0/8 nodes are available: 5 Insufficient cpu`) as soon as the state is seen, and fails after `kubevirt_unschedulable_grace`. Raise `kubevirt_memory_request` or a CPU request beyond what a node can offer and this is the error you will get.
96
108
  - Long-running commands aren't killed by idle-timeout drops, even when output streams in only one direction (e.g., a Windows scanner running for many minutes). The `port-forward` proxy no longer enforces a client-side read-silence timeout (it relied on net-ssh keepalive packets that net-ssh suppresses while server output is flowing) and instead sends a WebSocket protocol ping every 60s to keep the upstream tunnel alive. net-ssh keepalive defaults are also tightened to `keepalive_interval: 60`, `keepalive_maxcount: 5` — Beaker already enables `keepalive: true` — which protects `multus` and `nodeport` modes against NAT/conntrack timeouts. Override per-host under `ssh:` if needed.
97
109
 
98
110
  ### VM Image Formats
@@ -254,6 +266,29 @@ All resources created are labeled for traceability and cleanup:
254
266
 
255
267
  These labels are used during `cleanup` to remove VMs, secrets, and services associated with the test group.
256
268
 
269
+ ### Supplying the test-group identifier
270
+
271
+ By default the test-group identifier is `beaker-<random hex>`, generated per process. Set `BEAKER_KUBEVIRT_TEST_GROUP` to choose it instead:
272
+
273
+ ```bash
274
+ BEAKER_KUBEVIRT_TEST_GROUP="beaker-ci-${CI_JOB_ID}" bundle exec rake beaker
275
+ ```
276
+
277
+ This matters when something outside the beaker process has to clean up after it. In CI, a cancelled job can be killed before beaker tears anything down, and with a random identifier the job has no way to name the resources it leaked. With a known identifier the job can delete by `beaker/test-group=<its own identifier>` once beaker is gone, without disturbing other runs that share the namespace:
278
+
279
+ ```bash
280
+ kubectl -n "$namespace" delete vm,dv,secret,svc -l "beaker/test-group=beaker-ci-${CI_JOB_ID}"
281
+ ```
282
+
283
+ The identifier must be unique to each run that can be live at the same time. It names the VMs, DataVolumes and Secrets as well as labelling them, so two runs that share it collide on create, and either run's cleanup deletes the other's resources. `CI_JOB_ID` is unique per job on GitLab. On GitHub Actions, `GITHUB_RUN_ID` is not: every job and matrix leg of a run shares it, and so do re-run attempts. Combine it with the attempt, the job and the matrix index, keeping the result within 63 characters (a longer value is refused rather than truncated, since truncation would cut off the part that tells runs apart):
284
+
285
+ ```yaml
286
+ env:
287
+ BEAKER_KUBEVIRT_TEST_GROUP: beaker-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index }}
288
+ ```
289
+
290
+ Values that are not valid Kubernetes label values are sanitized (and a warning is logged), so prefer an identifier that is already label-safe — otherwise the label will not be the string you set.
291
+
257
292
  ## Requirements
258
293
 
259
294
  - KubeVirt and Kubernetes cluster access via `kubeconfig`
data/examples/hosts.yaml CHANGED
@@ -8,6 +8,11 @@ HOSTS:
8
8
  kubevirt_network_mode: port-forward
9
9
  kubevirt_ssh_key: ~/.ssh/id_rsa.pub
10
10
  kubevirt_cpus: 2
11
+ # Guest core count is not the pod's CPU request: without kubevirt_cpu_request
12
+ # KubeVirt divides the cores by the cluster's cpuAllocationRatio (default 10),
13
+ # so these 2 cores request only 200m and get throttled on a busy node.
14
+ # kubevirt_cpu_request: 2
15
+ # kubevirt_cpu_limit: 2
11
16
  kubevirt_memory: 4Gi
12
17
  # Extra memory added to the guest for the virt-launcher container memory
13
18
  # limit. Bump this for Windows guests that get OOMKilled with the default.
@@ -1,6 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require 'securerandom'
4
+ require 'digest'
4
5
  require 'yaml'
5
6
  require 'base64'
6
7
  require 'socket'
@@ -53,10 +54,61 @@ module Beaker
53
54
 
54
55
  # VMI phases that indicate the VM will never reach Running.
55
56
  VMI_TERMINAL_PHASES = %w[Failed Succeeded].freeze
57
+
58
+ # Values KubeVirt accepts for a disk's cache mode
59
+ DISK_CACHE_MODES = %w[none writethrough writeback].freeze
60
+ # Taint effects a toleration may name
61
+ TOLERATION_EFFECTS = %w[NoSchedule PreferNoSchedule NoExecute].freeze
62
+ # Fields a kubevirt_tolerations entry may set
63
+ TOLERATION_FIELDS = %w[key value effect].freeze
64
+ # Labels beaker sets itself; cleanup and the NodePort service select on them
65
+ RESERVED_LABEL_KEYS = %w[beaker/test-group beaker/host kubevirt.io/vm].freeze
66
+ # Kubernetes label values, and the name part of label and annotation keys
67
+ LABEL_NAME_RE = /\A(?:[A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]\z/
68
+ # The optional prefix of a label or annotation key (an RFC 1123 subdomain)
69
+ KEY_PREFIX_RE = /\A[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9](?:[-a-z0-9]*[a-z0-9])?)*\z/
70
+ KEY_PREFIX_MAX = 253
56
71
  # virt-launcher container waiting reasons we treat as fatal.
57
- FATAL_CONTAINER_WAITING_REASONS = %w[CrashLoopBackOff ImagePullBackOff ErrImagePull].freeze
72
+ FATAL_CONTAINER_WAITING_REASONS = %w[CrashLoopBackOff].freeze
73
+ # virt-launcher container waiting reasons that mean an image pull is being
74
+ # retried. kubelet re-attempts these with increasing backoff, and a cold
75
+ # pull-through cache, a slow upstream or a transient network error all clear
76
+ # without intervention — so they are only fatal once they persist (see
77
+ # DEFAULT_IMAGE_PULL_GRACE).
78
+ IMAGE_PULL_WAITING_REASONS = %w[ImagePullBackOff ErrImagePull].freeze
79
+ # How long an image pull may stay in backoff before we give up on it, unless
80
+ # overridden with :kubevirt_image_pull_grace. Still bounded by the overall
81
+ # wait_for_vm_ready timeout, which this does not extend.
82
+ DEFAULT_IMAGE_PULL_GRACE = 600
83
+ # Pull failures kubelet's retry will never fix — a bad tag, a missing repo or
84
+ # a rejected credential. Failing these immediately keeps a typo'd image as
85
+ # fast to diagnose as it was before the grace window existed. Transport-level
86
+ # failures (EOF, connection reset, timeouts) deliberately do not match. The
87
+ # patterns are anchored to the registry's and containerd's own wording: a
88
+ # bare "not found" would catch a cold pull-through cache's blob 404, and a
89
+ # bare "denied" would catch `connect: permission denied` from a network flap,
90
+ # both of which are what the window is for. containerd reports a missing tag
91
+ # as `failed to resolve reference "<ref>": <ref>: not found`; resolving is
92
+ # the tag/manifest lookup, never a blob fetch.
93
+ DEFINITIVE_IMAGE_PULL_FAILURE_RE = Regexp.union(
94
+ /manifest unknown/i,
95
+ /failed to resolve reference .*: not found/i,
96
+ /pull access denied/i,
97
+ /requested access to the resource is denied/i,
98
+ /unauthorized: authentication required/i,
99
+ )
58
100
  # virt-launcher container termination reasons we treat as fatal.
59
101
  FATAL_CONTAINER_TERMINATED_REASONS = %w[OOMKilled Error ContainerCannotRun].freeze
102
+ # PodScheduled=False reasons that mean the scheduler cannot place the virt-launcher pod.
103
+ # SchedulingGated is deliberately excluded: a gated pod is being held on purpose by
104
+ # whoever set the gate, and is not the cluster failing to find room.
105
+ UNSCHEDULABLE_POD_REASONS = %w[Unschedulable SchedulerError].freeze
106
+ # How long the virt-launcher pod may stay unschedulable before provisioning gives up,
107
+ # unless overridden with :kubevirt_unschedulable_grace. This cannot be immediate: a
108
+ # cluster autoscaler treats an unschedulable pod as its cue to add a node, and a node
109
+ # returning from a drain or reboot frees capacity on its own — both take minutes. Still
110
+ # bounded by the overall wait_for_vm_ready timeout, which this does not extend.
111
+ DEFAULT_UNSCHEDULABLE_GRACE = 300
60
112
 
61
113
  ##
62
114
  # Create a new instance of the KubeVirt hypervisor object
@@ -72,6 +124,12 @@ module Beaker
72
124
  # @option options [String] :kubevirt_network_mode Network mode (port-forward, nodeport, multus)
73
125
  # @option options [String] :kubevirt_ssh_key SSH public key to inject
74
126
  # @option options [String] :kubevirt_cpus CPU resources for VM
127
+ # @option options [String] :kubevirt_cpu_request CPU request for the VM pod, as a Kubernetes
128
+ # CPU quantity ('2', '1500m'). Unset by default, which leaves KubeVirt to derive the request
129
+ # from the guest core count divided by the cluster's cpuAllocationRatio (default 10). Set it
130
+ # for guests that must not be throttled while booting -- Windows especially.
131
+ # @option options [String] :kubevirt_cpu_limit CPU limit for the VM pod, as a Kubernetes CPU
132
+ # quantity. Unset by default, which leaves the compute container burstable.
75
133
  # @option options [String] :kubevirt_memory Memory resources for VM
76
134
  # @option options [String] :kubevirt_memory_overhead Extra memory added to guest for the
77
135
  # container memory limit (default: '512Mi'). Needed for Windows guests where KubeVirt's
@@ -89,7 +147,21 @@ module Beaker
89
147
  # (period_seconds * failure_threshold = 600s) accommodates slow Windows first-boots.
90
148
  # @option options [Integer] :timeout Timeout for operations. When the readiness probe is
91
149
  # enabled, the effective wait is max(:timeout, probe budget + 60s).
150
+ # @option options [Integer] :kubevirt_image_pull_grace How long a containerDisk pull may
151
+ # stay in ImagePullBackOff/ErrImagePull before it is treated as fatal (default: 600s).
152
+ # kubelet retries image pulls, so a cold pull-through cache or a slow registry clears on
153
+ # its own. Bounded by (and does not extend) the effective :timeout above, and skipped
154
+ # entirely when the pull failure is definitive (an unknown manifest or tag, or the
155
+ # registry refusing access).
156
+ # Set it to 0 to switch the window off and fail on the first backoff, as this check
157
+ # did before the option existed.
158
+ # @option options [Integer] :kubevirt_unschedulable_grace How long the virt-launcher pod may
159
+ # remain unschedulable before it is treated as fatal (default: 300s). An unschedulable pod
160
+ # is what triggers a cluster autoscaler to add a node, so this cannot be immediate.
161
+ # Bounded by (and does not extend) the effective :timeout above.
92
162
  # @option options [Boolean] :kubevirt_disable_virtio Disable virtio devices (for compatibility with Windows)
163
+ # @option options [String] :kubevirt_disk_cache Cache mode for the root disk: none, writethrough
164
+ # or writeback. Unset leaves the choice to KubeVirt.
93
165
  def initialize(kubevirt_hosts, options)
94
166
  require 'beaker/hypervisor/kubevirt_helper'
95
167
 
@@ -105,9 +177,16 @@ module Beaker
105
177
  @hosts = kubevirt_hosts
106
178
  # Ensure the helper gets the validated namespace
107
179
  @kubevirt_helper = KubevirtHelper.new(@options)
108
- @test_group_identifier = "beaker-#{SecureRandom.hex(4)}"
180
+ @test_group_identifier = configured_test_group_identifier || "beaker-#{SecureRandom.hex(4)}"
109
181
  @cleanup_called = false
110
182
  @cleanup_mutex = Mutex.new
183
+ # "<pod_name>/<container>" => monotonic timestamp the container was first
184
+ # seen backing off an image pull. Cleared once it is running or terminated.
185
+ @image_pull_backoff_since = {}
186
+ # vm_name => { uid:, since: } — the virt-launcher pod's UID and the monotonic
187
+ # timestamp it was first seen unschedulable. Cleared as soon as the scheduler
188
+ # places it; restarted when a pod with a different UID replaces it.
189
+ @unschedulable_since = {}
111
190
 
112
191
  # Register at_exit handler to ensure cleanup happens even on non-success exits
113
192
  # This handles cases like Ctrl+C, errors, or test failures that occur after
@@ -277,7 +356,7 @@ module Beaker
277
356
  if vm_image&.start_with?('http://', 'https://')
278
357
  base_name = vm_image.split('/').last
279
358
  # Create a unique datavolume name by including the VM name in it
280
- host['dv_name'] = sanitize_k8s_name("#{vm_name}-#{base_name}-dv")
359
+ host['dv_name'] = bounded_k8s_name("#{vm_name}-#{base_name}-dv")
281
360
  elsif vm_image && !vm_image.start_with?('docker://', 'oci://')
282
361
  # For PVC sources, we also need to clone to avoid sharing the same disk
283
362
  source_pvc = vm_image.sub(%r{^pvc://}, '')
@@ -287,7 +366,7 @@ module Beaker
287
366
  else
288
367
  source_pvc_name = source_pvc
289
368
  end
290
- host['dv_name'] = sanitize_k8s_name("#{vm_name}-#{source_pvc_name}")
369
+ host['dv_name'] = bounded_k8s_name("#{vm_name}-#{source_pvc_name}")
291
370
 
292
371
  end
293
372
 
@@ -309,7 +388,7 @@ module Beaker
309
388
  def create_cloud_init_secret(host, cloud_init_data)
310
389
  raise 'Cloud-init data must be provided' unless cloud_init_data
311
390
 
312
- secret_name = "#{host['vm_name']}-cloud-init"
391
+ secret_name = bounded_k8s_name("#{host['vm_name']}-cloud-init")
313
392
  @logger.debug("Creating cloud-init secret #{secret_name} in namespace #{@namespace}")
314
393
 
315
394
  secret_spec = {
@@ -337,7 +416,25 @@ module Beaker
337
416
  # @return [String] The generated VM name
338
417
  def generate_vm_name(host)
339
418
  host_name = host.respond_to?(:name) ? host.name : host['name']
340
- sanitize_k8s_name("#{@test_group_identifier}-#{host_name}")
419
+ bounded_k8s_name("#{@test_group_identifier}-#{host_name}")
420
+ end
421
+
422
+ ##
423
+ # Sanitize a name to a DNS label, keeping distinct inputs distinct when
424
+ # they exceed the 63-char cap.
425
+ #
426
+ # Plain truncation drops the tail, which is where the host name (or a
427
+ # suffix such as "-ssh") sits, so hosts sharing a long test-group
428
+ # identifier would collide. At the cap the tail is replaced with a digest
429
+ # of the untruncated name instead.
430
+ # @param [String] full_name The unsanitized name
431
+ # @return [String] A valid DNS label of at most 63 characters
432
+ def bounded_k8s_name(full_name)
433
+ name = sanitize_k8s_name(full_name)
434
+ return name if name.length < K8S_NAME_MAX
435
+
436
+ digest = Digest::SHA256.hexdigest(full_name)[0, 8]
437
+ "#{name[0, K8S_NAME_MAX - digest.length - 1].sub(/-+\z/, '')}-#{digest}"
341
438
  end
342
439
 
343
440
  ##
@@ -442,12 +539,128 @@ module Beaker
442
539
  find_ssh_key_pair[:public_key]
443
540
  end
444
541
 
542
+ ##
543
+ # An externally supplied test-group identifier, from
544
+ # BEAKER_KUBEVIRT_TEST_GROUP.
545
+ #
546
+ # A caller that outlives the beaker process needs to be able to find this
547
+ # run's resources. CI is the motivating case: when a job is cancelled, the
548
+ # beaker process can be killed before it tears anything down, and a random
549
+ # per-process identifier leaves the job no way to name what it leaked.
550
+ # Deleting by `beaker/test-group=<the identifier it chose>` cleans up that
551
+ # run without touching other runs sharing the namespace.
552
+ #
553
+ # @return [String, nil] the sanitized identifier, or nil when unset or blank
554
+ def configured_test_group_identifier
555
+ raw = ENV.fetch('BEAKER_KUBEVIRT_TEST_GROUP', '')
556
+ return nil if raw.strip.empty?
557
+
558
+ # Truncating would cut off the tail, which is usually the part that tells
559
+ # runs apart (job ID, matrix index), and give concurrent runs one label.
560
+ if raw.strip.length > LABEL_VALUE_MAX
561
+ raise ArgumentError, "BEAKER_KUBEVIRT_TEST_GROUP must be at most #{LABEL_VALUE_MAX} characters, " \
562
+ "got #{raw.strip.length}: #{raw.inspect}"
563
+ end
564
+
565
+ identifier = sanitize_k8s_label_value(raw)
566
+ if identifier != raw
567
+ @logger.warn(
568
+ "BEAKER_KUBEVIRT_TEST_GROUP #{raw.inspect} is not usable as a Kubernetes " \
569
+ "label value; using #{identifier.inspect} instead",
570
+ )
571
+ end
572
+ identifier
573
+ end
574
+
445
575
  def get_labels(host)
446
576
  raw_name = host.respond_to?(:name) ? host.name : host['name']
447
- {
577
+ user_labels(host).merge(
448
578
  'beaker/test-group' => @test_group_identifier,
449
579
  'beaker/host' => sanitize_k8s_label_value(raw_name),
450
- }
580
+ )
581
+ end
582
+
583
+ ##
584
+ # Extra labels for every resource that carries beaker's own labels, from
585
+ # BEAKER_kubevirt_labels, the host or the global CONFIG, in that order.
586
+ # Cleanup finds resources by beaker/test-group, so the keys beaker sets
587
+ # itself are refused rather than silently overridden.
588
+ # @param [Host] host The host configuration
589
+ # @return [Hash{String=>String}] The labels, empty when unset
590
+ def user_labels(host)
591
+ labels = key_value_option(host, 'kubevirt_labels')
592
+ reserved = labels.keys & RESERVED_LABEL_KEYS
593
+ raise ArgumentError, "kubevirt_labels may not set #{reserved.join(', ')}; beaker sets those itself" unless reserved.empty?
594
+
595
+ labels.each do |key, value|
596
+ next if value.empty? || (value.length <= LABEL_VALUE_MAX && LABEL_NAME_RE.match?(value))
597
+
598
+ raise ArgumentError, "kubevirt_labels value for #{key.inspect} must be at most #{LABEL_VALUE_MAX} characters of " \
599
+ "letters, digits, '-', '_' or '.', starting and ending with a letter or digit, got #{value.inspect}"
600
+ end
601
+ labels
602
+ end
603
+
604
+ ##
605
+ # Extra annotations for the VM template, and so for the VMI and its
606
+ # virt-launcher pod, from BEAKER_kubevirt_annotations, the host or the
607
+ # global CONFIG, in that order.
608
+ # @param [Host] host The host configuration
609
+ # @return [Hash{String=>String}, nil] The annotations, or nil when unset
610
+ def user_annotations(host)
611
+ annotations = key_value_option(host, 'kubevirt_annotations')
612
+ annotations unless annotations.empty?
613
+ end
614
+
615
+ ##
616
+ # A hash option that BEAKER_<name> may replace with a comma-separated list
617
+ # of key=value pairs. Only the first = in a pair separates key from value.
618
+ # @param [Host] host The host configuration
619
+ # @param [String] name The option name
620
+ # @return [Hash{String=>String}] The option, empty when unset
621
+ def key_value_option(host, name)
622
+ env = env_option(name)
623
+ value = env ? parse_key_values(name, env) : host[name] || @options[name.to_sym]
624
+ return {} if value.nil?
625
+ raise ArgumentError, "#{name} must be a hash of keys to values, got #{value.inspect}" unless value.is_a?(Hash)
626
+
627
+ value.to_h do |key, val|
628
+ key = key.to_s
629
+ unless qualified_key?(key)
630
+ raise ArgumentError, "#{name} key #{key.inspect} is not a valid Kubernetes key: expected an optional DNS subdomain " \
631
+ "prefix and '/', then a name of at most #{LABEL_VALUE_MAX} characters of letters, digits, " \
632
+ "'-', '_' or '.', starting and ending with a letter or digit"
633
+ end
634
+
635
+ raise ArgumentError, "#{name} value for #{key.inspect} must be a single value, got #{val.inspect}" if val.is_a?(Hash) || val.is_a?(Array)
636
+
637
+ [key, val.to_s]
638
+ end
639
+ end
640
+
641
+ ##
642
+ # Whether a string is a valid label or annotation key: an optional
643
+ # DNS-subdomain prefix and '/', then a name. The API server rejects
644
+ # anything else with a 422, deep into provisioning.
645
+ # @param [String] key
646
+ # @return [Boolean]
647
+ def qualified_key?(key)
648
+ prefix, slash, name = key.rpartition('/')
649
+ return false unless name.length <= LABEL_VALUE_MAX && LABEL_NAME_RE.match?(name)
650
+
651
+ slash.empty? || (prefix.length <= KEY_PREFIX_MAX && KEY_PREFIX_RE.match?(prefix))
652
+ end
653
+
654
+ def parse_key_values(name, value)
655
+ pairs = value.split(',').map(&:strip).reject(&:empty?)
656
+ raise ArgumentError, "BEAKER_#{name} is set but has no key=value entries: #{value.inspect}" if pairs.empty?
657
+
658
+ pairs.to_h do |pair|
659
+ key, val = pair.split('=', 2).map(&:strip)
660
+ raise ArgumentError, "BEAKER_#{name} entries must be key=value, got #{pair.inspect}" if val.nil? || key.empty?
661
+
662
+ [key, val]
663
+ end
451
664
  end
452
665
 
453
666
  ##
@@ -483,19 +696,146 @@ module Beaker
483
696
  end
484
697
  end
485
698
 
699
+ ##
700
+ # The value of the BEAKER_<name> environment variable, which overrides the
701
+ # host and CONFIG settings of the option it names so that a CI environment
702
+ # can set it for every nodeset it runs, as beaker-google's BEAKER_gce_*
703
+ # variables do.
704
+ # @param [String] name The option name, e.g. 'kubevirt_disk_cache'
705
+ # @return [String, nil] The stripped value, or nil when unset or blank
706
+ def env_option(name)
707
+ value = ENV.fetch("BEAKER_#{name}", nil)&.strip
708
+ value unless value.nil? || value.empty?
709
+ end
710
+
711
+ ##
712
+ # Cache mode for the root disk, from BEAKER_kubevirt_disk_cache, the host
713
+ # or the global CONFIG, in that order.
714
+ # Unset leaves the choice to KubeVirt, which uses none when the backing
715
+ # store supports O_DIRECT and writethrough otherwise. writeback lets the
716
+ # virt-launcher's page cache absorb guest writes until the guest flushes,
717
+ # which suits throwaway test VMs on slow or shared storage.
718
+ # @param [Host] host The host configuration
719
+ # @return [String, nil] One of DISK_CACHE_MODES, or nil when unset
720
+ def disk_cache(host)
721
+ env = env_option('kubevirt_disk_cache')
722
+ value = env || host['kubevirt_disk_cache'] || @options[:kubevirt_disk_cache]
723
+ return nil if value.nil? || value.to_s.empty?
724
+
725
+ cache = value.to_s.downcase
726
+ unless DISK_CACHE_MODES.include?(cache)
727
+ source = env ? 'BEAKER_kubevirt_disk_cache' : 'kubevirt_disk_cache'
728
+ raise ArgumentError, "#{source} must be one of #{DISK_CACHE_MODES.join(', ')}, got #{value.to_s.inspect}"
729
+ end
730
+
731
+ cache
732
+ end
733
+
734
+ ##
735
+ # Node labels the VM's pod must match, from BEAKER_kubevirt_node_selector,
736
+ # the host or the global CONFIG, in that order. The variable is a
737
+ # comma-separated list of key=value pairs.
738
+ # @param [Host] host The host configuration
739
+ # @return [Hash{String=>String}, nil] The selector, or nil when unset
740
+ def node_selector(host)
741
+ env = env_option('kubevirt_node_selector')
742
+ selector = env ? parse_key_values('kubevirt_node_selector', env) : host['kubevirt_node_selector'] || @options[:kubevirt_node_selector]
743
+ return nil if selector.nil?
744
+ raise ArgumentError, "kubevirt_node_selector must be a hash of label keys to values, got #{selector.inspect}" unless selector.is_a?(Hash)
745
+ return nil if selector.empty?
746
+
747
+ selector.to_h do |key, value|
748
+ raise ArgumentError, "kubevirt_node_selector value for #{key.to_s.inspect} must be a single label value, got #{value.inspect}" unless scalar?(value)
749
+
750
+ [key.to_s, value.to_s]
751
+ end
752
+ end
753
+
754
+ def scalar?(value)
755
+ case value
756
+ # Not Float: YAML reads an unquoted 1.10 as 1.1, which would select the
757
+ # wrong nodes without an error. A quoted value is unambiguous.
758
+ when String, Symbol, Integer, true, false then true
759
+ else false
760
+ end
761
+ end
762
+
763
+ ##
764
+ # Tolerations for the VM's pod, from BEAKER_kubevirt_tolerations, the host
765
+ # or the global CONFIG, in that order. Each entry has a key, an effect and
766
+ # an optional value; one without a value tolerates the key whatever its
767
+ # value. The variable is a comma-separated list of key=value:Effect or
768
+ # key:Effect entries.
769
+ # @param [Host] host The host configuration
770
+ # @return [Array<Hash>, nil] Kubernetes tolerations, or nil when unset
771
+ def tolerations(host)
772
+ env = env_option('kubevirt_tolerations')
773
+ list = env ? parse_tolerations(env) : host['kubevirt_tolerations'] || @options[:kubevirt_tolerations]
774
+ return nil if list.nil?
775
+ raise ArgumentError, "kubevirt_tolerations must be a list of key/value/effect entries, got #{list.inspect}" unless list.is_a?(Array)
776
+ return nil if list.empty?
777
+
778
+ list.map { |entry| build_toleration(entry) }
779
+ end
780
+
781
+ def parse_tolerations(value)
782
+ entries = value.split(',').map(&:strip).reject(&:empty?)
783
+ raise ArgumentError, "BEAKER_kubevirt_tolerations is set but has no entries: #{value.inspect}" if entries.empty?
784
+
785
+ entries.map do |entry|
786
+ taint, separator, effect = entry.rpartition(':')
787
+ raise ArgumentError, "BEAKER_kubevirt_tolerations entries must be key=value:Effect or key:Effect, got #{entry.inspect}" if separator.empty?
788
+
789
+ key, toleration_value = taint.split('=', 2)
790
+ { 'key' => key, 'value' => toleration_value, 'effect' => effect }
791
+ end
792
+ end
793
+
794
+ def build_toleration(entry)
795
+ raise ArgumentError, "Each kubevirt_tolerations entry must be a hash, got #{entry.inspect}" unless entry.is_a?(Hash)
796
+
797
+ # operator and tolerationSeconds are not supported. Dropping them silently
798
+ # would change what the toleration matches or how long it lasts.
799
+ unknown = entry.keys.map(&:to_s) - TOLERATION_FIELDS
800
+ unless unknown.empty?
801
+ raise ArgumentError, "kubevirt_tolerations entry #{entry.inspect} has unsupported fields #{unknown.join(', ')}; " \
802
+ "only #{TOLERATION_FIELDS.join(', ')} are accepted (an entry without a value tolerates any value)"
803
+ end
804
+
805
+ # key? rather than ||, so a YAML `value: false` is the string "false" and
806
+ # not a missing value, which would widen the toleration to Exists.
807
+ key, value, effect = %w[key value effect].map { |field| entry.key?(field) ? entry[field] : entry[field.to_sym] }
808
+ key = key.to_s.strip
809
+ effect = effect.to_s.strip
810
+ raise ArgumentError, "kubevirt_tolerations entry #{entry.inspect} has no key" if key.empty?
811
+ raise ArgumentError, "kubevirt_tolerations effect must be one of #{TOLERATION_EFFECTS.join(', ')}, got #{effect.inspect}" unless TOLERATION_EFFECTS.include?(effect)
812
+
813
+ # Only an absent value means Exists. An explicit empty string is Equal "",
814
+ # which matches only taints whose value is empty.
815
+ if value.nil?
816
+ { 'key' => key, 'operator' => 'Exists', 'effect' => effect }
817
+ else
818
+ { 'key' => key, 'operator' => 'Equal', 'value' => value.to_s.strip, 'effect' => effect }
819
+ end
820
+ end
821
+
486
822
  ##
487
823
  # Generate the hardware devices specification for the VM
488
824
  # @param [Host] host The host configuration
489
825
  # @return [Hash] Hardware devices specification
490
826
  def generate_hardware_spec(host)
827
+ rootdisk = {
828
+ 'name' => 'rootdisk',
829
+ 'disk' => {
830
+ 'bus' => disk_bus(host),
831
+ },
832
+ }
833
+ cache = disk_cache(host)
834
+ rootdisk['cache'] = cache if cache
835
+
491
836
  {
492
837
  'disks' => [
493
- {
494
- 'name' => 'rootdisk',
495
- 'disk' => {
496
- 'bus' => disk_bus(host),
497
- },
498
- },
838
+ rootdisk,
499
839
  {
500
840
  'name' => 'cidata',
501
841
  'disk' => {
@@ -536,6 +876,15 @@ module Beaker
536
876
  memory_request = "#{memory_request}Mi" if /\A\d+\z/.match?(memory_request.to_s)
537
877
  memory_limit = "#{parse_memory_mib(memory) + parse_memory_mib(overhead)}Mi"
538
878
 
879
+ # Both unset by default: KubeVirt then derives the pod's CPU request from the guest core
880
+ # count divided by the cluster's cpuAllocationRatio, and sets no limit at all.
881
+ cpu_request = host['kubevirt_cpu_request'] || @options[:kubevirt_cpu_request]
882
+ cpu_limit = host['kubevirt_cpu_limit'] || @options[:kubevirt_cpu_limit]
883
+ resource_requests = { 'memory' => memory_request }
884
+ resource_requests['cpu'] = cpu_request.to_s if cpu_request
885
+ resource_limits = { 'memory' => memory_limit }
886
+ resource_limits['cpu'] = cpu_limit.to_s if cpu_limit
887
+
539
888
  vm_image = host['kubevirt_vm_image'] || @options[:kubevirt_vm_image]
540
889
  host_name = host.respond_to?(:name) ? host.name : host['name']
541
890
 
@@ -561,7 +910,8 @@ module Beaker
561
910
  'labels' => get_labels(host).merge({
562
911
  'kubevirt.io/vm' => vm_name,
563
912
  }),
564
- },
913
+ 'annotations' => user_annotations(host),
914
+ }.compact,
565
915
  'spec' => {
566
916
  'domain' => {
567
917
  'cpu' => {
@@ -573,8 +923,8 @@ module Beaker
573
923
  'guest' => memory.to_s,
574
924
  },
575
925
  'resources' => {
576
- 'requests' => { 'memory' => memory_request },
577
- 'limits' => { 'memory' => memory_limit },
926
+ 'requests' => resource_requests,
927
+ 'limits' => resource_limits,
578
928
  },
579
929
  'devices' => generate_hardware_spec(host),
580
930
  'features' => {
@@ -592,6 +942,8 @@ module Beaker
592
942
  },
593
943
  },
594
944
  'hostname' => host_name,
945
+ 'nodeSelector' => node_selector(host),
946
+ 'tolerations' => tolerations(host),
595
947
  'networks' => generate_networks_spec(host),
596
948
  'readinessProbe' => generate_readiness_probe_spec(host),
597
949
  'volumes' => [
@@ -637,6 +989,18 @@ module Beaker
637
989
  end
638
990
  end
639
991
 
992
+ ##
993
+ # Storage class for the root disk's DataVolume, from
994
+ # BEAKER_kubevirt_storage_class, the host or the global CONFIG, in that
995
+ # order. Unset leaves the choice to the cluster's default class. A
996
+ # containerDisk root has no DataVolume, so it is unaffected.
997
+ # @param [Host] host The host configuration
998
+ # @return [String, nil] The storage class name, or nil when unset
999
+ def storage_class(host)
1000
+ name = (env_option('kubevirt_storage_class') || host['kubevirt_storage_class'] || @options[:kubevirt_storage_class]).to_s.strip
1001
+ name unless name.empty?
1002
+ end
1003
+
640
1004
  ##
641
1005
  # Generate a DataVolume Template for the root disk
642
1006
  # @param [String] vm_image The VM image specification
@@ -660,6 +1024,9 @@ module Beaker
660
1024
  },
661
1025
  }
662
1026
 
1027
+ storage_class_name = storage_class(host)
1028
+ dv_spec['spec']['storage']['storageClassName'] = storage_class_name if storage_class_name
1029
+
663
1030
  # If a custom service account is specified, add it to the DataVolume spec
664
1031
  # so it can access the source PVC when required (including cross-namespace clones)
665
1032
  dv_spec['spec']['serviceAccountName'] = @service_account if @service_account
@@ -779,7 +1146,7 @@ module Beaker
779
1146
 
780
1147
  raise "VMI #{vm_name} entered terminal phase #{phase} before becoming Ready" if VMI_TERMINAL_PHASES.include?(phase)
781
1148
 
782
- check_virt_launcher_health!(vm_name)
1149
+ check_virt_launcher_health!(host)
783
1150
 
784
1151
  sleep SLEEPWAIT
785
1152
  end
@@ -831,24 +1198,47 @@ module Beaker
831
1198
 
832
1199
  ##
833
1200
  # Inspect the virt-launcher pod backing a VMI and raise with a specific reason
834
- # if it has already failed (OOMKilled, image pull errors, crash loops, etc.)
835
- # so we fail fast instead of waiting the full timeout.
836
- # @param [String] vm_name
837
- def check_virt_launcher_health!(vm_name)
1201
+ # if it has already failed (OOMKilled, crash loops, a container that cannot run)
1202
+ # or if the scheduler cannot place it, so we fail fast instead of waiting the
1203
+ # full timeout. Image pull backoff is handled separately by
1204
+ # check_image_pull_backoff!, since kubelet retries pulls and an early
1205
+ # observation does not mean provisioning has failed.
1206
+ # @param [Host] host The host being provisioned, carrying 'vm_name'
1207
+ def check_virt_launcher_health!(host)
1208
+ vm_name = host['vm_name']
838
1209
  pod = @kubevirt_helper.get_virt_launcher_pod(vm_name)
839
1210
  return unless pod
840
1211
 
1212
+ # Image-pull backoff is tracked per virt-launcher pod rather than per VMI.
1213
+ # A pod replaced mid-provision -- rescheduled, or evicted while pulling --
1214
+ # starts the pull again on a node whose cache may be cold, and must not
1215
+ # inherit the window its predecessor already spent. Keying on the pod also
1216
+ # means a momentary absence (API lag, or a `pods.first` that briefly sees
1217
+ # the successor) neither resets nor shortens the window a live pod is
1218
+ # accumulating.
1219
+ pod_name = pod.dig('metadata', 'name') || vm_name
1220
+
841
1221
  phase = pod.dig('status', 'phase')
842
1222
  raise "virt-launcher pod for #{vm_name} entered phase #{phase}" if phase == 'Failed'
843
1223
 
1224
+ check_pod_schedulable!(host, vm_name, pod)
1225
+
844
1226
  statuses = Array(pod.dig('status', 'containerStatuses')) +
845
1227
  Array(pod.dig('status', 'initContainerStatuses'))
846
1228
  statuses.each do |cs|
847
1229
  name = cs['name']
848
1230
  waiting_reason = cs.dig('state', 'waiting', 'reason')
849
- if FATAL_CONTAINER_WAITING_REASONS.include?(waiting_reason)
850
- raise "virt-launcher container #{name} for #{vm_name} is #{waiting_reason}: " \
851
- "#{cs.dig('state', 'waiting', 'message')}"
1231
+ if IMAGE_PULL_WAITING_REASONS.include?(waiting_reason)
1232
+ check_image_pull_backoff!(host, pod_name, name, waiting_reason, cs.dig('state', 'waiting', 'message').to_s)
1233
+ else
1234
+ # Only a container that has got past the pull ends its window. A
1235
+ # transient other waiting state between two backoffs must not restart it.
1236
+ @image_pull_backoff_since.delete("#{pod_name}/#{name}") if cs.dig('state', 'running') || cs.dig('state', 'terminated')
1237
+
1238
+ if FATAL_CONTAINER_WAITING_REASONS.include?(waiting_reason)
1239
+ raise "virt-launcher container #{name} for #{vm_name} is #{waiting_reason}: " \
1240
+ "#{cs.dig('state', 'waiting', 'message')}"
1241
+ end
852
1242
  end
853
1243
 
854
1244
  last_term = cs.dig('lastState', 'terminated') || cs.dig('state', 'terminated')
@@ -860,6 +1250,137 @@ module Beaker
860
1250
  end
861
1251
  end
862
1252
 
1253
+ ##
1254
+ # Decide whether a container backing off an image pull has waited long enough
1255
+ # to be called a failure. kubelet retries pulls with increasing backoff, so a
1256
+ # single observation of ImagePullBackOff proves nothing: an on-demand
1257
+ # pull-through cache warming a multi-GB image, a slow upstream registry or a
1258
+ # dropped connection all resolve on a later attempt. Raise only once the
1259
+ # backoff has outlived the grace window — or immediately when the message says
1260
+ # retrying is pointless, or when the window has been switched off.
1261
+ # @param [Host] host The host being provisioned, carrying 'vm_name'
1262
+ # @param [String] pod_name Name of the virt-launcher pod reporting the backoff
1263
+ # @param [String] container Container name reporting the backoff
1264
+ # @param [String] reason Waiting reason (ImagePullBackOff / ErrImagePull)
1265
+ # @param [String] message Waiting message from kubelet
1266
+ def check_image_pull_backoff!(host, pod_name, container, reason, message)
1267
+ vm_name = host['vm_name']
1268
+ grace = image_pull_grace(host)
1269
+
1270
+ # Two ways to skip the window and fail on this very observation: kubelet's
1271
+ # retry will never fix a definitive failure, and a grace of zero is the
1272
+ # documented way to opt out of the window entirely. Both keep the message
1273
+ # the check raised before the window existed.
1274
+ raise "virt-launcher container #{container} for #{vm_name} is #{reason}: #{message}" if DEFINITIVE_IMAGE_PULL_FAILURE_RE.match?(message) || grace <= 0
1275
+
1276
+ key = "#{pod_name}/#{container}"
1277
+ now = monotonic_time
1278
+ first_seen = !@image_pull_backoff_since.key?(key)
1279
+ @image_pull_backoff_since[key] ||= now
1280
+ elapsed = (now - @image_pull_backoff_since[key]).round
1281
+
1282
+ if elapsed > grace
1283
+ raise "virt-launcher container #{container} for #{vm_name} is still #{reason} after #{elapsed}s, " \
1284
+ "exceeding the :kubevirt_image_pull_grace window of #{grace}s: #{message}"
1285
+ end
1286
+
1287
+ # Warn on first sight: when the ready timeout expires before the window
1288
+ # does, this is the only record of why the VM never became ready.
1289
+ if first_seen
1290
+ @logger.warn("virt-launcher container #{container} for #{vm_name} is #{reason}, allowing #{grace}s " \
1291
+ "for kubelet to retry the pull: #{message}")
1292
+ else
1293
+ @logger.debug("virt-launcher container #{container} for #{vm_name} is #{reason} " \
1294
+ "(#{elapsed}s of #{grace}s grace); waiting for kubelet to retry the pull")
1295
+ end
1296
+ end
1297
+
1298
+ ##
1299
+ # How long an image pull may stay in backoff before it is treated as fatal.
1300
+ # Per-host first so a suite can give a multi-GB Windows containerDisk more room
1301
+ # than a small cloud image, then the global option, then the default. Zero
1302
+ # switches the window off rather than setting an empty one.
1303
+ # @param [Host] host The host being provisioned
1304
+ # @return [Integer] seconds
1305
+ def image_pull_grace(host)
1306
+ value = host['kubevirt_image_pull_grace'] || @options[:kubevirt_image_pull_grace]
1307
+ return DEFAULT_IMAGE_PULL_GRACE if value.nil?
1308
+
1309
+ grace = Integer(value, exception: false)
1310
+ raise ArgumentError, "kubevirt_image_pull_grace must be a whole number of seconds, got #{value.inspect}" if grace.nil? || grace.negative?
1311
+
1312
+ grace
1313
+ end
1314
+
1315
+ ##
1316
+ # Raise once the scheduler has been unable to place the virt-launcher pod for
1317
+ # longer than the grace window.
1318
+ #
1319
+ # An unschedulable pod is invisible to every other check in
1320
+ # check_virt_launcher_health!: its phase is Pending rather than Failed, and both
1321
+ # containerStatuses and initContainerStatuses are absent entirely, because no
1322
+ # container was ever assigned to a node. So the wait loop spins until the overall
1323
+ # timeout and reports "Timeout waiting for VM ... to be ready" — the same message
1324
+ # a slow guest boot produces. The one failure the cluster can describe precisely
1325
+ # ("0/8 nodes are available: 5 Insufficient cpu") is the one that arrives with no
1326
+ # explanation at all.
1327
+ #
1328
+ # Being unschedulable is not immediately terminal, which is why the grace window
1329
+ # exists rather than raising on first sight. The reason is logged as soon as it is
1330
+ # seen, so it is never silent even when the window is set wide.
1331
+ #
1332
+ # @param [Host] host The host being provisioned
1333
+ # @param [String] vm_name
1334
+ # @param [Hash] pod The virt-launcher pod
1335
+ def check_pod_schedulable!(host, vm_name, pod)
1336
+ condition = Array(pod.dig('status', 'conditions')).find { |c| c['type'] == 'PodScheduled' }
1337
+ reason = (condition && condition['status'] == 'False') ? condition['reason'] : nil
1338
+
1339
+ unless UNSCHEDULABLE_POD_REASONS.include?(reason)
1340
+ @unschedulable_since.delete(vm_name)
1341
+ return
1342
+ end
1343
+
1344
+ message = condition['message'].to_s
1345
+ grace = unschedulable_grace(host)
1346
+ uid = pod.dig('metadata', 'uid')
1347
+ tracked = @unschedulable_since[vm_name]
1348
+ # A replacement pod must get its own window, not inherit the elapsed time of
1349
+ # the one it replaced.
1350
+ if tracked.nil? || tracked[:uid] != uid
1351
+ tracked = @unschedulable_since[vm_name] = { uid: uid, since: monotonic_time }
1352
+ @logger.warn("virt-launcher pod for #{vm_name} is #{reason}, allowing #{grace}s " \
1353
+ "for capacity: #{message}")
1354
+ end
1355
+
1356
+ waited = monotonic_time - tracked[:since]
1357
+ return if waited < grace
1358
+
1359
+ raise "virt-launcher pod for #{vm_name} is still #{reason} after #{waited.round}s: #{message}"
1360
+ end
1361
+
1362
+ ##
1363
+ # Seconds the virt-launcher pod may remain unschedulable before it is fatal.
1364
+ # @param [Host] host
1365
+ # @return [Integer]
1366
+ def unschedulable_grace(host)
1367
+ value = host['kubevirt_unschedulable_grace'] || @options[:kubevirt_unschedulable_grace]
1368
+ return DEFAULT_UNSCHEDULABLE_GRACE if value.nil?
1369
+
1370
+ grace = Integer(value, exception: false)
1371
+ raise ArgumentError, "kubevirt_unschedulable_grace must be a whole number of seconds, got #{value.inspect}" if grace.nil? || grace.negative?
1372
+
1373
+ grace
1374
+ end
1375
+
1376
+ ##
1377
+ # Monotonic seconds, so a wall-clock adjustment mid-wait cannot shorten or
1378
+ # extend a grace window.
1379
+ # @return [Float]
1380
+ def monotonic_time
1381
+ Process.clock_gettime(Process::CLOCK_MONOTONIC)
1382
+ end
1383
+
863
1384
  ##
864
1385
  # The guest-side port SSH listens on, shared between the VMI readiness probe
865
1386
  # and client-side networking setup so they can never drift apart.
@@ -964,7 +1485,7 @@ module Beaker
964
1485
  # @param [Host] host The host
965
1486
  def setup_nodeport(host)
966
1487
  vm_name = host['vm_name']
967
- service_name = "#{vm_name}-ssh"
1488
+ service_name = bounded_k8s_name("#{vm_name}-ssh")
968
1489
 
969
1490
  @logger.debug("Creating NodePort service for VM #{vm_name}")
970
1491
  service = @kubevirt_helper.create_nodeport_service(vm_name, service_name, @test_group_identifier)
@@ -547,7 +547,7 @@ class KubeVirtPortForwarder
547
547
  end
548
548
 
549
549
  # Manages state transitions with thread safety.
550
- # rubocop:disable Naming/PredicateMethod
550
+ # rubocop:disable-next Naming/PredicateMethod
551
551
  def state_transition_to(new_state)
552
552
  @mutex.synchronize do
553
553
  case new_state
@@ -564,7 +564,6 @@ class KubeVirtPortForwarder
564
564
  end
565
565
  true
566
566
  end
567
- # rubocop:enable Naming/PredicateMethod
568
567
 
569
568
  # Convert kubeclient SSL options to Faye::WebSocket/EventMachine TLS options.
570
569
  # @param ssl_options [Hash] The SSL options from kubeclient
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module BeakerKubevirt
4
- VERSION = '1.0.0'
4
+ VERSION = '1.1.0'
5
5
  end
@@ -6,6 +6,6 @@ require 'beaker/hypervisor/kubevirt'
6
6
  describe Beaker::Hypervisor::Kubevirt do
7
7
  it 'can run a command' do
8
8
  result = on default, 'echo hello'
9
- expect(result.stdout).to match(/hello/)
9
+ expect(result.stdout).to include('hello')
10
10
  end
11
11
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: beaker-kubevirt
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Steven Pritchard
@@ -126,14 +126,14 @@ dependencies:
126
126
  requirements:
127
127
  - - "~>"
128
128
  - !ruby/object:Gem::Version
129
- version: 5.2.0
129
+ version: 5.3.0
130
130
  type: :development
131
131
  prerelease: false
132
132
  version_requirements: !ruby/object:Gem::Requirement
133
133
  requirements:
134
134
  - - "~>"
135
135
  - !ruby/object:Gem::Version
136
- version: 5.2.0
136
+ version: 5.3.0
137
137
  description: Enables Beaker to provision VMs using KubeVirt for automated acceptance
138
138
  testing of Puppet code.
139
139
  email:
@@ -182,7 +182,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
182
182
  - !ruby/object:Gem::Version
183
183
  version: '0'
184
184
  requirements: []
185
- rubygems_version: 4.0.16
185
+ rubygems_version: 4.0.20
186
186
  specification_version: 4
187
187
  summary: KubeVirt hypervisor support for Beaker acceptance testing
188
188
  test_files: []