beaker-kubevirt 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +24 -0
- data/README.md +35 -0
- data/examples/hosts.yaml +5 -0
- data/lib/beaker/hypervisor/kubevirt.rb +547 -26
- data/lib/beaker/hypervisor/port_forward.rb +1 -2
- data/lib/beaker/kubevirt/version.rb +1 -1
- data/tests/spec/beaker/hypervisor/kubevirt_spec.rb +1 -1
- metadata +4 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5b2973046f9f272d05b8397362b0cacd9fdb2a7aec80be0e9459d36bce83ee4c
|
|
4
|
+
data.tar.gz: a6b64fd54c8276ceb60a6904f636e9ff7a380849b652a3c7e51ab5f28848e4f4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 75a0558dd8ad93cc7ef7c70ed129d05754a3998165768f0b94da4046cdd33e3e8d9a20617982c89095023225136afbcf54d1a09014d4957b91a992ca31d22f0e
|
|
7
|
+
data.tar.gz: b8716d9af99b6d13f86f2ab1df4bca2c6894392a6a2572863fbc5735578d426f1cb212ca9fcd1d81e29e1161ea58e6feeeda281a8b11bf8dc7b91c83b8a528c7
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,30 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [1.0.0](https://github.com/voxpupuli/beaker-kubevirt/tree/1.0.0) (2026-11-05)
|
|
6
|
+
|
|
7
|
+
[Full Changelog](https://github.com/voxpupuli/beaker-kubevirt/compare/1.0.0...1.0.0)
|
|
8
|
+
|
|
9
|
+
**Closed issues:**
|
|
10
|
+
|
|
11
|
+
- Allow the test-group identifier to be supplied via the environment [\#63](https://github.com/voxpupuli/beaker-kubevirt/issues/63)
|
|
12
|
+
- ImagePullBackOff treated as fatal aborts provisioning that would otherwise succeed [\#54](https://github.com/voxpupuli/beaker-kubevirt/issues/54)
|
|
13
|
+
|
|
14
|
+
**Merged pull requests:**
|
|
15
|
+
|
|
16
|
+
- Update voxpupuli-rubocop to ~\> 5.3.0 and fix new offenses [\#86](https://github.com/voxpupuli/beaker-kubevirt/pull/86) ([silug](https://github.com/silug))
|
|
17
|
+
- Add kubevirt\_labels and kubevirt\_annotations [\#85](https://github.com/voxpupuli/beaker-kubevirt/pull/85) ([jaevans](https://github.com/jaevans))
|
|
18
|
+
- Add kubevirt\_storage\_class for the root disk [\#84](https://github.com/voxpupuli/beaker-kubevirt/pull/84) ([jaevans](https://github.com/jaevans))
|
|
19
|
+
- Add kubevirt\_node\_selector and kubevirt\_tolerations [\#83](https://github.com/voxpupuli/beaker-kubevirt/pull/83) ([jaevans](https://github.com/jaevans))
|
|
20
|
+
- Honour BEAKER\_kubevirt\_disk\_cache [\#82](https://github.com/voxpupuli/beaker-kubevirt/pull/82) ([jaevans](https://github.com/jaevans))
|
|
21
|
+
- Add kubevirt\_disk\_cache option for the root disk cache mode [\#77](https://github.com/voxpupuli/beaker-kubevirt/pull/77) ([silug](https://github.com/silug))
|
|
22
|
+
- Update voxpupuli-rubocop requirement from ~\> 5.2.0 to ~\> 5.3.0 [\#72](https://github.com/voxpupuli/beaker-kubevirt/pull/72) ([dependabot[bot]](https://github.com/apps/dependabot))
|
|
23
|
+
- Add kubevirt\_cpu\_request and kubevirt\_cpu\_limit [\#66](https://github.com/voxpupuli/beaker-kubevirt/pull/66) ([jaevans](https://github.com/jaevans))
|
|
24
|
+
- Detect an unschedulable virt-launcher pod instead of waiting out the timeout [\#65](https://github.com/voxpupuli/beaker-kubevirt/pull/65) ([jaevans](https://github.com/jaevans))
|
|
25
|
+
- feat: allow the test-group identifier to be supplied via the environment [\#64](https://github.com/voxpupuli/beaker-kubevirt/pull/64) ([jaevans](https://github.com/jaevans))
|
|
26
|
+
- Remove CodeQL workflow in favor of default setup [\#62](https://github.com/voxpupuli/beaker-kubevirt/pull/62) ([silug](https://github.com/silug))
|
|
27
|
+
- Give image pull backoff a grace window before failing [\#55](https://github.com/voxpupuli/beaker-kubevirt/pull/55) ([jaevans](https://github.com/jaevans))
|
|
28
|
+
|
|
5
29
|
## [1.0.0](https://github.com/voxpupuli/beaker-kubevirt/tree/1.0.0) (2026-07-17)
|
|
6
30
|
|
|
7
31
|
[Full Changelog](https://github.com/voxpupuli/beaker-kubevirt/compare/a388e545ac4aa46006e53f006ad5af1a243bdbbd...1.0.0)
|
data/README.md
CHANGED
|
@@ -78,21 +78,33 @@ CONFIG:
|
|
|
78
78
|
| `networks` | Custom network configuration | No | Auto-generated | HOSTS (per-host) |
|
|
79
79
|
| `kubevirt_ssh_key` | SSH public key path or content | Yes | Auto-detect from `~/.ssh/` | HOSTS (per-host) |
|
|
80
80
|
| `kubevirt_cpus` | CPU cores for VM | No | `1` | HOSTS (per-host) |
|
|
81
|
+
| `kubevirt_cpu_request` | CPU request for the VM pod, as a Kubernetes CPU quantity (`2`, `1500m`). Unset leaves KubeVirt to derive it from `kubevirt_cpus` divided by the cluster's `cpuAllocationRatio`. Set it for guests that must not be throttled while booting. | No | unset | HOSTS (per-host), CONFIG |
|
|
82
|
+
| `kubevirt_cpu_limit` | CPU limit for the VM pod, as a Kubernetes CPU quantity. Unset leaves the compute container burstable. | No | unset | HOSTS (per-host), CONFIG |
|
|
81
83
|
| `kubevirt_memory` | Memory for VM | No | `2Gi` | HOSTS (per-host) |
|
|
82
84
|
| `kubevirt_memory_overhead` | Extra memory added to the guest for the virt-launcher container memory limit. Raise this for Windows guests that OOMKill with the default. | No | `512Mi` | HOSTS (per-host), CONFIG |
|
|
83
85
|
| `kubevirt_memory_request` | Memory request for the VM pod. | No | Same as `kubevirt_memory` | HOSTS (per-host), CONFIG |
|
|
84
86
|
| `kubevirt_disk_size` | Size of the root disk | No | `10Gi` | HOSTS (per-host) |
|
|
87
|
+
| `kubevirt_storage_class` | Storage class for the root disk's DataVolume, used when `kubevirt_vm_image` is an `http(s)://` URL or a PVC to clone. Unset uses the cluster's default storage class. A `docker://`/`oci://` containerDisk has no DataVolume and ignores it. Cloning a PVC into a different class may make CDI copy the data instead of using a CSI clone. A non-empty `BEAKER_kubevirt_storage_class` environment variable overrides the host and CONFIG values. | No | cluster default | `BEAKER_kubevirt_storage_class`, HOSTS (per-host), CONFIG |
|
|
85
88
|
| `kubevirt_cloud_init` | Custom cloud-init YAML | No | Auto-generated | HOSTS (per-host) |
|
|
86
89
|
| `kubevirt_vm_ssh_port` | SSH port inside the VM | No | `22` | HOSTS (per-host) |
|
|
87
90
|
| `kubevirt_readiness_probe_disabled` | Skip the VMI SSH readinessProbe. Default is `false` for pod-network modes and `true` for `multus` (where a probe from the virt-launcher netns typically can't reach a bridge-only guest). | No | mode-dependent | HOSTS (per-host), CONFIG |
|
|
88
91
|
| `kubevirt_readiness_probe` | Probe tuning hash (snake_case keys): `initial_delay_seconds` (30), `period_seconds` (10), `timeout_seconds` (3), `failure_threshold` (60), `success_threshold` (1). The default failure budget (600s) accommodates slow Windows first-boots. When the probe is enabled, `timeout` is auto-raised to cover this budget. | No | see description | HOSTS (per-host), CONFIG |
|
|
92
|
+
| `kubevirt_unschedulable_grace` | Seconds the virt-launcher pod may stay unschedulable (`PodScheduled=False`, reason `Unschedulable` or `SchedulerError`) before provisioning gives up and reports the scheduler's own message. Cannot be immediate — an unschedulable pod is precisely what tells a cluster autoscaler to add a node. Bounded by the effective `timeout`, which this does not raise. | No | `300` | HOSTS (per-host), CONFIG |
|
|
93
|
+
| `kubevirt_image_pull_grace` | Seconds a containerDisk pull may sit in `ImagePullBackOff`/`ErrImagePull` before provisioning gives up. kubelet retries pulls, so a cold pull-through cache, a slow upstream registry or a dropped connection normally clears on a later attempt. Bounded by the effective `timeout` (which this does not raise), and skipped when the registry's error is definitive — `manifest unknown`, containerd's `failed to resolve reference …: not found` for a missing tag, `pull access denied`, `requested access to the resource is denied`, `unauthorized: authentication required` — so a typo'd tag or a refused credential fails on the first poll that shows the error. A blob 404 from a cold cache, or a `permission denied` from the network, gets the window. Set it to `0` to switch the window off and fail on the first backoff, as provisioning did before this option existed. The first backoff seen for each container is logged as a warning, so a run that reaches `timeout` before the window ends still records why. | No | `600` | HOSTS (per-host), CONFIG |
|
|
89
94
|
| `kubevirt_disable_virtio` | Disable virtio devices (for Windows compatibility). If set to true the disk bus will be set to `sata` and the network adapter will be model `e1000` | No | `false` | HOSTS (per-host) |
|
|
95
|
+
| `kubevirt_disk_cache` | Cache mode for the root disk: `none`, `writethrough` or `writeback`. Unset leaves the choice to KubeVirt (`none` when the backing store supports O_DIRECT). `writeback` lets the virt-launcher's page cache absorb guest writes until the guest flushes; use it for throwaway test VMs on slow or shared storage, where a lost VM only costs a re-run. A non-empty `BEAKER_kubevirt_disk_cache` environment variable overrides both. | No | unset | `BEAKER_kubevirt_disk_cache`, HOSTS (per-host), CONFIG |
|
|
96
|
+
| `kubevirt_node_selector` | Hash of node labels the VM's pod must match, e.g. `{ 'node-pool.example.com/ci' => 'true' }`. Select a group of nodes by a label they share, not a single node by name. Quote values that look like decimals (`'1.10'`): an unquoted float is refused, since YAML would read `1.10` as `1.1`. `BEAKER_kubevirt_node_selector`, a comma-separated list of `key=value` pairs, replaces the host and CONFIG values when non-empty. | No | unset | `BEAKER_kubevirt_node_selector`, HOSTS (per-host), CONFIG |
|
|
97
|
+
| `kubevirt_tolerations` | List of taints the VM's pod tolerates. Each entry is a hash with `key`, `effect` (`NoSchedule`, `PreferNoSchedule` or `NoExecute`) and an optional `value`; an entry without a value tolerates the key whatever its value. Other fields (`operator`, `tolerationSeconds`) are refused rather than ignored. Tolerating a taint only allows the VM onto those nodes; pair it with `kubevirt_node_selector` to keep it on them. `BEAKER_kubevirt_tolerations`, a comma-separated list of `key=value:Effect` or `key:Effect` entries, replaces the host and CONFIG values when non-empty. | No | unset | `BEAKER_kubevirt_tolerations`, HOSTS (per-host), CONFIG |
|
|
98
|
+
| `kubevirt_labels` | Hash of extra labels for every resource beaker labels: the VM, its template (and so the VMI and virt-launcher pod), the root DataVolume and the cloud-init Secret. `beaker/test-group`, `beaker/host` and `kubevirt.io/vm` are beaker's own and raise an error if set here, as does a key or value that is not valid Kubernetes label syntax (values are at most 63 characters of letters, digits, `-`, `_` and `.`). `BEAKER_kubevirt_labels`, a comma-separated list of `key=value` pairs, replaces the host and CONFIG values when non-empty. A host's hash replaces the CONFIG hash rather than merging with it. | No | unset | `BEAKER_kubevirt_labels`, HOSTS (per-host), CONFIG |
|
|
99
|
+
| `kubevirt_annotations` | Hash of annotations for the VM template only (and so the VMI and virt-launcher pod), e.g. `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"`. Keys must be valid Kubernetes keys; values are free text. `BEAKER_kubevirt_annotations`, a comma-separated list of `key=value` pairs, replaces the host and CONFIG values when non-empty. A host's hash replaces the CONFIG hash rather than merging with it; values given that way cannot contain commas, so set complex values in the hosts file. | No | unset | `BEAKER_kubevirt_annotations`, HOSTS (per-host), CONFIG |
|
|
90
100
|
|
|
91
101
|
**Important**: The `namespace`, `kubeconfig`, and `kubecontext` options must be specified in the global `CONFIG` section, not per-host. All VMs will be created in the same Kubernetes namespace.
|
|
92
102
|
|
|
93
103
|
Notes:
|
|
94
104
|
- Several per-host options support global fallbacks via `CONFIG` (e.g., `kubevirt_cpus`, `kubevirt_memory`, `kubevirt_vm_ssh_port`).
|
|
95
105
|
- The `networks` key for Multus is intentionally unprefixed (use `networks`, not `kubevirt_networks`).
|
|
106
|
+
- `kubevirt_cpus` sets the guest's core count, which is not the same thing as the pod's CPU request. Left to itself, KubeVirt divides the core count by the cluster's `cpuAllocationRatio` — 10 by default — so a 2-core guest lands in a burstable pod requesting `200m`. On a busy node that guest is throttled to roughly a tenth of what it advertises, and a slow first boot can outlast the readiness probe's failure budget while nothing anywhere reports an error. Set `kubevirt_cpu_request` when boot time matters; Windows guests are the usual case.
|
|
107
|
+
- A virt-launcher pod the scheduler cannot place used to be invisible: its phase is `Pending`, not `Failed`, and it carries no container statuses at all, so provisioning waited out the full `timeout` and reported `Timeout waiting for VM ... to be ready` — indistinguishable from a slow guest boot. It now reports the scheduler's own message (`0/8 nodes are available: 5 Insufficient cpu`) as soon as the state is seen, and fails after `kubevirt_unschedulable_grace`. Raise `kubevirt_memory_request` or a CPU request beyond what a node can offer and this is the error you will get.
|
|
96
108
|
- Long-running commands aren't killed by idle-timeout drops, even when output streams in only one direction (e.g., a Windows scanner running for many minutes). The `port-forward` proxy no longer enforces a client-side read-silence timeout (it relied on net-ssh keepalive packets that net-ssh suppresses while server output is flowing) and instead sends a WebSocket protocol ping every 60s to keep the upstream tunnel alive. net-ssh keepalive defaults are also tightened to `keepalive_interval: 60`, `keepalive_maxcount: 5` — Beaker already enables `keepalive: true` — which protects `multus` and `nodeport` modes against NAT/conntrack timeouts. Override per-host under `ssh:` if needed.
|
|
97
109
|
|
|
98
110
|
### VM Image Formats
|
|
@@ -254,6 +266,29 @@ All resources created are labeled for traceability and cleanup:
|
|
|
254
266
|
|
|
255
267
|
These labels are used during `cleanup` to remove VMs, secrets, and services associated with the test group.
|
|
256
268
|
|
|
269
|
+
### Supplying the test-group identifier
|
|
270
|
+
|
|
271
|
+
By default the test-group identifier is `beaker-<random hex>`, generated per process. Set `BEAKER_KUBEVIRT_TEST_GROUP` to choose it instead:
|
|
272
|
+
|
|
273
|
+
```bash
|
|
274
|
+
BEAKER_KUBEVIRT_TEST_GROUP="beaker-ci-${CI_JOB_ID}" bundle exec rake beaker
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
This matters when something outside the beaker process has to clean up after it. In CI, a cancelled job can be killed before beaker tears anything down, and with a random identifier the job has no way to name the resources it leaked. With a known identifier the job can delete by `beaker/test-group=<its own identifier>` once beaker is gone, without disturbing other runs that share the namespace:
|
|
278
|
+
|
|
279
|
+
```bash
|
|
280
|
+
kubectl -n "$namespace" delete vm,dv,secret,svc -l "beaker/test-group=beaker-ci-${CI_JOB_ID}"
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
The identifier must be unique to each run that can be live at the same time. It names the VMs, DataVolumes and Secrets as well as labelling them, so two runs that share it collide on create, and either run's cleanup deletes the other's resources. `CI_JOB_ID` is unique per job on GitLab. On GitHub Actions, `GITHUB_RUN_ID` is not: every job and matrix leg of a run shares it, and so do re-run attempts. Combine it with the attempt, the job and the matrix index, keeping the result within 63 characters (a longer value is refused rather than truncated, since truncation would cut off the part that tells runs apart):
|
|
284
|
+
|
|
285
|
+
```yaml
|
|
286
|
+
env:
|
|
287
|
+
BEAKER_KUBEVIRT_TEST_GROUP: beaker-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index }}
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
Values that are not valid Kubernetes label values are sanitized (and a warning is logged), so prefer an identifier that is already label-safe — otherwise the label will not be the string you set.
|
|
291
|
+
|
|
257
292
|
## Requirements
|
|
258
293
|
|
|
259
294
|
- KubeVirt and Kubernetes cluster access via `kubeconfig`
|
data/examples/hosts.yaml
CHANGED
|
@@ -8,6 +8,11 @@ HOSTS:
|
|
|
8
8
|
kubevirt_network_mode: port-forward
|
|
9
9
|
kubevirt_ssh_key: ~/.ssh/id_rsa.pub
|
|
10
10
|
kubevirt_cpus: 2
|
|
11
|
+
# Guest core count is not the pod's CPU request: without kubevirt_cpu_request
|
|
12
|
+
# KubeVirt divides the cores by the cluster's cpuAllocationRatio (default 10),
|
|
13
|
+
# so these 2 cores request only 200m and get throttled on a busy node.
|
|
14
|
+
# kubevirt_cpu_request: 2
|
|
15
|
+
# kubevirt_cpu_limit: 2
|
|
11
16
|
kubevirt_memory: 4Gi
|
|
12
17
|
# Extra memory added to the guest for the virt-launcher container memory
|
|
13
18
|
# limit. Bump this for Windows guests that get OOMKilled with the default.
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'securerandom'
|
|
4
|
+
require 'digest'
|
|
4
5
|
require 'yaml'
|
|
5
6
|
require 'base64'
|
|
6
7
|
require 'socket'
|
|
@@ -53,10 +54,61 @@ module Beaker
|
|
|
53
54
|
|
|
54
55
|
# VMI phases that indicate the VM will never reach Running.
|
|
55
56
|
VMI_TERMINAL_PHASES = %w[Failed Succeeded].freeze
|
|
57
|
+
|
|
58
|
+
# Values KubeVirt accepts for a disk's cache mode
|
|
59
|
+
DISK_CACHE_MODES = %w[none writethrough writeback].freeze
|
|
60
|
+
# Taint effects a toleration may name
|
|
61
|
+
TOLERATION_EFFECTS = %w[NoSchedule PreferNoSchedule NoExecute].freeze
|
|
62
|
+
# Fields a kubevirt_tolerations entry may set
|
|
63
|
+
TOLERATION_FIELDS = %w[key value effect].freeze
|
|
64
|
+
# Labels beaker sets itself; cleanup and the NodePort service select on them
|
|
65
|
+
RESERVED_LABEL_KEYS = %w[beaker/test-group beaker/host kubevirt.io/vm].freeze
|
|
66
|
+
# Kubernetes label values, and the name part of label and annotation keys
|
|
67
|
+
LABEL_NAME_RE = /\A(?:[A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]\z/
|
|
68
|
+
# The optional prefix of a label or annotation key (an RFC 1123 subdomain)
|
|
69
|
+
KEY_PREFIX_RE = /\A[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9](?:[-a-z0-9]*[a-z0-9])?)*\z/
|
|
70
|
+
KEY_PREFIX_MAX = 253
|
|
56
71
|
# virt-launcher container waiting reasons we treat as fatal.
|
|
57
|
-
FATAL_CONTAINER_WAITING_REASONS = %w[CrashLoopBackOff
|
|
72
|
+
FATAL_CONTAINER_WAITING_REASONS = %w[CrashLoopBackOff].freeze
|
|
73
|
+
# virt-launcher container waiting reasons that mean an image pull is being
|
|
74
|
+
# retried. kubelet re-attempts these with increasing backoff, and a cold
|
|
75
|
+
# pull-through cache, a slow upstream or a transient network error all clear
|
|
76
|
+
# without intervention — so they are only fatal once they persist (see
|
|
77
|
+
# DEFAULT_IMAGE_PULL_GRACE).
|
|
78
|
+
IMAGE_PULL_WAITING_REASONS = %w[ImagePullBackOff ErrImagePull].freeze
|
|
79
|
+
# How long an image pull may stay in backoff before we give up on it, unless
|
|
80
|
+
# overridden with :kubevirt_image_pull_grace. Still bounded by the overall
|
|
81
|
+
# wait_for_vm_ready timeout, which this does not extend.
|
|
82
|
+
DEFAULT_IMAGE_PULL_GRACE = 600
|
|
83
|
+
# Pull failures kubelet's retry will never fix — a bad tag, a missing repo or
|
|
84
|
+
# a rejected credential. Failing these immediately keeps a typo'd image as
|
|
85
|
+
# fast to diagnose as it was before the grace window existed. Transport-level
|
|
86
|
+
# failures (EOF, connection reset, timeouts) deliberately do not match. The
|
|
87
|
+
# patterns are anchored to the registry's and containerd's own wording: a
|
|
88
|
+
# bare "not found" would catch a cold pull-through cache's blob 404, and a
|
|
89
|
+
# bare "denied" would catch `connect: permission denied` from a network flap,
|
|
90
|
+
# both of which are what the window is for. containerd reports a missing tag
|
|
91
|
+
# as `failed to resolve reference "<ref>": <ref>: not found`; resolving is
|
|
92
|
+
# the tag/manifest lookup, never a blob fetch.
|
|
93
|
+
DEFINITIVE_IMAGE_PULL_FAILURE_RE = Regexp.union(
|
|
94
|
+
/manifest unknown/i,
|
|
95
|
+
/failed to resolve reference .*: not found/i,
|
|
96
|
+
/pull access denied/i,
|
|
97
|
+
/requested access to the resource is denied/i,
|
|
98
|
+
/unauthorized: authentication required/i,
|
|
99
|
+
)
|
|
58
100
|
# virt-launcher container termination reasons we treat as fatal.
|
|
59
101
|
FATAL_CONTAINER_TERMINATED_REASONS = %w[OOMKilled Error ContainerCannotRun].freeze
|
|
102
|
+
# PodScheduled=False reasons that mean the scheduler cannot place the virt-launcher pod.
|
|
103
|
+
# SchedulingGated is deliberately excluded: a gated pod is being held on purpose by
|
|
104
|
+
# whoever set the gate, and is not the cluster failing to find room.
|
|
105
|
+
UNSCHEDULABLE_POD_REASONS = %w[Unschedulable SchedulerError].freeze
|
|
106
|
+
# How long the virt-launcher pod may stay unschedulable before provisioning gives up,
|
|
107
|
+
# unless overridden with :kubevirt_unschedulable_grace. This cannot be immediate: a
|
|
108
|
+
# cluster autoscaler treats an unschedulable pod as its cue to add a node, and a node
|
|
109
|
+
# returning from a drain or reboot frees capacity on its own — both take minutes. Still
|
|
110
|
+
# bounded by the overall wait_for_vm_ready timeout, which this does not extend.
|
|
111
|
+
DEFAULT_UNSCHEDULABLE_GRACE = 300
|
|
60
112
|
|
|
61
113
|
##
|
|
62
114
|
# Create a new instance of the KubeVirt hypervisor object
|
|
@@ -72,6 +124,12 @@ module Beaker
|
|
|
72
124
|
# @option options [String] :kubevirt_network_mode Network mode (port-forward, nodeport, multus)
|
|
73
125
|
# @option options [String] :kubevirt_ssh_key SSH public key to inject
|
|
74
126
|
# @option options [String] :kubevirt_cpus CPU resources for VM
|
|
127
|
+
# @option options [String] :kubevirt_cpu_request CPU request for the VM pod, as a Kubernetes
|
|
128
|
+
# CPU quantity ('2', '1500m'). Unset by default, which leaves KubeVirt to derive the request
|
|
129
|
+
# from the guest core count divided by the cluster's cpuAllocationRatio (default 10). Set it
|
|
130
|
+
# for guests that must not be throttled while booting -- Windows especially.
|
|
131
|
+
# @option options [String] :kubevirt_cpu_limit CPU limit for the VM pod, as a Kubernetes CPU
|
|
132
|
+
# quantity. Unset by default, which leaves the compute container burstable.
|
|
75
133
|
# @option options [String] :kubevirt_memory Memory resources for VM
|
|
76
134
|
# @option options [String] :kubevirt_memory_overhead Extra memory added to guest for the
|
|
77
135
|
# container memory limit (default: '512Mi'). Needed for Windows guests where KubeVirt's
|
|
@@ -89,7 +147,21 @@ module Beaker
|
|
|
89
147
|
# (period_seconds * failure_threshold = 600s) accommodates slow Windows first-boots.
|
|
90
148
|
# @option options [Integer] :timeout Timeout for operations. When the readiness probe is
|
|
91
149
|
# enabled, the effective wait is max(:timeout, probe budget + 60s).
|
|
150
|
+
# @option options [Integer] :kubevirt_image_pull_grace How long a containerDisk pull may
|
|
151
|
+
# stay in ImagePullBackOff/ErrImagePull before it is treated as fatal (default: 600s).
|
|
152
|
+
# kubelet retries image pulls, so a cold pull-through cache or a slow registry clears on
|
|
153
|
+
# its own. Bounded by (and does not extend) the effective :timeout above, and skipped
|
|
154
|
+
# entirely when the pull failure is definitive (an unknown manifest or tag, or the
|
|
155
|
+
# registry refusing access).
|
|
156
|
+
# Set it to 0 to switch the window off and fail on the first backoff, as this check
|
|
157
|
+
# did before the option existed.
|
|
158
|
+
# @option options [Integer] :kubevirt_unschedulable_grace How long the virt-launcher pod may
|
|
159
|
+
# remain unschedulable before it is treated as fatal (default: 300s). An unschedulable pod
|
|
160
|
+
# is what triggers a cluster autoscaler to add a node, so this cannot be immediate.
|
|
161
|
+
# Bounded by (and does not extend) the effective :timeout above.
|
|
92
162
|
# @option options [Boolean] :kubevirt_disable_virtio Disable virtio devices (for compatibility with Windows)
|
|
163
|
+
# @option options [String] :kubevirt_disk_cache Cache mode for the root disk: none, writethrough
|
|
164
|
+
# or writeback. Unset leaves the choice to KubeVirt.
|
|
93
165
|
def initialize(kubevirt_hosts, options)
|
|
94
166
|
require 'beaker/hypervisor/kubevirt_helper'
|
|
95
167
|
|
|
@@ -105,9 +177,16 @@ module Beaker
|
|
|
105
177
|
@hosts = kubevirt_hosts
|
|
106
178
|
# Ensure the helper gets the validated namespace
|
|
107
179
|
@kubevirt_helper = KubevirtHelper.new(@options)
|
|
108
|
-
@test_group_identifier = "beaker-#{SecureRandom.hex(4)}"
|
|
180
|
+
@test_group_identifier = configured_test_group_identifier || "beaker-#{SecureRandom.hex(4)}"
|
|
109
181
|
@cleanup_called = false
|
|
110
182
|
@cleanup_mutex = Mutex.new
|
|
183
|
+
# "<pod_name>/<container>" => monotonic timestamp the container was first
|
|
184
|
+
# seen backing off an image pull. Cleared once it is running or terminated.
|
|
185
|
+
@image_pull_backoff_since = {}
|
|
186
|
+
# vm_name => { uid:, since: } — the virt-launcher pod's UID and the monotonic
|
|
187
|
+
# timestamp it was first seen unschedulable. Cleared as soon as the scheduler
|
|
188
|
+
# places it; restarted when a pod with a different UID replaces it.
|
|
189
|
+
@unschedulable_since = {}
|
|
111
190
|
|
|
112
191
|
# Register at_exit handler to ensure cleanup happens even on non-success exits
|
|
113
192
|
# This handles cases like Ctrl+C, errors, or test failures that occur after
|
|
@@ -277,7 +356,7 @@ module Beaker
|
|
|
277
356
|
if vm_image&.start_with?('http://', 'https://')
|
|
278
357
|
base_name = vm_image.split('/').last
|
|
279
358
|
# Create a unique datavolume name by including the VM name in it
|
|
280
|
-
host['dv_name'] =
|
|
359
|
+
host['dv_name'] = bounded_k8s_name("#{vm_name}-#{base_name}-dv")
|
|
281
360
|
elsif vm_image && !vm_image.start_with?('docker://', 'oci://')
|
|
282
361
|
# For PVC sources, we also need to clone to avoid sharing the same disk
|
|
283
362
|
source_pvc = vm_image.sub(%r{^pvc://}, '')
|
|
@@ -287,7 +366,7 @@ module Beaker
|
|
|
287
366
|
else
|
|
288
367
|
source_pvc_name = source_pvc
|
|
289
368
|
end
|
|
290
|
-
host['dv_name'] =
|
|
369
|
+
host['dv_name'] = bounded_k8s_name("#{vm_name}-#{source_pvc_name}")
|
|
291
370
|
|
|
292
371
|
end
|
|
293
372
|
|
|
@@ -309,7 +388,7 @@ module Beaker
|
|
|
309
388
|
def create_cloud_init_secret(host, cloud_init_data)
|
|
310
389
|
raise 'Cloud-init data must be provided' unless cloud_init_data
|
|
311
390
|
|
|
312
|
-
secret_name = "#{host['vm_name']}-cloud-init"
|
|
391
|
+
secret_name = bounded_k8s_name("#{host['vm_name']}-cloud-init")
|
|
313
392
|
@logger.debug("Creating cloud-init secret #{secret_name} in namespace #{@namespace}")
|
|
314
393
|
|
|
315
394
|
secret_spec = {
|
|
@@ -337,7 +416,25 @@ module Beaker
|
|
|
337
416
|
# @return [String] The generated VM name
|
|
338
417
|
def generate_vm_name(host)
|
|
339
418
|
host_name = host.respond_to?(:name) ? host.name : host['name']
|
|
340
|
-
|
|
419
|
+
bounded_k8s_name("#{@test_group_identifier}-#{host_name}")
|
|
420
|
+
end
|
|
421
|
+
|
|
422
|
+
##
|
|
423
|
+
# Sanitize a name to a DNS label, keeping distinct inputs distinct when
|
|
424
|
+
# they exceed the 63-char cap.
|
|
425
|
+
#
|
|
426
|
+
# Plain truncation drops the tail, which is where the host name (or a
|
|
427
|
+
# suffix such as "-ssh") sits, so hosts sharing a long test-group
|
|
428
|
+
# identifier would collide. At the cap the tail is replaced with a digest
|
|
429
|
+
# of the untruncated name instead.
|
|
430
|
+
# @param [String] full_name The unsanitized name
|
|
431
|
+
# @return [String] A valid DNS label of at most 63 characters
|
|
432
|
+
def bounded_k8s_name(full_name)
|
|
433
|
+
name = sanitize_k8s_name(full_name)
|
|
434
|
+
return name if name.length < K8S_NAME_MAX
|
|
435
|
+
|
|
436
|
+
digest = Digest::SHA256.hexdigest(full_name)[0, 8]
|
|
437
|
+
"#{name[0, K8S_NAME_MAX - digest.length - 1].sub(/-+\z/, '')}-#{digest}"
|
|
341
438
|
end
|
|
342
439
|
|
|
343
440
|
##
|
|
@@ -442,12 +539,128 @@ module Beaker
|
|
|
442
539
|
find_ssh_key_pair[:public_key]
|
|
443
540
|
end
|
|
444
541
|
|
|
542
|
+
##
|
|
543
|
+
# An externally supplied test-group identifier, from
|
|
544
|
+
# BEAKER_KUBEVIRT_TEST_GROUP.
|
|
545
|
+
#
|
|
546
|
+
# A caller that outlives the beaker process needs to be able to find this
|
|
547
|
+
# run's resources. CI is the motivating case: when a job is cancelled, the
|
|
548
|
+
# beaker process can be killed before it tears anything down, and a random
|
|
549
|
+
# per-process identifier leaves the job no way to name what it leaked.
|
|
550
|
+
# Deleting by `beaker/test-group=<the identifier it chose>` cleans up that
|
|
551
|
+
# run without touching other runs sharing the namespace.
|
|
552
|
+
#
|
|
553
|
+
# @return [String, nil] the sanitized identifier, or nil when unset or blank
|
|
554
|
+
def configured_test_group_identifier
|
|
555
|
+
raw = ENV.fetch('BEAKER_KUBEVIRT_TEST_GROUP', '')
|
|
556
|
+
return nil if raw.strip.empty?
|
|
557
|
+
|
|
558
|
+
# Truncating would cut off the tail, which is usually the part that tells
|
|
559
|
+
# runs apart (job ID, matrix index), and give concurrent runs one label.
|
|
560
|
+
if raw.strip.length > LABEL_VALUE_MAX
|
|
561
|
+
raise ArgumentError, "BEAKER_KUBEVIRT_TEST_GROUP must be at most #{LABEL_VALUE_MAX} characters, " \
|
|
562
|
+
"got #{raw.strip.length}: #{raw.inspect}"
|
|
563
|
+
end
|
|
564
|
+
|
|
565
|
+
identifier = sanitize_k8s_label_value(raw)
|
|
566
|
+
if identifier != raw
|
|
567
|
+
@logger.warn(
|
|
568
|
+
"BEAKER_KUBEVIRT_TEST_GROUP #{raw.inspect} is not usable as a Kubernetes " \
|
|
569
|
+
"label value; using #{identifier.inspect} instead",
|
|
570
|
+
)
|
|
571
|
+
end
|
|
572
|
+
identifier
|
|
573
|
+
end
|
|
574
|
+
|
|
445
575
|
def get_labels(host)
|
|
446
576
|
raw_name = host.respond_to?(:name) ? host.name : host['name']
|
|
447
|
-
|
|
577
|
+
user_labels(host).merge(
|
|
448
578
|
'beaker/test-group' => @test_group_identifier,
|
|
449
579
|
'beaker/host' => sanitize_k8s_label_value(raw_name),
|
|
450
|
-
|
|
580
|
+
)
|
|
581
|
+
end
|
|
582
|
+
|
|
583
|
+
##
|
|
584
|
+
# Extra labels for every resource that carries beaker's own labels, from
|
|
585
|
+
# BEAKER_kubevirt_labels, the host or the global CONFIG, in that order.
|
|
586
|
+
# Cleanup finds resources by beaker/test-group, so the keys beaker sets
|
|
587
|
+
# itself are refused rather than silently overridden.
|
|
588
|
+
# @param [Host] host The host configuration
|
|
589
|
+
# @return [Hash{String=>String}] The labels, empty when unset
|
|
590
|
+
def user_labels(host)
|
|
591
|
+
labels = key_value_option(host, 'kubevirt_labels')
|
|
592
|
+
reserved = labels.keys & RESERVED_LABEL_KEYS
|
|
593
|
+
raise ArgumentError, "kubevirt_labels may not set #{reserved.join(', ')}; beaker sets those itself" unless reserved.empty?
|
|
594
|
+
|
|
595
|
+
labels.each do |key, value|
|
|
596
|
+
next if value.empty? || (value.length <= LABEL_VALUE_MAX && LABEL_NAME_RE.match?(value))
|
|
597
|
+
|
|
598
|
+
raise ArgumentError, "kubevirt_labels value for #{key.inspect} must be at most #{LABEL_VALUE_MAX} characters of " \
|
|
599
|
+
"letters, digits, '-', '_' or '.', starting and ending with a letter or digit, got #{value.inspect}"
|
|
600
|
+
end
|
|
601
|
+
labels
|
|
602
|
+
end
|
|
603
|
+
|
|
604
|
+
##
|
|
605
|
+
# Extra annotations for the VM template, and so for the VMI and its
|
|
606
|
+
# virt-launcher pod, from BEAKER_kubevirt_annotations, the host or the
|
|
607
|
+
# global CONFIG, in that order.
|
|
608
|
+
# @param [Host] host The host configuration
|
|
609
|
+
# @return [Hash{String=>String}, nil] The annotations, or nil when unset
|
|
610
|
+
def user_annotations(host)
|
|
611
|
+
annotations = key_value_option(host, 'kubevirt_annotations')
|
|
612
|
+
annotations unless annotations.empty?
|
|
613
|
+
end
|
|
614
|
+
|
|
615
|
+
##
|
|
616
|
+
# A hash option that BEAKER_<name> may replace with a comma-separated list
|
|
617
|
+
# of key=value pairs. Only the first = in a pair separates key from value.
|
|
618
|
+
# @param [Host] host The host configuration
|
|
619
|
+
# @param [String] name The option name
|
|
620
|
+
# @return [Hash{String=>String}] The option, empty when unset
|
|
621
|
+
def key_value_option(host, name)
|
|
622
|
+
env = env_option(name)
|
|
623
|
+
value = env ? parse_key_values(name, env) : host[name] || @options[name.to_sym]
|
|
624
|
+
return {} if value.nil?
|
|
625
|
+
raise ArgumentError, "#{name} must be a hash of keys to values, got #{value.inspect}" unless value.is_a?(Hash)
|
|
626
|
+
|
|
627
|
+
value.to_h do |key, val|
|
|
628
|
+
key = key.to_s
|
|
629
|
+
unless qualified_key?(key)
|
|
630
|
+
raise ArgumentError, "#{name} key #{key.inspect} is not a valid Kubernetes key: expected an optional DNS subdomain " \
|
|
631
|
+
"prefix and '/', then a name of at most #{LABEL_VALUE_MAX} characters of letters, digits, " \
|
|
632
|
+
"'-', '_' or '.', starting and ending with a letter or digit"
|
|
633
|
+
end
|
|
634
|
+
|
|
635
|
+
raise ArgumentError, "#{name} value for #{key.inspect} must be a single value, got #{val.inspect}" if val.is_a?(Hash) || val.is_a?(Array)
|
|
636
|
+
|
|
637
|
+
[key, val.to_s]
|
|
638
|
+
end
|
|
639
|
+
end
|
|
640
|
+
|
|
641
|
+
##
|
|
642
|
+
# Whether a string is a valid label or annotation key: an optional
|
|
643
|
+
# DNS-subdomain prefix and '/', then a name. The API server rejects
|
|
644
|
+
# anything else with a 422, deep into provisioning.
|
|
645
|
+
# @param [String] key
|
|
646
|
+
# @return [Boolean]
|
|
647
|
+
def qualified_key?(key)
|
|
648
|
+
prefix, slash, name = key.rpartition('/')
|
|
649
|
+
return false unless name.length <= LABEL_VALUE_MAX && LABEL_NAME_RE.match?(name)
|
|
650
|
+
|
|
651
|
+
slash.empty? || (prefix.length <= KEY_PREFIX_MAX && KEY_PREFIX_RE.match?(prefix))
|
|
652
|
+
end
|
|
653
|
+
|
|
654
|
+
def parse_key_values(name, value)
|
|
655
|
+
pairs = value.split(',').map(&:strip).reject(&:empty?)
|
|
656
|
+
raise ArgumentError, "BEAKER_#{name} is set but has no key=value entries: #{value.inspect}" if pairs.empty?
|
|
657
|
+
|
|
658
|
+
pairs.to_h do |pair|
|
|
659
|
+
key, val = pair.split('=', 2).map(&:strip)
|
|
660
|
+
raise ArgumentError, "BEAKER_#{name} entries must be key=value, got #{pair.inspect}" if val.nil? || key.empty?
|
|
661
|
+
|
|
662
|
+
[key, val]
|
|
663
|
+
end
|
|
451
664
|
end
|
|
452
665
|
|
|
453
666
|
##
|
|
@@ -483,19 +696,146 @@ module Beaker
|
|
|
483
696
|
end
|
|
484
697
|
end
|
|
485
698
|
|
|
699
|
+
##
|
|
700
|
+
# The value of the BEAKER_<name> environment variable, which overrides the
|
|
701
|
+
# host and CONFIG settings of the option it names so that a CI environment
|
|
702
|
+
# can set it for every nodeset it runs, as beaker-google's BEAKER_gce_*
|
|
703
|
+
# variables do.
|
|
704
|
+
# @param [String] name The option name, e.g. 'kubevirt_disk_cache'
|
|
705
|
+
# @return [String, nil] The stripped value, or nil when unset or blank
|
|
706
|
+
def env_option(name)
|
|
707
|
+
value = ENV.fetch("BEAKER_#{name}", nil)&.strip
|
|
708
|
+
value unless value.nil? || value.empty?
|
|
709
|
+
end
|
|
710
|
+
|
|
711
|
+
##
|
|
712
|
+
# Cache mode for the root disk, from BEAKER_kubevirt_disk_cache, the host
|
|
713
|
+
# or the global CONFIG, in that order.
|
|
714
|
+
# Unset leaves the choice to KubeVirt, which uses none when the backing
|
|
715
|
+
# store supports O_DIRECT and writethrough otherwise. writeback lets the
|
|
716
|
+
# virt-launcher's page cache absorb guest writes until the guest flushes,
|
|
717
|
+
# which suits throwaway test VMs on slow or shared storage.
|
|
718
|
+
# @param [Host] host The host configuration
|
|
719
|
+
# @return [String, nil] One of DISK_CACHE_MODES, or nil when unset
|
|
720
|
+
def disk_cache(host)
|
|
721
|
+
env = env_option('kubevirt_disk_cache')
|
|
722
|
+
value = env || host['kubevirt_disk_cache'] || @options[:kubevirt_disk_cache]
|
|
723
|
+
return nil if value.nil? || value.to_s.empty?
|
|
724
|
+
|
|
725
|
+
cache = value.to_s.downcase
|
|
726
|
+
unless DISK_CACHE_MODES.include?(cache)
|
|
727
|
+
source = env ? 'BEAKER_kubevirt_disk_cache' : 'kubevirt_disk_cache'
|
|
728
|
+
raise ArgumentError, "#{source} must be one of #{DISK_CACHE_MODES.join(', ')}, got #{value.to_s.inspect}"
|
|
729
|
+
end
|
|
730
|
+
|
|
731
|
+
cache
|
|
732
|
+
end
|
|
733
|
+
|
|
734
|
+
##
|
|
735
|
+
# Node labels the VM's pod must match, from BEAKER_kubevirt_node_selector,
|
|
736
|
+
# the host or the global CONFIG, in that order. The variable is a
|
|
737
|
+
# comma-separated list of key=value pairs.
|
|
738
|
+
# @param [Host] host The host configuration
|
|
739
|
+
# @return [Hash{String=>String}, nil] The selector, or nil when unset
|
|
740
|
+
def node_selector(host)
|
|
741
|
+
env = env_option('kubevirt_node_selector')
|
|
742
|
+
selector = env ? parse_key_values('kubevirt_node_selector', env) : host['kubevirt_node_selector'] || @options[:kubevirt_node_selector]
|
|
743
|
+
return nil if selector.nil?
|
|
744
|
+
raise ArgumentError, "kubevirt_node_selector must be a hash of label keys to values, got #{selector.inspect}" unless selector.is_a?(Hash)
|
|
745
|
+
return nil if selector.empty?
|
|
746
|
+
|
|
747
|
+
selector.to_h do |key, value|
|
|
748
|
+
raise ArgumentError, "kubevirt_node_selector value for #{key.to_s.inspect} must be a single label value, got #{value.inspect}" unless scalar?(value)
|
|
749
|
+
|
|
750
|
+
[key.to_s, value.to_s]
|
|
751
|
+
end
|
|
752
|
+
end
|
|
753
|
+
|
|
754
|
+
def scalar?(value)
|
|
755
|
+
case value
|
|
756
|
+
# Not Float: YAML reads an unquoted 1.10 as 1.1, which would select the
|
|
757
|
+
# wrong nodes without an error. A quoted value is unambiguous.
|
|
758
|
+
when String, Symbol, Integer, true, false then true
|
|
759
|
+
else false
|
|
760
|
+
end
|
|
761
|
+
end
|
|
762
|
+
|
|
763
|
+
##
|
|
764
|
+
# Tolerations for the VM's pod, from BEAKER_kubevirt_tolerations, the host
|
|
765
|
+
# or the global CONFIG, in that order. Each entry has a key, an effect and
|
|
766
|
+
# an optional value; one without a value tolerates the key whatever its
|
|
767
|
+
# value. The variable is a comma-separated list of key=value:Effect or
|
|
768
|
+
# key:Effect entries.
|
|
769
|
+
# @param [Host] host The host configuration
|
|
770
|
+
# @return [Array<Hash>, nil] Kubernetes tolerations, or nil when unset
|
|
771
|
+
def tolerations(host)
|
|
772
|
+
env = env_option('kubevirt_tolerations')
|
|
773
|
+
list = env ? parse_tolerations(env) : host['kubevirt_tolerations'] || @options[:kubevirt_tolerations]
|
|
774
|
+
return nil if list.nil?
|
|
775
|
+
raise ArgumentError, "kubevirt_tolerations must be a list of key/value/effect entries, got #{list.inspect}" unless list.is_a?(Array)
|
|
776
|
+
return nil if list.empty?
|
|
777
|
+
|
|
778
|
+
list.map { |entry| build_toleration(entry) }
|
|
779
|
+
end
|
|
780
|
+
|
|
781
|
+
def parse_tolerations(value)
|
|
782
|
+
entries = value.split(',').map(&:strip).reject(&:empty?)
|
|
783
|
+
raise ArgumentError, "BEAKER_kubevirt_tolerations is set but has no entries: #{value.inspect}" if entries.empty?
|
|
784
|
+
|
|
785
|
+
entries.map do |entry|
|
|
786
|
+
taint, separator, effect = entry.rpartition(':')
|
|
787
|
+
raise ArgumentError, "BEAKER_kubevirt_tolerations entries must be key=value:Effect or key:Effect, got #{entry.inspect}" if separator.empty?
|
|
788
|
+
|
|
789
|
+
key, toleration_value = taint.split('=', 2)
|
|
790
|
+
{ 'key' => key, 'value' => toleration_value, 'effect' => effect }
|
|
791
|
+
end
|
|
792
|
+
end
|
|
793
|
+
|
|
794
|
+
def build_toleration(entry)
|
|
795
|
+
raise ArgumentError, "Each kubevirt_tolerations entry must be a hash, got #{entry.inspect}" unless entry.is_a?(Hash)
|
|
796
|
+
|
|
797
|
+
# operator and tolerationSeconds are not supported. Dropping them silently
|
|
798
|
+
# would change what the toleration matches or how long it lasts.
|
|
799
|
+
unknown = entry.keys.map(&:to_s) - TOLERATION_FIELDS
|
|
800
|
+
unless unknown.empty?
|
|
801
|
+
raise ArgumentError, "kubevirt_tolerations entry #{entry.inspect} has unsupported fields #{unknown.join(', ')}; " \
|
|
802
|
+
"only #{TOLERATION_FIELDS.join(', ')} are accepted (an entry without a value tolerates any value)"
|
|
803
|
+
end
|
|
804
|
+
|
|
805
|
+
# key? rather than ||, so a YAML `value: false` is the string "false" and
|
|
806
|
+
# not a missing value, which would widen the toleration to Exists.
|
|
807
|
+
key, value, effect = %w[key value effect].map { |field| entry.key?(field) ? entry[field] : entry[field.to_sym] }
|
|
808
|
+
key = key.to_s.strip
|
|
809
|
+
effect = effect.to_s.strip
|
|
810
|
+
raise ArgumentError, "kubevirt_tolerations entry #{entry.inspect} has no key" if key.empty?
|
|
811
|
+
raise ArgumentError, "kubevirt_tolerations effect must be one of #{TOLERATION_EFFECTS.join(', ')}, got #{effect.inspect}" unless TOLERATION_EFFECTS.include?(effect)
|
|
812
|
+
|
|
813
|
+
# Only an absent value means Exists. An explicit empty string is Equal "",
|
|
814
|
+
# which matches only taints whose value is empty.
|
|
815
|
+
if value.nil?
|
|
816
|
+
{ 'key' => key, 'operator' => 'Exists', 'effect' => effect }
|
|
817
|
+
else
|
|
818
|
+
{ 'key' => key, 'operator' => 'Equal', 'value' => value.to_s.strip, 'effect' => effect }
|
|
819
|
+
end
|
|
820
|
+
end
|
|
821
|
+
|
|
486
822
|
##
|
|
487
823
|
# Generate the hardware devices specification for the VM
|
|
488
824
|
# @param [Host] host The host configuration
|
|
489
825
|
# @return [Hash] Hardware devices specification
|
|
490
826
|
def generate_hardware_spec(host)
|
|
827
|
+
rootdisk = {
|
|
828
|
+
'name' => 'rootdisk',
|
|
829
|
+
'disk' => {
|
|
830
|
+
'bus' => disk_bus(host),
|
|
831
|
+
},
|
|
832
|
+
}
|
|
833
|
+
cache = disk_cache(host)
|
|
834
|
+
rootdisk['cache'] = cache if cache
|
|
835
|
+
|
|
491
836
|
{
|
|
492
837
|
'disks' => [
|
|
493
|
-
|
|
494
|
-
'name' => 'rootdisk',
|
|
495
|
-
'disk' => {
|
|
496
|
-
'bus' => disk_bus(host),
|
|
497
|
-
},
|
|
498
|
-
},
|
|
838
|
+
rootdisk,
|
|
499
839
|
{
|
|
500
840
|
'name' => 'cidata',
|
|
501
841
|
'disk' => {
|
|
@@ -536,6 +876,15 @@ module Beaker
|
|
|
536
876
|
memory_request = "#{memory_request}Mi" if /\A\d+\z/.match?(memory_request.to_s)
|
|
537
877
|
memory_limit = "#{parse_memory_mib(memory) + parse_memory_mib(overhead)}Mi"
|
|
538
878
|
|
|
879
|
+
# Both unset by default: KubeVirt then derives the pod's CPU request from the guest core
|
|
880
|
+
# count divided by the cluster's cpuAllocationRatio, and sets no limit at all.
|
|
881
|
+
cpu_request = host['kubevirt_cpu_request'] || @options[:kubevirt_cpu_request]
|
|
882
|
+
cpu_limit = host['kubevirt_cpu_limit'] || @options[:kubevirt_cpu_limit]
|
|
883
|
+
resource_requests = { 'memory' => memory_request }
|
|
884
|
+
resource_requests['cpu'] = cpu_request.to_s if cpu_request
|
|
885
|
+
resource_limits = { 'memory' => memory_limit }
|
|
886
|
+
resource_limits['cpu'] = cpu_limit.to_s if cpu_limit
|
|
887
|
+
|
|
539
888
|
vm_image = host['kubevirt_vm_image'] || @options[:kubevirt_vm_image]
|
|
540
889
|
host_name = host.respond_to?(:name) ? host.name : host['name']
|
|
541
890
|
|
|
@@ -561,7 +910,8 @@ module Beaker
|
|
|
561
910
|
'labels' => get_labels(host).merge({
|
|
562
911
|
'kubevirt.io/vm' => vm_name,
|
|
563
912
|
}),
|
|
564
|
-
|
|
913
|
+
'annotations' => user_annotations(host),
|
|
914
|
+
}.compact,
|
|
565
915
|
'spec' => {
|
|
566
916
|
'domain' => {
|
|
567
917
|
'cpu' => {
|
|
@@ -573,8 +923,8 @@ module Beaker
|
|
|
573
923
|
'guest' => memory.to_s,
|
|
574
924
|
},
|
|
575
925
|
'resources' => {
|
|
576
|
-
'requests' =>
|
|
577
|
-
'limits' =>
|
|
926
|
+
'requests' => resource_requests,
|
|
927
|
+
'limits' => resource_limits,
|
|
578
928
|
},
|
|
579
929
|
'devices' => generate_hardware_spec(host),
|
|
580
930
|
'features' => {
|
|
@@ -592,6 +942,8 @@ module Beaker
|
|
|
592
942
|
},
|
|
593
943
|
},
|
|
594
944
|
'hostname' => host_name,
|
|
945
|
+
'nodeSelector' => node_selector(host),
|
|
946
|
+
'tolerations' => tolerations(host),
|
|
595
947
|
'networks' => generate_networks_spec(host),
|
|
596
948
|
'readinessProbe' => generate_readiness_probe_spec(host),
|
|
597
949
|
'volumes' => [
|
|
@@ -637,6 +989,18 @@ module Beaker
|
|
|
637
989
|
end
|
|
638
990
|
end
|
|
639
991
|
|
|
992
|
+
##
|
|
993
|
+
# Storage class for the root disk's DataVolume, from
|
|
994
|
+
# BEAKER_kubevirt_storage_class, the host or the global CONFIG, in that
|
|
995
|
+
# order. Unset leaves the choice to the cluster's default class. A
|
|
996
|
+
# containerDisk root has no DataVolume, so it is unaffected.
|
|
997
|
+
# @param [Host] host The host configuration
|
|
998
|
+
# @return [String, nil] The storage class name, or nil when unset
|
|
999
|
+
def storage_class(host)
|
|
1000
|
+
name = (env_option('kubevirt_storage_class') || host['kubevirt_storage_class'] || @options[:kubevirt_storage_class]).to_s.strip
|
|
1001
|
+
name unless name.empty?
|
|
1002
|
+
end
|
|
1003
|
+
|
|
640
1004
|
##
|
|
641
1005
|
# Generate a DataVolume Template for the root disk
|
|
642
1006
|
# @param [String] vm_image The VM image specification
|
|
@@ -660,6 +1024,9 @@ module Beaker
|
|
|
660
1024
|
},
|
|
661
1025
|
}
|
|
662
1026
|
|
|
1027
|
+
storage_class_name = storage_class(host)
|
|
1028
|
+
dv_spec['spec']['storage']['storageClassName'] = storage_class_name if storage_class_name
|
|
1029
|
+
|
|
663
1030
|
# If a custom service account is specified, add it to the DataVolume spec
|
|
664
1031
|
# so it can access the source PVC when required (including cross-namespace clones)
|
|
665
1032
|
dv_spec['spec']['serviceAccountName'] = @service_account if @service_account
|
|
@@ -779,7 +1146,7 @@ module Beaker
|
|
|
779
1146
|
|
|
780
1147
|
raise "VMI #{vm_name} entered terminal phase #{phase} before becoming Ready" if VMI_TERMINAL_PHASES.include?(phase)
|
|
781
1148
|
|
|
782
|
-
check_virt_launcher_health!(
|
|
1149
|
+
check_virt_launcher_health!(host)
|
|
783
1150
|
|
|
784
1151
|
sleep SLEEPWAIT
|
|
785
1152
|
end
|
|
@@ -831,24 +1198,47 @@ module Beaker
|
|
|
831
1198
|
|
|
832
1199
|
##
|
|
833
1200
|
# Inspect the virt-launcher pod backing a VMI and raise with a specific reason
|
|
834
|
-
# if it has already failed (OOMKilled,
|
|
835
|
-
# so we fail fast instead of waiting the
|
|
836
|
-
#
|
|
837
|
-
|
|
1201
|
+
# if it has already failed (OOMKilled, crash loops, a container that cannot run)
|
|
1202
|
+
# or if the scheduler cannot place it, so we fail fast instead of waiting the
|
|
1203
|
+
# full timeout. Image pull backoff is handled separately by
|
|
1204
|
+
# check_image_pull_backoff!, since kubelet retries pulls and an early
|
|
1205
|
+
# observation does not mean provisioning has failed.
|
|
1206
|
+
# @param [Host] host The host being provisioned, carrying 'vm_name'
|
|
1207
|
+
def check_virt_launcher_health!(host)
|
|
1208
|
+
vm_name = host['vm_name']
|
|
838
1209
|
pod = @kubevirt_helper.get_virt_launcher_pod(vm_name)
|
|
839
1210
|
return unless pod
|
|
840
1211
|
|
|
1212
|
+
# Image-pull backoff is tracked per virt-launcher pod rather than per VMI.
|
|
1213
|
+
# A pod replaced mid-provision -- rescheduled, or evicted while pulling --
|
|
1214
|
+
# starts the pull again on a node whose cache may be cold, and must not
|
|
1215
|
+
# inherit the window its predecessor already spent. Keying on the pod also
|
|
1216
|
+
# means a momentary absence (API lag, or a `pods.first` that briefly sees
|
|
1217
|
+
# the successor) neither resets nor shortens the window a live pod is
|
|
1218
|
+
# accumulating.
|
|
1219
|
+
pod_name = pod.dig('metadata', 'name') || vm_name
|
|
1220
|
+
|
|
841
1221
|
phase = pod.dig('status', 'phase')
|
|
842
1222
|
raise "virt-launcher pod for #{vm_name} entered phase #{phase}" if phase == 'Failed'
|
|
843
1223
|
|
|
1224
|
+
check_pod_schedulable!(host, vm_name, pod)
|
|
1225
|
+
|
|
844
1226
|
statuses = Array(pod.dig('status', 'containerStatuses')) +
|
|
845
1227
|
Array(pod.dig('status', 'initContainerStatuses'))
|
|
846
1228
|
statuses.each do |cs|
|
|
847
1229
|
name = cs['name']
|
|
848
1230
|
waiting_reason = cs.dig('state', 'waiting', 'reason')
|
|
849
|
-
if
|
|
850
|
-
|
|
851
|
-
|
|
1231
|
+
if IMAGE_PULL_WAITING_REASONS.include?(waiting_reason)
|
|
1232
|
+
check_image_pull_backoff!(host, pod_name, name, waiting_reason, cs.dig('state', 'waiting', 'message').to_s)
|
|
1233
|
+
else
|
|
1234
|
+
# Only a container that has got past the pull ends its window. A
|
|
1235
|
+
# transient other waiting state between two backoffs must not restart it.
|
|
1236
|
+
@image_pull_backoff_since.delete("#{pod_name}/#{name}") if cs.dig('state', 'running') || cs.dig('state', 'terminated')
|
|
1237
|
+
|
|
1238
|
+
if FATAL_CONTAINER_WAITING_REASONS.include?(waiting_reason)
|
|
1239
|
+
raise "virt-launcher container #{name} for #{vm_name} is #{waiting_reason}: " \
|
|
1240
|
+
"#{cs.dig('state', 'waiting', 'message')}"
|
|
1241
|
+
end
|
|
852
1242
|
end
|
|
853
1243
|
|
|
854
1244
|
last_term = cs.dig('lastState', 'terminated') || cs.dig('state', 'terminated')
|
|
@@ -860,6 +1250,137 @@ module Beaker
|
|
|
860
1250
|
end
|
|
861
1251
|
end
|
|
862
1252
|
|
|
1253
|
+
##
|
|
1254
|
+
# Decide whether a container backing off an image pull has waited long enough
|
|
1255
|
+
# to be called a failure. kubelet retries pulls with increasing backoff, so a
|
|
1256
|
+
# single observation of ImagePullBackOff proves nothing: an on-demand
|
|
1257
|
+
# pull-through cache warming a multi-GB image, a slow upstream registry or a
|
|
1258
|
+
# dropped connection all resolve on a later attempt. Raise only once the
|
|
1259
|
+
# backoff has outlived the grace window — or immediately when the message says
|
|
1260
|
+
# retrying is pointless, or when the window has been switched off.
|
|
1261
|
+
# @param [Host] host The host being provisioned, carrying 'vm_name'
|
|
1262
|
+
# @param [String] pod_name Name of the virt-launcher pod reporting the backoff
|
|
1263
|
+
# @param [String] container Container name reporting the backoff
|
|
1264
|
+
# @param [String] reason Waiting reason (ImagePullBackOff / ErrImagePull)
|
|
1265
|
+
# @param [String] message Waiting message from kubelet
|
|
1266
|
+
def check_image_pull_backoff!(host, pod_name, container, reason, message)
|
|
1267
|
+
vm_name = host['vm_name']
|
|
1268
|
+
grace = image_pull_grace(host)
|
|
1269
|
+
|
|
1270
|
+
# Two ways to skip the window and fail on this very observation: kubelet's
|
|
1271
|
+
# retry will never fix a definitive failure, and a grace of zero is the
|
|
1272
|
+
# documented way to opt out of the window entirely. Both keep the message
|
|
1273
|
+
# the check raised before the window existed.
|
|
1274
|
+
raise "virt-launcher container #{container} for #{vm_name} is #{reason}: #{message}" if DEFINITIVE_IMAGE_PULL_FAILURE_RE.match?(message) || grace <= 0
|
|
1275
|
+
|
|
1276
|
+
key = "#{pod_name}/#{container}"
|
|
1277
|
+
now = monotonic_time
|
|
1278
|
+
first_seen = !@image_pull_backoff_since.key?(key)
|
|
1279
|
+
@image_pull_backoff_since[key] ||= now
|
|
1280
|
+
elapsed = (now - @image_pull_backoff_since[key]).round
|
|
1281
|
+
|
|
1282
|
+
if elapsed > grace
|
|
1283
|
+
raise "virt-launcher container #{container} for #{vm_name} is still #{reason} after #{elapsed}s, " \
|
|
1284
|
+
"exceeding the :kubevirt_image_pull_grace window of #{grace}s: #{message}"
|
|
1285
|
+
end
|
|
1286
|
+
|
|
1287
|
+
# Warn on first sight: when the ready timeout expires before the window
|
|
1288
|
+
# does, this is the only record of why the VM never became ready.
|
|
1289
|
+
if first_seen
|
|
1290
|
+
@logger.warn("virt-launcher container #{container} for #{vm_name} is #{reason}, allowing #{grace}s " \
|
|
1291
|
+
"for kubelet to retry the pull: #{message}")
|
|
1292
|
+
else
|
|
1293
|
+
@logger.debug("virt-launcher container #{container} for #{vm_name} is #{reason} " \
|
|
1294
|
+
"(#{elapsed}s of #{grace}s grace); waiting for kubelet to retry the pull")
|
|
1295
|
+
end
|
|
1296
|
+
end
|
|
1297
|
+
|
|
1298
|
+
##
|
|
1299
|
+
# How long an image pull may stay in backoff before it is treated as fatal.
|
|
1300
|
+
# Per-host first so a suite can give a multi-GB Windows containerDisk more room
|
|
1301
|
+
# than a small cloud image, then the global option, then the default. Zero
|
|
1302
|
+
# switches the window off rather than setting an empty one.
|
|
1303
|
+
# @param [Host] host The host being provisioned
|
|
1304
|
+
# @return [Integer] seconds
|
|
1305
|
+
def image_pull_grace(host)
|
|
1306
|
+
value = host['kubevirt_image_pull_grace'] || @options[:kubevirt_image_pull_grace]
|
|
1307
|
+
return DEFAULT_IMAGE_PULL_GRACE if value.nil?
|
|
1308
|
+
|
|
1309
|
+
grace = Integer(value, exception: false)
|
|
1310
|
+
raise ArgumentError, "kubevirt_image_pull_grace must be a whole number of seconds, got #{value.inspect}" if grace.nil? || grace.negative?
|
|
1311
|
+
|
|
1312
|
+
grace
|
|
1313
|
+
end
|
|
1314
|
+
|
|
1315
|
+
##
|
|
1316
|
+
# Raise once the scheduler has been unable to place the virt-launcher pod for
|
|
1317
|
+
# longer than the grace window.
|
|
1318
|
+
#
|
|
1319
|
+
# An unschedulable pod is invisible to every other check in
|
|
1320
|
+
# check_virt_launcher_health!: its phase is Pending rather than Failed, and both
|
|
1321
|
+
# containerStatuses and initContainerStatuses are absent entirely, because no
|
|
1322
|
+
# container was ever assigned to a node. So the wait loop spins until the overall
|
|
1323
|
+
# timeout and reports "Timeout waiting for VM ... to be ready" — the same message
|
|
1324
|
+
# a slow guest boot produces. The one failure the cluster can describe precisely
|
|
1325
|
+
# ("0/8 nodes are available: 5 Insufficient cpu") is the one that arrives with no
|
|
1326
|
+
# explanation at all.
|
|
1327
|
+
#
|
|
1328
|
+
# Being unschedulable is not immediately terminal, which is why the grace window
|
|
1329
|
+
# exists rather than raising on first sight. The reason is logged as soon as it is
|
|
1330
|
+
# seen, so it is never silent even when the window is set wide.
|
|
1331
|
+
#
|
|
1332
|
+
# @param [Host] host The host being provisioned
|
|
1333
|
+
# @param [String] vm_name
|
|
1334
|
+
# @param [Hash] pod The virt-launcher pod
|
|
1335
|
+
def check_pod_schedulable!(host, vm_name, pod)
|
|
1336
|
+
condition = Array(pod.dig('status', 'conditions')).find { |c| c['type'] == 'PodScheduled' }
|
|
1337
|
+
reason = (condition && condition['status'] == 'False') ? condition['reason'] : nil
|
|
1338
|
+
|
|
1339
|
+
unless UNSCHEDULABLE_POD_REASONS.include?(reason)
|
|
1340
|
+
@unschedulable_since.delete(vm_name)
|
|
1341
|
+
return
|
|
1342
|
+
end
|
|
1343
|
+
|
|
1344
|
+
message = condition['message'].to_s
|
|
1345
|
+
grace = unschedulable_grace(host)
|
|
1346
|
+
uid = pod.dig('metadata', 'uid')
|
|
1347
|
+
tracked = @unschedulable_since[vm_name]
|
|
1348
|
+
# A replacement pod must get its own window, not inherit the elapsed time of
|
|
1349
|
+
# the one it replaced.
|
|
1350
|
+
if tracked.nil? || tracked[:uid] != uid
|
|
1351
|
+
tracked = @unschedulable_since[vm_name] = { uid: uid, since: monotonic_time }
|
|
1352
|
+
@logger.warn("virt-launcher pod for #{vm_name} is #{reason}, allowing #{grace}s " \
|
|
1353
|
+
"for capacity: #{message}")
|
|
1354
|
+
end
|
|
1355
|
+
|
|
1356
|
+
waited = monotonic_time - tracked[:since]
|
|
1357
|
+
return if waited < grace
|
|
1358
|
+
|
|
1359
|
+
raise "virt-launcher pod for #{vm_name} is still #{reason} after #{waited.round}s: #{message}"
|
|
1360
|
+
end
|
|
1361
|
+
|
|
1362
|
+
##
|
|
1363
|
+
# Seconds the virt-launcher pod may remain unschedulable before it is fatal.
|
|
1364
|
+
# @param [Host] host
|
|
1365
|
+
# @return [Integer]
|
|
1366
|
+
def unschedulable_grace(host)
|
|
1367
|
+
value = host['kubevirt_unschedulable_grace'] || @options[:kubevirt_unschedulable_grace]
|
|
1368
|
+
return DEFAULT_UNSCHEDULABLE_GRACE if value.nil?
|
|
1369
|
+
|
|
1370
|
+
grace = Integer(value, exception: false)
|
|
1371
|
+
raise ArgumentError, "kubevirt_unschedulable_grace must be a whole number of seconds, got #{value.inspect}" if grace.nil? || grace.negative?
|
|
1372
|
+
|
|
1373
|
+
grace
|
|
1374
|
+
end
|
|
1375
|
+
|
|
1376
|
+
##
|
|
1377
|
+
# Monotonic seconds, so a wall-clock adjustment mid-wait cannot shorten or
|
|
1378
|
+
# extend a grace window.
|
|
1379
|
+
# @return [Float]
|
|
1380
|
+
def monotonic_time
|
|
1381
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
1382
|
+
end
|
|
1383
|
+
|
|
863
1384
|
##
|
|
864
1385
|
# The guest-side port SSH listens on, shared between the VMI readiness probe
|
|
865
1386
|
# and client-side networking setup so they can never drift apart.
|
|
@@ -964,7 +1485,7 @@ module Beaker
|
|
|
964
1485
|
# @param [Host] host The host
|
|
965
1486
|
def setup_nodeport(host)
|
|
966
1487
|
vm_name = host['vm_name']
|
|
967
|
-
service_name = "#{vm_name}-ssh"
|
|
1488
|
+
service_name = bounded_k8s_name("#{vm_name}-ssh")
|
|
968
1489
|
|
|
969
1490
|
@logger.debug("Creating NodePort service for VM #{vm_name}")
|
|
970
1491
|
service = @kubevirt_helper.create_nodeport_service(vm_name, service_name, @test_group_identifier)
|
|
@@ -547,7 +547,7 @@ class KubeVirtPortForwarder
|
|
|
547
547
|
end
|
|
548
548
|
|
|
549
549
|
# Manages state transitions with thread safety.
|
|
550
|
-
# rubocop:disable Naming/PredicateMethod
|
|
550
|
+
# rubocop:disable-next Naming/PredicateMethod
|
|
551
551
|
def state_transition_to(new_state)
|
|
552
552
|
@mutex.synchronize do
|
|
553
553
|
case new_state
|
|
@@ -564,7 +564,6 @@ class KubeVirtPortForwarder
|
|
|
564
564
|
end
|
|
565
565
|
true
|
|
566
566
|
end
|
|
567
|
-
# rubocop:enable Naming/PredicateMethod
|
|
568
567
|
|
|
569
568
|
# Convert kubeclient SSL options to Faye::WebSocket/EventMachine TLS options.
|
|
570
569
|
# @param ssl_options [Hash] The SSL options from kubeclient
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: beaker-kubevirt
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.1.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Steven Pritchard
|
|
@@ -126,14 +126,14 @@ dependencies:
|
|
|
126
126
|
requirements:
|
|
127
127
|
- - "~>"
|
|
128
128
|
- !ruby/object:Gem::Version
|
|
129
|
-
version: 5.
|
|
129
|
+
version: 5.3.0
|
|
130
130
|
type: :development
|
|
131
131
|
prerelease: false
|
|
132
132
|
version_requirements: !ruby/object:Gem::Requirement
|
|
133
133
|
requirements:
|
|
134
134
|
- - "~>"
|
|
135
135
|
- !ruby/object:Gem::Version
|
|
136
|
-
version: 5.
|
|
136
|
+
version: 5.3.0
|
|
137
137
|
description: Enables Beaker to provision VMs using KubeVirt for automated acceptance
|
|
138
138
|
testing of Puppet code.
|
|
139
139
|
email:
|
|
@@ -182,7 +182,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
182
182
|
- !ruby/object:Gem::Version
|
|
183
183
|
version: '0'
|
|
184
184
|
requirements: []
|
|
185
|
-
rubygems_version: 4.0.
|
|
185
|
+
rubygems_version: 4.0.20
|
|
186
186
|
specification_version: 4
|
|
187
187
|
summary: KubeVirt hypervisor support for Beaker acceptance testing
|
|
188
188
|
test_files: []
|