cs-image-system-gcloud-runtime-plugin 0.1.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- cs_image_system/gcloud_runtime/__init__.py +3 -0
- cs_image_system/gcloud_runtime/bootstrap.py +351 -0
- cs_image_system/gcloud_runtime/gcp_packer_source.py +135 -0
- cs_image_system/gcloud_runtime/gcp_provider_specific_image.py +38 -0
- cs_image_system/gcloud_runtime/gcp_runtime_builders.py +608 -0
- cs_image_system/gcloud_runtime/gcp_runtime_models.py +168 -0
- cs_image_system/gcloud_runtime/gcp_utils.py +498 -0
- cs_image_system/gcloud_runtime/main.py +28 -0
- cs_image_system/py.typed +0 -0
- cs_image_system_gcloud_runtime_plugin-0.1.1.dist-info/METADATA +1168 -0
- cs_image_system_gcloud_runtime_plugin-0.1.1.dist-info/RECORD +14 -0
- cs_image_system_gcloud_runtime_plugin-0.1.1.dist-info/WHEEL +4 -0
- cs_image_system_gcloud_runtime_plugin-0.1.1.dist-info/entry_points.txt +5 -0
- cs_image_system_gcloud_runtime_plugin-0.1.1.dist-info/licenses/LICENSE +202 -0
|
@@ -0,0 +1,1168 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: cs-image-system-gcloud-runtime-plugin
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: The gcloud runtime type: Google Compute Engine as a place where images bake and instances run
|
|
5
|
+
Project-URL: Repository, https://github.com/infrastructurebuilder/cs-image-system-3
|
|
6
|
+
Project-URL: Documentation, https://github.com/infrastructurebuilder/cs-image-system-3/blob/develop/docs/OPERATIONS.md
|
|
7
|
+
Project-URL: Package README, https://github.com/infrastructurebuilder/cs-image-system-3/blob/develop/packages/gcloud-runtime-plugin/README.md
|
|
8
|
+
Author-email: Mykel Alvis <mykelalvis@infrastructurebuilder.org>
|
|
9
|
+
License-Expression: Apache-2.0
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Requires-Python: >=3.13
|
|
12
|
+
Requires-Dist: cs-image-system-hashicorp-utils==0.1.1
|
|
13
|
+
Requires-Dist: cs-image-system-system==0.1.1
|
|
14
|
+
Requires-Dist: google-api-core>=2.31
|
|
15
|
+
Requires-Dist: google-auth>=2.55
|
|
16
|
+
Requires-Dist: google-cloud-compute>=1.49
|
|
17
|
+
Requires-Dist: pydantic>=2.13
|
|
18
|
+
Requires-Dist: python-hcl2>=8.1
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
|
|
21
|
+
# cs-image-system-gcloud-runtime-plugin
|
|
22
|
+
|
|
23
|
+
The `gcloud` runtime type. A runtime is where images are baked and instances
|
|
24
|
+
run; this package makes Google Compute Engine (GCE) one. It registers one
|
|
25
|
+
runtime model (`GCPCloudBuilderModel`), one runtime builder
|
|
26
|
+
(`GCPCloudBuilder`) and one version checker for the `gcloud` executable. At
|
|
27
|
+
load it lists the project's networks, subnetworks, routes and firewall rules
|
|
28
|
+
with the Compute API and checks the declared networking against them. During
|
|
29
|
+
a run it resolves vendor images by project and filter (or by image family),
|
|
30
|
+
emits the `googlecompute` packer source for every image baked here, bakes
|
|
31
|
+
and reaches instances through Identity-Aware Proxy (IAP) TCP forwarding,
|
|
32
|
+
answers the state query from the Compute API (images, booted images, power
|
|
33
|
+
states and instance identities), relabels released images, starts and stops
|
|
34
|
+
instances for bounded tasks, and deletes images by name. Credentials are
|
|
35
|
+
Application Default Credentials; the model declares none.
|
|
36
|
+
|
|
37
|
+
## What it registers
|
|
38
|
+
|
|
39
|
+
The entry point is declared in [pyproject.toml](pyproject.toml):
|
|
40
|
+
|
|
41
|
+
```toml
|
|
42
|
+
[project.entry-points."cs_image_system.plugins.runtime"]
|
|
43
|
+
gcloud = "cs_image_system.gcloud_runtime.main:initialize"
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
`initialize()` in [main.py](src/cs_image_system/gcloud_runtime/main.py)
|
|
47
|
+
returns a `GcpRuntimePluginMetadata` whose services map the key `gcloud` to
|
|
48
|
+
three classes and whose `builders_for_models` binds the model to its builder.
|
|
49
|
+
|
|
50
|
+
| Class | `csis_name()` | `csis_classifier()` | Registered under |
|
|
51
|
+
|---|---|---|---|
|
|
52
|
+
| `GCPCloudBuilderModel` ([gcp_runtime_models.py](src/cs_image_system/gcloud_runtime/gcp_runtime_models.py)) | `gcloud` | `VCT.CLOUD_BUILDER_MODEL` | `cloud_builder_model` and, because the loader re-registers cloud models as runtimes, `runtime_builder_model` |
|
|
53
|
+
| `GCPCloudBuilder` ([gcp_runtime_builders.py](src/cs_image_system/gcloud_runtime/gcp_runtime_builders.py)) | `gcloud` | `VCT.RUNTIME_BUILDER` | `runtime_builder` |
|
|
54
|
+
| `GCPCLIVersionChecker` ([gcp_runtime_builders.py](src/cs_image_system/gcloud_runtime/gcp_runtime_builders.py)) | `gcloud` | `VCT.VERSION_CHECKER` | `version_checker` |
|
|
55
|
+
|
|
56
|
+
A YAML entry selects the runtime with `type: gcloud` under
|
|
57
|
+
`runtime_builders:`. There are no type aliases; `gcloud` is the only type
|
|
58
|
+
key. An entry's own `aliases:` list gives that runtime extra names (the
|
|
59
|
+
fixture's `gcloud-east1` also answers to `gcp`, `google` and `us-east1`).
|
|
60
|
+
|
|
61
|
+
The version checker is looked up by an executable's `name` first and its
|
|
62
|
+
`type` second, so the fixture's `- name: gcloud` executable in
|
|
63
|
+
[executables.yml](../../tests/fixtures/config/cfg/executables.yml) is parsed
|
|
64
|
+
by it: the checker runs `gcloud --version` and matches
|
|
65
|
+
`Google Cloud SDK ([\d\.]+)` on the first line of the output, so the version
|
|
66
|
+
checked is the SDK's (`585.0.0`), not the `core` component's date-shaped
|
|
67
|
+
one (stage 48, 2026-09-17; before that the checker read the `core` line).
|
|
68
|
+
The fixture requires `>=500`.
|
|
69
|
+
|
|
70
|
+
Present in the package but not registered as a service:
|
|
71
|
+
|
|
72
|
+
- `GcpProviderSpecificImage` ([gcp_provider_specific_image.py](src/cs_image_system/gcloud_runtime/gcp_provider_specific_image.py)), reached through the builder's `provider_specific_image_class()` hook.
|
|
73
|
+
|
|
74
|
+
Stage 63 removed three dead pieces of code that earlier versions of this
|
|
75
|
+
README listed here (removed 2026-09-25): the unused constant `GCP_CLI`, the
|
|
76
|
+
unregistered `DummyGroupBuilderModel` copy of the
|
|
77
|
+
[dummy plugin](../dummy-plugin/README.md)'s group model that sat at the
|
|
78
|
+
bottom of the models module, and the `get_image_ssh_user` helper in
|
|
79
|
+
`gcp_utils.py`, which nothing called (the bake user comes from the
|
|
80
|
+
bake-user order, see `bake_ssh_username` below).
|
|
81
|
+
|
|
82
|
+
**The GCP section of the bootstrap** (stage 70 step 5), through the
|
|
83
|
+
`cs_image_system.bootstrap` entry-point group: `gcp =
|
|
84
|
+
cs_image_system.gcloud_runtime.bootstrap:gcp_section`. It owns the
|
|
85
|
+
questions and the terraform for this cloud's one-time initialisation --
|
|
86
|
+
the workload identity pool and GitHub provider of CI_SETUP.md 3.4, the
|
|
87
|
+
READ-ONLY service account and the optional WRITE one, their project
|
|
88
|
+
roles and `workloadIdentityUser` bindings (`tfmodules/bootstrap_gcp`).
|
|
89
|
+
The section is wanted by default where the tree declares a GCE runtime
|
|
90
|
+
or a `gcs` state backend. Its defaults come from the raw
|
|
91
|
+
`cfg/runtime-builders.yml` and `cfg/state-backends.yml` (a starter's
|
|
92
|
+
placeholders are no defaults), and every "does it already exist?"
|
|
93
|
+
question is a probe through `gcloud` as the active account (`projects
|
|
94
|
+
describe`, `iam workload-identity-pools [providers] describe`, `iam
|
|
95
|
+
service-accounts describe`, `storage buckets describe`): found, absent,
|
|
96
|
+
or -- with no credentials -- unanswerable, which `--quiet` refuses by
|
|
97
|
+
name. What exists is read as a data source and never rewritten (a
|
|
98
|
+
provider's condition may admit other repositories); every role and
|
|
99
|
+
binding is one member added. See OPERATIONS.md, "The bootstrap".
|
|
100
|
+
|
|
101
|
+
## Models
|
|
102
|
+
|
|
103
|
+
### `GCPCloudNetworkingModel`
|
|
104
|
+
|
|
105
|
+
Extends `CloudNetworkingConfig` in
|
|
106
|
+
[cloud_builder.py](../base/src/cs_image_system/base/models/cloud_builder.py),
|
|
107
|
+
which is `RuntimeNetworkingModel` in
|
|
108
|
+
[runtime.py](../base/src/cs_image_system/base/models/runtime.py) under
|
|
109
|
+
another name.
|
|
110
|
+
|
|
111
|
+
Fields it adds:
|
|
112
|
+
|
|
113
|
+
| Field | Type | Default | Meaning |
|
|
114
|
+
|---|---|---|---|
|
|
115
|
+
| `network_tags` | `list[str]` | `[]` | GCE network tags. Firewall rules apply by target tag, so these are the closest analog to AWS security group ids. Emitted as the packer source's `tags` and the instance module's `network_tags`. A tag no firewall rule targets logs a warning at load. |
|
|
116
|
+
| `_model_id` | `str \| None` | `None` (not an init field) | Back-reference to the owning runtime model. |
|
|
117
|
+
|
|
118
|
+
Base fields it inherits:
|
|
119
|
+
|
|
120
|
+
| Field | Type | Default | Meaning |
|
|
121
|
+
|---|---|---|---|
|
|
122
|
+
| `name` | `str` | the network name | Empty or `default` falls back to `network`. |
|
|
123
|
+
| `network` | `str` | `default` | The VPC network name. `default` resolves at load to the network literally named `default` (GCE has no default flag). |
|
|
124
|
+
| `subnets` | `list[RuntimeSubnetModel]` | required, non-empty | `name`, `subnet_id`, `is_default`, `public`, `cidr`, `availability_zone`, `config`. `subnet_id` is a subnetwork name, a full self link, or a `projects/<p>/regions/<r>/subnetworks/<n>` path. The first entry with `is_default: true` is the default subnet. |
|
|
125
|
+
| `availability_zones` | `list[RuntimeAvailabilityZoneModel]` | `[]` | Entries `{name, is_default}`. Read only by `validate`'s zone-compatibility check (stage 52); the GCE emitters use the runtime's `zone` instead. |
|
|
126
|
+
|
|
127
|
+
It constrains `name` to be non-empty after stripping. `get_subnet_id()`
|
|
128
|
+
raises when no default subnet exists. There is no security group concept
|
|
129
|
+
here: `security_group_ids` and the like are not fields and are rejected.
|
|
130
|
+
|
|
131
|
+
### `GCPCloudBuilderModel`
|
|
132
|
+
|
|
133
|
+
Extends `CloudBuilderModel` in
|
|
134
|
+
[cloud_builder.py](../base/src/cs_image_system/base/models/cloud_builder.py),
|
|
135
|
+
which extends `RuntimeBuilderModel` in
|
|
136
|
+
[runtime.py](../base/src/cs_image_system/base/models/runtime.py), which
|
|
137
|
+
extends `BuilderModel` and `NameTyped` in
|
|
138
|
+
[builder_model.py](../base/src/cs_image_system/base/models/builder_model.py).
|
|
139
|
+
|
|
140
|
+
Fields it adds:
|
|
141
|
+
|
|
142
|
+
| Field | Type | Default | Meaning |
|
|
143
|
+
|---|---|---|---|
|
|
144
|
+
| `project_id` | `str \| None` | `None` | The GCP project. `self` in an owner list resolves to it; without it network discovery is skipped with a warning and image resolution and bakes fail. Reported by `runtime describe`. |
|
|
145
|
+
| `zone` | `str \| None` | `None` | The zone build VMs and instances run in; the packer `zone`, the `google` provider's `zone`, and the scope of `inventory()`, `serial_console()`, the boot-image probe, the power-state and identity probes and `start_instance`/`stop_instance`. |
|
|
146
|
+
| `service_account_email` | `str \| None` | `None` | The service account attached to build VMs (`service_account_email` in the packer source). Without it packer attaches the project's default compute service account. Nothing passes it to the instance module: an instance gets whatever the module attaches. |
|
|
147
|
+
| `default_disk_size` | `int \| None` | `None` | Boot disk size in GB for every image baked on this runtime; wins over the image's `primary_disk_size`. A GCE instance's boot disk is exactly its image's disk. |
|
|
148
|
+
| `bake_preemptible` | `bool` | `False` | Bake on a preemptible (spot) VM; a preempted bake re-runs. |
|
|
149
|
+
| `state_configuration` | `str` | `DEFAULT` | Foreign key to a state backend. The storage and instance roots on this runtime resolve their backend as their own builder's `state_configuration`, else this value, else the default backend (`validate` resolves the bindings the same way). |
|
|
150
|
+
| `ssh_username` | `str` | `DEFAULT` | The bake's SSH user for images here that name none more specifically: step 4 of the bake-user order ([CONFIGURATION 5.1.1](../../docs/CONFIGURATION.md#511-the-bake-ssh-user)). Until stage 63 it overrode every entry. |
|
|
151
|
+
| `session_mechanism` | `str \| None` | `None` | `iap` is the only value. The model accepts any string; the builder's `session_mechanism()` raises on anything else the first time a run or `validate` asks for it. |
|
|
152
|
+
| `networking` | `GCPCloudNetworkingModel \| None` | `None` | Narrows the base type and makes it optional. Missing networking logs a warning at load. |
|
|
153
|
+
| `network_map` | `dict` | discovered | Not an init field. `{network: {"subnets": [...]}}` from the project. |
|
|
154
|
+
| `default_network` | `str \| None` | discovered | Not an init field. `default` when a network of that name exists. |
|
|
155
|
+
| `all_firewall_rules` | `dict[str, dict]` | discovered | Not an init field. `{rule_name: rule}` from the project. |
|
|
156
|
+
|
|
157
|
+
Base fields it inherits, and what this runtime does with them:
|
|
158
|
+
|
|
159
|
+
| Field | From | Default | Meaning here |
|
|
160
|
+
|---|---|---|---|
|
|
161
|
+
| `name`, `type`, `description`, `aliases` | `NameTyped` | `name`/`type` required | `type` is `gcloud`. |
|
|
162
|
+
| `executable`, `is_default`, `config`, `gitignore`, `tags` | `BuilderModel` | | `executable` (default `gcloud` on this model, stage 63 item 14) names the `executables` entry every gcloud call on this runtime runs through (`gcloud_binary()`), and `validate` refuses it undeclared; `is_default` makes this the runtime an unqualified reference resolves to. `tags` are accepted and read by nothing on this runtime: image labels are the lineage tags plus the image's own, and the instance's labels are the instance's own. For the same reason `validate`'s label-key check (stage 63, "Label keys" below) does not look at the runtime's own `tags`. |
|
|
163
|
+
| `region` | `CloudBuilderModel` | required | The `google` provider's `region`; the client config's `region`. |
|
|
164
|
+
| `credentials` | `RuntimeBuilderModel` | `CredentialsBase()` | Not narrowed. The base declares no fields and forbids extra keys, so `credentials:` accepts nothing; clients use Application Default Credentials. |
|
|
165
|
+
| `default_machine_type` | `RuntimeBuilderModel` | required | Machine type when neither an OS-builder runtime entry nor an image names one; always the instance module's `machine_type`. |
|
|
166
|
+
| `default_image_builder` | `RuntimeBuilderModel` | `DEFAULT` | The image builder an OS-builder runtime entry with `image_builder: default` resolves to. |
|
|
167
|
+
| `default_owners` | `RuntimeBuilderModel` | `None` | Appended to every vendor-image query's owner list (through the runtime entry's `get_owners()`), then mapped to projects. The lookup that reads it keyed the runtime by the wrong name until stage 63 item 22, so it was never found before 2026-09-25. |
|
|
168
|
+
| `default_config_username` | `RuntimeBuilderModel` | `None` | The bake SSH user when nothing more specific names one: step 5 of the bake-user order ([CONFIGURATION 5.1.1](../../docs/CONFIGURATION.md#511-the-bake-ssh-user)), read since stage 63 item 22. |
|
|
169
|
+
| `ephemeral` | `RuntimeBuilderModel` | `False` | See "Retention and the GCE cycle". The fixture's `gcloud-east1` is ephemeral. |
|
|
170
|
+
| `retention_keep` | `RuntimeBuilderModel` | `None` | Builds kept per series when the image declares no retention. |
|
|
171
|
+
| `on_failure`, `teardown_after` | `RuntimeBuilderModel` | `None` | Runtime-level defaults for ephemeral instances. |
|
|
172
|
+
|
|
173
|
+
It overrides `finalize()` to run `update_networking()` after the base
|
|
174
|
+
finalization, and `csis_name()` to return `gcloud`. It carries the class
|
|
175
|
+
attribute `type = "gcloud"`.
|
|
176
|
+
|
|
177
|
+
Not supported: any key inside `credentials:` (including a service-account
|
|
178
|
+
key file; `gcp_utils` can read `service_account_key_file` from a session
|
|
179
|
+
config, but no model field carries it, so that path is unreachable from
|
|
180
|
+
YAML), session mechanisms other than `iap`, and `availability_zones` as a
|
|
181
|
+
zone selector (use `zone`). Because no field can reach a key file, the
|
|
182
|
+
load-time warning for a missing project no longer suggests one (stage 63;
|
|
183
|
+
until 2026-09-25 it told the operator to configure a "service account key
|
|
184
|
+
file"): it says that no `project_id` is declared and none is in the
|
|
185
|
+
Application Default Credentials.
|
|
186
|
+
|
|
187
|
+
## The builder
|
|
188
|
+
|
|
189
|
+
`GCPCloudBuilder` extends `CloudBuilderBase` in
|
|
190
|
+
[builder_base_cloud.py](../base/src/cs_image_system/base/basic/builder_base_cloud.py),
|
|
191
|
+
an empty specialisation of `RuntimeBuilderBase` in
|
|
192
|
+
[builder_base_runtime.py](../base/src/cs_image_system/base/basic/builder_base_runtime.py).
|
|
193
|
+
The runtime builder writes no files of its own: `generate_items_during`
|
|
194
|
+
defers to the base and returns nothing. Everything it contributes reaches
|
|
195
|
+
`generated/` through the image, instance and release builders that call its
|
|
196
|
+
hooks. In the order a run reaches them:
|
|
197
|
+
|
|
198
|
+
| When | Hook | What it does |
|
|
199
|
+
|---|---|---|
|
|
200
|
+
| Configuration load | `GCPCloudBuilderModel.finalize()` | Runs `update_networking()` (network discovery below). |
|
|
201
|
+
| Resolution | `provider_specific_image_class()` | Returns `GcpProviderSpecificImage`. |
|
|
202
|
+
| Resolution | `query_provider_image(subconfig)` | Finds the vendor image for one OS-builder runtime entry (image query below). Returns `(image_name, owning_project, raw_result)` or `None`. |
|
|
203
|
+
| Resolution | `query_provider_image_by_id(subconfig, image_id)` | Stage 63: the image an entry pins with `image_id` (a GCE image NAME), searched in the entry's owner projects with the filter `name = "<id>"` instead of the query's filters. Same return shape; `None` when no owner project has an image of that name. |
|
|
204
|
+
| Image generation | `packer_source_type()` | `googlecompute`. |
|
|
205
|
+
| Image generation | `packer_source_blocks(...)` | The `source "googlecompute"` block for one image. |
|
|
206
|
+
| Image generation, `validate` | `session_mechanism()` | `iap` or `None`; raises on any other declared value. Also read by the launch-parameter record (`session`), by the instance builder (no public IP under `iap`) and by `validate`'s dead-end rule (a base image with no admin key AND no session mechanism). |
|
|
207
|
+
| Image generation | `session_agent_commands(os_family)` | Installs `google-guest-agent` when missing (`apt-get` for `debian`/`ubuntu`, `yum` otherwise) and enables it together with `sshd`/`ssh`. Empty when the mechanism is not `iap`. |
|
|
208
|
+
| Image generation | `session_verify_commands(os_family)` | Assertions that the guest agent is present and enabled. Empty when the mechanism is not `iap`. |
|
|
209
|
+
| `validate` | `label_key_problem(key)` | Implemented by this runtime (stage 63; the base returns `None`, because AWS tags take any key). Returns a reason when `gce_label(key)` does not start with a lowercase letter: `GCE label keys must start with a lowercase letter; '<key>' becomes '<label>'`; otherwise `None`. `validate`'s `check_label_keys` asks it for every tag key declared on an image, instance or storage that lands on this runtime (see "Label keys" below). |
|
|
210
|
+
| Image generation | `default_bake_user(family)`, `bake_ssh_username(image)`, `one_bake_user_per_chain()` | `default_bake_user` is `packer` for every family (the last step of the bake-user order). `bake_ssh_username(image)` is the RESOLVED user for that image, the one the packer source bakes as, and the ansible provisioner names it (stage 63 item 23; it read only the runtime's field and said `packer` otherwise). `one_bake_user_per_chain()` is True, so `validate` refuses a chain whose images resolve to different users (finding 43). |
|
|
211
|
+
| Image generation | `bake_finalize_commands(os_family)` | The last provisioner of every bake here: adds the build user to `google-sudoers` when that group exists, so the guest agent's first-boot user cleanup succeeds and metadata SSH keys get provisioned. Emitted whatever the session mechanism. |
|
|
212
|
+
| After a bake | `build_id_from_artifact(artifact_id)` | The packer manifest's `artifact_id` is the image name itself; returned unchanged. |
|
|
213
|
+
| After a bake | `retag_image(image_id, tags)` | The packer builder stamps the recorded `csis_parent` and `csis_fingerprint` back onto the image (zero-drift-report). |
|
|
214
|
+
| Instance verification | `serial_console(name)` | Serial port 1 output of the instance, read-only. |
|
|
215
|
+
| Instance verification | `verify_instance(name, expected_build, expect_mounts, timeout)` | Polls the serial console every 15 s until the guest agent reports the startup scripts finished (`Finished running startup scripts`, or the older `startup-script exit status 0`) or an explicit failure line appears, compares the booted image with the expected build, and counts the kernel's `XFS (...): Ending clean mount` lines against the declared data disks. |
|
|
216
|
+
| Instance operations | `run_session_command(name, script, timeout)` | `<declared gcloud> compute ssh <name> --project <p> --zone <z> --tunnel-through-iap --quiet --command <script>` as the operator's gcloud account. Refused with `declares no session_mechanism` before anything runs when the runtime declares none (stage 63 item 14; it used to tunnel anyway and fail as IAP's problem). Used by `verify instance` (post-bake tests), `unmount storage`, the alias writer and the reachability wait. |
|
|
217
|
+
| Instance operations | `can_query_instance_power_state()`, `query_instance_power_state(name)` | `True`; GCE's `Instance.status` mapped onto the system's vocabulary: `PROVISIONING`/`STAGING` starting, `RUNNING` running, `STOPPING`/`SUSPENDING` stopping, `SUSPENDED` suspended, `TERMINATED` **stopped** (on GCE it means the machine exists and can be started; a deleted instance is simply not found and answers `absent`), `REPAIRING` unknown. `None` means the runtime could not answer, never that the machine is off. |
|
|
218
|
+
| Instance operations | `can_set_instance_power_state()`, `start_instance(name)`, `stop_instance(name)` | `True`; `instances.start`/`instances.stop`, waiting on the operation, then re-reading the state. Only `running_for_task` calls them (a bounded task that needs a running machine, which puts it back). |
|
|
219
|
+
| Instance operations | `can_query_instance_identity()`, `query_instance_identity(name)` | `True`; the numeric instance id and the internal hostname (a custom `hostname` when set, else `<name>.c.<project>.internal`). Feeds the generation ledger and the provider-alias writer. |
|
|
220
|
+
| `empty --runtime` | `inventory()` | Instances and disks in the zone and every custom image in the project (Compute API), plus the project's buckets (`gcloud storage buckets list`). |
|
|
221
|
+
| Retention | `dispose_image(build_id)` | Deletes the image named by the build id and waits for the operation. `NotFound` returns `False` (already gone). |
|
|
222
|
+
| State reconciliation | `retag_image(image_id, tags)` | `setLabels` on one of the system's own images: the current labels merged with the given tags, both sides passed through `gce_label()`, with the image's label fingerprint. Also the hook behind `lineage relabel` and `restamp`. |
|
|
223
|
+
| State query | `query_images(series)` | The images in the project whose `csis_series` label equals `gce_label(<s>)` for one of the series `s` asked about: `{image_id, name, state, created, tags}` (labels as tags). An empty `series` list keeps every image that carries a `csis_series` label. Stage 63: until 2026-09-25 the argument was ignored and every csis-labelled image in the project came back. The state query asks for every recorded series plus every declared OS builder and image; `lineage relabel` asks for the series of the builds it relabels. |
|
|
224
|
+
| State query | `can_query_instance_boot_image()`, `query_instance_boot_image(name)` | `True`; the instance's boot disk's `source_image` basename, or `None` when the instance, disk, zone or project cannot be resolved. Read-only, never fatal. |
|
|
225
|
+
| Release | `release_commands(build_id, tags)` | Base default: none. A release on GCE is recorded and relabelled through `retag_image`, not through a CLI command. |
|
|
226
|
+
|
|
227
|
+
`gce_name()` and `gce_label()` in
|
|
228
|
+
[gcp_packer_source.py](src/cs_image_system/gcloud_runtime/gcp_packer_source.py)
|
|
229
|
+
are the one place every csis name and tag is mapped to GCE's constrained
|
|
230
|
+
forms: names to `[a-z]([-a-z0-9]*[a-z0-9])?` (63 characters, prefixed `i-`
|
|
231
|
+
when they would start with a digit), labels to lowercase `[a-z0-9_-]`.
|
|
232
|
+
Every hook that names a GCE resource goes through them.
|
|
233
|
+
|
|
234
|
+
### Label keys
|
|
235
|
+
|
|
236
|
+
`gce_label()` lowercases a key and replaces what GCE cannot keep, but it
|
|
237
|
+
cannot invent a first letter, and GCE requires a label KEY to start with a
|
|
238
|
+
lowercase letter. A tag key such as `2024`, `-env` or `_x` therefore
|
|
239
|
+
becomes a label key GCE refuses. Since stage 63 the runtime says so through
|
|
240
|
+
`label_key_problem(key)`, and `validate` runs `check_label_keys` in
|
|
241
|
+
[validate.py](../base/src/cs_image_system/base/commands/validate.py) over
|
|
242
|
+
every declared tag key that becomes a label here: an image's tags for each
|
|
243
|
+
runtime it bakes on, an instance's tags on its runtime, and a storage's
|
|
244
|
+
tags on its storage builder's runtime. A key GCE cannot take is refused,
|
|
245
|
+
naming who declared it:
|
|
246
|
+
|
|
247
|
+
```text
|
|
248
|
+
<image|instance|storage> '<name>': tag key '<key>' on runtime <rt>: GCE label keys must start with a lowercase letter; '<key>' becomes '<label>'; rename the key
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
The key is never renamed silently: the operator renames it in the YAML.
|
|
252
|
+
Before stage 63 (until 2026-09-25) such a key passed `validate` and
|
|
253
|
+
generation and was refused by GCE at apply. Label VALUES are not checked.
|
|
254
|
+
The runtime's own `tags` are not checked because nothing on this runtime
|
|
255
|
+
reads them.
|
|
256
|
+
|
|
257
|
+
### Credentials
|
|
258
|
+
|
|
259
|
+
`GCPCloudBuilderModel.self_to_gcp_client_config()` returns `project`,
|
|
260
|
+
`region` and `zone` (and `credentials`, which is always empty). The Compute
|
|
261
|
+
clients in [gcp_utils.py](src/cs_image_system/gcloud_runtime/gcp_utils.py)
|
|
262
|
+
are built without explicit credentials, so the Google client library's
|
|
263
|
+
Application Default Credentials apply: `gcloud auth application-default
|
|
264
|
+
login` on a workstation, `GOOGLE_APPLICATION_CREDENTIALS`, or the runner's
|
|
265
|
+
attached service account. `run_session_command` and `inventory()` shell out
|
|
266
|
+
to `gcloud`, which uses the operator's active gcloud account. Nothing in the
|
|
267
|
+
tree names a key.
|
|
268
|
+
|
|
269
|
+
### Network discovery at load
|
|
270
|
+
|
|
271
|
+
`update_networking()` in
|
|
272
|
+
[gcp_runtime_models.py](src/cs_image_system/gcloud_runtime/gcp_runtime_models.py)
|
|
273
|
+
first calls `resolve_project()`; with no project it logs a warning and
|
|
274
|
+
returns, so a tree without `project_id` loads but cannot resolve images or
|
|
275
|
+
bake. With a project it calls `get_network_map_and_default_network()`, which
|
|
276
|
+
lists networks, routes, firewall rules and an aggregated subnetwork list once
|
|
277
|
+
each and builds:
|
|
278
|
+
|
|
279
|
+
- `network_map`: for every network, a `subnets` list of `{subnet_id, self_link, region, cidr, is_public, tags, config}`. GCE routes are network-scoped, so a network is public when it has a `0.0.0.0/0` or `::/0` route whose next hop is `default-internet-gateway`, and every subnetwork in it inherits that.
|
|
280
|
+
- `default_network`: `default` when a network of that name exists.
|
|
281
|
+
- `all_firewall_rules`: every rule by name.
|
|
282
|
+
|
|
283
|
+
Then it validates the declaration: `network: default` becomes the network
|
|
284
|
+
named `default` (an error when the project has none); the network must
|
|
285
|
+
exist; every declared `subnet_id` must match a subnetwork by name, full self
|
|
286
|
+
link, or `projects/...` path suffix; every `network_tags` entry that no
|
|
287
|
+
firewall rule targets logs a warning.
|
|
288
|
+
|
|
289
|
+
### The image query
|
|
290
|
+
|
|
291
|
+
`query_provider_image()` takes one OS-builder runtime entry
|
|
292
|
+
(`OSBuilderBaseImageBuilderSubconfig` in
|
|
293
|
+
[os_builder_runtime_config.py](../base/src/cs_image_system/base/models/os_builder_runtime_config.py)).
|
|
294
|
+
`remap_for_image_query()` builds a Compute `list()` query:
|
|
295
|
+
|
|
296
|
+
- `projects` is the entry's `get_owners()` (the OS builder's owners, the runtime's `default_owners`, the entry's own), resolved by `resolve_projects()`: `self` becomes the session project; the aliases `debian`, `ubuntu`, `rhel`, `centos`, `rocky`, `fedora`, `suse`, `windows`, `cos` and `google` become the public image projects (`debian-cloud`, `ubuntu-os-cloud`, `rhel-cloud`, ...); AWS-only aliases (`amazon`, `aws-marketplace`, `aws-backup-vault`) and anything that is not a valid project id (an AWS account number, for example) are dropped; a valid project id such as `almalinux-cloud` is kept as is.
|
|
297
|
+
- `query.filters` become a filter expression: `name`, `family`, `state`/`status`, `architecture`, `description` and `creation_date` map to image fields; `state: available` becomes `status = "READY"` (forced when absent); architectures map to `X86_64`/`ARM64`; `tag:<key>` becomes `labels.<key>`; AWS-only keys (`root_device_type`, `virtualization_type`, `block_device_mapping.*`, ...) are dropped instead of excluding every result; keys the map does not know go to a post-query exact-match filter.
|
|
298
|
+
- A `family` key with no filter expression resolves through `getFromFamily` instead.
|
|
299
|
+
|
|
300
|
+
`query_image()` lists every candidate project, applies the post-query
|
|
301
|
+
filter, and returns the newest `creation_timestamp` with the owning
|
|
302
|
+
`project` added. The builder returns `(name, project, raw)`; the base
|
|
303
|
+
registers a `GcpProviderSpecificImage` in the resolved state.
|
|
304
|
+
|
|
305
|
+
When the entry declares `image_id` (stage 63), the base's resolve step
|
|
306
|
+
calls `query_provider_image_by_id(entry, image_id)` instead. A GCE image
|
|
307
|
+
is named uniquely within its project, so the id is an image name and is
|
|
308
|
+
searched where the vendor query would search: `remap_for_image_query()`
|
|
309
|
+
builds the same `projects` list (the entry's merged owners, or a
|
|
310
|
+
`query.owners`), then the builder drops the query's `family` and `filter`
|
|
311
|
+
and sets the filter to `name = "<id>"`; the post-query filter is not
|
|
312
|
+
applied, and a match whose status is not `READY` is refused by name
|
|
313
|
+
(stage 67: `GCP runtime <rt>: pinned image '<id>' is <status>, not READY; a
|
|
314
|
+
bake from it would fail`). The owning project
|
|
315
|
+
is read from the result as for the query (else `self`). No match returns
|
|
316
|
+
`None`, and the base stops resolution with `OS builder <name>: image_id
|
|
317
|
+
'<id>' on runtime <rt> is not known to the provider`. Pinning makes the
|
|
318
|
+
base bake reproducible: the same vendor image every time, where the query
|
|
319
|
+
takes the newest match (a vendor family publishes a new image often).
|
|
320
|
+
|
|
321
|
+
`GcpProviderSpecificImage.get_query_assets()` describes the image as a
|
|
322
|
+
generic query (`filters = { name = "<name>" }` plus `owners` when resolved;
|
|
323
|
+
`owners = ["self"]`, `filters = { name = "<pattern>*" }`, `most_recent`
|
|
324
|
+
when deferred). The packer source does not use these assets: GCE's image
|
|
325
|
+
families are the series, so `googlecompute_source()` writes `source_image`
|
|
326
|
+
(a resolved or pinned parent) or `source_image_family` (a deferred parent)
|
|
327
|
+
directly.
|
|
328
|
+
|
|
329
|
+
[tests/test_gcp_utils.py](tests/test_gcp_utils.py) and
|
|
330
|
+
[tests/test_gcp_provider_specific_image.py](tests/test_gcp_provider_specific_image.py)
|
|
331
|
+
pin the remap, the project resolution, the newest-wins rule and the asset
|
|
332
|
+
shapes without network access.
|
|
333
|
+
|
|
334
|
+
### Image families
|
|
335
|
+
|
|
336
|
+
Every bake sets `image_family = gce_name(<series>)`, so "the latest build of
|
|
337
|
+
series X" is GCE's own `source_image_family = X` lookup. A pinned parent
|
|
338
|
+
bakes from `source_image = gce_name(<build id>)`; the build id is the image
|
|
339
|
+
name. `query_images()` reads the `csis_series` label rather than the family
|
|
340
|
+
so the state query compares the same facts on both clouds, and since stage
|
|
341
|
+
63 it filters on that label: only images whose `csis_series` equals
|
|
342
|
+
`gce_label(<series>)` for a series it was asked about come back (the label
|
|
343
|
+
carries the sanitised form, so the comparison is made in that form).
|
|
344
|
+
|
|
345
|
+
### IAP
|
|
346
|
+
|
|
347
|
+
With `session_mechanism: iap` the packer source gets `use_iap = true` and
|
|
348
|
+
`iap_tunnel_launch_wait = 120`, so packer reaches the build VM through the
|
|
349
|
+
IAP tunnel instead of its external address; the ephemeral external IP stays
|
|
350
|
+
for package egress only. `run_session_command` tunnels the same way, and the
|
|
351
|
+
GCE instance builder launches instances with `public_ip = false`. The
|
|
352
|
+
operator prerequisites are outside the tree: a firewall rule allowing
|
|
353
|
+
`35.235.240.0/20` to port 22 and `roles/iap.tunnelResourceAccessor` for the
|
|
354
|
+
account that opens tunnels. The bake installs `google-guest-agent` so
|
|
355
|
+
metadata SSH keys are provisioned at first boot.
|
|
356
|
+
|
|
357
|
+
### The state query and instance hooks
|
|
358
|
+
|
|
359
|
+
`query_images`, `query_instance_boot_image`, `query_instance_power_state`,
|
|
360
|
+
`query_instance_identity`, `inventory`, `serial_console`, `retag_image`,
|
|
361
|
+
`dispose_image`, `start_instance` and `stop_instance` use `compute_v1`
|
|
362
|
+
clients built from the session config. The state query in
|
|
363
|
+
[state_query.py](../base/src/cs_image_system/base/state_query.py) reads
|
|
364
|
+
`query_images` to classify `missing`, `foreign` and `changed` images
|
|
365
|
+
(asking for every recorded series and every declared OS builder and image,
|
|
366
|
+
so since stage 63 an image labelled with a series the tree neither records
|
|
367
|
+
nor declares is not listed on GCE; until 2026-09-25 every csis-labelled
|
|
368
|
+
image in the project came back whatever was asked), asks
|
|
369
|
+
`query_instance_boot_image` because `can_query_instance_boot_image()` is
|
|
370
|
+
`True`, and asks `query_instance_power_state` before calling a silent boot
|
|
371
|
+
probe "unavailable": a stopped machine is a `note`, not drift.
|
|
372
|
+
`verify_instance` is the deferred verification step of the instance
|
|
373
|
+
lifecycle, run inside `running_for_task` (a stopped instance is started for
|
|
374
|
+
the check and stopped again afterwards; one that cannot be started is a
|
|
375
|
+
SKIP, recorded as such). `inventory()` feeds `empty --runtime`, which
|
|
376
|
+
subtracts the declared storages' cloud names and the released builds and
|
|
377
|
+
reports what is left.
|
|
378
|
+
|
|
379
|
+
### Retention and the GCE cycle
|
|
380
|
+
|
|
381
|
+
The closing `retention` lifecycle in
|
|
382
|
+
[retention.py](../base/src/cs_image_system/base/retention.py) runs
|
|
383
|
+
`dispose image --retention` at execution time. For each series on this
|
|
384
|
+
runtime the number of builds kept is, in order: `0` when the runtime is
|
|
385
|
+
`ephemeral`, the image's `retention.keep`, the runtime's `retention_keep`,
|
|
386
|
+
else everything. Builds beyond that are disposed through `dispose_image()`,
|
|
387
|
+
except released builds and builds an instance is pinned to or was launched
|
|
388
|
+
from, which are reported as retention debt and kept. Declared storages are
|
|
389
|
+
never touched by retention.
|
|
390
|
+
|
|
391
|
+
The fixture's `gcloud-east1` declares `ephemeral: true`, and this is the
|
|
392
|
+
GCE cycle: a run bakes its images, launches its ephemeral instance, verifies
|
|
393
|
+
it from the serial console, tears it down, and the closing retention step
|
|
394
|
+
disposes of every image baked on the runtime. What persists across runs is
|
|
395
|
+
the declared storages (a persistent disk and a bucket in the fixture) and
|
|
396
|
+
any released build. `empty --runtime gcloud-east1` proves the runtime is in
|
|
397
|
+
that state afterwards.
|
|
398
|
+
|
|
399
|
+
### Cost posture
|
|
400
|
+
|
|
401
|
+
A successful run on an ephemeral GCE runtime leaves standing: the declared
|
|
402
|
+
storages and any released build. It may also leave standing: a
|
|
403
|
+
non-ephemeral instance, and an ephemeral instance whose verification failed
|
|
404
|
+
under `on_failure: keep` (`teardown_after` lets a later run tear it down).
|
|
405
|
+
`bake_preemptible: true` bakes on spot VMs; `default_disk_size` fixes the
|
|
406
|
+
boot disk of every image, and therefore of every instance, at a small size.
|
|
407
|
+
On a non-ephemeral GCE runtime the AWS rules apply: images stand until
|
|
408
|
+
retention or an explicit `dispose image` removes them.
|
|
409
|
+
|
|
410
|
+
## Emission
|
|
411
|
+
|
|
412
|
+
The plugin's output is visible in the golden emission under
|
|
413
|
+
[tests/fixtures/v2_golden/generated](../../tests/fixtures/v2_golden/generated).
|
|
414
|
+
|
|
415
|
+
- [pckr-gce-ans-image-generation-source-basic-rh-10-block-000.pkr.hcl](../../tests/fixtures/v2_golden/generated/base-image/pckr-gce-ans/image-generation/block-000/pckr-gce-ans-image-generation-source-basic-rh-10-block-000.pkr.hcl):
|
|
416
|
+
the `source "googlecompute"` block written by `googlecompute_source()`:
|
|
417
|
+
`project_id`, `zone`, `source_image` and `source_image_project_id` (a
|
|
418
|
+
resolved vendor image and its project), `image_name` (`gce_name` of the
|
|
419
|
+
final name), `image_family` (the series), `image_labels` (the merged
|
|
420
|
+
lineage tags, sanitised), `machine_type` (the runtime entry's, else the
|
|
421
|
+
runtime default), `service_account_email`, `use_iap` and
|
|
422
|
+
`iap_tunnel_launch_wait` (the mechanism is `iap`), `preemptible`,
|
|
423
|
+
`disk_size` (from `lineage.bake_disk_size`, the one rule the input
|
|
424
|
+
fingerprint also hashes (stage 63); here the runtime's
|
|
425
|
+
`default_disk_size`), `ssh_username` and
|
|
426
|
+
`subnetwork` (the default subnet). `network` is emitted only when it
|
|
427
|
+
resolves to a non-default name, and `tags` only when `network_tags` are
|
|
428
|
+
declared.
|
|
429
|
+
- [pckr-gce-ans-image-generation-block-000-build.pkr.hcl](../../tests/fixtures/v2_golden/generated/base-image/pckr-gce-ans/image-generation/block-000/pckr-gce-ans-image-generation-block-000-build.pkr.hcl):
|
|
430
|
+
the `# debug session mechanism (iap)` shell provisioner
|
|
431
|
+
(`session_agent_commands`), the `# verify: guest agent baked for IAP
|
|
432
|
+
sessions` lines inside the in-bake verification (`session_verify_commands`),
|
|
433
|
+
and the closing `# runtime bake finalization (gcloud-east1)` provisioner
|
|
434
|
+
(`bake_finalize_commands`). The `googlecompute` entry in the neighbouring
|
|
435
|
+
`-plugins.pkr.hcl` comes from the image builder's `required_plugins`.
|
|
436
|
+
- [tofu-gce-instance-generation.tf](../../tests/fixtures/v2_golden/generated/instance-image/tofu-gce/instance-generation/tofu-gce-instance-generation.tf)
|
|
437
|
+
and
|
|
438
|
+
[tofu-gce-instance-generation-instance-gce_test.tf](../../tests/fixtures/v2_golden/generated/instance-image/tofu-gce/instance-generation/tofu-gce-instance-generation-instance-gce_test.tf):
|
|
439
|
+
written by the GCE instance plugin from this model. `provider "google" {
|
|
440
|
+
project, region, zone, alias }`; the instance module's `machine_type`
|
|
441
|
+
(the runtime default), `zone`, `subnetwork` (the default subnet),
|
|
442
|
+
`image_family` (the series), `public_ip = false` under `iap` and, when
|
|
443
|
+
declared, `network_tags`. The `.tfbackend.hcl` beside them is an S3
|
|
444
|
+
backend written by the state-backend plugin; GCE roots keep their state
|
|
445
|
+
there.
|
|
446
|
+
- Runtime facts: `cs-image-system runtime describe gcloud-east1`
|
|
447
|
+
([runtime_facts.py](../base/src/cs_image_system/base/commands/runtime_facts.py))
|
|
448
|
+
reports `runtime`, `type`, `project_id`, `zone`, `region`,
|
|
449
|
+
`default_machine_type`, `ephemeral`, `retention_keep`, the `images`,
|
|
450
|
+
`storages`, `instances` and `ephemeral_instances` declared on it, the
|
|
451
|
+
`builders` bound to it and its `emission` directories;
|
|
452
|
+
`empty --runtime gcloud-east1` adds the live `inventory` and the
|
|
453
|
+
`leftovers`. Commands, not files.
|
|
454
|
+
|
|
455
|
+
## Example configuration
|
|
456
|
+
|
|
457
|
+
From
|
|
458
|
+
[tests/fixtures/config/cfg/runtime-builders.yml](../../tests/fixtures/config/cfg/runtime-builders.yml):
|
|
459
|
+
|
|
460
|
+
```yaml
|
|
461
|
+
runtime_builders:
|
|
462
|
+
- name: gcloud-east1
|
|
463
|
+
aliases:
|
|
464
|
+
- gcp
|
|
465
|
+
- google
|
|
466
|
+
- us-east1
|
|
467
|
+
type: gcloud
|
|
468
|
+
# Credentials come from the environment only (Application Default
|
|
469
|
+
# Credentials or GOOGLE_APPLICATION_CREDENTIALS); nothing is declared here.
|
|
470
|
+
project_id: csis-sandbox
|
|
471
|
+
zone: us-east1-b
|
|
472
|
+
session_mechanism: iap
|
|
473
|
+
# build VMs run as the least-privilege runner account (the packer source
|
|
474
|
+
# carries it; the instance module attaches no service account)
|
|
475
|
+
service_account_email: csis-runner@csis-sandbox.iam.gserviceaccount.com
|
|
476
|
+
# bake disk (= every instance's boot disk) at the vendor image's 10 GB floor
|
|
477
|
+
default_disk_size: 10
|
|
478
|
+
bake_preemptible: true
|
|
479
|
+
# nothing baked here survives a successful run; declared storages persist
|
|
480
|
+
ephemeral: true
|
|
481
|
+
tags:
|
|
482
|
+
Project: csis-sandbox
|
|
483
|
+
Environment: Development
|
|
484
|
+
region: us-east1
|
|
485
|
+
default_machine_type: e2-micro
|
|
486
|
+
default_image_builder: pckr-gce-ans
|
|
487
|
+
networking:
|
|
488
|
+
network: default
|
|
489
|
+
subnets:
|
|
490
|
+
- name: subnet1
|
|
491
|
+
subnet_id: projects/csis-sandbox/regions/us-east1/subnetworks/default
|
|
492
|
+
is_default: true
|
|
493
|
+
public: true
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
## Prerequisites and integration
|
|
497
|
+
|
|
498
|
+
Everything below exists outside the system; the plugin creates none of it
|
|
499
|
+
and generates no IAM change. For each item: how the plugin finds it.
|
|
500
|
+
|
|
501
|
+
**A GCP project with the Compute Engine API enabled.** Found through
|
|
502
|
+
`project_id` on the runtime entry; it is the `project` of every Compute
|
|
503
|
+
client call, the packer source's `project_id`, the `google` provider's
|
|
504
|
+
`project` and the project `self` resolves to in an owner list. Without it
|
|
505
|
+
the tree loads with a warning and nothing that touches GCE works. The
|
|
506
|
+
Compute API must be enabled for the project: the first thing the plugin
|
|
507
|
+
does with it is list its networks at load.
|
|
508
|
+
|
|
509
|
+
**Application Default Credentials (ADC) for a principal that can act in
|
|
510
|
+
that project.** The model declares no credentials (`credentials:` accepts
|
|
511
|
+
no key), so every `compute_v1` client is built with no explicit credentials
|
|
512
|
+
and the Google auth library's default chain applies, in its order:
|
|
513
|
+
`GOOGLE_APPLICATION_CREDENTIALS` naming a credentials file, else
|
|
514
|
+
`~/.config/gcloud/application_default_credentials.json` (written by `gcloud
|
|
515
|
+
auth application-default login`), else the metadata server of an attached
|
|
516
|
+
service account. The operator's form, from
|
|
517
|
+
[docs/OPERATIONS.md](../../docs/OPERATIONS.md) ("Credentials and sessions"):
|
|
518
|
+
|
|
519
|
+
```sh
|
|
520
|
+
gcloud auth application-default login --impersonate-service-account=csis-runner@<project>.iam.gserviceaccount.com
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
which needs `roles/iam.serviceAccountTokenCreator` on the runner for the
|
|
524
|
+
operator's own Google user. `cs-image-system preflight` and every `run` and
|
|
525
|
+
`state` check that the ADC file exists at the same path (presence only: an
|
|
526
|
+
impersonated or authorized-user ADC refreshes itself, so no expiry is
|
|
527
|
+
readable) and print one `session:` line for it. In CI the
|
|
528
|
+
`google-github-actions/auth` action turns `GCP_WORKLOAD_IDENTITY_PROVIDER`
|
|
529
|
+
and `GCP_SERVICE_ACCOUNT` into ADC (read-only in CI: the GCE runtime is out
|
|
530
|
+
of CI by the cost decision, and the perform job only records and guards).
|
|
531
|
+
|
|
532
|
+
What that principal must be allowed to do, by hook: list networks, routes,
|
|
533
|
+
firewalls and subnetworks (load); list and get images in the project and in
|
|
534
|
+
the public image projects the owner aliases name (`resolve`, the state
|
|
535
|
+
query, `lineage relabel`, `empty`); `getFromFamily` (a family query);
|
|
536
|
+
`images.delete` (retention, `dispose image`) and `images.setLabels` (the
|
|
537
|
+
post-bake retag, `relabel`, `restamp`); `instances.get`, `instances.list`,
|
|
538
|
+
`instances.getSerialPortOutput`, `instances.start`, `instances.stop` and
|
|
539
|
+
`disks.get`/`disks.list` in the zone (verification, the state query,
|
|
540
|
+
`empty`, the power-state and identity probes). Bakes run as this principal
|
|
541
|
+
too: packer creates the build VM and the image, and attaching
|
|
542
|
+
`service_account_email` to the build VM needs `roles/iam.serviceAccountUser`
|
|
543
|
+
on that account (found live: a least-privilege runner cannot `actAs` the
|
|
544
|
+
default compute service account, which is why the field exists).
|
|
545
|
+
|
|
546
|
+
**A service account for build VMs.** Optional. Found through
|
|
547
|
+
`service_account_email`; emitted into the packer source only. Without it
|
|
548
|
+
packer attaches the project's default compute service account.
|
|
549
|
+
|
|
550
|
+
**A VPC network and a subnetwork in the project.** Found through
|
|
551
|
+
`networking.network` (`default` means the network named `default`; the
|
|
552
|
+
project must have one) and `networking.subnets[].subnet_id`; both are
|
|
553
|
+
checked against the project at every load, so a load needs the API and the
|
|
554
|
+
credentials above even for a dry run or `validate`. The subnetwork's region
|
|
555
|
+
should be the runtime's `region` and contain the runtime's `zone`; the
|
|
556
|
+
plugin does not check that, GCE refuses the VM at bake or launch if it is
|
|
557
|
+
wrong. Under `iap` instances get no public IP, so anything an instance must
|
|
558
|
+
reach on the internet needs a route the operator provides (the bake VM keeps
|
|
559
|
+
an ephemeral external IP for package egress).
|
|
560
|
+
|
|
561
|
+
**IAP TCP forwarding, when `session_mechanism: iap`.** Three things the
|
|
562
|
+
tree never creates, from OPERATIONS.md ("IAP sessions on GCE"): a firewall
|
|
563
|
+
rule allowing `tcp:22` from `35.235.240.0/20` on the instances' network
|
|
564
|
+
(`network_tags` on the runtime select tagged rules; a tag no rule targets
|
|
565
|
+
is warned about at load); `roles/iap.tunnelResourceAccessor` for the runner
|
|
566
|
+
service account, because bakes tunnel too (`use_iap = true`); and the same
|
|
567
|
+
role for each operator's own Google user, because `run_session_command`
|
|
568
|
+
runs `gcloud compute ssh --tunnel-through-iap` as the operator's active
|
|
569
|
+
gcloud account, not as the impersonated ADC. The operator's SSH key must be
|
|
570
|
+
agent-loaded or passphrase-less: nothing in a run can prompt (found live
|
|
571
|
+
2026-09-09). The images bake the Google guest agent so metadata SSH keys
|
|
572
|
+
are provisioned at first boot; the base image's package manager must be
|
|
573
|
+
able to install `google-guest-agent` (`apt-get` on `debian`/`ubuntu`,
|
|
574
|
+
`yum` elsewhere; the vendor images carry it already).
|
|
575
|
+
|
|
576
|
+
**The `gcloud` CLI.** Two hooks shell out to it: `run_session_command`
|
|
577
|
+
(`gcloud compute ssh`) and `inventory()` (`gcloud storage buckets list`,
|
|
578
|
+
so `empty --runtime` needs `storage.buckets.list` for the gcloud account).
|
|
579
|
+
Both run the `binary` of the `executables` entry the runtime's
|
|
580
|
+
`executable` names (default `gcloud`; stage 63 item 14, before which they
|
|
581
|
+
called `gcloud` from `PATH`), so the entry must be declared: `validate`
|
|
582
|
+
refuses the runtime without it. Its version is checked
|
|
583
|
+
when it is declared under `executables:` with `name: gcloud` (the checker
|
|
584
|
+
is registered under that name): the fixture declares
|
|
585
|
+
`binary: /usr/local/bin/gcloud` and `version: ">=500"`, the Google Cloud
|
|
586
|
+
SDK version; `validate` and every run refuse when the binary is missing,
|
|
587
|
+
the version cannot be parsed from `gcloud --version`, or it is below the
|
|
588
|
+
floor. Packer's `googlecompute` plugin is a prerequisite of the packer
|
|
589
|
+
image builder (`required_plugins` on the `packer-gce` builder), not of this
|
|
590
|
+
package.
|
|
591
|
+
|
|
592
|
+
**A live AWS session as well.** The GCE roots keep their terraform state
|
|
593
|
+
in the S3 backend by standing decision, and the tree's AWS runtimes
|
|
594
|
+
validate their networking at the same load, so a GCE lifecycle, `empty
|
|
595
|
+
--runtime gcloud-east1` and every configuration load need the AWS
|
|
596
|
+
profile's session too (found live 2026-09-08: an expired SSO session
|
|
597
|
+
stopped GCE work with GCP fully authenticated). See
|
|
598
|
+
[docs/OPERATIONS.md](../../docs/OPERATIONS.md) for the profile.
|
|
599
|
+
|
|
600
|
+
**Python dependencies**, from [pyproject.toml](pyproject.toml):
|
|
601
|
+
`google-cloud-compute>=1.49`, `google-api-core>=2.31`, `google-auth>=2.55`,
|
|
602
|
+
`python-hcl2>=8.1`, `pydantic>=2.13`, and the system's own
|
|
603
|
+
`cs-image-system-system` and `cs-image-system-hashicorp-utils` at the same
|
|
604
|
+
version. Python 3.13 or later.
|
|
605
|
+
|
|
606
|
+
Environment variables the plugin or its checks read: `GOOGLE_APPLICATION_CREDENTIALS`
|
|
607
|
+
(optional; the ADC file), and in CI `GCP_WORKLOAD_IDENTITY_PROVIDER` and
|
|
608
|
+
`GCP_SERVICE_ACCOUNT` (read by the auth action, not by the plugin). The
|
|
609
|
+
preflight also refuses any `GOOGLE_*` variable that is set but empty.
|
|
610
|
+
|
|
611
|
+
## Configuration reference
|
|
612
|
+
|
|
613
|
+
The YAML this plugin owns is one entry of `runtime_builders:` in
|
|
614
|
+
`cfg/runtime-builders.yml` with `type: gcloud`; the manual's section is
|
|
615
|
+
[docs/CONFIGURATION.md](../../docs/CONFIGURATION.md) 4.4. Every field the
|
|
616
|
+
plugin's model accepts, and who reads it:
|
|
617
|
+
|
|
618
|
+
| Field | Type | Default | Meaning |
|
|
619
|
+
|---|---|---|---|
|
|
620
|
+
| `name` | str | required | the runtime's name; other builders name it through `runtime:`; the `# runtime bake finalization (<name>)` provisioner carries it |
|
|
621
|
+
| `type` | str | required | `gcloud` |
|
|
622
|
+
| `description` | str or null | null | free text; accepted, not read by this plugin |
|
|
623
|
+
| `aliases` | list[str] | `[]` | extra names the runtime answers to |
|
|
624
|
+
| `executable` | str or null | `gcloud` | the `executables` entry whose `binary` every gcloud call on this runtime runs (the session command, `inventory()`, the tf-gcp storage lookups and generated scripts); `validate` refuses a name that is not declared. Stage 63 item 14: until 2026-09-25 every call was a bare `gcloud` from `PATH` |
|
|
625
|
+
| `is_default` | bool | false | the runtime an unqualified runtime reference resolves to |
|
|
626
|
+
| `config` | mapping | `{}` | free-form; accepted, not read |
|
|
627
|
+
| `gitignore` | list[str] | `[]` | accepted, not read by this plugin |
|
|
628
|
+
| `tags` | mapping[str, str] | `{}` | accepted, not read: no image label, instance label or storage label comes from the runtime's tags |
|
|
629
|
+
| `region` | str | required | the `google` provider's `region`; the client config's `region` |
|
|
630
|
+
| `project_id` | str or null | null | the project (see above); `runtime describe` reports it |
|
|
631
|
+
| `zone` | str or null | null | the zone of build VMs and instances and the scope of every zonal query; `runtime describe` reports it |
|
|
632
|
+
| `service_account_email` | str or null | null | the packer source's `service_account_email`; instances are not given it |
|
|
633
|
+
| `default_disk_size` | int or null | null | the packer source's `disk_size` in GB, the first step of `lineage.bake_disk_size`; null falls back, for a base image, to its OS builder entry's `default_primary_disk_size` and then the OS builder's, and for an instance image to its `primary_disk_size` (stage 63) |
|
|
634
|
+
| `bake_preemptible` | bool | false | `preemptible = true` on the packer source |
|
|
635
|
+
| `state_configuration` | str | `default` | the state backend rung between a root's own and the default backend |
|
|
636
|
+
| `ssh_username` | str | `default` | the bake's ssh user for images that name none more specifically (step 4 of the bake-user order, CONFIGURATION 5.1.1); with nothing named anywhere, `packer` |
|
|
637
|
+
| `session_mechanism` | str or null | null | `iap`, or nothing; any other string is refused by the builder when first read |
|
|
638
|
+
| `default_machine_type` | str | required | the bake's `machine_type` when the OS-builder runtime entry names none, and always the instance module's `machine_type` |
|
|
639
|
+
| `default_image_builder` | str | `default` | the image builder an OS-builder runtime entry with `image_builder: default` resolves to |
|
|
640
|
+
| `default_owners` | list[str] or null | null | extra owners for every vendor-image query on this runtime, resolved to projects |
|
|
641
|
+
| `credentials` | mapping | `{}` | must be empty; any key is a validation error |
|
|
642
|
+
| `default_config_username` | str or null | null | the bake ssh user when no image entry, OS entry, `config_username` or runtime `ssh_username` names one (stage 63 item 22) |
|
|
643
|
+
| `ephemeral` | bool | false | every image baked here is disposed by the closing retention lifecycle |
|
|
644
|
+
| `retention_keep` | int or null | null | builds kept per series when the image declares no `retention`; null keeps all |
|
|
645
|
+
| `on_failure` | str or null | null | ephemeral default: `keep` or `teardown` |
|
|
646
|
+
| `teardown_after` | str or null | null | ephemeral default: `<number>` followed by `m`, `h` or `d` |
|
|
647
|
+
| `networking` | mapping or null | null | below; null loads with a warning and bakes and instances then emit no `subnetwork` |
|
|
648
|
+
|
|
649
|
+
`networking:`
|
|
650
|
+
|
|
651
|
+
| Field | Type | Default | Meaning |
|
|
652
|
+
|---|---|---|---|
|
|
653
|
+
| `name` | str | the network | a label; must not be empty after stripping |
|
|
654
|
+
| `network` | str | `default` | the network name; `default` becomes the network literally named `default` at load |
|
|
655
|
+
| `subnets` | list | required, at least one, one `is_default: true` | see below |
|
|
656
|
+
| `availability_zones` | list | `[]` | `{name, is_default}`; only `validate`'s zone-compatibility check reads the default one |
|
|
657
|
+
| `network_tags` | list[str] | `[]` | the packer source's `tags` and the instance module's `network_tags` |
|
|
658
|
+
|
|
659
|
+
`networking.subnets[]:`
|
|
660
|
+
|
|
661
|
+
| Field | Type | Default | Meaning |
|
|
662
|
+
|---|---|---|---|
|
|
663
|
+
| `name` | str | the subnet id | label |
|
|
664
|
+
| `subnet_id` | str | required | a subnetwork name, self link, or `projects/<p>/regions/<r>/subnetworks/<n>` path; checked against the project at load; the default one is the packer source's and the instance module's `subnetwork` |
|
|
665
|
+
| `is_default` | bool | false | the subnet bakes and instances use |
|
|
666
|
+
| `public` | bool | false | accepted, not read (the plugin discovers publicness from the routes) |
|
|
667
|
+
| `cidr` | str or null | null | accepted, not read |
|
|
668
|
+
| `availability_zone` | str or null | null | the zone `validate`'s compatibility check attributes to the runtime when no default availability zone is declared |
|
|
669
|
+
| `config` | mapping | `{}` | accepted, not read |
|
|
670
|
+
|
|
671
|
+
Fields the plugin reads from YAML it does not own:
|
|
672
|
+
|
|
673
|
+
| Where | Field | Read by |
|
|
674
|
+
|---|---|---|
|
|
675
|
+
| the OS builder's runtime entry (`os_builders[].runtimes[]`) | `owners`, `query.filters`, `query.owners` | the image query |
|
|
676
|
+
| the same entry | `image_id` | stage 63: a GCE image name that replaces the query's filters; searched in the entry's owner projects |
|
|
677
|
+
| the same entry | `default_primary_disk_size` | stage 63: a base image's `disk_size` when the runtime declares no `default_disk_size` |
|
|
678
|
+
| the same entry | `machine_type`, `default_machine_type` | the packer source's `machine_type`, before the runtime default |
|
|
679
|
+
| the same entry | `ssh_username` | the packer source's `ssh_username` and the ansible provisioner's `user`, for this OS and every image built on it, whatever the runtime declares |
|
|
680
|
+
| the image | `primary_disk_size` | an instance image's `disk_size` when the runtime declares no `default_disk_size` (a base image's is its entry's `default_primary_disk_size`, else the OS builder's) |
|
|
681
|
+
| the image | `source_image`, `parent_policy`, the pin | `source_image` (pinned or resolved) or `source_image_family` (deferred) |
|
|
682
|
+
| `cfg/executables.yml` | the `gcloud` entry's `binary` and `version` | the version checker |
|
|
683
|
+
| the instance | `runtime`, `image`, `storages`, `tags` | the GCE instance plugin, which reads `zone`, `default_machine_type`, `networking` and `session_mechanism` from this model |
|
|
684
|
+
|
|
685
|
+
Not in the model, and refused at load because unknown keys are errors:
|
|
686
|
+
`account_id`, `security_group_ids`, `session_instance_profile`,
|
|
687
|
+
`iam_instance_profile`, `ena_support`, `sriov_support`, `runtime_classifier`,
|
|
688
|
+
and any key inside `credentials:`.
|
|
689
|
+
|
|
690
|
+
### Variations
|
|
691
|
+
|
|
692
|
+
- **`session_mechanism: iap` vs unset.** With `iap` the packer source gets
|
|
693
|
+
`use_iap = true` and `iap_tunnel_launch_wait = 120`, the bake gains the
|
|
694
|
+
guest-agent install provisioner and the guest-agent assertions in its
|
|
695
|
+
in-bake verification, the instance module gets `public_ip = false`, the
|
|
696
|
+
launch record's `session` is `iap`, and `run_session_command` works. Unset,
|
|
697
|
+
the bake reaches the build VM over its external IP on tcp:22 (which needs
|
|
698
|
+
a firewall rule the operator owns), instances keep a public IP, no agent
|
|
699
|
+
is installed or asserted, `run_session_command` still runs `gcloud
|
|
700
|
+
compute ssh --tunnel-through-iap` (it does not check the mechanism, so it
|
|
701
|
+
fails without the IAP wiring), and `validate` refuses a base image on
|
|
702
|
+
this runtime that also declares no admin public key (no debug path). The
|
|
703
|
+
finalize provisioner is emitted either way.
|
|
704
|
+
- **`ephemeral: true` vs `false`.** Ephemeral: the closing retention step
|
|
705
|
+
disposes every build baked here that is not released, pinned or launched
|
|
706
|
+
from, whatever `retention_keep` or the images' `retention` say; `empty
|
|
707
|
+
--runtime` then expects no custom image but released ones. Durable:
|
|
708
|
+
builds stand until `retention.keep`, `retention_keep` or an explicit
|
|
709
|
+
`dispose image`.
|
|
710
|
+
- **Pinned vs deferred vs resolved parent.** A pinned parent bakes from
|
|
711
|
+
`source_image = gce_name(<build id>)`; a resolved vendor image from
|
|
712
|
+
`source_image = <name>` plus `source_image_project_id = ["<project>"]`
|
|
713
|
+
(omitted when the owner is `self` or unset); a deferred parent (built
|
|
714
|
+
earlier in the run or by a previous one) from `source_image_family =
|
|
715
|
+
gce_name(<series>)`, which GCE resolves to the newest image of the family.
|
|
716
|
+
`parent_policy: follow` moves the pin after the bake and the post-bake
|
|
717
|
+
retag writes the resolved parent onto the image.
|
|
718
|
+
- **`default_disk_size` set vs null.** The packer source's `disk_size`
|
|
719
|
+
is `lineage.bake_disk_size` (stage 63), the rule the input fingerprint
|
|
720
|
+
also hashes. Set: every image here bakes at that size and everything
|
|
721
|
+
else is ignored. Null: a base image takes its OS builder entry's
|
|
722
|
+
`default_primary_disk_size` for this runtime when declared, else the OS
|
|
723
|
+
builder's `default_primary_disk_size` (200 GB unset); an instance image
|
|
724
|
+
takes its own `primary_disk_size`. Stage 63 moved no GCE fingerprint:
|
|
725
|
+
the runtime's `default_disk_size` already won, and still does.
|
|
726
|
+
- **A queried vs a pinned vendor image.** Without `image_id` on the OS
|
|
727
|
+
builder's entry, resolution runs the filter query and takes the newest
|
|
728
|
+
match, so the next base bake may start from a newer vendor image. With
|
|
729
|
+
`image_id` (stage 63) that one image name is looked up in the entry's
|
|
730
|
+
owner projects; the id is part of the fingerprint's vendor source.
|
|
731
|
+
- **`ssh_username` set vs `default`.** Either way the packer source and
|
|
732
|
+
the ansible provisioner name the same user, the one the bake-user order
|
|
733
|
+
resolves: an entry's user wins over this field, this field over the
|
|
734
|
+
runtime's `default_config_username`, and with nothing named anywhere it
|
|
735
|
+
is `packer`. Before stage 63 the runtime's value overrode the entry, and
|
|
736
|
+
with it unset the provisioner said `packer` even when the entry named
|
|
737
|
+
another user.
|
|
738
|
+
- **`network: default` vs a name.** `default` is replaced at load by the
|
|
739
|
+
network named `default` and the packer source emits no `network` line;
|
|
740
|
+
a named network is checked to exist and emitted as `network`.
|
|
741
|
+
- **`network_tags` declared vs empty.** Declared: `tags = [...]` on the
|
|
742
|
+
packer source, `network_tags` on the instance module, a load-time warning
|
|
743
|
+
per tag no firewall rule targets. Empty: neither line is emitted.
|
|
744
|
+
- **Owner aliases vs project ids vs AWS-only owners.** `self` becomes
|
|
745
|
+
`project_id`; distro aliases become the public image projects; a valid
|
|
746
|
+
project id is used as is; `amazon`, `aws-marketplace`, `aws-backup-vault`
|
|
747
|
+
and anything that cannot be a project id (an AWS account number) are
|
|
748
|
+
dropped, so a shared OS-builder query works on both clouds.
|
|
749
|
+
- **`filters` with AWS-only keys.** `root_device_type`,
|
|
750
|
+
`virtualization_type`, `block_device_mapping.*` and the rest of
|
|
751
|
+
`AWS_ONLY_QUERY_KEYS` are dropped with a DEBUG line; an unknown key is
|
|
752
|
+
applied as an exact post-query match on the image dictionary (a typo
|
|
753
|
+
therefore matches nothing and the resolution fails, by design).
|
|
754
|
+
- **`state: available` vs anything else.** `available` maps to `status =
|
|
755
|
+
"READY"`, which is also forced when no status is given; `pending`/`failed`
|
|
756
|
+
map to `PENDING`/`FAILED`; another value is upper-cased as given.
|
|
757
|
+
- **`bake_preemptible: true` vs `false`.** True emits `preemptible = true`;
|
|
758
|
+
a preempted bake fails and is re-run by the next run (nothing is recorded
|
|
759
|
+
for it). False emits nothing.
|
|
760
|
+
- **`project_id` set vs null.** Null: the load warns and skips network
|
|
761
|
+
discovery and validation; `resolve` then fails with `Owner 'self'
|
|
762
|
+
requires a project ...` and every Compute hook raises `no project ...`.
|
|
763
|
+
The warning names the missing `project_id` (stage 63; it named a
|
|
764
|
+
"service account key file" until 2026-09-25, which no field of this
|
|
765
|
+
runtime can declare, and the project is never read from the
|
|
766
|
+
Application Default Credentials).
|
|
767
|
+
- **A tag key that starts with a lowercase letter vs one that does not.**
|
|
768
|
+
A key whose `gce_label` form starts with `a`-`z` (`Project` becomes
|
|
769
|
+
`project`) passes and is emitted as a label. A key whose label form
|
|
770
|
+
starts with anything else (`2024`, `-env`, `_x`) on an image, instance or
|
|
771
|
+
storage that lands on this runtime is refused by `validate` (stage 63,
|
|
772
|
+
`check_label_keys`); the same key on an AWS runtime passes, because the
|
|
773
|
+
base `label_key_problem` answers `None` there. Until 2026-09-25 it passed
|
|
774
|
+
`validate` everywhere and GCE refused it at apply.
|
|
775
|
+
- **`query_images` with series vs with none.** Given series, only images
|
|
776
|
+
whose `csis_series` label is one of them (in `gce_label` form) come back;
|
|
777
|
+
given an empty list, every csis-labelled image in the project does
|
|
778
|
+
(stage 63; the argument was ignored until 2026-09-25).
|
|
779
|
+
- **Dry run vs real run.** The plugin itself does not read the flag. A dry
|
|
780
|
+
run still loads (so network discovery still calls the API), still
|
|
781
|
+
resolves vendor images (the `resolve` phase queries the image projects),
|
|
782
|
+
and still emits the packer source and the terraform arguments; nothing is
|
|
783
|
+
baked, verified, disposed or relabelled because the runner scripts are
|
|
784
|
+
enumerated, not executed. `dispose image` under `--dry-run` prints its
|
|
785
|
+
plan and calls no hook; `lineage relabel` is dry by default.
|
|
786
|
+
- **Apply flag on vs off.** With `apply_instances: [gcloud-east1]` (or
|
|
787
|
+
`--apply-runtime gcloud-east1`) the instance root applies, the launch is
|
|
788
|
+
recorded, the verification runs, `mark_launched` fires, the generation
|
|
789
|
+
ledger asks `query_instance_identity` and the alias writer asks it again.
|
|
790
|
+
Off, the root plans and gates only and none of the instance hooks run.
|
|
791
|
+
- **`--only-runtime` another runtime.** No GCE image bakes and no GCE root
|
|
792
|
+
is generated or planned (a run scoped away from this runtime cannot fail
|
|
793
|
+
on its family lookup); retention disposes within the scoped runtime only.
|
|
794
|
+
- **A stopped instance.** `TERMINATED` maps to `stopped`; the state query
|
|
795
|
+
writes a `note` instead of an `unavailable` line, `verify instance` starts
|
|
796
|
+
the machine for the check and stops it again, `verify login` and the
|
|
797
|
+
alias writer skip it, and nothing else touches its power state.
|
|
798
|
+
- **Encrypted vs clear values.** The plugin reads no encrypted value: every
|
|
799
|
+
field it owns is a name, an id or a number. An `ENC[age:...]` marker in
|
|
800
|
+
one of them would reach the API as the literal marker.
|
|
801
|
+
|
|
802
|
+
## What it tests and verifies
|
|
803
|
+
|
|
804
|
+
**At load (pydantic, then `finalize()`).** Unknown keys anywhere in the
|
|
805
|
+
entry are refused with the field path (`extra="forbid"`); `credentials:`
|
|
806
|
+
with any key is refused; a networking block must have a non-empty `name`
|
|
807
|
+
and `network`, at least one subnet and one `is_default: true`; a runtime
|
|
808
|
+
must declare `default_machine_type`. Then `update_networking()`: with a
|
|
809
|
+
project it calls the Compute API and refuses the load (a `ValueError`; the
|
|
810
|
+
CLI exits 1 for any command that loads the tree, `validate` included) when
|
|
811
|
+
`network: default` names nothing, the network is absent from the project,
|
|
812
|
+
or a `subnet_id` matches no subnetwork of that network; it logs a warning
|
|
813
|
+
per network tag no firewall rule targets, a warning when `networking` is
|
|
814
|
+
absent, and a warning when no project can be resolved. The verdict lands
|
|
815
|
+
in the log; the discovered `network_map`, `default_network` and
|
|
816
|
+
`all_firewall_rules` stay on the model for the emitters. The test suite
|
|
817
|
+
stubs `get_network_map_and_default_network` with the fixture's own subnet
|
|
818
|
+
ids (see `tests/v2_support.py`), so no test reaches the API.
|
|
819
|
+
|
|
820
|
+
**At `validate`.** The `gcloud` executable's binary exists and its SDK
|
|
821
|
+
version satisfies the declared requirement (one INFO line names every
|
|
822
|
+
version found; a failure names the tool); `session_mechanism()` is asked
|
|
823
|
+
for the dead-end rule, so an unsupported value surfaces here as a
|
|
824
|
+
`ValueError`; the zone-compatibility check reads `networking`'s default
|
|
825
|
+
availability zone or the default subnet's `availability_zone` against the
|
|
826
|
+
instances and zonal storages on the runtime; the state-location check
|
|
827
|
+
resolves each root's backend through `state_configuration`; and (stage 63)
|
|
828
|
+
`check_label_keys` asks `label_key_problem` about every tag key declared on
|
|
829
|
+
an image baked here, an instance on this runtime and a storage whose
|
|
830
|
+
builder's runtime is this one, refusing a key whose label form does not
|
|
831
|
+
start with a lowercase letter (it used to pass here and fail at apply
|
|
832
|
+
until 2026-09-25). Every failure is one line naming the object, and
|
|
833
|
+
`validate` exits 1.
|
|
834
|
+
|
|
835
|
+
**At generation.** `session_mechanism()` again (the packer provisioners,
|
|
836
|
+
the instance builder, the launch record). The packer source is emitted
|
|
837
|
+
with the sanitised names; the golden emission under
|
|
838
|
+
`tests/fixtures/v2_golden` pins its exact shape, and `just test` fails on
|
|
839
|
+
any drift of it. `test_v2_explore_gcp.py` asserts the emitted commands: the
|
|
840
|
+
finalize provisioner is the last one of every GCE bake, IAP runtimes emit
|
|
841
|
+
`use_iap` and the tunnel wait and non-IAP ones do not, the disk size comes
|
|
842
|
+
from the runtime, the device name and the mounted by-id path agree.
|
|
843
|
+
`test_v2_gce_cycle.py` pins that `bake_preemptible` lands on the source and
|
|
844
|
+
that `retag_image` merges the lineage truth into the labels. The package's
|
|
845
|
+
own tests pin the query remap and project resolution.
|
|
846
|
+
|
|
847
|
+
**In the bake (packer, on the build VM).** The in-bake verification
|
|
848
|
+
provisioner runs `session_verify_commands` under `set -e`: the guest agent
|
|
849
|
+
binary exists and the unit is enabled; a failed assertion fails the bake
|
|
850
|
+
and packer deletes the build VM. The finalize provisioner runs last and is
|
|
851
|
+
tolerant (`|| true`).
|
|
852
|
+
|
|
853
|
+
**After a bake.** The packer builder records the build from the manifest
|
|
854
|
+
(`build_id_from_artifact` returns the image name) and calls `retag_image`
|
|
855
|
+
to stamp `csis_parent` and `csis_fingerprint` from the record; the labels
|
|
856
|
+
and the record then agree, which the state query later checks.
|
|
857
|
+
|
|
858
|
+
**After the instance apply (verification).** `verify instance <name>`
|
|
859
|
+
runs inside `running_for_task`; the hook polls the serial console until
|
|
860
|
+
the startup scripts report completion, checks the booted image against the
|
|
861
|
+
pin or launch record (or, for an image built this run, against the lineage
|
|
862
|
+
records of the instance's series on this runtime), counts clean XFS mounts
|
|
863
|
+
against the declared data disks, and then runs the image's declared
|
|
864
|
+
post-bake tests over `run_session_command`. The verdict lands in
|
|
865
|
+
`meta-state/verifications.yaml` (`ok`, `checks`, the last 20 serial
|
|
866
|
+
console lines mentioning `startup`, `XFS` or
|
|
867
|
+
`google_metadata_script_runner` as `evidence`), post-bake results in
|
|
868
|
+
`meta-state/image-tests.yaml`, one `verify <name>: <check>: ok|FAILED --
|
|
869
|
+
<detail>` log line per check, and a failed verdict raises
|
|
870
|
+
`VerificationFailed`, which stops the runner with the instance standing
|
|
871
|
+
(`on_failure: keep`) or after its teardown (`teardown`); the run exits 1.
|
|
872
|
+
A stopped instance that cannot be started is recorded as `skipped` with no
|
|
873
|
+
verdict.
|
|
874
|
+
|
|
875
|
+
**After the apply (generations and aliases).** `query_instance_identity`
|
|
876
|
+
gives the numeric instance id that defines the instance's generation in
|
|
877
|
+
`meta-state/instance-state.yaml`; the alias writer reads it again, waits
|
|
878
|
+
until `run_session_command` answers, and writes the AltNames over the same
|
|
879
|
+
session (log lines `now also answers to [...]` or a `WARNING`/`ERROR`
|
|
880
|
+
naming why not).
|
|
881
|
+
|
|
882
|
+
**In the state query.** `query_images` lists the images whose
|
|
883
|
+
`csis_series` label is `gce_label` of a series the query asked about (the
|
|
884
|
+
recorded series and the declared OS builders and images; stage 63, it
|
|
885
|
+
listed every csis-labelled image until 2026-09-25); the base classifies `missing` (recorded, not found),
|
|
886
|
+
`foreign` (labelled, not recorded) and `changed` (labels disagree with the
|
|
887
|
+
record) images. `query_instance_boot_image` compares a pinned instance's
|
|
888
|
+
booted image with its pin (`changed`, or a `note` when a replacement is
|
|
889
|
+
pending) and finds standing ephemerals; `query_instance_power_state` turns
|
|
890
|
+
a stopped machine into a `note`. Any hook that raises becomes an
|
|
891
|
+
`unavailable:` line naming the label and the error, never a claim. The
|
|
892
|
+
report is `generated/state-report.json` (run-local) and the log;
|
|
893
|
+
`state query --strict` exits 1 on any drift but `stale`.
|
|
894
|
+
|
|
895
|
+
**In `empty --runtime`.** `inventory()` is subtracted by the declared
|
|
896
|
+
storages' cloud names and the released builds; any leftover instance,
|
|
897
|
+
image, disk or bucket is printed and the command exits 1; a hook that
|
|
898
|
+
cannot answer exits 2.
|
|
899
|
+
|
|
900
|
+
**In retention and `dispose image`.** `dispose_image` reports
|
|
901
|
+
`deleted=True|False` per build in the log (`False` when GCE already had no
|
|
902
|
+
such image); the lineage record is dropped either way.
|
|
903
|
+
|
|
904
|
+
`tests/test_v2_bootstrap_gcp.py` holds the bootstrap's GCP section: the
|
|
905
|
+
entry point and the tree-derived gates (a GCE-only tree's quiet interview
|
|
906
|
+
asks nothing of AWS), placeholders and an unaskable project refused by
|
|
907
|
+
name, a fresh project getting the guide's pool, provider and accounts, an
|
|
908
|
+
existing shared provider read as it stands and deciding which attribute
|
|
909
|
+
the bindings name, the by-hand note when its condition does not name
|
|
910
|
+
this repository, the module held to the guide's attribute mapping and to
|
|
911
|
+
single-member grants only, and a real `tofu validate`.
|
|
912
|
+
|
|
913
|
+
## When it fails
|
|
914
|
+
|
|
915
|
+
Failures that have happened, newest first. Dates are the ledger's
|
|
916
|
+
([docs/history/LEDGER.md](../../docs/history/LEDGER.md)) or the test
|
|
917
|
+
docstrings'.
|
|
918
|
+
|
|
919
|
+
- **2026-09-21 -- a stopped machine reported as "could not answer".** The
|
|
920
|
+
boot-image probe answered `None` for an instance the operator had
|
|
921
|
+
stopped, the state query wrote `unavailable`, and the records described a
|
|
922
|
+
machine that was not there. Since stage 57 the plugin maps GCE's
|
|
923
|
+
`TERMINATED` to `stopped` (it is not deleted: a deleted instance is `not
|
|
924
|
+
found` and maps to `absent`), the query asks the power state before
|
|
925
|
+
calling silence unavailable, and the line is a `note` (`instances/<name>:
|
|
926
|
+
stopped; its pinned build ... still stand ...`). Nothing to do; a stopped
|
|
927
|
+
machine is the operator's decision.
|
|
928
|
+
- **2026-09-10 -- `Script disconnected unexpectedly` at the last scriptlet
|
|
929
|
+
of the security-update transaction.** A GCE base bake over the IAP tunnel
|
|
930
|
+
died after 3 min 24 s as the transaction closed (`pam`, `dbus-broker`,
|
|
931
|
+
`libssh`, `curl`, `python3`...); packer cleaned its build VM, nothing was
|
|
932
|
+
left on GCE, and a scoped retry (`run base-image --only
|
|
933
|
+
basic-rh-10@gcloud-east1`) baked cleanly. Transient, not reproduced; the
|
|
934
|
+
watch item is packer's `expect_disconnect` or a pause on the OS-update
|
|
935
|
+
provisioner of IAP runtimes if it recurs. Look at packer's log in the
|
|
936
|
+
bake's block directory; retry the bake alone.
|
|
937
|
+
- **2026-09-10 -- `family/imgfile-basic-dask` returned 404 at the GCE
|
|
938
|
+
instance plan.** A run scoped to the AWS runtime still planned the GCE
|
|
939
|
+
instance root, whose module resolves a deferred image through
|
|
940
|
+
`data "google_compute_image" { family = ... }`, and the previous cycle's
|
|
941
|
+
retention had disposed every GCE image. Since then a run scoped to one
|
|
942
|
+
runtime (`--only-runtime`, or `--apply-runtime` implying it) generates
|
|
943
|
+
and plans no other runtime's root. If it appears in an unscoped run: the
|
|
944
|
+
family is empty because retention emptied it; bake the series first.
|
|
945
|
+
- **2026-09-09 -- `read_passphrase: can't open /dev/tty`.** The operator's
|
|
946
|
+
`google_compute_engine` key was passphrase-protected and the session hook
|
|
947
|
+
(`gcloud compute ssh`, non-interactive) cannot prompt, so every
|
|
948
|
+
`run_session_command` failed. Use an agent-loaded or passphrase-less key
|
|
949
|
+
(the project's `ssh-keys` metadata was re-provisioned by the guest agent
|
|
950
|
+
within seconds of the change).
|
|
951
|
+
- **2026-09-08 -- `changed` drift: `fingerprint differs` on a follow image
|
|
952
|
+
(finding 66).** A `parent_policy: follow` image baked in the same run as
|
|
953
|
+
its parent got its labels from packer at generation time (parent
|
|
954
|
+
`series:...`, a placeholder fingerprint); the lineage record was written
|
|
955
|
+
after the bake; `retag_image` was a no-op on GCE, so the strict preflight
|
|
956
|
+
refused the next run. `retag_image` now does `setLabels`. For images
|
|
957
|
+
already standing: `lineage relabel --runtime <rt>` (`just cloud-relabel
|
|
958
|
+
<rt> no`; dry by default) re-labels every recorded build whose labels
|
|
959
|
+
disagree with its record.
|
|
960
|
+
- **2026-09-08 -- `4047: Failed to lookup instance` and an SSH timeout
|
|
961
|
+
against a healthy build VM (finding 64).** A bake that followed another
|
|
962
|
+
build in the same run hit IAP's instance-lookup lag for longer than
|
|
963
|
+
packer's default 30 s tunnel wait; four reproductions, serial consoles
|
|
964
|
+
showed sshd up in 30 s. IAP bakes now set `iap_tunnel_launch_wait = 120`.
|
|
965
|
+
If it recurs, `PACKER_LOG=1` shows the tunnel retries; the wait is a
|
|
966
|
+
constant in `gcp_packer_source.py`.
|
|
967
|
+
- **2026-09-08 -- `No valid credential sources found ... backend s3 ...
|
|
968
|
+
the SSO session has expired` at the GCE instance plan (finding 65).** The
|
|
969
|
+
AWS session lapsed mid-cycle; the GCE roots' state is in S3 and every
|
|
970
|
+
load validates the AWS runtimes' networking. Renew the AWS session (`aws
|
|
971
|
+
sso login --profile <p>`) and rerun; the preflight now reads session
|
|
972
|
+
lifetimes before a run. `empty --runtime gcloud-east1` needs the AWS
|
|
973
|
+
session for the same reason.
|
|
974
|
+
- **2026-09-08 -- verification failed on `booted image` with a launch
|
|
975
|
+
record of `unbound` (finding 63).** The instance's image was baked in the
|
|
976
|
+
same run, so nothing was pinned and the check compared the booted image
|
|
977
|
+
with the literal `unbound`; the instance was left standing by the rule.
|
|
978
|
+
Verification now requires the booted image to be a recorded build of the
|
|
979
|
+
instance's image series on its runtime. The recovery was the next
|
|
980
|
+
instance-image run resuming the sequence.
|
|
981
|
+
- **2026-09-07 -- `gce-verify` never saw `startup-script exit status 0`
|
|
982
|
+
(finding 56).** Guest agent 20260715 logs `Finished running startup
|
|
983
|
+
scripts` instead; the instance was healthy the whole time. The hook
|
|
984
|
+
accepts both lines. A verification that reports `no completion on the
|
|
985
|
+
serial console within <timeout>s` on a newer agent means a third
|
|
986
|
+
spelling: read `serial_console` output (the `evidence` in
|
|
987
|
+
`verifications.yaml`) and extend the pattern.
|
|
988
|
+
- **2026-09-07 -- a dask bake timed out waiting for SSH through the IAP
|
|
989
|
+
tunnel (finding 55).** In the window GitHub's release downloads were
|
|
990
|
+
returning 504; the retry booted, provisioned in about 90 s and imaged
|
|
991
|
+
cleanly. Later understood as finding 64 above.
|
|
992
|
+
- **2026-09-06 -- every bake timed out `waiting for SSH` (finding 54).**
|
|
993
|
+
Packer reached build VMs over their external IP on tcp:22, which only
|
|
994
|
+
worked while the default VPC's `default-allow-ssh` rule existed; deleting
|
|
995
|
+
it (deliberate hygiene) broke every bake. IAP runtimes now bake through
|
|
996
|
+
the tunnel (`use_iap = true`); the operator prerequisite is
|
|
997
|
+
`roles/iap.tunnelResourceAccessor` for the runner service account. The
|
|
998
|
+
same symptom today means that role is missing.
|
|
999
|
+
- **2026-09-05 -- the instance boot disk was 200 GB, about $8/month
|
|
1000
|
+
(finding 51).** The boot disk inherited the image's `disk_size`, which
|
|
1001
|
+
inherited the OS builder's 200 GB default (a RHEL vendor minimum carried
|
|
1002
|
+
forward). `default_disk_size` on the runtime now wins; the fixture bakes
|
|
1003
|
+
at the vendor image's 10 GB floor (a boot disk cannot be smaller than its
|
|
1004
|
+
source image).
|
|
1005
|
+
- **2026-09-05 -- every SSH/IAP session to the launched instance refused
|
|
1006
|
+
(finding 47).** At first boot the guest agent removes users absent from
|
|
1007
|
+
metadata; when the bake user's `google-sudoers` membership was missing
|
|
1008
|
+
(the agents race over it during bakes) `gpasswd` exited 3 and the agent
|
|
1009
|
+
aborted its whole metadata-ssh-key setup, so no operator key was ever
|
|
1010
|
+
provisioned. `bake_finalize_commands` now guarantees the membership as
|
|
1011
|
+
the bake's last act. The symptom returning means the finalize provisioner
|
|
1012
|
+
is not the last one in the emitted `-build.pkr.hcl` (the test
|
|
1013
|
+
`test_gce_bakes_end_with_the_finalize_provisioner` guards it).
|
|
1014
|
+
- **2026-09-05 -- ansible tasks died `unreachable / Failed to create
|
|
1015
|
+
temporary directory` (finding 48).** Packer's ansible provisioner
|
|
1016
|
+
defaulted `ansible_user` to the operator's local login, not the build
|
|
1017
|
+
VM's ssh user, so ansible built `~/.ansible/tmp` under
|
|
1018
|
+
`/home/<local-user>`; intermittent because packer's `use_proxy`
|
|
1019
|
+
auto-detection masked it. `bake_ssh_username(image)` pins the provisioner
|
|
1020
|
+
`user` to the bake's user, the one the packer source resolves (since
|
|
1021
|
+
stage 63 item 23; it read only the runtime's field before).
|
|
1022
|
+
- **2026-09-04 -- the base bake used ssh user `ec2-user` while the
|
|
1023
|
+
instance bake fell back to `packer` (finding 43).** The new-generation
|
|
1024
|
+
guest agent removes the other-name user at first boot and aborts key
|
|
1025
|
+
provisioning; one GCE chain must bake every image as ONE user. Declare
|
|
1026
|
+
`ssh_username` on the OS-builder runtime entry (it covers the chain), or
|
|
1027
|
+
on the runtime. Since stage 63 `validate` refuses a chain whose images
|
|
1028
|
+
would bake as different users, and the GCE family default is `packer`.
|
|
1029
|
+
- **2026-09-03 -- a dask bake fell through to the runtime's 1 GB machine
|
|
1030
|
+
and OOMed at 65 minutes (finding 41).** The packer source's
|
|
1031
|
+
`machine_type` now follows the AWS-parity chain: the OS-builder runtime
|
|
1032
|
+
entry's `machine_type`, its `default_machine_type`, then the runtime's
|
|
1033
|
+
`default_machine_type`. A bake that needs memory declares it on the
|
|
1034
|
+
entry; the runtime default is the free-tier `e2-micro` in the fixture.
|
|
1035
|
+
- **2026-09-03 -- underscores in GCE names (findings 34-37).** GCE refuses
|
|
1036
|
+
`_` in resource and device names; every name now goes through
|
|
1037
|
+
`gce_name()`. A `gce_data` disk attached under that name while the
|
|
1038
|
+
startup script mounted `/dev/disk/by-id/google-gce-data` (finding 50,
|
|
1039
|
+
2026-09-05) is the same lesson on the instance side: one sanitisation
|
|
1040
|
+
serves both.
|
|
1041
|
+
|
|
1042
|
+
Failures the code raises that have not happened live, by where they
|
|
1043
|
+
surface:
|
|
1044
|
+
|
|
1045
|
+
- **Load.** ``No usable GCP project could be resolved for cloud builder
|
|
1046
|
+
<name> (no `project_id` declared on the runtime). Skipping network validation; image
|
|
1047
|
+
resolution and build execution will not work until valid GCP credentials
|
|
1048
|
+
are configured.`` (WARNING): no `project_id`; add one. (Stage 63: until
|
|
1049
|
+
2026-09-25 this warning pointed at a "service account key file", which
|
|
1050
|
+
no field of this runtime can declare.) `No networking configuration provided for GCP cloud builder
|
|
1051
|
+
<name>` (WARNING): bakes and instances will carry no `subnetwork`.
|
|
1052
|
+
`No network specified ... and no 'default' network found in GCP project`,
|
|
1053
|
+
`Network <n> ... not found in GCP project`, `Subnetwork <id> ... not
|
|
1054
|
+
found in network <n>` (ERROR, then a `ValueError` that ends the load,
|
|
1055
|
+
exit 1): fix the declaration or the project. `GCP Error: <api error>`
|
|
1056
|
+
(`ValueError`): the API refused the listing, usually permissions or a
|
|
1057
|
+
disabled Compute API. An auth error (`DefaultCredentialsError`, no ADC)
|
|
1058
|
+
is not wrapped and ends the load with the library's own message: run the
|
|
1059
|
+
ADC login above. `Network tag <t> ... is not targeted by any firewall
|
|
1060
|
+
rule` (WARNING): the tag selects nothing.
|
|
1061
|
+
- **`validate` / generation.** `GCP runtime <name>: unknown session
|
|
1062
|
+
mechanism '<x>' (supported: iap)` (`ValueError`): only `iap` exists.
|
|
1063
|
+
`gcloud: binary '/usr/local/bin/gcloud' not found`, `gcloud:
|
|
1064
|
+
GCPCLIVersionChecker could not parse a version from ...`, `gcloud <v>
|
|
1065
|
+
does not meet its requirement >=500 (cfg/executables.yml)`: install or
|
|
1066
|
+
upgrade the SDK, or move the floor in `executables.yml`.
|
|
1067
|
+
`Executable gcloud specified for provider <runtime> not found in
|
|
1068
|
+
executables list.`: the runtime's `executable` (default `gcloud`, stage
|
|
1069
|
+
63 item 14) names no declared entry; declare `- name: gcloud` with its
|
|
1070
|
+
`binary`, or point `executable` at the entry you have. The same cause at
|
|
1071
|
+
a session or lookup reads `GCP runtime <name>: executable 'gcloud' is
|
|
1072
|
+
not declared in cfg/executables.yml`.
|
|
1073
|
+
`<image|instance|storage> '<name>': tag key '<key>' on runtime <rt>: GCE
|
|
1074
|
+
label keys must start with a lowercase letter; '<key>' becomes
|
|
1075
|
+
'<label>'; rename the key` (stage 63, from `validate`, exit 1): a tag key
|
|
1076
|
+
declared on that image, instance or storage would become a label key GCE
|
|
1077
|
+
refuses (a key such as `2024`). Rename the key in the YAML; nothing
|
|
1078
|
+
renames it for you. Until 2026-09-25 the key passed `validate` and GCE
|
|
1079
|
+
refused it at apply.
|
|
1080
|
+
- **A session command (post-bake verify, unmount, the alias writer).**
|
|
1081
|
+
`GCP runtime <name> declares no session_mechanism, so no session command
|
|
1082
|
+
can reach its instances (supported: iap)`: declare
|
|
1083
|
+
`session_mechanism: iap` (and its firewall rule and grant) or run
|
|
1084
|
+
nothing that needs a session on that runtime.
|
|
1085
|
+
- **Resolution (`resolve`, the base-image phase of a run).** `Owner
|
|
1086
|
+
'self' requires a project in the session configuration (set project_id
|
|
1087
|
+
on the runtime builder ...)`; `No usable GCP projects resolved from
|
|
1088
|
+
query ...` (every owner was dropped as invalid); `Failed to query images
|
|
1089
|
+
in project '<p>' with filter: <error>` and `Failed to query image family
|
|
1090
|
+
'<f>' in project '<p>': <error>` (`ImageQueryError`: permissions or an
|
|
1091
|
+
unknown project); `No resolved Image identifiers for OS <name> in
|
|
1092
|
+
predefined_resolve` (the base's message when the query matched nothing:
|
|
1093
|
+
loosen `filters`, check the owner and the `status`); `OS builder <name>:
|
|
1094
|
+
image_id '<id>' on runtime <rt> is not known to the provider` (stage 63:
|
|
1095
|
+
the pinned `image_id` names no image in the entry's owner projects;
|
|
1096
|
+
check the name and that its project is among the owners); `Could not get
|
|
1097
|
+
owning project for image <name>` (a result with neither `project` nor a
|
|
1098
|
+
`self_link`; not expected from the API). Each ends the run, exit 1.
|
|
1099
|
+
- **The state query.** `state query images/<rt> unavailable: GCP runtime
|
|
1100
|
+
<rt>: no project to query` or an API error: an `unavailable:` line, never
|
|
1101
|
+
drift; `state query --strict` still refuses on a session that will not
|
|
1102
|
+
outlast the run, and a plain run warns. `instances/<name>: booted image
|
|
1103
|
+
(runtime <rt> could not answer)`: the boot probe returned `None` and the
|
|
1104
|
+
machine is not stopped; the DEBUG log carries the exception.
|
|
1105
|
+
- **Verification.** `verify <name>: startup scripts: FAILED -- an explicit
|
|
1106
|
+
failure line` (a `startup-script exit status <non-zero>` or a `failed`/
|
|
1107
|
+
`error` startup line on the console; read the `evidence`); `no completion
|
|
1108
|
+
on the serial console within <t>s` (the agent never reported; raise
|
|
1109
|
+
`--timeout`, read the console); `booted image: FAILED -- booted <x>,
|
|
1110
|
+
expected <y>` (the instance did not boot its pin: `upgrade instance` and
|
|
1111
|
+
replace, or `state query` to see the mismatch); `booted <x>, which
|
|
1112
|
+
lineage does not record for <image> on <rt>`; `data disks mounted:
|
|
1113
|
+
FAILED -- <n> clean XFS mount(s) on the console, <m> declared` (the
|
|
1114
|
+
startup script did not format or mount a pd: the device name and the
|
|
1115
|
+
by-id path must agree); `declared tests: FAILED -- k of n failed: ...`
|
|
1116
|
+
(the image's post-bake suite; the per-build record in
|
|
1117
|
+
`image-tests.yaml` names the assertion; `release` refuses the build).
|
|
1118
|
+
`GCP runtime <rt>: no project/zone to read a serial console`
|
|
1119
|
+
(`RuntimeError`): declare `zone`. A `VerificationFailed` leaves the
|
|
1120
|
+
instance standing under `keep` and fails the run.
|
|
1121
|
+
- **Session commands.** A non-zero exit from `gcloud compute ssh` is
|
|
1122
|
+
returned with the combined output: `unmount storage` writes a failed
|
|
1123
|
+
receipt and raises `unmount of <mount> on <instance> failed (exit
|
|
1124
|
+
<rc>)`, the alias writer logs `alias write failed (<rc>): <tail>`,
|
|
1125
|
+
`wait_until_reachable` keeps polling until its deadline, and a post-bake
|
|
1126
|
+
suite shows as `no result`. The usual causes are the missing IAP role or
|
|
1127
|
+
firewall rule, a passphrase-protected key, or a machine still booting.
|
|
1128
|
+
`GCP runtime <rt>: no project/zone for a session command`: declare
|
|
1129
|
+
`zone`.
|
|
1130
|
+
- **Power state.** `GCE runtime <rt>: instance '<n>' reports unrecognised
|
|
1131
|
+
status '<s>'; treating it as unknown` (WARNING): a status outside the
|
|
1132
|
+
eight the map knows; nothing acts on it. `GCP runtime <rt>: no
|
|
1133
|
+
project/zone to start an instance` / `... to stop an instance`
|
|
1134
|
+
(`RuntimeError`). `<n>: FAILED to stop it again after <why>: <error>. It
|
|
1135
|
+
was switched off before this run and is now RUNNING -- stop it by hand.`
|
|
1136
|
+
(ERROR from `running_for_task`): the one case that leaves a billable
|
|
1137
|
+
change behind; stop the instance.
|
|
1138
|
+
- **`empty --runtime`.** `GCP runtime <rt>: no project/zone to inventory`;
|
|
1139
|
+
`gcloud storage buckets list: <stderr tail>` (the gcloud account cannot
|
|
1140
|
+
list buckets, or is not logged in): exit 2. A leftover named under
|
|
1141
|
+
`leftovers` is exit 1: dispose or decommission it through the recorded
|
|
1142
|
+
path, never by hand.
|
|
1143
|
+
- **Retention and `dispose image`.** `GCE image <id> not found in <p>;
|
|
1144
|
+
nothing to delete` (WARNING, `deleted=False`): the image was already
|
|
1145
|
+
gone; the record is dropped anyway. `GCP runtime <rt>: no project to
|
|
1146
|
+
dispose in` (`RuntimeError`). An API error (permissions, an image in
|
|
1147
|
+
use by a disk) propagates and stops the retention step; the record then
|
|
1148
|
+
still exists and the next run retries.
|
|
1149
|
+
- **Relabel.** `relabel: could not retag <id>: <error>` (WARNING,
|
|
1150
|
+
`retagged=False`): the record is right and the state query keeps showing
|
|
1151
|
+
the tag. `relabel: <id> is not in <rt>; nothing to relabel`: the image
|
|
1152
|
+
is gone; the state query reports it `missing`. The post-bake retag has no
|
|
1153
|
+
guard: an error there fails the bake's record step.
|
|
1154
|
+
- **Names.** `gce_name()` never fails; two csis names that differ only in
|
|
1155
|
+
characters it strips (`a_b` and `a-b`, or case) collide on GCE, and the
|
|
1156
|
+
second bake or launch fails with the API's `already exists`. Keep names
|
|
1157
|
+
distinct in the lowercase `[a-z0-9-]` alphabet.
|
|
1158
|
+
|
|
1159
|
+
## Related
|
|
1160
|
+
|
|
1161
|
+
- [aws-runtime-plugin](../aws-runtime-plugin/README.md): the other runtime type, with the same hook surface on EC2.
|
|
1162
|
+
- [default-os-plugin](../default-os-plugin/README.md): the OS builders whose runtime entries this plugin resolves to GCE images.
|
|
1163
|
+
- [tf-gcp-plugin](../tf-gcp-plugin/README.md): the GCE instance and storage roots that read this model.
|
|
1164
|
+
- [dummy-plugin](../dummy-plugin/README.md): the extension template (this package no longer carries a copy of its group model; removed in stage 63).
|
|
1165
|
+
- Base classes: [runtime.py](../base/src/cs_image_system/base/models/runtime.py), [cloud_builder.py](../base/src/cs_image_system/base/models/cloud_builder.py), [credentials.py](../base/src/cs_image_system/base/models/credentials.py), [provider_specific_image.py](../base/src/cs_image_system/base/models/provider_specific_image.py), [builder_base_runtime.py](../base/src/cs_image_system/base/basic/builder_base_runtime.py), [abstract_version_checker.py](../base/src/cs_image_system/base/basic/abstract_version_checker.py), [power_state.py](../base/src/cs_image_system/base/power_state.py).
|
|
1166
|
+
- Consumers of the hooks: [state_query.py](../base/src/cs_image_system/base/state_query.py), [retention.py](../base/src/cs_image_system/base/retention.py), [dispose.py](../base/src/cs_image_system/base/commands/dispose.py), [verify_instance.py](../base/src/cs_image_system/base/commands/verify_instance.py), [runtime_facts.py](../base/src/cs_image_system/base/commands/runtime_facts.py), [relabel.py](../base/src/cs_image_system/base/commands/relabel.py), [provider_aliases.py](../base/src/cs_image_system/base/provider_aliases.py), [generations.py](../base/src/cs_image_system/base/generations.py), [validate.py](../base/src/cs_image_system/base/commands/validate.py).
|
|
1167
|
+
- [docs/OPERATIONS.md](../../docs/OPERATIONS.md) for the credential contract and the operator's cycles; [docs/DESIGN.md](../../docs/DESIGN.md) for the design; [docs/PLUGINS.md](../../docs/PLUGINS.md) for the package index.
|
|
1168
|
+
- [The configuration reference](../../docs/CONFIGURATION.md) -- every field of the YAML this plugin reads, with an example.
|