cs-image-system-tf-ebs-instance-plugin 0.1.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- cs_image_system/py.typed +0 -0
- cs_image_system/tf_ebs_instance_plugin/__init__.py +3 -0
- cs_image_system/tf_ebs_instance_plugin/helpers.py +8 -0
- cs_image_system/tf_ebs_instance_plugin/main.py +74 -0
- cs_image_system/tf_ebs_instance_plugin/tf_instance_builder.py +677 -0
- cs_image_system/tf_ebs_instance_plugin/tf_instance_models.py +31 -0
- cs_image_system/tf_ebs_instance_plugin/tf_storage_builder.py +833 -0
- cs_image_system/tf_ebs_instance_plugin/tf_storage_models.py +95 -0
- cs_image_system_tf_ebs_instance_plugin-0.1.1.dist-info/METADATA +1376 -0
- cs_image_system_tf_ebs_instance_plugin-0.1.1.dist-info/RECORD +13 -0
- cs_image_system_tf_ebs_instance_plugin-0.1.1.dist-info/WHEEL +4 -0
- cs_image_system_tf_ebs_instance_plugin-0.1.1.dist-info/entry_points.txt +2 -0
- cs_image_system_tf_ebs_instance_plugin-0.1.1.dist-info/licenses/LICENSE +202 -0
|
@@ -0,0 +1,1376 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: cs-image-system-tf-ebs-instance-plugin
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: The OpenTofu plugin for AWS: instances and EBS, EFS and S3 storages as module calls
|
|
5
|
+
Project-URL: Repository, https://github.com/infrastructurebuilder/cs-image-system-3
|
|
6
|
+
Project-URL: Documentation, https://github.com/infrastructurebuilder/cs-image-system-3/blob/develop/docs/OPERATIONS.md
|
|
7
|
+
Project-URL: Package README, https://github.com/infrastructurebuilder/cs-image-system-3/blob/develop/packages/tf-ebs-instance-plugin/README.md
|
|
8
|
+
Author-email: Mykel Alvis <mykelalvis@infrastructurebuilder.org>
|
|
9
|
+
License-Expression: Apache-2.0
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Requires-Python: >=3.13
|
|
12
|
+
Requires-Dist: cs-image-system-aws-runtime-plugin==0.1.1
|
|
13
|
+
Requires-Dist: cs-image-system-hashicorp-utils==0.1.1
|
|
14
|
+
Requires-Dist: cs-image-system-system==0.1.1
|
|
15
|
+
Requires-Dist: pydantic>=2.13
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# cs-image-system-tf-ebs-instance-plugin
|
|
19
|
+
|
|
20
|
+
This package is the OpenTofu plugin for AWS. It registers one **instance
|
|
21
|
+
builder** (`tofu`), which turns every declared instance into a `module` call
|
|
22
|
+
against `tfmodules/aws_instance`, and three **storage builders**
|
|
23
|
+
(`tf-aws-ebs`, `tf-aws-efs`, `tf-aws-s3`), which turn every declared storage
|
|
24
|
+
into a `module` call against the matching `tfmodules/aws_storage_*` module.
|
|
25
|
+
Instances and storages live in one plugin so the instance root can bind to
|
|
26
|
+
the storage roots' outputs through `terraform_remote_state`. Each builder owns
|
|
27
|
+
one terraform root (a workspace), writes a partial backend configuration for
|
|
28
|
+
it, and defers its `plan -> gate -> apply` sequence to the lifecycle runner
|
|
29
|
+
script. The GCE instance root (`tofu-gce`) and the GCP storage builders are
|
|
30
|
+
**not** here: they live in
|
|
31
|
+
[cs-image-system-tf-gcp-plugin](../tf-gcp-plugin/README.md) and subclass the
|
|
32
|
+
classes in this package.
|
|
33
|
+
|
|
34
|
+
## What it registers
|
|
35
|
+
|
|
36
|
+
The entry point, from [pyproject.toml](pyproject.toml):
|
|
37
|
+
|
|
38
|
+
```toml
|
|
39
|
+
[project.entry-points."cs_image_system.plugins.instance"]
|
|
40
|
+
tf-ebs-instance-plugin = "cs_image_system.tf_ebs_instance_plugin.main:initialize"
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
`initialize()` in [main.py](src/cs_image_system/tf_ebs_instance_plugin/main.py)
|
|
44
|
+
returns a `TFEbsTofuTypes` metadata object (plugin metadata version `1`,
|
|
45
|
+
Python `3.13`). The classification of every class comes from its own
|
|
46
|
+
`csis_classifier()`, not from the entry-point group, which is why storage
|
|
47
|
+
builders can be registered from an `instance` entry point.
|
|
48
|
+
|
|
49
|
+
| Service name (`type:` of a builder entry) | Model class | Builder class | Classifications (VCT) | Also registered |
|
|
50
|
+
|---|---|---|---|---|
|
|
51
|
+
| `tofu` | `TofuInstanceBuilderModel` | `TofuInstanceBuilder` | `INSTANCE_BUILDER_MODEL`, `INSTANCE_BUILDER` | `TofuVersionChecker` (`VERSION_CHECKER`) |
|
|
52
|
+
| `tf-aws` | `TofuStorageBuilderModel` | `TofuStorageBuilder` | `STORAGE_BUILDER_MODEL`, `STORAGE_BUILDER` | `TofuStorageVersionChecker` (`VERSION_CHECKER`) |
|
|
53
|
+
| `tf-aws-ebs` | `TofuEbsStorageBuilderModel` | `TofuEbsStorageBuilder` | `STORAGE_BUILDER_MODEL`, `STORAGE_BUILDER` | |
|
|
54
|
+
| `tf-aws-efs` | `TofuEfsStorageBuilderModel` | `TofuEfsStorageBuilder` | `STORAGE_BUILDER_MODEL`, `STORAGE_BUILDER` | |
|
|
55
|
+
| `tf-aws-s3` | `TofuS3StorageBuilderModel` | `TofuS3StorageBuilder` | `STORAGE_BUILDER_MODEL`, `STORAGE_BUILDER` | |
|
|
56
|
+
|
|
57
|
+
`tf-aws` is a base. Its `module_dirname()` and `module_args()` raise
|
|
58
|
+
`NotImplementedError`, so a builder entry with `type: tf-aws` cannot emit
|
|
59
|
+
anything; it exists so the three AWS subclasses and the GCP plugin share one
|
|
60
|
+
implementation. Each subclass binds the base's model type parameter to its
|
|
61
|
+
own model; `TofuS3StorageBuilder` has had its own parameter bound to
|
|
62
|
+
`TofuS3StorageBuilderModel` since stage 63 (a typing change only, so
|
|
63
|
+
`self.model` is known to carry the S3 fields; no behaviour changed).
|
|
64
|
+
|
|
65
|
+
The version checkers are selected by an **executable** entry, not by a
|
|
66
|
+
builder: an entry in `cfg/executables.yml` with `type: tofu` is checked by
|
|
67
|
+
`TofuVersionChecker`, which runs `<binary> --version -json` and reads
|
|
68
|
+
`terraform_version`. `TofuStorageVersionChecker` does the same under the name
|
|
69
|
+
`tf-aws`.
|
|
70
|
+
|
|
71
|
+
### Two levels of `type:`
|
|
72
|
+
|
|
73
|
+
1. A **builder entry** under `instance_builders:` or `storage_builders:`
|
|
74
|
+
selects a plugin class by `type:` = the service name above (`tofu`,
|
|
75
|
+
`tf-aws-ebs`, ...). The entry's `name:` (for example `open-tofu`,
|
|
76
|
+
`aws-ebs`) becomes the terraform workspace name and the directory under
|
|
77
|
+
`generated/`. Its `aliases:` register extra names for the same builder.
|
|
78
|
+
2. An **instance entry** under `instances:` selects a builder by `type:` =
|
|
79
|
+
that builder's name or alias (`type: open-tofu`). `type: default`, or no
|
|
80
|
+
`type:` at all, resolves to the instance builder with `is_default: true`.
|
|
81
|
+
A **storage entry** under `storages:` does the same against the storage
|
|
82
|
+
builders (`type: aws-efs`; `default` resolves to the one with
|
|
83
|
+
`is_default: true`).
|
|
84
|
+
|
|
85
|
+
## Models
|
|
86
|
+
|
|
87
|
+
All models are pydantic dataclasses (`CSIS_MODEL_CONFIG`): unknown keys are
|
|
88
|
+
refused at load. The base class chain is
|
|
89
|
+
`NameTyped -> BuilderModel -> RuntimeEnabledBuilderModel -> InstanceBuilderModel | StorageBuilderModel`
|
|
90
|
+
in [builder_model.py](../base/src/cs_image_system/base/models/builder_model.py),
|
|
91
|
+
[instance_builder.py](../base/src/cs_image_system/base/models/instance_builder.py)
|
|
92
|
+
and [storage_builder.py](../base/src/cs_image_system/base/models/storage_builder.py).
|
|
93
|
+
|
|
94
|
+
Fields every builder model inherits from the base:
|
|
95
|
+
|
|
96
|
+
| Field | Type | Default | Meaning |
|
|
97
|
+
|---|---|---|---|
|
|
98
|
+
| `name` | str | required | Builder name; normalized (lowercase, spaces to `_`). `/` and `\` are refused. |
|
|
99
|
+
| `type` | str | required | The service name that selects the plugin class. |
|
|
100
|
+
| `description` | str \| None | None | Free text. |
|
|
101
|
+
| `aliases` | set[str] | {} | Extra names the registry resolves to this builder. |
|
|
102
|
+
| `executable` | str \| None | None (base) | Name of an entry in `cfg/executables.yml`. |
|
|
103
|
+
| `is_default` | bool | false | The builder chosen for `type: default`. |
|
|
104
|
+
| `config` | dict | {} | Arbitrary map; not read by this plugin. |
|
|
105
|
+
| `gitignore` | list[str] | [] | Extra ignore patterns; not read by this plugin. |
|
|
106
|
+
| `tags` | dict[str, str] | {} | Not read by this plugin (the instance's and storage's own tags are). |
|
|
107
|
+
| `runtime` | str | `default` | Foreign key to a runtime builder; `default` resolves to the default runtime. Required at generation. |
|
|
108
|
+
|
|
109
|
+
The base refuses a `parameters:` key on any builder with a message naming
|
|
110
|
+
`variables:` as the replacement.
|
|
111
|
+
|
|
112
|
+
### `TofuInstanceBuilderModel` (`tofu`)
|
|
113
|
+
|
|
114
|
+
Defined in
|
|
115
|
+
[tf_instance_models.py](src/cs_image_system/tf_ebs_instance_plugin/tf_instance_models.py).
|
|
116
|
+
Extends `InstanceBuilderModel`
|
|
117
|
+
([instance_builder.py](../base/src/cs_image_system/base/models/instance_builder.py)),
|
|
118
|
+
which adds no fields of its own over `RuntimeEnabledBuilderModel`.
|
|
119
|
+
|
|
120
|
+
| Field | Type | Default | Meaning |
|
|
121
|
+
|---|---|---|---|
|
|
122
|
+
| `executable` | str \| None | `"tofu"` | Overrides the base default of None. |
|
|
123
|
+
| `required_plugins` | list[`TFTofuPluginModel`] | [] | Terraform providers for the root's `required_providers`: `name` (required), `version` (required), `source` (optional), `config` (accepted, not read). Empty means one `aws` provider from `hashicorp/aws` with no version constraint. |
|
|
124
|
+
| `state_configuration` | str (foreign key to `STATE_BACKEND_MODEL`) | `default` | The state backend for this workspace, by backend name. `default` resolves through the chain: the runtime's `state_configuration`, else the backend with `is_default: true`. |
|
|
125
|
+
|
|
126
|
+
Not supported: `variables:` (an instance root's module inputs are computed
|
|
127
|
+
from the instance and the runtime), `modifications:` (those belong on
|
|
128
|
+
images), `parameters:` (refused).
|
|
129
|
+
|
|
130
|
+
### The `Instance` item this builder reads
|
|
131
|
+
|
|
132
|
+
The builder consumes base `Instance` items from
|
|
133
|
+
[instance.py](../base/src/cs_image_system/base/models/instance.py). The
|
|
134
|
+
fields that reach the emission:
|
|
135
|
+
|
|
136
|
+
| Field | Type | Default | Meaning | Allowed values |
|
|
137
|
+
|---|---|---|---|---|
|
|
138
|
+
| `type` | str | `default` | The instance builder (name or alias). | any registered instance builder |
|
|
139
|
+
| `image` | str | None | The image to launch; required for generation. The image's `group` is the instance's owning group. | any declared image |
|
|
140
|
+
| `runtime` | str | `default` | The runtime builder. | any runtime |
|
|
141
|
+
| `storages` | list[`StorageMapping`] | [] | Per-instance attachments: `name` (a storage), `mount_point` (default `/mnt/storage`), `min_size` (default 100; not read by this plugin). The same storage listed twice keeps the later mapping. | storages the image's group may use |
|
|
142
|
+
| `userdata` | str | "" | Extra shell lines appended to the generated launch script, before the completion marker. | |
|
|
143
|
+
| `image_policy` | str | `pinned` | `pinned` keeps the launched build until an explicit upgrade; `follow` plans a gated replacement whenever the image's series head moves. | `pinned`, `follow` |
|
|
144
|
+
| `ephemeral` | bool | false | Launch, verify and tear down in the same run. | |
|
|
145
|
+
| `on_failure` | str \| None | None | What a failed verification of an ephemeral does: `keep` leaves it standing and fails the run; `teardown` tears it down and still fails the run. None inherits the runtime builder's value, else `keep`. | `keep`, `teardown` |
|
|
146
|
+
| `teardown_after` | str \| None | None | Duration (`"2h"`, `"30m"`, `"1d"`) after a failed verification past which the next run tears a standing ephemeral down without re-verifying. | |
|
|
147
|
+
| `machine_type` | str | "" | EC2 instance type; empty means the runtime's `default_machine_type`. | |
|
|
148
|
+
| `availability_zone` | str \| None | None | Declared zone. Not read by the module call (an instance lands in the runtime's default subnet); read by the zone-compatibility validator. | |
|
|
149
|
+
| `tags` | dict[str, str] | {} | Passed to the module as `tags`. | |
|
|
150
|
+
| `description`, `config`, `aliases` | | | Not read by this plugin. | |
|
|
151
|
+
|
|
152
|
+
A per-instance `groups:` key is refused: ownership lives on the image.
|
|
153
|
+
|
|
154
|
+
### `TofuStorageBuilderModel` (`tf-aws`)
|
|
155
|
+
|
|
156
|
+
Defined in
|
|
157
|
+
[tf_storage_models.py](src/cs_image_system/tf_ebs_instance_plugin/tf_storage_models.py).
|
|
158
|
+
Extends `StorageBuilderModel`
|
|
159
|
+
([storage_builder.py](../base/src/cs_image_system/base/models/storage_builder.py)),
|
|
160
|
+
which adds the private `_storages` list the loader fills with the builder's
|
|
161
|
+
storage items.
|
|
162
|
+
|
|
163
|
+
| Field | Type | Default | Meaning |
|
|
164
|
+
|---|---|---|---|
|
|
165
|
+
| `executable` | str \| None | `"tofu"` | Overrides the base default. |
|
|
166
|
+
| `required_plugins` | list[`TFTofuPluginModel`] | [] | As for the instance builder; empty means `aws` from `hashicorp/aws`. |
|
|
167
|
+
| `state_configuration` | str (foreign key) | `default` | State backend by name; `default` resolves through the same chain (runtime's, then the default backend). |
|
|
168
|
+
| `variables` | `ModuleVariables` | `ModuleVariables()` | Declared inputs to the module call. Each subclass narrows the type. |
|
|
169
|
+
|
|
170
|
+
### The `variables:` models
|
|
171
|
+
|
|
172
|
+
`ModuleVariables`
|
|
173
|
+
([module_variables.py](../base/src/cs_image_system/base/models/module_variables.py))
|
|
174
|
+
is the base: every field is optional and `None` means "not declared", so the
|
|
175
|
+
module's own default applies. `as_module_args()` returns the declared fields
|
|
176
|
+
except `tags`; `merged_tags(item_tags)` puts the builder's tags under the
|
|
177
|
+
storage item's. Each provider subclass lists exactly its module's variable
|
|
178
|
+
names, so a misspelt key (`performance-mode`, `volume-type`) is refused at
|
|
179
|
+
load instead of being dropped.
|
|
180
|
+
|
|
181
|
+
| Class | Fields (all optional) | Module |
|
|
182
|
+
|---|---|---|
|
|
183
|
+
| `EbsVariables` | `volume_type: str`, `encrypted: bool`, `tags` | [aws_storage_ebs/variables.tf](../../tfmodules/aws_storage_ebs/variables.tf) |
|
|
184
|
+
| `EfsVariables` | `performance_mode: str`, `encrypted: bool`, `tags` | [aws_storage_efs/variables.tf](../../tfmodules/aws_storage_efs/variables.tf) |
|
|
185
|
+
| `S3Variables` | `force_destroy: bool`, `tags` | [aws_storage_s3/variables.tf](../../tfmodules/aws_storage_s3/variables.tf) |
|
|
186
|
+
|
|
187
|
+
Precedence in the module call: what the builder computes from the storage
|
|
188
|
+
item (name, groups, lifecycle, restore snapshot, placement) always wins;
|
|
189
|
+
declared `variables` win over the module's defaults; tags merge with the
|
|
190
|
+
item's over the builder's.
|
|
191
|
+
|
|
192
|
+
### `TofuEbsStorageBuilderModel` (`tf-aws-ebs`)
|
|
193
|
+
|
|
194
|
+
| Field | Type | Default | Meaning |
|
|
195
|
+
|---|---|---|---|
|
|
196
|
+
| `size` | int | 100 | Volume size in GB. A builder field, not a variable: `size` under `variables:` is refused. |
|
|
197
|
+
| `variables` | `EbsVariables` | `EbsVariables()` | `volume_type`, `encrypted`, `tags`. |
|
|
198
|
+
|
|
199
|
+
### `TofuEfsStorageBuilderModel` (`tf-aws-efs`)
|
|
200
|
+
|
|
201
|
+
| Field | Type | Default | Meaning |
|
|
202
|
+
|---|---|---|---|
|
|
203
|
+
| `variables` | `EfsVariables` | `EfsVariables()` | `performance_mode`, `encrypted`, `tags`. |
|
|
204
|
+
|
|
205
|
+
### `TofuS3StorageBuilderModel` (`tf-aws-s3`)
|
|
206
|
+
|
|
207
|
+
| Field | Type | Default | Meaning |
|
|
208
|
+
|---|---|---|---|
|
|
209
|
+
| `bucket_name` | str | required | The DEFAULT bucket for this builder's storages (stage 67): a storage's own `bucket_name` wins, else this, else the storage's name -- one rule (`_bucket_for`) for the module call, the wipe and the state query. Between 2026-09-25 and 26 it was read by nothing (the state query had read it for every storage, so two S3 storages on one builder both reported this one bucket). `get_bucket_name()` still returns it and nothing calls it. |
|
|
210
|
+
| `variables` | `S3Variables` | `S3Variables()` | `force_destroy`, `tags`. |
|
|
211
|
+
|
|
212
|
+
### The `Storage` item the storage builders read
|
|
213
|
+
|
|
214
|
+
Base `Storage` from
|
|
215
|
+
[storage.py](../base/src/cs_image_system/base/models/storage.py):
|
|
216
|
+
|
|
217
|
+
| Field | Type | Default | Meaning | Read by |
|
|
218
|
+
|---|---|---|---|---|
|
|
219
|
+
| `type` | str | `default` | The storage builder (name or alias). | all |
|
|
220
|
+
| `runtime` | str | `default` | Accepted; the builder's runtime governs the root and the provider. | none |
|
|
221
|
+
| `groups` | list[str] | [] | Groups allowed to use the storage. Each gets a private root-level subtree `/<group>/`. The value `ALL` is refused. | EBS (launch script), EFS (access points), S3 (prefixes) |
|
|
222
|
+
| `public_read` | bool | false | Anyone may mount read-only; POSIX permissions still govern. | EFS, S3 |
|
|
223
|
+
| `share_mode` | str | `"2770"` | Subtree mode: `2770` group-private, `2775` read-shared. | EBS (launch script), EFS |
|
|
224
|
+
| `state` | str | `active` | The **requested** lifecycle state. | all |
|
|
225
|
+
| `availability_zone` | str \| None | None | The zone a zonal storage lives in. On EBS the storage's own declaration wins over the runtime's default zone in the module call. | EBS |
|
|
226
|
+
| `bucket_name` | str \| None | None | S3 bucket name; falls back to the storage name. | S3 |
|
|
227
|
+
| `config` | dict | {} | `file_system_id` binds an existing EFS filesystem instead of creating one. | EFS |
|
|
228
|
+
| `lifecycle` | dict \| None | None | Data lifecycle. S3: `transition_days`, `storage_class`, `expire_days`, `prefix`. EFS: `ia_days`, `archive_days`. Refused on EBS. | EFS, S3 |
|
|
229
|
+
| `tags` | dict[str, str] | {} | Merged over the builder's `variables.tags`. | all |
|
|
230
|
+
| `mount_point`, `source`, `ephemeral`, `generative`, `singleton`, `is_default` | | | Not read by these builders. The mount point that matters is the one on the instance's `StorageMapping`. | none |
|
|
231
|
+
|
|
232
|
+
Allowed `state` values: `active`, `archived`, `destroyed`. Allowed
|
|
233
|
+
`share_mode` values: `2770`, `2775`.
|
|
234
|
+
|
|
235
|
+
## The builder
|
|
236
|
+
|
|
237
|
+
Every builder hook is keyed on an `ExecutionLifecyclePhase`. For a phase the
|
|
238
|
+
runner calls, in order, `generate_items_before`, `get_commands_to_run_before`,
|
|
239
|
+
`generate_items_during`, `get_commands_to_run_during`,
|
|
240
|
+
`generate_items_after`, `get_commands_to_run_after`. Each `get_commands_*`
|
|
241
|
+
hook returns two lists: commands that run at generation time, and
|
|
242
|
+
**deferred** commands that reach the lifecycle's `run-<lifecycle>.sh`. At
|
|
243
|
+
finalization the runner calls `pre_finalize_phase`, then the phase's deferred
|
|
244
|
+
commands, then `post_finalize_phase`. A file for phase `P` of builder `B` is
|
|
245
|
+
written at `<B>/<P>/<B>-<P>[-<label>]<suffix>`; the backend/init plumbing
|
|
246
|
+
comes from `TerraformRootMixin` in
|
|
247
|
+
[roots.py](../hashicorp-utils/src/cs_image_system/hashicorp_utils/roots.py).
|
|
248
|
+
|
|
249
|
+
### `TofuInstanceBuilder` (phase `instance-generation`)
|
|
250
|
+
|
|
251
|
+
Source: [tf_instance_builder.py](src/cs_image_system/tf_ebs_instance_plugin/tf_instance_builder.py).
|
|
252
|
+
|
|
253
|
+
The root is emitted only when `_root_has_work()`: the builder's runtime is
|
|
254
|
+
in the run's `--only-runtime` scope, and either instances are declared or a
|
|
255
|
+
recorded (launched) instance is no longer declared. An instance-less root is
|
|
256
|
+
terraform + provider + backend only, and its plan is exactly the pending
|
|
257
|
+
destroys.
|
|
258
|
+
|
|
259
|
+
**1. `generate_items_before`** writes the root file
|
|
260
|
+
`<B>-instance-generation.tf` and the partial backend file:
|
|
261
|
+
|
|
262
|
+
- provider requirements from `required_plugins`, or `aws`/`hashicorp/aws`;
|
|
263
|
+
- a `provider "aws"` block with `region` from the runtime and `profile` from
|
|
264
|
+
the runtime's `credentials.profile_name`, aliased to the workspace name
|
|
265
|
+
(`open_tofu`); module calls bind it with `providers = { aws = aws.open_tofu }`;
|
|
266
|
+
- one `variable "<label>_ami_id"` (string, default `""`) per instance, where
|
|
267
|
+
`<label>` is the instance name with `-` and `.` turned into `_`;
|
|
268
|
+
- `variable "ephemeral_present"` (bool, default `true`) when any instance is
|
|
269
|
+
ephemeral;
|
|
270
|
+
- `variable "sft_enrollment_token"` (string, sensitive, default `""`);
|
|
271
|
+
- the workspace's backend binding (`state_configuration`), a
|
|
272
|
+
`terraform_remote_state` reference to every storage builder that owns
|
|
273
|
+
storages, and one to every identity workspace that owns an instance
|
|
274
|
+
image's group (so gids enter by reference);
|
|
275
|
+
- the `terraform {}` block (with an empty `backend "s3" {}` when
|
|
276
|
+
`config.use_state_backends` is true), the provider blocks, the variable
|
|
277
|
+
blocks, the remote-state data sources;
|
|
278
|
+
- `<B>-instance-generation.tfbackend.hcl`, the partial backend configuration;
|
|
279
|
+
- when the runtime names a concrete VPC (`networking.network`), a
|
|
280
|
+
`data "aws_vpc" "csis_instances"` and a
|
|
281
|
+
`resource "aws_security_group" "csis_instances"`: SSH ingress from the
|
|
282
|
+
runtime's `ssh_ingress_security_group_ids` (or the VPC CIDR when none), all
|
|
283
|
+
egress, `create_before_destroy`. Both bind the aliased provider explicitly.
|
|
284
|
+
|
|
285
|
+
**2. `generate_items_during`** writes, per instance,
|
|
286
|
+
`<B>-instance-generation-instance-<label>.tf` and `user-data-<label>.sh.tftpl`.
|
|
287
|
+
The `.tf` file holds `module "instance_<label>"` with `source` pointing at
|
|
288
|
+
`tfmodules/aws_instance` (resolved from `config.module_source_base`) and
|
|
289
|
+
these arguments:
|
|
290
|
+
|
|
291
|
+
| Argument | Value |
|
|
292
|
+
|---|---|
|
|
293
|
+
| `count` | `var.ephemeral_present ? 1 : 0`, only for ephemeral instances |
|
|
294
|
+
| `name` | the instance name |
|
|
295
|
+
| `ami_id` | the pinned build id when meta-state pins the instance; else the resolved provider-specific image id; else `var.<label>_ami_id` |
|
|
296
|
+
| `ami_name_pattern` | `"<deferred name pattern>*"`, only while the image is deferred (built later in the same run) |
|
|
297
|
+
| `instance_type` | `machine_type`, or the runtime's default machine type |
|
|
298
|
+
| `associate_public_ip_address` | `false` |
|
|
299
|
+
| `vpc_security_group_ids` | `[aws_security_group.csis_instances.id, <runtime addl_security_groups>...]` when the VPC is known |
|
|
300
|
+
| `subnet_id` | the runtime's default subnet |
|
|
301
|
+
| `iam_instance_profile` | the runtime's `session_instance_profile()` when it has one |
|
|
302
|
+
| `tags` | the instance's tags |
|
|
303
|
+
| `user_data` | `templatefile("${path.module}/user-data-<label>.sh.tftpl", {...})` |
|
|
304
|
+
| `ebs_volumes` | `{ <storage> = { volume_id = <remote-state ref>.volume_id, device_name = "/dev/xvdf" } }` for every EBS mount |
|
|
305
|
+
|
|
306
|
+
The template variables are all terraform references: `group_gid` =
|
|
307
|
+
`data.terraform_remote_state.<identity ws>.outputs.group_gids["<group>"]`;
|
|
308
|
+
`ebs` = per-storage `{volume_id, device_name}`; `efs` = per-storage
|
|
309
|
+
`{file_system_id, access_point_id}` (the group's access point, or the
|
|
310
|
+
`public_read` one); `sft_enrollment_token` = `var.sft_enrollment_token`,
|
|
311
|
+
or `(var.sft_enrollment_token != "" ? var.sft_enrollment_token : <group
|
|
312
|
+
builder's token reference>)` when the group's identity plugin supplies one.
|
|
313
|
+
|
|
314
|
+
**Launch parameters.** `compute_launch_params()` in
|
|
315
|
+
[launch_params.py](../base/src/cs_image_system/base/launch_params.py)
|
|
316
|
+
derives, from validated configuration only: the image, the pinned build (or
|
|
317
|
+
`unbound`), the owning group (the image's), the identity type, the mounts
|
|
318
|
+
(storage, capability type, builder, mount point, group, share mode,
|
|
319
|
+
public_read, and for EBS the device `/dev/xvdf`, `/dev/xvdg`, ... in
|
|
320
|
+
declaration order), the enrollment trigger from the group builder, the
|
|
321
|
+
runtime's session mechanism, the canonical hostname (`<name>-NNN`, the
|
|
322
|
+
declared name plus the generation the machine will be), the pool alias when
|
|
323
|
+
one is drawn, `ephemeral`, and any `userdata`.
|
|
324
|
+
`user_data_template()` renders them as a bash script: set the hostname,
|
|
325
|
+
create each mount point, wait for the device (EBS falls back to the
|
|
326
|
+
`/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_<id>` name), `mkfs -t xfs`
|
|
327
|
+
when blank, add an `fstab` line, mount, create and `chgrp`/`chmod` the
|
|
328
|
+
group's subtree, write the enrollment token when one is supplied, then the
|
|
329
|
+
instance's own `userdata` lines, then the completion marker
|
|
330
|
+
`/var/lib/csis/launch-applied`. The parameters are recorded in
|
|
331
|
+
`meta-state/launch-params.yaml` and are **immutable after launch**: a change
|
|
332
|
+
is refused by a validator unless it is a mount removal (a detach) or the
|
|
333
|
+
instance is being replaced.
|
|
334
|
+
|
|
335
|
+
**3. `get_commands_to_run_after`** returns, at generation time, `fmt`,
|
|
336
|
+
`init` and `validate` in the root directory. `init` is `init -backend=false`
|
|
337
|
+
in a dry run; in a real run it is
|
|
338
|
+
`init -reconfigure -backend-config=<B>-instance-generation.tfbackend.hcl`
|
|
339
|
+
when a backend is bound, else bare `init`. No `plan` runs at generation.
|
|
340
|
+
|
|
341
|
+
The **deferred** commands (these are what `run-instance-image.sh` contains)
|
|
342
|
+
come from `gated_apply_commands()`:
|
|
343
|
+
|
|
344
|
+
| Step | Command (in the root directory) |
|
|
345
|
+
|---|---|
|
|
346
|
+
| 1 | `rm -f tfplan` |
|
|
347
|
+
| 2 | `tofu init -input=false -reconfigure [-backend-config=<file>]` |
|
|
348
|
+
| 3 | `cs-image-system --root-dir ... --no-dry-run unmount storage --instance <i> --storage <s> --mount-point <mp> --workdir <dir>` for every mount a launched instance's declaration dropped |
|
|
349
|
+
| 4 | `tofu plan -input=false -out=tfplan [-replace=<addr>...]` |
|
|
350
|
+
| 5 | `cs-image-system gate-plan --planfile tfplan --tofu <binary> [--allow-destroy <addr>]... [--require-unmounted <instance>:<storage>]...` |
|
|
351
|
+
| 6 | only when this root may apply: `cs-image-system apply-check --lifecycle instances --root <B> --root-alias <runtime> [--overlay <path>]... [--apply-runtime <rt>]` |
|
|
352
|
+
| 7 | only when this root may apply: `tofu apply -input=false tfplan` |
|
|
353
|
+
|
|
354
|
+
"May apply" is `config.apply_instances`: `true`, or a list naming the
|
|
355
|
+
builder or its runtime (`--apply-runtime <rt>` sets that list for the run).
|
|
356
|
+
The gate fails unless every destroy in the plan is whitelisted. The
|
|
357
|
+
whitelist holds: `module.instance_<label>` for every recorded instance that
|
|
358
|
+
is no longer declared (a decommission, scoped to this runtime by the build's
|
|
359
|
+
lineage record); `module.instance_<label>.aws_volume_attachment.this` for
|
|
360
|
+
every detach; and, for every instance this plan replaces,
|
|
361
|
+
`module.instance_<label>.aws_instance.this` (the `-replace` address itself)
|
|
362
|
+
and `module.instance_<label>.aws_volume_attachment.this` (an attachment
|
|
363
|
+
binds a volume to the instance's id, so replacing the instance replaces the
|
|
364
|
+
attachment). `-replace=module.instance_<label>.aws_instance.this` is added
|
|
365
|
+
for instances with a pending explicit upgrade or a `follow` target.
|
|
366
|
+
|
|
367
|
+
**Ephemeral instances.** When the root may apply and it has ephemeral
|
|
368
|
+
instances, `_ephemeral_commands()` appends, after step 7: for each
|
|
369
|
+
ephemeral, `cs-image-system ... verify instance <name>` (with
|
|
370
|
+
`--record-only` when its policy is `teardown`), unless a failed verification
|
|
371
|
+
older than `teardown_after` is on record (then no re-verification); then a
|
|
372
|
+
second `rm -f tfplan`, `init`, `plan ... -var=ephemeral_present=false`,
|
|
373
|
+
`gate-plan` whitelisting each `module.instance_<label>`, `apply-check`, and
|
|
374
|
+
`apply`. Under `keep`, a failed `verify` stops the runner before the
|
|
375
|
+
teardown, so the instance stays standing. After a real run the ephemerals'
|
|
376
|
+
launch records and pins are forgotten; under `teardown` a failed verdict
|
|
377
|
+
still fails the run from that hook.
|
|
378
|
+
|
|
379
|
+
**4. `pre_finalize_phase`** runs before the deferred commands, and only for
|
|
380
|
+
a root inside the run's runtime scope (`--only-runtime`, explicit or implied
|
|
381
|
+
by `--apply-runtime`; stage 63, it used to act for out-of-scope roots too
|
|
382
|
+
until 2026-09-25) and only for the `instance-image` lifecycle (the `release`
|
|
383
|
+
and `retention` lifecycles extend the same phase and used to get the file
|
|
384
|
+
too). For every instance
|
|
385
|
+
whose provider-specific image is now resolved (the image builder's post-hook
|
|
386
|
+
has read the packer manifest), it writes
|
|
387
|
+
`<B>/instance-generation/instances.auto.tfvars` with one line
|
|
388
|
+
`<label>_ami_id = "<ami id>"`, so the module uses the exact artifact instead
|
|
389
|
+
of the name-pattern fallback. The file is run-local: a `--commit` never
|
|
390
|
+
stages `*.tfvars`.
|
|
391
|
+
|
|
392
|
+
**5. `post_finalize_phase`** runs after the deferred commands succeed and only
|
|
393
|
+
when the root is inside the run's runtime scope (stage 63: a root out of
|
|
394
|
+
scope applied nothing to bind) and applied. For every non-ephemeral instance: a `follow` target
|
|
395
|
+
moves the pin to the new head; an unpinned instance is bound to the resolved
|
|
396
|
+
image id (first bind); when the image was deferred and the runtime can query
|
|
397
|
+
the booted image, the pin is bound from what actually booted, provided
|
|
398
|
+
lineage records it.
|
|
399
|
+
|
|
400
|
+
### `TofuStorageBuilder` and its subclasses (phase `storage-generation`)
|
|
401
|
+
|
|
402
|
+
Source: [tf_storage_builder.py](src/cs_image_system/tf_ebs_instance_plugin/tf_storage_builder.py).
|
|
403
|
+
|
|
404
|
+
The root is emitted when `_has_work()`: the runtime is in scope, and the
|
|
405
|
+
builder owns declared storages or recorded storages that left the YAML.
|
|
406
|
+
|
|
407
|
+
**Storage state machine.** The YAML declares the *requested* state; the
|
|
408
|
+
authoritative current state is in `meta-state`. The legality rules are in
|
|
409
|
+
[storage_state.py](../base/src/cs_image_system/base/storage_state.py):
|
|
410
|
+
|
|
411
|
+
| From | To | Conditions |
|
|
412
|
+
|---|---|---|
|
|
413
|
+
| never applied | `active` | a new storage must start `active` |
|
|
414
|
+
| `active` | `archived` | unattached; the builder must support archiving |
|
|
415
|
+
| `archived` | `active` | a restore |
|
|
416
|
+
| `active` | `destroyed` | unattached |
|
|
417
|
+
| `archived` | `destroyed` | |
|
|
418
|
+
| `destroyed` | `active` | a new generation of the name; no data continuity |
|
|
419
|
+
| recorded, not destroyed, no longer declared | `destroyed` | the record must name the owning builder |
|
|
420
|
+
|
|
421
|
+
Attaching a storage that is not `active` is an error. A storage that is
|
|
422
|
+
destroyed stays a tombstone in the records.
|
|
423
|
+
|
|
424
|
+
**1. `generate_items_before`** writes `<B>-storage-generation.tf`: provider
|
|
425
|
+
requirements, the aliased `provider "aws"` block (region, profile), the
|
|
426
|
+
backend binding, one `terraform_remote_state` reference per identity
|
|
427
|
+
workspace whose gids this root consumes, the terraform/provider/remote-state
|
|
428
|
+
blocks, and `<B>-storage-generation.tfbackend.hcl`.
|
|
429
|
+
|
|
430
|
+
**2. `generate_items_during`** writes, per *live* storage (requested state
|
|
431
|
+
not `destroyed`, and not `archived` on a builder that supports archiving),
|
|
432
|
+
`<B>-storage-generation-storage-<label>.tf` with a comment naming the allowed
|
|
433
|
+
groups, `public_read` and the requested state, then
|
|
434
|
+
`module "storage_<label>"` (source `tfmodules/<module_dirname>`, the aliased
|
|
435
|
+
provider, `module_args(storage)`), then
|
|
436
|
+
`output "storage_<label>" { value = module.storage_<label> }` for the
|
|
437
|
+
instance root to read. Tombstones and archived storages get a comment-only
|
|
438
|
+
file.
|
|
439
|
+
|
|
440
|
+
Gids never appear as literals: `gid_reference(group)` is
|
|
441
|
+
`data.terraform_remote_state.<identity ws>.outputs.group_gids["<group>"]`,
|
|
442
|
+
and `group_subtrees(storage)` maps each allowed group to
|
|
443
|
+
`{gid, path: "/<group>", permissions: <share_mode>}`.
|
|
444
|
+
|
|
445
|
+
**3. `get_commands_to_run_after`** returns `fmt`, `init`, `validate` at
|
|
446
|
+
generation time and, deferred:
|
|
447
|
+
|
|
448
|
+
- for every tombstone and every undeclared storage whose current state is
|
|
449
|
+
not `destroyed`: the builder's `transition_actions(cur -> destroyed)`,
|
|
450
|
+
then `module.storage_<label>` on the destroy whitelist (unless the storage
|
|
451
|
+
is already archived on an archiving builder, whose volume is already gone);
|
|
452
|
+
- on archiving builders, for every storage requested `archived` whose
|
|
453
|
+
current state is `active`: `transition_actions(active -> archived)`, then
|
|
454
|
+
the whitelisted destroy; and for every live storage currently recorded
|
|
455
|
+
`archived`: `post_transition_actions(archived -> active)` after the apply;
|
|
456
|
+
- then `rm -f tfplan`, `init`, `plan`, `gate-plan --allow-destroy ...`, and,
|
|
457
|
+
when `config.apply_storage` allows this root (builder name or runtime),
|
|
458
|
+
`apply-check --lifecycle storage --root <B> --root-alias <runtime>` and
|
|
459
|
+
`apply -input=false tfplan`, followed by the post-transition scripts.
|
|
460
|
+
|
|
461
|
+
Transition scripts are written beside the root as
|
|
462
|
+
`archive-<label>.sh`, `unarchive-<label>.sh`, `restore-<label>.sh` and run
|
|
463
|
+
with `bash`.
|
|
464
|
+
|
|
465
|
+
**`query_state()`** answers `cs-image-system state query`: for every owned
|
|
466
|
+
storage (declared ones and recorded ones that left the YAML), `present`, the
|
|
467
|
+
capability type and the provider record found by `Name` tag (or, when
|
|
468
|
+
recorded `archived`, the archive snapshot).
|
|
469
|
+
|
|
470
|
+
#### Per-subclass behaviour
|
|
471
|
+
|
|
472
|
+
| | `TofuEbsStorageBuilder` (`tf-aws-ebs`) | `TofuEfsStorageBuilder` (`tf-aws-efs`) | `TofuS3StorageBuilder` (`tf-aws-s3`) |
|
|
473
|
+
|---|---|---|---|
|
|
474
|
+
| Module | [aws_storage_ebs](../../tfmodules/aws_storage_ebs/variables.tf) | [aws_storage_efs](../../tfmodules/aws_storage_efs/variables.tf) | [aws_storage_s3](../../tfmodules/aws_storage_s3/variables.tf) |
|
|
475
|
+
| Capability type (what a base image declares in `storage_types`) | `ebs` | `efs` | `s3` |
|
|
476
|
+
| Attachment cardinality | single | many | many |
|
|
477
|
+
| Zonal (`is_zonal()`) | yes: the zone takes part in the compatibility check and changing it replaces the volume | no | no |
|
|
478
|
+
| POSIX | yes | yes | no (groups become prefixes; `public_read` is a same-account read policy) |
|
|
479
|
+
| Supports `archived` | yes: snapshot named `csis-<label>-archive`, then the volume is destroyed; restore creates the volume from that snapshot and deletes it after the apply | no | no |
|
|
480
|
+
| `lifecycle:` keys | refused | `ia_days` in {1,7,14,30,60,90,180,270,365}; `archive_days` in {90,180,270,365} | `transition_days` (positive int), `storage_class` in {STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER_IR, GLACIER, DEEP_ARCHIVE}, `expire_days` (positive int), `prefix`; a class needs `transition_days`; at least one of the two day counts |
|
|
481
|
+
| `destroyed` transition action | delete the archive snapshot when the volume was archived | none | `aws s3 rm s3://<bucket> --recursive [--profile <p>]` before the destroy |
|
|
482
|
+
| Base-image prerequisites | none (a comment) | `nfs-common` (Debian family) or `amazon-efs-utils`/`nfs-utils`; verify: `mount.efs`/`mount.nfs`/`mount.nfs4` present | AWS CLI installed via the official zip (`unzip` first: it is not on RHEL-family images); verify: `aws` present |
|
|
483
|
+
| State query | `describe_volumes` by `Name` tag, or `describe_snapshots` when archived | `describe_file_systems` matched by name, plus the lifecycle policy | `get_bucket_tagging` on the **storage's** bucket (its `bucket_name`, else its name: the bucket its module call creates; stage 63, it read the builder's `bucket_name` until 2026-09-25); `NoSuchBucket` means absent, any other failure is "unavailable" |
|
|
484
|
+
|
|
485
|
+
Module arguments (`module_args(storage)`), in emission order:
|
|
486
|
+
|
|
487
|
+
| Builder | Arguments |
|
|
488
|
+
|---|---|
|
|
489
|
+
| EBS | declared `variables` (`volume_type`, `encrypted`); `name`; `size` (the builder's); `snapshot_name = csis-<label>-archive` when the storage is recorded archived and declared active (restore); `availability_zone` from the storage's own `availability_zone`, else the runtime's `networking.default_availability_zone`, else `subnet_id` from the runtime's default subnet (the module derives the zone from it); merged `tags` |
|
|
490
|
+
| EFS | declared `variables` (`performance_mode`, `encrypted`); `name`; `existing_file_system_id` from `config.file_system_id`; `transition_to_ia = AFTER_<ia_days>_DAYS`; `transition_to_archive = AFTER_<archive_days>_DAYS`; `access_points` = the group subtrees (gid by reference); `public_read = true` when set; when the runtime declares `networking` with private subnets and `addl_security_groups`: `vpc_id`, `mount_target_subnet_ids` (one private subnet per availability zone, in declaration order) and `client_security_group_ids` (the runtime's `addl_security_groups`), so the module creates the mount targets; merged `tags` |
|
|
491
|
+
| S3 | declared `variables` (`force_destroy`); `bucket_name` = the storage's `bucket_name` or its name; `lifecycle_rules = [{id = "csis", prefix, transition_days, storage_class, expire_days}]` when a lifecycle is declared; `group_prefixes` = sorted allowed groups; `public_read = true` when set; merged `tags` |
|
|
492
|
+
|
|
493
|
+
Undeclared `variables` are absent from the call, so the module default
|
|
494
|
+
applies.
|
|
495
|
+
|
|
496
|
+
## Emission
|
|
497
|
+
|
|
498
|
+
Everything below is from the frozen golden emission over the test fixture.
|
|
499
|
+
|
|
500
|
+
Instance root (`open-tofu`), under
|
|
501
|
+
[generated/instance-image/open-tofu/instance-generation/](../../tests/fixtures/v2_golden/generated/instance-image/open-tofu/instance-generation/):
|
|
502
|
+
|
|
503
|
+
| File | Content |
|
|
504
|
+
|---|---|
|
|
505
|
+
| [open-tofu-instance-generation.tf](../../tests/fixtures/v2_golden/generated/instance-image/open-tofu/instance-generation/open-tofu-instance-generation.tf) | `terraform {}` with `required_providers` and `backend "s3" {}`; `provider "aws"` (region `us-east-2`, profile `noaa`, alias `open_tofu`); `variable "test_ami_id"`, `variable "test2_ami_id"`, `variable "sft_enrollment_token"`; `data "terraform_remote_state"` for `aws_efs`, `aws_ebs`, `aws_s3`, `gcp_pd`, `gcp_gcs`, `oktagroups`; `data "aws_vpc"` and `resource "aws_security_group" "csis_instances"` |
|
|
506
|
+
| [open-tofu-instance-generation-instance-test2.tf](../../tests/fixtures/v2_golden/generated/instance-image/open-tofu/instance-generation/open-tofu-instance-generation-instance-test2.tf) | `module "instance_test2"` calling `../../../../../tfmodules/aws_instance` with `ami_id = var.test2_ami_id`, `ami_name_pattern = "imgfile-basic-dask-pckr-ebs-ans*"`, `instance_type = "t3.medium"`, the security groups, `subnet_id`, `iam_instance_profile`, `user_data = templatefile(...)`, and `ebs_volumes` for `mnt_data` on `/dev/xvdf` |
|
|
507
|
+
| [open-tofu-instance-generation-instance-test.tf](../../tests/fixtures/v2_golden/generated/instance-image/open-tofu/instance-generation/open-tofu-instance-generation-instance-test.tf) | the same shape for `test`, with no storages |
|
|
508
|
+
| [user-data-test2.sh.tftpl](../../tests/fixtures/v2_golden/generated/instance-image/open-tofu/instance-generation/user-data-test2.sh.tftpl) | the launch script: hostname, the `mnt_data` mount at `/mnt/data`, the `coops` subtree with mode `2770`, the enrollment block |
|
|
509
|
+
| [open-tofu-instance-generation.tfbackend.hcl](../../tests/fixtures/v2_golden/generated/instance-image/open-tofu/instance-generation/open-tofu-instance-generation.tfbackend.hcl) | `bucket`, `key = "statefiles/csia-image-system-test/open_tofu.tfstate"`, `region`, `encrypt = true`, `use_lockfile = true`, `profile` |
|
|
510
|
+
| [run-instance-image.sh](../../tests/fixtures/v2_golden/generated/instance-image/run-instance-image.sh) | the deferred sequence per root, each step entered through `cs-image-system materialize .`: `rm -f tfplan`, `init -input=false -reconfigure -backend-config=...`, `plan -input=false -out=tfplan`, `gate-plan` (no apply: the fixture sets `apply_instances: false`) |
|
|
511
|
+
|
|
512
|
+
Storage roots, under
|
|
513
|
+
[generated/storage/](../../tests/fixtures/v2_golden/generated/storage/):
|
|
514
|
+
|
|
515
|
+
| File | Content |
|
|
516
|
+
|---|---|
|
|
517
|
+
| [aws-ebs/storage-generation/aws-ebs-storage-generation.tf](../../tests/fixtures/v2_golden/generated/storage/aws-ebs/storage-generation/aws-ebs-storage-generation.tf) | terraform block, `provider "aws"` alias `aws_ebs`, remote state of `oktagroups` |
|
|
518
|
+
| [aws-ebs/storage-generation/aws-ebs-storage-generation-storage-mnt_data.tf](../../tests/fixtures/v2_golden/generated/storage/aws-ebs/storage-generation/aws-ebs-storage-generation-storage-mnt_data.tf) | `module "storage_mnt_data"` against `tfmodules/aws_storage_ebs`: `volume_type = "gp3"`, `encrypted = true`, `name`, `size = 100`, `subnet_id`, merged `tags`; `output "storage_mnt_data"` |
|
|
519
|
+
| [aws-efs/storage-generation/aws-efs-storage-generation-storage-efs_storage.tf](../../tests/fixtures/v2_golden/generated/storage/aws-efs/storage-generation/aws-efs-storage-generation-storage-efs_storage.tf) | `module "storage_efs_storage"`: `performance_mode`, `encrypted`, `transition_to_ia = "AFTER_30_DAYS"`, `access_points` for `coops` and `stofs` with gids by reference and permissions `2775` |
|
|
520
|
+
| [aws-s3/storage-generation/aws-s3-storage-generation-storage-default_bucket.tf](../../tests/fixtures/v2_golden/generated/storage/aws-s3/storage-generation/aws-s3-storage-generation-storage-default_bucket.tf) | `module "storage_default_bucket"`: `bucket_name`, `lifecycle_rules` (30 days to `STANDARD_IA`), `public_read = true`, merged `tags` |
|
|
521
|
+
| `<B>-storage-generation.tfbackend.hcl` | one per root, e.g. [aws-ebs-storage-generation.tfbackend.hcl](../../tests/fixtures/v2_golden/generated/storage/aws-ebs/storage-generation/aws-ebs-storage-generation.tfbackend.hcl) |
|
|
522
|
+
| [run-storage.sh](../../tests/fixtures/v2_golden/generated/storage/run-storage.sh) | the deferred sequence per root |
|
|
523
|
+
|
|
524
|
+
Meta-state written from these lifecycles:
|
|
525
|
+
[meta-state/launch-params.yaml](../../tests/fixtures/v2_golden/meta-state/launch-params.yaml)
|
|
526
|
+
(the recorded launch parameters) and
|
|
527
|
+
[meta-state/storage.yaml](../../tests/fixtures/v2_golden/meta-state/storage.yaml)
|
|
528
|
+
(the storage read-model: builder, plugin, cardinality, attachments, states).
|
|
529
|
+
A real run with applies on also writes, through the base package's
|
|
530
|
+
after-apply hooks, `storage-state.yaml` (the storage transitions), `pins.yaml`
|
|
531
|
+
(instance pins, moved by this builder's post-finalize hook),
|
|
532
|
+
`instance-state.yaml` (instance generations) and `verifications.yaml` (the
|
|
533
|
+
ephemerals' verdicts); none of these appear in the dry golden.
|
|
534
|
+
|
|
535
|
+
Modules called: [tfmodules/aws_instance](../../tfmodules/aws_instance/main.tf),
|
|
536
|
+
[tfmodules/aws_storage_ebs](../../tfmodules/aws_storage_ebs/main.tf),
|
|
537
|
+
[tfmodules/aws_storage_efs](../../tfmodules/aws_storage_efs/main.tf),
|
|
538
|
+
[tfmodules/aws_storage_s3](../../tfmodules/aws_storage_s3/main.tf). The
|
|
539
|
+
`source` is computed from `config.module_source_base` (default
|
|
540
|
+
`../tfmodules`, relative to the configuration root) as a relative path from
|
|
541
|
+
the workspace directory; absolute paths and remote URLs pass through.
|
|
542
|
+
|
|
543
|
+
## Example configuration
|
|
544
|
+
|
|
545
|
+
From the test fixture. The instance builder,
|
|
546
|
+
[cfg/instance-builders.yml](../../tests/fixtures/config/cfg/instance-builders.yml):
|
|
547
|
+
|
|
548
|
+
```yaml
|
|
549
|
+
instance_builders:
|
|
550
|
+
- name: open-tofu
|
|
551
|
+
type: tofu
|
|
552
|
+
is_default: true
|
|
553
|
+
runtime: aws-east2-runtime
|
|
554
|
+
executable: open-tofu-1
|
|
555
|
+
required_plugins:
|
|
556
|
+
- name: aws
|
|
557
|
+
source: hashicorp/aws
|
|
558
|
+
version: '>= 4.0.0'
|
|
559
|
+
state_configuration: s3-east2
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
The storage builders,
|
|
563
|
+
[cfg/storage-builders.yml](../../tests/fixtures/config/cfg/storage-builders.yml):
|
|
564
|
+
|
|
565
|
+
```yaml
|
|
566
|
+
storage_builders:
|
|
567
|
+
- name: aws-efs
|
|
568
|
+
type: tf-aws-efs
|
|
569
|
+
executable: open-tofu-1
|
|
570
|
+
runtime: aws-east2-runtime
|
|
571
|
+
variables:
|
|
572
|
+
performance_mode: generalPurpose
|
|
573
|
+
encrypted: true
|
|
574
|
+
tags:
|
|
575
|
+
Project: MyProject
|
|
576
|
+
Environment: Development
|
|
577
|
+
- name: aws-ebs
|
|
578
|
+
type: tf-aws-ebs
|
|
579
|
+
executable: open-tofu-1
|
|
580
|
+
is_default: true
|
|
581
|
+
runtime: aws-east2-runtime
|
|
582
|
+
size: 100
|
|
583
|
+
variables:
|
|
584
|
+
volume_type: gp3
|
|
585
|
+
encrypted: true
|
|
586
|
+
tags:
|
|
587
|
+
Project: MyProject
|
|
588
|
+
Environment: Development
|
|
589
|
+
- name: aws-s3
|
|
590
|
+
type: tf-aws-s3
|
|
591
|
+
executable: open-tofu-1
|
|
592
|
+
runtime: aws-east2-runtime
|
|
593
|
+
bucket_name: noscsb-csis-test-514190660293-default-bucket
|
|
594
|
+
variables:
|
|
595
|
+
tags:
|
|
596
|
+
Project: MyProject
|
|
597
|
+
Environment: Development
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
The executable both name,
|
|
601
|
+
[cfg/executables.yml](../../tests/fixtures/config/cfg/executables.yml):
|
|
602
|
+
|
|
603
|
+
```yaml
|
|
604
|
+
executables:
|
|
605
|
+
- name: open-tofu-1
|
|
606
|
+
type: tofu
|
|
607
|
+
binary: /usr/local/bin/tofu
|
|
608
|
+
version: ">1,<2"
|
|
609
|
+
```
|
|
610
|
+
|
|
611
|
+
The storages, [storages/storage0.yaml](../../tests/fixtures/config/storages/storage0.yaml):
|
|
612
|
+
|
|
613
|
+
```yaml
|
|
614
|
+
storages:
|
|
615
|
+
- name: mnt_data
|
|
616
|
+
mount_point: /mnt/data
|
|
617
|
+
is_default: true
|
|
618
|
+
groups:
|
|
619
|
+
- coops
|
|
620
|
+
state: active
|
|
621
|
+
- name: default-bucket
|
|
622
|
+
is_default: false
|
|
623
|
+
runtime: default
|
|
624
|
+
type: aws-s3
|
|
625
|
+
singleton: true
|
|
626
|
+
bucket_name: noscsb-csis-test-514190660293-default-bucket
|
|
627
|
+
public_read: true
|
|
628
|
+
lifecycle:
|
|
629
|
+
transition_days: 30
|
|
630
|
+
storage_class: STANDARD_IA
|
|
631
|
+
tags:
|
|
632
|
+
env: production
|
|
633
|
+
team: devops
|
|
634
|
+
- name: efs-storage
|
|
635
|
+
type: aws-efs
|
|
636
|
+
groups:
|
|
637
|
+
- coops
|
|
638
|
+
- stofs
|
|
639
|
+
share_mode: "2775"
|
|
640
|
+
lifecycle:
|
|
641
|
+
ia_days: 30
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
`mnt_data` has no `type:`, so it belongs to `aws-ebs` (`is_default: true`).
|
|
645
|
+
|
|
646
|
+
An instance that mounts it,
|
|
647
|
+
[instances/instances.yaml](../../tests/fixtures/config/instances/instances.yaml):
|
|
648
|
+
|
|
649
|
+
```yaml
|
|
650
|
+
instances:
|
|
651
|
+
- name: test2
|
|
652
|
+
machine_type: t3.medium
|
|
653
|
+
image: imgfile-basic-dask
|
|
654
|
+
description: Test deploy of CSB Base Image AlmaLinux 9 - Snapshot
|
|
655
|
+
tags:
|
|
656
|
+
hype: test
|
|
657
|
+
storages:
|
|
658
|
+
- name: mnt_data
|
|
659
|
+
mount_point: /mnt/data
|
|
660
|
+
```
|
|
661
|
+
|
|
662
|
+
`test2` has no `type:` and no `runtime:`, so it is built by `open-tofu` on
|
|
663
|
+
`aws-east2-runtime`. Its owning group is `coops`, the group of
|
|
664
|
+
`imgfile-basic-dask`, which is on `mnt_data`'s allowed list.
|
|
665
|
+
|
|
666
|
+
The flags that gate the roots, in
|
|
667
|
+
[cfg/_config.yml](../../tests/fixtures/config/cfg/_config.yml) under
|
|
668
|
+
`config:`: `use_state_backends: true`, `apply_instances: false`,
|
|
669
|
+
`apply_storage: false`, `module_source_base: ../tfmodules`. The optional
|
|
670
|
+
`require_released_builds: true` makes a validator refuse an instance pinned
|
|
671
|
+
to a build that is not a released build of its image, with one grace (the
|
|
672
|
+
series head while its own proof is under way; see "What it tests and
|
|
673
|
+
verifies").
|
|
674
|
+
|
|
675
|
+
## Prerequisites and integration
|
|
676
|
+
|
|
677
|
+
Everything the plugin needs from outside the system, and how it finds each
|
|
678
|
+
thing. Where a value is "read from the runtime", the runtime is the `aws`
|
|
679
|
+
runtime builder the builder entry's `runtime:` names
|
|
680
|
+
([cs-image-system-aws-runtime-plugin](../aws-runtime-plugin/README.md),
|
|
681
|
+
model in
|
|
682
|
+
[aws_runtime_models.py](../aws-runtime-plugin/src/cs_image_system/aws_runtime/aws_runtime_models.py)).
|
|
683
|
+
The manuals' versions of the same facts are
|
|
684
|
+
[CONFIGURATION.md section 4](../../docs/CONFIGURATION.md) (the runtime
|
|
685
|
+
fields) and [OPERATIONS.md "Credentials contract"](../../docs/OPERATIONS.md).
|
|
686
|
+
|
|
687
|
+
| Prerequisite | Why | How the plugin finds it |
|
|
688
|
+
|---|---|---|
|
|
689
|
+
| An OpenTofu (or terraform) binary | every root is `fmt`/`init`/`validate`d at generation and planned, gated and applied by the runner | the builder's `executable:` names an entry in `cfg/executables.yml`; that entry's `binary:` is the path and `type: tofu` selects `TofuVersionChecker`, which runs `<binary> --version -json`. The entry's `version:` requirement is enforced: `validate` and every run refuse a binary outside it. The default `executable` value is the literal `"tofu"`, so an entry of that name must exist unless the builder names another. |
|
|
690
|
+
| The terraform modules | each root is a `module` call against `tfmodules/aws_instance`, `aws_storage_ebs`, `aws_storage_efs` or `aws_storage_s3` | `config.module_source_base` in `cfg/_config.yml` (default `../tfmodules`, relative to the configuration root; an absolute path or a git/registry URL passes through). A fixture copy needs the `tfmodules` directory beside it. |
|
|
691
|
+
| An AWS account, region and credentials | the `provider "aws"` block, every plan and apply, the state queries (boto3), the archive/restore/wipe scripts (the AWS CLI) | `region` from the runtime's `region:`; `profile` from the runtime's `credentials.profile_name` (emitted into the provider block, the backend file and the CLI scripts as `--profile`). With no profile named, the provider and boto3 fall back to the environment (`AWS_PROFILE`, or `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`/`AWS_SESSION_TOKEN`). An SSO profile needs a live session (`aws sso login --profile <p>`): the load itself refuses on an expired one, and a session that lapses mid-run is environmental. |
|
|
692
|
+
| The `aws` CLI on the runner's `PATH` | `archive-<label>.sh` (`describe-snapshots`, `create-snapshot`, `wait snapshot-completed`), `unarchive-<label>.sh` and `restore-<label>.sh` (`delete-snapshot`), and the S3 wipe `aws s3 rm --recursive` before a bucket's destroy | run as deferred steps of `run-storage.sh` with `bash` and `aws` found on `PATH`; the EBS scripts pass `--region` and `--profile` from the runtime, the S3 wipe passes `--profile` only. Nothing checks the CLI's presence before the step runs. |
|
|
693
|
+
| `cs-image-system` on `PATH` and the configuration identity | the runner scripts call `cs-image-system materialize`, `gate-plan`, `apply-check`, `unmount storage` and `verify instance`; `materialize` and every configuration-loading step open `ENC[age:...]` values | `just` and `uv run` put the CLI on `PATH`; `CSIS_CONFIG_IDENTITY` must be in the environment of whoever runs the script (OPERATIONS "Encrypted values"). |
|
|
694
|
+
| A VPC, subnets and existing security groups | instances launch in the runtime's default subnet, wear a security group this root creates plus the groups the runtime lists, and take SSH only from the relay's group; EFS mount targets need one private subnet per zone and the clients' groups | all from the runtime's `networking:` block: `network` (the VPC id; when it is unset or a placeholder, no `aws_vpc`/`aws_security_group` blocks are emitted and the module gets no `vpc_security_group_ids`), `default_subnet_id`, `default_availability_zone`, `subnets[]` (`subnet_id`, `availability_zone`, `public`), `ssh_ingress_security_group_ids`, `addl_security_groups`. Existing groups, subnets and routes are never modified (the standing network rule); only the `csis-instances` group and the EFS module's mount-target group are created. |
|
|
695
|
+
| No internet gateway is assumed | instances get no public IP and sftd advertises the private address | `associate_public_ip_address = false` is hard-coded; the launch script writes `AccessAddress` from `hostname -I` when the group enrolls through `sftd-token`. |
|
|
696
|
+
| Session access (SSM) and its instance profile | the runtime's `verify instance`, the unmount before a detach, the post-bake tests and the alias pass all reach the machine over the session mechanism; the instance must be able to reach SSM | the runtime's `session_mechanism: ssm` and `session_instance_profile:`; the profile is passed as `iam_instance_profile` and must exist in the account (operator wiring; the system generates no IAM). |
|
|
697
|
+
| The identity root, applied first | gids and the enrollment token enter by `terraform_remote_state` from the identity workspace's outputs (`group_gids`, and the group builder's token reference) | the group builder of the image's owning group (`group_builder_of`) names the workspace; the storage lifecycle and the instance-image lifecycle run after `identity` in every run, and the reference resolves only when that state has been applied. A group with no identity builder is a generation-time error on a storage root. |
|
|
698
|
+
| The state backend | the workspace's own state, the partial backend file, and the remote-state data sources that read the storage and identity roots | `state_configuration` on the builder, else the runtime's, else the backend marked `is_default` in `cfg/state-backends*.yml` (an S3 backend needs the same AWS session; see [cs-image-system-tf-s3-state-plugin](../tf-s3-state-plugin/README.md)). Everything about state is gated by `config.use_state_backends`; note under "Variations" what happens when it is off. |
|
|
699
|
+
| `TF_VAR_sft_enrollment_token` (optional) | an explicit override of the enrollment token, and the only source when the group's identity plugin mints none | read by tofu at plan/apply as `var.sft_enrollment_token`; the variable is sensitive and never recorded. |
|
|
700
|
+
| `TF_PLUGIN_CACHE_DIR` (optional) | provider downloads are reused across roots | exported by the Justfile; a run started outside `just` needs it in the environment to benefit. One tofu process at a time: the recipes take `.tofu-plugin-cache/.lock`. |
|
|
701
|
+
| The alias pool (optional) | a memorable name for every new durable machine | `meta-state/aliases.txt` in the configuration root, hand-appended; absent means no aliases and no error. |
|
|
702
|
+
| Base images that declare the storage type | an instance may mount a storage only when its image's root base build was stamped with that capability type (`ebs`, `efs`, `s3`); the EFS and S3 prerequisites (mount tooling, the AWS CLI) are baked then | the base image's `storage_types:`; the plugin supplies `base_image_prerequisites()` and `verify_commands()` per type to the base-image bake. |
|
|
703
|
+
|
|
704
|
+
Nothing here is read from this package's own environment variables: the
|
|
705
|
+
plugin declares none.
|
|
706
|
+
|
|
707
|
+
## Configuration reference
|
|
708
|
+
|
|
709
|
+
The field tables under "Models" above are the complete list of fields on the
|
|
710
|
+
builder models this plugin owns (`TofuInstanceBuilderModel`,
|
|
711
|
+
`TofuStorageBuilderModel` and its three subclasses, the `variables:` models)
|
|
712
|
+
and of the `Instance`, `StorageMapping` and `Storage` fields the builders
|
|
713
|
+
read, with the ones that are accepted but not read named as such. This
|
|
714
|
+
section lists everything else the plugin reads: the `config:` keys, the
|
|
715
|
+
runtime fields, the executable entry, the meta-state it consults, and the
|
|
716
|
+
run options that change its output. Manual references:
|
|
717
|
+
[CONFIGURATION.md sections 2.3, 8, 11.3 and 11.4](../../docs/CONFIGURATION.md).
|
|
718
|
+
|
|
719
|
+
`config:` keys in `cfg/_config.yml` (and in overlays):
|
|
720
|
+
|
|
721
|
+
| Key | Type | Default | Meaning for this plugin |
|
|
722
|
+
|---|---|---|---|
|
|
723
|
+
| `apply_instances` | bool or list[str] | `false` | whether the instance root's runner carries `apply-check` + `apply`; a list names builders or runtimes. Also gates the ephemeral verify/teardown block, the post-finalize pin binds, and (through the base hooks) the `launched` marker, the forgets and the alias draw. |
|
|
724
|
+
| `apply_storage` | bool or list[str] | `false` | the same for each storage root, and for the post-transition (restore) scripts. |
|
|
725
|
+
| `use_state_backends` | bool | `false` | when true: `backend "s3" {}` in the terraform block, the `.tfbackend.hcl` file, the `-backend-config=` init argument and every `data "terraform_remote_state"` block are emitted. |
|
|
726
|
+
| `module_source_base` | str | `../tfmodules` | where the four modules are. |
|
|
727
|
+
| `require_released_builds` | bool | `false` | read by the base validator, not by the builder: an instance's pin must be a released build, or the series head under the release grace. |
|
|
728
|
+
|
|
729
|
+
Runtime fields (the `aws` runtime the builder's `runtime:` names):
|
|
730
|
+
|
|
731
|
+
| Field | Read by | Used for |
|
|
732
|
+
|---|---|---|
|
|
733
|
+
| `region` (`get_region()`) | both | the provider block's `region`; the EBS/EFS/S3 CLI flags and boto3 sessions |
|
|
734
|
+
| `credentials.profile_name` (`get_credentials()`) | both | the provider block's `profile`; `--profile` on the CLI scripts; the boto3 session. It is the only source of the profile: stage 63 removed the storage builder's dead fallback to a `profile` attribute on the runtime model (`rtb.model.profile`, which no runtime declares; removed 2026-09-25). |
|
|
735
|
+
| `state_configuration` | both | the second rung of the backend chain |
|
|
736
|
+
| `networking.network` | instance | the VPC for `data "aws_vpc"` and the `csis-instances` security group; also gates `vpc_security_group_ids` |
|
|
737
|
+
| `networking.ssh_ingress_security_group_ids` | instance | the SSH ingress sources of `csis-instances`; empty falls back to the VPC's CIDR |
|
|
738
|
+
| `networking.addl_security_groups` | instance, EFS | extra groups every instance wears; the EFS mount targets' client groups |
|
|
739
|
+
| `networking.default_subnet_id` | instance, EBS | `subnet_id` of the instance; the EBS zone fallback |
|
|
740
|
+
| `networking.default_availability_zone` | EBS | the volume's zone when the storage declares none |
|
|
741
|
+
| `networking.subnets[]` (`subnet_id`, `availability_zone`, `public`) | EFS | one private subnet per zone becomes a mount target |
|
|
742
|
+
| `default_machine_type` | instance | `instance_type` when the instance's `machine_type` is empty |
|
|
743
|
+
| `session_mechanism`, `session_instance_profile` | instance | `iam_instance_profile` when the mechanism is `ssm` and a profile is named |
|
|
744
|
+
| `on_failure`, `teardown_after` | instance | the ephemeral defaults an instance inherits |
|
|
745
|
+
| `self_to_aws_client_config()` | storage | the boto3 session of the state query |
|
|
746
|
+
|
|
747
|
+
The executable entry (`cfg/executables.yml`, `type: tofu`): `name` (what the
|
|
748
|
+
builder's `executable:` names), `binary` (the path the roots run), `version`
|
|
749
|
+
(the requirement the checker enforces).
|
|
750
|
+
|
|
751
|
+
Meta-state the builders read at generation (the configuration root's
|
|
752
|
+
`meta-state/`):
|
|
753
|
+
|
|
754
|
+
| File | Read for |
|
|
755
|
+
|---|---|
|
|
756
|
+
| `pins.yaml` | the instance's pinned build (`ami_id`), pending replacements (`-replace`), the decommission whitelist |
|
|
757
|
+
| `launch-params.yaml` | launched instances and their recorded mounts (detaches), the decommission whitelist, the recorded hostname and alias |
|
|
758
|
+
| `lineage.yaml` | the pinned build's runtime (scoping the decommission whitelist to this root), series heads (`follow`), what a booted image is |
|
|
759
|
+
| `storage-state.yaml` | each storage's current state (transitions, restore, the archive lookup in the state query) and the facts of undeclared storages (builder, bucket name) |
|
|
760
|
+
| `verifications.yaml` | a standing ephemeral's last verdict (`teardown_after`) |
|
|
761
|
+
| `instance-state.yaml` | the generation counters behind the canonical hostname |
|
|
762
|
+
| `aliases.txt` | the alias pool |
|
|
763
|
+
|
|
764
|
+
Run options that change the emission: `--dry-run` (the default) versus
|
|
765
|
+
`--no-dry-run`; `--only-runtime <rt>` and `--apply-runtime <rt>`; `--overlay`
|
|
766
|
+
and `--undeclare instance:<name>` / `{name: X, undeclare: true}` (a
|
|
767
|
+
decommission for one invocation); `--migrate-state <root>` (the runner's
|
|
768
|
+
init becomes `-migrate-state -force-copy` and the plan `-detailed-exitcode`,
|
|
769
|
+
wrapped in `state-migration begin`/`finish`).
|
|
770
|
+
|
|
771
|
+
### Variations
|
|
772
|
+
|
|
773
|
+
- **Ephemeral versus durable instance.** When `ephemeral: true` is declared,
|
|
774
|
+
the module call gets `count = var.ephemeral_present ? 1 : 0` and the root
|
|
775
|
+
declares `variable "ephemeral_present"`; when the root may apply, the
|
|
776
|
+
runner appends `verify instance <name>` and a second plan/gate/apply with
|
|
777
|
+
`-var=ephemeral_present=false` whitelisting `module.instance_<label>`;
|
|
778
|
+
the post-finalize hook binds no pin for it; no alias is drawn from the
|
|
779
|
+
pool; its hostname counts on the `ephemeral` generation counter, so a
|
|
780
|
+
durable machine of the same name never moves. A durable instance keeps
|
|
781
|
+
its module call unconditional, is bound to its build after the first
|
|
782
|
+
applied launch, and is replaced only through an upgrade or a follow.
|
|
783
|
+
- **`on_failure: keep` versus `teardown`, and `teardown_after`.** Under
|
|
784
|
+
`keep` (the default, inherited from the runtime when the instance says
|
|
785
|
+
nothing) a failed `verify instance` stops the runner before the teardown
|
|
786
|
+
and the machine stays standing; under `teardown` the verify runs with
|
|
787
|
+
`--record-only`, the teardown proceeds, and the failed verdict fails the
|
|
788
|
+
run from the after-apply hook after the record is forgotten. When a
|
|
789
|
+
failed verification older than `teardown_after` is on record, the next
|
|
790
|
+
run emits no `verify` step for that instance and tears it down directly.
|
|
791
|
+
- **`image_policy: pinned` versus `follow`.** Pinned: `ami_id` is the
|
|
792
|
+
recorded pin and a newer series head changes nothing until
|
|
793
|
+
`upgrade instance`. Follow: when the series head is newer than the pin,
|
|
794
|
+
the plan carries `-replace=module.instance_<label>.aws_instance.this`,
|
|
795
|
+
the replace address and the instance's volume attachments are
|
|
796
|
+
whitelisted, `ami_id` is the head, the canonical hostname takes the next
|
|
797
|
+
generation, and the pin moves in the post-finalize hook after the apply.
|
|
798
|
+
- **A pending upgrade.** `upgrade instance <name>` leaves a pending
|
|
799
|
+
replacement marker; the next generation treats the instance exactly like
|
|
800
|
+
a follow target (the `-replace`, the attachment whitelist, the new
|
|
801
|
+
hostname and alias), and `mark_launched` clears the marker after the
|
|
802
|
+
apply. Under `require_released_builds: true` the unreleased pin is
|
|
803
|
+
admitted by the release grace while the proof is under way.
|
|
804
|
+
- **Image resolved, deferred, or pinned.** `ami_id` is the pin when
|
|
805
|
+
meta-state has one; else the resolved provider-specific image id; else
|
|
806
|
+
(the image bakes later in this run) `var.<label>_ami_id` with
|
|
807
|
+
`ami_name_pattern` as the module's fallback, and `pre_finalize_phase`
|
|
808
|
+
writes `instances.auto.tfvars` once the packer manifest is read. With no
|
|
809
|
+
provider-specific image at all, no `ami_id` is emitted and a warning is
|
|
810
|
+
logged.
|
|
811
|
+
- **Dry run versus real run.** A dry run runs `init -backend=false` at
|
|
812
|
+
generation, enumerates the deferred commands in the log without running
|
|
813
|
+
them, runs neither finalize hook, draws no alias (it logs the name it
|
|
814
|
+
would take), forgets no record and binds no pin. A real run inits with
|
|
815
|
+
`-reconfigure -backend-config=<file>` (or bare `init` with no backend),
|
|
816
|
+
executes the runner, and the hooks act.
|
|
817
|
+
- **Apply flag on, off, or a list.** Off: the runner still plans and gates
|
|
818
|
+
every root (a decommission can be dry-run repeatedly) but carries no
|
|
819
|
+
`apply-check`/`apply`, no ephemeral block, and the post-finalize hook
|
|
820
|
+
returns at once. A list (`[gcloud-east1]`, or `--apply-runtime`) enables
|
|
821
|
+
only the roots whose builder name or runtime is listed; the other roots
|
|
822
|
+
still plan and gate. `apply-check` re-reads the flag at execution time.
|
|
823
|
+
- **`--only-runtime <rt>` (explicit or implied by `--apply-runtime`).**
|
|
824
|
+
A builder whose runtime is not `<rt>` emits no root at all and contributes
|
|
825
|
+
no commands; with no scope every root of every runtime is generated and
|
|
826
|
+
planned. Both finalize hooks of an out-of-scope instance builder return
|
|
827
|
+
at once (stage 63): `pre_finalize_phase` writes no `instances.auto.tfvars`
|
|
828
|
+
and `post_finalize_phase` binds nothing, matching the root's own empty
|
|
829
|
+
emission. Until 2026-09-25 another runtime's builder still ran
|
|
830
|
+
`pre_finalize_phase`, wrote tfvars for a root that emitted nothing, and
|
|
831
|
+
logged `No resolved provider-specific images for instances; modules will
|
|
832
|
+
use their name-pattern fallback` on every scoped run (visible in every GCE
|
|
833
|
+
cycle).
|
|
834
|
+
- **`use_state_backends` on versus off.** On: the terraform block gets an
|
|
835
|
+
empty `backend "s3" {}`, the `.tfbackend.hcl` file is written, init
|
|
836
|
+
carries `-backend-config=`, and every `data "terraform_remote_state"`
|
|
837
|
+
block is emitted. Off: none of those, while the module calls and the
|
|
838
|
+
launch template still reference `data.terraform_remote_state.<ws>...`
|
|
839
|
+
for gids, enrollment tokens, volume ids and access points, so an
|
|
840
|
+
instance with an owning group or a mount, or a storage with allowed
|
|
841
|
+
groups, produces a root that does not validate. The flag is effectively
|
|
842
|
+
required for anything but a groupless, mountless emission.
|
|
843
|
+
- **`required_plugins` declared or not.** Declared: exactly those providers
|
|
844
|
+
with their sources and version constraints. Not declared: one `aws`
|
|
845
|
+
provider from `hashicorp/aws` with no constraint.
|
|
846
|
+
- **`state_configuration` own, runtime's, or default.** The workspace binds
|
|
847
|
+
to the first that names a backend; a consumer's remote-state data source
|
|
848
|
+
carries the producer's backend, whichever type it is (the golden reads
|
|
849
|
+
`oktagroups` from a `local` backend and the storage roots from a second
|
|
850
|
+
S3 bucket).
|
|
851
|
+
- **A concrete VPC or not.** With `networking.network` set to a real id,
|
|
852
|
+
the root creates `csis-instances` and the module wears it plus
|
|
853
|
+
`addl_security_groups`; with `ssh_ingress_security_group_ids` empty the
|
|
854
|
+
SSH ingress is the VPC's own CIDR instead of a group reference. Without
|
|
855
|
+
a VPC no security group is created and `vpc_security_group_ids` is
|
|
856
|
+
absent (the module leaves it null).
|
|
857
|
+
- **`machine_type` set or empty; a session profile or none.** Empty falls
|
|
858
|
+
back to the runtime's `default_machine_type`; `iam_instance_profile` is
|
|
859
|
+
emitted only under `session_mechanism: ssm` with a
|
|
860
|
+
`session_instance_profile`.
|
|
861
|
+
- **Requested storage state: `active`, `archived`, `destroyed`; and an
|
|
862
|
+
entry that left the YAML.** `active` emits the module call and the
|
|
863
|
+
output; `archived` on EBS emits a comment-only file, `archive-<label>.sh`
|
|
864
|
+
(snapshot, wait) before the plan and whitelists `module.storage_<label>`;
|
|
865
|
+
`archived` on EFS or S3 is refused at validate; `destroyed` emits a
|
|
866
|
+
comment-only tombstone file, the builder's destroy action (S3: the
|
|
867
|
+
bucket wipe; EBS from archived: the snapshot delete) and the whitelist;
|
|
868
|
+
an undeclared recorded storage is planned like `destroyed` from its
|
|
869
|
+
record's facts, without any file.
|
|
870
|
+
- **A restore (recorded `archived`, declared `active`).** The EBS module
|
|
871
|
+
call carries `snapshot_name = csis-<label>-archive`; after a successful
|
|
872
|
+
apply `restore-<label>.sh` deletes the snapshot.
|
|
873
|
+
- **EBS placement.** `availability_zone` from the storage's own declaration
|
|
874
|
+
wins; else the runtime's `default_availability_zone`; else `subnet_id`
|
|
875
|
+
from the runtime's default subnet and the module derives the zone.
|
|
876
|
+
Declaring a zone pins the volume; a zone change replaces it.
|
|
877
|
+
- **EFS created or adopted; networking or not.** With
|
|
878
|
+
`config.file_system_id` the module adopts that filesystem
|
|
879
|
+
(`existing_file_system_id`) and applies no lifecycle policy of its own;
|
|
880
|
+
otherwise it creates one. With a runtime `networking` block that has
|
|
881
|
+
private subnets and `addl_security_groups`, the call carries `vpc_id`,
|
|
882
|
+
`mount_target_subnet_ids` and `client_security_group_ids`; with either
|
|
883
|
+
list empty none of the three is passed and the module makes no mount
|
|
884
|
+
target, so nothing can mount the filesystem.
|
|
885
|
+
- **S3 bucket name.** The storage's `bucket_name`, else its name, is the
|
|
886
|
+
bucket the module manages, the wipe empties and (stage 63) the state
|
|
887
|
+
query reads, so two S3 storages on one builder each report their own
|
|
888
|
+
bucket. The builder's own `bucket_name` is still required but nothing
|
|
889
|
+
reads it; until 2026-09-25 it was the bucket the state query read, and
|
|
890
|
+
every storage on the builder reported that one bucket.
|
|
891
|
+
- **Groups versus `public_read`.** Allowed groups become EBS subtrees in the
|
|
892
|
+
launch script, EFS access points (`access_points`) and S3 prefixes
|
|
893
|
+
(`group_prefixes`), each gid by reference; `public_read: true` adds the
|
|
894
|
+
EFS root access point / the S3 same-account read policy, and the instance
|
|
895
|
+
template mounts a `public_read` EFS through the public access point
|
|
896
|
+
rather than the group's.
|
|
897
|
+
- **Declared `variables` or not; tags.** A declared variable is passed
|
|
898
|
+
verbatim; an undeclared one is absent so the module default applies
|
|
899
|
+
(`volume_type` `gp3`, `encrypted` true, `performance_mode`
|
|
900
|
+
`generalPurpose`, `force_destroy` false). Tags are the builder's
|
|
901
|
+
`variables.tags` under the storage item's, and are absent when both are
|
|
902
|
+
empty.
|
|
903
|
+
- **Encrypted versus clear values.** The plugin treats an `ENC[age:...]`
|
|
904
|
+
value like any other string: the committed emission carries the
|
|
905
|
+
ciphertext and each runner step enters the root through
|
|
906
|
+
`cs-image-system materialize .`, which writes the plaintext mirror the
|
|
907
|
+
tofu commands run in (OPERATIONS "Encrypted values"). Nothing in this
|
|
908
|
+
package decrypts.
|
|
909
|
+
|
|
910
|
+
## What it tests and verifies
|
|
911
|
+
|
|
912
|
+
What this plugin and the base machinery around it check, when, and where
|
|
913
|
+
the verdict lands. "Validator" means a function registered on the run's
|
|
914
|
+
validator list: `cs-image-system validate` and every `run` execute them
|
|
915
|
+
before anything is emitted, and one failure is exit 1 with every message
|
|
916
|
+
printed and recorded in `generated/run-summary.json` under
|
|
917
|
+
`validation_errors`.
|
|
918
|
+
|
|
919
|
+
**At load (pydantic, before any validator).** The verdict is a
|
|
920
|
+
`ValidationError` naming the file and key; the load stops.
|
|
921
|
+
|
|
922
|
+
- Unknown keys on any model this package owns are refused
|
|
923
|
+
(`CSIS_MODEL_CONFIG`); `parameters:` on a builder is refused with a
|
|
924
|
+
message naming `variables:` as the replacement.
|
|
925
|
+
- `variables:` keys are typed per provider: `performance-mode`,
|
|
926
|
+
`volume-type`, or `size` under `variables:` on `tf-aws-ebs` are refused.
|
|
927
|
+
- `bucket_name` is required on a `tf-aws-s3` builder (still required,
|
|
928
|
+
although since stage 63 nothing reads it: the state query takes the
|
|
929
|
+
storage's bucket).
|
|
930
|
+
- A `Storage` refuses `state` outside `active`/`archived`/`destroyed`,
|
|
931
|
+
`share_mode` outside `2770`/`2775`, the `ALL` group value, an empty
|
|
932
|
+
`runtime` or `type`.
|
|
933
|
+
- An `Instance` refuses a `groups:` key; a `StorageMapping` refuses an
|
|
934
|
+
empty `mount_point` or a non-positive `min_size` (its `aliases` are
|
|
935
|
+
dropped with a warning).
|
|
936
|
+
- A builder `name` containing `/` or `\` is refused.
|
|
937
|
+
|
|
938
|
+
**At `validate` (and at the start of every `run`).** Base validators, in
|
|
939
|
+
[v2_validation.py](../base/src/cs_image_system/base/v2_validation.py),
|
|
940
|
+
[launch_params.py](../base/src/cs_image_system/base/launch_params.py),
|
|
941
|
+
[lineage.py](../base/src/cs_image_system/base/lineage.py),
|
|
942
|
+
[release.py](../base/src/cs_image_system/base/release.py),
|
|
943
|
+
[validate.py](../base/src/cs_image_system/base/commands/validate.py) and
|
|
944
|
+
[alias_pool.py](../base/src/cs_image_system/base/alias_pool.py), some of
|
|
945
|
+
which call back into this plugin:
|
|
946
|
+
|
|
947
|
+
- the storage attach rule: a storage's allowed groups exist; an instance's
|
|
948
|
+
owning group is on the list (or the storage is `public_read`);
|
|
949
|
+
- attachment cardinality: two instances on one `ebs` storage are refused,
|
|
950
|
+
naming the multi-host builders on that runtime (this plugin's
|
|
951
|
+
`attachment_cardinality()`);
|
|
952
|
+
- `lifecycle:` declarations, through this plugin's `validate_lifecycle()`:
|
|
953
|
+
refused outright on EBS; the EFS and S3 keys and values as tabled above;
|
|
954
|
+
- the storage state machine: illegal transitions, a new storage not
|
|
955
|
+
starting `active`, a transition out of `active` while attached,
|
|
956
|
+
`archived` on a builder that cannot archive (`supports_archive()`), an
|
|
957
|
+
undeclared storage whose record names no builder, an attachment of a
|
|
958
|
+
storage that is not `active`;
|
|
959
|
+
- capabilities: a mounted storage's type (`capability_type()`) must be in
|
|
960
|
+
the image's root base build's stamped `storage_types`;
|
|
961
|
+
- launch-parameter immutability: a launched instance whose recomputed
|
|
962
|
+
parameters differ from `launch-params.yaml` (other than a mount removal
|
|
963
|
+
or a sanctioned replacement) is refused naming the changed keys;
|
|
964
|
+
- claimed hostnames: only when the instance-image lifecycle is requested
|
|
965
|
+
and `apply_instances` is on, each canonical hostname is checked against
|
|
966
|
+
the group's OPA registry; a registered name for an unlaunched instance,
|
|
967
|
+
two registrations for a launched one, or a registry that did not answer
|
|
968
|
+
are refusals;
|
|
969
|
+
- policies: `image_policy`, `on_failure` and `teardown_after` values;
|
|
970
|
+
- released pins: under `require_released_builds`, a pin that is not a
|
|
971
|
+
released build is refused unless the release grace admits it (the
|
|
972
|
+
series head on its runtime, verified in bake, no failed post-bake
|
|
973
|
+
record, and the instance is a pending replacement onto it or stands on
|
|
974
|
+
it) -- the grace is logged as a warning, the refusal names what is
|
|
975
|
+
missing;
|
|
976
|
+
- availability zones: more than one distinct zone across an instance, the
|
|
977
|
+
zonal storages it mounts (`is_zonal()`) and its runtime's effective
|
|
978
|
+
subnet is refused naming each claimant;
|
|
979
|
+
- the alias pool: a free line that is not a legal hostname label, repeats
|
|
980
|
+
another, or is a declared instance name;
|
|
981
|
+
- every declared executable exists and its version satisfies the
|
|
982
|
+
requirement (`TofuVersionChecker`).
|
|
983
|
+
|
|
984
|
+
**At generation.** Each emitted root runs `tofu fmt`, `tofu init`
|
|
985
|
+
(`-backend=false` in a dry run) and `tofu validate` in its directory; a
|
|
986
|
+
non-zero exit fails the generation with a log line naming the builder and
|
|
987
|
+
the command. A storage whose allowed group has no identity builder raises
|
|
988
|
+
`ValueError` from `gid_reference`. An instance with no provider-specific
|
|
989
|
+
image logs a warning and gets no `ami_id`. A `tf-aws` builder entry raises
|
|
990
|
+
`NotImplementedError` at its first module call. The alias draw logs what it
|
|
991
|
+
took or would take. Nothing plans at generation.
|
|
992
|
+
|
|
993
|
+
**In the runner (apply time).** `gate-plan` refuses a missing or stale
|
|
994
|
+
planfile and any destroy that is not whitelisted, and refuses a detach
|
|
995
|
+
without a successful receipt in `unmount-receipts/<instance>__<storage>.json`
|
|
996
|
+
(the unmount step writes it after running the unmount over SSM);
|
|
997
|
+
`apply-check` refuses when the flag is off now, an overlay is gone, or no
|
|
998
|
+
`cfg/_config.yml` is found; `verify instance <name>` (ephemerals) runs the
|
|
999
|
+
runtime's checks (over SSM on AWS: the completion marker, the mounts, the
|
|
1000
|
+
booted AMI) plus the image's `tests.post_bake`, records the verdict in
|
|
1001
|
+
`meta-state/verifications.yaml` and the post-bake result in
|
|
1002
|
+
`meta-state/image-tests.yaml`, and raises on failure unless `--record-only`.
|
|
1003
|
+
`archive-<label>.sh` exits 1 when no volume of that name exists. All of
|
|
1004
|
+
these stop the runner under `set -euo pipefail`; the run summary records
|
|
1005
|
+
the lifecycle's apply as `failed` and the run exits 1.
|
|
1006
|
+
|
|
1007
|
+
**After apply (post-finalize and the base hooks), only when the root's
|
|
1008
|
+
apply flag was on and the root is inside the run's runtime scope (stage
|
|
1009
|
+
63).** `post_finalize_phase` binds first pins and moves
|
|
1010
|
+
`follow` pins in `meta-state/pins.yaml` (a booted image lineage does not
|
|
1011
|
+
record is logged and left unbound). The base hooks then mark instances
|
|
1012
|
+
`launched`, open or close generations in `instance-state.yaml` and confirm
|
|
1013
|
+
them against the provider's instance id, retire the replaced machine's OPA
|
|
1014
|
+
registration, forget decommissioned and ephemeral records, re-record
|
|
1015
|
+
detached mounts, write the AltNames aliases, and record storage
|
|
1016
|
+
transitions in `storage-state.yaml`. Every verdict is a log line and a
|
|
1017
|
+
meta-state write; a `--commit` run commits them.
|
|
1018
|
+
|
|
1019
|
+
**In the state query.** `query_state()` on each storage builder reports
|
|
1020
|
+
every owned storage as `present` or not with the provider record;
|
|
1021
|
+
[state_query.py](../base/src/cs_image_system/base/state_query.py) turns that
|
|
1022
|
+
into drift lines in `generated/state-report.json`: `missing [HARD]` for a
|
|
1023
|
+
recorded non-destroyed storage that is absent, `stale` for a destroyed
|
|
1024
|
+
record that still exists, for a provider state other than
|
|
1025
|
+
available/in-use/active/ready, and for a declared lifecycle the resource
|
|
1026
|
+
does not carry yet, `foreign` for a present storage with no record, and an
|
|
1027
|
+
`unavailable:` line when the lookup raised (an S3 permission or transport
|
|
1028
|
+
error is raised on purpose so it is never read as absent). The instance
|
|
1029
|
+
root has no state query of its own: an instance's booted image, power
|
|
1030
|
+
state, generation and aliases are read by the runtime plugin.
|
|
1031
|
+
|
|
1032
|
+
## When it fails
|
|
1033
|
+
|
|
1034
|
+
The failures this plugin produces or takes part in, with what the operator
|
|
1035
|
+
sees, what it means, where to look and what to do. Failures that have
|
|
1036
|
+
happened live come first, dated; the sources are
|
|
1037
|
+
[OPERATIONS.md](../../docs/OPERATIONS.md), the findings ledger in
|
|
1038
|
+
[docs/history/LEDGER.md](../../docs/history/LEDGER.md), the tests named
|
|
1039
|
+
below and the commit messages on this package.
|
|
1040
|
+
|
|
1041
|
+
### Failures that have happened
|
|
1042
|
+
|
|
1043
|
+
- **2026-09-02 (findings 20 and 21): an apply-less storage run recorded
|
|
1044
|
+
`None -> active`.** The fixture carried no `apply_storage` key, so the run
|
|
1045
|
+
planned and gated with zero applies, and the transition recorder wrote
|
|
1046
|
+
the requested state anyway; the state query's `missing [HARD]` caught the
|
|
1047
|
+
lie. Since then every reality-claiming recorder (storage transitions, the
|
|
1048
|
+
`launched` marker, pin binds, the forgets) runs only when the lifecycle's
|
|
1049
|
+
apply flag was on. What to expect now: a dry or apply-less run leaves
|
|
1050
|
+
`storage-state.yaml`, `launch-params.yaml` `launched` and `pins.yaml`
|
|
1051
|
+
exactly as they were.
|
|
1052
|
+
- **Stage 1, 2026-08-31 to 2026-09-02 (finding 22): the S3 state query
|
|
1053
|
+
reported the system's own bucket as HARD-missing** because it looked for
|
|
1054
|
+
a `Name` tag the module never sets. `_lookup` now treats a reachable
|
|
1055
|
+
bucket as present; only `NoSuchBucket`/`404` is absent and any other
|
|
1056
|
+
error is re-raised into an `unavailable:` line.
|
|
1057
|
+
- **Stage 1 (finding 24): `apply_*` flags gated generation only**, so a
|
|
1058
|
+
runner script generated under a flag kept applying after the flag was
|
|
1059
|
+
turned off. Now every apply is preceded by `cs-image-system apply-check`,
|
|
1060
|
+
which re-reads `cfg/_config.yml` and the overlays at execution and exits
|
|
1061
|
+
3 with `apply_instances is false NOW in <path>/cfg/_config.yml ... not
|
|
1062
|
+
applying.`
|
|
1063
|
+
- **Stage 1 (finding 25): a `t2.micro` OOM-killed the SSM agent** minutes
|
|
1064
|
+
after boot. `machine_type` on the instance overrides the runtime default;
|
|
1065
|
+
size the machine for its image.
|
|
1066
|
+
- **Stage 1 (finding 26): the module's default public IP poisoned sftd's
|
|
1067
|
+
advertised address** in a VPC with no internet gateway, so `sft ssh`
|
|
1068
|
+
hung. The module call now sets `associate_public_ip_address = false` and
|
|
1069
|
+
the launch script pins `AccessAddress` to the private IP.
|
|
1070
|
+
- **Stage 1 (findings 27, 29, 30, 31): instances wore only the VPC default
|
|
1071
|
+
security group**, which blocked the Okta gateway's inbound SSH relay. The
|
|
1072
|
+
root now creates `csis-instances` with SSH ingress from
|
|
1073
|
+
`ssh_ingress_security_group_ids` (never a CIDR of the shared networks),
|
|
1074
|
+
`create_before_destroy` (a delete-first replacement hit
|
|
1075
|
+
`DependencyViolation` because instances wear the group), and every
|
|
1076
|
+
instance also wears `addl_security_groups` (the relay requires the
|
|
1077
|
+
OKTA-GATEWAY group on the target). Symptom without them: `sft ssh` hangs
|
|
1078
|
+
or is refused while every step reports success.
|
|
1079
|
+
- **Stage 1 (finding 28): user-data died at `mkfs`** because on Nitro
|
|
1080
|
+
instances (`t3` and later) the EBS volume never appears as `/dev/xvdf`
|
|
1081
|
+
and RHEL-family AMIs create no legacy symlink; everything after the
|
|
1082
|
+
`mkfs` line never ran and the mount was missing. The launch script waits
|
|
1083
|
+
for either the declared device or the
|
|
1084
|
+
`/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_<volume id>` name.
|
|
1085
|
+
Where to look: the instance's cloud-init/user-data log, and `verify
|
|
1086
|
+
instance` (the completion marker `/var/lib/csis/launch-applied` is
|
|
1087
|
+
written only when every line above it succeeded).
|
|
1088
|
+
- **Stage 1 (finding 33): `gate-plan` accepted a stale planfile** left by a
|
|
1089
|
+
failed plan. Every sequence now begins with `rm -f tfplan` and the gate
|
|
1090
|
+
refuses with `STALE PLANFILE: tfplan is older than <file>` (exit 3) or
|
|
1091
|
+
`tfplan does not exist`.
|
|
1092
|
+
- **2026-09-04 (finding 45): the root-level `aws_vpc` and
|
|
1093
|
+
`aws_security_group` blocks bound the DEFAULT aws provider**, which has no
|
|
1094
|
+
configuration, so they only ever planned from a shell with ambient AWS
|
|
1095
|
+
credentials. Both now carry `provider = aws.<alias>`. Symptom: a plan
|
|
1096
|
+
that fails on the VPC lookup with a credentials or region error while
|
|
1097
|
+
the module calls are fine.
|
|
1098
|
+
- **2026-09-05 (findings 52 and 53), tearing down the first GCE instance,
|
|
1099
|
+
fixed in both instance builders
|
|
1100
|
+
([test_v2_decommission.py](../../tests/test_v2_decommission.py)):**
|
|
1101
|
+
undeclaring the last instance on a runtime made its root vanish, so
|
|
1102
|
+
nothing ever planned the destroy and the VM would have stayed orphaned;
|
|
1103
|
+
and the forget of the launch record ran at generation, so a dry run
|
|
1104
|
+
erased it before any destroy and the gate then had nothing to whitelist.
|
|
1105
|
+
Now `_root_has_work()` emits an instance-less root whenever a recorded
|
|
1106
|
+
instance is undeclared, the decommission whitelist is scoped to the root
|
|
1107
|
+
of the runtime the pinned build was baked on, and the forget is an
|
|
1108
|
+
after-apply hook. A decommission can be dry-run as often as needed.
|
|
1109
|
+
- **2026-09-05/06 (finding 49): a pin was re-bound to a new series head
|
|
1110
|
+
without replacing the instance**, so meta-state claimed a build reality
|
|
1111
|
+
did not run. The post-finalize hook now binds from what actually booted
|
|
1112
|
+
when the parent was deferred, and the state query reports
|
|
1113
|
+
`booted image <x> != pinned build <y>` as `changed`.
|
|
1114
|
+
- **2026-09-09 (ledger 68,
|
|
1115
|
+
[test_v2_ephemeral.py](../../tests/test_v2_ephemeral.py)): a torn-down
|
|
1116
|
+
ephemeral kept its launch record** because a `verify assert` as the
|
|
1117
|
+
runner's last step failed the script before the after-apply hook ran.
|
|
1118
|
+
Under `on_failure: teardown` the verdict now fails the run from
|
|
1119
|
+
`forget_ephemerals`, after the record is forgotten.
|
|
1120
|
+
- **2026-09-10 (ledger 70): a run scoped to AWS planned the GCE instance
|
|
1121
|
+
root too**, whose image-family lookup returned 404 because the previous
|
|
1122
|
+
cycle had disposed every GCE image, and the run failed before the dask
|
|
1123
|
+
bake. Under `--only-runtime` (explicit or implied by `--apply-runtime`)
|
|
1124
|
+
another runtime's instance and storage roots emit nothing, and since
|
|
1125
|
+
stage 63 (2026-09-25) the out-of-scope instance builder's finalize hooks
|
|
1126
|
+
do nothing either.
|
|
1127
|
+
- **Until 2026-09-25 (stage 63): every scoped run logged `No resolved
|
|
1128
|
+
provider-specific images for instances; modules will use their
|
|
1129
|
+
name-pattern fallback`** (seen in every GCE cycle), because the AWS
|
|
1130
|
+
instance builder's `pre_finalize_phase` still ran for a root the scope
|
|
1131
|
+
had left empty and wrote an `instances.auto.tfvars` nothing read. Both
|
|
1132
|
+
finalize hooks now return early for a root outside the runtime scope.
|
|
1133
|
+
- **Until 2026-09-25 (stage 63): two S3 storages on one builder reported
|
|
1134
|
+
the same bucket.** The state query read the builder's `bucket_name`, not
|
|
1135
|
+
the storage's, so the second storage's presence and tags were the
|
|
1136
|
+
first's. `_lookup` now asks for the storage's own bucket (its
|
|
1137
|
+
`bucket_name`, else its name)
|
|
1138
|
+
([test_v2_defects_dead.py](../../tests/test_v2_defects_dead.py),
|
|
1139
|
+
`test_the_s3_lookup_asks_for_the_storages_own_bucket`).
|
|
1140
|
+
- **2026-09-19 (stage 52,
|
|
1141
|
+
[test_v2_availability_zones.py](../../tests/test_v2_availability_zones.py)):
|
|
1142
|
+
a zone was inferred from the subnet and never checked.** Pointing the
|
|
1143
|
+
runtime at a subnet in another zone planned `forces replacement, 1 to
|
|
1144
|
+
add, 1 to destroy` on a 100 GiB data volume; the gate refused it as an
|
|
1145
|
+
unwhitelisted destroy, but at apply time and naming the volume rather
|
|
1146
|
+
than the cause. Now a subnet, a storage and an instance may declare
|
|
1147
|
+
`availability_zone`, `validate` refuses an incompatible set naming each
|
|
1148
|
+
claimant, and the EBS builder passes the declared zone to the module.
|
|
1149
|
+
- **2026-09-20 (stage 19,
|
|
1150
|
+
[test_storage_module_calls.py](tests/test_storage_module_calls.py)): no
|
|
1151
|
+
instance in the account could mount an EFS storage.** The module made a
|
|
1152
|
+
filesystem and its access points and no mount targets, so `mount.efs`
|
|
1153
|
+
could not even resolve the filesystem's DNS name; `verify instance`
|
|
1154
|
+
failed on the mount. The module now creates one mount target per zone and
|
|
1155
|
+
a security group for them, and the builder passes the runtime's private
|
|
1156
|
+
subnets (one per zone) and `addl_security_groups` as the clients. If the
|
|
1157
|
+
runtime declares no `networking`, or no private subnets, or no
|
|
1158
|
+
`addl_security_groups`, the call carries none of the three and the
|
|
1159
|
+
filesystem is again unmountable: check the emitted
|
|
1160
|
+
`<B>-storage-generation-storage-<label>.tf` for `mount_target_subnet_ids`.
|
|
1161
|
+
- **2026-09-20: `coops-model` was destroyed through the gate but its pin
|
|
1162
|
+
and launch parameters stayed**, which with `require_released_builds` on
|
|
1163
|
+
made `validate` refuse every run. The mechanism is proven in the fixture,
|
|
1164
|
+
so the live failure remains unexplained; the recourse is
|
|
1165
|
+
`cs-image-system forget instance <name>`, which drops the records without
|
|
1166
|
+
editing meta-state by hand.
|
|
1167
|
+
- **2026-09-22, the first live replacement (`coops-model` ->
|
|
1168
|
+
`coops-model-002`), four traps:** the gate refused
|
|
1169
|
+
`module.instance_coops_model.aws_volume_attachment.this["mnt_data"]` as
|
|
1170
|
+
an unwhitelisted destroy (the attachment binds the volume to the
|
|
1171
|
+
instance id, so the `-replace` replaces it too) -- now
|
|
1172
|
+
`_replacement_attachments()` whitelists it; the strict state query
|
|
1173
|
+
reported the booted image behind the moved pin as drift and refused the
|
|
1174
|
+
very run that finishes the upgrade -- now a pending replacement is a
|
|
1175
|
+
`note`; the replaced machine's OPA registration outlived it and would
|
|
1176
|
+
have claimed the new machine's alias -- now `mark_launched` retires it;
|
|
1177
|
+
and the alias pass ran before the new machine had enrolled and wrote
|
|
1178
|
+
nothing -- now it waits. Proof in
|
|
1179
|
+
[test_v2_gate5_capabilities_activation.py](../../tests/test_v2_gate5_capabilities_activation.py)
|
|
1180
|
+
and [test_v2_provider_aliases.py](../../tests/test_v2_provider_aliases.py);
|
|
1181
|
+
the procedure is OPERATIONS "A model image, end to end".
|
|
1182
|
+
- **2026-09-20 and 2026-09-22: `require_released_builds` had to be switched
|
|
1183
|
+
off by hand** for the window between `upgrade instance` and the release,
|
|
1184
|
+
because a durable instance whose volume allows one attachment can only
|
|
1185
|
+
prove a build on itself. Since 2026-09-23 (stage 61 item 4) the release
|
|
1186
|
+
grace admits the series head while its own proof is under way; the
|
|
1187
|
+
refusal now reads `instance '<name>' is pinned to build <id> of '<image>',
|
|
1188
|
+
which is not a released build (config.require_released_builds; no grace:
|
|
1189
|
+
<what is missing>)`. Proved live the same day as the third release
|
|
1190
|
+
(`coops-model-003`, the pool's first draw).
|
|
1191
|
+
- **Until 2026-09-23 (stage 61 item 5): the `release` and `retention`
|
|
1192
|
+
lifecycles wrote `instances.auto.tfvars` under their own generation
|
|
1193
|
+
directories**, where no instance root exists; the file lingered untracked
|
|
1194
|
+
after every release run. `pre_finalize_phase` now acts for the
|
|
1195
|
+
`instance-image` lifecycle alone
|
|
1196
|
+
([test_v2_hygiene_five.py](../../tests/test_v2_hygiene_five.py)).
|
|
1197
|
+
Since stage 63 both finalize hooks also return early for a root outside
|
|
1198
|
+
the run's runtime scope
|
|
1199
|
+
([test_v2_defects_dead.py](../../tests/test_v2_defects_dead.py),
|
|
1200
|
+
`test_an_out_of_scope_instance_root_writes_no_tfvars`).
|
|
1201
|
+
- **Found live, undated: the S3 prerequisite bake failed on RHEL-family
|
|
1202
|
+
images** because `unzip` is not installed there (`curl` is). The
|
|
1203
|
+
prerequisite now installs `unzip` before fetching the AWS CLI zip.
|
|
1204
|
+
|
|
1205
|
+
### Failures the code raises that have not happened
|
|
1206
|
+
|
|
1207
|
+
Each with the message or symptom, what it means, where to look, what to do.
|
|
1208
|
+
|
|
1209
|
+
- `NotImplementedError: TofuStorageBuilder must name its terraform module`
|
|
1210
|
+
at generation: a builder entry declares `type: tf-aws`. Use
|
|
1211
|
+
`tf-aws-ebs`, `tf-aws-efs` or `tf-aws-s3`.
|
|
1212
|
+
- `ValueError: Storage builder <B>: group '<g>' has no identity builder` at
|
|
1213
|
+
generation: a storage's allowed group is not owned by any group builder,
|
|
1214
|
+
so no `group_gids` output can be referenced. Declare the group under a
|
|
1215
|
+
group builder, or remove it from `groups:` (the validator's `storage
|
|
1216
|
+
'<name>' allows unknown group '<g>'` usually catches it first).
|
|
1217
|
+
- Log warning `No provider-specific image for image '<img>' of instance
|
|
1218
|
+
<name>; the module call will not resolve an AMI`: the image is not baked
|
|
1219
|
+
on the instance's runtime and lineage has no head. The emitted call has
|
|
1220
|
+
no `ami_id`; `tofu validate` passes but the plan fails on the module's
|
|
1221
|
+
`data "aws_ami"` with an empty pattern. Bake the image on that runtime
|
|
1222
|
+
first.
|
|
1223
|
+
- Log info `No resolved provider-specific images for instances; modules
|
|
1224
|
+
will use their name-pattern fallback` at pre-finalize: nothing baked this
|
|
1225
|
+
run, so no `instances.auto.tfvars`; the module resolves `ami_id` from the
|
|
1226
|
+
pin or the most recent AMI matching the pattern. Expected on a launch
|
|
1227
|
+
that re-bakes nothing. Since stage 63 it is never logged for a root
|
|
1228
|
+
outside the run's `--only-runtime` scope (the hook returns before it).
|
|
1229
|
+
- Log warning `Instance <name> booted <ami>, which lineage does not record;
|
|
1230
|
+
pin left unbound`: the parent was deferred and the machine booted an
|
|
1231
|
+
image outside lineage (a foreign AMI matched the name pattern). Run
|
|
1232
|
+
`state import` or dispose the foreign image, then `upgrade instance` onto
|
|
1233
|
+
a recorded build.
|
|
1234
|
+
- `DESTROY NOT WHITELISTED: <address>` from `gate-plan` (exit 3): the plan
|
|
1235
|
+
destroys something no operation sanctioned. Common causes on these
|
|
1236
|
+
roots: `volume_type`, `size` or `availability_zone` changed on an EBS
|
|
1237
|
+
storage (all force replacement; declare what exists, or archive and
|
|
1238
|
+
restore), a subnet in another zone, a renamed storage (a new module
|
|
1239
|
+
address is a create plus a destroy; retire the old name through
|
|
1240
|
+
`state: destroyed` or by removing the entry), an instance's changed
|
|
1241
|
+
image without an upgrade. Read `tofu show tfplan` in the root; nothing
|
|
1242
|
+
applied.
|
|
1243
|
+
- `DETACH NOT UNMOUNTED: <instance>:<storage> has no successful unmount
|
|
1244
|
+
receipt` (exit 3): the unmount step did not write
|
|
1245
|
+
`unmount-receipts/<instance>__<storage>.json` with `ok: true`. Look at the
|
|
1246
|
+
runner's `unmount storage` output; when the machine is gone or the mount
|
|
1247
|
+
point was never mounted, `unmount storage --confirm ...` records the
|
|
1248
|
+
operator's word instead.
|
|
1249
|
+
- `unmount of <mount point> on <instance> failed (exit <n>): ...`: the
|
|
1250
|
+
session command could not unmount (a process holds the mount, or SSM
|
|
1251
|
+
cannot reach the machine). The runner stops before the plan.
|
|
1252
|
+
- `STALE PLANFILE: tfplan does not exist` or `... is older than <file>`
|
|
1253
|
+
(exit 3): the plan step failed or the root changed after it. Re-run the
|
|
1254
|
+
lifecycle; never hand the gate an old file.
|
|
1255
|
+
- `apply_<lifecycle> is false NOW in <path>/cfg/_config.yml [for root
|
|
1256
|
+
'<B>'] ... not applying.` (exit 3), `apply-check: overlay <path> is gone;
|
|
1257
|
+
refusing to apply`, `apply-check: no cfg/_config.yml found from the
|
|
1258
|
+
working directory up`: the execution-time guard. Turn the flag on (or
|
|
1259
|
+
list the root/runtime), restore the overlay, or run from a checkout that
|
|
1260
|
+
holds the tree.
|
|
1261
|
+
- `instance <name> failed verification: <check>: <detail>` from `verify
|
|
1262
|
+
instance`: an ephemeral's launch did not complete (the marker
|
|
1263
|
+
`/var/lib/csis/launch-applied` is missing: a launch-script line failed,
|
|
1264
|
+
including a declared `userdata` line), a mount is absent, the booted AMI
|
|
1265
|
+
is not the expected build, or a post-bake assertion failed. Under `keep`
|
|
1266
|
+
the machine stands for inspection and the state query reports `ephemeral
|
|
1267
|
+
instance is STANDING ...` until it is decommissioned (undeclare it and
|
|
1268
|
+
run with the root allowed to apply); under `teardown` the run still fails
|
|
1269
|
+
after the teardown. `verify <name>: SKIPPED -- the machine is stopped ...`
|
|
1270
|
+
records nothing.
|
|
1271
|
+
- `Ephemeral instance <name>: teardown_after elapsed since its failed
|
|
1272
|
+
verification; tearing down without re-verifying` (log): expected on the
|
|
1273
|
+
run after the duration.
|
|
1274
|
+
- `instance '<name>' was launched (run <r>) and its launch parameters
|
|
1275
|
+
changed (<keys>); launch parameters are immutable after launch -- replace
|
|
1276
|
+
the instance (upgrade instance, or decommission and redeclare) (N26)`:
|
|
1277
|
+
a mount was added, a mount point or `userdata` changed, the image or
|
|
1278
|
+
group changed. Removing a mount is the one in-place change; anything else
|
|
1279
|
+
is a replacement.
|
|
1280
|
+
- `instance '<name>': canonical hostname '<h>' is already registered to N
|
|
1281
|
+
server(s) in group '<g>' ...` / `... could not check whether canonical
|
|
1282
|
+
hostname ... (group '<g>'s server registry did not answer); refusing to
|
|
1283
|
+
launch on silence`: only on an applies-on instance-image run. Retire the
|
|
1284
|
+
stale registration (a decommission does it; or `DELETE` the server under
|
|
1285
|
+
the resource group's project), or restore the OPA credentials.
|
|
1286
|
+
- `instance '<name>' is pinned to build <id> of '<image>', which is not a
|
|
1287
|
+
released build (config.require_released_builds; no grace: ...)`: release
|
|
1288
|
+
the build, `upgrade instance` onto a released one, or read what the grace
|
|
1289
|
+
needs (the build must be the series head, verified in bake, with no failed
|
|
1290
|
+
post-bake record, and the instance a pending replacement onto it or
|
|
1291
|
+
standing on it).
|
|
1292
|
+
- `storage '<name>' (<builder>) is single-attach but instances [...] all
|
|
1293
|
+
attach it (N16); multi-host storage builders on <rt>: [...]`: an EBS
|
|
1294
|
+
volume attaches to one instance; use EFS or S3 for a shared storage.
|
|
1295
|
+
- `storage '<name>' cannot move <cur> -> <req> while attached by [...]
|
|
1296
|
+
(detach in one run, transition in the next; N21)`, `storage '<name>'
|
|
1297
|
+
(<builder>) requests archived, which its builder does not realize`,
|
|
1298
|
+
`storage '<name>' has never been applied; a new storage must start
|
|
1299
|
+
'active'`, `storage '<name>': transition <a> -> <b> is not legal`,
|
|
1300
|
+
`instances [...] attach storage '<name>' which is <state>`: the state
|
|
1301
|
+
machine at validate. Two runs for detach-then-archive.
|
|
1302
|
+
- `storage '<name>': lifecycle.<key> must be one of [...]`, `... unknown
|
|
1303
|
+
lifecycle keys [...] (efs: ia_days, archive_days)`, `... lifecycle
|
|
1304
|
+
declares neither transition_days nor expire_days`, `storage '<name>'
|
|
1305
|
+
(<builder>) declares a data lifecycle, which its builder does not realize
|
|
1306
|
+
(no lifecycle on ebs storages)`: fix the declaration per the table
|
|
1307
|
+
above.
|
|
1308
|
+
- `instance '<name>' (image '<img>', group '<g>') may not attach storage
|
|
1309
|
+
'<s>': allowed groups are [...] (N2)` and `instance '<name>' (image
|
|
1310
|
+
'<img>') attaches storage '<s>' of type '<t>' which base '<b>' does not
|
|
1311
|
+
declare`: add the group to the storage's `groups:` (or `public_read`),
|
|
1312
|
+
or declare the type on the base image and bake forward.
|
|
1313
|
+
- `no volume named <name> to archive` (exit 1 from `archive-<label>.sh`):
|
|
1314
|
+
the volume the `Name` tag should find is gone. Check the state query; if
|
|
1315
|
+
the archive snapshot already exists the script reuses it and exits 0.
|
|
1316
|
+
- `aws: command not found` or `bash: aws: No such file or directory` in a
|
|
1317
|
+
storage runner: the AWS CLI is not on the runner's `PATH`; install it
|
|
1318
|
+
where the script runs.
|
|
1319
|
+
- `Reference to undeclared resource` from `tofu validate` at generation:
|
|
1320
|
+
`use_state_backends` is off while the emission references remote state
|
|
1321
|
+
(any instance with a group or mount, any storage with groups). Turn the
|
|
1322
|
+
flag on and declare a backend.
|
|
1323
|
+
- `Backend configuration changed` from `tofu init`: only when a root is
|
|
1324
|
+
initialised by hand without `-reconfigure`; the generated commands always
|
|
1325
|
+
carry it. A root whose state must move is `--migrate-state <root>`, never
|
|
1326
|
+
a hand `init`.
|
|
1327
|
+
- `Instance <name>: the alias pool is EMPTY; launching <hostname> without
|
|
1328
|
+
an alias (append names to aliases.txt to refill it)` (warning): the
|
|
1329
|
+
launch proceeds; append names.
|
|
1330
|
+
- `Instance <name>: its OPA registration as '<h>' was NOT retired (<e>); a
|
|
1331
|
+
stale server will answer to that name until it is deregistered by hand`
|
|
1332
|
+
(error, never fatal to the forget): deregister by hand under the
|
|
1333
|
+
resource group's project.
|
|
1334
|
+
- State query lines `storages/<B>: ... unavailable` (a lookup raised: a
|
|
1335
|
+
permission error, an expired session, no network) and `storage <name>
|
|
1336
|
+
recorded <state> but not found in reality [HARD]` (the volume, filesystem
|
|
1337
|
+
or bucket is gone out of band): a HARD line refuses every run until the
|
|
1338
|
+
record is corrected through `state import`/the forget paths, never by
|
|
1339
|
+
hand-editing meta-state.
|
|
1340
|
+
- `Instance builder <B> does not specify a runtime provider.` /
|
|
1341
|
+
`... has runtime provider <rt> which is not configured`: the builder's
|
|
1342
|
+
`runtime:` resolves to nothing; name a declared runtime.
|
|
1343
|
+
|
|
1344
|
+
## Related
|
|
1345
|
+
|
|
1346
|
+
- [cs-image-system-tf-gcp-plugin](../tf-gcp-plugin/README.md): the GCE
|
|
1347
|
+
instance root and the GCP storage builders, subclasses of this package.
|
|
1348
|
+
- [cs-image-system-tf-s3-state-plugin](../tf-s3-state-plugin/README.md): the
|
|
1349
|
+
S3 state backend the `state_configuration` field names.
|
|
1350
|
+
- [cs-image-system-aws-runtime-plugin](../aws-runtime-plugin/README.md): the
|
|
1351
|
+
runtime whose region, credentials, networking and session mechanism these
|
|
1352
|
+
roots read, and whose hooks verify instances and read their booted image.
|
|
1353
|
+
- [hashicorp-utils collector.py](../hashicorp-utils/src/cs_image_system/hashicorp_utils/collector.py):
|
|
1354
|
+
the run-scoped `TerraformCollector` that renders the terraform, provider,
|
|
1355
|
+
variable, backend and remote-state blocks.
|
|
1356
|
+
- [hashicorp-utils roots.py](../hashicorp-utils/src/cs_image_system/hashicorp_utils/roots.py):
|
|
1357
|
+
`TerraformRootMixin`, the init arguments and the gated apply sequence.
|
|
1358
|
+
- [base launch_params.py](../base/src/cs_image_system/base/launch_params.py),
|
|
1359
|
+
[base storage_state.py](../base/src/cs_image_system/base/storage_state.py),
|
|
1360
|
+
[base lineage.py](../base/src/cs_image_system/base/lineage.py),
|
|
1361
|
+
[base alias_pool.py](../base/src/cs_image_system/base/alias_pool.py),
|
|
1362
|
+
[base generations.py](../base/src/cs_image_system/base/generations.py),
|
|
1363
|
+
[base release.py](../base/src/cs_image_system/base/release.py): launch
|
|
1364
|
+
parameters, the storage state machine, pins and image policies, the alias
|
|
1365
|
+
pool, instance generations, the release grace.
|
|
1366
|
+
- [base commands/gate.py](../base/src/cs_image_system/base/commands/gate.py),
|
|
1367
|
+
[base commands/unmount.py](../base/src/cs_image_system/base/commands/unmount.py),
|
|
1368
|
+
[base commands/verify_instance.py](../base/src/cs_image_system/base/commands/verify_instance.py):
|
|
1369
|
+
the gate, the unmount receipt, the verification.
|
|
1370
|
+
- [tests/test_instance_module_calls.py](tests/test_instance_module_calls.py)
|
|
1371
|
+
and [tests/test_storage_module_calls.py](tests/test_storage_module_calls.py):
|
|
1372
|
+
the unit tests that pin the module arguments described here.
|
|
1373
|
+
- [docs/DESIGN.md](../../docs/DESIGN.md), [docs/OPERATIONS.md](../../docs/OPERATIONS.md)
|
|
1374
|
+
and [docs/CONFIGURATION.md](../../docs/CONFIGURATION.md): the system
|
|
1375
|
+
design, the operator's view of the lifecycles, and the configuration
|
|
1376
|
+
manual.
|