cs-image-system-gcloud-runtime-plugin 0.1.4__tar.gz → 0.1.6__tar.gz

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/.gitignore

@@ -13,6 +13,8 @@ build/
 .venv/
 .DS_Store
 packages/**/*.code-workspace
+# the IDE rewrites the root workspace file whenever folders are opened: it is the editor's, not the repository's (hygiene XXI)
+/*.code-workspace
 trial
 tfmodules/tftest
 YAML_DUMP.yaml

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cs-image-system-gcloud-runtime-plugin
-Version: 0.1.4
+Version: 0.1.6
 Summary: The gcloud runtime type: Google Compute Engine as a place where images bake and instances run
 Project-URL: Repository, https://github.com/infrastructurebuilder/cs-image-system-3
 Project-URL: Documentation, https://github.com/infrastructurebuilder/cs-image-system-3/blob/develop/docs/OPERATIONS.md
@@ -9,9 +9,9 @@ Author-email: Mykel Alvis <mykelalvis@infrastructurebuilder.org>
 License-Expression: Apache-2.0
 License-File: LICENSE
 Requires-Python: >=3.13
-Requires-Dist: cs-image-system-base==0.1.4
-Requires-Dist: cs-image-system-contract==0.1.4
-Requires-Dist: cs-image-system-hashicorp-utils==0.1.4
+Requires-Dist: cs-image-system-base==0.1.6
+Requires-Dist: cs-image-system-contract==0.1.6
+Requires-Dist: cs-image-system-hashicorp-utils==0.1.6
 Requires-Dist: google-api-core>=2.31
 Requires-Dist: google-auth>=2.55
 Requires-Dist: google-cloud-compute>=1.49
@@ -86,7 +86,12 @@ cs_image_system.gcloud_runtime.bootstrap:gcp_section`. It owns the
 questions and the terraform for this cloud's one-time initialisation --
 the workload identity pool and GitHub provider of CI_SETUP.md 3.4, the
 READ-ONLY service account and the optional WRITE one, their project
-roles and `workloadIdentityUser` bindings (`tfmodules/bootstrap_gcp`).
+roles and `workloadIdentityUser` bindings, and since hygiene XX the GCS
+state bucket when the tree keeps its state in GCS and the project lacks
+it (versioned, uniform access, no public access, the CI identities'
+object access; managed from then on, with the AWS section's fork)
+(`tfmodules/bootstrap_gcp`). The section renders its coverage audit for
+the generated README: what it made, read, or leaves by hand and why.
 The section is wanted by default where the tree declares a GCE runtime
 or a `gcs` state backend. Its defaults come from the raw
 `cfg/runtime-builders.yml` and `cfg/state-backends.yml` (a starter's
@@ -291,46 +296,83 @@ firewall rule targets logs a warning.
 
 `query_provider_image()` takes one OS-builder runtime entry
 (`OSBuilderBaseImageBuilderSubconfig` in
-[os_builder_runtime_config.py](../base/src/cs_image_system/base/models/os_builder_runtime_config.py)).
-`remap_for_image_query()` builds a Compute `list()` query:
-
-- `projects` is the entry's `get_owners()` (the OS builder's owners, the runtime's `default_owners`, the entry's own), resolved by `resolve_projects()`: `self` becomes the session project; the aliases `debian`, `ubuntu`, `rhel`, `centos`, `rocky`, `fedora`, `suse`, `windows`, `cos` and `google` become the public image projects (`debian-cloud`, `ubuntu-os-cloud`, `rhel-cloud`, ...); AWS-only aliases (`amazon`, `aws-marketplace`, `aws-backup-vault`) and anything that is not a valid project id (an AWS account number, for example) are dropped; a valid project id such as `almalinux-cloud` is kept as is.
-- `query.filters` become a filter expression: `name`, `family`, `state`/`status`, `architecture`, `description` and `creation_date` map to image fields; `state: available` becomes `status = "READY"` (forced when absent); architectures map to `X86_64`/`ARM64`; `tag:<key>` becomes `labels.<key>`; AWS-only keys (`root_device_type`, `virtualization_type`, `block_device_mapping.*`, ...) are dropped instead of excluding every result; keys the map does not know go to a post-query exact-match filter.
-- A `family` key with no filter expression resolves through `getFromFamily` instead.
-
-`query_image()` lists every candidate project, applies the post-query
-filter, and returns the newest `creation_timestamp` with the owning
-`project` added. The builder returns `(name, project, raw)`; the base
-registers a `GcpProviderSpecificImage` in the resolved state.
+[os_builder_runtime_config.py](../base/src/cs_image_system/base/models/os_builder_runtime_config.py))
+and asks it for its effective query, `entry.image_query()`: an
+`ImageQuery` in the SYSTEM's vocabulary
+([contract/image.py](../contract/src/cs_image_system/contract/image.py);
+stage 106). That query is the image-source catalog's `gcloud` block for
+the OS builder's family (the release's layer under the tree's
+`cfg/image-sources*.yml`), with the OS builder's own `query` landing on it
+key by key and the runtime entry's landing on that; the publishers
+declared anywhere (the OS builder's `owners`, this runtime's
+`default_owners`, the entry's `owners`) replace the catalog's. The builder
+translates it with `translate_image_query()` (the table below) into
+`projects` and a Compute `list()` filter expression; `query_image()` in
+[gcp_utils.py](src/cs_image_system/gcloud_runtime/gcp_utils.py) resolves
+the projects (`resolve_projects()`: `self` becomes the session project;
+the aliases `debian`, `ubuntu`, `rhel`, `centos`, `rocky`, `fedora`,
+`suse`, `windows`, `cos` and `google` become the public image projects;
+AWS-only aliases and anything that is not a valid project id are dropped;
+a valid project id such as `almalinux-cloud` is kept as is), lists every
+candidate project, and returns the newest `creation_timestamp` with the
+owning `project` added; with `newest: false` more than one match raises
+`ImageQueryError` as ambiguous. The builder returns `(name, project,
+raw)`; the base registers a `GcpProviderSpecificImage` in the resolved
+state. `validate` has already refused an entry whose query uses a key this
+runtime cannot translate (the EC2 particulars, which this runtime used to
+drop with a debug line) or has no source at all, and has printed every
+entry's query in both vocabularies.
+
+### Image query translation
+
+What `translate_image_query()` does with each key of the system's
+vocabulary (`supported_query_keys()` is the set of rows not marked
+refused; [tests/test_docs_image_sources.py](../../tests/test_docs_image_sources.py)
+holds this table to it). The answer is `{"projects": [...], "terms":
+{<image field>: value}, "filter": "<the list() expression>",
+"most_recent": bool}`.
+
+| System key | Compute Engine field | Note |
+| --- | --- | --- |
+| `name` | `name` | a glob over the image name |
+| `family` | `family` | the vendor's image family; the latest READY image of it is the match |
+| `version` | refused | the catalog fills `{version}` into `name` or `family` |
+| `architecture` | `architecture` | `x86_64` becomes `X86_64`, `arm64` becomes `ARM64` |
+| `publisher` | the projects | project ids and the aliases above |
+| `state` | `status` | `available` becomes `READY`, forced when unset |
+| `virtualization` | refused | an EC2 particular |
+| `root_device` | refused | an EC2 particular |
+| `tags` | `labels.<key>` | one term per tag |
+| `newest` | `most_recent` | true when unset; false refuses more than one match |
+| `native` | verbatim | each key an image field, passed as written into the expression |
 
 When the entry declares `image_id` (stage 63), the base's resolve step
 calls `query_provider_image_by_id(entry, image_id)` instead. A GCE image
 is named uniquely within its project, so the id is an image name and is
-searched where the vendor query would search: `remap_for_image_query()`
-builds the same `projects` list (the entry's merged owners, or a
-`query.owners`), then the builder drops the query's `family` and `filter`
-and sets the filter to `name = "<id>"`; the post-query filter is not
-applied, and a match whose status is not `READY` is refused by name
-(stage 67: `GCP runtime <rt>: pinned image '<id>' is <status>, not READY; a
-bake from it would fail`). The owning project
-is read from the result as for the query (else `self`). No match returns
-`None`, and the base stops resolution with `OS builder <name>: image_id
-'<id>' on runtime <rt> is not known to the provider`. Pinning makes the
-base bake reproducible: the same vendor image every time, where the query
-takes the newest match (a vendor family publishes a new image often).
+searched where the vendor query would search: the effective query's
+publishers as the projects, and the filter `name = "<id>"`; a match whose
+status is not `READY` is refused by name (stage 67: `GCP runtime <rt>:
+pinned image '<id>' is <status>, not READY; a bake from it would fail`).
+The owning project is read from the result as for the query (else
+`self`). No match returns `None`, and the base stops resolution with `OS
+builder <name>: image_id '<id>' on runtime <rt> is not known to the
+provider`. Pinning makes the base bake reproducible: the same vendor image
+every time, where the query takes the newest match (a vendor family
+publishes a new image often).
 
 `GcpProviderSpecificImage.get_query_assets()` describes the image as a
 generic query (`filters = { name = "<name>" }` plus `owners` when resolved;
-`owners = ["self"]`, `filters = { name = "<pattern>*" }`, `most_recent`
-when deferred). The packer source does not use these assets: GCE's image
+when deferred, an `ImageQuery` for `<pattern>*` from `self`, translated
+and quoted: `owners = ["self"]`, `filters = { name = "<pattern>*" }`,
+`most_recent`). The packer source does not use these assets: GCE's image
 families are the series, so `googlecompute_source()` writes `source_image`
 (a resolved or pinned parent) or `source_image_family` (a deferred parent)
 directly.
 
 [tests/test_gcp_utils.py](tests/test_gcp_utils.py) and
 [tests/test_gcp_provider_specific_image.py](tests/test_gcp_provider_specific_image.py)
-pin the remap, the project resolution, the newest-wins rule and the asset
-shapes without network access.
+pin the translation, the project resolution, the newest-wins rule and the
+asset shapes without network access.
 
 ### Image families
 

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/README.md

@@ -65,7 +65,12 @@ cs_image_system.gcloud_runtime.bootstrap:gcp_section`. It owns the
 questions and the terraform for this cloud's one-time initialisation --
 the workload identity pool and GitHub provider of CI_SETUP.md 3.4, the
 READ-ONLY service account and the optional WRITE one, their project
-roles and `workloadIdentityUser` bindings (`tfmodules/bootstrap_gcp`).
+roles and `workloadIdentityUser` bindings, and since hygiene XX the GCS
+state bucket when the tree keeps its state in GCS and the project lacks
+it (versioned, uniform access, no public access, the CI identities'
+object access; managed from then on, with the AWS section's fork)
+(`tfmodules/bootstrap_gcp`). The section renders its coverage audit for
+the generated README: what it made, read, or leaves by hand and why.
 The section is wanted by default where the tree declares a GCE runtime
 or a `gcs` state backend. Its defaults come from the raw
 `cfg/runtime-builders.yml` and `cfg/state-backends.yml` (a starter's
@@ -270,46 +275,83 @@ firewall rule targets logs a warning.
 
 `query_provider_image()` takes one OS-builder runtime entry
 (`OSBuilderBaseImageBuilderSubconfig` in
-[os_builder_runtime_config.py](../base/src/cs_image_system/base/models/os_builder_runtime_config.py)).
-`remap_for_image_query()` builds a Compute `list()` query:
-
-- `projects` is the entry's `get_owners()` (the OS builder's owners, the runtime's `default_owners`, the entry's own), resolved by `resolve_projects()`: `self` becomes the session project; the aliases `debian`, `ubuntu`, `rhel`, `centos`, `rocky`, `fedora`, `suse`, `windows`, `cos` and `google` become the public image projects (`debian-cloud`, `ubuntu-os-cloud`, `rhel-cloud`, ...); AWS-only aliases (`amazon`, `aws-marketplace`, `aws-backup-vault`) and anything that is not a valid project id (an AWS account number, for example) are dropped; a valid project id such as `almalinux-cloud` is kept as is.
-- `query.filters` become a filter expression: `name`, `family`, `state`/`status`, `architecture`, `description` and `creation_date` map to image fields; `state: available` becomes `status = "READY"` (forced when absent); architectures map to `X86_64`/`ARM64`; `tag:<key>` becomes `labels.<key>`; AWS-only keys (`root_device_type`, `virtualization_type`, `block_device_mapping.*`, ...) are dropped instead of excluding every result; keys the map does not know go to a post-query exact-match filter.
-- A `family` key with no filter expression resolves through `getFromFamily` instead.
-
-`query_image()` lists every candidate project, applies the post-query
-filter, and returns the newest `creation_timestamp` with the owning
-`project` added. The builder returns `(name, project, raw)`; the base
-registers a `GcpProviderSpecificImage` in the resolved state.
+[os_builder_runtime_config.py](../base/src/cs_image_system/base/models/os_builder_runtime_config.py))
+and asks it for its effective query, `entry.image_query()`: an
+`ImageQuery` in the SYSTEM's vocabulary
+([contract/image.py](../contract/src/cs_image_system/contract/image.py);
+stage 106). That query is the image-source catalog's `gcloud` block for
+the OS builder's family (the release's layer under the tree's
+`cfg/image-sources*.yml`), with the OS builder's own `query` landing on it
+key by key and the runtime entry's landing on that; the publishers
+declared anywhere (the OS builder's `owners`, this runtime's
+`default_owners`, the entry's `owners`) replace the catalog's. The builder
+translates it with `translate_image_query()` (the table below) into
+`projects` and a Compute `list()` filter expression; `query_image()` in
+[gcp_utils.py](src/cs_image_system/gcloud_runtime/gcp_utils.py) resolves
+the projects (`resolve_projects()`: `self` becomes the session project;
+the aliases `debian`, `ubuntu`, `rhel`, `centos`, `rocky`, `fedora`,
+`suse`, `windows`, `cos` and `google` become the public image projects;
+AWS-only aliases and anything that is not a valid project id are dropped;
+a valid project id such as `almalinux-cloud` is kept as is), lists every
+candidate project, and returns the newest `creation_timestamp` with the
+owning `project` added; with `newest: false` more than one match raises
+`ImageQueryError` as ambiguous. The builder returns `(name, project,
+raw)`; the base registers a `GcpProviderSpecificImage` in the resolved
+state. `validate` has already refused an entry whose query uses a key this
+runtime cannot translate (the EC2 particulars, which this runtime used to
+drop with a debug line) or has no source at all, and has printed every
+entry's query in both vocabularies.
+
+### Image query translation
+
+What `translate_image_query()` does with each key of the system's
+vocabulary (`supported_query_keys()` is the set of rows not marked
+refused; [tests/test_docs_image_sources.py](../../tests/test_docs_image_sources.py)
+holds this table to it). The answer is `{"projects": [...], "terms":
+{<image field>: value}, "filter": "<the list() expression>",
+"most_recent": bool}`.
+
+| System key | Compute Engine field | Note |
+| --- | --- | --- |
+| `name` | `name` | a glob over the image name |
+| `family` | `family` | the vendor's image family; the latest READY image of it is the match |
+| `version` | refused | the catalog fills `{version}` into `name` or `family` |
+| `architecture` | `architecture` | `x86_64` becomes `X86_64`, `arm64` becomes `ARM64` |
+| `publisher` | the projects | project ids and the aliases above |
+| `state` | `status` | `available` becomes `READY`, forced when unset |
+| `virtualization` | refused | an EC2 particular |
+| `root_device` | refused | an EC2 particular |
+| `tags` | `labels.<key>` | one term per tag |
+| `newest` | `most_recent` | true when unset; false refuses more than one match |
+| `native` | verbatim | each key an image field, passed as written into the expression |
 
 When the entry declares `image_id` (stage 63), the base's resolve step
 calls `query_provider_image_by_id(entry, image_id)` instead. A GCE image
 is named uniquely within its project, so the id is an image name and is
-searched where the vendor query would search: `remap_for_image_query()`
-builds the same `projects` list (the entry's merged owners, or a
-`query.owners`), then the builder drops the query's `family` and `filter`
-and sets the filter to `name = "<id>"`; the post-query filter is not
-applied, and a match whose status is not `READY` is refused by name
-(stage 67: `GCP runtime <rt>: pinned image '<id>' is <status>, not READY; a
-bake from it would fail`). The owning project
-is read from the result as for the query (else `self`). No match returns
-`None`, and the base stops resolution with `OS builder <name>: image_id
-'<id>' on runtime <rt> is not known to the provider`. Pinning makes the
-base bake reproducible: the same vendor image every time, where the query
-takes the newest match (a vendor family publishes a new image often).
+searched where the vendor query would search: the effective query's
+publishers as the projects, and the filter `name = "<id>"`; a match whose
+status is not `READY` is refused by name (stage 67: `GCP runtime <rt>:
+pinned image '<id>' is <status>, not READY; a bake from it would fail`).
+The owning project is read from the result as for the query (else
+`self`). No match returns `None`, and the base stops resolution with `OS
+builder <name>: image_id '<id>' on runtime <rt> is not known to the
+provider`. Pinning makes the base bake reproducible: the same vendor image
+every time, where the query takes the newest match (a vendor family
+publishes a new image often).
 
 `GcpProviderSpecificImage.get_query_assets()` describes the image as a
 generic query (`filters = { name = "<name>" }` plus `owners` when resolved;
-`owners = ["self"]`, `filters = { name = "<pattern>*" }`, `most_recent`
-when deferred). The packer source does not use these assets: GCE's image
+when deferred, an `ImageQuery` for `<pattern>*` from `self`, translated
+and quoted: `owners = ["self"]`, `filters = { name = "<pattern>*" }`,
+`most_recent`). The packer source does not use these assets: GCE's image
 families are the series, so `googlecompute_source()` writes `source_image`
 (a resolved or pinned parent) or `source_image_family` (a deferred parent)
 directly.
 
 [tests/test_gcp_utils.py](tests/test_gcp_utils.py) and
 [tests/test_gcp_provider_specific_image.py](tests/test_gcp_provider_specific_image.py)
-pin the remap, the project resolution, the newest-wins rule and the asset
-shapes without network access.
+pin the translation, the project resolution, the newest-wins rule and the
+asset shapes without network access.
 
 ### Image families
 

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/pyproject.toml

@@ -4,7 +4,7 @@
 
 [project]
 name = "cs-image-system-gcloud-runtime-plugin"
-version = "0.1.4"
+version = "0.1.6"
 description = "The gcloud runtime type: Google Compute Engine as a place where images bake and instances run"
 readme = "README.md"
 authors = [{ name = "Mykel Alvis", email = "mykelalvis@infrastructurebuilder.org" }]
@@ -12,9 +12,9 @@ license = "Apache-2.0"
 license-files = ["LICENSE"]
 requires-python = ">=3.13"
 dependencies = [
-    "cs-image-system-contract==0.1.4",
-    "cs-image-system-base==0.1.4",
-    "cs-image-system-hashicorp-utils==0.1.4",
+    "cs-image-system-contract==0.1.6",
+    "cs-image-system-base==0.1.6",
+    "cs-image-system-hashicorp-utils==0.1.6",
     "google-cloud-compute>=1.49",
     "google-api-core>=2.31",
     "google-auth>=2.55",

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/src/cs_image_system/gcloud_runtime/bootstrap.py

@@ -32,7 +32,7 @@ import json
 from typing import Any
 
 from cs_image_system.base.bootstrap.facts import Facts, default_state_backend, first_runtime_name, real, runtime_of_type
-from cs_image_system.base.bootstrap.questions import Answers, Question, Raw, Rendered, Secret, Section
+from cs_image_system.base.bootstrap.questions import BY_HAND, MADE, READ, Answers, Question, Raw, Rendered, Secret, Section
 
 MODULE = "bootstrap_gcp"
 ISSUER = "https://token.actions.githubusercontent.com"
@@ -143,6 +143,13 @@ def _bucket_exists(facts: Facts, answers: Answers) -> bool | None:
     return _describe(facts, "storage", "buckets", "describe", f"gs://{answers.get('state_bucket')}")[0]
 
 
+def _bucket_location_default(facts: Facts, answers: Answers) -> str:
+    """The region of the GCE runtime's zone (``us-east1-b`` -> ``us-east1``),
+    else the multi-region ``US``."""
+    zone = real(_runtime(facts).get("zone")) or ""
+    return zone.rsplit("-", 1)[0] if zone.count("-") >= 2 else "US"
+
+
 def _roles_default(exists_key: str, roles: str):
     """The roles a NEW account is granted; none for one that exists."""
     def default(facts: Facts, answers: Answers) -> str:
@@ -213,7 +220,15 @@ QUESTIONS: tuple[Question, ...] = (
              default=lambda f, a: _gcs(f, "prefix").strip("/") or None,
              help="the default GCS backend's `prefix` in cfg/state-backends.yml"),
     Question("state_bucket_exists", "Does that bucket already exist?", kind="bool",
-             when=lambda a: bool(a.get("state_bucket")), default=_bucket_exists, help=UNASKABLE),
+             when=lambda a: bool(a.get("state_bucket")), default=_bucket_exists, help=UNASKABLE, observed=True),
+    # hygiene XX item 2: the bucket is MADE when absent, as the AWS section
+    # makes its own; whether this root manages it is a choice recorded once
+    Question("state_bucket_managed", "Does this root MANAGE the state bucket: made by an apply of this root and kept "
+             "here, so a destroy of the root removes it? (a bucket that stood before this root is read, never "
+             "managed; one this root makes is managed from then on)", kind="bool",
+             when=lambda a: bool(a.get("state_bucket")), default=lambda f, a: not a.get("state_bucket_exists")),
+    Question("state_bucket_location", "The bucket's location when this root makes it (a region, or a multi-region "
+             "such as US)", when=lambda a: bool(a.get("state_bucket")), default=_bucket_location_default),
     Question("labels", "Labels for what this root makes, as k=v,k=v", default=_labels_default, may_be_empty=True),
 )
 
@@ -257,10 +272,30 @@ def render(answers: Answers) -> Rendered:
         "gcp_write_account_exists": bool(answers.get("write_account_exists")) if want_write else False,
         "gcp_write_roles": _split(answers.get("write_roles")) if want_write else [],
     })
+    bucket = str(answers.get("state_bucket") or "")
+    bucket_exists = bool(answers.get("state_bucket_exists")) if bucket else False
+    bucket_managed = (bool(answers.get("state_bucket_managed")) or not bucket_exists) if bucket else False
+    bucket_prefix = str(answers.get("state_prefix") or "").strip("/") if bucket else ""
+    labels = {k: v for k, v in (p.split("=", 1) for p in _split(answers.get("labels")) if "=" in p)}
+    tfvars.update({
+        "gcp_state_bucket": bucket,
+        "gcp_state_prefix": bucket_prefix,
+        "gcp_state_bucket_exists": bucket_exists,
+        "gcp_state_bucket_managed": bucket_managed,
+        "gcp_state_bucket_location": str(answers.get("state_bucket_location") or "US") if bucket else "US",
+        "gcp_labels": labels,
+    })
+    variables = variables + (
+        ("gcp_state_bucket", "string", "the GCS state bucket (empty when the tree keeps no state in GCS)"),
+        ("gcp_state_prefix", "string", "the state prefix within that bucket"),
+        ("gcp_state_bucket_exists", "bool", "the bucket already stands (observed at every interview; decides the backend)"),
+        ("gcp_state_bucket_managed", "bool", "this root manages the state bucket: made by an apply of it and kept here"),
+        ("gcp_state_bucket_location", "string", "the bucket's location when this root makes it"),
+        ("gcp_labels", "map(string)", "labels on what this root makes that carries labels"),
+    )
     backend = None
-    if answers.get("state_bucket") and answers.get("state_bucket_exists"):
-        backend = ("gcs", {"bucket": answers["state_bucket"],
-                           "prefix": f"{str(answers['state_prefix']).strip('/')}/bootstrap"})
+    if bucket and bucket_exists:
+        backend = ("gcs", {"bucket": bucket, "prefix": f"{bucket_prefix}/bootstrap"})
     by_hand = ["The network and the firewall rules: the team's, never modified by the system."]
     if provider_exists and not names_repository(condition, answers):
         by_hand.insert(0, f"The existing provider `{answers['provider']}` has the attribute condition `{condition or '(none)'}`, "
@@ -272,7 +307,6 @@ def render(answers: Answers) -> Rendered:
         by_hand.append("No WRITE service account: `GCP_APPLY_SERVICE_ACCOUNT` is set to the READ-ONLY account's "
                        "address (CI never writes to a runtime it does not perform on), or delete those lines from "
                        "the `perform` job (CI_SETUP.md 3.4 step 4).")
-    labels = {k: v for k, v in (p.split("=", 1) for p in _split(answers.get("labels")) if "=" in p)}
     destroy_takes: list[str] = []
     for key, roles in (("read", _split(answers.get("read_roles"))),
                        ("write", _split(answers.get("write_roles")) if want_write else [])):
@@ -287,7 +321,9 @@ def render(answers: Answers) -> Rendered:
         module_args={**{{"provider": "provider_id"}.get(n, n): Raw(f"var.gcp_{n}") for n in names},
                      **{k: Raw(f"var.gcp_{k}") for k in ("principal_value", "attribute_condition", "read_roles",
                                                          "want_write", "write_account", "write_account_exists",
-                                                         "write_roles")}},
+                                                         "write_roles", "state_bucket", "state_prefix",
+                                                         "state_bucket_exists", "state_bucket_managed",
+                                                         "state_bucket_location", "labels")}},
         variables=variables,
         tfvars=tfvars,
         outputs=(
@@ -297,6 +333,7 @@ def render(answers: Answers) -> Rendered:
              "the READ-ONLY service account's address: the GCP_SERVICE_ACCOUNT secret"),
             ("gcp_write_service_account", f"module.{MODULE}.write_service_account",
              "the address CI may write as (the READ-ONLY one when there is no WRITE account): the GCP_APPLY_SERVICE_ACCOUNT secret"),
+            ("gcp_state_bucket", f"module.{MODULE}.state_bucket", "the GCS state bucket, when the tree keeps its state in GCS"),
         ),
         required_providers={"google": {"source": "hashicorp/google", "version": ">= 5.0"}},
         provider_blocks=('provider "google" {\n  project = var.gcp_project   # the credentials: GOOGLE_OAUTH_ACCESS_TOKEN '
@@ -334,10 +371,39 @@ def render(answers: Answers) -> Rendered:
             "every role and binding here is one member added; nothing existing is replaced"
             + ("; an account that already exists is granted nothing unless you named the roles" if any(
                 answers.get(f"{k}_account_exists") for k in ("read", "write")) else ""),
+            *((f"the GCS state bucket `{bucket}` "
+               + ("exists and is MANAGED here (made by an earlier apply of this root; a destroy of the root removes it): "
+                  "this root's state is bound to it (decision D8)" if bucket_exists and bucket_managed else
+                  "exists: this root's state is bound to it (decision D8); the bucket stood before this root and is not managed here"
+                  if bucket_exists else
+                  f"is CREATED here in {answers.get('state_bucket_location') or 'US'} (versioned, uniform access, no public "
+                  "access; the CI identities get object access on it) and managed from then on: local state for this first apply"),)
+              if bucket else ("no GCS state bucket: the tree keeps its state elsewhere",)),
             *((f"labels: {', '.join(f'{k}={v}' for k, v in sorted(labels.items()))} (service accounts carry no labels; "
-               "kept for what later iterations make)",) if labels else ()),
+               "the bucket does, when made here)",) if labels else ()),
         ),
         backend=backend,
+        coverage=(
+            (f"the workload identity pool `{answers['pool']}`", READ if answers["pool_exists"] else MADE,
+             "read as it stands" if answers["pool_exists"] else "the project lacked it"),
+            (f"the pool's GitHub provider `{answers['provider']}`", READ if provider_exists else MADE,
+             "read as it stands, never rewritten: other repositories may share its condition" if provider_exists
+             else "the project lacked it: made with a condition admitting this repository alone"),
+            (f"the READ-ONLY service account `{answers['read_account']}`",
+             READ if answers["read_account_exists"] else MADE,
+             "read as it stands; granted only the roles you named" if answers["read_account_exists"] else "the project lacked it"),
+            ((f"the WRITE service account `{answers.get('write_account')}`",
+              READ if answers.get("write_account_exists") else MADE,
+              "read as it stands; granted only the roles you named" if answers.get("write_account_exists")
+              else "the project lacked it") if want_write
+             else ("a WRITE service account", BY_HAND, "not wanted: CI performs on no GCE runtime, so none is made or read")),
+            ((f"the GCS state bucket `{bucket}`", MADE if bucket_managed else READ,
+              "made by an apply of this root and managed from then on, the CI identities' object access included"
+              if bucket_managed else "stood before this root: read, never managed")
+             if bucket else ("a GCS state bucket", BY_HAND, "the tree declares no gcs backend: nothing to make")),
+            ("the network and the firewall rules", BY_HAND,
+             "the team's, by decision: the system never creates or modifies network configuration"),
+        ),
     )
 
 

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/src/cs_image_system/gcloud_runtime/gcp_provider_specific_image.py

@@ -28,11 +28,10 @@ class GcpProviderSpecificImage(ProviderSpecificImage):
                 ret["owners"] = [f'"{self.owner}"']
             return ret
         # Deferred: find the most recent image matching the output-name pattern
-        # produced by a build in this project.
-        return {
-            "owners": ['"self"'],
-            "filters": {
-                "name": f'"{self.deferred_name_pattern}*"',
-            },
-            "most_recent": True,
-        }
+        # produced by a build in this project. stage 106: an ImageQuery in the
+        # system's vocabulary, translated like a vendor query, then quoted.
+        from cs_image_system.contract.image import ImageQuery
+        from .gcp_utils import translate_image_query
+        native = translate_image_query(ImageQuery(name=f"{self.deferred_name_pattern}*", publisher=("self",), newest=True))
+        filters = {k: f'"{v}"' for k, v in native["terms"].items() if k != "status"}
+        return {"owners": [f'"{p}"' for p in native["projects"]], "filters": filters, "most_recent": True}

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/src/cs_image_system/gcloud_runtime/gcp_runtime_builders.py

@@ -20,7 +20,8 @@ from cs_image_system.contract.core import PluginArtifactProtocol
 from .gcp_provider_specific_image import GcpProviderSpecificImage
 from .gcp_runtime_models import GCPCloudBuilderModel
 from cs_image_system.gcloud_runtime import gcp_utils
-from .gcp_utils import remap_for_image_query, query_image
+from .gcp_utils import GCP_QUERY_KEYS, query_image, translate_image_query
+from cs_image_system.contract.image import ImageQuery
 
 GCP_RUNTIME: str = "gcloud"
 
@@ -41,12 +42,18 @@ class GCPCloudBuilder(CloudBuilderBase[GCPCloudBuilderModel], PluginArtifactProt
     def provider_specific_image_class(self) -> type[ProviderSpecificImage]:
         return GcpProviderSpecificImage
 
+    # --- the image query (stage 106)
+    def supported_query_keys(self) -> frozenset[str]:
+        return GCP_QUERY_KEYS
+
+    def translate_image_query(self, query: ImageQuery) -> dict[str, Any]:
+        return translate_image_query(query)
+
     def query_provider_image(self, os_builder: OSBuilderBaseImageBuilderSubconfig) -> tuple[str, str, Mapping[str,Any]] | None:
         retval: str | None = None
-        q, missed = remap_for_image_query(os_builder) if os_builder else ({}, {})
+        q = self.translate_image_query(os_builder.image_query()) if os_builder else {}
         kv = query_image(
                 query=q,
-                post_query_filter=missed,
                 session_config=self.model.self_to_gcp_client_config(),
             )
         if kv:
@@ -71,9 +78,7 @@ class GCPCloudBuilder(CloudBuilderBase[GCPCloudBuilderModel], PluginArtifactProt
         """The image an entry pins by ``image_id`` (stage 63): a GCE image is
         named uniquely within its project, so the name is searched in the
         entry's owner projects, as the vendor query is."""
-        q, _missed = remap_for_image_query(os_builder)
-        q = {k: v for k, v in q.items() if k not in ("family", "filter")}
-        q["filter"] = f'name = "{image_id}"'
+        q = {"projects": list(os_builder.image_query().publisher), "filter": f'name = "{image_id}"'}
         kv = query_image(query=q, session_config=self.model.self_to_gcp_client_config())
         if not kv:
             return None
@@ -155,6 +160,28 @@ class GCPCloudBuilder(CloudBuilderBase[GCPCloudBuilderModel], PluginArtifactProt
             return f"GCE label keys must start with a lowercase letter; {key!r} becomes {label!r}"
         return None
 
+    def image_name_as_made(self, name: str) -> str:
+        """What GCE makes of a build name (stage 105): ``gce_name`` --
+        lowercase, dashes for what it cannot keep, a letter first, at most
+        63 characters -- the name the packer source writes."""
+        from .gcp_packer_source import gce_name
+        return gce_name(name)
+
+    def image_name_problem(self, name: str) -> str | None:
+        """Why a build's full name cannot be a GCE image name (stage 105):
+        only its length. ``gce_name`` can lowercase and rewrite, but it
+        shortens by cutting the end, where the timestamp is, so a name
+        longer than 63 characters once rewritten would lose what keeps
+        builds apart."""
+        import re
+        made = re.sub(r"[^a-z0-9-]+", "-", str(name).lower()).strip("-")
+        if made and not made[0].isalpha():
+            made = f"i-{made}"
+        if len(made) > 63:
+            return (f"a GCE image name is at most 63 characters; {name!r} becomes {made!r} "
+                    f"({len(made)}) and would be cut where the timestamp is")
+        return None
+
     def one_bake_user_per_chain(self) -> bool:
         return True                             # finding 43
 

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/src/cs_image_system/gcloud_runtime/gcp_utils.py

@@ -8,6 +8,8 @@ from __future__ import annotations
 
 import json
 import logging
+
+from cs_image_system.contract.image import ImageQuery
 import re
 from collections.abc import Mapping
 from typing import Any, Sequence, cast
@@ -16,7 +18,6 @@ from google.api_core import exceptions as gcp_exceptions
 from google.cloud import compute_v1
 
 from cs_image_system.contract.constants import SELF
-from cs_image_system.base.models.os_builder_runtime_config import OSBuilderBaseImageBuilderSubconfig
 
 log = logging.getLogger(__name__)
 
@@ -204,65 +205,42 @@ def build_filter_expression(terms: Mapping[str, Any]) -> str:
     return " AND ".join(parts)
 
 
-def remap_for_image_query(
-    rc: OSBuilderBaseImageBuilderSubconfig,
-) -> tuple[dict[str, Any], dict[str, Any]]:
-    """Convert a generic image-builder subconfig query to a GCP image query.
-
-    Returns a tuple of (query, missed) where ``query`` carries the projects to
-    search and a Compute Engine filter expression, and ``missed`` holds
-    unmappable keys applied as a post-query filter (mirrors the AWS remap).
-    """
-    ret: dict[str, Any] = {}
-    owners: Sequence[Any] = rc.get_owners() if rc else []
-    ret["projects"] = list(owners)  # resolved to real projects at query time
-    missed: dict[str, Any] = {}
-    filter_terms: dict[str, Any] = {}
-
-    for topk, topv in rc.get_query().items():
-        if topk == "owners":
-            ret["projects"] = topv if isinstance(topv, list) else [topv]
+#: stage 106: the keys of the system's image-query vocabulary Compute
+#: Engine's images.list can answer, and the field each becomes.
+GCP_QUERY_KEYS: frozenset[str] = frozenset({"name", "family", "architecture", "publisher", "state", "tags",
+                                            "newest", "native"})
+GCP_FIELD_NAMES: dict[str, str] = {"name": "name", "family": "family", "architecture": "architecture", "state": "status"}
+
+
+def translate_image_query(query: ImageQuery) -> dict[str, Any]:
+    """The system's query in Compute Engine's words (stage 106):
+    ``{"projects": [...], "terms": {<image field>: value}, "filter": <the
+    list() expression>, "most_recent": bool}``; ``query_image`` takes it.
+    ``state`` defaults to ``available`` (READY); ``architecture`` and
+    ``state`` values are mapped; ``tags`` become ``labels.<k>``. A key the
+    API cannot answer (the EC2 particulars) raises ValueError naming it
+    (``validate`` refuses it first); ``native`` keys are image fields,
+    passed verbatim."""
+    unsupported = sorted(k for k in query.keys() if k not in GCP_QUERY_KEYS)
+    if unsupported:
+        raise ValueError(f"GCE cannot translate image query key(s) {', '.join(unsupported)}")
+    terms: dict[str, Any] = {}
+    for key, name in GCP_FIELD_NAMES.items():
+        value = getattr(query, key)
+        if value is None:
             continue
-        if topk == "filters":
-            if not isinstance(topv, dict):
-                raise ValueError(f"Expected 'filters' value to be a dict, got {type(topv)}")
-            terms = dict(topv)
-            # Ensure we only get READY images, but allow override.
-            terms["state"] = terms.get("state", "available")
-            for _k, _v in terms.items():
-                if _k in AWS_ONLY_QUERY_KEYS or _k.startswith("block_device_mapping"):
-                    log.debug(f"Dropping AWS-only image query key {_k!r} for GCP query.")
-                    continue
-                if _k.startswith("tag:"):
-                    filter_terms[f"labels.{_k[4:]}"] = _v
-                    continue
-                field_name = GCP_DI_MAP.get(_k)
-                if not field_name:
-                    missed[_k] = _v
-                    continue
-                if field_name == "status":
-                    _v = _gcp_status(_v)
-                elif field_name == "architecture":
-                    _v = _gcp_architecture(_v)
-                filter_terms[field_name] = _v
-            continue
-        field_name = GCP_DI_MAP.get(topk)
-        if not field_name:
-            if topk in AWS_ONLY_QUERY_KEYS:
-                log.debug(f"Dropping AWS-only image query key {topk!r} for GCP query.")
-            else:
-                missed[topk] = topv
-            continue
-        if field_name == "status":
-            topv = _gcp_status(topv)
-        elif field_name == "architecture":
-            topv = _gcp_architecture(topv)
-        filter_terms[field_name] = topv
-
-    if "status" not in filter_terms:
-        filter_terms["status"] = "READY"
-    ret["filter"] = build_filter_expression(filter_terms)
-    return (ret, missed)
+        if name == "status":
+            value = _gcp_status(value)
+        elif name == "architecture":
+            value = _gcp_architecture(value)
+        terms[name] = value
+    terms.setdefault("status", "READY")
+    for k, v in sorted(query.tags.items()):
+        terms[f"labels.{k}"] = v
+    for k, v in query.native.items():
+        terms[str(k)] = v
+    return {"projects": list(query.publisher), "terms": terms, "filter": build_filter_expression(terms),
+            "most_recent": True if query.newest is None else bool(query.newest)}
 
 
 def _image_to_dict(image: compute_v1.Image, project: str) -> dict[str, Any]:
@@ -339,6 +317,7 @@ def query_image(
     results: list[dict[str, Any]] = []
     filter_expr = query.get("filter", "")
     family = query.get("family")
+    most_recent = query.get("most_recent", True)
     for project in projects:
         if filter_expr:
             results.extend(_query_by_filters(client, project, filter_expr))
@@ -365,6 +344,8 @@ def query_image(
     if not ilist:
         return None
     if len(ilist) > 1:
+        if not most_recent:
+            raise ImageQueryError(f"{len(ilist)} images match and the query says newest: false; narrow it")
         ilist.sort(key=lambda x: x.get("creation_timestamp", ""), reverse=True)
 
     return ilist[0]

{cs_image_system_gcloud_runtime_plugin-0.1.4 → cs_image_system_gcloud_runtime_plugin-0.1.6}/tests/test_gcp_utils.py

@@ -3,7 +3,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 """Unit tests for the GCP-native image query utilities (no network access)."""
-from typing import Any, cast
+from typing import Any
 
 import pytest
 from google.cloud import compute_v1
@@ -13,9 +13,10 @@ from cs_image_system.gcloud_runtime.gcp_utils import (
     build_filter_expression,
     get_image_owner,
     query_image,
-    remap_for_image_query,
     resolve_projects,
+    translate_image_query,
 )
+from cs_image_system.contract.image import ImageQuery
 
 
 class FakeSubconfig:
@@ -51,41 +52,32 @@ def _img(name: str, ts: str, **kw: Any) -> compute_v1.Image:
     return compute_v1.Image(name=name, creation_timestamp=ts, **kw)
 
 
-def test_remap_translates_aws_shaped_query_to_gcp_filter():
-    rc = FakeSubconfig(
-        owners=["self", "amazon"],
-        query={
-            "filters": {
-                "name": "RHEL-9.5*x86_64*",
-                "root_device_type": "ebs",           # AWS-only: dropped
-                "virtualization_type": "hvm",        # AWS-only: dropped
-                "architecture": "x86_64",
-                "state": "available",
-            }
-        },
-    )
-    q, missed = remap_for_image_query(cast(Any, rc))
-    assert q["projects"] == ["self", "amazon"]
-    assert '(name = "RHEL-9.5*x86_64*")' in q["filter"]
-    assert '(status = "READY")' in q["filter"]
-    assert '(architecture = "X86_64")' in q["filter"]
-    assert "root_device_type" not in q["filter"]
-    assert "virtualization" not in q["filter"]
-    # AWS-only keys must not leak into the post-query filter either
-    assert missed == {}
-
-
-def test_remap_defaults_status_ready_and_maps_tags_to_labels():
-    rc = FakeSubconfig(owners=[], query={"filters": {"tag:Environment": "prod"}})
-    q, _ = remap_for_image_query(cast(Any, rc))
-    assert '(labels.Environment = "prod")' in q["filter"]
-    assert '(status = "READY")' in q["filter"]
-
-
-def test_remap_unknown_keys_go_to_missed():
-    rc = FakeSubconfig(owners=[], query={"filters": {"totally_custom": "x"}})
-    _, missed = remap_for_image_query(cast(Any, rc))
-    assert missed == {"totally_custom": "x"}
+def test_translate_puts_the_systems_query_in_compute_engines_words():
+    # stage 106: the vocabulary is the system's; the values are mapped
+    q = translate_image_query(ImageQuery.from_mapping({
+        "publisher": ["self", "rhel-cloud"], "name": "rhel-9-v*", "architecture": "x86_64", "state": "available"}))
+    assert q["projects"] == ["self", "rhel-cloud"]
+    assert q["terms"] == {"name": "rhel-9-v*", "architecture": "X86_64", "status": "READY"}
+    assert '(name = "rhel-9-v*")' in q["filter"] and '(status = "READY")' in q["filter"]
+    assert '(architecture = "X86_64")' in q["filter"] and q["most_recent"] is True
+
+
+def test_translate_defaults_status_ready_maps_tags_to_labels_and_passes_native():
+    q = translate_image_query(ImageQuery.from_mapping({"tag:Environment": "prod", "family": "debian-12",
+                                                       "native": {"deprecated.state": "ACTIVE"}}))
+    assert q["terms"] == {"family": "debian-12", "status": "READY", "labels.Environment": "prod",
+                          "deprecated.state": "ACTIVE"}
+
+
+def test_translate_refuses_the_ec2_particulars_by_name():
+    # the keys GCE used to drop with a debug line are refused, named (validate refuses them first)
+    with pytest.raises(ValueError, match="root_device, virtualization"):
+        translate_image_query(ImageQuery.from_mapping({"filters": {"root_device_type": "ebs", "virtualization_type": "hvm"}}))
+
+
+def test_an_unknown_key_is_refused_by_the_vocabulary_not_buried():
+    with pytest.raises(ValueError, match="totally_custom"):
+        ImageQuery.from_mapping({"filters": {"totally_custom": "x"}})
 
 
 def test_build_filter_expression_quotes_strings_and_lowers_bools():