squidient 4.0.0__tar.gz → 4.0.2__tar.gz

squidient-4.0.2/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: squidient
+Version: 4.0.2
+Summary: CI/CD framework for alya
+Project-URL: Homepage, https://alya.gitlab.bsc.es/alya/ci-cd/squidient
+Project-URL: Bug Tracker, https://alya.gitlab.bsc.es/alya/ci-cd/squidient/issues
+Author-email: Damien Dosimont <damien.dosimont@gmail.com>
+License-File: LICENSE.txt
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.7
+Requires-Dist: importlib-resources; python_version < '3.9'
+Requires-Dist: lxml
+Requires-Dist: mariasql>=5
+Requires-Dist: packaging>=21.3
+Description-Content-Type: text/markdown
+
+# Squidient
+
+Squidient is a scheduler for testing, benchmarking and deploying HPC programs on SLURM machines.
+
+## Refreshing Workspace Data
+
+Squidient records a fingerprint of its packaged `data/` in `version/data.json`
+when `squidient init .` completes. A completed full configuration records
+that fingerprint in `configuration.json.dataRevision`. Staging and monitoring stop
+with exit code 1 if either differs from the installed package, including for older
+workspaces without this metadata. No confirmation prompt is displayed. The check
+runs before Terraform provisioning, remote connections, or job submission.
+A package release that leaves `data/` unchanged does not require a refresh.
+
+Refreshing is explicit: back up any customized workspace files and configuration
+first, then run:
+
+```sh
+squidient init .
+squidient configure --all
+```
+
+`init` overwrites packaged workspace files but does not regenerate
+`configuration.json`. `configure --all` regenerates the configuration; reapply
+your local settings as necessary. Partial reconfiguration does not mark an old
+configuration as current. The check never overwrites customizations automatically.
+Help and cleanup commands, including `terraform stop`, remain available.
+The former `fileFormat` counter and `version/version.json` are no longer used;
+legacy configurations must be refreshed before workflow execution when their
+data revision cannot be established.
+
+## MR Concurrency Checks
+
+Each MR staging/monitoring `run` job checks the GitLab API before installing
+Squidient or starting a campaign. `version:final` repeats the check before its
+release-version validation. A running or canceling MR run/version job in another
+pipeline makes the current job fail immediately, with the conflicting pipeline
+and job identifiers. There is no waiting, automatic cancellation or write API
+operation. Jobs in the same parent pipeline, including its staging and monitoring
+children, are excluded so those two workflows can still run together.
+
+The check uses `CI_API_V4_URL` and the existing `SQUIDIENT_READ_API_TOKEN`.
+Pagination, response sizes and request/check durations are bounded. API failures,
+invalid metadata or missing credentials prevent campaign execution. Master and
+trigger workflows are unchanged. Retried child runs are detected independently
+of their parent pipeline's completion status.
+
+This is a best-effort API snapshot, not an atomic lock: simultaneous starts can
+race. That risk is accepted instead of queueing campaigns or using a remote lock.
+Retry a rejected job after the other campaign finishes. TestPyPI publication is
+automatic after successful validations for eligible version-changing MRs.
+
+## CI Report Comments
+
+The three `post` jobs delegate MR comments to Taskmaster's declared `task=comment`
+input. The helper sends a JSON `inputs` object, never arbitrary pipeline variables.
+Define `TASKMASTER_PROJECT_ID` and `TASKMASTER_TOKEN` in Squidient's CI/CD settings.
+The latter is a Taskmaster pipeline trigger token, not a GitLab write API token.
+Mask it in job logs and make it available to the MR pipelines that need to post;
+the protected write API tokens remain exclusively in Taskmaster for this operation.
+`TASKMASTER_REF` optionally selects the trusted execution reference and defaults
+to `master`. GitLab provides `CI_API_V4_URL` automatically.
+
+Taskmaster selects its own write credential using the target project: Squidient
+for MR staging/monitoring reports, or Alya for triggered staging reports. Titles
+and report links are unchanged. No write API credential is used by the `post`
+scripts. Execution jobs use `SQUIDIENT_READ_API_TOKEN` for the watchdog
+and MR lock checks; triggered Alya staging jobs also use `ALYA_READ_API_TOKEN` to
+watch the source Alya pipeline. Neither `ALYA_WRITE_API_TOKEN` nor
+`SQUIDIENT_WRITE_API_TOKEN` is required by Squidient's CI anymore. Their use for
+publishing comments is confined to Taskmaster.
+
+The caller only submits the request; successful submission does not mean that
+Taskmaster has already posted the comment. Check the indicated Taskmaster pipeline
+for the publication result. Posting remains non-blocking for the Squidient
+pipeline. Automatic retries are disabled to reduce duplicate comments; manually
+rerunning a post job can still create duplicates. Credentials, comment bodies
+and API response bodies are not printed by the submission helper.
+
+The Gazix, Taskmaster, Alya and Squidient migrations may be merged together.
+Gazix 2.0.0 must finish publishing before Taskmaster jobs can install it. Configure
+Taskmaster's protected write credentials and a protected execution ref, then run
+its `tools/activate_inputs.py` preflight and explicit activation after both known
+master callers have migrated. This disables arbitrary pipeline-variable overrides
+in Taskmaster; there is no legacy-variable fallback. Only Alya and
+Squidient on the same GitLab instance are currently authorized by Taskmaster;
+an explicitly configured source API URL on another instance is rejected rather
+than silently posting to the wrong project.
+

squidient-4.0.2/README.md

@@ -0,0 +1,89 @@
+# Squidient
+
+Squidient is a scheduler for testing, benchmarking and deploying HPC programs on SLURM machines.
+
+## Refreshing Workspace Data
+
+Squidient records a fingerprint of its packaged `data/` in `version/data.json`
+when `squidient init .` completes. A completed full configuration records
+that fingerprint in `configuration.json.dataRevision`. Staging and monitoring stop
+with exit code 1 if either differs from the installed package, including for older
+workspaces without this metadata. No confirmation prompt is displayed. The check
+runs before Terraform provisioning, remote connections, or job submission.
+A package release that leaves `data/` unchanged does not require a refresh.
+
+Refreshing is explicit: back up any customized workspace files and configuration
+first, then run:
+
+```sh
+squidient init .
+squidient configure --all
+```
+
+`init` overwrites packaged workspace files but does not regenerate
+`configuration.json`. `configure --all` regenerates the configuration; reapply
+your local settings as necessary. Partial reconfiguration does not mark an old
+configuration as current. The check never overwrites customizations automatically.
+Help and cleanup commands, including `terraform stop`, remain available.
+The former `fileFormat` counter and `version/version.json` are no longer used;
+legacy configurations must be refreshed before workflow execution when their
+data revision cannot be established.
+
+## MR Concurrency Checks
+
+Each MR staging/monitoring `run` job checks the GitLab API before installing
+Squidient or starting a campaign. `version:final` repeats the check before its
+release-version validation. A running or canceling MR run/version job in another
+pipeline makes the current job fail immediately, with the conflicting pipeline
+and job identifiers. There is no waiting, automatic cancellation or write API
+operation. Jobs in the same parent pipeline, including its staging and monitoring
+children, are excluded so those two workflows can still run together.
+
+The check uses `CI_API_V4_URL` and the existing `SQUIDIENT_READ_API_TOKEN`.
+Pagination, response sizes and request/check durations are bounded. API failures,
+invalid metadata or missing credentials prevent campaign execution. Master and
+trigger workflows are unchanged. Retried child runs are detected independently
+of their parent pipeline's completion status.
+
+This is a best-effort API snapshot, not an atomic lock: simultaneous starts can
+race. That risk is accepted instead of queueing campaigns or using a remote lock.
+Retry a rejected job after the other campaign finishes. TestPyPI publication is
+automatic after successful validations for eligible version-changing MRs.
+
+## CI Report Comments
+
+The three `post` jobs delegate MR comments to Taskmaster's declared `task=comment`
+input. The helper sends a JSON `inputs` object, never arbitrary pipeline variables.
+Define `TASKMASTER_PROJECT_ID` and `TASKMASTER_TOKEN` in Squidient's CI/CD settings.
+The latter is a Taskmaster pipeline trigger token, not a GitLab write API token.
+Mask it in job logs and make it available to the MR pipelines that need to post;
+the protected write API tokens remain exclusively in Taskmaster for this operation.
+`TASKMASTER_REF` optionally selects the trusted execution reference and defaults
+to `master`. GitLab provides `CI_API_V4_URL` automatically.
+
+Taskmaster selects its own write credential using the target project: Squidient
+for MR staging/monitoring reports, or Alya for triggered staging reports. Titles
+and report links are unchanged. No write API credential is used by the `post`
+scripts. Execution jobs use `SQUIDIENT_READ_API_TOKEN` for the watchdog
+and MR lock checks; triggered Alya staging jobs also use `ALYA_READ_API_TOKEN` to
+watch the source Alya pipeline. Neither `ALYA_WRITE_API_TOKEN` nor
+`SQUIDIENT_WRITE_API_TOKEN` is required by Squidient's CI anymore. Their use for
+publishing comments is confined to Taskmaster.
+
+The caller only submits the request; successful submission does not mean that
+Taskmaster has already posted the comment. Check the indicated Taskmaster pipeline
+for the publication result. Posting remains non-blocking for the Squidient
+pipeline. Automatic retries are disabled to reduce duplicate comments; manually
+rerunning a post job can still create duplicates. Credentials, comment bodies
+and API response bodies are not printed by the submission helper.
+
+The Gazix, Taskmaster, Alya and Squidient migrations may be merged together.
+Gazix 2.0.0 must finish publishing before Taskmaster jobs can install it. Configure
+Taskmaster's protected write credentials and a protected execution ref, then run
+its `tools/activate_inputs.py` preflight and explicit activation after both known
+master callers have migrated. This disables arbitrary pipeline-variable overrides
+in Taskmaster; there is no legacy-variable fallback. Only Alya and
+Squidient on the same GitLab instance are currently authorized by Taskmaster;
+an explicitly configured source API URL on another instance is rejected rather
+than silently posting to the wrong project.
+

{squidient-4.0.0 → squidient-4.0.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "squidient"
-version = "4.0.0"
+version = "4.0.2"
 authors = [
   { name="Damien Dosimont", email="damien.dosimont@gmail.com" },
 ]

{squidient-4.0.0 → squidient-4.0.2}/src/squidient/data/terraform/aws-ec2-t3.small-x2/t3.small/t3.small.tf

@@ -95,7 +95,7 @@ resource "aws_instance" "slurm_head" {
 
   # Head gets a single "primary" compute hostname for its slurm.conf
   user_data = templatefile(
-    "${path.module}/cloud-init-slurm-fedora.sh",
+    "${path.module}/../../common/cloud-init-slurm-fedora.sh",
     {
       ROLE             = "head"
       HEAD_HOSTNAME    = "slurm-head"
@@ -131,7 +131,7 @@ resource "aws_instance" "slurm_compute" {
   depends_on = [aws_instance.slurm_head]
 
   user_data = templatefile(
-    "${path.module}/cloud-init-slurm-fedora.sh",
+    "${path.module}/../../common/cloud-init-slurm-fedora.sh",
     {
       ROLE             = "compute"
       HEAD_HOSTNAME    = "slurm-head"

{squidient-4.0.0 → squidient-4.0.2}/src/squidient/data/terraform/aws-ec2-t3.small-x6/t3.small/t3.small.tf

@@ -95,7 +95,7 @@ resource "aws_instance" "slurm_head" {
 
   # Head gets a single "primary" compute hostname for its slurm.conf
   user_data = templatefile(
-    "${path.module}/cloud-init-slurm-fedora.sh",
+    "${path.module}/../../common/cloud-init-slurm-fedora.sh",
     {
       ROLE              = "head"
       HEAD_HOSTNAME     = "slurm-head"
@@ -131,7 +131,7 @@ resource "aws_instance" "slurm_compute" {
   depends_on = [aws_instance.slurm_head]
 
   user_data = templatefile(
-    "${path.module}/cloud-init-slurm-fedora.sh",
+    "${path.module}/../../common/cloud-init-slurm-fedora.sh",
     {
       ROLE              = "compute"
       HEAD_HOSTNAME     = "slurm-head"

{squidient-4.0.0/src/squidient/data/terraform/aws-ec2-t3.small-x6/t3.small → squidient-4.0.2/src/squidient/data/terraform/common}/cloud-init-slurm-fedora.sh

@@ -2,7 +2,7 @@
 set -eux
 
 # ------------------------------------------------------------
-# Role and topology (filled by Terraform via templatefile)
+# Role and topology (provided by each Terraform configuration)
 # ------------------------------------------------------------
 ROLE="${ROLE}"
 HEAD_HOSTNAME="${HEAD_HOSTNAME}"
@@ -154,13 +154,67 @@ else
       sleep 3
     done
 
-    if mountpoint -q /shared; then
-      COMPUTE_IP="$(hostname -I | awk '{print $1}')"
-      echo "$COMPUTE_IP" > "/shared/${COMPUTE_HOSTNAME}.ip" || true
-    fi
   fi
 fi
 
+# ------------------------------------------------------------
+# Share the complete cluster host map with every MPI launch node
+# ------------------------------------------------------------
+if [ "${ROLE}" != "head" ] && ! mountpoint -q /shared; then
+  echo "Shared storage is not mounted; cannot resolve cluster nodes" >&2
+  exit 1
+fi
+
+LOCAL_IP="$(hostname -I | awk '{print $1}')"
+printf '%s\n' "$LOCAL_IP" > "/shared/$HOSTNAME.ip"
+
+all_ready=false
+for i in $(seq 1 300); do
+  all_ready=true
+  for node in "${HEAD_HOSTNAME}" ${COMPUTE_NODE_LIST}; do
+    if [ ! -s "/shared/$node.ip" ]; then
+      all_ready=false
+    fi
+  done
+  if $all_ready; then break; fi
+  sleep 2
+done
+
+if ! $all_ready; then
+  echo "Timed out waiting for all cluster node addresses in /shared" >&2
+  exit 1
+fi
+
+python3 - /shared /etc/hosts "${HEAD_HOSTNAME}" "${COMPUTE_NODE_LIST}" <<'CLUSTER_HOSTS_PY'
+import ipaddress
+from pathlib import Path
+import sys
+
+shared = Path(sys.argv[1])
+hosts = Path(sys.argv[2])
+names = [sys.argv[3]] + sys.argv[4].split()
+addresses = {
+    name: str(ipaddress.IPv4Address((shared / (name + ".ip")).read_text().strip()))
+    for name in names
+}
+
+# Replace cluster aliases while preserving unrelated host entries and comments.
+lines = []
+for line in hosts.read_text().splitlines():
+    body, separator, comment = line.partition("#")
+    fields = body.split()
+    if len(fields) > 1 and any(alias in addresses for alias in fields[1:]):
+        aliases = [alias for alias in fields[1:] if alias not in addresses]
+        if not aliases and not separator:
+            continue
+        line = " ".join([fields[0]] + aliases) if aliases else ""
+        if separator:
+            line += (" " if line else "") + "#" + comment
+    lines.append(line)
+lines.extend(addresses[name] + " " + name for name in names)
+hosts.write_text("\n".join(lines) + "\n")
+CLUSTER_HOSTS_PY
+
 # ------------------------------------------------------------
 # Munge setup
 # ------------------------------------------------------------

squidient-4.0.0/PKG-INFO

@@ -1,49 +0,0 @@
-Metadata-Version: 2.4
-Name: squidient
-Version: 4.0.0
-Summary: CI/CD framework for alya
-Project-URL: Homepage, https://alya.gitlab.bsc.es/alya/ci-cd/squidient
-Project-URL: Bug Tracker, https://alya.gitlab.bsc.es/alya/ci-cd/squidient/issues
-Author-email: Damien Dosimont <damien.dosimont@gmail.com>
-License-File: LICENSE.txt
-Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
-Classifier: Operating System :: POSIX :: Linux
-Classifier: Programming Language :: Python :: 3
-Requires-Python: >=3.7
-Requires-Dist: importlib-resources; python_version < '3.9'
-Requires-Dist: lxml
-Requires-Dist: mariasql>=5
-Requires-Dist: packaging>=21.3
-Description-Content-Type: text/markdown
-
-# Squidient
-
-Squidient is a scheduler for testing, benchmarking and deploying HPC programs on SLURM machines.
-
-## Refreshing Workspace Data
-
-Squidient records a fingerprint of its packaged `data/` in `version/data.json`
-when `squidient init .` completes. A completed full configuration records
-that fingerprint in `configuration.json.dataRevision`. Staging and monitoring stop
-with exit code 1 if either differs from the installed package, including for older
-workspaces without this metadata. No confirmation prompt is displayed. The check
-runs before Terraform provisioning, remote connections, or job submission.
-A package release that leaves `data/` unchanged does not require a refresh.
-
-Refreshing is explicit: back up any customized workspace files and configuration
-first, then run:
-
-```sh
-squidient init .
-squidient configure --all
-```
-
-`init` overwrites packaged workspace files but does not regenerate
-`configuration.json`. `configure --all` regenerates the configuration; reapply
-your local settings as necessary. Partial reconfiguration does not mark an old
-configuration as current. The check never overwrites customizations automatically.
-Help and cleanup commands, including `terraform stop`, remain available.
-The former `fileFormat` counter and `version/version.json` are no longer used;
-legacy configurations must be refreshed before workflow execution when their
-data revision cannot be established.
-

squidient-4.0.0/README.md

@@ -1,31 +0,0 @@
-# Squidient
-
-Squidient is a scheduler for testing, benchmarking and deploying HPC programs on SLURM machines.
-
-## Refreshing Workspace Data
-
-Squidient records a fingerprint of its packaged `data/` in `version/data.json`
-when `squidient init .` completes. A completed full configuration records
-that fingerprint in `configuration.json.dataRevision`. Staging and monitoring stop
-with exit code 1 if either differs from the installed package, including for older
-workspaces without this metadata. No confirmation prompt is displayed. The check
-runs before Terraform provisioning, remote connections, or job submission.
-A package release that leaves `data/` unchanged does not require a refresh.
-
-Refreshing is explicit: back up any customized workspace files and configuration
-first, then run:
-
-```sh
-squidient init .
-squidient configure --all
-```
-
-`init` overwrites packaged workspace files but does not regenerate
-`configuration.json`. `configure --all` regenerates the configuration; reapply
-your local settings as necessary. Partial reconfiguration does not mark an old
-configuration as current. The check never overwrites customizations automatically.
-Help and cleanup commands, including `terraform stop`, remain available.
-The former `fileFormat` counter and `version/version.json` are no longer used;
-legacy configurations must be refreshed before workflow execution when their
-data revision cannot be established.
-