scale-forecasting 1.2.0__tar.gz → 1.2.1__tar.gz

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/CHANGELOG.md

@@ -11,6 +11,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [1.2.1] - 2026-10-14
+
+### Added
+
+- **Atomic Version Bump Automation (`make bump VERSION=X.Y.Z`):**
+  - Added [`bump_version()`](./src/scale_forecasting/agent_surfaces.py) (`python -m scale_forecasting.agent_surfaces --bump-version X.Y.Z`) and `make bump VERSION=X.Y.Z` in [`Makefile`](./Makefile) to atomically update all 7 version-locked files ([`pyproject.toml`](./pyproject.toml), [`src/scale_forecasting/__init__.py`](./src/scale_forecasting/__init__.py), [`plugin.json`](./plugin.json), [`skills/scale-forecasting/SKILL.md`](./skills/scale-forecasting/SKILL.md), [`CHANGELOG.md`](./CHANGELOG.md), [`CITATION.cff`](./CITATION.cff), and [`docker/cloudbuild.yaml`](./docker/cloudbuild.yaml)), refresh `uv.lock` and `docker/requirements.txt`, sync the editable `.venv` `.dist-info`, and regenerate all 6 AI agent surface files.
+- **Version-Tagged Runtime Containers (`:vX.Y.Z`) & Launcher $\leftrightarrow$ Worker Version Parity Guard (`code_delivery.py`):**
+  - [`docker/cloudbuild.yaml`](./docker/cloudbuild.yaml) and `make image` in [`Makefile`](./Makefile) now tag and push both `:latest` and `:v<version>` (`:v1.2.1`) to Artifact Registry alongside the content-addressed `venv_archive` tarball.
+  - Added [`verify_version_parity()`](./src/scale_forecasting/code_delivery.py), wired into `spark_explode._stamp_executed_fanout` and `ray_engine._stamp_executed_sizing` (shared across `ray`, `vertex`, `gce`, and `gke`), stamping `launcher` and `worker` `scale_forecasting.__version__` (`$.version`) into `run_registry.job_telemetry` and raising `ConfigError` on mismatch when `SF_STRICT_VERSION_PARITY=1`.
+- **Complete 7-Runtime Provisioning vs. Execution Telemetry (`$.platform`):**
+  - Extended [`batch_telemetry.platform_phases()`](./src/scale_forecasting/batch_telemetry.py) (`_to_epoch_seconds` / `_rfc3339_seconds`) to accept protobuf `Timestamp`, `datetime`, epoch seconds/milliseconds, and RFC 3339 strings, and wired `$.platform` (`provision_s`, `execute_s`, `total_wall_s`) into `RayProbe`, `GceProbe`, `GkeProbe`, and `VertexAutoMLProbe` in [`src/scale_forecasting/probes/runtimes.py`](./src/scale_forecasting/probes/runtimes.py) as well as ephemeral submitters ([`gce_submit.py`](./src/scale_forecasting/gce_submit.py), [`gke_submit.py`](./src/scale_forecasting/gke_submit.py), [`ray_submit.py`](./src/scale_forecasting/ray_submit.py), [`cluster_submit.py`](./src/scale_forecasting/cluster_submit.py), [`automl_submit.py`](./src/scale_forecasting/automl_submit.py)) so `v_run_jobs.provision_seconds` and `execute_seconds` are populated across all compute runtimes.
+- **Cross-Version BigQuery Registry Compatibility Tripwire ([`tests/unit/test_cross_version_resilience.py`](./tests/unit/test_cross_version_resilience.py)):**
+  - Verifies backward and forward compatibility of `Forecaster.from_run_id()`, `ops.match_reusable_models()` & `render_borrow_model_sql()`, `review._assemble_review()` & DataFrame builders, and `diagnostics.py` (`compute_fva`, `compute_inventory_policy`, `compute_anomalies`, `compute_change_points`) across historical `v1.0.0`, `v1.1.0`, and `v1.2.0` registry payloads.
+
+### Changed
+
+- **Interactive Showcase Notebook Refresh ([`notebooks/01_bigquery_native_sql.ipynb`](./notebooks/01_bigquery_native_sql.ipynb) & [`notebooks/08_multi_engine_master_workflow.ipynb`](./notebooks/08_multi_engine_master_workflow.ipynb)):**
+  - Re-executed and persisted `01_bigquery_native_sql.ipynb` (`arima_plus`, `arima_plus_xreg`, `timesfm` `TimesFM 3.0` multivariate, `timesfm@univariate` `TimesFM 2.5`, Covariate FVA, Two-Tier `ML.EXPLAIN_FORECAST` attributions, anomaly/change-point detection, and $(s, S)$ inventory replenishment policy) and `08_multi_engine_master_workflow.ipynb` (4-family parallel DAG with `[models-foundation]` `chronos_bolt`, FVA, TreeSHAP attributions, `v_run_jobs` provisioning telemetry, anomaly/change-point detection, and inventory replenishment policy) against the `v1.2.1` runtime container.
+
+---
+
 ## [1.2.0] - 2026-10-14
 
 ### Added

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: scale-forecasting
-Version: 1.2.0
+Version: 1.2.1
 Summary: Time-series forecasting on Google Cloud at any scale: one JSON configuration runs every model family on Dataproc Serverless, Ray, Vertex AI, GKE, Compute Engine, or BigQuery ML and writes forecasts and leaderboards back to BigQuery.
 Project-URL: Homepage, https://github.com/statmike/scale-forecasting
 Project-URL: Documentation, https://statmike.github.io/scale-forecasting/

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "scale-forecasting"
-version = "1.2.0"
+version = "1.2.1"
 description = "Time-series forecasting on Google Cloud at any scale: one JSON configuration runs every model family on Dataproc Serverless, Ray, Vertex AI, GKE, Compute Engine, or BigQuery ML and writes forecasts and leaderboards back to BigQuery."
 dynamic = ["readme"]
 # Two different pins, on purpose. `requires-python` is what *the package* installs on: the pure

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/__init__.py

@@ -53,7 +53,7 @@ from .settings import Settings
 try:
     __version__: str = _pkg_version("scale-forecasting")
 except PackageNotFoundError:  # pragma: no cover - zip-staged cloud worker without .dist-info
-    __version__ = "1.2.0"
+    __version__ = "1.2.1"
 
 # Lazy (heavy): name -> (module, attribute). Resolved on first access by ``__getattr__`` (PEP 562)
 # so the model modules (statsmodels et al., ~5s) load only when a caller actually reaches for the

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/agent_surfaces.py

@@ -23,6 +23,7 @@ CLI usage::
     python -m scale_forecasting.agent_surfaces --write
     python -m scale_forecasting.agent_surfaces --check
     python -m scale_forecasting.agent_surfaces --probe-env
+    python -m scale_forecasting.agent_surfaces --bump-version X.Y.Z
 """
 
 from __future__ import annotations
@@ -30,6 +31,7 @@ from __future__ import annotations
 import argparse
 import json
 import os
+import re
 import sys
 from importlib.metadata import version as _pkg_version
 from pathlib import Path
@@ -73,6 +75,7 @@ __all__ = [
     "GPU_CATALOG",
     "RUNTIME_CATALOG",
     "build_run_config_json_schema",
+    "bump_version",
     "check_agent_surfaces",
     "expected_agent_surface_files",
     "main",
@@ -1530,6 +1533,138 @@ def check_agent_surfaces(repo_root: Path | None = None) -> list[str]:
     return drifted
 
 
+_SEMVER_RE = re.compile(r"^\d+\.\d+\.\d+(?:[a-zA-Z0-9.\-+]+)?$")
+
+
+def bump_version(
+    new_version: str,
+    repo_root: Path | None = None,
+    *,
+    release_date: str | None = None,
+) -> list[Path]:
+    """Atomically update the package version across all version-locked repository files.
+
+    Updates:
+    1. ``pyproject.toml`` (``[project].version``)
+    2. ``src/scale_forecasting/__init__.py`` (``__version__`` fallback)
+    3. ``plugin.json`` (``"version"``)
+    4. ``skills/scale-forecasting/SKILL.md`` (``metadata.version``)
+    5. ``CITATION.cff`` (``version`` and ``date-released``)
+    6. ``CHANGELOG.md`` (inserts ``## [<version>] - <date>`` section if not already present)
+    7. ``docker/cloudbuild.yaml`` (``_VERSION_TAG: "v<version>"`` when present)
+    8. All 6 generated agent surface files via `write_agent_surfaces` when ``mkdocs.yml`` exists.
+
+    Returns the list of modified file paths.
+    """
+    ver = new_version.strip().lstrip("v")
+    if not _SEMVER_RE.match(ver):
+        raise ValueError(f"Invalid semantic version {new_version!r}; expected 'MAJOR.MINOR.PATCH'.")
+    root = repo_root or _default_repo_root()
+    from datetime import UTC, datetime
+
+    rel_date = release_date or datetime.now(UTC).strftime("%Y-%m-%d")
+    updated: list[Path] = []
+
+    pyproject_path = root / "pyproject.toml"
+    if pyproject_path.is_file():
+        text = pyproject_path.read_text(encoding="utf-8")
+        new_text, n = re.subn(
+            r'(?m)^version\s*=\s*"[^"]+"',
+            f'version = "{ver}"',
+            text,
+            count=1,
+        )
+        if n:
+            pyproject_path.write_text(new_text, encoding="utf-8")
+            updated.append(pyproject_path)
+
+    init_path = root / "src" / "scale_forecasting" / "__init__.py"
+    if init_path.is_file():
+        text = init_path.read_text(encoding="utf-8")
+        new_text, n = re.subn(
+            r'(?m)^(\s*__version__\s*=\s*)"[^"]+"',
+            rf'\1"{ver}"',
+            text,
+            count=1,
+        )
+        if n:
+            init_path.write_text(new_text, encoding="utf-8")
+            updated.append(init_path)
+
+    plugin_path = root / "plugin.json"
+    if plugin_path.is_file():
+        plugin_data = json.loads(plugin_path.read_text(encoding="utf-8"))
+        plugin_data["version"] = ver
+        plugin_path.write_text(json.dumps(plugin_data, indent=2) + "\n", encoding="utf-8")
+        updated.append(plugin_path)
+
+    skill_path = root / "skills" / "scale-forecasting" / "SKILL.md"
+    if skill_path.is_file():
+        text = skill_path.read_text(encoding="utf-8")
+        new_text, n = re.subn(
+            r"(?m)^(\s*version:\s*)\S+",
+            rf"\g<1>{ver}",
+            text,
+            count=1,
+        )
+        if n:
+            skill_path.write_text(new_text, encoding="utf-8")
+            updated.append(skill_path)
+
+    citation_path = root / "CITATION.cff"
+    if citation_path.is_file():
+        text = citation_path.read_text(encoding="utf-8")
+        new_text = re.sub(
+            r'(?m)^version:\s*"[^"]+"',
+            f'version: "{ver}"',
+            text,
+            count=1,
+        )
+        new_text = re.sub(
+            r'(?m)^date-released:\s*"[^"]+"',
+            f'date-released: "{rel_date}"',
+            new_text,
+            count=1,
+        )
+        citation_path.write_text(new_text, encoding="utf-8")
+        updated.append(citation_path)
+
+    changelog_path = root / "CHANGELOG.md"
+    if changelog_path.is_file():
+        text = changelog_path.read_text(encoding="utf-8")
+        if f"## [{ver}]" not in text:
+            stub = (
+                f"## [Unreleased]\n\n---\n\n"
+                f"## [{ver}] - {rel_date}\n\n"
+                f"### Changed\n\n"
+                f"- Release `{ver}`.\n\n---"
+            )
+            if "## [Unreleased]\n\n---" in text:
+                new_text = text.replace("## [Unreleased]\n\n---", stub, 1)
+            else:
+                new_text = text + f"\n## [{ver}] - {rel_date}\n"
+            changelog_path.write_text(new_text, encoding="utf-8")
+            updated.append(changelog_path)
+
+    cloudbuild_path = root / "docker" / "cloudbuild.yaml"
+    if cloudbuild_path.is_file():
+        text = cloudbuild_path.read_text(encoding="utf-8")
+        new_text, n = re.subn(
+            r'(?m)^(\s*_VERSION_TAG:\s*)"[^"]+"',
+            rf'\1"v{ver}"',
+            text,
+            count=1,
+        )
+        if n:
+            cloudbuild_path.write_text(new_text, encoding="utf-8")
+            updated.append(cloudbuild_path)
+
+    if (root / "mkdocs.yml").is_file():
+        updated.extend(write_agent_surfaces(root))
+
+    return updated
+
+
 def main(argv: list[str] | None = None) -> int:
     """CLI entry point for generating, checking, or probing agent surfaces."""
     configure_cli_logging()
@@ -1553,12 +1688,23 @@ def main(argv: list[str] | None = None) -> int:
         action="store_true",
         help="Print a JSON report of installed extras, available models, and SF_* cloud readiness.",
     )
+    group.add_argument(
+        "--bump-version",
+        metavar="X.Y.Z",
+        help="Atomically bump package version across all version-locked files and agent surfaces.",
+    )
     args = parser.parse_args(argv)
 
     if args.probe_env:
         print(json.dumps(probe_environment(), indent=2, sort_keys=True))
         return 0
 
+    if args.bump_version:
+        updated = bump_version(args.bump_version)
+        for path in updated:
+            print(f"updated {path}")
+        return 0
+
     if args.write:
         written = write_agent_surfaces()
         for path in written:

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/automl_submit.py

@@ -239,7 +239,10 @@ def submit_automl(
                 },
             )
 
+    import time
+
     _ = wait
+    t_start = time.monotonic()
     outcome = automl_engine.run(
         cfg,
         models=automl_models,
@@ -247,6 +250,7 @@ def submit_automl(
         settings=settings,
         job_id=plan.display_name,
     )
+    wall_s = round(max(0.0, time.monotonic() - t_start), 1)
 
     # If the pipeline produced a concrete resource_name or pipeline_job_id, update the handle.
     models_telem = outcome.get("models") or {}
@@ -263,6 +267,12 @@ def submit_automl(
         region=settings.region,
         resource_name=str(resource_name) if resource_name else None,
     )
+    prov_s = first_telem.get("provision_s")
+    exec_s = first_telem.get("execute_s")
+    if exec_s is None:
+        exec_s = wall_s
+    if prov_s is None:
+        prov_s = round(max(0.0, wall_s - float(exec_s)), 1)
     if job_id is not None:
         with contextlib.suppress(Exception):
             update_job(
@@ -270,6 +280,11 @@ def submit_automl(
                 settings=settings,
                 merge_telemetry={
                     "probe_handle": final_handle.to_blob(),
+                    "platform": {
+                        "provision_s": prov_s,
+                        "execute_s": exec_s,
+                        "total_wall_s": wall_s,
+                    },
                     "vertex_automl": {
                         **plan.to_dict(),
                         "models_telemetry": models_telem,

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/batch_telemetry.py

@@ -37,19 +37,50 @@ def _batch_client(region: str) -> BatchControllerClient:
     )
 
 
+def _to_epoch_seconds(val: object) -> float | None:
+    """Coerce a protobuf ``Timestamp``, ``datetime``, epoch int/float, or ISO-8601 string."""
+    if val is None or isinstance(val, bool):
+        return None
+    ts_fn = getattr(val, "timestamp", None)
+    if callable(ts_fn):
+        try:
+            res = ts_fn()
+            return float(res) if isinstance(res, int | float) else None
+        except Exception:  # noqa: BLE001 - telemetry is best-effort, never fatal
+            return None
+    if isinstance(val, int | float):
+        fval = float(val)
+        # Ray JobInfo timestamps are epoch milliseconds (> 1e11 for modern dates).
+        return fval / 1000.0 if fval > 1e11 else fval
+    if isinstance(val, str) and val.strip():
+        from datetime import datetime
+
+        raw = val.strip()
+        try:
+            return (
+                float(raw)
+                if raw.replace(".", "", 1).isdigit()
+                else datetime.fromisoformat(raw.replace("Z", "+00:00")).timestamp()
+            )
+        except Exception:  # noqa: BLE001 - malformed timestamp string degrades cleanly
+            return None
+    return None
+
+
 def _rfc3339_seconds(a: object, b: object) -> float | None:
-    """Whole seconds between two Dataproc timestamp fields (``b - a``), or None.
+    """Seconds between two timestamps (``b - a``), or None.
 
-    Dataproc stamps ``create_time``/``state_time`` as ``google.protobuf.Timestamp``; both expose
-    ``.timestamp()`` (via the proto's datetime helper). Returns None if either is missing so a
-    partial batch object degrades cleanly rather than raising.
+    Accepts ``google.protobuf.Timestamp`` / ``datetime`` (``.timestamp()``), epoch seconds or
+    milliseconds (``int`` / ``float``), and ISO-8601 / RFC-3339 strings (such as Kubernetes
+    ``creationTimestamp`` / ``startTime`` / ``completionTime``). Returns None if either is missing
+    so a partial job object degrades cleanly rather than raising.
     """
-    ts_a = getattr(a, "timestamp", None)
-    ts_b = getattr(b, "timestamp", None)
-    if not callable(ts_a) or not callable(ts_b):
+    ea = _to_epoch_seconds(a)
+    eb = _to_epoch_seconds(b)
+    if ea is None or eb is None:
         return None
     try:
-        return round(ts_b() - ts_a(), 1)
+        return round(max(0.0, eb - ea), 1)
     except Exception:  # noqa: BLE001 - telemetry is best-effort, never fatal
         return None
 

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/cluster_submit.py

@@ -291,6 +291,9 @@ def submit_cluster_job(
         workers,
     )
 
+    import time
+
+    t_submit_start = time.perf_counter()
     if reuse:
         # A named cluster (standing, or the run's shared ephemeral): submit to its region — a shared
         # cluster may have hopped on a capacity failover, so trust the region the caller threads.
@@ -322,6 +325,8 @@ def submit_cluster_job(
 
     cluster_client = _cluster_client(region)
     job_client = _job_client(region)
+    t_job_start = time.perf_counter()
+    provision_s = round(max(0.0, t_job_start - t_submit_start), 1)
 
     # Before submit, so nothing this job writes can predate the liveness window the watchdog reads.
     since = launch_window_start()
@@ -355,6 +360,24 @@ def submit_cluster_job(
         # The fallback is kept for the case where we did name it and the response still came back
         # without one: the id is ours and we already know it, so there is no reason to return "".
         final_id = submitted_id or job_id or ""
+        if wait and job_id is not None:
+            import contextlib
+
+            from .registry.jobs import update_job
+
+            exec_s = round(max(0.0, time.perf_counter() - t_job_start), 1)
+            with contextlib.suppress(Exception):
+                update_job(
+                    job_id,
+                    settings=settings,
+                    merge_telemetry={
+                        "platform": {
+                            "provision_s": provision_s,
+                            "execute_s": exec_s,
+                            "total_wall_s": round(max(0.0, provision_s + exec_s), 1),
+                        }
+                    },
+                )
         if wait and state_name != "DONE":
             raise EngineError(
                 f"cluster job {final_id} terminal state {state_name}: {detail or '(no detail)'}"

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/code_delivery.py

@@ -194,3 +194,45 @@ def write_package_zip(dest_dir: str | Path | None = None) -> Path:
     out = base / f"scale_forecasting-{code_hash}.zip"
     out.write_bytes(data)
     return out
+
+
+def verify_version_parity(
+    launcher_version: str | None = None,
+    worker_version: str | None = None,
+    *,
+    strict: bool = False,
+) -> dict[str, Any]:
+    """Compare launcher and worker ``scale_forecasting.__version__`` strings.
+
+    Returns a structured telemetry dict ``{"launcher": str, "worker": str, "match": bool}``.
+    When ``launcher_version`` or ``worker_version`` is omitted, resolves
+    ``scale_forecasting.__version__`` from the active process (or ``SF_LAUNCHER_VERSION`` in
+    ``os.environ`` for the launcher side on a remote worker). Logs a warning when the versions
+    differ and raises `ConfigError` when ``strict=True``.
+    """
+    import os
+
+    import scale_forecasting
+
+    from .errors import ConfigError, get_logger
+
+    current_ver = scale_forecasting.__version__
+    resolved_worker = (worker_version or current_ver).strip()
+    resolved_launcher = (
+        launcher_version or os.environ.get("SF_LAUNCHER_VERSION") or current_ver
+    ).strip()
+    match = resolved_launcher == resolved_worker
+    if not match:
+        msg = (
+            f"scale-forecasting version mismatch between launcher ({resolved_launcher!r}) and "
+            f"worker ({resolved_worker!r}); rebuild runtime image with `make image` or sync "
+            "package versions."
+        )
+        get_logger(__name__).warning(msg)
+        if strict:
+            raise ConfigError(msg)
+    return {
+        "launcher": resolved_launcher,
+        "worker": resolved_worker,
+        "match": match,
+    }

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/engines/ray_engine.py

@@ -787,10 +787,12 @@ def _stamp_executed_sizing(
     inside a live run on the cluster head. A header that will not take an overlay costs a record,
     never a run.
     """
+    from ..code_delivery import verify_version_parity
     from ..registry.header import merge_header_telemetry
 
     try:
-        merge_header_telemetry(run_id, patch, settings=settings)
+        parity = verify_version_parity()
+        merge_header_telemetry(run_id, {**patch, "version": parity}, settings=settings)
     except Exception as exc:  # noqa: BLE001 - telemetry is an overlay, never fatal
         _log.warning("executed sizing capture failed (non-fatal): %r", exc)
 

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/engines/spark_explode.py

@@ -100,11 +100,17 @@ def _stamp_executed_fanout(
     Best-effort in the same sense as every other telemetry write: this runs inside a live run, and
     a header that will not take an overlay is a lost record, never a lost run.
     """
+    from ..code_delivery import verify_version_parity
     from ..registry.header import executed_sizing_path, merge_header_telemetry
 
     try:
         path = executed_sizing_path(_fanout_family(executed))
-        merge_header_telemetry(run_id, {path: {"fanout": fanout}}, settings=settings)
+        parity = verify_version_parity()
+        merge_header_telemetry(
+            run_id,
+            {path: {"fanout": fanout}, "version": parity},
+            settings=settings,
+        )
     except Exception as exc:  # noqa: BLE001 - telemetry is an overlay, never fatal
         _log.warning("executed fan-out capture failed (non-fatal): %r", exc)
 

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/gce_submit.py

@@ -325,6 +325,8 @@ def build_gce_startup_script(
         f"STATUS_OBJECT_ENC={shlex.quote(encoded_obj)}\n"
         "LOG_FILE=/tmp/sf_container.log\n"
         ': > "$LOG_FILE"\n'
+        "BOOT_EPOCH=$(date +%s)\n"
+        "EXEC_START_EPOCH=$BOOT_EPOCH\n"
         "\n"
         "get_token() {\n"
         "  curl -sf -H 'Metadata-Flavor: Google' \\\n"
@@ -337,12 +339,17 @@ def build_gce_startup_script(
         '  local st="$1"\n'
         '  local rc="$2"\n'
         '  local detail="$3"\n'
+        "  local end_epoch\n"
+        "  end_epoch=$(date +%s)\n"
+        "  local exec_s=$((end_epoch - EXEC_START_EPOCH))\n"
         '  if [ -n "$STATUS_BUCKET" ]; then\n'
         "    local tok\n"
         "    tok=$(get_token || true)\n"
         '    printf \'{"status":"%s","exit_code":%s,"instance":"%s",'
-        '"zone":"%s","detail":"%s"}\\n\' \\\n'
+        '"zone":"%s","detail":"%s","boot_epoch":%s,"exec_start_epoch":%s,'
+        '"end_epoch":%s,"execute_s":%s}\\n\' \\\n'
         '      "$st" "$rc" "$INSTANCE_NAME" "$ZONE" "$detail" \\\n'
+        '      "$BOOT_EPOCH" "$EXEC_START_EPOCH" "$end_epoch" "$exec_s" \\\n'
         '      | curl -sf -X POST -H "Authorization: Bearer ${tok}" \\\n'
         "        -H 'Content-Type: application/json' --data-binary @- \\\n"
         '        "https://storage.googleapis.com/upload/storage/v1/b/'
@@ -378,6 +385,7 @@ def build_gce_startup_script(
         'echo "$TOKEN" | docker login -u oauth2accesstoken '
         f'--password-stdin {registry_url_q} >>"$LOG_FILE" 2>&1\n'
         f'docker pull {shlex.quote(plan.image_uri)} >>"$LOG_FILE" 2>&1\n'
+        "EXEC_START_EPOCH=$(date +%s)\n"
         f'{docker_cmd} >>"$LOG_FILE" 2>&1\n'
     )
 
@@ -399,12 +407,17 @@ def build_gce_instance_spec_dict(
     3. ``disks[0].autoDelete = True`` so the boot persistent disk never outlives the VM.
     """
     startup_script = build_gce_startup_script(plan, driver_args, project_id=project_id, zone=zone)
+    on_host_maintenance = (
+        "MIGRATE"
+        if plan.accelerator_count == 0 and str(plan.machine_type).startswith("e2-")
+        else "TERMINATE"
+    )
     spec: dict[str, Any] = {
         "name": plan.instance_name,
         "machineType": f"zones/{zone}/machineTypes/{plan.machine_type}",
         "scheduling": {
             "automaticRestart": False,
-            "onHostMaintenance": "TERMINATE",
+            "onHostMaintenance": on_host_maintenance,
             "maxRunDuration": {"seconds": str(plan.max_run_duration_seconds)},
             "instanceTerminationAction": "DELETE",
         },
@@ -909,6 +922,7 @@ def submit_gce(
 
     if wait:
         assert chosen_zone is not None
+        t_poll_start = time.monotonic()
         try:
             marker = _poll_gce_instance(
                 session,
@@ -920,6 +934,34 @@ def submit_gce(
                 grace_s=int(infra.stall_grace_seconds),
                 since=since,
             )
+            if effective_job_id is not None:
+                from .batch_telemetry import platform_phases
+
+                phases = platform_phases(
+                    marker.get("create_epoch") or since or marker.get("boot_epoch"),
+                    marker.get("exec_start_epoch") or marker.get("boot_epoch"),
+                    marker.get("end_epoch"),
+                )
+                if phases.get("execute_s") is None and isinstance(
+                    marker.get("execute_s"), int | float
+                ):
+                    phases["execute_s"] = round(float(marker["execute_s"]), 1)
+                if phases.get("total_wall_s") is None:
+                    phases["total_wall_s"] = round(max(0.0, time.monotonic() - t_poll_start), 1)
+                exec_val = phases.get("execute_s")
+                wall_val = phases.get("total_wall_s")
+                if (
+                    phases.get("provision_s") is None
+                    and exec_val is not None
+                    and wall_val is not None
+                ):
+                    phases["provision_s"] = round(max(0.0, wall_val - exec_val), 1)
+                with contextlib.suppress(Exception):
+                    update_job(
+                        effective_job_id,
+                        settings=settings,
+                        merge_telemetry={"platform": phases},
+                    )
             if marker.get("status") != "SUCCEEDED":
                 detail = marker.get("detail") or f"exit_code={marker.get('exit_code')}"
                 raise EngineError(

{scale_forecasting-1.2.0 → scale_forecasting-1.2.1}/src/scale_forecasting/gke_submit.py

@@ -1292,6 +1292,8 @@ def submit_gke(
     created_ephemeral_cluster = False
     landed_location = cluster_location or settings.region
     cluster_obj: dict[str, Any] | None = None
+    t_submit_start = time.monotonic()
+    t_job_start: float | None = None
 
     if not plan.ephemeral_cluster:
         cluster_obj = get_gke_cluster(
@@ -1429,6 +1431,7 @@ def submit_gke(
             if not wait:
                 return run_id, plan.job_name, handle
 
+            t_job_start = time.monotonic()
             timeout_s = (
                 wait_timeout_seconds
                 if wait_timeout_seconds is not None
@@ -1444,6 +1447,29 @@ def submit_gke(
             while time.monotonic() < deadline:
                 cur_job = k8s.get_job(plan.namespace, plan.job_name)
                 state, detail = k8s_job_state(cur_job)
+                if state in {"SUCCEEDED", "FAILED"} and job_id:
+                    from .batch_telemetry import platform_phases
+
+                    t_end = time.monotonic()
+                    meta = (cur_job or {}).get("metadata") or {}
+                    st = (cur_job or {}).get("status") or {}
+                    phases = platform_phases(
+                        (cluster_obj or {}).get("createTime") or meta.get("creationTimestamp"),
+                        st.get("startTime") or meta.get("creationTimestamp"),
+                        st.get("completionTime"),
+                    )
+                    if phases.get("provision_s") is None and t_job_start is not None:
+                        phases["provision_s"] = round(max(0.0, t_job_start - t_submit_start), 1)
+                    if phases.get("execute_s") is None and t_job_start is not None:
+                        phases["execute_s"] = round(max(0.0, t_end - t_job_start), 1)
+                    if phases.get("total_wall_s") is None:
+                        phases["total_wall_s"] = round(max(0.0, t_end - t_submit_start), 1)
+                    with contextlib.suppress(Exception):
+                        update_job(
+                            job_id,
+                            settings=settings,
+                            merge_telemetry={"platform": phases},
+                        )
                 if state == "SUCCEEDED":
                     _log.info(
                         "gke job %s SUCCEEDED on cluster %s (%s)",