gen3-dataops-toolkit 3.3.0__tar.gz → 3.5.0__tar.gz

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gen3-dataops-toolkit
-Version: 3.3.0
+Version: 3.5.0
 Summary: Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments
 License: Apache-2.0
 Author: JoshuaHarris391
@@ -182,6 +182,47 @@ Synthetic data is only schema-valid against the dictionary that generated it, so
 refuses a batch that doesn't match the version being uploaded (override with
 `--allow-version-mismatch`).
 
+### Synthetic data: LLM configuration
+
+`synth generate --llm` and `synth deploy` generate LLM-realistic values with
+[gen3-metadata-simulator](https://github.com/AustralianBioCommons/gen3-metadata-simulator).
+The provider and model resolve with precedence **CLI flags > SSM > default**:
+the CDK config's optional `llm` block publishes `app/llm_provider` /
+`app/llm_model` to SSM, so every operator gets the deployment's values, and
+`--llm-provider` / `--llm-model` override them for one run (e.g. to try a
+model before adding it to the CDK config). Environments deployed without the
+block fall back to provider `anthropic`, and the `--llm` path errors with
+guidance when no model is configured anywhere.
+
+Only the API key stays local — as a *path* to the file holding it, never the
+key itself, set once per operator:
+
+```bash
+g3dt config set llm_api_key_file ~/.g3dt/anthropic_api_key
+g3dt synth generate AusDiab_Simulated --llm -n 5 -e test
+```
+
+(or per run with `--llm-api-key-file`; the vendor env var `ANTHROPIC_API_KEY`
+/ `OPENAI_API_KEY` also works as a fallback.) The old `~/.g3dt/.env`
+(`LLM_PROVIDER`/`LLM_MODEL`/`LLM_API_KEY_FILE`) is **no longer read**.
+`g3dt config show --env <env>` prints the resolved provider, model, and key
+path.
+
+### Kubernetes restart targets
+
+`k8s restart-schema`, `k8s restart-ms`, `dict deploy`, and `synth deploy`
+restart the commons' schema microservices **serially, in a configured order**,
+waiting for each rollout to go Healthy before the next; `k8s restart-etl` and
+the deploy flows run a named ETL cronjob. Both targets resolve with precedence
+**CLI flags > SSM > default**: the CDK config's optional `k8s` block publishes
+`app/restart_services` (comma-separated deployment names) and
+`app/etl_cronjob`, `--restart-services` / `--etl-cronjob` override them for
+one run, and environments deployed without the block keep the classic Gen3 set
+(`sheepdog-deployment,peregrine-deployment,guppy-deployment,portal-deployment`
+/ `etl-cronjob`). A commons that manages some service outside this flow (e.g.
+a manually redeployed frontend) simply omits it from the list in its wrapper
+config. `g3dt config show --env <env>` prints the resolved values.
+
 ## Verifying download access (check-download)
 
 Registration alone does not prove a file can be downloaded. Two failure modes

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/README.md

@@ -148,6 +148,47 @@ Synthetic data is only schema-valid against the dictionary that generated it, so
 refuses a batch that doesn't match the version being uploaded (override with
 `--allow-version-mismatch`).
 
+### Synthetic data: LLM configuration
+
+`synth generate --llm` and `synth deploy` generate LLM-realistic values with
+[gen3-metadata-simulator](https://github.com/AustralianBioCommons/gen3-metadata-simulator).
+The provider and model resolve with precedence **CLI flags > SSM > default**:
+the CDK config's optional `llm` block publishes `app/llm_provider` /
+`app/llm_model` to SSM, so every operator gets the deployment's values, and
+`--llm-provider` / `--llm-model` override them for one run (e.g. to try a
+model before adding it to the CDK config). Environments deployed without the
+block fall back to provider `anthropic`, and the `--llm` path errors with
+guidance when no model is configured anywhere.
+
+Only the API key stays local — as a *path* to the file holding it, never the
+key itself, set once per operator:
+
+```bash
+g3dt config set llm_api_key_file ~/.g3dt/anthropic_api_key
+g3dt synth generate AusDiab_Simulated --llm -n 5 -e test
+```
+
+(or per run with `--llm-api-key-file`; the vendor env var `ANTHROPIC_API_KEY`
+/ `OPENAI_API_KEY` also works as a fallback.) The old `~/.g3dt/.env`
+(`LLM_PROVIDER`/`LLM_MODEL`/`LLM_API_KEY_FILE`) is **no longer read**.
+`g3dt config show --env <env>` prints the resolved provider, model, and key
+path.
+
+### Kubernetes restart targets
+
+`k8s restart-schema`, `k8s restart-ms`, `dict deploy`, and `synth deploy`
+restart the commons' schema microservices **serially, in a configured order**,
+waiting for each rollout to go Healthy before the next; `k8s restart-etl` and
+the deploy flows run a named ETL cronjob. Both targets resolve with precedence
+**CLI flags > SSM > default**: the CDK config's optional `k8s` block publishes
+`app/restart_services` (comma-separated deployment names) and
+`app/etl_cronjob`, `--restart-services` / `--etl-cronjob` override them for
+one run, and environments deployed without the block keep the classic Gen3 set
+(`sheepdog-deployment,peregrine-deployment,guppy-deployment,portal-deployment`
+/ `etl-cronjob`). A commons that manages some service outside this flow (e.g.
+a manually redeployed frontend) simply omits it from the list in its wrapper
+config. `g3dt config show --env <env>` prints the resolved values.
+
 ## Verifying download access (check-download)
 
 Registration alone does not prove a file can be downloaded. Two failure modes

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "gen3-dataops-toolkit"
-version = "3.3.0"
+version = "3.5.0"
 description = "Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments"
 authors = ["JoshuaHarris391 <harjo391@gmail.com>"]
 readme = "README.md"

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/cli/config_cmds.py

@@ -118,6 +118,15 @@ def show(
     typer.echo(f"  namespace          : {e.namespace}")
     typer.echo(f"  cluster_name       : {e.cluster_name}")
     typer.echo(f"  ec2_instance_id    : {e.ec2_instance_id}")
+    # Synthetic-data LLM facts: provider/model from SSM (the CDK's optional
+    # llm block); only the key *path* is local, from the marker.
+    typer.echo(f"  llm_provider       : {e.llm_provider}")
+    typer.echo(f"  llm_model          : {e.llm_model or '(not set — pass --llm-model or add the llm block to the CDK config)'}")
+    typer.echo(f"  llm_api_key_file   : {config.llm_api_key_file() or '(not set — g3dt config set llm_api_key_file <path>)'}")
+    # k8s restart targets: from SSM (the CDK's optional k8s block), restarted
+    # in the listed order by restart-schema/restart-ms/dict deploy/synth deploy.
+    typer.echo(f"  restart_services   : {e.restart_services}")
+    typer.echo(f"  etl_cronjob        : {e.etl_cronjob}")
     if study:
         s = study_of(study, env)
         typer.secho(f"Study: {study} -> {s.key}", bold=True)
@@ -199,6 +208,30 @@ def diff(
         check(f"gen3.{camel}", gen3.get(camel), rc.get(f"app/{leaf}"))
     check("toolkitVersion", inputs.get("toolkitVersion"), rc.get("meta/toolkitVersion"))
 
+    # Optional inputs: compared only when the file defines them — an absent
+    # input legitimately publishes no SSM parameter (or, for the dictionary
+    # fields, predates their addition), so absence on both sides is not drift.
+    for camel, leaf in {
+        "dictionaryBaseUrl": "dictionary_base_url",
+        "dictionaryPath": "dictionary_path",
+    }.items():
+        if camel in gen3:
+            check(f"gen3.{camel}", gen3.get(camel), rc.get(f"app/{leaf}"))
+    llm = inputs.get("llm") or {}
+    for camel, leaf in {"provider": "llm_provider", "model": "llm_model"}.items():
+        if camel in llm:
+            check(f"llm.{camel}", llm.get(camel), rc.get(f"app/{leaf}"))
+    k8s = inputs.get("k8s") or {}
+    if "schemaRestartServices" in k8s:
+        # The CDK publishes the list comma-joined; compare the same shape.
+        check(
+            "k8s.schemaRestartServices",
+            ",".join(k8s.get("schemaRestartServices") or []),
+            rc.get("app/restart_services"),
+        )
+    if "etlCronjob" in k8s:
+        check("k8s.etlCronjob", k8s.get("etlCronjob"), rc.get("app/etl_cronjob"))
+
     if not drift:
         typer.secho(
             f"No drift: SSM /{project}/{config.env_base(env)} matches {file}.",

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/cli/dict_cmds.py

@@ -103,6 +103,11 @@ def deploy(
     version: str = typer.Option(
         None, "--version", help="Dictionary git tag (default: the env's version)."
     ),
+    restart_services: Optional[str] = typer.Option(
+        None, "--restart-services",
+        help="Comma-separated deployment names restarted after the upload, in "
+        "order; default: the env's SSM app/restart_services.",
+    ),
 ) -> None:
     """Pull + upload the dictionary and restart Gen3 schema microservices.
 
@@ -123,7 +128,10 @@ def deploy(
     """
     e = env_of(env)
     warn_if_overridden(e, version)
+    env_vars = script_env(e, _version(e, version))
+    if restart_services:
+        env_vars["G3DT_RESTART_SERVICES"] = restart_services
     runner.run(
         runner.bash_script("services/dictionary/deploy_dd.sh", env),
-        env=script_env(e, _version(e, version)),
+        env=env_vars,
     )

gen3_dataops_toolkit-3.5.0/src/g3dt/cli/k8s.py

@@ -0,0 +1,104 @@
+"""`g3dt k8s` — restart Gen3 microservices / ETL via ArgoCD (LOCAL only).
+
+These use ``argocd login --sso`` (a browser flow), so they cannot run headless
+on EC2. The wrapped scripts receive their settings as ``G3DT_*`` environment
+variables resolved from SSM — they read no config files.
+
+The restart targets resolve with precedence CLI flags > SSM > default: the
+CDK config's optional ``k8s`` block publishes ``app/restart_services`` (a
+comma-separated list, restarted in order) and ``app/etl_cronjob``;
+``--restart-services`` / ``--etl-cronjob`` override them for one run, and
+environments deployed without the block keep the classic Gen3 set
+(sheepdog, peregrine, guppy, portal / etl-cronjob).
+"""
+from __future__ import annotations
+
+from typing import Optional
+
+import typer
+
+from g3dt.config import script_env
+from g3dt.cli._internal import runner
+from g3dt.cli._internal.resolve import env_of
+
+app = typer.Typer(no_args_is_help=True, help="ArgoCD / Kubernetes restarts (local).")
+
+_SCHEMA = "services/k8s_ops/argocd_restart_schema.sh"
+_ETL = "services/k8s_ops/argocd_restart_etl.sh"
+_ETL_AND_MS = "services/k8s_ops/restart_etl_and_ms.sh"
+
+_RESTART_SERVICES_HELP = (
+    "Comma-separated deployment names to restart, in order; default: the "
+    "env's SSM app/restart_services (the CDK config's k8s.schemaRestartServices)."
+)
+_ETL_CRONJOB_HELP = (
+    "ETL cronjob name; default: the env's SSM app/etl_cronjob "
+    "(the CDK config's k8s.etlCronjob)."
+)
+
+
+def restart_env(e, restart_services: Optional[str] = None,
+                etl_cronjob: Optional[str] = None) -> dict:
+    """script_env plus per-run restart-target overrides (flags beat SSM)."""
+    env_vars = script_env(e)
+    if restart_services:
+        env_vars["G3DT_RESTART_SERVICES"] = restart_services
+    if etl_cronjob:
+        env_vars["G3DT_ETL_CRONJOB"] = etl_cronjob
+    return env_vars
+
+
+@app.command(name="restart-schema")
+def restart_schema(
+    env: str = typer.Option(..., "--env", "-e", help="Environment, e.g. test."),
+    sync: bool = typer.Option(False, "--sync", "-s", help="argocd app sync first."),
+    restart_services: Optional[str] = typer.Option(
+        None, "--restart-services", help=_RESTART_SERVICES_HELP
+    ),
+) -> None:
+    """Restart the schema microservices, in the env's configured order."""
+    e = env_of(env)
+    args = ["-d", e.domain, "-a", e.app_name, "-n", e.namespace]
+    if sync:
+        args.append("-s")
+    runner.run(
+        runner.bash_script(_SCHEMA, *args),
+        env=restart_env(e, restart_services=restart_services),
+    )
+
+
+@app.command(name="restart-etl")
+def restart_etl(
+    env: str = typer.Option(..., "--env", "-e", help="Environment, e.g. test."),
+    sync: bool = typer.Option(False, "--sync", "-s", help="argocd app sync first."),
+    etl_cronjob: Optional[str] = typer.Option(
+        None, "--etl-cronjob", help=_ETL_CRONJOB_HELP
+    ),
+) -> None:
+    """Create + run the ETL cronjob and wait for completion."""
+    e = env_of(env)
+    args = ["-e", env]
+    if sync:
+        args.append("-s")
+    runner.run(
+        runner.bash_script(_ETL, *args),
+        env=restart_env(e, etl_cronjob=etl_cronjob),
+    )
+
+
+@app.command(name="restart-ms")
+def restart_ms(
+    env: str = typer.Option(..., "--env", "-e", help="Environment, e.g. test."),
+    restart_services: Optional[str] = typer.Option(
+        None, "--restart-services", help=_RESTART_SERVICES_HELP
+    ),
+    etl_cronjob: Optional[str] = typer.Option(
+        None, "--etl-cronjob", help=_ETL_CRONJOB_HELP
+    ),
+) -> None:
+    """Restart both ETL and schema microservices (wraps restart_etl_and_ms.sh)."""
+    e = env_of(env)
+    runner.run(
+        runner.bash_script(_ETL_AND_MS, env),
+        env=restart_env(e, restart_services=restart_services, etl_cronjob=etl_cronjob),
+    )

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/cli/synth.py

@@ -9,9 +9,14 @@ Every command accepts ``--env``; targeting a **production** environment (any env
 whose name contains ``prod``) shows a warning and requires typing the env name to
 confirm — it cannot be bypassed.
 
-Generation defaults to keyless ``random`` data (no API calls). Pass ``--llm`` for
-LLM-realistic values, which needs an API key configured in a ``.env`` in the
-working directory (``LLM_PROVIDER`` / ``LLM_MODEL`` / ``LLM_API_KEY_FILE``).
+Generation defaults to keyless ``random`` data (no API calls). Pass ``--llm``
+for LLM-realistic values. The LLM provider and model come from the
+environment's SSM tree (the CDK config's optional ``llm`` block, published as
+``app/llm_provider`` / ``app/llm_model``) and can be overridden per run with
+``--llm-provider`` / ``--llm-model``. Only the API key stays local: point at
+its file once with ``g3dt config set llm_api_key_file <path>`` (or per run
+with ``--llm-api-key-file``); the vendor env var ``ANTHROPIC_API_KEY`` /
+``OPENAI_API_KEY`` also works. The old ``~/.g3dt/.env`` is no longer read.
 """
 from __future__ import annotations
 
@@ -21,6 +26,7 @@ from typing import Optional
 
 import typer
 
+from g3dt import config
 from g3dt.config import (
     dictionary_filename,
     dictionary_url,
@@ -38,6 +44,45 @@ app = typer.Typer(
 
 SYNTH_DIR = Path("~/.g3dt/synth_metadata").expanduser()
 
+
+def _llm_env_overrides(
+    e,
+    llm_provider: Optional[str],
+    llm_model: Optional[str],
+    llm_api_key_file: Optional[Path],
+) -> dict:
+    """Resolve the effective LLM settings for a run: flags > SSM > default.
+
+    Returns env-var overrides for the generator script, which forwards
+    provider/model to gen3-metadata-simulator as CLI flags (the simulator's
+    own precedence puts flags first). Exits with guidance when no model is
+    configured anywhere; the key-file path is optional — the simulator falls
+    back to the vendor env var and raises its own error if neither exists.
+    """
+    effective_provider = llm_provider or e.llm_provider
+    effective_model = llm_model or e.llm_model
+    if not effective_model:
+        typer.secho(
+            "No LLM model configured. Set the llm block in the CDK config "
+            "(published to SSM as app/llm_model) and redeploy, or pass "
+            "--llm-model for this run.",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(1)
+    overrides = {
+        "G3DT_LLM_PROVIDER": effective_provider,
+        "G3DT_LLM_MODEL": effective_model,
+    }
+    key_file = (
+        str(Path(llm_api_key_file).expanduser())
+        if llm_api_key_file
+        else config.llm_api_key_file()
+    )
+    if key_file:
+        overrides["LLM_API_KEY_FILE"] = key_file
+    return overrides
+
 #: Written into each generated batch so `synth upload` can tell which dictionary
 #: produced it. A batch is only valid against that dictionary, and the directory
 #: name alone cannot be trusted: --schema and --version are separate options, so
@@ -110,19 +155,49 @@ def deploy(
     env: str = typer.Option(
         "test", "--env", "-e", help="Target environment (prod requires typed confirmation)."
     ),
+    llm_provider: Optional[str] = typer.Option(
+        None, "--llm-provider",
+        help="LLM vendor override (anthropic|openai); default: the env's SSM app/llm_provider.",
+    ),
+    llm_model: Optional[str] = typer.Option(
+        None, "--llm-model",
+        help="LLM model override; default: the env's SSM app/llm_model.",
+    ),
+    llm_api_key_file: Optional[Path] = typer.Option(
+        None, "--llm-api-key-file", exists=True, dir_okay=False,
+        help="Path to the file holding the LLM API key; default: the marker's "
+        "llm_api_key_file (set once: g3dt config set llm_api_key_file <path>).",
+    ),
+    restart_services: Optional[str] = typer.Option(
+        None, "--restart-services",
+        help="Comma-separated deployment names restarted during the deploy, in "
+        "order; default: the env's SSM app/restart_services.",
+    ),
+    etl_cronjob: Optional[str] = typer.Option(
+        None, "--etl-cronjob",
+        help="ETL cronjob name; default: the env's SSM app/etl_cronjob.",
+    ),
 ) -> None:
     """Full end-to-end synthetic deploy (dict + LLM-generate + upload + restarts).
 
     Wraps services/synthetic_data/full_deploy_dd_and_synth.sh (LLM-backed
-    generation). Requires an LLM key configured in .env.
+    generation). Provider/model and the restart targets come from the env's
+    SSM tree unless overridden; the API key path comes from
+    --llm-api-key-file or the marker.
     """
     e = env_of(env)
     safety.confirm_prod_strict("synthetic full deploy", env)
+    env_vars = script_env(e)
+    env_vars.update(_llm_env_overrides(e, llm_provider, llm_model, llm_api_key_file))
+    if restart_services:
+        env_vars["G3DT_RESTART_SERVICES"] = restart_services
+    if etl_cronjob:
+        env_vars["G3DT_ETL_CRONJOB"] = etl_cronjob
     runner.run(
         runner.bash_script(
             "services/synthetic_data/full_deploy_dd_and_synth.sh", env
         ),
-        env=script_env(e),
+        env=env_vars,
     )
 
 
@@ -147,8 +222,22 @@ def generate(
     llm: bool = typer.Option(
         False,
         "--llm",
-        help="Generate LLM-realistic values; reads LLM config from a .env in the "
-        "working directory. Default is keyless random data (no API key, no API calls).",
+        help="Generate LLM-realistic values; provider/model resolve from the "
+        "env's SSM tree (override with --llm-provider/--llm-model). Default is "
+        "keyless random data (no API key, no API calls).",
+    ),
+    llm_provider: Optional[str] = typer.Option(
+        None, "--llm-provider",
+        help="LLM vendor override (anthropic|openai); default: the env's SSM app/llm_provider.",
+    ),
+    llm_model: Optional[str] = typer.Option(
+        None, "--llm-model",
+        help="LLM model override; default: the env's SSM app/llm_model.",
+    ),
+    llm_api_key_file: Optional[Path] = typer.Option(
+        None, "--llm-api-key-file", exists=True, dir_okay=False,
+        help="Path to the file holding the LLM API key; default: the marker's "
+        "llm_api_key_file (set once: g3dt config set llm_api_key_file <path>).",
     ),
     seed: int = typer.Option(None, "--seed", help="RNG seed for reproducible output."),
     schema: str = typer.Option(
@@ -227,11 +316,16 @@ def generate(
         args += ["--num-records", num_records]
     if seed is not None:
         args += ["--seed", str(seed)]
+    env_vars = script_env(e, ver)
+    if effective_provider == "llm":
+        env_vars.update(
+            _llm_env_overrides(e, llm_provider, llm_model, llm_api_key_file)
+        )
     runner.run(
         runner.bash_script(
             "services/synthetic_data/generate_synth_metadata.sh", *args
         ),
-        env=script_env(e, ver),
+        env=env_vars,
     )
     # Only after a successful generate: runner.run raises on failure, so a
     # half-written batch is never stamped as valid.

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/config.py

@@ -50,11 +50,32 @@ DEFAULT_REGION = "ap-southeast-2"
 DEFAULT_DICT_BASE_URL = "https://raw.githubusercontent.com"
 DEFAULT_DICT_PATH = "dictionary/prod_dict/acdc_schema.json"
 
+#: Synthetic-data LLM facts, published by the CDK's OPTIONAL ``llm`` config
+#: block as ``app/llm_provider`` / ``app/llm_model`` and consumed by
+#: gen3-metadata-simulator through ``g3dt synth``. Optional app inputs:
+#: environments deployed without the block fall back to this provider default.
+#: The model deliberately has no default — ``g3dt synth`` errors with guidance
+#: when the ``--llm`` path is used and no model is configured anywhere.
+DEFAULT_LLM_PROVIDER = "anthropic"
+
+#: Kubernetes restart targets, published by the CDK's OPTIONAL ``k8s`` config
+#: block as ``app/restart_services`` (comma-separated, restarted in order) and
+#: ``app/etl_cronjob``. Optional app inputs: environments deployed without the
+#: block keep the classic Gen3 set. Consumed by every restart path — `g3dt
+#: k8s restart-schema/restart-etl/restart-ms`, `dict deploy`, `synth deploy` —
+#: via $G3DT_RESTART_SERVICES / $G3DT_ETL_CRONJOB in the service scripts.
+DEFAULT_RESTART_SERVICES = (
+    "sheepdog-deployment,peregrine-deployment,guppy-deployment,portal-deployment"
+)
+DEFAULT_ETL_CRONJOB = "etl-cronjob"
+
 #: Marker locations, most specific first.
 MARKER_PATHS = ("g3dt.yaml", "~/.g3dt/g3dt.yaml", "/etc/g3dt/g3dt.yaml")
 
-#: Marker keys `g3dt config set` may write.
-SETTABLE_MARKER_KEYS = ("project", "region", "default_env")
+#: Marker keys `g3dt config set` may write. ``llm_api_key_file`` is the path
+#: to the file holding the synth LLM API key — the one LLM setting that stays
+#: local (the provider and model come from SSM; the key never leaves the box).
+SETTABLE_MARKER_KEYS = ("project", "region", "default_env", "llm_api_key_file")
 
 #: Gen3 app facts mirrored to SSM /{project}/{env}/app/* by the CDK.
 REQUIRED_APP_KEYS = (
@@ -150,6 +171,18 @@ def require_project(marker: Optional[dict] = None) -> str:
     return project
 
 
+def llm_api_key_file(marker: Optional[dict] = None) -> Optional[str]:
+    """Path to the file holding the synth LLM API key, from the marker.
+
+    Set once per operator with ``g3dt config set llm_api_key_file <path>``.
+    Returns ``None`` when unset — gen3-metadata-simulator then falls back to
+    the vendor's standard env var (``ANTHROPIC_API_KEY`` / ``OPENAI_API_KEY``).
+    """
+    m = marker if marker is not None else load_marker()
+    value = m.get("llm_api_key_file")
+    return str(Path(str(value)).expanduser()) if value else None
+
+
 def set_marker_value(key: str, value: str) -> Tuple[Optional[str], str, Path]:
     """Set one bootstrap key in the user's marker file and write it back.
 
@@ -230,6 +263,14 @@ class EnvConfig:
     # hand-built EnvConfig still composes a valid URL.
     dictionary_base_url: str = DEFAULT_DICT_BASE_URL
     dictionary_path: str = DEFAULT_DICT_PATH
+    # Optional synthetic-data LLM inputs (SSM app/llm_provider, app/llm_model).
+    # The model has no default on purpose: synth's --llm path checks and errors
+    # with guidance rather than silently picking a model.
+    llm_provider: str = DEFAULT_LLM_PROVIDER
+    llm_model: Optional[str] = None
+    # Optional k8s restart targets (SSM app/restart_services, app/etl_cronjob).
+    restart_services: str = DEFAULT_RESTART_SERVICES
+    etl_cronjob: str = DEFAULT_ETL_CRONJOB
 
 
 def _app_or_default(rc, leaf: str, default: str) -> str:
@@ -289,6 +330,14 @@ def resolve_env(env: str, project: Optional[str] = None) -> EnvConfig:
             rc, "dictionary_base_url", DEFAULT_DICT_BASE_URL
         ),
         dictionary_path=_app_or_default(rc, "dictionary_path", DEFAULT_DICT_PATH),
+        # Same optional-app-fact contract: the CDK publishes these only when
+        # the config has an llm block, so absence means "use the defaults".
+        llm_provider=_app_or_default(rc, "llm_provider", DEFAULT_LLM_PROVIDER),
+        llm_model=(_app_or_default(rc, "llm_model", "") or None),
+        restart_services=_app_or_default(
+            rc, "restart_services", DEFAULT_RESTART_SERVICES
+        ),
+        etl_cronjob=_app_or_default(rc, "etl_cronjob", DEFAULT_ETL_CRONJOB),
     )
 
 
@@ -395,6 +444,10 @@ def script_env(e: EnvConfig, version: Optional[str] = None) -> Dict[str, str]:
         "G3DT_NAMESPACE": e.namespace,
         "G3DT_CLUSTER_NAME": e.cluster_name,
         "G3DT_SCHEMA_REPO": e.schema_repo,
+        "G3DT_LLM_PROVIDER": e.llm_provider,
+        "G3DT_LLM_MODEL": e.llm_model,
+        "G3DT_RESTART_SERVICES": e.restart_services,
+        "G3DT_ETL_CRONJOB": e.etl_cronjob,
     }
     env.update({k: v for k, v in values.items() if v is not None})
     return env

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/services/k8s_ops/argocd_restart_etl.sh

@@ -6,7 +6,7 @@ usage() {
     echo "  -d DOMAIN      The domain for argocd login (example: cd.cad.test.biocommons.org.au)"
     echo "  -a APPNAME     The application name (example: uatgen3)"
     echo "  -n NAMESPACE   The namespace for the resources (example: cad)"
-    echo "  -c ETL_CRONJOB The name of the ETL cronjob to run (default: etl-cronjob)"
+    echo "  -c ETL_CRONJOB The name of the ETL cronjob to run (default: \$G3DT_ETL_CRONJOB — the env's SSM app/etl_cronjob, set by g3dt — else etl-cronjob)"
     echo "  -t CONTAINER   The name of the container to check logs from (default: tube)"
     echo "  -l             Bypass login"
     echo "  -s             Sync the argocd app before restarting resources"
@@ -15,8 +15,9 @@ usage() {
 
 set -eo pipefail
 
-# default values
-ETL_CRONJOB="etl-cronjob"
+# default values. The cronjob name comes from the environment's SSM tree
+# (app/etl_cronjob, exported by g3dt as G3DT_ETL_CRONJOB).
+ETL_CRONJOB="${G3DT_ETL_CRONJOB:-etl-cronjob}"
 CONTAINER_TO_CHECK="tube"
 LOGIN_REQUIRED=true
 SYNC_APP=false

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/services/k8s_ops/argocd_restart_ms.sh

@@ -6,8 +6,11 @@ usage() {
     echo "Usage: $0 [-d DOMAIN] [-a APPNAME] [-r RESOURCES] [-n NAMESPACE] [-k KIND] [-l] [-s]"
     echo "  -d DOMAIN      The domain for argocd login (example: cd.cad.test.biocommons.org.au)"
     echo "  -a APPNAME     The application name (example: uatgen3)"
-    echo "  -r RESOURCES   Comma-separated string of microservice names to restart (example: sheepdog-deployment,peregrine-deployment )"
-    echo "  -n NAMESPACE   The namespace for the resources (example: cad)"
+    echo "  -r RESOURCES   Comma-separated microservice names, restarted in order"
+    echo "                 (default: \$G3DT_RESTART_SERVICES — the env's SSM"
+    echo "                 app/restart_services, set by g3dt — else the classic"
+    echo "                 sheepdog,peregrine,guppy,portal set)"
+    echo "  -n NAMESPACE   The namespace for the resources (default: \$G3DT_NAMESPACE)"
     echo "  -k KIND        The kind of resource to restart (default: Deployment)"
     echo "  -l             Bypass login"
     echo "  -s             Sync the argocd app before restarting resources"
@@ -16,7 +19,10 @@ usage() {
 
 set -eo pipefail
 
-# Parse command line arguments
+# Parse command line arguments. The restart set defaults to the environment's
+# SSM app/restart_services (exported by g3dt as G3DT_RESTART_SERVICES).
+IFS=',' read -r -a RESOURCES <<< "${G3DT_RESTART_SERVICES:-sheepdog-deployment,peregrine-deployment,guppy-deployment,portal-deployment}"
+NAMESPACE="${G3DT_NAMESPACE:-}"
 LOGIN_REQUIRED=true
 KIND="Deployment"
 SYNC_APP=false
@@ -52,6 +58,11 @@ while getopts "d:a:r:n:k:hls" opt; do
     esac
 done
 
+if [ -z "$NAMESPACE" ]; then
+    echo "No namespace given: pass -n, or run via the g3dt CLI (which sets G3DT_NAMESPACE)." >&2
+    exit 1
+fi
+
 # Check if argocd is installed
 if ! command -v argocd &> /dev/null
 then

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/services/k8s_ops/argocd_restart_schema.sh

@@ -6,8 +6,11 @@ usage() {
     echo "Usage: $0 [-d DOMAIN] [-a APPNAME] [-r RESOURCES] [-n NAMESPACE] [-k KIND] [-l] [-s]"
     echo "  -d DOMAIN      The domain for argocd login (example: cd.cad.test.biocommons.org.au)"
     echo "  -a APPNAME     The application name (example: uatgen3)"
-    echo "  -r RESOURCES   Comma-separated string of microservice names to restart (default: \"sheepdog-deployment\" \"peregrine-deployment\" \"guppy-deployment\" \"portal-deployment\")"
-    echo "  -n NAMESPACE   The namespace for the resources (default: cad)"
+    echo "  -r RESOURCES   Comma-separated microservice names, restarted in order"
+    echo "                 (default: \$G3DT_RESTART_SERVICES — the env's SSM"
+    echo "                 app/restart_services, set by g3dt — else the classic"
+    echo "                 sheepdog,peregrine,guppy,portal set)"
+    echo "  -n NAMESPACE   The namespace for the resources (default: \$G3DT_NAMESPACE)"
     echo "  -k KIND        The kind of resource to restart (default: Deployment)"
     echo "  -l             Bypass login"
     echo "  -s             Run 'argocd app sync' before restarts"
@@ -16,9 +19,11 @@ usage() {
 
 set -eo pipefail
 
-# Set default values
-RESOURCES=("sheepdog-deployment" "peregrine-deployment" "guppy-deployment" "portal-deployment")
-NAMESPACE="cad"
+# Set default values. The restart set comes from the environment's SSM tree
+# (app/restart_services, exported by g3dt as G3DT_RESTART_SERVICES); the
+# classic Gen3 set is the fallback for direct invocations outside g3dt.
+IFS=',' read -r -a RESOURCES <<< "${G3DT_RESTART_SERVICES:-sheepdog-deployment,peregrine-deployment,guppy-deployment,portal-deployment}"
+NAMESPACE="${G3DT_NAMESPACE:-}"
 KIND="Deployment"
 LOGIN_REQUIRED=true
 SYNC_APP=false
@@ -56,6 +61,11 @@ while getopts "d:a:r:n:k:hls" opt; do
     esac
 done
 
+if [ -z "$NAMESPACE" ]; then
+    echo "No namespace given: pass -n, or run via the g3dt CLI (which sets G3DT_NAMESPACE)." >&2
+    exit 1
+fi
+
 # Check if argocd is installed
 if ! command -v argocd &> /dev/null
 then

{gen3_dataops_toolkit-3.3.0 → gen3_dataops_toolkit-3.5.0}/src/g3dt/services/synthetic_data/generate_synth_metadata.sh

@@ -8,14 +8,13 @@
 # The tool takes a LOCAL bundled Gen3 schema file (pulled by pull_dict.sh into
 # ~/.g3dt/schemas/acdc_schema_<version>.json, or $G3DT_SCHEMA_DIR if set). The
 # default provider is keyless 'random'; pass --provider llm for LLM-realistic
-# values, in which case LLM config is read from ~/.g3dt/.env (or $G3DT_ENV_FILE
-# if set): LLM_PROVIDER / LLM_MODEL / LLM_API_KEY_FILE.
+# values. The LLM vendor/model arrive as $G3DT_LLM_PROVIDER / $G3DT_LLM_MODEL
+# (resolved by g3dt: CLI flags > SSM app/llm_* > default) and are forwarded to
+# the simulator as flags; the API key file path arrives as $LLM_API_KEY_FILE.
+# The old ~/.g3dt/.env is no longer read.
 
 set -euo pipefail
 
-# LLM provider config file (lives outside the installed package).
-ENV_FILE="${G3DT_ENV_FILE:-$HOME/.g3dt/.env}"
-
 usage() {
     cat <<EOF
 Usage: $(basename "$0") --schema <path> --version <ver> [options]
@@ -32,8 +31,8 @@ Options:
   --num-records N|n1,n2  Records per study: one number for all, or a comma list
                          (one per study). Default: ${DEFAULT_NUM_RECORDS}
   --provider random|llm  Value strategy. Default: ${DEFAULT_PROVIDER}
-                         'random' needs no key; 'llm' reads LLM config from
-                         ${ENV_FILE}.
+                         'random' needs no key; 'llm' uses \$G3DT_LLM_PROVIDER /
+                         \$G3DT_LLM_MODEL / \$LLM_API_KEY_FILE (set by g3dt).
   --seed N               RNG seed for reproducible output.
   --output-root DIR      Root output dir. Default: ${DEFAULT_OUTPUT_ROOT}
   -h, --help             Show this help and exit.
@@ -123,9 +122,16 @@ for i in "${!STUDY_ARRAY[@]}"; do
          --num-records "$N"
          --provider "$PROVIDER")
     [[ -n "$SEED" ]] && CMD+=(--seed "$SEED")
-    # Point the LLM provider at the user-level env file regardless of the caller's CWD.
-    if [[ "$PROVIDER" == "llm" && -f "${ENV_FILE}" ]]; then
-        CMD+=(--env-file "${ENV_FILE}")
+    if [[ "$PROVIDER" == "llm" ]]; then
+        # Vendor/model resolved by g3dt (flags > SSM > default) and forwarded
+        # as simulator flags, because the simulator's own precedence puts
+        # flags above any .env or environment variable.
+        [[ -n "${G3DT_LLM_PROVIDER:-}" ]] && CMD+=(--llm-provider "$G3DT_LLM_PROVIDER")
+        [[ -n "${G3DT_LLM_MODEL:-}" ]] && CMD+=(--llm-model "$G3DT_LLM_MODEL")
+        # Neutralize any .env in the caller's CWD: /dev/null exists (satisfies
+        # the simulator's exists=True check) and dotenv-parses to empty, so
+        # only the flags above and the inherited $LLM_API_KEY_FILE apply.
+        CMD+=(--env-file /dev/null)
     fi
     "${CMD[@]}"
 done

gen3_dataops_toolkit-3.3.0/src/g3dt/cli/k8s.py

@@ -1,54 +0,0 @@
-"""`g3dt k8s` — restart Gen3 microservices / ETL via ArgoCD (LOCAL only).
-
-These use ``argocd login --sso`` (a browser flow), so they cannot run headless
-on EC2. The wrapped scripts receive their settings as ``G3DT_*`` environment
-variables resolved from SSM — they read no config files.
-"""
-from __future__ import annotations
-
-import typer
-
-from g3dt.config import script_env
-from g3dt.cli._internal import runner
-from g3dt.cli._internal.resolve import env_of
-
-app = typer.Typer(no_args_is_help=True, help="ArgoCD / Kubernetes restarts (local).")
-
-_SCHEMA = "services/k8s_ops/argocd_restart_schema.sh"
-_ETL = "services/k8s_ops/argocd_restart_etl.sh"
-_ETL_AND_MS = "services/k8s_ops/restart_etl_and_ms.sh"
-
-
-@app.command(name="restart-schema")
-def restart_schema(
-    env: str = typer.Option(..., "--env", "-e", help="Environment, e.g. test."),
-    sync: bool = typer.Option(False, "--sync", "-s", help="argocd app sync first."),
-) -> None:
-    """Restart sheepdog/peregrine/guppy/portal (schema microservices)."""
-    e = env_of(env)
-    args = ["-d", e.domain, "-a", e.app_name, "-n", e.namespace]
-    if sync:
-        args.append("-s")
-    runner.run(runner.bash_script(_SCHEMA, *args), env=script_env(e))
-
-
-@app.command(name="restart-etl")
-def restart_etl(
-    env: str = typer.Option(..., "--env", "-e", help="Environment, e.g. test."),
-    sync: bool = typer.Option(False, "--sync", "-s", help="argocd app sync first."),
-) -> None:
-    """Create + run the ETL cronjob and wait for completion."""
-    e = env_of(env)
-    args = ["-e", env]
-    if sync:
-        args.append("-s")
-    runner.run(runner.bash_script(_ETL, *args), env=script_env(e))
-
-
-@app.command(name="restart-ms")
-def restart_ms(
-    env: str = typer.Option(..., "--env", "-e", help="Environment, e.g. test."),
-) -> None:
-    """Restart both ETL and schema microservices (wraps restart_etl_and_ms.sh)."""
-    e = env_of(env)
-    runner.run(runner.bash_script(_ETL_AND_MS, env), env=script_env(e))