gen3-dataops-toolkit 3.7.4__tar.gz → 3.8.1__tar.gz

{gen3_dataops_toolkit-3.7.4 → gen3_dataops_toolkit-3.8.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gen3-dataops-toolkit
-Version: 3.7.4
+Version: 3.8.1
 Summary: Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments
 License: Apache-2.0
 Author: JoshuaHarris391
@@ -46,8 +46,9 @@ restarts. The dbt half of the platform lives in
 [gen3-dbt-template](https://github.com/AustralianBioCommons/gen3-dbt-template).
 
 **No AWS resource name is compiled into this package.** The same wheel
-operates any project: it is targeted purely by `--env`, the project's SSM tree
-(`/{project}/{env}/...`), and a tiny local bootstrap marker.
+operates any project: it is targeted by a **context** — a named
+(project, env, profile, region) tuple — and the project's SSM tree
+(`/{project}/{env}/...`).
 
 ## Install
 
@@ -55,41 +56,50 @@ operates any project: it is targeted purely by `--env`, the project's SSM tree
 pip install gen3-dataops-toolkit
 ```
 
-## Bootstrap (the only local configuration)
-
-`g3dt` needs to know just the project and region — everything else comes from
-SSM. Create `~/.g3dt/g3dt.yaml`:
-
-```yaml
-project: etl                # your projectId
-region: ap-southeast-2
-default_env: test
-profiles:                   # optional: AWS named profile per env
-  test: etl_test            # (omit entirely on EC2/CodeBuild — ambient
-  staging: etl_staging      #  role credentials are used)
-studies:                    # optional: the project's study registry;
-  mystudy_test:             # alternatively upload it once per env to
-    project_id: MyStudy     # s3://<metadata-bucket>/config/studies.yaml
-    program_id: program1
-    s3_metadata_path: s3://my-bucket/metadata/mystudy/
+## Bootstrap: discover, then use
+
+```bash
+g3dt config discover <aws-profile> --add    # logs the profile in if needed,
+                                            # lists every deployed env its
+                                            # account holds, registers them
+g3dt config contexts                        # list them (current marked *)
+g3dt config use myproj/test                 # point g3dt at one
+```
+
+That writes `~/.g3dt/g3dt.yaml` for you. **Every command prints the active
+context first** (on stderr) — read that line before anything else:
+
+```
+ctx myproj/test → project=myproj env=test profile=myproj_test region=ap-southeast-2
 ```
 
+Production contexts are marked `[PROD]`, switching to one asks for
+confirmation, and destructive actions on them require typing the context
+name — `--yes` never bypasses that.
+
+The marker can also be written by hand (design doc:
+`docs/design/contexts.md`); the study registry lives either in a top-level
+`studies:` block or per env at `s3://<metadata-bucket>/config/studies.yaml`.
 Search order: `./g3dt.yaml` → `~/.g3dt/g3dt.yaml` → `/etc/g3dt/g3dt.yaml`
-(the EC2 job box's copy, written by CDK user-data). Env vars override:
-`G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV`.
+(the EC2 job box's copy, written by CDK user-data). Legacy markers
+(`project`/`default_env`/`profiles:` keys) keep working unchanged, as do the
+env-var overrides `G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV` — that is
+the file-less CodeBuild/EC2 path.
 
 ## Quick start
 
 ```bash
-g3dt config envs                 # environments with a deployed SSM tree
-g3dt config show --env test      # every resolved name — the safety check
-g3dt ec2 up --env test           # start the env's job box (SSM-managed)
-g3dt metadata upload --study mystudy --env test --on ec2
+g3dt config show                 # every resolved name — the safety check
+g3dt ec2 up                      # start the context's job box (SSM-managed)
+g3dt metadata upload --study mystudy --on ec2
 g3dt jobs logs <run-id> --follow # live logs; laptop can sleep, job keeps going
-g3dt ec2 down --env test         # or let the auto-stop alarm handle it
+g3dt ec2 down                    # or let the auto-stop alarm handle it
 g3dt docs                        # the full operations overview
 ```
 
+(`--env <name>` still works everywhere and selects the matching context;
+CodeBuild's `g3dt config dbt-env --env $ENV` contract is unchanged.)
+
 ## How configuration works
 
 There are exactly two kinds of configuration:

{gen3_dataops_toolkit-3.7.4 → gen3_dataops_toolkit-3.8.1}/README.md

@@ -12,8 +12,9 @@ restarts. The dbt half of the platform lives in
 [gen3-dbt-template](https://github.com/AustralianBioCommons/gen3-dbt-template).
 
 **No AWS resource name is compiled into this package.** The same wheel
-operates any project: it is targeted purely by `--env`, the project's SSM tree
-(`/{project}/{env}/...`), and a tiny local bootstrap marker.
+operates any project: it is targeted by a **context** — a named
+(project, env, profile, region) tuple — and the project's SSM tree
+(`/{project}/{env}/...`).
 
 ## Install
 
@@ -21,41 +22,50 @@ operates any project: it is targeted purely by `--env`, the project's SSM tree
 pip install gen3-dataops-toolkit
 ```
 
-## Bootstrap (the only local configuration)
-
-`g3dt` needs to know just the project and region — everything else comes from
-SSM. Create `~/.g3dt/g3dt.yaml`:
-
-```yaml
-project: etl                # your projectId
-region: ap-southeast-2
-default_env: test
-profiles:                   # optional: AWS named profile per env
-  test: etl_test            # (omit entirely on EC2/CodeBuild — ambient
-  staging: etl_staging      #  role credentials are used)
-studies:                    # optional: the project's study registry;
-  mystudy_test:             # alternatively upload it once per env to
-    project_id: MyStudy     # s3://<metadata-bucket>/config/studies.yaml
-    program_id: program1
-    s3_metadata_path: s3://my-bucket/metadata/mystudy/
+## Bootstrap: discover, then use
+
+```bash
+g3dt config discover <aws-profile> --add    # logs the profile in if needed,
+                                            # lists every deployed env its
+                                            # account holds, registers them
+g3dt config contexts                        # list them (current marked *)
+g3dt config use myproj/test                 # point g3dt at one
+```
+
+That writes `~/.g3dt/g3dt.yaml` for you. **Every command prints the active
+context first** (on stderr) — read that line before anything else:
+
+```
+ctx myproj/test → project=myproj env=test profile=myproj_test region=ap-southeast-2
 ```
 
+Production contexts are marked `[PROD]`, switching to one asks for
+confirmation, and destructive actions on them require typing the context
+name — `--yes` never bypasses that.
+
+The marker can also be written by hand (design doc:
+`docs/design/contexts.md`); the study registry lives either in a top-level
+`studies:` block or per env at `s3://<metadata-bucket>/config/studies.yaml`.
 Search order: `./g3dt.yaml` → `~/.g3dt/g3dt.yaml` → `/etc/g3dt/g3dt.yaml`
-(the EC2 job box's copy, written by CDK user-data). Env vars override:
-`G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV`.
+(the EC2 job box's copy, written by CDK user-data). Legacy markers
+(`project`/`default_env`/`profiles:` keys) keep working unchanged, as do the
+env-var overrides `G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV` — that is
+the file-less CodeBuild/EC2 path.
 
 ## Quick start
 
 ```bash
-g3dt config envs                 # environments with a deployed SSM tree
-g3dt config show --env test      # every resolved name — the safety check
-g3dt ec2 up --env test           # start the env's job box (SSM-managed)
-g3dt metadata upload --study mystudy --env test --on ec2
+g3dt config show                 # every resolved name — the safety check
+g3dt ec2 up                      # start the context's job box (SSM-managed)
+g3dt metadata upload --study mystudy --on ec2
 g3dt jobs logs <run-id> --follow # live logs; laptop can sleep, job keeps going
-g3dt ec2 down --env test         # or let the auto-stop alarm handle it
+g3dt ec2 down                    # or let the auto-stop alarm handle it
 g3dt docs                        # the full operations overview
 ```
 
+(`--env <name>` still works everywhere and selects the matching context;
+CodeBuild's `g3dt config dbt-env --env $ENV` contract is unchanged.)
+
 ## How configuration works
 
 There are exactly two kinds of configuration:

{gen3_dataops_toolkit-3.7.4 → gen3_dataops_toolkit-3.8.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "gen3-dataops-toolkit"
-version = "3.7.4"
+version = "3.8.1"
 description = "Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments"
 authors = ["JoshuaHarris391 <harjo391@gmail.com>"]
 readme = "README.md"

gen3_dataops_toolkit-3.8.1/src/g3dt/cli/_internal/resolve.py

@@ -0,0 +1,85 @@
+"""Thin wrappers that turn config resolution errors into clean CLI exits.
+
+3.8.0 additions: :func:`active_env` / :func:`announce_context` implement the
+universal context banner (design doc docs/design/contexts.md section 6), and
+:func:`rc_of` is the single resolved-config idiom that replaces the hand-rolled
+``load_marker``/``require_project``/``env_base``/``aws_profile_for`` block that
+used to be copy-pasted across command modules (one copy of which forgot the
+``_ec2`` → ambient rule — the class of bug this unification removes).
+"""
+from __future__ import annotations
+
+from typing import Optional
+
+import typer
+
+from g3dt import config, contexts
+
+
+def env_of(env: str) -> config.EnvConfig:
+    try:
+        return config.resolve_env(env)
+    except config.ConfigError as exc:
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(1)
+
+
+def study_of(study: str, env: str) -> config.StudyConfig:
+    try:
+        return config.resolve_study(study, env)
+    except config.ConfigError as exc:
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(1)
+
+
+def active_env(env: Optional[str]) -> str:
+    """Resolve the acting context, print the banner, return the effective env.
+
+    Every command that takes ``--env`` calls this first. The returned string
+    keeps a caller-supplied ``_ec2`` suffix intact (dispatch/auth mechanics
+    downstream depend on it, and the remote wire form never changes).
+    """
+    try:
+        ctx, effective = contexts.resolve_context(
+            ctx_name=contexts.override(), env=env
+        )
+    except config.ConfigError as exc:
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(1)
+    contexts.set_active(ctx)
+    contexts.print_banner(ctx, effective)
+    return effective
+
+
+def announce_context() -> None:
+    """Banner-only entry point for commands that take no ``--env``.
+
+    Never fails: with nothing configured it prints the "(none configured)"
+    banner and returns — `jobs list`, `version` etc. must keep working on a
+    bare machine.
+    """
+    try:
+        ctx, effective = contexts.resolve_context(required=False)
+    except config.ConfigError:
+        ctx, effective = None, None
+    contexts.set_active(ctx)
+    contexts.print_banner(ctx, effective)
+
+
+def rc_of(env: str):
+    """Resolved SSM config for ``env`` with the correct credential rule.
+
+    ``_ec2``-suffixed envs always use the ambient chain (never a laptop
+    profile) — the rule three call sites applied and two forgot.
+    """
+    from g3dt import resolver
+
+    marker = config.load_marker()
+    project = config.require_project(marker)
+    base = config.env_base(env)
+    profile = None if env.endswith("_ec2") else config.aws_profile_for(env, marker)
+    try:
+        return resolver.resolve(project, base, profile=profile)
+    except config.ConfigError as exc:
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(1)

{gen3_dataops_toolkit-3.7.4 → gen3_dataops_toolkit-3.8.1}/src/g3dt/cli/_internal/runner.py

@@ -38,8 +38,20 @@ def python_script(relpath: str, *args: str) -> List[str]:
 
 
 def bash_script(relpath: str, *args: str) -> List[str]:
-    """Build an argv that runs a packaged shell script via bash."""
-    return ["bash", str(package_path(relpath)), *[str(a) for a in args]]
+    """Build an argv that runs a packaged shell script via bash.
+
+    The CLI's own interpreter is passed through as ``G3DT_PYTHON`` (via an
+    ``env`` prefix, so no process-global mutation) because wrapped scripts
+    that shell back into packaged Python services must use the interpreter
+    that owns this g3dt installation. A bare ``python3`` is NOT that on
+    Amazon Linux 2023, where the system default is 3.9 but g3dt is installed
+    under 3.11 — observed live on acdc/staging as ``ModuleNotFoundError:
+    No module named 'g3dt'`` from every study in an upload-all run.
+    """
+    return [
+        "env", f"G3DT_PYTHON={sys.executable}",
+        "bash", str(package_path(relpath)), *[str(a) for a in args],
+    ]
 
 
 def run(

gen3_dataops_toolkit-3.8.1/src/g3dt/cli/_internal/safety.py

@@ -0,0 +1,130 @@
+"""Confirmation and environment guards for destructive / production operations.
+
+These mirror (and strengthen) the guards already baked into the shell scripts:
+the test-only ``synth deploy`` guard, the prod aborts in the bulk scripts, and
+the optional delete confirmation prompts.
+
+3.8.0: the guards are context-aware. Production is any env/study key whose
+name contains ``prod`` (unchanged), OR an active context classified production
+(``production: true`` flag, or 'prod' in its name — see
+``contexts.is_production``). When a *named* context from the marker is active,
+the typed confirmation token is the **context name** (e.g. ``acdc/prod``);
+legacy and synthetic contexts keep the historical env-name/target tokens, so
+older muscle memory and the pinned safety tests are unaffected. ``--yes``
+never bypasses a production prompt, and confirmation always happens locally,
+before any EC2 dispatch.
+"""
+from __future__ import annotations
+
+from typing import Optional
+
+import typer
+
+from g3dt.config import env_base
+
+
+def is_prod(env: str) -> bool:
+    """True if the environment name refers to production."""
+    return "prod" in env.lower()
+
+
+def _active_prod_context():
+    """The active context when it is production-classified, else ``None``."""
+    from g3dt import contexts
+
+    ctx = contexts.active()
+    if ctx is not None and contexts.is_production(ctx):
+        return ctx
+    return None
+
+
+def _named(ctx) -> bool:
+    """True for a context the operator configured by name (marker source)."""
+    return ctx is not None and ctx.source == "marker"
+
+
+def _typed_gate(header: str, token: str) -> None:
+    typer.secho(header, fg=typer.colors.RED, bold=True)
+    # default="" so an empty entry (just pressing Enter) returns immediately
+    # and aborts, instead of click re-prompting forever.
+    typed = typer.prompt(
+        f"Type '{token}' to confirm", default="", show_default=False
+    )
+    if typed.strip() != token:
+        typer.secho(
+            "Confirmation did not match. Aborting.",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(1)
+
+
+def require_test_env(env: str) -> None:
+    """Abort unless ``env`` is the test environment (``test`` or ``test_ec2``).
+
+    A hard guard for any command that must never run outside test. (The ``synth``
+    commands no longer use this — they allow any env and gate prod with
+    :func:`confirm_prod_strict` instead.)
+    """
+    if env_base(env) != "test":
+        typer.secho(
+            f"Refusing: this command is only allowed for the 'test' "
+            f"environment (got '{env}').",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(2)
+
+
+def confirm_destructive(action: str, target: str, env: str, assume_yes: bool) -> None:
+    """Gate a destructive operation with an appropriate confirmation.
+
+    * Production (env name, study key, or the active context's
+      classification): ALWAYS require a typed confirmation, even with
+      ``--yes`` (so automation can never silently delete prod data). The
+      token is the active context's name when one is configured, else the
+      ``target``.
+    * Non-production: a simple y/N prompt, skippable with ``--yes``.
+
+    Confirmation always happens locally, before any EC2 dispatch (SSM has no
+    TTY), after which the remote job is invoked with ``--yes``.
+    """
+    ctx = _active_prod_context()
+    if is_prod(env) or ctx is not None:
+        token = ctx.name if _named(ctx) else target
+        _typed_gate(
+            f"PRODUCTION {action} targeting '{target}' (env={env}"
+            + (f", ctx={ctx.name}" if ctx is not None else "")
+            + ").",
+            token,
+        )
+        return
+
+    if assume_yes:
+        return
+    if not typer.confirm(f"{action} targeting '{target}' (env={env}). Proceed?"):
+        typer.secho("Aborted.", fg=typer.colors.YELLOW)
+        raise typer.Exit(1)
+
+
+def confirm_prod_strict(action: str, env: str) -> None:
+    """Warn and require a typed confirmation before any action on production.
+
+    Production is any env whose name contains ``prod`` (see :func:`is_prod`)
+    or an active production-classified context. Non-production returns
+    immediately (no prompt). The confirmation cannot be bypassed, so
+    automation can never silently act on prod. The token is the active
+    context's name when one is configured, else the env name.
+
+    Used by the ``synth`` commands, which may target any configured environment.
+    """
+    ctx = _active_prod_context()
+    if not is_prod(env) and ctx is None:
+        return
+    token = ctx.name if _named(ctx) else env
+    _typed_gate(
+        f"PRODUCTION {action} targeting env '{env}'"
+        + (f" (ctx={ctx.name})" if ctx is not None else "")
+        + ".",
+        token,
+    )