gen3-dataops-toolkit 4.2.0__tar.gz → 5.0.0__tar.gz

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gen3-dataops-toolkit
-Version: 4.2.0
+Version: 5.0.0
 Summary: Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments
 License: Apache-2.0
 Author: JoshuaHarris391
@@ -37,7 +37,7 @@ Description-Content-Type: text/markdown
 Operate Gen3 AWS data-pipeline environments from one pip-installable CLI.
 
 `g3dt` is the tooling half of the Gen3 DataOps platform: the
-[gen3-aws-data-pipeline](https://github.com/AustralianBioCommons/gen3-aws-data-pipeline)
+[aws-gen3-pipeline](https://github.com/AustralianBioCommons/aws-gen3-pipeline)
 CDK app deploys a complete pipeline per project/environment and publishes every
 resource name to AWS SSM Parameter Store; `g3dt` resolves those names at
 runtime and gives operators one command surface for dictionary deploys,
@@ -53,9 +53,15 @@ operates any project: it is targeted by a **context** — a named
 ## Install
 
 ```bash
-pip install gen3-dataops-toolkit
+pipx install gen3-dataops-toolkit        # a CLI in its own venv (recommended)
+pip install gen3-dataops-toolkit         # or into an environment you manage
 ```
 
+Synthetic-data generation needs the simulator too: `g3dt synth
+install-simulator` installs it into whichever environment owns `g3dt`
+(for pipx that is `pipx inject gen3-dataops-toolkit gen3-metadata-simulator`).
+`g3dt --version` reports the installed toolkit version.
+
 ## Bootstrap: discover, then use
 
 ```bash
@@ -111,7 +117,7 @@ CodeBuild's `g3dt config dbt-env --env $ENV` contract is unchanged.)
 There are exactly two kinds of configuration:
 
 - **INPUTS** — human-authored values, committed as
-  `config/<projectId>.<env>.json` in the CDK repo and read only by
+  `config/<project>.<env>.json` in the deployment wrapper repo and read only by
   `cdk deploy`. To change what an environment *declares*, edit that file and
   redeploy — the value flows to SSM.
 - **OUTPUTS** — every resource name the CDK creates plus the mirrored Gen3
@@ -169,6 +175,35 @@ gate fails -> inspect the results table -> fix data -> re-run until green.
 multi-study caller resolves the schema once, lists the validation prefix
 once, and batches all studies into a single Iceberg INSERT.
 
+### The study registry: what an upload sends, and from where
+
+Each study the pipeline can upload is one JSON record in the env's SSM tree
+(`/{project}/{env}/studies/<name>`) holding its Gen3 `project_id`,
+`program_id` and `s3_metadata_path` — the `s3://` prefix of that study's
+release JSONs (`DataImportOrder.txt` plus one `<node>.json` per node, exactly
+as the release export writes them). The environment lives in the path, so
+staging and prod records can never cross-resolve. `g3dt config` shows the
+registry but never edits deployed settings; the registry is the one thing the
+toolkit itself writes, through `g3dt study` (design: `docs/design/studies.md`):
+
+```bash
+g3dt study list                                   # bare names (config studies is an alias)
+g3dt study show cdah                              # record + liveness check of its S3 path
+g3dt study add cdah --project-id CAUGHT-CAD --program-id program1 \
+    --path s3://my-gold/release_jsons/v2.0.0/cdah/
+g3dt study set cdah --path s3://my-gold/release_jsons/v2.1.0/cdah/ --dry-run
+g3dt study set cdah --path s3://my-gold/release_jsons/v2.1.0/cdah/   # config study-set is an alias
+g3dt study repoint --release v2.1.0               # every study at once (--latest: newest tag)
+g3dt study remove cdah                            # registry entry only; data untouched
+```
+
+`set` and `repoint` check a new path in S3 before writing anything (the same
+`DataImportOrder.txt` + node-JSON checks `metadata upload` makes), so a typo
+cannot point an upload at nothing; `set --no-verify` records a path whose
+release is not exported yet. Every write on a production context requires
+typing the context name. To upload one release without moving the registry,
+use `g3dt metadata upload --release <tag>` (5.0.0).
+
 ### Where the data dictionary comes from
 
 Composed from the env's inputs as
@@ -194,6 +229,20 @@ version until the CDK config catches up —
 `g3dt config diff --env <env> --file <wrapper>/config/<project>.<env>.json`
 reports exactly that gap and exits 1, so it can gate CI.
 
+### Deleting metadata: where the node order comes from
+
+`delete metadata` walks nodes children-before-parents. The order is resolved
+from the first available of: an explicit `--import-order <path|s3://…>`
+(failures are fatal — an explicit source is never silently skipped), the
+registered study's release bucket (`DataImportOrder.txt` next to the release's
+node JSONs), a `DataImportOrder.txt` in the current directory, or a
+topological sort derived from the dictionary itself — the `--dict-version`
+tag's bundle when given (downloaded to `~/.g3dt/schemas` if needed), else the
+env's deployed dictionary. Deriving matters when deleting data submitted
+under an older dictionary whose node layout differs from today's: pass
+`--dict-version <old-tag>`. With `--on ec2`, `--import-order` must be an
+`s3://` URI (a laptop path does not exist on the box).
+
 Synthetic data is only schema-valid against the dictionary that generated it, so
 `synth generate` records the dictionary version in each batch and `synth upload`
 refuses a batch that doesn't match the version being uploaded (override with
@@ -274,6 +323,15 @@ one run, and environments deployed without the block keep the classic Gen3 set
 a manually redeployed frontend) simply omits it from the list in its wrapper
 config. `g3dt config show --env <env>` prints the resolved values.
 
+**Syncing ArgoCD is opt-in.** None of these flows runs `argocd app sync`
+unless you pass `--sync` (available on `k8s restart-schema`, `k8s restart-etl`,
+`k8s restart-ms`, `dict deploy` and `synth deploy`). Add it only when the
+commons app is behind the merged revision; a sync that fails on unrelated
+drift (for example a Job whose template changed, which Kubernetes refuses to
+patch) aborts the whole command before anything restarts. Before 5.0.0
+`restart-ms` and `synth deploy` always synced first, and `restart-ms` ignored
+`--restart-services`; both are fixed.
+
 ## Verifying download access (check-download)
 
 Registration alone does not prove a file can be downloaded. Two failure modes

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/README.md

@@ -3,7 +3,7 @@
 Operate Gen3 AWS data-pipeline environments from one pip-installable CLI.
 
 `g3dt` is the tooling half of the Gen3 DataOps platform: the
-[gen3-aws-data-pipeline](https://github.com/AustralianBioCommons/gen3-aws-data-pipeline)
+[aws-gen3-pipeline](https://github.com/AustralianBioCommons/aws-gen3-pipeline)
 CDK app deploys a complete pipeline per project/environment and publishes every
 resource name to AWS SSM Parameter Store; `g3dt` resolves those names at
 runtime and gives operators one command surface for dictionary deploys,
@@ -19,9 +19,15 @@ operates any project: it is targeted by a **context** — a named
 ## Install
 
 ```bash
-pip install gen3-dataops-toolkit
+pipx install gen3-dataops-toolkit        # a CLI in its own venv (recommended)
+pip install gen3-dataops-toolkit         # or into an environment you manage
 ```
 
+Synthetic-data generation needs the simulator too: `g3dt synth
+install-simulator` installs it into whichever environment owns `g3dt`
+(for pipx that is `pipx inject gen3-dataops-toolkit gen3-metadata-simulator`).
+`g3dt --version` reports the installed toolkit version.
+
 ## Bootstrap: discover, then use
 
 ```bash
@@ -77,7 +83,7 @@ CodeBuild's `g3dt config dbt-env --env $ENV` contract is unchanged.)
 There are exactly two kinds of configuration:
 
 - **INPUTS** — human-authored values, committed as
-  `config/<projectId>.<env>.json` in the CDK repo and read only by
+  `config/<project>.<env>.json` in the deployment wrapper repo and read only by
   `cdk deploy`. To change what an environment *declares*, edit that file and
   redeploy — the value flows to SSM.
 - **OUTPUTS** — every resource name the CDK creates plus the mirrored Gen3
@@ -135,6 +141,35 @@ gate fails -> inspect the results table -> fix data -> re-run until green.
 multi-study caller resolves the schema once, lists the validation prefix
 once, and batches all studies into a single Iceberg INSERT.
 
+### The study registry: what an upload sends, and from where
+
+Each study the pipeline can upload is one JSON record in the env's SSM tree
+(`/{project}/{env}/studies/<name>`) holding its Gen3 `project_id`,
+`program_id` and `s3_metadata_path` — the `s3://` prefix of that study's
+release JSONs (`DataImportOrder.txt` plus one `<node>.json` per node, exactly
+as the release export writes them). The environment lives in the path, so
+staging and prod records can never cross-resolve. `g3dt config` shows the
+registry but never edits deployed settings; the registry is the one thing the
+toolkit itself writes, through `g3dt study` (design: `docs/design/studies.md`):
+
+```bash
+g3dt study list                                   # bare names (config studies is an alias)
+g3dt study show cdah                              # record + liveness check of its S3 path
+g3dt study add cdah --project-id CAUGHT-CAD --program-id program1 \
+    --path s3://my-gold/release_jsons/v2.0.0/cdah/
+g3dt study set cdah --path s3://my-gold/release_jsons/v2.1.0/cdah/ --dry-run
+g3dt study set cdah --path s3://my-gold/release_jsons/v2.1.0/cdah/   # config study-set is an alias
+g3dt study repoint --release v2.1.0               # every study at once (--latest: newest tag)
+g3dt study remove cdah                            # registry entry only; data untouched
+```
+
+`set` and `repoint` check a new path in S3 before writing anything (the same
+`DataImportOrder.txt` + node-JSON checks `metadata upload` makes), so a typo
+cannot point an upload at nothing; `set --no-verify` records a path whose
+release is not exported yet. Every write on a production context requires
+typing the context name. To upload one release without moving the registry,
+use `g3dt metadata upload --release <tag>` (5.0.0).
+
 ### Where the data dictionary comes from
 
 Composed from the env's inputs as
@@ -160,6 +195,20 @@ version until the CDK config catches up —
 `g3dt config diff --env <env> --file <wrapper>/config/<project>.<env>.json`
 reports exactly that gap and exits 1, so it can gate CI.
 
+### Deleting metadata: where the node order comes from
+
+`delete metadata` walks nodes children-before-parents. The order is resolved
+from the first available of: an explicit `--import-order <path|s3://…>`
+(failures are fatal — an explicit source is never silently skipped), the
+registered study's release bucket (`DataImportOrder.txt` next to the release's
+node JSONs), a `DataImportOrder.txt` in the current directory, or a
+topological sort derived from the dictionary itself — the `--dict-version`
+tag's bundle when given (downloaded to `~/.g3dt/schemas` if needed), else the
+env's deployed dictionary. Deriving matters when deleting data submitted
+under an older dictionary whose node layout differs from today's: pass
+`--dict-version <old-tag>`. With `--on ec2`, `--import-order` must be an
+`s3://` URI (a laptop path does not exist on the box).
+
 Synthetic data is only schema-valid against the dictionary that generated it, so
 `synth generate` records the dictionary version in each batch and `synth upload`
 refuses a batch that doesn't match the version being uploaded (override with
@@ -240,6 +289,15 @@ one run, and environments deployed without the block keep the classic Gen3 set
 a manually redeployed frontend) simply omits it from the list in its wrapper
 config. `g3dt config show --env <env>` prints the resolved values.
 
+**Syncing ArgoCD is opt-in.** None of these flows runs `argocd app sync`
+unless you pass `--sync` (available on `k8s restart-schema`, `k8s restart-etl`,
+`k8s restart-ms`, `dict deploy` and `synth deploy`). Add it only when the
+commons app is behind the merged revision; a sync that fails on unrelated
+drift (for example a Job whose template changed, which Kubernetes refuses to
+patch) aborts the whole command before anything restarts. Before 5.0.0
+`restart-ms` and `synth deploy` always synced first, and `restart-ms` ignored
+`--restart-services`; both are fixed.
+
 ## Verifying download access (check-download)
 
 Registration alone does not prove a file can be downloaded. Two failure modes

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "gen3-dataops-toolkit"
-version = "4.2.0"
+version = "5.0.0"
 description = "Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments"
 authors = ["JoshuaHarris391 <harjo391@gmail.com>"]
 readme = "README.md"

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/src/g3dt/cli/config_cmds.py

@@ -24,6 +24,7 @@ from typing import List, Optional
 import typer
 
 from g3dt import config, contexts
+from g3dt.cli import study_cmds
 from g3dt.cli._internal import resolve
 from g3dt.cli._internal.resolve import env_of, study_of
 from g3dt.cli._internal.helptext import ENV_OPT
@@ -35,8 +36,11 @@ app = typer.Typer(
          "stored in the local g3dt.yaml marker. Every command acts on the "
          "current context — switch with 'g3dt config use <name>', one-shot "
          "with --ctx. Discovery scans one SSO profile's AWS account at a "
-         "time ('g3dt config discover <profile>'). Run 'g3dt docs' for the "
-         "full model.",
+         "time ('g3dt config discover <profile>'). Deployed settings are "
+         "read-only here (they are CDK inputs); study records are edited "
+         "with 'g3dt config study-set' / 'g3dt study set' and moved to a "
+         "release with 'g3dt study repoint'. Run 'g3dt docs' for the full "
+         "model.",
 )
 
 
@@ -530,12 +534,51 @@ def envs() -> None:
 def studies(
     env: Optional[str] = typer.Option(None, "--env", "-e", help=ENV_OPT),
 ) -> None:
-    """List the env's registered studies (alias of `g3dt study list`)."""
-    from g3dt.cli import study_cmds
+    """List the env's registered studies (alias of `g3dt study list`).
 
+    Inspect one with `g3dt config show --study <name>`; change its S3 path
+    or Gen3 ids with `g3dt config study-set <name> ...`.
+    """
     study_cmds.list_impl(env)
 
 
+@app.command("study-set")
+def study_set(
+    name: str = typer.Argument(..., help="Study name (see `g3dt config studies`)."),
+    env: Optional[str] = typer.Option(None, "--env", "-e", help=ENV_OPT),
+    project_id: Optional[str] = typer.Option(
+        None, "--project-id", help=study_cmds.SET_PROJECT_ID_HELP
+    ),
+    program_id: Optional[str] = typer.Option(
+        None, "--program-id", help=study_cmds.SET_PROGRAM_ID_HELP
+    ),
+    path: Optional[str] = typer.Option(None, "--path", help=study_cmds.SET_PATH_HELP),
+    dry_run: bool = typer.Option(
+        False, "--dry-run", "-d", help=study_cmds.SET_DRY_RUN_HELP
+    ),
+    no_verify: bool = typer.Option(
+        False, "--no-verify", help=study_cmds.SET_NO_VERIFY_HELP
+    ),
+) -> None:
+    """Update a study's S3 release location or Gen3 ids (alias of `g3dt study set`).
+
+    Only the fields you pass change. A new --path is checked in S3 first
+    (DataImportOrder.txt plus node JSONs) so an upload can never be pointed
+    at nothing. To move every study to a new release at once use
+    `g3dt study repoint`.
+
+    Examples:
+      g3dt config study-set cdah --path s3://my-gold/release_jsons/v2.1.0/cdah/ --dry-run
+      g3dt config study-set cdah --path s3://my-gold/release_jsons/v2.1.0/cdah/
+
+    Targeting production requires typing the context/env name to confirm.
+    """
+    study_cmds.set_impl(
+        name, env, project_id, program_id, path,
+        dry_run=dry_run, verify=not no_verify,
+    )
+
+
 @app.command()
 def show(
     env: Optional[str] = typer.Option(None, "--env", "-e", help=ENV_OPT),
@@ -586,6 +629,11 @@ def show(
         typer.echo(f"  project_id       : {s.project_id}")
         typer.echo(f"  program_id       : {s.program_id}")
         typer.echo(f"  s3_metadata_path : {s.s3_metadata_path}")
+        typer.secho(
+            f"  change with      : g3dt config study-set {s.key} --path s3://... "
+            f"| g3dt study repoint --release <tag>",
+            fg=typer.colors.BRIGHT_BLACK,
+        )
     if full:
         rc = resolve.rc_of(env)
         typer.secho(

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/src/g3dt/cli/delete_cmds.py

@@ -14,6 +14,12 @@ synthetic projects are never registered), bare names default to version
 ``data_version`` property via GraphQL rather than Athena receipts (synthetic
 uploads write none).
 
+The node deletion order comes from, in order: an explicit ``--import-order``
+(path or s3:// URI; failures are fatal, never silently skipped), the
+registered study's release bucket, a ``DataImportOrder.txt`` in the current
+directory, or a topological sort derived from the dictionary itself — the
+``--dict-version`` bundle when given, else the env's deployed dictionary.
+
 Every command confirms before acting. Production always requires typing the
 target id, even with ``--yes``. Deleting ALL versions always prompts, even with
 ``--yes``. Confirmation happens locally before any EC2 dispatch (SSM has no
@@ -186,6 +192,21 @@ def metadata(
         help="Gen3 program for --synthetic (default: program1). "
              "Invalid without --synthetic.",
     ),
+    import_order: Optional[str] = typer.Option(
+        None,
+        "--import-order",
+        help="Path or s3:// URI of DataImportOrder.txt. Default: auto — the "
+             "study's release bucket (registered studies), then "
+             "./DataImportOrder.txt, then derived from the dictionary. "
+             "With --on ec2 only s3:// URIs are accepted.",
+    ),
+    dict_version: Optional[str] = typer.Option(
+        None,
+        "--dict-version",
+        help="Dictionary git tag to derive the node order from (verbatim, "
+             "e.g. v1.3.0). Default: the env's deployed dictionary. Only "
+             "used when the order is derived.",
+    ),
     yes: bool = typer.Option(
         False, "--yes", "-y", help="Skip the non-prod prompt (specific-version only)."
     ),
@@ -212,6 +233,26 @@ def metadata(
             err=True,
         )
         raise typer.Exit(2)
+    if import_order and dict_version:
+        typer.secho(
+            "--import-order names the exact file; --dict-version derives one "
+            "— pass only one.",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(2)
+    # A laptop path forwarded to the EC2 box would resolve against the box's
+    # filesystem — at best a crash after confirmation, at worst a same-named
+    # DIFFERENT file ordering the delete. s3:// URIs (and --dict-version)
+    # resolve identically anywhere, so only those may travel.
+    if on == Target.ec2 and import_order and not import_order.startswith("s3://"):
+        typer.secho(
+            "--import-order with --on ec2 must be an s3:// URI (a local path "
+            "does not exist on the box). Upload the file, or run locally.",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(2)
     if version is None:
         fallback = None
     elif synthetic:
@@ -266,6 +307,10 @@ def metadata(
             # optional on the wire, but an explicit value keeps the contract
             # visible in logs and SSM command history.
             a += ["--synthetic", "--program-id", program_id or "program1"]
+        if import_order:
+            a += ["--import-order", import_order]
+        if dict_version:
+            a += ["--dict-version", dict_version]
         return a
 
     def remote_cli(env_name):
@@ -284,6 +329,11 @@ def metadata(
             a.append("--synthetic")
             if program_id is not None:
                 a += ["--program-id", program_id]
+        if import_order:
+            # Guaranteed s3:// by the pre-dispatch gate above.
+            a += ["--import-order", import_order]
+        if dict_version:
+            a += ["--dict-version", dict_version]
         return a
 
     dispatch.run_or_dispatch(

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/src/g3dt/cli/dict_cmds.py

@@ -14,12 +14,21 @@ from typing import Optional
 
 import typer
 
-from g3dt.config import dictionary_filename, dictionary_url, script_env
+from g3dt.config import (
+    ConfigError,
+    dictionary_filename,
+    dictionary_url,
+    normalize_s3_location,
+    script_env,
+)
 from g3dt.cli._internal import resolve, runner, safety
 from g3dt.cli._internal.resolve import env_of
 from g3dt.cli._internal.helptext import ENV_OPT
 
-app = typer.Typer(no_args_is_help=True, help="Data dictionary operations (local).")
+app = typer.Typer(
+    no_args_is_help=True,
+    help="Data dictionary operations: pull, upload, deploy, status (local).",
+)
 
 SCHEMA_DIR = Path("~/.g3dt/schemas").expanduser()
 
@@ -78,6 +87,82 @@ def pull(
     )
 
 
+@app.command()
+def status(
+    env: Optional[str] = typer.Option(None, "--env", "-e", help=ENV_OPT),
+    strict: bool = typer.Option(
+        False, "--strict",
+        help="Exit 1 when the deployed version differs from the declared one.",
+    ),
+) -> None:
+    """Show which dictionary version is deployed, next to the declared one.
+
+    Declared is the env's `dictionary_version` (a CDK input — what
+    `g3dt config show` reports). Deployed is the `version` stamped on the S3
+    object at `schema_s3_uri` by `dict upload`, i.e. what the commons'
+    services actually read on their next restart. The two drift after
+    `dict deploy --version <tag>` until the CDK config is updated and
+    redeployed (`g3dt config diff` shows that half). Read-only.
+
+    Examples:
+      g3dt dict status --env staging
+      g3dt dict status --env staging --strict    # drift gate for scripts
+    """
+    from botocore.exceptions import ClientError
+
+    env = resolve.active_env(env)
+    e = env_of(env)
+    _, session = resolve.rc_session_of(env)
+    try:
+        location = normalize_s3_location(e.schema_s3_uri, param="schema_s3_uri")
+    except ConfigError as exc:
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(1)
+    if "/" not in location:
+        typer.secho(
+            f"schema_s3_uri '{e.schema_s3_uri}' has no object key — expected "
+            f"<bucket>/<key>.", fg=typer.colors.RED, err=True,
+        )
+        raise typer.Exit(1)
+    bucket, key = location.split("/", 1)
+    deployed, uploaded = None, None
+    try:
+        head = session.client("s3").head_object(Bucket=bucket, Key=key)
+        deployed = head.get("Metadata", {}).get("version")
+        uploaded = head.get("LastModified")
+    except ClientError as exc:
+        code = exc.response.get("Error", {}).get("Code", "")
+        if code not in ("404", "NoSuchKey", "NotFound"):
+            typer.secho(
+                f"Cannot read s3://{location} ({code or exc}).",
+                fg=typer.colors.RED, err=True,
+            )
+            raise typer.Exit(1)
+
+    declared = e.dictionary_version
+    typer.echo(f"declared : {declared}   (SSM app/dictionary_version)")
+    if uploaded is None:
+        typer.echo(f"deployed : (nothing at s3://{location} — run `g3dt dict deploy`)")
+        state = "MISSING"
+    else:
+        stamp = deployed or "(no version stamp on the object)"
+        when = uploaded.strftime("%Y-%m-%d %H:%M %Z") if hasattr(uploaded, "strftime") else uploaded
+        typer.echo(f"deployed : {stamp}   (s3://{location}, uploaded {when})")
+        state = "in sync" if deployed == declared else "DRIFT"
+    if state == "in sync":
+        typer.secho("status   : in sync", fg=typer.colors.GREEN)
+        return
+    typer.secho(
+        f"status   : {state} — the services will read {deployed or 'nothing'} "
+        f"while the CDK config declares {declared}. Deploy with "
+        f"`g3dt dict deploy`, or update dictionaryVersion in the wrapper "
+        f"config and redeploy.",
+        fg=typer.colors.YELLOW,
+    )
+    if strict:
+        raise typer.Exit(1)
+
+
 @app.command()
 def upload(
     env: Optional[str] = typer.Option(None, "--env", "-e", help=ENV_OPT),
@@ -88,6 +173,13 @@ def upload(
 ) -> None:
     """Upload the (already pulled) dictionary JSON to the env's S3 location.
 
+    Stamps the object's S3 metadata `version` from the JSON's _settings, which
+    is what `g3dt dict status` reads back.
+
+    Examples:
+      g3dt dict upload --env staging
+      g3dt dict upload --env staging --version v1.1.7
+
     Targeting production requires typing the context/env name to confirm.
     """
     env = resolve.active_env(env)
@@ -116,17 +208,25 @@ def deploy(
     restart_services: Optional[str] = typer.Option(
         None, "--restart-services",
         help="Comma-separated deployment names restarted after the upload, in "
-        "order; default: the env's SSM app/restart_services.",
+        "order; default: the env's SSM app/restart_services (the CDK "
+        "config's k8s.schemaRestartServices).",
+    ),
+    sync: bool = typer.Option(
+        False, "--sync",
+        help="Run 'argocd app sync' on the commons app before the restart "
+        "(off by default; add it when the app is behind the merged revision).",
     ),
 ) -> None:
     """Pull + upload the dictionary and restart Gen3 schema microservices.
 
     Wraps services/dictionary/deploy_dd.sh. Requires an interactive ArgoCD SSO
-    login, so it runs locally only.
+    login, so it runs locally only. The restart never syncs the ArgoCD app
+    unless --sync is given.
 
     The version defaults to the env's `dictionary_version`, a CDK INPUT: edit
-    config/<project>.<env>.json in gen3-aws-data-pipeline and `cdk deploy` to
-    change what an env declares. Pass --version to deploy a different tag now
+    config/<project>.<env>.json in your deployment wrapper (the repo that pins
+    the aws-gen3-pipeline template) and `cdk deploy` to change what an env
+    declares. Pass --version to deploy a different tag now
     without that round trip — which is how one dictionary gets promoted across
     environments. `g3dt config diff` reports the resulting drift until the CDK
     config catches up.
@@ -146,6 +246,8 @@ def deploy(
     env_vars = script_env(e, _version(e, version))
     if restart_services:
         env_vars["G3DT_RESTART_SERVICES"] = restart_services
+    if sync:
+        env_vars["G3DT_SYNC"] = "1"
     runner.run(
         runner.bash_script("services/dictionary/deploy_dd.sh", env),
         env=env_vars,

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/src/g3dt/cli/indexd_cmds.py

@@ -35,7 +35,9 @@ def register(
     env: Optional[str] = typer.Option(None, "--env", "-e", help=ENV_OPT),
     dry_run: bool = typer.Option(
         False, "--dry-run", "-d",
-        help="Scan + write file_metadata only; skip indexd."
+        help="Scan the prefixes and record what was found in the warehouse's "
+        "file_metadata table, but register NOTHING with indexd. Note this "
+        "still writes that table (unlike --dry-run elsewhere in g3dt)."
     ),
     force: bool = typer.Option(
         False, "--force",

{gen3_dataops_toolkit-4.2.0 → gen3_dataops_toolkit-5.0.0}/src/g3dt/cli/jobs.py

@@ -7,7 +7,15 @@ import typer
 
 from g3dt.cli._internal import dispatch, registry, resolve
 
-app = typer.Typer(no_args_is_help=True, help="Track EC2-dispatched jobs.")
+app = typer.Typer(
+    no_args_is_help=True,
+    help="Track EC2-dispatched jobs.\n\n"
+    "Every `--on ec2` dispatch (metadata upload / upload-all, delete "
+    "metadata, indexd register) prints a run id and records it in this "
+    "laptop's ~/.g3dt/runs.json; these commands read that file and the "
+    "run's SSM Run Command status / CloudWatch logs. No --env: the run id "
+    "already names its environment.",
+)
 
 
 def _status_color(state: str) -> str:
@@ -79,9 +87,21 @@ def stop(run_id: str = typer.Argument(..., help="Run id from dispatch.")) -> Non
 
 @app.command()
 def logs(
-    run_id: str = typer.Argument(..., help="Run id from dispatch."),
-    follow: bool = typer.Option(False, "--follow", "-f", help="Stream new output as it arrives."),
+    run_id: str = typer.Argument(
+        ..., help="Run id printed by the --on ec2 dispatch (see `g3dt jobs list`)."
+    ),
+    follow: bool = typer.Option(
+        False, "--follow", "-f",
+        help="Stream new output as it arrives, until the run ends.",
+    ),
 ) -> None:
-    """Print (and optionally follow) the CloudWatch logs for a dispatched run."""
+    """Print (and optionally follow) the CloudWatch logs for a dispatched run.
+
+    Empty output in the first minute is normal: the log stream is created
+    when the box starts writing.
+
+    Examples:
+      g3dt jobs logs 20260908T093000-metadata-upload --follow
+    """
     resolve.announce_context()
     dispatch.logs(run_id, follow=follow)