gen3-dataops-toolkit 2.1.2__tar.gz → 2.3.0__tar.gz

gen3_dataops_toolkit-2.3.0/PKG-INFO

@@ -0,0 +1,234 @@
+Metadata-Version: 2.4
+Name: gen3-dataops-toolkit
+Version: 2.3.0
+Summary: Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments
+License: Apache-2.0
+Author: JoshuaHarris391
+Author-email: harjo391@gmail.com
+Requires-Python: >=3.9.5,<4.0.0
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: awswrangler (>=3.14.0,<4.0.0)
+Requires-Dist: boto3
+Requires-Dist: gen3 (>=4.27.4,<5.0.0)
+Requires-Dist: gen3-metadata (>=1.4.0,<2.0.0)
+Requires-Dist: gen3_validator (>=2.0.0,<3.0.0)
+Requires-Dist: numpy (<2.0.0)
+Requires-Dist: openpyxl (>=3.1.0)
+Requires-Dist: pyarrow (>=14.0.0,<19.0.0)
+Requires-Dist: pyjwt (>=2.10.1,<3.0.0)
+Requires-Dist: python-dotenv
+Requires-Dist: pytz (>=2025.2,<2026.0)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Requires-Dist: s3fs (==2025.10.0)
+Requires-Dist: tenacity (>=8.2,<10.0)
+Requires-Dist: typer (>=0.12)
+Requires-Dist: tzlocal (>=5.3.1,<6.0.0)
+Project-URL: Repository, https://github.com/AustralianBioCommons/gen3-dataops-toolkit
+Description-Content-Type: text/markdown
+
+# gen3-dataops-toolkit (`g3dt`)
+
+Operate Gen3 AWS data-pipeline environments from one pip-installable CLI.
+
+`g3dt` is the tooling half of the Gen3 DataOps platform: the
+[gen3-aws-data-pipeline](https://github.com/AustralianBioCommons/gen3-aws-data-pipeline)
+CDK app deploys a complete pipeline per project/environment and publishes every
+resource name to AWS SSM Parameter Store; `g3dt` resolves those names at
+runtime and gives operators one command surface for dictionary deploys,
+metadata upload/delete, indexd registration, EC2 job dispatch, and Kubernetes
+restarts. The dbt half of the platform lives in
+[gen3-dbt-template](https://github.com/AustralianBioCommons/gen3-dbt-template).
+
+**No AWS resource name is compiled into this package.** The same wheel
+operates any project: it is targeted purely by `--env`, the project's SSM tree
+(`/{project}/{env}/...`), and a tiny local bootstrap marker.
+
+## Install
+
+```bash
+pip install gen3-dataops-toolkit
+```
+
+## Bootstrap (the only local configuration)
+
+`g3dt` needs to know just the project and region — everything else comes from
+SSM. Create `~/.g3dt/g3dt.yaml`:
+
+```yaml
+project: etl                # your projectId
+region: ap-southeast-2
+default_env: test
+profiles:                   # optional: AWS named profile per env
+  test: etl_test            # (omit entirely on EC2/CodeBuild — ambient
+  staging: etl_staging      #  role credentials are used)
+studies:                    # optional: the project's study registry;
+  mystudy_test:             # alternatively upload it once per env to
+    project_id: MyStudy     # s3://<metadata-bucket>/config/studies.yaml
+    program_id: program1
+    s3_metadata_path: s3://my-bucket/metadata/mystudy/
+```
+
+Search order: `./g3dt.yaml` → `~/.g3dt/g3dt.yaml` → `/etc/g3dt/g3dt.yaml`
+(the EC2 job box's copy, written by CDK user-data). Env vars override:
+`G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV`.
+
+## Quick start
+
+```bash
+g3dt config envs                 # environments with a deployed SSM tree
+g3dt config show --env test      # every resolved name — the safety check
+g3dt ec2 up --env test           # start the env's job box (SSM-managed)
+g3dt metadata upload --study mystudy --env test --on ec2
+g3dt jobs logs <run-id> --follow # live logs; laptop can sleep, job keeps going
+g3dt ec2 down --env test         # or let the auto-stop alarm handle it
+g3dt docs                        # the full operations overview
+```
+
+## How configuration works
+
+There are exactly two kinds of configuration:
+
+- **INPUTS** — human-authored values, committed as
+  `config/<projectId>.<env>.json` in the CDK repo and read only by
+  `cdk deploy`. To change what an environment *declares*, edit that file and
+  redeploy — the value flows to SSM.
+- **OUTPUTS** — every resource name the CDK creates plus the mirrored Gen3
+  app facts, published to SSM under `/{project}/{env}/...` on deploy. `g3dt`
+  reads these live (cached one round-trip per invocation) and never stores
+  them locally.
+
+Because the CLI and the infrastructure read the same parameters, they cannot
+disagree — and because each environment has its own tree (including its own
+`ec2/instanceId`), running a job against the wrong environment's resources is
+structurally impossible.
+
+
+## CI isolation and the release contract
+
+**Only the dbt template's `ci` target is prefixed.** `g3dt config dbt-env`
+emits, alongside the real names, the CI-isolation variants the template's
+`ci` target consumes: `G3DT_DB_RAW_SILVER_CI` / `G3DT_DB_RAW_GOLD_CI`
+(`ci_` + the real database name) and `G3DT_S3_SILVER_DATA_DIR_CI` /
+`G3DT_S3_GOLD_DATA_DIR_CI` (`dbt_ci/` under the same buckets). Commit-
+triggered CI builds land there; every other target (default, local) and the
+release build keep the real, unprefixed names — so CI can never advance the
+warehouse's Iceberg snapshots that releases pin. The library enforces the
+other half: `find_db_for_model` always skips `ci_`-prefixed databases, so
+`g3dt release write` can never pin a release to a CI-build snapshot.
+
+**Snapshot pinning.** `AthenaValidationWriter.construct_json` /
+`AthenaGoldWriter.construct_json` honour a pre-set `snapshot_id` (reading the
+table `FOR VERSION AS OF` that snapshot) and only fetch the latest snapshot
+when unpinned — the contract the release-JSON export relies on for
+reproducible releases.
+
+**Concurrency.** `release_writer.run` processes models with a bounded thread
+pool (`max_workers`, default 8) and fails at the end naming every failed
+model (inserts are idempotent — re-run to fill the remainder). The S3
+writers (`write_release_jsons_to_s3`, `write_validation_json_to_s3`) accept
+`s3_client=` (pass one per worker thread) and `key_prefix=` (write a
+verification tree without touching real artifacts).
+
+**The validation gate.** `g3dt.validate.run_validation_gate(glue_database,
+athena_s3_output, aws_region, workgroup)` queries the latest
+`validation_id` in `full_validation_results` for REAL failures — the
+known-noise patterns in `VALIDATION_GATE_IGNORED_ERRORS` and synthetic
+studies are excluded. The validator Glue job fails when rows come back, so a
+green validation Step Function means schema-clean data; the operator loop is
+gate fails -> inspect the results table -> fix data -> re-run until green.
+`validate_pipeline` also accepts pre-computed loop-invariants
+(`schema=`/`resolver=`/`metadata_table=`) and `write_iceberg=False` so a
+multi-study caller resolves the schema once, lists the validation prefix
+once, and batches all studies into a single Iceberg INSERT.
+
+### Where the data dictionary comes from
+
+Composed from the env's inputs as
+`{dictionary_base_url}/{schema_repo}/refs/tags/{dictionary_version}/{dictionary_path}`.
+Only `schema_repo` and `dictionary_version` are required; `app/dictionary_base_url`
+and `app/dictionary_path` are optional and default to raw GitHub and the schema
+repo's conventional layout, so environments deployed before they existed keep
+working. `g3dt config show --env <env>` prints the composed URL.
+
+### Promoting a dictionary across environments
+
+A dictionary version is *content*, not infrastructure: it changes far more often
+than buckets or clusters do. Rather than a `cdk deploy` per environment per
+version, `dict pull`, `dict upload` and `dict deploy` all accept `--version`:
+
+```bash
+g3dt dict deploy --env test    --version v1.1.7
+g3dt dict deploy --env staging --version v1.1.7   # same tag, no cdk deploy
+```
+
+An override does not persist, so `config show` keeps reporting the declared
+version until the CDK config catches up — `g3dt config diff --env <env>` reports
+exactly that gap and exits 1, so it can gate CI.
+
+Synthetic data is only schema-valid against the dictionary that generated it, so
+`synth generate` records the dictionary version in each batch and `synth upload`
+refuses a batch that doesn't match the version being uploaded (override with
+`--allow-version-mismatch`).
+
+## Verifying download access (check-download)
+
+Registration alone does not prove a file can be downloaded. Two failure modes
+are invisible until a user clicks the file in the portal: an Indexd record
+with no storage URL (nothing to download, ever), and a record Fence refuses
+to sign a URL for. `g3dt indexd check-download` walks the exact chain the
+portal hits — Indexd record → storage URL → DRS object → access methods →
+Fence signed URL — and reports PASS/FAIL per object, exiting non-zero if any
+object fails so it can gate a deployment step.
+
+Run it before a release, and after registering new files. The env selects the
+API key secret and the key's JWT selects the commons, so there is no URL to
+pass (and none to get wrong).
+
+```bash
+g3dt indexd check-download --env staging                    # sample the 25 newest
+g3dt indexd check-download --env staging --limit 50
+g3dt indexd check-download --env prod PREFIX/<uuid-1> PREFIX/<uuid-2>
+```
+
+With no GUIDs, the newest objects for the env's commons are sampled from the
+indexd registry (latest revision per baseid). The registry may live in a
+different AWS account than the commons being checked; if the env's AWS
+profile cannot reach it, pass GUIDs explicitly.
+
+Reading a failure:
+
+| Symptom | Meaning |
+|---|---|
+| `Indexd status: 404` | the object is not registered — a registration problem, not a download one |
+| `urls: []` / no access methods | registered but with no storage location; it can never download |
+| `Access endpoint … 401` | authorization: the API key's user lacks `read-storage` on the record's `authz` resource — an authz gap, not a broken key |
+| `Access endpoint … 500` | Fence has the permission but failed to sign — a service-side fault |
+
+On a 401, compare what the record requires
+(`https://commons.example.org/index/<did>`, the `authz` field) with what the
+key's user actually holds (`https://commons.example.org/user/user`):
+downloads require `read-storage` on the record's authz resource, which a user
+holding only `create` does not have.
+
+## Development
+
+```bash
+poetry install
+poetry run python3 -m pytest
+```
+
+## Provenance
+
+This toolkit was ported (working tree only) from
+[AustralianBioCommons/acdc-aws-etl-pipeline](https://github.com/AustralianBioCommons/acdc-aws-etl-pipeline),
+the ACDC ETL monolith, as part of the Gen3 DataOps platform refactor (2026).
+It starts at version **2.0.0**; versions ≤ 1.2.0 on PyPI are the legacy
+`acdc_aws_etl_pipeline` package, which continues to operate the legacy ACDC
+pipeline unchanged.
+

gen3_dataops_toolkit-2.3.0/README.md

@@ -0,0 +1,199 @@
+# gen3-dataops-toolkit (`g3dt`)
+
+Operate Gen3 AWS data-pipeline environments from one pip-installable CLI.
+
+`g3dt` is the tooling half of the Gen3 DataOps platform: the
+[gen3-aws-data-pipeline](https://github.com/AustralianBioCommons/gen3-aws-data-pipeline)
+CDK app deploys a complete pipeline per project/environment and publishes every
+resource name to AWS SSM Parameter Store; `g3dt` resolves those names at
+runtime and gives operators one command surface for dictionary deploys,
+metadata upload/delete, indexd registration, EC2 job dispatch, and Kubernetes
+restarts. The dbt half of the platform lives in
+[gen3-dbt-template](https://github.com/AustralianBioCommons/gen3-dbt-template).
+
+**No AWS resource name is compiled into this package.** The same wheel
+operates any project: it is targeted purely by `--env`, the project's SSM tree
+(`/{project}/{env}/...`), and a tiny local bootstrap marker.
+
+## Install
+
+```bash
+pip install gen3-dataops-toolkit
+```
+
+## Bootstrap (the only local configuration)
+
+`g3dt` needs to know just the project and region — everything else comes from
+SSM. Create `~/.g3dt/g3dt.yaml`:
+
+```yaml
+project: etl                # your projectId
+region: ap-southeast-2
+default_env: test
+profiles:                   # optional: AWS named profile per env
+  test: etl_test            # (omit entirely on EC2/CodeBuild — ambient
+  staging: etl_staging      #  role credentials are used)
+studies:                    # optional: the project's study registry;
+  mystudy_test:             # alternatively upload it once per env to
+    project_id: MyStudy     # s3://<metadata-bucket>/config/studies.yaml
+    program_id: program1
+    s3_metadata_path: s3://my-bucket/metadata/mystudy/
+```
+
+Search order: `./g3dt.yaml` → `~/.g3dt/g3dt.yaml` → `/etc/g3dt/g3dt.yaml`
+(the EC2 job box's copy, written by CDK user-data). Env vars override:
+`G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV`.
+
+## Quick start
+
+```bash
+g3dt config envs                 # environments with a deployed SSM tree
+g3dt config show --env test      # every resolved name — the safety check
+g3dt ec2 up --env test           # start the env's job box (SSM-managed)
+g3dt metadata upload --study mystudy --env test --on ec2
+g3dt jobs logs <run-id> --follow # live logs; laptop can sleep, job keeps going
+g3dt ec2 down --env test         # or let the auto-stop alarm handle it
+g3dt docs                        # the full operations overview
+```
+
+## How configuration works
+
+There are exactly two kinds of configuration:
+
+- **INPUTS** — human-authored values, committed as
+  `config/<projectId>.<env>.json` in the CDK repo and read only by
+  `cdk deploy`. To change what an environment *declares*, edit that file and
+  redeploy — the value flows to SSM.
+- **OUTPUTS** — every resource name the CDK creates plus the mirrored Gen3
+  app facts, published to SSM under `/{project}/{env}/...` on deploy. `g3dt`
+  reads these live (cached one round-trip per invocation) and never stores
+  them locally.
+
+Because the CLI and the infrastructure read the same parameters, they cannot
+disagree — and because each environment has its own tree (including its own
+`ec2/instanceId`), running a job against the wrong environment's resources is
+structurally impossible.
+
+
+## CI isolation and the release contract
+
+**Only the dbt template's `ci` target is prefixed.** `g3dt config dbt-env`
+emits, alongside the real names, the CI-isolation variants the template's
+`ci` target consumes: `G3DT_DB_RAW_SILVER_CI` / `G3DT_DB_RAW_GOLD_CI`
+(`ci_` + the real database name) and `G3DT_S3_SILVER_DATA_DIR_CI` /
+`G3DT_S3_GOLD_DATA_DIR_CI` (`dbt_ci/` under the same buckets). Commit-
+triggered CI builds land there; every other target (default, local) and the
+release build keep the real, unprefixed names — so CI can never advance the
+warehouse's Iceberg snapshots that releases pin. The library enforces the
+other half: `find_db_for_model` always skips `ci_`-prefixed databases, so
+`g3dt release write` can never pin a release to a CI-build snapshot.
+
+**Snapshot pinning.** `AthenaValidationWriter.construct_json` /
+`AthenaGoldWriter.construct_json` honour a pre-set `snapshot_id` (reading the
+table `FOR VERSION AS OF` that snapshot) and only fetch the latest snapshot
+when unpinned — the contract the release-JSON export relies on for
+reproducible releases.
+
+**Concurrency.** `release_writer.run` processes models with a bounded thread
+pool (`max_workers`, default 8) and fails at the end naming every failed
+model (inserts are idempotent — re-run to fill the remainder). The S3
+writers (`write_release_jsons_to_s3`, `write_validation_json_to_s3`) accept
+`s3_client=` (pass one per worker thread) and `key_prefix=` (write a
+verification tree without touching real artifacts).
+
+**The validation gate.** `g3dt.validate.run_validation_gate(glue_database,
+athena_s3_output, aws_region, workgroup)` queries the latest
+`validation_id` in `full_validation_results` for REAL failures — the
+known-noise patterns in `VALIDATION_GATE_IGNORED_ERRORS` and synthetic
+studies are excluded. The validator Glue job fails when rows come back, so a
+green validation Step Function means schema-clean data; the operator loop is
+gate fails -> inspect the results table -> fix data -> re-run until green.
+`validate_pipeline` also accepts pre-computed loop-invariants
+(`schema=`/`resolver=`/`metadata_table=`) and `write_iceberg=False` so a
+multi-study caller resolves the schema once, lists the validation prefix
+once, and batches all studies into a single Iceberg INSERT.
+
+### Where the data dictionary comes from
+
+Composed from the env's inputs as
+`{dictionary_base_url}/{schema_repo}/refs/tags/{dictionary_version}/{dictionary_path}`.
+Only `schema_repo` and `dictionary_version` are required; `app/dictionary_base_url`
+and `app/dictionary_path` are optional and default to raw GitHub and the schema
+repo's conventional layout, so environments deployed before they existed keep
+working. `g3dt config show --env <env>` prints the composed URL.
+
+### Promoting a dictionary across environments
+
+A dictionary version is *content*, not infrastructure: it changes far more often
+than buckets or clusters do. Rather than a `cdk deploy` per environment per
+version, `dict pull`, `dict upload` and `dict deploy` all accept `--version`:
+
+```bash
+g3dt dict deploy --env test    --version v1.1.7
+g3dt dict deploy --env staging --version v1.1.7   # same tag, no cdk deploy
+```
+
+An override does not persist, so `config show` keeps reporting the declared
+version until the CDK config catches up — `g3dt config diff --env <env>` reports
+exactly that gap and exits 1, so it can gate CI.
+
+Synthetic data is only schema-valid against the dictionary that generated it, so
+`synth generate` records the dictionary version in each batch and `synth upload`
+refuses a batch that doesn't match the version being uploaded (override with
+`--allow-version-mismatch`).
+
+## Verifying download access (check-download)
+
+Registration alone does not prove a file can be downloaded. Two failure modes
+are invisible until a user clicks the file in the portal: an Indexd record
+with no storage URL (nothing to download, ever), and a record Fence refuses
+to sign a URL for. `g3dt indexd check-download` walks the exact chain the
+portal hits — Indexd record → storage URL → DRS object → access methods →
+Fence signed URL — and reports PASS/FAIL per object, exiting non-zero if any
+object fails so it can gate a deployment step.
+
+Run it before a release, and after registering new files. The env selects the
+API key secret and the key's JWT selects the commons, so there is no URL to
+pass (and none to get wrong).
+
+```bash
+g3dt indexd check-download --env staging                    # sample the 25 newest
+g3dt indexd check-download --env staging --limit 50
+g3dt indexd check-download --env prod PREFIX/<uuid-1> PREFIX/<uuid-2>
+```
+
+With no GUIDs, the newest objects for the env's commons are sampled from the
+indexd registry (latest revision per baseid). The registry may live in a
+different AWS account than the commons being checked; if the env's AWS
+profile cannot reach it, pass GUIDs explicitly.
+
+Reading a failure:
+
+| Symptom | Meaning |
+|---|---|
+| `Indexd status: 404` | the object is not registered — a registration problem, not a download one |
+| `urls: []` / no access methods | registered but with no storage location; it can never download |
+| `Access endpoint … 401` | authorization: the API key's user lacks `read-storage` on the record's `authz` resource — an authz gap, not a broken key |
+| `Access endpoint … 500` | Fence has the permission but failed to sign — a service-side fault |
+
+On a 401, compare what the record requires
+(`https://commons.example.org/index/<did>`, the `authz` field) with what the
+key's user actually holds (`https://commons.example.org/user/user`):
+downloads require `read-storage` on the record's authz resource, which a user
+holding only `create` does not have.
+
+## Development
+
+```bash
+poetry install
+poetry run python3 -m pytest
+```
+
+## Provenance
+
+This toolkit was ported (working tree only) from
+[AustralianBioCommons/acdc-aws-etl-pipeline](https://github.com/AustralianBioCommons/acdc-aws-etl-pipeline),
+the ACDC ETL monolith, as part of the Gen3 DataOps platform refactor (2026).
+It starts at version **2.0.0**; versions ≤ 1.2.0 on PyPI are the legacy
+`acdc_aws_etl_pipeline` package, which continues to operate the legacy ACDC
+pipeline unchanged.

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "gen3-dataops-toolkit"
-version = "2.1.2"
+version = "2.3.0"
 description = "Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments"
 authors = ["JoshuaHarris391 <harjo391@gmail.com>"]
 readme = "README.md"
@@ -28,6 +28,7 @@ awswrangler = ">=3.14.0,<4.0.0"
 pyarrow = ">=14.0.0,<19.0.0"
 gen3-metadata = ">=1.4.0,<2.0.0"
 typer = ">=0.12"
+openpyxl = ">=3.1.0"
 
 [tool.poetry.scripts]
 g3dt = "g3dt.cli.main:main"

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.3.0}/src/g3dt/cli/_internal/safety.py

@@ -33,18 +33,6 @@ def require_test_env(env: str) -> None:
         raise typer.Exit(2)
 
 
-def abort_if_prod(env: str) -> None:
-    """Hard abort for bulk operations that must never touch production."""
-    if is_prod(env):
-        typer.secho(
-            f"Refusing bulk operation against a production environment "
-            f"('{env}').",
-            fg=typer.colors.RED,
-            err=True,
-        )
-        raise typer.Exit(2)
-
-
 def confirm_destructive(action: str, target: str, env: str, assume_yes: bool) -> None:
     """Gate a destructive operation with an appropriate confirmation.
 

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.3.0}/src/g3dt/cli/config_cmds.py

@@ -86,6 +86,10 @@ def show(
     typer.echo(f"  aws_profile        : {e.aws_profile or '(ambient credentials)'}")
     typer.echo(f"  aws_secret_name    : {e.aws_secret_name}")
     typer.echo(f"  dictionary_version : {e.dictionary_version}")
+    # The composed URL, not the three parts: "which dictionary will this env
+    # actually fetch?" now spans schema_repo plus two optional inputs, so showing
+    # the resolved result is what makes the answer checkable before a deploy.
+    typer.echo(f"  dictionary_url     : {config.dictionary_url(e)}")
     typer.echo(f"  schema_s3_uri      : {e.schema_s3_uri}")
     typer.echo(f"  schema_repo        : {e.schema_repo}")
     typer.echo(f"  domain             : {e.domain}")
@@ -208,15 +212,24 @@ def dbt_env(
         typer.secho(str(exc), fg=typer.colors.RED, err=True)
         raise typer.Exit(1)
 
+    raw_silver_db = rc.get("glue/db/rawSilver")
+    raw_gold_db = rc.get("glue/db/rawGold")
     values = {
         "G3DT_REGION": rc.region,
         "G3DT_ATHENA_WORKGROUP": rc.athena_workgroup,
         "G3DT_ATHENA_OUTPUT": rc.athena_output_location,
         "G3DT_DB_RAW_BRONZE": rc.get("glue/db/rawBronze"),
-        "G3DT_DB_RAW_SILVER": rc.get("glue/db/rawSilver"),
-        "G3DT_DB_RAW_GOLD": rc.get("glue/db/rawGold"),
+        "G3DT_DB_RAW_SILVER": raw_silver_db,
+        "G3DT_DB_RAW_GOLD": raw_gold_db,
         "G3DT_S3_SILVER_DATA_DIR": f"s3://{rc.get('buckets/rawSilver')}/dbt/",
         "G3DT_S3_GOLD_DATA_DIR": f"s3://{rc.get('buckets/rawGold')}/dbt/",
+        # CI isolation: the dbt template's `ci` target builds into these
+        # instead — same grammar as the CDK's ci_ databases, same buckets
+        # under a dbt_ci/ prefix. Real names above are never prefixed.
+        "G3DT_DB_RAW_SILVER_CI": f"ci_{raw_silver_db}" if raw_silver_db else None,
+        "G3DT_DB_RAW_GOLD_CI": f"ci_{raw_gold_db}" if raw_gold_db else None,
+        "G3DT_S3_SILVER_DATA_DIR_CI": f"s3://{rc.get('buckets/rawSilver')}/dbt_ci/",
+        "G3DT_S3_GOLD_DATA_DIR_CI": f"s3://{rc.get('buckets/rawGold')}/dbt_ci/",
     }
     if profile:
         # A named profile means a laptop run: select the dbt target that
@@ -239,6 +252,11 @@ def set_value(
     settings — dictionary_version, domain, buckets, ... — are CDK INPUTS: edit
     config/<project>.<env>.json in gen3-aws-data-pipeline and `cdk deploy`;
     the values flow to SSM, which is what every consumer reads.
+
+    The one exception is per-invocation: `dict pull/upload/deploy --version`
+    fetches a different dictionary tag without a redeploy (for promoting one
+    dictionary across environments). That does not persist — `config diff`
+    reports the gap until the CDK config is updated to match.
     """
     try:
         old, new, path = config.set_marker_value(key, value)