gen3-dataops-toolkit 2.1.2__tar.gz → 2.2.0__tar.gz

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gen3-dataops-toolkit
-Version: 2.1.2
+Version: 2.2.0
 Summary: Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments
 License: Apache-2.0
 Author: JoshuaHarris391
@@ -95,8 +95,8 @@ There are exactly two kinds of configuration:
 
 - **INPUTS** — human-authored values, committed as
   `config/<projectId>.<env>.json` in the CDK repo and read only by
-  `cdk deploy`. To change a deployed setting (e.g. the dictionary version),
-  edit that file and redeploy — the value flows to SSM.
+  `cdk deploy`. To change what an environment *declares*, edit that file and
+  redeploy — the value flows to SSM.
 - **OUTPUTS** — every resource name the CDK creates plus the mirrored Gen3
   app facts, published to SSM under `/{project}/{env}/...` on deploy. `g3dt`
   reads these live (cached one round-trip per invocation) and never stores
@@ -107,6 +107,74 @@ disagree — and because each environment has its own tree (including its own
 `ec2/instanceId`), running a job against the wrong environment's resources is
 structurally impossible.
 
+
+## CI isolation and the release contract
+
+**Only the dbt template's `ci` target is prefixed.** `g3dt config dbt-env`
+emits, alongside the real names, the CI-isolation variants the template's
+`ci` target consumes: `G3DT_DB_RAW_SILVER_CI` / `G3DT_DB_RAW_GOLD_CI`
+(`ci_` + the real database name) and `G3DT_S3_SILVER_DATA_DIR_CI` /
+`G3DT_S3_GOLD_DATA_DIR_CI` (`dbt_ci/` under the same buckets). Commit-
+triggered CI builds land there; every other target (default, local) and the
+release build keep the real, unprefixed names — so CI can never advance the
+warehouse's Iceberg snapshots that releases pin. The library enforces the
+other half: `find_db_for_model` always skips `ci_`-prefixed databases, so
+`g3dt release write` can never pin a release to a CI-build snapshot.
+
+**Snapshot pinning.** `AthenaValidationWriter.construct_json` /
+`AthenaGoldWriter.construct_json` honour a pre-set `snapshot_id` (reading the
+table `FOR VERSION AS OF` that snapshot) and only fetch the latest snapshot
+when unpinned — the contract the release-JSON export relies on for
+reproducible releases.
+
+**Concurrency.** `release_writer.run` processes models with a bounded thread
+pool (`max_workers`, default 8) and fails at the end naming every failed
+model (inserts are idempotent — re-run to fill the remainder). The S3
+writers (`write_release_jsons_to_s3`, `write_validation_json_to_s3`) accept
+`s3_client=` (pass one per worker thread) and `key_prefix=` (write a
+verification tree without touching real artifacts).
+
+**The validation gate.** `g3dt.validate.run_validation_gate(glue_database,
+athena_s3_output, aws_region, workgroup)` queries the latest
+`validation_id` in `full_validation_results` for REAL failures — the
+known-noise patterns in `VALIDATION_GATE_IGNORED_ERRORS` and synthetic
+studies are excluded. The validator Glue job fails when rows come back, so a
+green validation Step Function means schema-clean data; the operator loop is
+gate fails -> inspect the results table -> fix data -> re-run until green.
+`validate_pipeline` also accepts pre-computed loop-invariants
+(`schema=`/`resolver=`/`metadata_table=`) and `write_iceberg=False` so a
+multi-study caller resolves the schema once, lists the validation prefix
+once, and batches all studies into a single Iceberg INSERT.
+
+### Where the data dictionary comes from
+
+Composed from the env's inputs as
+`{dictionary_base_url}/{schema_repo}/refs/tags/{dictionary_version}/{dictionary_path}`.
+Only `schema_repo` and `dictionary_version` are required; `app/dictionary_base_url`
+and `app/dictionary_path` are optional and default to raw GitHub and the schema
+repo's conventional layout, so environments deployed before they existed keep
+working. `g3dt config show --env <env>` prints the composed URL.
+
+### Promoting a dictionary across environments
+
+A dictionary version is *content*, not infrastructure: it changes far more often
+than buckets or clusters do. Rather than a `cdk deploy` per environment per
+version, `dict pull`, `dict upload` and `dict deploy` all accept `--version`:
+
+```bash
+g3dt dict deploy --env test    --version v1.1.7
+g3dt dict deploy --env staging --version v1.1.7   # same tag, no cdk deploy
+```
+
+An override does not persist, so `config show` keeps reporting the declared
+version until the CDK config catches up — `g3dt config diff --env <env>` reports
+exactly that gap and exits 1, so it can gate CI.
+
+Synthetic data is only schema-valid against the dictionary that generated it, so
+`synth generate` records the dictionary version in each batch and `synth upload`
+refuses a batch that doesn't match the version being uploaded (override with
+`--allow-version-mismatch`).
+
 ## Development
 
 ```bash

gen3_dataops_toolkit-2.2.0/README.md

@@ -0,0 +1,159 @@
+# gen3-dataops-toolkit (`g3dt`)
+
+Operate Gen3 AWS data-pipeline environments from one pip-installable CLI.
+
+`g3dt` is the tooling half of the Gen3 DataOps platform: the
+[gen3-aws-data-pipeline](https://github.com/AustralianBioCommons/gen3-aws-data-pipeline)
+CDK app deploys a complete pipeline per project/environment and publishes every
+resource name to AWS SSM Parameter Store; `g3dt` resolves those names at
+runtime and gives operators one command surface for dictionary deploys,
+metadata upload/delete, indexd registration, EC2 job dispatch, and Kubernetes
+restarts. The dbt half of the platform lives in
+[gen3-dbt-template](https://github.com/AustralianBioCommons/gen3-dbt-template).
+
+**No AWS resource name is compiled into this package.** The same wheel
+operates any project: it is targeted purely by `--env`, the project's SSM tree
+(`/{project}/{env}/...`), and a tiny local bootstrap marker.
+
+## Install
+
+```bash
+pip install gen3-dataops-toolkit
+```
+
+## Bootstrap (the only local configuration)
+
+`g3dt` needs to know just the project and region — everything else comes from
+SSM. Create `~/.g3dt/g3dt.yaml`:
+
+```yaml
+project: etl                # your projectId
+region: ap-southeast-2
+default_env: test
+profiles:                   # optional: AWS named profile per env
+  test: etl_test            # (omit entirely on EC2/CodeBuild — ambient
+  staging: etl_staging      #  role credentials are used)
+studies:                    # optional: the project's study registry;
+  mystudy_test:             # alternatively upload it once per env to
+    project_id: MyStudy     # s3://<metadata-bucket>/config/studies.yaml
+    program_id: program1
+    s3_metadata_path: s3://my-bucket/metadata/mystudy/
+```
+
+Search order: `./g3dt.yaml` → `~/.g3dt/g3dt.yaml` → `/etc/g3dt/g3dt.yaml`
+(the EC2 job box's copy, written by CDK user-data). Env vars override:
+`G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV`.
+
+## Quick start
+
+```bash
+g3dt config envs                 # environments with a deployed SSM tree
+g3dt config show --env test      # every resolved name — the safety check
+g3dt ec2 up --env test           # start the env's job box (SSM-managed)
+g3dt metadata upload --study mystudy --env test --on ec2
+g3dt jobs logs <run-id> --follow # live logs; laptop can sleep, job keeps going
+g3dt ec2 down --env test         # or let the auto-stop alarm handle it
+g3dt docs                        # the full operations overview
+```
+
+## How configuration works
+
+There are exactly two kinds of configuration:
+
+- **INPUTS** — human-authored values, committed as
+  `config/<projectId>.<env>.json` in the CDK repo and read only by
+  `cdk deploy`. To change what an environment *declares*, edit that file and
+  redeploy — the value flows to SSM.
+- **OUTPUTS** — every resource name the CDK creates plus the mirrored Gen3
+  app facts, published to SSM under `/{project}/{env}/...` on deploy. `g3dt`
+  reads these live (cached one round-trip per invocation) and never stores
+  them locally.
+
+Because the CLI and the infrastructure read the same parameters, they cannot
+disagree — and because each environment has its own tree (including its own
+`ec2/instanceId`), running a job against the wrong environment's resources is
+structurally impossible.
+
+
+## CI isolation and the release contract
+
+**Only the dbt template's `ci` target is prefixed.** `g3dt config dbt-env`
+emits, alongside the real names, the CI-isolation variants the template's
+`ci` target consumes: `G3DT_DB_RAW_SILVER_CI` / `G3DT_DB_RAW_GOLD_CI`
+(`ci_` + the real database name) and `G3DT_S3_SILVER_DATA_DIR_CI` /
+`G3DT_S3_GOLD_DATA_DIR_CI` (`dbt_ci/` under the same buckets). Commit-
+triggered CI builds land there; every other target (default, local) and the
+release build keep the real, unprefixed names — so CI can never advance the
+warehouse's Iceberg snapshots that releases pin. The library enforces the
+other half: `find_db_for_model` always skips `ci_`-prefixed databases, so
+`g3dt release write` can never pin a release to a CI-build snapshot.
+
+**Snapshot pinning.** `AthenaValidationWriter.construct_json` /
+`AthenaGoldWriter.construct_json` honour a pre-set `snapshot_id` (reading the
+table `FOR VERSION AS OF` that snapshot) and only fetch the latest snapshot
+when unpinned — the contract the release-JSON export relies on for
+reproducible releases.
+
+**Concurrency.** `release_writer.run` processes models with a bounded thread
+pool (`max_workers`, default 8) and fails at the end naming every failed
+model (inserts are idempotent — re-run to fill the remainder). The S3
+writers (`write_release_jsons_to_s3`, `write_validation_json_to_s3`) accept
+`s3_client=` (pass one per worker thread) and `key_prefix=` (write a
+verification tree without touching real artifacts).
+
+**The validation gate.** `g3dt.validate.run_validation_gate(glue_database,
+athena_s3_output, aws_region, workgroup)` queries the latest
+`validation_id` in `full_validation_results` for REAL failures — the
+known-noise patterns in `VALIDATION_GATE_IGNORED_ERRORS` and synthetic
+studies are excluded. The validator Glue job fails when rows come back, so a
+green validation Step Function means schema-clean data; the operator loop is
+gate fails -> inspect the results table -> fix data -> re-run until green.
+`validate_pipeline` also accepts pre-computed loop-invariants
+(`schema=`/`resolver=`/`metadata_table=`) and `write_iceberg=False` so a
+multi-study caller resolves the schema once, lists the validation prefix
+once, and batches all studies into a single Iceberg INSERT.
+
+### Where the data dictionary comes from
+
+Composed from the env's inputs as
+`{dictionary_base_url}/{schema_repo}/refs/tags/{dictionary_version}/{dictionary_path}`.
+Only `schema_repo` and `dictionary_version` are required; `app/dictionary_base_url`
+and `app/dictionary_path` are optional and default to raw GitHub and the schema
+repo's conventional layout, so environments deployed before they existed keep
+working. `g3dt config show --env <env>` prints the composed URL.
+
+### Promoting a dictionary across environments
+
+A dictionary version is *content*, not infrastructure: it changes far more often
+than buckets or clusters do. Rather than a `cdk deploy` per environment per
+version, `dict pull`, `dict upload` and `dict deploy` all accept `--version`:
+
+```bash
+g3dt dict deploy --env test    --version v1.1.7
+g3dt dict deploy --env staging --version v1.1.7   # same tag, no cdk deploy
+```
+
+An override does not persist, so `config show` keeps reporting the declared
+version until the CDK config catches up — `g3dt config diff --env <env>` reports
+exactly that gap and exits 1, so it can gate CI.
+
+Synthetic data is only schema-valid against the dictionary that generated it, so
+`synth generate` records the dictionary version in each batch and `synth upload`
+refuses a batch that doesn't match the version being uploaded (override with
+`--allow-version-mismatch`).
+
+## Development
+
+```bash
+poetry install
+poetry run python3 -m pytest
+```
+
+## Provenance
+
+This toolkit was ported (working tree only) from
+[AustralianBioCommons/acdc-aws-etl-pipeline](https://github.com/AustralianBioCommons/acdc-aws-etl-pipeline),
+the ACDC ETL monolith, as part of the Gen3 DataOps platform refactor (2026).
+It starts at version **2.0.0**; versions ≤ 1.2.0 on PyPI are the legacy
+`acdc_aws_etl_pipeline` package, which continues to operate the legacy ACDC
+pipeline unchanged.

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "gen3-dataops-toolkit"
-version = "2.1.2"
+version = "2.2.0"
 description = "Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments"
 authors = ["JoshuaHarris391 <harjo391@gmail.com>"]
 readme = "README.md"

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.2.0}/src/g3dt/cli/config_cmds.py

@@ -86,6 +86,10 @@ def show(
     typer.echo(f"  aws_profile        : {e.aws_profile or '(ambient credentials)'}")
     typer.echo(f"  aws_secret_name    : {e.aws_secret_name}")
     typer.echo(f"  dictionary_version : {e.dictionary_version}")
+    # The composed URL, not the three parts: "which dictionary will this env
+    # actually fetch?" now spans schema_repo plus two optional inputs, so showing
+    # the resolved result is what makes the answer checkable before a deploy.
+    typer.echo(f"  dictionary_url     : {config.dictionary_url(e)}")
     typer.echo(f"  schema_s3_uri      : {e.schema_s3_uri}")
     typer.echo(f"  schema_repo        : {e.schema_repo}")
     typer.echo(f"  domain             : {e.domain}")
@@ -208,15 +212,24 @@ def dbt_env(
         typer.secho(str(exc), fg=typer.colors.RED, err=True)
         raise typer.Exit(1)
 
+    raw_silver_db = rc.get("glue/db/rawSilver")
+    raw_gold_db = rc.get("glue/db/rawGold")
     values = {
         "G3DT_REGION": rc.region,
         "G3DT_ATHENA_WORKGROUP": rc.athena_workgroup,
         "G3DT_ATHENA_OUTPUT": rc.athena_output_location,
         "G3DT_DB_RAW_BRONZE": rc.get("glue/db/rawBronze"),
-        "G3DT_DB_RAW_SILVER": rc.get("glue/db/rawSilver"),
-        "G3DT_DB_RAW_GOLD": rc.get("glue/db/rawGold"),
+        "G3DT_DB_RAW_SILVER": raw_silver_db,
+        "G3DT_DB_RAW_GOLD": raw_gold_db,
         "G3DT_S3_SILVER_DATA_DIR": f"s3://{rc.get('buckets/rawSilver')}/dbt/",
         "G3DT_S3_GOLD_DATA_DIR": f"s3://{rc.get('buckets/rawGold')}/dbt/",
+        # CI isolation: the dbt template's `ci` target builds into these
+        # instead — same grammar as the CDK's ci_ databases, same buckets
+        # under a dbt_ci/ prefix. Real names above are never prefixed.
+        "G3DT_DB_RAW_SILVER_CI": f"ci_{raw_silver_db}" if raw_silver_db else None,
+        "G3DT_DB_RAW_GOLD_CI": f"ci_{raw_gold_db}" if raw_gold_db else None,
+        "G3DT_S3_SILVER_DATA_DIR_CI": f"s3://{rc.get('buckets/rawSilver')}/dbt_ci/",
+        "G3DT_S3_GOLD_DATA_DIR_CI": f"s3://{rc.get('buckets/rawGold')}/dbt_ci/",
     }
     if profile:
         # A named profile means a laptop run: select the dbt target that
@@ -239,6 +252,11 @@ def set_value(
     settings — dictionary_version, domain, buckets, ... — are CDK INPUTS: edit
     config/<project>.<env>.json in gen3-aws-data-pipeline and `cdk deploy`;
     the values flow to SSM, which is what every consumer reads.
+
+    The one exception is per-invocation: `dict pull/upload/deploy --version`
+    fetches a different dictionary tag without a redeploy (for promoting one
+    dictionary across environments). That does not persist — `config diff`
+    reports the gap until the CDK config is updated to match.
     """
     try:
         old, new, path = config.set_marker_value(key, value)

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.2.0}/src/g3dt/cli/dict_cmds.py

@@ -10,22 +10,16 @@ can point at its own dictionary repo. Downloads land in ``~/.g3dt/schemas/``
 from __future__ import annotations
 
 from pathlib import Path
+from typing import Optional
 
 import typer
 
-from g3dt.config import script_env
+from g3dt.config import dictionary_filename, dictionary_url, script_env
 from g3dt.cli._internal import runner
 from g3dt.cli._internal.resolve import env_of
 
 app = typer.Typer(no_args_is_help=True, help="Data dictionary operations (local).")
 
-#: Raw-GitHub URL template; the trailing path is the schema repo's layout
-#: convention (see AustralianBioCommons/acdc-schema-json), not a project name.
-_DICT_URL_TMPL = (
-    "https://raw.githubusercontent.com/{repo}/"
-    "refs/tags/{version}/dictionary/prod_dict/acdc_schema.json"
-)
-
 SCHEMA_DIR = Path("~/.g3dt/schemas").expanduser()
 
 
@@ -33,8 +27,23 @@ def _version(env_cfg, override):
     return override or env_cfg.dictionary_version
 
 
-def dict_url(env_cfg, version: str) -> str:
-    return _DICT_URL_TMPL.format(repo=env_cfg.schema_repo, version=version)
+def warn_if_overridden(env_cfg, version: Optional[str]) -> None:
+    """Say so, loudly, when the deployed version isn't the one SSM declares.
+
+    An override is legitimate -- promoting one dictionary through environments
+    shouldn't need a `cdk deploy` per env -- but it leaves SSM (and so
+    `g3dt config show`) describing a different version than the bucket holds.
+    Naming both keeps that discoverable instead of silent; `g3dt config diff`
+    is what reconciles it once the CDK config catches up.
+    """
+    if version and version != env_cfg.dictionary_version:
+        typer.secho(
+            f"Overriding the declared version: SSM says "
+            f"{env_cfg.dictionary_version}, using {version}. `config show` will "
+            f"keep reporting {env_cfg.dictionary_version} until "
+            f"config/<project>.{env_cfg.name}.json is updated and redeployed.",
+            fg=typer.colors.YELLOW,
+        )
 
 
 @app.command()
@@ -46,15 +55,23 @@ def pull(
 ) -> None:
     """Download the dictionary JSON from the env's schema repo.
 
+    Where it comes from is config: `app/schema_repo` plus the optional
+    `app/dictionary_base_url` and `app/dictionary_path`.
+
     Examples:
       g3dt dict pull --env test
       g3dt dict pull --env staging --version v1.1.5
     """
     e = env_of(env)
-    url = dict_url(e, _version(e, version))
+    warn_if_overridden(e, version)
+    v = _version(e, version)
     runner.run(
-        runner.bash_script("services/dictionary/pull_dict.sh", url),
-        env=script_env(e),
+        runner.bash_script(
+            "services/dictionary/pull_dict.sh",
+            dictionary_url(e, v),
+            dictionary_filename(e, v),
+        ),
+        env=script_env(e, v),
     )
 
 
@@ -67,36 +84,46 @@ def upload(
 ) -> None:
     """Upload the (already pulled) dictionary JSON to the env's S3 location."""
     e = env_of(env)
+    warn_if_overridden(e, version)
     v = _version(e, version)
-    local_file = str(SCHEMA_DIR / f"acdc_schema_{v}.json")
+    local_file = str(SCHEMA_DIR / dictionary_filename(e, v))
     s3_uri = f"s3://{e.schema_s3_uri}"
     args = [local_file, s3_uri]
     if e.aws_profile:
         args.append(e.aws_profile)
     runner.run(
         runner.python_script("services/dictionary/upload_dictionary.py", *args),
-        env=script_env(e),
+        env=script_env(e, v),
     )
 
 
 @app.command()
 def deploy(
     env: str = typer.Option(..., "--env", "-e", help="Environment, e.g. test."),
+    version: str = typer.Option(
+        None, "--version", help="Dictionary git tag (default: the env's version)."
+    ),
 ) -> None:
     """Pull + upload the dictionary and restart Gen3 schema microservices.
 
     Wraps services/dictionary/deploy_dd.sh. Requires an interactive ArgoCD SSO
     login, so it runs locally only.
 
-    The deployed version is the env's `dictionary_version` — a CDK INPUT. To
-    change it, edit config/<project>.<env>.json in gen3-aws-data-pipeline and
-    `cdk deploy` (the value flows to SSM), then re-run this command.
+    The version defaults to the env's `dictionary_version`, a CDK INPUT: edit
+    config/<project>.<env>.json in gen3-aws-data-pipeline and `cdk deploy` to
+    change what an env declares. Pass --version to deploy a different tag now
+    without that round trip — which is how one dictionary gets promoted across
+    environments. `g3dt config diff` reports the resulting drift until the CDK
+    config catches up.
 
     Examples:
       g3dt dict deploy --env test
+      g3dt dict deploy --env test --version v1.1.7
+      g3dt dict deploy --env staging --version v1.1.7   # promote the same tag
     """
     e = env_of(env)
+    warn_if_overridden(e, version)
     runner.run(
         runner.bash_script("services/dictionary/deploy_dd.sh", env),
-        env=script_env(e),
+        env=script_env(e, _version(e, version)),
     )

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.2.0}/src/g3dt/cli/main.py

@@ -73,6 +73,16 @@ Typical release runbook (staging shown; repeat for prod with care)
   3. g3dt jobs logs <run-id> --follow
   4. g3dt k8s restart-etl --env staging
 
+Promoting one dictionary across environments
+  The source repo/path are env inputs; usually only the tag changes, and it
+  changes far more often than infrastructure does. So --version deploys a tag
+  without a `cdk deploy` per env:
+    g3dt dict deploy --env test    --version v1.1.7
+    g3dt dict deploy --env staging --version v1.1.7
+  The override does not persist: `g3dt config show` keeps reporting the declared
+  version, and `g3dt config diff --env <env>` reports the gap (exit 1) until
+  config/<project>.<env>.json is updated to match.
+
 Data releases (the dbt pipeline; see the project's dbt repo)
   git tag data-v1.4.0 && git push origin data-v1.4.0
   g3dt pipeline status --env staging           which stage is running/failed
@@ -81,6 +91,9 @@ Data releases (the dbt pipeline; see the project's dbt repo)
 
 Synthetic data (test only, all local)
   g3dt synth deploy --env test
+  Batches are only schema-valid against the dictionary that generated them, so
+  each one records its dictionary version and `g3dt synth upload` refuses a
+  batch that does not match the version being uploaded.
 
 EC2 / SSM prerequisites
   - The env's job box is created by the CDK (ec2-job-runner stack): SSM-managed,

{gen3_dataops_toolkit-2.1.2 → gen3_dataops_toolkit-2.2.0}/src/g3dt/cli/synth.py

@@ -15,14 +15,21 @@ working directory (``LLM_PROVIDER`` / ``LLM_MODEL`` / ``LLM_API_KEY_FILE``).
 """
 from __future__ import annotations
 
+import json
 from pathlib import Path
+from typing import Optional
 
 import typer
 
-from g3dt.config import script_env
+from g3dt.config import (
+    dictionary_filename,
+    dictionary_url,
+    dictionary_version_of,
+    script_env,
+)
 from g3dt.cli._internal import runner, safety
 from g3dt.cli._internal.resolve import env_of
-from g3dt.cli.dict_cmds import SCHEMA_DIR, dict_url
+from g3dt.cli.dict_cmds import SCHEMA_DIR, warn_if_overridden
 
 app = typer.Typer(
     no_args_is_help=True,
@@ -31,6 +38,72 @@ app = typer.Typer(
 
 SYNTH_DIR = Path("~/.g3dt/synth_metadata").expanduser()
 
+#: Written into each generated batch so `synth upload` can tell which dictionary
+#: produced it. A batch is only valid against that dictionary, and the directory
+#: name alone cannot be trusted: --schema and --version are separate options, so
+#: the label is exactly the thing that can be wrong.
+PROVENANCE_FILE = ".g3dt-provenance.json"
+
+
+def _write_provenance(batch_dir: Path, e, ver: str, schema_path: str) -> None:
+    """Record which dictionary produced this batch."""
+    batch_dir.mkdir(parents=True, exist_ok=True)
+    (batch_dir / PROVENANCE_FILE).write_text(
+        json.dumps(
+            {
+                "dictionary_version": ver,
+                "dictionary_url": dictionary_url(e, ver),
+                "schema_file": Path(schema_path).name,
+                "generated_for_env": e.name,
+            },
+            indent=2,
+        )
+        + "\n"
+    )
+
+
+def _check_batch_matches_env(batch_dir: Path, e, ver: str, allow_mismatch: bool) -> None:
+    """Refuse to upload a batch generated against a different dictionary.
+
+    Synthetic records are only schema-valid against the dictionary version used
+    to generate them, so pushing a v1.0.0 batch into an environment running
+    v1.1.0 produces data Gen3 may reject or, worse, silently accept as wrong.
+    The environment's own version is the authority here.
+
+    A batch with no provenance file predates this check and cannot be attributed
+    after the fact, so it warns rather than blocks.
+    """
+    marker = batch_dir / PROVENANCE_FILE
+    if not marker.is_file():
+        typer.secho(
+            f"No provenance in {batch_dir} — cannot confirm which dictionary "
+            f"generated it. Proceeding; regenerate the batch to record it.",
+            fg=typer.colors.YELLOW,
+        )
+        return
+    batch_version = json.loads(marker.read_text()).get("dictionary_version")
+    if batch_version == ver:
+        return
+    if allow_mismatch:
+        typer.secho(
+            f"Version mismatch allowed by flag: batch was generated against "
+            f"{batch_version}, uploading to {e.name} which runs {ver}.",
+            fg=typer.colors.YELLOW,
+        )
+        return
+    typer.secho(
+        f"Refusing to upload: this batch was generated against dictionary "
+        f"{batch_version}, but '{e.name}' runs {ver}. Synthetic data is only "
+        f"valid against the dictionary that produced it.\n"
+        f"Either regenerate for {ver} (g3dt synth generate ... --version {ver}), "
+        f"deploy {batch_version} to '{e.name}' first "
+        f"(g3dt dict deploy --env {e.name} --version {batch_version}), "
+        f"or override with --allow-version-mismatch.",
+        fg=typer.colors.RED,
+        err=True,
+    )
+    raise typer.Exit(1)
+
 
 @app.command()
 def deploy(
@@ -113,14 +186,34 @@ def generate(
             raise typer.Exit(1)
 
     ver = version or e.dictionary_version
-    schema_path = schema or str(SCHEMA_DIR / f"acdc_schema_{ver}.json")
+    schema_path = schema or str(SCHEMA_DIR / dictionary_filename(e, ver))
+
+    # A batch is only valid against the dictionary that generated it, and the
+    # output directory is named for `ver` -- so an explicit --schema carrying a
+    # different version stamp would silently mislabel the whole batch.
+    if schema:
+        schema_version = dictionary_version_of(Path(schema).name)
+        if schema_version and schema_version != ver:
+            typer.secho(
+                f"--schema is {Path(schema).name} (dictionary {schema_version}) "
+                f"but the batch would be labelled {ver}. Synthetic data is only "
+                f"valid against the dictionary that produced it — pass "
+                f"--version {schema_version}, or drop --schema to use {ver}.",
+                fg=typer.colors.RED,
+                err=True,
+            )
+            raise typer.Exit(1)
 
     # Ensure the schema is available locally; pull it if missing.
     if not Path(schema_path).exists():
         typer.secho(f"Schema not found locally; pulling {ver}...", fg=typer.colors.YELLOW)
         runner.run(
-            runner.bash_script("services/dictionary/pull_dict.sh", dict_url(e, ver)),
-            env=script_env(e),
+            runner.bash_script(
+                "services/dictionary/pull_dict.sh",
+                dictionary_url(e, ver),
+                dictionary_filename(e, ver),
+            ),
+            env=script_env(e, ver),
         )
 
     effective_provider = "llm" if llm else provider
@@ -138,8 +231,11 @@ def generate(
         runner.bash_script(
             "services/synthetic_data/generate_synth_metadata.sh", *args
         ),
-        env=script_env(e),
+        env=script_env(e, ver),
     )
+    # Only after a successful generate: runner.run raises on failure, so a
+    # half-written batch is never stamped as valid.
+    _write_provenance(SYNTH_DIR / ver, e, ver, schema_path)
 
 
 @app.command()
@@ -150,11 +246,26 @@ def upload(
     version: str = typer.Option(
         None, "--version", help="Dictionary version dir (default: the env's version)."
     ),
+    allow_version_mismatch: bool = typer.Option(
+        False,
+        "--allow-version-mismatch",
+        help="Upload even if the batch was generated against another dictionary.",
+    ),
 ) -> None:
-    """Upload generated synthetic metadata to Gen3 (reads local files)."""
+    """Upload generated synthetic metadata to Gen3 (reads local files).
+
+    The batch is checked against the dictionary version being uploaded for:
+    synthetic records are only schema-valid against the dictionary that produced
+    them. Defaults to the env's declared version, so the common case verifies
+    against what the environment actually runs.
+    """
     e = env_of(env)
     safety.confirm_prod_strict("synthetic metadata upload", env)
+    # A promoted dictionary (dict deploy --version) leaves SSM behind, so an
+    # explicit --version here is how you say "this env really runs that tag".
+    warn_if_overridden(e, version)
     v = version or e.dictionary_version
+    _check_batch_matches_env(SYNTH_DIR / v, e, v, allow_version_mismatch)
     base_dir = str(SYNTH_DIR / v) + "/"
     args = ["--base-dir", base_dir, "--aws-secret-name", e.aws_secret_name]
     if e.aws_profile:
@@ -163,7 +274,7 @@ def upload(
         runner.python_script(
             "services/synthetic_data/upload_synth_metadata_sheepdog.py", *args
         ),
-        env=script_env(e),
+        env=script_env(e, v),
     )