docker-devkit 0.1.17__tar.gz → 0.1.18__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/AGENTS.md +17 -4
  2. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/PKG-INFO +1 -1
  3. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/README.md +4 -0
  4. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/pyproject.toml +2 -1
  5. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/build_docker.py +12 -1
  6. docker_devkit-0.1.18/src/docker_devkit/mirror.py +115 -0
  7. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/up.py +25 -2
  8. docker_devkit-0.1.18/tests/test_mirror.py +356 -0
  9. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/.gitattributes +0 -0
  10. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/.github/workflows/ci.yml +0 -0
  11. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/.github/workflows/release.yml +0 -0
  12. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/.gitignore +0 -0
  13. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/.python-version +0 -0
  14. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/CLAUDE.md +0 -0
  15. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/LICENSE +0 -0
  16. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/NOTICE +0 -0
  17. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/plan.md +0 -0
  18. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/publish-config.json +0 -0
  19. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/ruff.base.toml +0 -0
  20. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/ruff.toml +0 -0
  21. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/__init__.py +0 -0
  22. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/context_sha.py +0 -0
  23. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/detect_gpu.py +0 -0
  24. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/documents.py +0 -0
  25. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/down.py +0 -0
  26. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/image_refs.py +0 -0
  27. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/lifecycle.py +0 -0
  28. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/modes.py +0 -0
  29. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/py.typed +0 -0
  30. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/src/docker_devkit/score_generate.py +0 -0
  31. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/tests/test_context_sha.py +0 -0
  32. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/tests/test_default_targets.py +0 -0
  33. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/tests/test_documents.py +0 -0
  34. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/tests/test_image_refs.py +0 -0
  35. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/tests/test_lifecycle.py +0 -0
  36. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/tests/test_score_generate.py +0 -0
  37. {docker_devkit-0.1.17 → docker_devkit-0.1.18}/uv.lock +0 -0
@@ -2,7 +2,7 @@
2
2
 
3
3
  ## What this is
4
4
 
5
- `docker-devkit` owns the Docker-stack lifecycle commands — `up`, `down`, `build` — plus the helpers they share (`detect_gpu`, `modes`, `context_sha`). It is a small, generic package that any repo shipping a compose graph can depend on to get the same bring-up/tear-down / cross-build flow, with the compose assembly declared per consumer in `[tool.docker-devkit.lifecycle]` so a repo that authors its own images and a repo that only OCI-includes an upstream artifact both work through the same commands.
5
+ `docker-devkit` owns the Docker-stack lifecycle commands — `up`, `down`, `build`, `mirror` — plus the helpers they share (`detect_gpu`, `modes`, `context_sha`). It is a small, generic package that any repo shipping a compose graph can depend on to get the same bring-up/tear-down / cross-build flow, with the compose assembly declared per consumer in `[tool.docker-devkit.lifecycle]` so a repo that authors its own images and a repo that only OCI-includes an upstream artifact both work through the same commands.
6
6
 
7
7
  The package is `docker_devkit` (src-layout under `src/docker_devkit/`); all dependencies resolve from PyPI (`bashrun>=0.1.0`; git-source pins only in scratch branches testing unreleased changes). The repo and package renamed twice on the way here — `stack-lifecycle` → `stack-toolkit` (2026-09-20, before any publish) and `stack-toolkit` → `docker-devkit` (2026-09-21, member of the `-devkit` family). The PyPI identity is fresh: `docker-devkit` starts its own tag ledger at `0.1.0`; the terminal `stack-toolkit` distributions (≤0.1.1) are deprecation signposts pointing here, not this package's history.
8
8
 
@@ -12,13 +12,13 @@ Publishing rides `release.yml`, triggered by a successful CI run on a `main` pus
12
12
 
13
13
  ## Shape
14
14
 
15
- Entry points (`[project.scripts]`): `up` → `up.py:app`, `down` → `down.py:app`, `build` → `build_docker.py:app`, `generate-score` → `score_generate.py:app`. All accept `--help`.
15
+ Entry points (`[project.scripts]`): `up` → `up.py:app`, `down` → `down.py:app`, `build` → `build_docker.py:app`, `mirror` → `mirror.py:app`, `generate-score` → `score_generate.py:app`. All accept `--help`.
16
16
 
17
17
  | Module | Role |
18
18
  |---|---|
19
- | `up.py` | `docker compose up`. Flags: `--attached`/`-a`, `--quiet-pull`/`-q`, `--build`, `--gpu auto\|cuda\|rocm\|none`, `--dev`. |
19
+ | `up.py` | `docker compose up`. Flags: `--attached`/`-a`, `--quiet-pull`/`-q`, `--build`, `--gpu auto\|cuda\|rocm\|none`, `--dev`, `--allow-upstream-fallback`. |
20
20
  | `down.py` | `docker compose down`. Flags: `--volumes`/`-v` (also removes named volumes), `--gpu`. |
21
- | `build_docker.py` | `run_build()` + the `build` command — cross-builds all images per the resolved image manifest, writing the paired lock (one `NAME=path:tag@sha256:…` entry per `x-base-images` declaration; `@digest`-pinned declarations pass through unresolved). `--mode ci` hard-requires `x-registry-cache` in the manifest. |
21
+ | `build_docker.py` | `run_build()` + the `build` command — cross-builds all images per the resolved image manifest, writing the paired lock (one `NAME=path:tag@sha256:…` entry per `x-base-images` declaration; `@digest`-pinned declarations pass through unresolved; mirror-prefixed declarations resolve their digest against the upstream reference). `--mode ci` hard-requires `x-registry-cache` in the manifest. |
22
22
  | `score_generate.py` | `generate-score` — deterministic Score-manifest generation from a consumer's `stack/score/` sources: `${ALL_CAPS}` placeholder expansion into tempdir-rendered workload copies, idempotent `score-compose`/`score-k8s` init (state preserved across runs), and a determinism suite (document sort, state-path normalisation, 0644 modes, LF newlines). Consumer policy — workloads, provisioners, patch templates, output paths, storage classes, publishes, tool versions — is declared in the consumer's root `pyproject.toml` under `[tool.docker-devkit.generate-score]` (`ScoreConfig`); `load_score_config()` reads it, `ensure_score_tools()` fetches the pinned score-compose/score-k8s release binaries (for CI staleness gates). Output and state paths resolve against the repo root: the compose half runs in `score_dir` (its `.score-compose` state stays there, gitignored), the k8s half runs in the configured `k8s_state` parent so the committed `.score-k8s` state lands in the generated tier. `--local` writes the k8s artifact to the configured gitignored local path with the local storage class. |
23
23
  | `lifecycle.py` | `LifecycleConfig` — the `[tool.docker-devkit.lifecycle]` declaration (`files`: compose files in layering order with `{gpu}` templating; `dev_file`: optional dev overlay); `load_lifecycle_config()` (walks `tool → docker-devkit → lifecycle`, `None` when absent), `resolve_manifest()` / `require_manifest()` (the raising form) / `resolve_lock()` (the image manifest and its lock are a named pair — `workloads/images.yml` + `workloads/images.lock`, with the legacy root `compose.bake.yml` + `.env.lock` pair still resolved so repos can upgrade before moving files), `enforce_bake_declaration()` (a manifest under either name requires a declaration), `expand_compose_files()` (assembles the `-f` list; a `{gpu}` entry is dropped when the GPU resolves to `none`; `dev_file` appended when a dev overlay applies). |
24
24
  | `detect_gpu.py` | `detect_gpu()` → `cuda`/`rocm`/`none` from host devices; the `Gpu` literal. |
@@ -26,6 +26,7 @@ Entry points (`[project.scripts]`): `up` → `up.py:app`, `down` → `down.py:ap
26
26
  | `context_sha.py` | `compute_service_shas(repo_root, bake: BakeDocument)` — per-service `tree-<hash>` image tags over the `.dockerignore`-allowlisted git tree. The workload-tree root is derived as the single top-level directory containing every tagged service's Dockerfile (a bake graph spanning several top-level directories is a loud error); files under it that belong to no tagged service are excluded from every SHA rather than shared into all of them. |
27
27
  | `documents.py` | `BakeDocument` + `parse_bake(path)` — the typed bake-file model every consumer of bake data goes through (build, up, down, generate-score, the reference collectors). A field validator enforces at parse time that every `x-base-images` value is `path:tag` or `path@digest` (a `${…}` hole or a bare path is a validation error, killing the silent-unpinned `FROM` → `:latest` failure mode). `digest_pinned(reference)` is the shared tail test. |
28
28
  | `image_refs.py` | Image-reference collection, split by source. `declared_references(root)` enumerates bake `x-base-images` declarations from `workloads/images.yml` and the legacy `compose*.bake.yml` names (parsed and validated via `BakeDocument`) — the single source consumer mirroring consumes. `stray_references(root, …globs)` scans Dockerfile `FROM`, Dockerfile `COPY --from=` (slash-bearing captures only, so build-stage names are skipped), and yaml `image:` lines — hand-written literal refs that bypass the declaration system — across the `workloads/` and `docker/` Dockerfile trees and the `stack/score/` + `score/` source trees; `unpinned_references(root)` polices exactly those strays (every slash-bearing reference must carry a tag, a digest, or a trailing `${build-arg}` suffix — consumer CIs fail on its result). `bake_base_image_refs(bake: BakeDocument)` adapts declarations to `ImageReference` pairs; `version_coupling_violations(root, couplings)` (each `VersionCoupling` names a pyproject-declared version and the `VersionSite` regexes that must restate it — bake `x-base-images` values or raw file text — so non-derivable couplings like uv `required-version` ↔ uv-base-image tag can't drift silently; the mechanism lives here, the registry in consumer repos' preflight) is unchanged. Remote digest resolution goes through `imagetools inspect`. Compose files are parsed by nothing in this package. |
29
+ | `mirror.py` | The mirror namespace's config, inversion, and verbs. `MirrorConfig` (`[tool.docker-devkit.mirror]`, key `prefix`), `load_mirror_config()` (walks `tool → docker-devkit → mirror`, `None` when absent), `is_mirrored`/`upstream_ref` (prefix inversion preserving the `:tag@sha256:…` tail), `reference_exists` (imagetools existence probe honoring docker config creds), `mirror_targets` (declared refs filtered by prefix, sorted), the `mirror` verb (crane install + per-target `crane copy`), and `up`'s fallback helpers (`upstream_fallback`, `write_fallback_overlay`). Constraints in "Mirror namespace and upstream fallback". |
29
30
 
30
31
  ## Constraints
31
32
 
@@ -47,6 +48,18 @@ Entry points (`[project.scripts]`): `up` → `up.py:app`, `down` → `down.py:ap
47
48
 
48
49
  `build` injects the `tree-<hash>` tags of locally-built images into the environment rather than writing them anywhere; `up`/`down` do the same via `compute_service_shas`. The lock stays a pure third-party pin file — a CI invariant forbids built-image digests in the committed, reviewed lock.
49
50
 
51
+ ### Mirror namespace and upstream fallback
52
+
53
+ **Context.** Third-party images are consumed through an org-owned mirror namespace (`ghcr.io/outernet-foundation/mirror/<upstream-host>/<path>…`) instead of pulling upstream registries directly: manifests at upstream get GC'd, and an upstream yank once broke external cloners — digest pins alone don't save you when the manifest is gone. Durability for strangers beats upstream egress: strangers, CI, and prod always pull the mirror. Populating it is CI's job, single writer — engineers never push to the org namespace, and error text must never teach them to (no "run this populate command" hints).
54
+
55
+ **Constraint.** The mirror is declared, not inferred: `[tool.docker-devkit.mirror]` with `prefix` in the consumer's root pyproject. Absent table → all mirror behavior off; `up`/`build` behave exactly as they did before it existed. Under the table:
56
+
57
+ - `build` resolves lock digests for mirror-prefixed declarations against the upstream reference (prefix stripped, tag/digest tail preserved) and records them under the mirror name. Valid because `crane copy` produces byte-identical manifests — upstream and mirror digests are the same digest; a manifest re-wrap (`docker buildx imagetools`-shaped tooling) would change the top-level digest and break lock equality. This makes declare → re-lock walkable before CI has mirrored anything; lock lines are never hand-authored.
58
+ - `up` fails closed: after lock resolution it probes each mirror-prefixed lock entry; a missing mirror image is a hard `RuntimeError`, not a warning — people ignore warnings, and warnings are errors someone was too lazy to fix. The only escape is the per-run `--allow-upstream-fallback` flag. Under the flag, missing entries become digest-preserving address substitutions (same bytes from the upstream host), printed as plain statements — the operator asked for fallback, the run reports what it did. The rewrite is authoritative on both consumption paths: an appended `--env-file` overlay (compose's later env files win) plus `os.environ` before `build`'s bake (shell env wins over env files; `run_build` re-applies the overlay after its own lock update so bake is never clobbered). CI never passes the flag — the mirror job runs before anything needs the images.
59
+ - The `mirror` verb (crane install from a pinned release, per-target `crane copy`, count footer) uses bashrun + plain prints, deliberately not ci-devkit's `ci_step`: docker-devkit and ci-devkit are siblings in the dependency DAG and the mirror verb is the only thing that would ever draw that edge. Zero new dependencies is a hard rule.
60
+
61
+ **Consequences.** Non-mirror lock entries (e.g. `placeframe/*` publishes) are not probed and keep failing naturally at pull. There is no config-level ban on the flag: an air-gap fails closed anyway (no route to upstream — the pull dies with or without it).
62
+
50
63
  ### Package split rationale
51
64
 
52
65
  Lifecycle (`up`/`down`/`build`) and codegen (client generation, datamodel generation, workspace locking) have disjoint dependency surfaces. Bundling them would force a consumer that only brings a stack up to install `sqlacodegen`, `psycopg`, `datamodel-code-generator`. Keeping the lifecycle in its own small package lets consumers git-reference just this without dragging the codegen surface behind it.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: docker-devkit
3
- Version: 0.1.17
3
+ Version: 0.1.18
4
4
  Summary: Docker-stack lifecycle commands (up/down/build) driven by a pyproject-declared compose assembly
5
5
  License-File: LICENSE
6
6
  License-File: NOTICE
@@ -21,14 +21,18 @@ uv run up # docker compose up (auto-detects GPU; assembles
21
21
  uv run up --dev # layer the declared dev overlay (bind-mount/debug bring-up)
22
22
  uv run up --build # build images locally first (requires workloads/images.yml)
23
23
  uv run up --gpu none # override GPU auto-detection
24
+ uv run up --allow-upstream-fallback # per-run escape: pull unmirrored lock entries from upstream at the same digest
24
25
  uv run down # docker compose down
25
26
  uv run down -v # also remove named volumes
26
27
  uv run build # cross-build all images per workloads/images.yml
27
28
  uv run build --lock-only # refresh the image lock without building
29
+ uv run mirror # populate the org mirror namespace from mirror-prefixed declarations (CI)
28
30
  uv run generate-score # regenerate Score compose/k8s manifests per [tool.docker-devkit.generate-score]
29
31
  uv run generate-score --local # local-cluster variant (local storage class, gitignored output)
30
32
  ```
31
33
 
34
+ Repos that consume third-party images through the org mirror namespace declare `[tool.docker-devkit.mirror]` (`prefix = "ghcr.io/outernet-foundation/mirror"`); `up` then fails closed on any lock entry not mirrored yet, and `build` resolves lock digests against upstream. See the "Mirror namespace and upstream fallback" section in [`AGENTS.md`](./AGENTS.md).
35
+
32
36
  ## Consuming from another repo
33
37
 
34
38
  Install from PyPI and use its entry points:
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "docker-devkit"
3
- version = "0.1.17"
3
+ version = "0.1.18"
4
4
  description = "Docker-stack lifecycle commands (up/down/build) driven by a pyproject-declared compose assembly"
5
5
  requires-python = ">=3.13"
6
6
  dependencies = [
@@ -28,6 +28,7 @@ build-backend = "hatchling.build"
28
28
  up = "docker_devkit.up:app"
29
29
  down = "docker_devkit.down:app"
30
30
  build = "docker_devkit.build_docker:app"
31
+ mirror = "docker_devkit.mirror:app"
31
32
  generate-score = "docker_devkit.score_generate:app"
32
33
 
33
34
  [tool.hatch.build.targets.wheel]
@@ -17,6 +17,7 @@ from .context_sha import compute_service_shas
17
17
 
18
18
  from .image_refs import bake_base_image_refs, resolve_remote_digest
19
19
  from .lifecycle import require_manifest, resolve_lock
20
+ from .mirror import is_mirrored, load_mirror_config, upstream_ref
20
21
  from .modes import parse_env_file
21
22
 
22
23
 
@@ -71,10 +72,12 @@ def run_build(
71
72
  gpu_only: bool = False,
72
73
  no_cache: bool = False,
73
74
  targets_opt: list[str] | None = None,
75
+ env_overlay: dict[str, str] | None = None,
74
76
  ) -> None:
75
77
  root = Path.cwd()
76
78
  manifest = require_manifest(root)
77
79
  lock_file = resolve_lock(root, manifest)
80
+ mirror_config = load_mirror_config(root)
78
81
 
79
82
  bake = parse_bake(manifest)
80
83
 
@@ -103,6 +106,9 @@ def run_build(
103
106
  gpu = detect_gpu()
104
107
 
105
108
  # One lock entry per declared third-party image: NAME=path:tag@sha256:…
109
+ # Mirror-prefixed declarations resolve their digest against upstream (crane copies
110
+ # are byte-identical, so the digest is valid under both names) and record it under
111
+ # the mirror name — declaring and re-locking works before CI has mirrored anything.
106
112
  for occurrence in bake_base_image_refs(bake):
107
113
  if digest_pinned(occurrence.reference):
108
114
  lock_data[occurrence.name] = occurrence.reference
@@ -113,7 +119,10 @@ def run_build(
113
119
  and lock_data[occurrence.name].startswith(occurrence.reference + "@")
114
120
  ):
115
121
  continue
116
- lock_data[occurrence.name] = f"{occurrence.reference}@{resolve_remote_digest(occurrence.reference)}"
122
+ inspect_target = occurrence.reference
123
+ if mirror_config is not None and is_mirrored(occurrence.reference, mirror_config.prefix):
124
+ inspect_target = upstream_ref(occurrence.reference, mirror_config.prefix)
125
+ lock_data[occurrence.name] = f"{occurrence.reference}@{resolve_remote_digest(inspect_target)}"
117
126
 
118
127
  # Update main lock file
119
128
  lock_file.write_text(
@@ -126,6 +135,8 @@ def run_build(
126
135
 
127
136
  # Update environment with external dependency image digests
128
137
  os.environ.update(lock_data)
138
+ if env_overlay:
139
+ os.environ.update(env_overlay)
129
140
 
130
141
  # Build command arguments
131
142
  command_arguments: list[str] = []
@@ -0,0 +1,115 @@
1
+ from __future__ import annotations
2
+
3
+ import os
4
+ import shlex
5
+ import tempfile
6
+ import tomllib
7
+ from pathlib import Path
8
+
9
+ import typer
10
+ from bashrun.bash import bash, bash_check
11
+ from pydantic import BaseModel, ConfigDict, field_validator
12
+
13
+ from .image_refs import ImageReference, declared_references
14
+
15
+ app = typer.Typer(add_completion=False, pretty_exceptions_show_locals=False)
16
+
17
+ CRANE_VERSION = "v0.22.1"
18
+
19
+
20
+ class MirrorConfig(BaseModel):
21
+ model_config = ConfigDict(extra="forbid")
22
+
23
+ prefix: str
24
+
25
+ @field_validator("prefix")
26
+ @classmethod
27
+ def _require_bare_prefix(cls, prefix: str) -> str:
28
+ if not prefix or prefix.endswith("/"):
29
+ raise ValueError("mirror prefix must be non-empty and carry no trailing slash")
30
+ return prefix
31
+
32
+
33
+ @app.command()
34
+ def mirror() -> None:
35
+ root = Path.cwd()
36
+ config = load_mirror_config(root)
37
+ if config is None:
38
+ raise RuntimeError("[tool.docker-devkit.mirror] with 'prefix' is required to mirror")
39
+
40
+ targets = mirror_targets(root, config.prefix)
41
+ print(f"Install crane {CRANE_VERSION}")
42
+ sudo = "sudo " if os.geteuid() != 0 else ""
43
+ archive = "go-containerregistry_Linux_x86_64.tar.gz"
44
+ bash(f"curl -fsSLO https://github.com/google/go-containerregistry/releases/download/{CRANE_VERSION}/{archive}")
45
+ bash(f"{sudo}tar -xzf {archive} -C /usr/local/bin crane")
46
+ Path(archive).unlink()
47
+ for target in targets:
48
+ upstream = upstream_ref(target.reference, config.prefix)
49
+ print(f"Mirror {upstream} -> {target.reference}")
50
+ bash(f"crane copy {upstream} {target.reference}")
51
+ print(f"Mirrored images: {len(targets)}")
52
+
53
+
54
+ def load_mirror_config(root: Path) -> MirrorConfig | None:
55
+ pyproject_path = root / "pyproject.toml"
56
+ if not pyproject_path.exists():
57
+ return None
58
+ data = tomllib.loads(pyproject_path.read_text(encoding="utf-8"))
59
+ section = data.get("tool", {}).get("docker-devkit", {}).get("mirror")
60
+ if section is None:
61
+ return None
62
+ return MirrorConfig.model_validate(section)
63
+
64
+
65
+ def is_mirrored(reference: str, prefix: str) -> bool:
66
+ return reference.startswith(f"{prefix}/")
67
+
68
+
69
+ def upstream_ref(reference: str, prefix: str) -> str:
70
+ return reference[len(prefix) + 1 :]
71
+
72
+
73
+ def reference_exists(reference: str) -> bool:
74
+ return bash_check(f"docker buildx imagetools inspect {shlex.quote(reference)}")
75
+
76
+
77
+ def mirror_targets(root: Path, prefix: str) -> list[ImageReference]:
78
+ return sorted(
79
+ (occurrence for occurrence in declared_references(root) if is_mirrored(occurrence.reference, prefix)),
80
+ key=lambda occurrence: occurrence.reference,
81
+ )
82
+
83
+
84
+ def upstream_fallback(lock_data: dict[str, str], prefix: str, *, allow_upstream_fallback: bool) -> dict[str, str]:
85
+ missing = {
86
+ name: value for name, value in lock_data.items() if is_mirrored(value, prefix) and not reference_exists(value)
87
+ }
88
+ if not missing:
89
+ return {}
90
+ if not allow_upstream_fallback:
91
+ value = next(iter(missing.values()))
92
+ reference, _, digest = value.rpartition("@")
93
+ raise RuntimeError(
94
+ f"Image {reference or value} (digest {digest or 'unknown'}) is not mirrored yet. "
95
+ "If this is a new upstream dependency that has not been through CI, pass --allow-upstream-fallback "
96
+ "to pull the same digest from upstream for this run."
97
+ )
98
+ return {name: upstream_ref(value, prefix) for name, value in missing.items()}
99
+
100
+
101
+ def write_fallback_overlay(substitutions: dict[str, str]) -> Path:
102
+ overlay = Path(tempfile.gettempdir()) / f"docker-devkit-upstream-fallback-{os.getpid()}.env"
103
+ overlay.write_text(
104
+ "\n".join(f"{name}={value}" for name, value in sorted(substitutions.items())) + "\n",
105
+ encoding="utf-8",
106
+ )
107
+ return overlay
108
+
109
+
110
+ def main() -> None:
111
+ app()
112
+
113
+
114
+ if __name__ == "__main__":
115
+ main()
@@ -16,7 +16,8 @@ from .lifecycle import (
16
16
  resolve_lock,
17
17
  resolve_manifest,
18
18
  )
19
- from .modes import resolve_auth_mode
19
+ from .mirror import load_mirror_config, upstream_fallback, write_fallback_overlay
20
+ from .modes import parse_env_file, resolve_auth_mode
20
21
 
21
22
  ENV_FILE = Path(".env")
22
23
 
@@ -41,6 +42,13 @@ def up(
41
42
  "--dev",
42
43
  help="Layer the declared dev overlay (compose.dev.yml shape) over the production stack for bind-mount/debug bring-up.",
43
44
  ),
45
+ allow_upstream_fallback: Annotated[
46
+ bool,
47
+ typer.Option(
48
+ "--allow-upstream-fallback",
49
+ help="Pull unmirrored lock entries from upstream at the same digest instead of failing closed.",
50
+ ),
51
+ ] = False,
44
52
  ) -> None:
45
53
  root = Path.cwd()
46
54
  config = load_lifecycle_config(root)
@@ -67,8 +75,21 @@ def up(
67
75
 
68
76
  auth_mode = resolve_auth_mode(ENV_FILE)
69
77
 
78
+ substitutions: dict[str, str] = {}
79
+ fallback_overlay: Path | None = None
80
+ mirror_config = load_mirror_config(root)
81
+ if mirror_config is not None:
82
+ substitutions = upstream_fallback(
83
+ parse_env_file(lock_file), mirror_config.prefix, allow_upstream_fallback=allow_upstream_fallback
84
+ )
85
+ if substitutions:
86
+ for name, upstream in sorted(substitutions.items()):
87
+ print(f"Falling back to upstream for {name}: {upstream}", flush=True)
88
+ os.environ.update(substitutions)
89
+ fallback_overlay = write_fallback_overlay(substitutions)
90
+
70
91
  if build:
71
- run_build(gpu=gpu)
92
+ run_build(gpu=gpu, env_overlay=substitutions)
72
93
 
73
94
  if manifest is not None:
74
95
  os.environ.update(compute_service_shas(root, parse_bake(manifest)))
@@ -80,6 +101,8 @@ def up(
80
101
  files_args = " ".join(f"-f {compose_file}" for compose_file in compose_files)
81
102
  profile_flag = "--profile keycloak " if auth_mode == "keycloak" else ""
82
103
  compose_args = f"{files_args} {profile_flag}--project-directory {root} --env-file .env --env-file {lock_file}"
104
+ if fallback_overlay is not None:
105
+ compose_args += f" --env-file {fallback_overlay}"
83
106
 
84
107
  up_command = f"docker compose {compose_args} up"
85
108
  if not build:
@@ -0,0 +1,356 @@
1
+ import os
2
+ import tempfile
3
+ from pathlib import Path
4
+
5
+ import pytest
6
+ from pydantic import ValidationError
7
+ from typer.testing import CliRunner
8
+
9
+ from docker_devkit import build_docker, mirror
10
+ from docker_devkit.documents import BakeDocument
11
+ from docker_devkit.image_refs import ImageReference
12
+ from docker_devkit.mirror import MirrorConfig
13
+
14
+ MIRROR_PREFIX = "ghcr.io/outernet-foundation/mirror"
15
+ FAKE_DIGEST = "sha256:" + "a" * 64
16
+
17
+ MIRROR_REFERENCES = [
18
+ f"{MIRROR_PREFIX}/docker.io/library/python:3.13-slim@{FAKE_DIGEST}",
19
+ f"{MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15-python3.13-trixie-slim",
20
+ f"{MIRROR_PREFIX}/quay.io/keycloak/keycloak:26.3.5@{FAKE_DIGEST}",
21
+ f"{MIRROR_PREFIX}/docker.io/livekit/livekit-server:v1.12.0@{FAKE_DIGEST}",
22
+ ]
23
+
24
+ MANIFEST = (
25
+ "x-base-images:\n"
26
+ f" UV_IMAGE: {MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15\n"
27
+ " API_IMAGE: ghcr.io/outernet-foundation/placeframe/api:1.2.3\n"
28
+ "services:\n"
29
+ " api:\n"
30
+ " build:\n"
31
+ " dockerfile: workloads/api/Dockerfile\n"
32
+ " tags:\n"
33
+ " - localhost:5000/api:${API_SHA}\n"
34
+ )
35
+
36
+ LOCKED_ERROR = (
37
+ f"Image {MIRROR_PREFIX}/docker.io/library/alpine:3.20 (digest {FAKE_DIGEST}) is not mirrored yet. "
38
+ "If this is a new upstream dependency that has not been through CI, pass --allow-upstream-fallback "
39
+ "to pull the same digest from upstream for this run."
40
+ )
41
+
42
+ runner = CliRunner()
43
+
44
+
45
+ class SelectiveExistence:
46
+ def __init__(self, present: str) -> None:
47
+ self.present = present
48
+ self.probed: list[str] = []
49
+
50
+ def exists(self, reference: str) -> bool:
51
+ self.probed.append(reference)
52
+ return reference == self.present
53
+
54
+
55
+ class BashCheckRecorder:
56
+ def __init__(self, *, exists: bool) -> None:
57
+ self.exists = exists
58
+ self.commands: list[str] = []
59
+
60
+ def check(self, command: str) -> bool:
61
+ self.commands.append(command)
62
+ return self.exists
63
+
64
+
65
+ class CommandRecorder:
66
+ def __init__(self) -> None:
67
+ self.commands: list[str] = []
68
+
69
+ def run(self, command: str) -> None:
70
+ self.commands.append(command)
71
+ if command.startswith("curl"):
72
+ Path("go-containerregistry_Linux_x86_64.tar.gz").write_text("", encoding="utf-8")
73
+
74
+
75
+ class DigestRecorder:
76
+ def __init__(self) -> None:
77
+ self.inspected: list[str] = []
78
+
79
+ def resolve(self, reference: str) -> str:
80
+ self.inspected.append(reference)
81
+ return FAKE_DIGEST
82
+
83
+
84
+ class BakeEnvProbe:
85
+ def __init__(self) -> None:
86
+ self.uv_image: str | None = None
87
+
88
+ def run(self, command: str) -> None:
89
+ self.uv_image = os.environ.get("UV_IMAGE")
90
+ Path("metadata.json").write_text('{"api": {}}', encoding="utf-8")
91
+
92
+
93
+ def _always_true(reference: str) -> bool:
94
+ return True
95
+
96
+
97
+ def _bash_check_false(command: str) -> bool:
98
+ return False
99
+
100
+
101
+ def _must_not_be_probed(reference: str) -> bool:
102
+ raise AssertionError(f"unexpected existence probe: {reference}")
103
+
104
+
105
+ def _no_service_shas(root: Path, bake: BakeDocument) -> dict[str, str]:
106
+ return {}
107
+
108
+
109
+ def _no_gc_limits(min_gb: int = 60) -> None:
110
+ return None
111
+
112
+
113
+ def _fixed_digest(reference: str) -> str:
114
+ return FAKE_DIGEST
115
+
116
+
117
+ class TestInversion:
118
+ @pytest.mark.parametrize("reference", MIRROR_REFERENCES)
119
+ def test_round_trips_through_the_prefix(self, reference: str):
120
+ assert mirror.is_mirrored(reference, MIRROR_PREFIX)
121
+ assert f"{MIRROR_PREFIX}/{mirror.upstream_ref(reference, MIRROR_PREFIX)}" == reference
122
+
123
+ @pytest.mark.parametrize("reference", MIRROR_REFERENCES)
124
+ def test_preserves_tag_and_digest_tail(self, reference: str):
125
+ assert mirror.upstream_ref(reference, MIRROR_PREFIX).rpartition("/")[2] == reference.rpartition("/")[2]
126
+
127
+ def test_dockerio_path_maps_to_fully_qualified_upstream(self):
128
+ reference = f"{MIRROR_PREFIX}/docker.io/livekit/livekit-server:v1.12.0@{FAKE_DIGEST}"
129
+ assert (
130
+ mirror.upstream_ref(reference, MIRROR_PREFIX) == f"docker.io/livekit/livekit-server:v1.12.0@{FAKE_DIGEST}"
131
+ )
132
+
133
+ def test_prefix_boundary_requires_slash(self):
134
+ assert not mirror.is_mirrored(f"{MIRROR_PREFIX}-evil/docker.io/library/alpine:3.20", MIRROR_PREFIX)
135
+
136
+ def test_publish_entries_are_not_mirrored(self):
137
+ assert not mirror.is_mirrored("ghcr.io/outernet-foundation/placeframe/api:1.2.3", MIRROR_PREFIX)
138
+
139
+
140
+ class TestLoadMirrorConfig:
141
+ def test_loads_declared_prefix(self, tmp_path: Path):
142
+ (tmp_path / "pyproject.toml").write_text(
143
+ f"[tool.docker-devkit.mirror]\nprefix = '{MIRROR_PREFIX}'\n", encoding="utf-8"
144
+ )
145
+
146
+ assert mirror.load_mirror_config(tmp_path) == MirrorConfig(prefix=MIRROR_PREFIX)
147
+
148
+ def test_absent_table_returns_none(self, tmp_path: Path):
149
+ (tmp_path / "pyproject.toml").write_text("[tool.docker-devkit.lifecycle]\nfiles = []\n", encoding="utf-8")
150
+
151
+ assert mirror.load_mirror_config(tmp_path) is None
152
+
153
+ def test_missing_pyproject_returns_none(self, tmp_path: Path):
154
+ assert mirror.load_mirror_config(tmp_path) is None
155
+
156
+ def test_rejects_unknown_keys(self, tmp_path: Path):
157
+ (tmp_path / "pyproject.toml").write_text(
158
+ "[tool.docker-devkit.mirror]\nprefix = 'ghcr.io/x/mirror'\nbogus = true\n", encoding="utf-8"
159
+ )
160
+
161
+ with pytest.raises(ValidationError):
162
+ mirror.load_mirror_config(tmp_path)
163
+
164
+ def test_rejects_trailing_slash_prefix(self, tmp_path: Path):
165
+ (tmp_path / "pyproject.toml").write_text(
166
+ "[tool.docker-devkit.mirror]\nprefix = 'ghcr.io/x/mirror/'\n", encoding="utf-8"
167
+ )
168
+
169
+ with pytest.raises(ValidationError):
170
+ mirror.load_mirror_config(tmp_path)
171
+
172
+
173
+ class TestMirrorTargets:
174
+ def test_filters_declared_references_by_prefix_and_sorts(self, tmp_path: Path):
175
+ (tmp_path / "workloads").mkdir()
176
+ (tmp_path / "workloads" / "images.yml").write_text(MANIFEST, encoding="utf-8")
177
+
178
+ assert mirror.mirror_targets(tmp_path, MIRROR_PREFIX) == [
179
+ ImageReference("UV_IMAGE", f"{MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15")
180
+ ]
181
+
182
+
183
+ class TestReferenceExists:
184
+ def test_probes_with_imagetools_inspect(self, monkeypatch: pytest.MonkeyPatch):
185
+ recorder = BashCheckRecorder(exists=True)
186
+ monkeypatch.setattr(mirror, "bash_check", recorder.check)
187
+
188
+ assert mirror.reference_exists("docker.io/library/alpine:3.20")
189
+ assert recorder.commands == ["docker buildx imagetools inspect docker.io/library/alpine:3.20"]
190
+
191
+ def test_returns_false_for_failed_probe(self, monkeypatch: pytest.MonkeyPatch):
192
+ monkeypatch.setattr(mirror, "bash_check", _bash_check_false)
193
+
194
+ assert not mirror.reference_exists("docker.io/library/alpine:3.20")
195
+
196
+
197
+ class TestUpstreamFallback:
198
+ def test_fails_closed_with_the_locked_message(self, monkeypatch: pytest.MonkeyPatch):
199
+ monkeypatch.setattr(mirror, "reference_exists", _bash_check_false)
200
+ lock_data = {"ALPINE_IMAGE": f"{MIRROR_PREFIX}/docker.io/library/alpine:3.20@{FAKE_DIGEST}"}
201
+
202
+ with pytest.raises(RuntimeError) as excinfo:
203
+ mirror.upstream_fallback(lock_data, MIRROR_PREFIX, allow_upstream_fallback=False)
204
+
205
+ assert str(excinfo.value) == LOCKED_ERROR
206
+
207
+ def test_flag_substitutes_exactly_the_missing_entries(self, monkeypatch: pytest.MonkeyPatch):
208
+ present = f"{MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15"
209
+ absent = f"{MIRROR_PREFIX}/docker.io/library/alpine:3.20@{FAKE_DIGEST}"
210
+ existence = SelectiveExistence(present)
211
+ monkeypatch.setattr(mirror, "reference_exists", existence.exists)
212
+ lock_data = {
213
+ "UV_IMAGE": present,
214
+ "ALPINE_IMAGE": absent,
215
+ "API_IMAGE": "ghcr.io/outernet-foundation/placeframe/api:1.2.3",
216
+ }
217
+
218
+ substitutions = mirror.upstream_fallback(lock_data, MIRROR_PREFIX, allow_upstream_fallback=True)
219
+
220
+ assert substitutions == {"ALPINE_IMAGE": f"docker.io/library/alpine:3.20@{FAKE_DIGEST}"}
221
+ assert existence.probed == [present, absent]
222
+
223
+ def test_returns_empty_when_everything_is_mirrored(self, monkeypatch: pytest.MonkeyPatch):
224
+ monkeypatch.setattr(mirror, "reference_exists", _always_true)
225
+ lock_data = {"ALPINE_IMAGE": f"{MIRROR_PREFIX}/docker.io/library/alpine:3.20@{FAKE_DIGEST}"}
226
+
227
+ assert mirror.upstream_fallback(lock_data, MIRROR_PREFIX, allow_upstream_fallback=False) == {}
228
+
229
+ def test_non_mirror_entries_are_never_probed(self, monkeypatch: pytest.MonkeyPatch):
230
+ monkeypatch.setattr(mirror, "reference_exists", _must_not_be_probed)
231
+ lock_data = {"API_IMAGE": "ghcr.io/outernet-foundation/placeframe/api:1.2.3"}
232
+
233
+ assert mirror.upstream_fallback(lock_data, MIRROR_PREFIX, allow_upstream_fallback=True) == {}
234
+
235
+
236
+ class TestWriteFallbackOverlay:
237
+ def test_writes_sorted_entries_under_a_pid_named_tempfile(self):
238
+ substitutions = {
239
+ "UV_IMAGE": "ghcr.io/astral-sh/uv:0.12.15",
240
+ "ALPINE_IMAGE": f"docker.io/library/alpine:3.20@{FAKE_DIGEST}",
241
+ }
242
+
243
+ overlay = mirror.write_fallback_overlay(substitutions)
244
+
245
+ assert overlay.parent == Path(tempfile.gettempdir())
246
+ assert overlay.name == f"docker-devkit-upstream-fallback-{os.getpid()}.env"
247
+ assert overlay.read_text(encoding="utf-8") == (
248
+ f"ALPINE_IMAGE=docker.io/library/alpine:3.20@{FAKE_DIGEST}\nUV_IMAGE=ghcr.io/astral-sh/uv:0.12.15\n"
249
+ )
250
+
251
+
252
+ class TestMirrorVerb:
253
+ def _write_tree(self, tmp_path: Path) -> None:
254
+ (tmp_path / "pyproject.toml").write_text(
255
+ f"[tool.docker-devkit.mirror]\nprefix = '{MIRROR_PREFIX}'\n", encoding="utf-8"
256
+ )
257
+ (tmp_path / "workloads").mkdir()
258
+ (tmp_path / "workloads" / "images.yml").write_text(MANIFEST, encoding="utf-8")
259
+
260
+ def test_copies_each_mirror_target_from_upstream(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
261
+ recorder = CommandRecorder()
262
+ monkeypatch.setattr(mirror, "bash", recorder.run)
263
+ monkeypatch.chdir(tmp_path)
264
+ self._write_tree(tmp_path)
265
+
266
+ result = runner.invoke(mirror.app, [])
267
+
268
+ assert result.exit_code == 0
269
+ assert [command for command in recorder.commands if command.startswith("crane copy")] == [
270
+ f"crane copy ghcr.io/astral-sh/uv:0.12.15 {MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15"
271
+ ]
272
+ assert "Mirrored images: 1" in result.stdout
273
+
274
+ def test_errors_without_mirror_table(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
275
+ monkeypatch.chdir(tmp_path)
276
+
277
+ result = runner.invoke(mirror.app, [])
278
+
279
+ assert result.exit_code != 0
280
+ assert isinstance(result.exception, RuntimeError)
281
+
282
+
283
+ class TestUpstreamAwareLockResolution:
284
+ def _write_tree(self, tmp_path: Path, *, mirror_table: bool) -> None:
285
+ table = f"[tool.docker-devkit.mirror]\nprefix = '{MIRROR_PREFIX}'\n" if mirror_table else ""
286
+ (tmp_path / "pyproject.toml").write_text(f"[project]\nname = 'stack'\n\n{table}", encoding="utf-8")
287
+ (tmp_path / "workloads").mkdir()
288
+ (tmp_path / "workloads" / "images.yml").write_text(MANIFEST, encoding="utf-8")
289
+
290
+ def test_resolves_mirror_declarations_against_upstream(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
291
+ self._write_tree(tmp_path, mirror_table=True)
292
+ monkeypatch.chdir(tmp_path)
293
+ recorder = DigestRecorder()
294
+ monkeypatch.setattr(build_docker, "resolve_remote_digest", recorder.resolve)
295
+ monkeypatch.setattr(build_docker, "compute_service_shas", _no_service_shas)
296
+
297
+ build_docker.run_build(lock_only=True)
298
+
299
+ assert recorder.inspected == [
300
+ "ghcr.io/astral-sh/uv:0.12.15",
301
+ "ghcr.io/outernet-foundation/placeframe/api:1.2.3",
302
+ ]
303
+ lock = (tmp_path / "workloads" / "images.lock").read_text(encoding="utf-8")
304
+ assert f"API_IMAGE=ghcr.io/outernet-foundation/placeframe/api:1.2.3@{FAKE_DIGEST}" in lock
305
+ assert f"UV_IMAGE={MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15@{FAKE_DIGEST}" in lock
306
+
307
+ def test_without_table_inspects_the_declared_reference(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
308
+ self._write_tree(tmp_path, mirror_table=False)
309
+ monkeypatch.chdir(tmp_path)
310
+ recorder = DigestRecorder()
311
+ monkeypatch.setattr(build_docker, "resolve_remote_digest", recorder.resolve)
312
+ monkeypatch.setattr(build_docker, "compute_service_shas", _no_service_shas)
313
+
314
+ build_docker.run_build(lock_only=True)
315
+
316
+ assert recorder.inspected == [
317
+ f"{MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15",
318
+ "ghcr.io/outernet-foundation/placeframe/api:1.2.3",
319
+ ]
320
+
321
+
322
+ class TestBakeConsumesEnvOverlay:
323
+ def _write_tree(self, tmp_path: Path) -> None:
324
+ (tmp_path / "pyproject.toml").write_text(
325
+ f"[project]\nname = 'stack'\n\n[tool.docker-devkit.mirror]\nprefix = '{MIRROR_PREFIX}'\n",
326
+ encoding="utf-8",
327
+ )
328
+ (tmp_path / "workloads").mkdir()
329
+ (tmp_path / "workloads" / "images.yml").write_text(MANIFEST, encoding="utf-8")
330
+
331
+ def _fake_bake(self, monkeypatch: pytest.MonkeyPatch) -> BakeEnvProbe:
332
+ probe = BakeEnvProbe()
333
+ monkeypatch.setattr(build_docker, "bash", probe.run)
334
+ monkeypatch.setattr(build_docker, "_check_gc_limits", _no_gc_limits)
335
+ monkeypatch.setattr(build_docker, "resolve_remote_digest", _fixed_digest)
336
+ monkeypatch.setattr(build_docker, "compute_service_shas", _no_service_shas)
337
+ return probe
338
+
339
+ def test_overlay_wins_over_lock_values(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
340
+ self._write_tree(tmp_path)
341
+ monkeypatch.chdir(tmp_path)
342
+ probe = self._fake_bake(monkeypatch)
343
+ upstream = f"ghcr.io/astral-sh/uv:0.12.15@{FAKE_DIGEST}"
344
+
345
+ build_docker.run_build(gpu="none", env_overlay={"UV_IMAGE": upstream})
346
+
347
+ assert probe.uv_image == upstream
348
+
349
+ def test_without_overlay_bake_sees_mirror_values(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
350
+ self._write_tree(tmp_path)
351
+ monkeypatch.chdir(tmp_path)
352
+ probe = self._fake_bake(monkeypatch)
353
+
354
+ build_docker.run_build(gpu="none")
355
+
356
+ assert probe.uv_image == f"{MIRROR_PREFIX}/ghcr.io/astral-sh/uv:0.12.15@{FAKE_DIGEST}"
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes