pants-backend-distroless 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (13) hide show
  1. pants_backend_distroless-0.1.0/LICENSE-MIT.txt +21 -0
  2. pants_backend_distroless-0.1.0/PKG-INFO +255 -0
  3. pants_backend_distroless-0.1.0/README.md +219 -0
  4. pants_backend_distroless-0.1.0/pants_backend_distroless/__init__.py +0 -0
  5. pants_backend_distroless-0.1.0/pants_backend_distroless/apt_resolver.py +616 -0
  6. pants_backend_distroless-0.1.0/pants_backend_distroless/lockfile.py +151 -0
  7. pants_backend_distroless-0.1.0/pants_backend_distroless/openpgp.py +487 -0
  8. pants_backend_distroless-0.1.0/pants_backend_distroless/register.py +20 -0
  9. pants_backend_distroless-0.1.0/pants_backend_distroless/sandboxed.py +124 -0
  10. pants_backend_distroless-0.1.0/pants_backend_distroless/subsystem.py +40 -0
  11. pants_backend_distroless-0.1.0/pants_backend_distroless/target_generation.py +211 -0
  12. pants_backend_distroless-0.1.0/pants_backend_distroless/target_types.py +177 -0
  13. pants_backend_distroless-0.1.0/pyproject.toml +53 -0
pants_backend_distroless-0.1.0/LICENSE-MIT.txt ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2022 Tom Solberg
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
pants_backend_distroless-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,255 @@
1
+ Metadata-Version: 2.1
2
+ Name: pants_backend_distroless
3
+ Version: 0.1.0
4
+ Summary: A distroless image plugin for the Pants build system
5
+ Keywords: pantsbuild,pants,backend,distroless,debian,apt,oci
6
+ Author-Email: Tom Solberg <me@sbg.dev>
7
+ License: MIT License
8
+
9
+ Copyright (c) 2022 Tom Solberg
10
+
11
+ Permission is hereby granted, free of charge, to any person obtaining a copy
12
+ of this software and associated documentation files (the "Software"), to deal
13
+ in the Software without restriction, including without limitation the rights
14
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
15
+ copies of the Software, and to permit persons to whom the Software is
16
+ furnished to do so, subject to the following conditions:
17
+
18
+ The above copyright notice and this permission notice shall be included in all
19
+ copies or substantial portions of the Software.
20
+
21
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
22
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
23
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
24
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
25
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
26
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
27
+ SOFTWARE.
28
+
29
+ Classifier: Development Status :: 4 - Beta
30
+ Classifier: Programming Language :: Python
31
+ Classifier: Topic :: Software Development :: Build Tools
32
+ Project-URL: Repository, https://github.com/tgolsson/pants-backends
33
+ Project-URL: Code, https://github.com/tgolsson/pants-backends/tree/main/pants-plugins/distroless
34
+ Project-URL: Changelog, https://github.com/tgolsson/pants-backends/tree/main/pants-plugins/distroless/CHANGELOG.md
35
+ Description-Content-Type: text/markdown
36
+
37
+ # Distroless backend for Pants
38
+
39
+ This release targets Pants **2.32.x**. Install it with
40
+ `pants-backend-distroless==0.1.0` in `[GLOBAL].plugins` in `pants.toml`, and enable
41
+ `pants_backend_distroless` in `[GLOBAL].backend_packages`.
42
+
43
+ [![PyPI](https://img.shields.io/pypi/v/pants-backend-distroless?label=Latest%20release)](https://pypi.org/project/pants-backend-distroless)
44
+
45
+ > **Warning**
46
+ > This plugin is in development. No stability is guaranteed! Contributions welcome.
47
+
48
+ This backend builds the parts of distroless-style images that come from Debian or Ubuntu packages, in the
49
+ spirit of [rules_distroless](https://github.com/GoogleContainerTools/rules_distroless): declare the
50
+ top-level packages against an apt repository snapshot, and `pants generate-lockfiles` resolves their
51
+ dependencies into a committed lockfile. The locked packages feed `oci_deb_layer` from
52
+ [`pants-backend-oci`](https://pypi.org/project/pants-backend-oci), which builds a layer from them
53
+ without running `apt`, `dpkg` or maintainer scripts. `pants-backend-oci` also has the helper layers such
54
+ an image needs: `oci_passwd_layer`, `oci_group_layer`, `oci_cacerts_layer` and `oci_locale_layer`.
55
+
56
+ ## Targets
57
+
58
+ ### `deb_packages`
59
+
60
+ A set of Debian packages resolved into a lockfile. Each resolve is named in `[apt].resolves`, which
61
+ maps it to its lockfile, and is declared by exactly one `deb_packages` target.
62
+
63
+ ``` toml
64
+ [apt.resolves]
65
+ bookworm = "3rdparty/debian/bookworm.lock.json"
66
+ ```
67
+
68
+ ``` python
69
+ deb_packages(
70
+ name="bookworm",
71
+ resolve="bookworm",
72
+ sources=["https://snapshot.debian.org/archive/debian/20241001T000000Z bookworm main"],
73
+ architectures=["amd64", "arm64"],
74
+ packages=["jq"],
75
+ # Already in the base image.
76
+ exclude=["libc6"],
77
+ # From the `debian-archive-keyring` package.
78
+ keyrings=["debian-archive-keyring.gpg"],
79
+ )
80
+
81
+ oci_deb_layer(
82
+ name="jq",
83
+ debs=[":bookworm#amd64/jq"],
84
+ )
85
+ ```
86
+
87
+ Run `pants generate-lockfiles --resolve=bookworm` to write the lockfile, and again whenever the
88
+ target changes.
89
+
90
+ The target generates, from its own fields alone:
91
+
92
+ * a `deb_package` target named `<arch>/<package>` for every architecture and requested package,
93
+ whose sources are the pinned `.deb` files of the package and of every locked package in its
94
+ closure, so `:bookworm#amd64/jq` brings `jq` and its whole closure;
95
+ * a `deb_package_group` target named `<arch>`, depending on all the requested packages, so
96
+ `:bookworm#amd64` is everything.
97
+
98
+ As with Python requirements, only the requested packages are targets: to depend on a package
99
+ directly, add it to `packages`. A requested virtual package, like `awk`, brings the package that
100
+ provides it.
101
+
102
+ The lockfile is read only when the `.deb` files are needed, such as when a layer is built. Until it's
103
+ generated, and while it's out of date, that fails with an error naming the lockfile and the
104
+ `generate-lockfiles` command; commands that don't need the `.deb` files, like `pants list ::`, work
105
+ without it. The lockfile records each package's closure, so apt dependency cycles are fine: `libc6`
106
+ and `libgcc-s1` depend on each other in Debian, and either one brings both.
107
+
108
+ A layer is for one architecture, so always pick one: `debs=[":bookworm"]` would bring the packages of
109
+ every architecture, which `oci_deb_layer` rejects.
110
+
111
+ | Argument | Meaning | Default value |
112
+ |------------------|-----------------------------------------------------------------------------------------------|-------------------------------------------------------|
113
+ | `name` | The target name | Same as any other target, which is the directory name |
114
+ | `resolve` | The resolve name, a key of `[apt].resolves`. | Required |
115
+ | `sources` | The repositories, each `<uri> <suite> <component>...` like a sources.list line without `deb`. | Required |
116
+ | `architectures` | The Debian architectures to lock, e.g. `amd64` and `arm64`. | Required |
117
+ | `packages` | The top-level packages. | Required |
118
+ | `exclude` | Packages treated as already installed, typically those the base image ships. | `[]` |
119
+ | `keyrings` | OpenPGP keyrings (`.gpg` or `.asc`) to verify the repositories' `Release` signatures with. | Required unless `allow_unsigned=True` |
120
+ | `allow_unsigned` | Resolve without verifying signatures. Can't be combined with `keyrings`. | `False` |
121
+ | `description` | A description of the target | |
122
+ | `tags` | List of tags | `[]` |
123
+
124
+ #### Sources
125
+
126
+ Pin each URI to a snapshot, such as `https://snapshot.debian.org/archive/debian/<timestamp>`, so
127
+ the lockfile can be regenerated and the packages stay downloadable. Several sources can be combined,
128
+ for example to add `bookworm-updates`, or `bookworm-security` from
129
+ `https://snapshot.debian.org/archive/debian-security/<timestamp>`. Every `Packages` index is checked
130
+ against the SHA256 in its suite's signed `Release` file, see [Signature verification](#signature-verification).
131
+
132
+ Ubuntu works the same way, from [snapshot.ubuntu.com](https://snapshot.ubuntu.com). Its updates
133
+ and security fixes are separate suites of the same archive, and unlike the live archive, where
134
+ arm64 lives on `ports.ubuntu.com`, the snapshot serves arm64 packages under `/ubuntu` too, so one
135
+ set of sources covers both architectures. `Phased-Update-Percentage` is ignored: a snapshot lock
136
+ takes a phased update like any other version. Ubuntu's packages are zstd-compressed, which
137
+ `oci_deb_layer` reads with its `zstd` tool.
138
+
139
+ ``` python
140
+ deb_packages(
141
+ name="noble",
142
+ resolve="noble",
143
+ sources=[
144
+ "https://snapshot.ubuntu.com/ubuntu/20241001T000000Z noble main",
145
+ "https://snapshot.ubuntu.com/ubuntu/20241001T000000Z noble-updates main",
146
+ "https://snapshot.ubuntu.com/ubuntu/20241001T000000Z noble-security main",
147
+ ],
148
+ architectures=["amd64", "arm64"],
149
+ packages=["jq", "ca-certificates"],
150
+ # Already in the ubuntu:noble base image.
151
+ exclude=["libc6", "debconf", "libssl3t64"],
152
+ # From the `ubuntu-keyring` package.
153
+ keyrings=["ubuntu-archive-keyring.gpg"],
154
+ )
155
+ ```
156
+
157
+ #### Signature verification
158
+
159
+ Like rules_distroless's `gpg_keys`, `generate-lockfiles` verifies every suite's `InRelease`, or its
160
+ `Release` and `Release.gpg` when there's no `InRelease`, against `keyrings`, and fails unless at
161
+ least one signature is good and made by a key in them. A bad signature from one of their keys fails
162
+ too. The hashes of the `Packages` indices are only taken from the verified file. It also fails
163
+ unless the signed file's `Suite` or `Codename` is the suite asked for, so another suite's validly
164
+ signed `Release` can't stand in for it. This is apt's rule, though apt only warns: a path like
165
+ debian-security's old `buster/updates` matches by what comes before its last `/`. Either `keyrings` or `allow_unsigned=True` must be set.
166
+
167
+ Use the archive's own keyring, from the snapshot you resolve against so it's pinned too:
168
+ `/usr/share/keyrings/debian-archive-keyring.gpg` from the `debian-archive-keyring` package, or
169
+ `/usr/share/keyrings/ubuntu-archive-keyring.gpg` from `ubuntu-keyring`. Copy it next to the
170
+ `deb_packages` target, since sources can't come from another directory.
171
+
172
+ Verification needs no `gpgv` or `sqv`: it's built into the resolver, and supports what Debian and
173
+ Ubuntu sign with, v4 RSA and Ed25519 keys with SHA-2 hashes. Older snapshots signed only with SHA-1
174
+ need `allow_unsigned=True`, which turns signature verification off entirely; the suite is still
175
+ checked. Every key in the keyrings is trusted as given: expiry and revocation aren't checked, since a
176
+ snapshot's `Release` files are historical. For the same reason, freshness (`Valid-Until`) isn't
177
+ checked either: the snapshot URL is what pins it. The keys aren't recorded in the lockfile, so
178
+ changing `keyrings` doesn't make it out of date.
179
+
180
+ #### Resolution
181
+
182
+ The `Pre-Depends` and `Depends` of the requested packages are resolved transitively, for each
183
+ architecture separately, much like rules_distroless does:
184
+
185
+ * A package resolves to its highest version, across all sources, that satisfies the constraint.
186
+ When sources have the same version, the first source wins.
187
+ * For alternatives (`a | b`), a package already in the closure is preferred; otherwise the first
188
+ alternative that exists wins.
189
+ * A virtual package (`Provides`) resolves to a provider already in the closure, else the only
190
+ provider, else a `Priority: required` one, else the highest version, else the first by name.
191
+ * The result doesn't depend on the order of `packages`.
192
+
193
+ `Recommends` and `Suggests` are not installed, and `Conflicts` and `Breaks` are not checked.
194
+
195
+ The resolver runs in a sandboxed process from `apt-resolver`, a [scie](https://github.com/a-scie/jump)
196
+ the plugin builds with pex from its own resolver code and a pinned python-build-standalone Python,
197
+ CPython `3.11.17` from release `20261009` (set `[apt].resolver_python_version` and
198
+ `[apt].resolver_pbs_release` to use another). Since it names nothing on the host, a resolve's
199
+ cached result can be shared across checkouts, and across machines through a remote cache. It is built once per machine, downloading
200
+ that Python and pex's `science` tool from GitHub the first time, and unpacks into the
201
+ `apt_resolver` named cache. The Python and `science` are checked only against the checksum files
202
+ published in the same GitHub releases, through pex.
203
+
204
+ #### Packages already in the base image
205
+
206
+ A layer replaces any file the base image has at the same path, and nothing warns about it. If a
207
+ locked package is also in the base, its files from the snapshot overwrite the base's copies, which
208
+ can mix versions when the base image was built from a different archive state than the snapshot.
209
+ List such packages in `exclude`: they count as installed, satisfy any dependency on them, and are not
210
+ traversed, so whatever only they need is left out as well. Images built on an empty base leave
211
+ `exclude` empty.
212
+
213
+ ## Moving from a Python base image
214
+
215
+ A few things change when a Python app moves from a pulled base such as `python:3.11-slim` to one
216
+ built from Debian or Ubuntu packages.
217
+
218
+ **The interpreter is `/usr/bin/python3.X`.** The official Python images install it under
219
+ `/usr/local/bin` and have `/usr/bin/env`. A package's interpreter is `/usr/bin/python3.X`, where
220
+ `3.X` is the suite's Python: 3.11 in Debian bookworm, 3.13 in trixie, 3.12 in Ubuntu noble. Build
221
+ the pex for that version, since its interpreter constraints and wheels must match. An image built
222
+ from packages has no `env` unless it locks `coreutils`, and a pex's default shebang is
223
+ `#!/usr/bin/env python3.X`, so the pex fails to start. Point the shebang at the interpreter, and
224
+ make the pex the entrypoint with `oci_image_build`, whose first package is the entrypoint:
225
+
226
+ ``` python
227
+ pex_binary(
228
+ name="app",
229
+ entry_point="app.py:main",
230
+ shebang="/usr/bin/python3.11",
231
+ )
232
+
233
+ oci_image_build(
234
+ name="image",
235
+ base=[...],
236
+ layers=[":python"],
237
+ packages=[":app"],
238
+ )
239
+ ```
240
+
241
+ `oci_python_image` doesn't use the shebang: it runs the pex as `python <pex>`, and no Debian or
242
+ Ubuntu package but `python-is-python3` provides `/usr/bin/python`. Lock that package to keep using
243
+ `oci_python_image`. `python3-minimal`, which `python3` pre-depends on, adds `/usr/bin/python3`.
244
+
245
+ **pex warns that `ldd` is missing.** On startup a pex may print `Failed to detect if this
246
+ interpreter is running under musl libc: [Errno 2] No such file or directory: 'ldd'`. pex runs `ldd`
247
+ to tell musl from glibc and, when it can't, assumes glibc, which is right for Debian and Ubuntu.
248
+ The warning is harmless. To hide it, set `emit_warnings=False` on the `pex_binary`, or
249
+ `PEX_EMIT_WARNINGS=false` in the image's environment; either hides all of pex's warnings. Locking
250
+ `libc-bin` doesn't help in an image without a shell, since Debian's `ldd` is a bash script.
251
+
252
+ **The Pants store grows quickly.** Rebuilding multi-GB images adds new layers to the Pants store on
253
+ every change. See
254
+ [Pants store size](https://github.com/tgolsson/pants-backends/tree/main/pants-plugins/oci#pants-store-size)
255
+ in the OCI README for a setting that keeps it in check.
pants_backend_distroless-0.1.0/README.md ADDED
@@ -0,0 +1,219 @@
1
+ # Distroless backend for Pants
2
+
3
+ This release targets Pants **2.32.x**. Install it with
4
+ `pants-backend-distroless==0.1.0` in `[GLOBAL].plugins` in `pants.toml`, and enable
5
+ `pants_backend_distroless` in `[GLOBAL].backend_packages`.
6
+
7
+ [![PyPI](https://img.shields.io/pypi/v/pants-backend-distroless?label=Latest%20release)](https://pypi.org/project/pants-backend-distroless)
8
+
9
+ > **Warning**
10
+ > This plugin is in development. No stability is guaranteed! Contributions welcome.
11
+
12
+ This backend builds the parts of distroless-style images that come from Debian or Ubuntu packages, in the
13
+ spirit of [rules_distroless](https://github.com/GoogleContainerTools/rules_distroless): declare the
14
+ top-level packages against an apt repository snapshot, and `pants generate-lockfiles` resolves their
15
+ dependencies into a committed lockfile. The locked packages feed `oci_deb_layer` from
16
+ [`pants-backend-oci`](https://pypi.org/project/pants-backend-oci), which builds a layer from them
17
+ without running `apt`, `dpkg` or maintainer scripts. `pants-backend-oci` also has the helper layers such
18
+ an image needs: `oci_passwd_layer`, `oci_group_layer`, `oci_cacerts_layer` and `oci_locale_layer`.
19
+
20
+ ## Targets
21
+
22
+ ### `deb_packages`
23
+
24
+ A set of Debian packages resolved into a lockfile. Each resolve is named in `[apt].resolves`, which
25
+ maps it to its lockfile, and is declared by exactly one `deb_packages` target.
26
+
27
+ ``` toml
28
+ [apt.resolves]
29
+ bookworm = "3rdparty/debian/bookworm.lock.json"
30
+ ```
31
+
32
+ ``` python
33
+ deb_packages(
34
+ name="bookworm",
35
+ resolve="bookworm",
36
+ sources=["https://snapshot.debian.org/archive/debian/20241001T000000Z bookworm main"],
37
+ architectures=["amd64", "arm64"],
38
+ packages=["jq"],
39
+ # Already in the base image.
40
+ exclude=["libc6"],
41
+ # From the `debian-archive-keyring` package.
42
+ keyrings=["debian-archive-keyring.gpg"],
43
+ )
44
+
45
+ oci_deb_layer(
46
+ name="jq",
47
+ debs=[":bookworm#amd64/jq"],
48
+ )
49
+ ```
50
+
51
+ Run `pants generate-lockfiles --resolve=bookworm` to write the lockfile, and again whenever the
52
+ target changes.
53
+
54
+ The target generates, from its own fields alone:
55
+
56
+ * a `deb_package` target named `<arch>/<package>` for every architecture and requested package,
57
+ whose sources are the pinned `.deb` files of the package and of every locked package in its
58
+ closure, so `:bookworm#amd64/jq` brings `jq` and its whole closure;
59
+ * a `deb_package_group` target named `<arch>`, depending on all the requested packages, so
60
+ `:bookworm#amd64` is everything.
61
+
62
+ As with Python requirements, only the requested packages are targets: to depend on a package
63
+ directly, add it to `packages`. A requested virtual package, like `awk`, brings the package that
64
+ provides it.
65
+
66
+ The lockfile is read only when the `.deb` files are needed, such as when a layer is built. Until it's
67
+ generated, and while it's out of date, that fails with an error naming the lockfile and the
68
+ `generate-lockfiles` command; commands that don't need the `.deb` files, like `pants list ::`, work
69
+ without it. The lockfile records each package's closure, so apt dependency cycles are fine: `libc6`
70
+ and `libgcc-s1` depend on each other in Debian, and either one brings both.
71
+
72
+ A layer is for one architecture, so always pick one: `debs=[":bookworm"]` would bring the packages of
73
+ every architecture, which `oci_deb_layer` rejects.
74
+
75
+ | Argument | Meaning | Default value |
76
+ |------------------|-----------------------------------------------------------------------------------------------|-------------------------------------------------------|
77
+ | `name` | The target name | Same as any other target, which is the directory name |
78
+ | `resolve` | The resolve name, a key of `[apt].resolves`. | Required |
79
+ | `sources` | The repositories, each `<uri> <suite> <component>...` like a sources.list line without `deb`. | Required |
80
+ | `architectures` | The Debian architectures to lock, e.g. `amd64` and `arm64`. | Required |
81
+ | `packages` | The top-level packages. | Required |
82
+ | `exclude` | Packages treated as already installed, typically those the base image ships. | `[]` |
83
+ | `keyrings` | OpenPGP keyrings (`.gpg` or `.asc`) to verify the repositories' `Release` signatures with. | Required unless `allow_unsigned=True` |
84
+ | `allow_unsigned` | Resolve without verifying signatures. Can't be combined with `keyrings`. | `False` |
85
+ | `description` | A description of the target | |
86
+ | `tags` | List of tags | `[]` |
87
+
88
+ #### Sources
89
+
90
+ Pin each URI to a snapshot, such as `https://snapshot.debian.org/archive/debian/<timestamp>`, so
91
+ the lockfile can be regenerated and the packages stay downloadable. Several sources can be combined,
92
+ for example to add `bookworm-updates`, or `bookworm-security` from
93
+ `https://snapshot.debian.org/archive/debian-security/<timestamp>`. Every `Packages` index is checked
94
+ against the SHA256 in its suite's signed `Release` file, see [Signature verification](#signature-verification).
95
+
96
+ Ubuntu works the same way, from [snapshot.ubuntu.com](https://snapshot.ubuntu.com). Its updates
97
+ and security fixes are separate suites of the same archive, and unlike the live archive, where
98
+ arm64 lives on `ports.ubuntu.com`, the snapshot serves arm64 packages under `/ubuntu` too, so one
99
+ set of sources covers both architectures. `Phased-Update-Percentage` is ignored: a snapshot lock
100
+ takes a phased update like any other version. Ubuntu's packages are zstd-compressed, which
101
+ `oci_deb_layer` reads with its `zstd` tool.
102
+
103
+ ``` python
104
+ deb_packages(
105
+ name="noble",
106
+ resolve="noble",
107
+ sources=[
108
+ "https://snapshot.ubuntu.com/ubuntu/20241001T000000Z noble main",
109
+ "https://snapshot.ubuntu.com/ubuntu/20241001T000000Z noble-updates main",
110
+ "https://snapshot.ubuntu.com/ubuntu/20241001T000000Z noble-security main",
111
+ ],
112
+ architectures=["amd64", "arm64"],
113
+ packages=["jq", "ca-certificates"],
114
+ # Already in the ubuntu:noble base image.
115
+ exclude=["libc6", "debconf", "libssl3t64"],
116
+ # From the `ubuntu-keyring` package.
117
+ keyrings=["ubuntu-archive-keyring.gpg"],
118
+ )
119
+ ```
120
+
121
+ #### Signature verification
122
+
123
+ Like rules_distroless's `gpg_keys`, `generate-lockfiles` verifies every suite's `InRelease`, or its
124
+ `Release` and `Release.gpg` when there's no `InRelease`, against `keyrings`, and fails unless at
125
+ least one signature is good and made by a key in them. A bad signature from one of their keys fails
126
+ too. The hashes of the `Packages` indices are only taken from the verified file. It also fails
127
+ unless the signed file's `Suite` or `Codename` is the suite asked for, so another suite's validly
128
+ signed `Release` can't stand in for it. This is apt's rule, though apt only warns: a path like
129
+ debian-security's old `buster/updates` matches by what comes before its last `/`. Either `keyrings` or `allow_unsigned=True` must be set.
130
+
131
+ Use the archive's own keyring, from the snapshot you resolve against so it's pinned too:
132
+ `/usr/share/keyrings/debian-archive-keyring.gpg` from the `debian-archive-keyring` package, or
133
+ `/usr/share/keyrings/ubuntu-archive-keyring.gpg` from `ubuntu-keyring`. Copy it next to the
134
+ `deb_packages` target, since sources can't come from another directory.
135
+
136
+ Verification needs no `gpgv` or `sqv`: it's built into the resolver, and supports what Debian and
137
+ Ubuntu sign with, v4 RSA and Ed25519 keys with SHA-2 hashes. Older snapshots signed only with SHA-1
138
+ need `allow_unsigned=True`, which turns signature verification off entirely; the suite is still
139
+ checked. Every key in the keyrings is trusted as given: expiry and revocation aren't checked, since a
140
+ snapshot's `Release` files are historical. For the same reason, freshness (`Valid-Until`) isn't
141
+ checked either: the snapshot URL is what pins it. The keys aren't recorded in the lockfile, so
142
+ changing `keyrings` doesn't make it out of date.
143
+
144
+ #### Resolution
145
+
146
+ The `Pre-Depends` and `Depends` of the requested packages are resolved transitively, for each
147
+ architecture separately, much like rules_distroless does:
148
+
149
+ * A package resolves to its highest version, across all sources, that satisfies the constraint.
150
+ When sources have the same version, the first source wins.
151
+ * For alternatives (`a | b`), a package already in the closure is preferred; otherwise the first
152
+ alternative that exists wins.
153
+ * A virtual package (`Provides`) resolves to a provider already in the closure, else the only
154
+ provider, else a `Priority: required` one, else the highest version, else the first by name.
155
+ * The result doesn't depend on the order of `packages`.
156
+
157
+ `Recommends` and `Suggests` are not installed, and `Conflicts` and `Breaks` are not checked.
158
+
159
+ The resolver runs in a sandboxed process from `apt-resolver`, a [scie](https://github.com/a-scie/jump)
160
+ the plugin builds with pex from its own resolver code and a pinned python-build-standalone Python,
161
+ CPython `3.11.17` from release `20261009` (set `[apt].resolver_python_version` and
162
+ `[apt].resolver_pbs_release` to use another). Since it names nothing on the host, a resolve's
163
+ cached result can be shared across checkouts, and across machines through a remote cache. It is built once per machine, downloading
164
+ that Python and pex's `science` tool from GitHub the first time, and unpacks into the
165
+ `apt_resolver` named cache. The Python and `science` are checked only against the checksum files
166
+ published in the same GitHub releases, through pex.
167
+
168
+ #### Packages already in the base image
169
+
170
+ A layer replaces any file the base image has at the same path, and nothing warns about it. If a
171
+ locked package is also in the base, its files from the snapshot overwrite the base's copies, which
172
+ can mix versions when the base image was built from a different archive state than the snapshot.
173
+ List such packages in `exclude`: they count as installed, satisfy any dependency on them, and are not
174
+ traversed, so whatever only they need is left out as well. Images built on an empty base leave
175
+ `exclude` empty.
176
+
177
+ ## Moving from a Python base image
178
+
179
+ A few things change when a Python app moves from a pulled base such as `python:3.11-slim` to one
180
+ built from Debian or Ubuntu packages.
181
+
182
+ **The interpreter is `/usr/bin/python3.X`.** The official Python images install it under
183
+ `/usr/local/bin` and have `/usr/bin/env`. A package's interpreter is `/usr/bin/python3.X`, where
184
+ `3.X` is the suite's Python: 3.11 in Debian bookworm, 3.13 in trixie, 3.12 in Ubuntu noble. Build
185
+ the pex for that version, since its interpreter constraints and wheels must match. An image built
186
+ from packages has no `env` unless it locks `coreutils`, and a pex's default shebang is
187
+ `#!/usr/bin/env python3.X`, so the pex fails to start. Point the shebang at the interpreter, and
188
+ make the pex the entrypoint with `oci_image_build`, whose first package is the entrypoint:
189
+
190
+ ``` python
191
+ pex_binary(
192
+ name="app",
193
+ entry_point="app.py:main",
194
+ shebang="/usr/bin/python3.11",
195
+ )
196
+
197
+ oci_image_build(
198
+ name="image",
199
+ base=[...],
200
+ layers=[":python"],
201
+ packages=[":app"],
202
+ )
203
+ ```
204
+
205
+ `oci_python_image` doesn't use the shebang: it runs the pex as `python <pex>`, and no Debian or
206
+ Ubuntu package but `python-is-python3` provides `/usr/bin/python`. Lock that package to keep using
207
+ `oci_python_image`. `python3-minimal`, which `python3` pre-depends on, adds `/usr/bin/python3`.
208
+
209
+ **pex warns that `ldd` is missing.** On startup a pex may print `Failed to detect if this
210
+ interpreter is running under musl libc: [Errno 2] No such file or directory: 'ldd'`. pex runs `ldd`
211
+ to tell musl from glibc and, when it can't, assumes glibc, which is right for Debian and Ubuntu.
212
+ The warning is harmless. To hide it, set `emit_warnings=False` on the `pex_binary`, or
213
+ `PEX_EMIT_WARNINGS=false` in the image's environment; either hides all of pex's warnings. Locking
214
+ `libc-bin` doesn't help in an image without a shell, since Debian's `ldd` is a bash script.
215
+
216
+ **The Pants store grows quickly.** Rebuilding multi-GB images adds new layers to the Pants store on
217
+ every change. See
218
+ [Pants store size](https://github.com/tgolsson/pants-backends/tree/main/pants-plugins/oci#pants-store-size)
219
+ in the OCI README for a setting that keeps it in check.