pubdeploy 0.8.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. pubdeploy/__init__.py +532 -0
  2. pubdeploy/channels.py +188 -0
  3. pubdeploy/cli.py +120 -0
  4. pubdeploy/config.py +140 -0
  5. pubdeploy/contract.py +87 -0
  6. pubdeploy/github.py +61 -0
  7. pubdeploy/hooks.py +82 -0
  8. pubdeploy/implied.py +179 -0
  9. pubdeploy/journal.json +207 -0
  10. pubdeploy/k8s.py +472 -0
  11. pubdeploy/machine.py +363 -0
  12. pubdeploy/model.py +229 -0
  13. pubdeploy/names.py +25 -0
  14. pubdeploy/order.py +101 -0
  15. pubdeploy/providers.py +75 -0
  16. pubdeploy/registry.py +96 -0
  17. pubdeploy/release-commit +1 -0
  18. pubdeploy/schema.json +127 -0
  19. pubdeploy/schema.py +238 -0
  20. pubdeploy/store.py +127 -0
  21. pubdeploy/testing/__init__.py +94 -0
  22. pubdeploy/testing/fixtures/deploy-app/.github/workflows/deploy.yml +11 -0
  23. pubdeploy/testing/fixtures/deploy-app/k8s/admin/network-policy.yaml +11 -0
  24. pubdeploy/testing/fixtures/deploy-app/k8s/app.yaml +63 -0
  25. pubdeploy/testing/fixtures/deploy-app/k8s/kustomization.yaml +6 -0
  26. pubdeploy/testing/fixtures/deploy-app/pubrules.json +27 -0
  27. pubdeploy/testing/fixtures/deploy-app/rendered.yaml +86 -0
  28. pubdeploy/testing/fixtures/deploy-app/zones.json +8 -0
  29. pubdeploy/testing/plugin.py +26 -0
  30. pubdeploy/testing/steps.py +1124 -0
  31. pubdeploy/testing/world.py +569 -0
  32. pubdeploy/types/__init__.py +181 -0
  33. pubdeploy/types/auth.py +199 -0
  34. pubdeploy/types/github.py +110 -0
  35. pubdeploy/types/k8s.py +609 -0
  36. pubdeploy/types/net.py +237 -0
  37. pubdeploy/types/rollout.py +186 -0
  38. pubdeploy/units.py +177 -0
  39. pubdeploy/walk.py +325 -0
  40. pubdeploy-0.8.0.dist-info/METADATA +520 -0
  41. pubdeploy-0.8.0.dist-info/RECORD +45 -0
  42. pubdeploy-0.8.0.dist-info/WHEEL +5 -0
  43. pubdeploy-0.8.0.dist-info/entry_points.txt +11 -0
  44. pubdeploy-0.8.0.dist-info/licenses/LICENSE +202 -0
  45. pubdeploy-0.8.0.dist-info/top_level.txt +1 -0
@@ -0,0 +1,520 @@
1
+ Metadata-Version: 2.4
2
+ Name: pubdeploy
3
+ Version: 0.8.0
4
+ Summary: The deploy walk: what a service needs before it deploys, whether it has it, and making it; and the checks a deploy to Kubernetes runs.
5
+ Author-email: David Schryer <schryer@gmail.com>
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Repository, https://github.com/Sustenaut/pubdeploy
8
+ Requires-Python: >=3.11
9
+ Description-Content-Type: text/markdown
10
+ License-File: LICENSE
11
+ Requires-Dist: PyYAML>=6
12
+ Requires-Dist: python-statemachine>=3.2.2
13
+ Dynamic: license-file
14
+
15
+ # pubdeploy
16
+
17
+ The deploy walk, split out of [pubrules](https://github.com/Sustenaut/pubrules):
18
+ what a service needs before it deploys (pubrules.json's `deploy` block),
19
+ whether it has it, and making what is missing; the checks a deploy to
20
+ Kubernetes runs (`pubdeploy k8s preflight` and `diagnose`); and the reusable
21
+ `k8s-deploy.yml` and `deploy-check.yml`, the `deploy-ssh` action and the
22
+ kustomize component `k8s/components/app`.
23
+
24
+ pubdeploy depends on PyYAML alone and imports nothing of pubrules's. The few
25
+ helpers it shares with pubrules (`gh`, the storage journal client, the
26
+ secrets-block reading) are copies, each marked with the pubrules original.
27
+
28
+ A repository with a deploy block runs its pinned pubdeploy as `./pubdeploy`,
29
+ which `pubrules sync` writes beside `./pubrules`, with each `deploy.providers`
30
+ wheel, so the need types of other packages (pubsec's, pg-backup's) load. It
31
+ runs the release wheel pubrules.json pins, through `uvx`, checking its sha256 on
32
+ every run:
33
+
34
+ ```sh
35
+ ./pubdeploy --version
36
+ ./pubdeploy types
37
+ ./with-secrets ./pubdeploy walk --fulfil
38
+ ```
39
+
40
+ ## What pubrules calls
41
+
42
+ pubrules imports nothing of pubdeploy. It loads two entry-point groups, both
43
+ pubdeploy's:
44
+
45
+ - **`sustenaut.sync`**, `check(root, config, check) -> (lines, failed)`:
46
+ `pubrules sync` calls it after its own work, and `--check` fails when it
47
+ says so. It is the deploy block's offline check (the block and its needs,
48
+ the manifests against them, the units against the deploy workflow).
49
+ It no longer seeds a scheduled walk's caller: the convergence engine walks
50
+ every deploy service. `pubdeploy sync-check [--check]` runs it alone.
51
+ - **`sustenaut.bump`**, `bump(root) -> lines`: `pubrules bump` calls it after
52
+ moving pubrules's own pins. It moves each `deploy.providers` wheel to its
53
+ latest release, then this repository's uses of pubdeploy's reusable
54
+ workflows to the pubdeploy that `deploy.providers` names, and rewrites a
55
+ caller of k8s-deploy.yml still started by `push` to `workflow_run`.
56
+ `pubdeploy bump` runs it alone.
57
+
58
+ ## Deploying a service to k3s
59
+
60
+ `k8s-deploy.yml` is a service's whole deploy workflow, for a server whose
61
+ deploy key runs one program (a forced command) that applies the manifests it
62
+ reads in the namespace its command names, and waits. It deploys a commit only
63
+ once the repository's CI has passed on it, never beside it:
64
+
65
+ ```yaml
66
+ name: deploy
67
+ on:
68
+ workflow_run: { workflows: [main], types: [completed], branches: [main] }
69
+ workflow_dispatch:
70
+ permissions: { actions: read, contents: read, packages: write }
71
+ jobs:
72
+ deploy:
73
+ uses: Sustenaut/pubdeploy/.github/workflows/k8s-deploy.yml@<commit> # vX.Y.Z
74
+ with:
75
+ image: ghcr.io/example/example-app
76
+ deploys: |
77
+ k8s example
78
+ k8s-other example-other DEPLOY_OTHER
79
+ check: docker run --rm "$IMAGE" python -c 'import example.main'
80
+ secrets: inherit
81
+ ```
82
+
83
+ - The `ready` job finds the commit and waits until it has passed. Started by
84
+ `workflow_run`, the commit is the one that run tested, and the run deploys
85
+ only when that run passed on a push to the default branch: a pull
86
+ request's or the merge queue's run of the same workflow skips every job.
87
+ However it starts, `ready` then waits, up to `wait` minutes (30), for the
88
+ latest push run on the commit of each workflow `require` names (`main
89
+ ?integrations`), and fails, naming it, if one did not pass. A name
90
+ starting with `?` is waited for only where the repository has such a
91
+ workflow, so `integrations` counts wherever there is one. `main` is the
92
+ workflow `pubrules sync` writes for pushes to main; a repository whose CI
93
+ is another workflow names it in `require`, and in `workflow_run`.
94
+ - The `build` job checks out that commit and builds `image`, tagged with it,
95
+ with pubhost's `actions/docker-build` (its layer cache scoped to the image's name,
96
+ and `build-args`), runs `check` with the image in `IMAGE`, and only then
97
+ pushes it to GitHub's container registry with the job's token.
98
+ - The `deploy` job runs once for each line of `deploys`: an overlay
99
+ directory, the unit the server's program deploys it as, and optionally a
100
+ variable that must be `on` for that unit to deploy (one whose namespace and
101
+ Secrets are not on the server yet). It sets the image in the overlay with
102
+ `kustomize edit set image`, runs `render` there if given, with the
103
+ caller's variables as JSON in `VARS` (to add a ConfigMap from them with
104
+ `jq -r .NAME <<<"$VARS"`, say), renders it with `kustomize
105
+ build`, and fails if the image is not in what it rendered. Every overlay is
106
+ rendered on every run, so a broken one fails even while it is off.
107
+ - A commit the default branch has moved on from is not deployed: the newer
108
+ commit's own run deploys it, so an older one never lands over it. A deploy
109
+ by hand (`workflow_dispatch`) deploys the commit it is given, once that
110
+ has passed.
111
+ - It deploys with `actions/deploy-ssh`: the unit as the command, the rendered
112
+ manifests as `input`, to `deploy@` the variable `SUSTENAUT_SERVER`, with the
113
+ host key in the variable `SUSTENAUT_KNOWN_HOSTS` and the key in the secret
114
+ `SUSTENAUT_DEPLOY_SSH_KEY`, after joining the tailnet as `tag:ci` with the
115
+ secrets `TS_GH_OAUTH_CLIENT_ID` and `TS_GH_OAUTH_SECRET`. Without
116
+ `SUSTENAUT_SERVER` the image is built and pushed, and nothing is deployed.
117
+ - The deploy job runs in the environment `environment` names, `server` by
118
+ default, so a public repository can keep the deploy secrets to main. A
119
+ secret or variable set on the repository instead reaches it too, and GitHub
120
+ makes the environment on its first run if it is not there.
121
+ - A unit deploys one run at a time (concurrency group `deploy-<unit>`), so the
122
+ caller sets no concurrency group of its own with that name.
123
+
124
+ `push: { branches: [main] }` still works in place of `workflow_run`: `ready`
125
+ then holds a runner while it waits for CI. Either way the caller grants
126
+ `actions: read`, with which `ready` reads the commit's runs.
127
+
128
+ **Moving to it from 0.30.** In the deploy workflow, move the pin; replace
129
+ `push` with the `workflow_run` above (on the CI workflow's name, `main` where
130
+ `pubrules sync` writes it); and add `actions: read` to `permissions`. A
131
+ repository without a `main` workflow also gives `require`. Nothing else
132
+ changes: inputs, secrets and variables are as they were.
133
+
134
+ **Checking the Secrets before anything is applied.** The runner sees no
135
+ cluster, only the forced command, so the check runs there, on the manifests
136
+ it reads, before it applies them: `pubdeploy k8s preflight` reads every Secret
137
+ the pods use (env `secretKeyRef`, `envFrom`, secret and projected volumes,
138
+ `imagePullSecrets`, of containers and init containers), each must be there
139
+ with every key a reference names, unless it is `optional: true`, and a
140
+ value of a known shape must have it. oauth2-proxy's cookie secret
141
+ (`OAUTH2_PROXY_COOKIE_SECRET`, or `--cookie-secret` on oauth2-proxy's image)
142
+ must be 16, 24 or 32 bytes as it is or after URL-safe base64 decoding, as
143
+ oauth2-proxy itself requires; standard base64, with `+` or `/`, is named as
144
+ the likely mistake. It fails in the seconds kubectl takes, naming the
145
+ Secret, the key and what reads it, and never prints a value. `pubdeploy k8s
146
+ diagnose` shows, for each pod that is not ready or restarted, its
147
+ containers' state and last exit, its warning events and its last log
148
+ lines (the crashed container's too).
149
+
150
+ `src/pubdeploy/k8s.py` needs nothing else of pubdeploy's, and PyYAML (which
151
+ cloud-init brings) only for YAML, so a server installs that one file, pinned
152
+ by the commit of a release and its sha256, and its deploy program calls it:
153
+
154
+ ```bash
155
+ check() { python3 /usr/local/lib/pubrules/k8s.py "$@" --namespace "$app" \
156
+ --kubectl "/usr/local/bin/k3s kubectl --kubeconfig $cfg"; }
157
+ check preflight < "$manifests" || exit 1
158
+ k apply -f "$manifests"
159
+ for d in $(k get deployment -o name); do
160
+ k rollout status "$d" --timeout=3m || { check diagnose; exit 1; }
161
+ done
162
+ ```
163
+
164
+ The deploy identity needs `get` on `secrets` in its namespace for the
165
+ preflight. It reads no more than it could before: it already makes pods
166
+ that mount any Secret there, and reads their logs. Three minutes is enough
167
+ for a rollout whose pods start within the app component's minute; a
168
+ longer wait only delays the failure.
169
+
170
+ Pin it like pubrules's workflows, to the commit of a pubdeploy release: the one
171
+ `deploy.providers` names, to which `pubrules bump` moves it.
172
+
173
+ **The app component.** `k8s/components/app` is a kustomize Component with the
174
+ hardening and probes every service's app shares. It patches the Deployment
175
+ named `app` and its container named `app`, which serves `/healthz` and
176
+ `/livez` on the port named `http`, and leaves every other Deployment (a
177
+ sign-in proxy, a database) alone:
178
+
179
+ ```yaml
180
+ components:
181
+ - https://github.com/Sustenaut/pubdeploy//k8s/components/app?ref=<commit> # vX.Y.Z
182
+ ```
183
+
184
+ - The pod runs as user 1000, not root (`runAsNonRoot`), under the runtime's
185
+ seccomp profile, with no API token (`automountServiceAccountToken: false`);
186
+ a token projected for another audience is a volume of its own and is kept.
187
+ - The container can't gain privileges, drops every capability and can't
188
+ write its root filesystem, so whatever it writes goes to a volume it mounts
189
+ (an `emptyDir` at `/tmp`, say).
190
+ - `startupProbe` waits up to a minute for `/healthz`; `livenessProbe`
191
+ restarts the container after a minute without `/livez`; `readinessProbe`
192
+ sends it requests only while `/healthz` answers. `/livez` should only say
193
+ the process is up; `/healthz` checks what the app can't serve without.
194
+
195
+ A kustomization's own patches come after its components', so a service that
196
+ needs something else (its API token, a longer start) patches it back in its
197
+ own `patches`. Pin the component by commit, as above: every service takes it,
198
+ so a change to it reaches each one only when that service moves its pin.
199
+
200
+ ## Declaring what a service needs
201
+
202
+ A service needs more than its manifests before it runs: a namespace, the
203
+ Secrets its pods read and what fills them, GitHub secrets and variables for
204
+ its deploy, the services it calls, its host in the DNS. pubrules.json's
205
+ `deploy` block says so, and `pubdeploy check` checks each, changing
206
+ nothing:
207
+
208
+ ```json
209
+ "deploy": {
210
+ "namespace": "example",
211
+ "host": "example.example.eu",
212
+ "workflow": "deploy.yml",
213
+ "gate": "DEPLOY_EXAMPLE",
214
+ "overlay": "k8s",
215
+ "needs": [
216
+ {"type": "bitwarden.generated", "name": "example-cookie-secret", "shape": "oauth2-proxy-cookie",
217
+ "secret": {"name": "example-page-secrets", "key": "cookie-secret"}},
218
+ {"type": "k8s.manifests", "path": "k8s/admin/network-policy.yaml"},
219
+ {"type": "upstream", "name": "storage", "health": "https://storage.example.eu/healthz"},
220
+ {"type": "service.health", "path": "/healthz", "job": "cronjob/worker", "after": ["upstream:storage"]}
221
+ ]
222
+ }
223
+ ```
224
+
225
+ `namespace` is the only key it must have. `workflow` (default `deploy.yml`)
226
+ must be in `.github/workflows`, and `overlay` (default `k8s`) must hold a
227
+ `kustomization.yaml`. Each need has a `type`, maybe a `name` (each type has a
228
+ default; a need's id is `TYPE:NAME`), maybe `after`, the ids it waits for,
229
+ and its type's keys; `pubdeploy types` lists them. A need with a `secret`
230
+ (`{"name", "key"?, "decode"?: "base64"}`) says it fills that Kubernetes
231
+ Secret.
232
+
233
+ The block and the other blocks imply the rest, so they are not written
234
+ twice: the namespace a `k8s.namespace`, a `deploy.rollout` and a
235
+ `k8s.secrets-declared`, the host a `dns.site` and a `service.health`, the gate
236
+ a `github.gate`; each entry of `secrets` and `github_app` a `github.secret` or
237
+ `github.variable`, and the server's settings a `github.deploy-settings`; each
238
+ of `variables.names` a `github.variable` that must be set; each need's
239
+ `secret`, and each Secret the rendered overlay reads, a `k8s.secret`.
240
+ `k8s.secrets-declared` holds the manifests and the needs to each other, and,
241
+ when the walk reads the cluster (not offline, not over the deploy channel), says
242
+ which Opaque Secret in the namespace nothing reads or declares, with its delete:
243
+ one left behind when a need went, such as a key's once the service reaches Google
244
+ without one.
245
+
246
+ `pubdeploy check` walks them in order: the namespace before what is in
247
+ it, what fills a Secret before the Secret, the gate after everything but the
248
+ rollout and the service's health. Each is `ready`, `missing`, `drift`,
249
+ `blocked` (missing while something it waits for is not ready) or `unknown`
250
+ (not checked, refused, or over its time), with what to do about it. It exits
251
+ 0 when every need is ready, 1 when one is not, and 2 for a bad block.
252
+
253
+ ```
254
+ ./with-secrets ./pubdeploy check # everything, Bitwarden included
255
+ ./pubdeploy check --offline # the manifests and the files only
256
+ ./pubdeploy check --zones ../infra/pubrules.json # the hosting block that keeps the zone
257
+ ./pubdeploy check --json # the walk as a deploy-journal/2 document
258
+ ```
259
+
260
+ It reads the cluster through `kubectl` (`--kubectl CMD` for another, such as
261
+ `k3s kubectl --kubeconfig FILE`), GitHub through `gh`, health addresses over
262
+ HTTPS, the DNS through `dig`, and Bitwarden only as `./with-secrets` gives
263
+ it: a Bitwarden need is looked for by name in the environment, and is
264
+ `unknown` outside with-secrets. It never prints a value: a secret is named,
265
+ with its length and kind. `--zones` reads only a file named `pubrules.json`, in
266
+ this repository or a checkout beside it, and may be given once per owner whose
267
+ zones the units' hosts are in. A host in a zone none of them holds is `unknown`,
268
+ not `missing`: its owner's zones were not given.
269
+
270
+ `pubrules sync --check` checks the block too, and runs the offline walk: every
271
+ Secret the manifests read must be one a need fills, with the keys they read,
272
+ and every Secret a need fills must be read, so a manifest that reads a new
273
+ Secret fails CI until the block says where it comes from. What it cannot
274
+ check (no kustomize, a zone kept in another repository) it says, and fails
275
+ nothing for.
276
+
277
+ A package may offer more need types under the entry-point group
278
+ `sustenaut.needs`, each `module:object` naming an instance (not a class)
279
+ with a `type`, a JSON `schema` of its keys that takes no other, a `grade`, a
280
+ `budget` in seconds (at most 10), `offline`, and `default_name`,
281
+ `depends_on`, `check` and `fulfil`; and optionally a `fulfil_budget` (at
282
+ most 900 seconds, default 60). It need import nothing of pubdeploy's: any
283
+ result with `state`, `summary`, `fix` and `details` serves. The walk takes
284
+ `type`, `name`, `after` and `unit` out of a need's params, and a `secret` on
285
+ any need implies the `k8s.secret` it fills. One pubdeploy cannot use is left
286
+ out with a warning. `deploy.providers` names the packages a repository's
287
+ needs use, and the managed `./pubdeploy` runs with each:
288
+
289
+ ```json
290
+ "providers": [{"distribution": "example-needs", "sha256": "…",
291
+ "wheel": "https://github.com/example-org/example-needs/releases/download/v1.0.0/example_needs-1.0.0-py3-none-any.whl"}]
292
+ ```
293
+
294
+ uv does not check a `--with` address's `#sha256`, so `./pubdeploy` fetches each
295
+ wheel once into the user's cache and checks it against its sha256 on every
296
+ run before uv sees it. `pubrules bump` moves each to its latest release
297
+ (pubdeploy's sustenaut.bump hook). A need whose type no installed package offers
298
+ names the providers that would.
299
+
300
+ ### Making what a service needs
301
+
302
+ `pubdeploy walk` is the check (`pubdeploy check` is it without `--fulfil`);
303
+ with `--fulfil` it then makes what is missing:
304
+
305
+ ```
306
+ ./with-secrets ./pubdeploy walk --fulfil --server server.example.eu
307
+ ./pubdeploy walk --unit other --json
308
+ ```
309
+
310
+ It checks every need first, then prints the plan, a line per need it will
311
+ act on, and waits for a typed `yes` (`--yes` skips that, for a scheduled
312
+ run). It visits the needs in order again: each missing one whose grade is
313
+ auto, whose dependencies are ready, is fulfilled and checked again, and
314
+ that check is its result, `fulfilled`. A fulfil that fails leaves the need
315
+ as it was, with why, and the walk goes on with what does not wait for it.
316
+ A Secret that is ready by its keys is made again, and what reads it
317
+ restarted, when the walk makes a new value of a need it is made from, such
318
+ as a generated secret that was misshapen: otherwise Bitwarden and the
319
+ cluster would differ. A propose need is never acted on: its fix is the change to make, such as
320
+ the JSON entry and the file it goes in; a human one's is the step. What auto
321
+ does, type by type:
322
+
323
+ | type | its fulfil | it needs |
324
+ | --- | --- | --- |
325
+ | k8s.namespace | `sustenaut-app NS` on the server | the admin channel |
326
+ | k8s.manifests | `kubectl apply --server-side --force-conflicts -f -` of the file, each object in the namespace it names (fields applied client-side before move to server-side ownership); its check also runs `kubectl diff --server-side --force-conflicts`, so an object there but changed is drift, and a dry run the channel may not make is unknown | kubectl that writes |
327
+ | k8s.cronjob, service.health | a Job from the CronJob, waited for; a Job made by hand, NAME-pubrules-... or NAME-now, counts as the CronJob's success, as kubectl does not make it the CronJob's | kubectl that writes |
328
+ | k8s.job-once | a one-off Job named as the need (a backfill, a migration), made with `kubectl create -f -` from a CronJob's template (`cronjob`) or a manifest holding one Job (`manifest`), `args` added to its first container's, `timeout` its active deadline, its time-to-live dropped so it stays as the record; the fulfil returns once it is made. Its check: ready when the Job completed, unknown while it runs, missing when it is not there, drift when it failed. A failed Job is made again only with `rerun: if-failed`, and the plan names the one Job that deletes ("no data is deleted") | kubectl that writes (and deletes Jobs, for a rerun) |
329
+ | github.gate | the gate set on, once every need before it is ready, and the workflow run | gh |
330
+ | deploy.rollout | the workflow run on the default branch, and waited for | gh |
331
+ | auth.sign-in-client | its Secret made on the server, from the client kept there; or the approval record written into `approvals` | the admin channel |
332
+
333
+ The eight types that hold a secret's value are
334
+ [pubsec](https://github.com/Sustenaut/pubsec)'s: `k8s.secret` (a Secret made
335
+ from the values of the needs that fill it, on kubectl's standard input),
336
+ `github.secret`, `github.variable` and `github.deploy-settings` (set from
337
+ Bitwarden through gh), `bitwarden.secret`, `bitwarden.generated`,
338
+ `bitwarden.adopt` (a value a cluster Secret already holds, copied into Bitwarden) and
339
+ `gcp.service-account-key`. pubsec's README, "Making what a service needs",
340
+ says what each checks and does. A repository with a deploy block pins its
341
+ own pubsec in `deploy.providers`, and `pubrules bump` moves it to pubsec's
342
+ latest release, as it does every provider. pubrules keeps a floor, the oldest
343
+ pubsec it works with (`PUBSEC_FLOOR`, which its suite tests): `pubrules sync`
344
+ lists the floor where none is listed and moves one below it up, and `sync
345
+ --check` fails without one or below it, so `./pubdeploy` always walks with
346
+ them. The floor moves only when pubrules needs a newer pubsec, such as for a
347
+ change to the provider contract. Where pubsec is not installed, a need of
348
+ one of its types says so.
349
+
350
+ `dns.site`, `storage.tenant` and `github.variable-entry` propose; `upstream`
351
+ and `k8s.secrets-declared` have no fulfil, nor has `k8s.jwks`, a person's step:
352
+ it compares a file that pins the cluster's service-account signing keys (for
353
+ Google's Workload Identity providers and storage's principals) with the
354
+ cluster's own, read over the admin channel or from the plane, as the deploy
355
+ channel cannot; with `keys` (2), the cluster must publish that many, one signing and
356
+ one its fallback. Before it fulfils a need that
357
+ wants the Bitwarden write token, at a terminal, the walk calls each
358
+ `sustenaut.walk` hook, `prepare(ctx, plan, terminal)`: pubsec's asks for the
359
+ token once, at a hidden prompt, and gives the walk its writer, kept only by
360
+ the process. pubdeploy itself holds no write token. A walk from GitHub Actions,
361
+ or with `--yes` and no terminal, asks for none, and those needs are unknown,
362
+ saying so.
363
+
364
+ ### A live run: the needs as a state machine
365
+
366
+ The convergence engine can run a unit's needs live, each on its own, in parallel where
367
+ they allow it. `pubdeploy machine` gives it the unit's needs as a state machine, and
368
+ `walk --need` walks one need at a time; both are here so every need stays runnable from a
369
+ laptop as well.
370
+
371
+ ```
372
+ ./pubdeploy machine --unit example # the machine, as XState v5 JSON
373
+ ./pubdeploy machine --unit example --mode check --modes upstream:storage=fulfil
374
+ # --mode check: every need a check; --mode fulfil: only the default, so a need the run
375
+ # section keeps to a check stays one; --modes, a need's own, is over both
376
+ ./with-secrets ./pubdeploy walk --fulfil --yes --need k8s.secret:example-files
377
+ ./pubdeploy check --need upstream:storage --budget-seconds 30
378
+ ```
379
+
380
+ **The machine** (`machine.py`) is a documented subset of XState v5 config, so Stately's
381
+ editor and inspector draw it and XState runs it as python-statemachine (`machine.load`)
382
+ does: `id`, `initial`, `states`, `type` (parallel, final), `on` (event to `{target,
383
+ guard?}` or a list), `entry`, `exit`, `meta`; guards and actions are names alone. The
384
+ root's state `running` is parallel, a region per need, keyed by the need's id as a name
385
+ may hold it, with the need in its `meta` (id, type, grade, mode, the keys it is after, its
386
+ seconds, and how a Stop treats it). A region goes waiting, checking, then ready, found
387
+ (missing or drift), unknown (what the walk could not judge), blocked or failed; in mode fulfil a found need
388
+ of grade auto (the one grade a walk acts on) goes on `approved` to fulfilling, which ends in a re-check;
389
+ `predecessor_failed` blocks a waiting need, and `stop` stops any need not final. A runner
390
+ blocks the needs after one that ends unknown, blocked, failed or stopped, and fails a run
391
+ only on a need failed or blocked: a run whose only needs not ready are unknown ends done,
392
+ with a note. Events
393
+ carry what the guards read: `checked.KEY` a `state`, `fulfilled.KEY` an `ok`.
394
+
395
+ **The run section** of the deploy block says how a live run runs the needs, and may only
396
+ narrow what is generated:
397
+
398
+ ```json
399
+ "run": {"schema": 1, "default_mode": "fulfil", "parallel": 4,
400
+ "needs": {"pg.database:auth": {"mode": "fulfil", "after": ["k8s.namespace:auth"]}}}
401
+ ```
402
+
403
+ Without it, every need is in `default_mode` (fulfil), after the needs the walk's order
404
+ already has it wait for, 4 at a time; the needs of a serial kind are chained, and so are
405
+ needs touching one resource (one Secret, Terraform state or database). A section may
406
+ change a need's mode, add after-edges and lower `parallel`; naming an unknown need, an
407
+ edge that makes needs wait for each other, or a higher `parallel` fails `pubrules sync
408
+ --check`. `schema` is its version: a newer one says to move to a newer pubdeploy, and an
409
+ older one is brought up by `machine.TRANSFORMERS`, one per change of shape.
410
+
411
+ **A need type's run metadata**: `serial` (chain a kind's needs) and `stop`, what a Stop does
412
+ while the need is fulfilled: `signal` (the walk is ended; the default), `shorten` (it ends
413
+ when the time left is up: k8s.cronjob, service.health) or `continue` (it runs on, with a
414
+ `stop_warning` saying why: deploy.rollout, k8s.job-once).
415
+
416
+ **`walk --need ID`** (repeat it) walks only those needs: the needs they wait for, by the
417
+ walk's order and the run's after-edges, are checked, never fulfilled; when one they wait
418
+ for directly is not ready, nothing is done (exit 2, naming it). Only the named needs are
419
+ said, journaled and, with `--fulfil`, fulfilled. `--budget-seconds S` caps every check and
420
+ fulfil at S seconds: what is left of a run's time. A live run's Stop for a need that shortens sends the walk
421
+ `SIGUSR1`: what is running ends at once as unknown ("a Stop shortened it"), and the walk
422
+ prints and journals what it has.
423
+
424
+ ### Provider contract
425
+
426
+ A package that offers need types under `sustenaut.needs`, such as pubsec or
427
+ pg-backup, or hooks under `sustenaut.walk`, imports nothing of pubdeploy in
428
+ its code. Its tests may import two modules of pubdeploy, and no other:
429
+ `pubdeploy.contract`, the contract below as code (its name sets, `Need`,
430
+ `Result` and `Ran`, and `problems(kind)` and `hook_problems(hook)`, the checks
431
+ the walk makes of a type or a hook as it loads one, in the words it prints),
432
+ and `pubdeploy.testing`, the deploy walk's test harness. The names are the
433
+ contract: a provider checks what it relies on by their inclusion, not by a
434
+ version, and a change that drops or renames one is a breaking change, said
435
+ in pubdeploy's changelog, with a follow-up in each provider. A scenario keeps
436
+ these lists and the module's equal.
437
+
438
+ Context: a check or a fulfil uses the walk's Context by these names: `root`,
439
+ `config`, `block`, `namespace`, `unit`, `channel` (admin, deploy, plain or
440
+ plane), `kubectl(args, input=None)`, `server(args, input)`, `gh(method, path,
441
+ body)`, `send(kind, name, repo, value, environment)`, `bitwarden_writer`
442
+ (None, or an object whose write(name, value, note) writes), `http_get(url,
443
+ timeout)`, `env`, `now()`, `rendered()`, `offline`, `slug`, `zones`,
444
+ `resolve(host, type)`, `values` (by Bitwarden name, bytes), `types`, `needs`,
445
+ `results`, `sleep(seconds)` and `dispatched`.
446
+
447
+ Result: a result is any object with `state`, `summary`, `fix` and `details`,
448
+ and optionally `fixable` and `step`.
449
+
450
+ Need: a need has `id`, `type`, `name`, `params`, `implied_by` and `after`.
451
+
452
+ Type: a type has `type`, `schema`, `grade`, `budget`, `offline`,
453
+ `default_name`, `depends_on`, `check` and `fulfil`, and optionally
454
+ `fulfil_budget`, `singleton`, `grade_for`, `requires`, `action`, `value`,
455
+ `secret_keys`, `secret_shape`, `made_from`, `serial`, `stop` and
456
+ `stop_warning`. Each entry point names an instance, not a class.
457
+
458
+ Hook: a hook under `sustenaut.walk` is a function, prepare(ctx, plan,
459
+ terminal), returning a line to print or None.
460
+
461
+ The channels: with `--server HOST` (or `SUSTENAUT_SERVER`) and no
462
+ `--kubectl`, kubectl is `ssh root@HOST k3s kubectl`, and a command can run
463
+ on the server: the admin channel, everything a fulfil does. Every command
464
+ goes over one SSH connection the walk opens before its first check and
465
+ closes at its end (ssh's `ControlMaster`, its socket in a directory only
466
+ the user may enter): a new connection over a tailnet takes about a second,
467
+ longer than a check's budget. `--kubectl 'ssh
468
+ -i KEY deploy@HOST {unit} kubectl-read'` is the server's read-only verb, a
469
+ JSON line a call over one connection, in the unit's namespace (`{unit}`):
470
+ every write is unknown there, the namespace is judged on the admin channel
471
+ only, and a Secret is read by its key names (`secret-keys`) or, on a server
472
+ without that verb, by name alone. Any other `--kubectl`, or `kubectl`, is
473
+ what its kubeconfig may do. A value moves only on a standard input: never
474
+ in an argv, a file, a log or the journal.
475
+
476
+ `deploy.units` adds units beside the top level's, each with its own
477
+ `namespace`, `overlay`, `gate` and `host`; they share the workflow and every
478
+ need with no `unit` (`"unit": "other"` scopes one). Each is walked and
479
+ journaled on its own; `--unit` walks one. When the workflow calls
480
+ k8s-deploy.yml, `pubrules sync --check` fails unless its `deploys` lines and
481
+ the units agree, and, from k8s-deploy.yml 0.31.0, unless the workflow
482
+ starts on `workflow_run` of main with `actions: read`, and on
483
+ `workflow_dispatch`, by which the gate's and the rollout's fulfils run it;
484
+ `pubrules bump` rewrites a `push` trigger when it moves that pin across 0.31.0.
485
+
486
+ `--json` prints a deploy-journal/2 document per unit (a list for more than
487
+ one): each need with its grade and whether it was fulfilled. `--journal URL`
488
+ (or `PUBRULES_JOURNAL_URL`) also writes it to storage, as a file of the tenant
489
+ `plane`, and moves the head `OWNER.REPO.UNIT` (lower case, as storage's head names are) to it, If-Match its
490
+ ETag, trying once more when another walk moved it first. The token is
491
+ `PUBRULES_STORAGE_TOKEN`, or in GitHub Actions the job's OIDC token for
492
+ `PUBRULES_STORAGE_AUDIENCE` (default the address's host). A journal that
493
+ could not be written is said, and fails nothing.
494
+
495
+ pubdeploy's `deploy-check.yml` walks it every 30 minutes, read-only, over the
496
+ deploy channel, with the deploy key, host and tailnet secrets k8s-deploy.yml
497
+ takes and the environment's read-only `BWS_ACCESS_TOKEN`, with
498
+ `--tolerate-unknown`: it fails when a need is missing, drift or blocked, and
499
+ a need the job cannot judge, such as a secret its token may not list, stays
500
+ unknown in the journal and fails nothing. `pubrules sync` no longer seeds its
501
+ caller, `.github/workflows/deploy-check.yml`: the convergence engine walks every
502
+ deploy service, and a repository removes its caller once the engine shows it.
503
+ And every repository with a `with-secrets.json` gets the managed
504
+ `with-secrets`: an older copied script is why `bitwarden.*` needs were
505
+ unknown, and `pubrules sync --check` says so.
506
+
507
+ ## Developing
508
+
509
+ ```sh
510
+ make venv # a hash-locked environment, this checkout installed editable
511
+ make functional # the Gherkin suite, with its coverage
512
+ make check # what CI runs
513
+ ```
514
+
515
+ The suite walks pubsec's seven need types (a test dependency), so the canary
516
+ and the fulfil scenarios cover the whole walk.
517
+
518
+ ## Licence
519
+
520
+ Apache-2.0.
@@ -0,0 +1,45 @@
1
+ pubdeploy/__init__.py,sha256=d2dwApWWPYf-puJTGAwa44stmeCUPuS2yA_WPUljs0U,26580
2
+ pubdeploy/channels.py,sha256=lvZwgWcpaIqBODyjalrvminPzcfE7Sbv0YU_AZ_6Ptw,8142
3
+ pubdeploy/cli.py,sha256=FZsXQuQbWI-EhGlatAmnnRt3deBGpwSsKnjq-hxQr74,7838
4
+ pubdeploy/config.py,sha256=y-5afit4_zXRlZsgjEG7ohkIDfD0JkrGTODBP6nCtWM,6068
5
+ pubdeploy/contract.py,sha256=75futAQTJHAzCU561SgGEk9yLFLIpfGcbyssHEKqxwI,4481
6
+ pubdeploy/github.py,sha256=GL4txguw-uf5dP08iMZsVvu2rrTTf8qsoSgj9mXhoOY,2701
7
+ pubdeploy/hooks.py,sha256=ZofOtPIGkLl82_X5EqUyHND8_J7C6MpbP4T5PFIQSqI,3860
8
+ pubdeploy/implied.py,sha256=7nnst7Ln8irqnxXSrayWI7XgE0LBANVGoo_60vrS4ic,8906
9
+ pubdeploy/journal.json,sha256=DUEcVv-EkP-bK-88HHFEJVa1cFpFgejgBbnoHbS12jQ,4315
10
+ pubdeploy/k8s.py,sha256=rAvZvUDJxRGV0jyYMsglsXEHyVYSXzFXic6KzLaeWTQ,22160
11
+ pubdeploy/machine.py,sha256=7eXFocJQPa9D7NfHVtKKjw3Tea_W9AIIOIQTt73j2zg,18298
12
+ pubdeploy/model.py,sha256=breDUbmOpbEq_dJPy3hwjBIFXAExtmigIkMMQ9AULYQ,9862
13
+ pubdeploy/names.py,sha256=ncv9ZE1a38-FBYuu8s_GxRT1hXtJRR5hkdthW2mkBs0,846
14
+ pubdeploy/order.py,sha256=LGM-hucq_-f9pNMffdv2l9AxOXyUJ0i6nKS58GroOj4,4067
15
+ pubdeploy/providers.py,sha256=y63P70RvuGjcL8pvdxW84AVb3Fn6FQzidj03nG7AjRs,3336
16
+ pubdeploy/registry.py,sha256=2pxdnByol-wn_HGzFtPNpmLQf7uI5Kh6H4WGhrtVXXI,4523
17
+ pubdeploy/release-commit,sha256=nplB5A5XFlLZJzlT5ePcZXRoLzxn9F9K8GzyEbL_v3I,41
18
+ pubdeploy/schema.json,sha256=SWnHfHQ5yEKTo0reMaEst0qS8Zh2hCE_wHlbK8PjgxM,4881
19
+ pubdeploy/schema.py,sha256=CoYq3WZzUZt6tamK4alYebFGYtBtVyMxYVulLlVbbWg,11865
20
+ pubdeploy/store.py,sha256=bTkmKtneUkdRX4mkioLT0jbfGTdw5DU8BPuIk1sDvEU,5864
21
+ pubdeploy/units.py,sha256=avzkjfIcEMrZ6nboOOAY_kO3y0MsIEkJP6UYW_DExXY,7953
22
+ pubdeploy/walk.py,sha256=gxztNFjk6iJP80HwWUPqsLCV35FnKgKtwuxjP-hIIbg,15937
23
+ pubdeploy/testing/__init__.py,sha256=JzBGfu-u2vud1gyKjzc_7_ZUwaLkWAp5iZaAOZczHJY,4301
24
+ pubdeploy/testing/plugin.py,sha256=yyC32b4hePZ-aHv0uNzeATy64BfEvuFSTIJLKYVmSQc,1084
25
+ pubdeploy/testing/steps.py,sha256=DskxQu9L3FmVGCgwkHPAJ0wyI7YEDiP-EiRfnbD5w0o,39338
26
+ pubdeploy/testing/world.py,sha256=tLwg1lABGPl9KOd0fh1EE9e_MBANTTkMKYPOqQEfyMk,30918
27
+ pubdeploy/testing/fixtures/deploy-app/pubrules.json,sha256=dIkFf_6_SLxEdwgVGT1KhHJ4ret29aFgZwLwqcATVc8,1208
28
+ pubdeploy/testing/fixtures/deploy-app/rendered.yaml,sha256=Y92-bSSB7xQPtQ_S5qME0aKGBQ1m_ZheefiSOZ1-A2I,1826
29
+ pubdeploy/testing/fixtures/deploy-app/zones.json,sha256=3haezqIOEvyCRFdASm9wo_-t3bBUXAMnaR3fwLsQi98,183
30
+ pubdeploy/testing/fixtures/deploy-app/.github/workflows/deploy.yml,sha256=00zLGKVdDwRDyi_1vV4RW1g0Cc1sajBd4vuyako1AY8,310
31
+ pubdeploy/testing/fixtures/deploy-app/k8s/app.yaml,sha256=e9v2MmfbZ6ZWI-xlot1Nvn03lEdrTDJO-x5bivP_MAI,1748
32
+ pubdeploy/testing/fixtures/deploy-app/k8s/kustomization.yaml,sha256=lX3g0aNjll7SVQRVE84k9I02K_7i5xH3IaygZxK6Qnk,178
33
+ pubdeploy/testing/fixtures/deploy-app/k8s/admin/network-policy.yaml,sha256=armH53JxoQtBu16vKv1CEl7pwNk8XeG5RlfrkJKZsoM,303
34
+ pubdeploy/types/__init__.py,sha256=2XASQljQG_gRKNzsaTb_m61s5K-w8DZSdpDh9BotJYg,7979
35
+ pubdeploy/types/auth.py,sha256=N2kS0c8P3QQapEcofO54cEFSvN3NcyPASlRsj6DtQIY,12501
36
+ pubdeploy/types/github.py,sha256=jtsyTw5NIPlWlPFZQfne3Tefp6xPeI2NQ6Qnyg3IHPU,5461
37
+ pubdeploy/types/k8s.py,sha256=-X1_zhuiJE0k8UNuP_cahE3H5GKwR4PMx74F8WDRdSw,33791
38
+ pubdeploy/types/net.py,sha256=yEHq-1nhhqYuI33Trfkt4WxISWDCak8pyvJtRzGRvSo,12517
39
+ pubdeploy/types/rollout.py,sha256=92jkabSsufu2CqXunRPJZyKYumUa56qMSC8_33X85uQ,10114
40
+ pubdeploy-0.8.0.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
41
+ pubdeploy-0.8.0.dist-info/METADATA,sha256=racyDGF-9liBp2NMdbF2QzmQMz1n9DTC4tcbL5fl_FM,29867
42
+ pubdeploy-0.8.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
43
+ pubdeploy-0.8.0.dist-info/entry_points.txt,sha256=kY0oI3X_lK0rL3OwAttY3WqeWrb6kk4H7lelQWHZhkE,209
44
+ pubdeploy-0.8.0.dist-info/top_level.txt,sha256=_nu3jRFeFiQNNeeZD8ApOZodICPXdUH7HO5ljtrEsIE,10
45
+ pubdeploy-0.8.0.dist-info/RECORD,,
@@ -0,0 +1,5 @@
1
+ Wheel-Version: 1.0
2
+ Generator: setuptools (84.2.1)
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
5
+
@@ -0,0 +1,11 @@
1
+ [console_scripts]
2
+ pubdeploy = pubdeploy.cli:main
3
+
4
+ [pytest11]
5
+ pubdeploy.testing = pubdeploy.testing.plugin
6
+
7
+ [sustenaut.bump]
8
+ pubdeploy = pubdeploy.hooks:bump
9
+
10
+ [sustenaut.sync]
11
+ pubdeploy = pubdeploy.hooks:check