splent-feature-autodeploy 0.2.0__tar.gz → 0.3.0__tar.gz

{splent_feature_autodeploy-0.2.0/src/splent_feature_autodeploy.egg-info → splent_feature_autodeploy-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: splent_feature_autodeploy
-Version: 0.2.0
+Version: 0.3.0
 License-Expression: LGPL-3.0-or-later
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
@@ -12,12 +12,14 @@ Dynamic: license-file
 # splent_feature_autodeploy
 
 Release-gated continuous delivery for a SPLENT product. A product that
-selects this feature runs a small agent next to its web container in
-production. The agent follows a registry image channel, for example
-`git.splent.io/splent-io/<product>:stable`, and when the channel points to a
-new image it redeploys the product. The database is backed up first, the
-new version has to pass its health check, and an automatic rollback of both the container and the database when
-the new version does not come up.
+selects this feature runs a small agent in a container of its own next to
+its web container in production. The agent follows a registry image
+channel, for example `git.splent.io/splent-io/<product>:stable`, and when
+the channel points to a new image it redeploys the product on that image's
+digest. It backs up first (off the host too, when the product has
+`splent_feature_backup`), migrates once, waits for the new version's
+`/readyz`, and rolls the container back, and the database when the deploy
+changed it, when the new version is not ready.
 
 The channel only moves when someone runs `splent product:release`, so nothing
 reaches production that nobody released. The agent never builds anything and
@@ -29,41 +31,92 @@ Every `AUTODEPLOY_INTERVAL` seconds, and whenever an administrator asks,
 the agent goes through these steps.
 
 1. Pull the channel image. If its id is the one the web container runs,
-   there is nothing to do.
+   there is nothing to do. Otherwise the image is named from now on by its
+   digest (`<registry>/<org>/<product>@sha256:...`), which no push can
+   move: that is what the new container runs and what the history records.
+   An image the registry knows no digest of runs by its id, just as fixed.
 2. Skip an image that failed before, until the channel points elsewhere.
    Honour the pause flag and the update window (`AUTODEPLOY_WINDOW`). A
    check somebody asks for (Check now) deploys outside the window too.
-3. Back up the database. `mariadb-dump --single-transaction --routines
-   --triggers` runs as root inside the database container, with the
-   credentials that container already holds, and the gzipped dump is kept
-   only if `mariadb-dump` exited cleanly and the dump ends with its
-   `Dump completed` trailer. No complete backup, no deploy.
-4. Record what the web container runs (`splent version --json`).
-5. Recreate the web container on the new image with everything it was
-   configured with, the way Watchtower does it. That is its name, its
-   environment (the container's own values on top of the new image's), its
-   labels, mounts and binds, port bindings, restart policy, healthcheck, and
-   its networks with their aliases. Cmd, Entrypoint and WorkingDir follow the new image when the old
-   container only inherited them, and are kept when it chose them (the
-   compose file's `command:`). The old container is renamed
-   `<name>-previous` and stopped, not removed.
-6. Wait for the healthcheck, up to `AUTODEPLOY_HEALTH_TIMEOUT` seconds. The
-   entrypoint runs every feature's migrations before gunicorn starts, so this
-   is generous.
-7. Once it is healthy, remove the previous container and write the history
-   entry (from and to image, product version, the features whose version
-   changed, duration).
-8. If it turns unhealthy, exits, restarts or runs out of time, keep the new
-   container's last 80 log lines, remove it, drop and recreate the database and import the
-   backup (so tables the new migrations created do not linger), put the
-   previous container back under its name, wait for it to be healthy, and
-   refuse that image from then on. The history entry says `rolled_back` and
-   carries the log lines.
+3. Back up.
+   - When the product has `splent_feature_backup`, run its off-host backup
+     in the running web container, as the app's user:
+     `splent feature:backup run --tag pre-deploy-<short id of the image
+     being replaced>` (`AUTODEPLOY_BACKUP_COMMAND`). It is encrypted to keys
+     the server does not hold, and it is the copy that survives the host.
+   - Then the agent's own dump, the one a rollback restores:
+     `mariadb-dump --single-transaction --routines --triggers` runs as root
+     inside the database container, with the credentials that container
+     already holds, and the gzipped dump is kept only if `mariadb-dump`
+     exited cleanly and the dump ends with its `Dump completed` trailer.
+
+   No complete backup, no deploy.
+4. Record what the web container runs (`splent version --json`), and a
+   fingerprint of the schema: every table's columns and every migration
+   revision (`alembic_*`, `splent_migrations`).
+5. Create the new web container on the image's digest with everything the
+   old one was configured with, the way Watchtower does it. That is its
+   name, its environment (the container's own values on top of the new
+   image's), its labels, mounts and binds, port bindings, restart policy,
+   healthcheck, and its networks with their aliases. Cmd, Entrypoint and
+   WorkingDir follow the new image when the old container only inherited
+   them, and are kept when it chose them (the compose file's `command:`).
+   The old container is renamed `<name>-previous` and stopped, not removed.
+6. Migrate, once: a one-off container of the new image, `<name>-migrate`,
+   with the web container's environment, mounts and networks, runs
+   `AUTODEPLOY_MIGRATE_COMMAND` (`splent db:upgrade`) as the app's user
+   while neither version serves. It answers to no alias, so nginx never
+   finds it, publishes no port and is removed once it exits. The new
+   container's entrypoint upgrades again as it starts and finds nothing
+   left to do.
+7. Start the new container and wait, up to `AUTODEPLOY_HEALTH_TIMEOUT`
+   seconds, for `/readyz` to answer 200 (`AUTODEPLOY_READY_PATH`, asked by
+   the container's name on `AUTODEPLOY_WEB_PORT`), while Docker's
+   healthcheck does not say unhealthy. `/readyz` is the framework's
+   (1.22.0 and later): 200 when the database and the services the product
+   declares answer. An image without it answers 404, and the healthcheck
+   decides for it.
+8. Once it is ready, remove the previous container and write the history
+   entry (from and to image, the digest, product version, the features
+   whose version changed, the off-host backup, duration).
+9. If the migration fails, or the new container turns unhealthy, exits,
+   restarts or is not ready in time, keep its last 80 log lines (the
+   migration's, when that is what failed), remove it, put the database
+   back when the rule below says so, put the previous container back under
+   its name, wait for it to be healthy, and refuse that image from then on.
+   The history entry says `rolled_back`, carries the log lines and says
+   whether the database went back and why.
+
+### When the database goes back
+
+A rollback restores the agent's pre-deploy dump (dropping the database
+first, so tables the new migrations created do not linger) when:
+
+- the migration failed, or the agent died during it. MariaDB commits DDL as
+  it goes, so a failed migration can leave half of its changes behind
+  without recording them;
+- the migration finished, and the schema fingerprint differs from the one
+  taken before the deploy. The previous version cannot run on the new
+  schema, and a down-migration is not something to trust with production
+  data;
+- the fingerprint cannot be read, which counts as changed;
+- `AUTODEPLOY_MIGRATE_COMMAND=off`: the new container migrated as it
+  started, and the agent cannot tell a failed migration from anything
+  else.
+
+It does not when the deploy stopped before anything ran against the
+database (a failed backup, a container that could not be created), and when
+the migration finished without changing the schema: the previous version
+comes back on the same database, and nothing written while the new one
+tried to start is lost. The dump is taken right before the old version
+stops, so a restore loses only what the new version wrote during its
+attempt.
 
 Every step is written to `/data/state.json` before it is taken. An agent
 restarted in the middle of a deploy (a reboot, a redeploy of the agent
-itself) finishes it if the new container turns healthy, and rolls it back
-otherwise.
+itself) finishes it if the new container turns ready, and rolls it back
+otherwise; one restarted during the migration rolls back with the
+database.
 
 | State                | Meaning                                                      |
 |----------------------|--------------------------------------------------------------|
@@ -137,11 +190,16 @@ with the product's names filled in.
 
 | Variable                     | Default                             | Meaning |
 |------------------------------|-------------------------------------|---------|
-| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE` |
+| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE`. A channel (`:stable`) or a version tag; `:latest`, or no tag at all, is refused |
 | `AUTODEPLOY_INTERVAL`        | `300`                               | Seconds between checks (at least 30) |
 | `AUTODEPLOY_WINDOW`          | empty                               | UTC span for deploys, like `02:00-05:00` or `22:00-04:00`. Empty means any time |
-| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the new container gets to turn healthy |
-| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Backups kept, newest first |
+| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the migrations get, and then the new container to turn ready |
+| `AUTODEPLOY_MIGRATE_COMMAND` | `splent db:upgrade`                 | What the one-off migration container runs. `off` leaves the migrations to the new container's start |
+| `AUTODEPLOY_READY_PATH`      | `/readyz`                           | The readiness probe. `off` waits for Docker's healthcheck alone |
+| `AUTODEPLOY_WEB_PORT`        | `5000`                              | The port the web container answers on, inside the product network |
+| `AUTODEPLOY_BACKUP_COMMAND`  | `auto`                              | The off-host backup before a deploy. `auto` is `splent_feature_backup`'s when the product has it, `off` none, anything else the command to run in the web container |
+| `AUTODEPLOY_APP_USER`        | `splent`                            | The user the migration and the backup command run as |
+| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Rollback dumps kept, newest first |
 | `AUTODEPLOY_WEB_CONTAINER`   | `<product>_web_deploy`              | The container that is replaced |
 | `AUTODEPLOY_DB_CONTAINER`    | `<product>_db_deploy`               | The MariaDB container that is backed up |
 | `AUTODEPLOY_AGENT_CONTAINER` | `<product>_autodeploy`              | The agent's container name |
@@ -202,8 +260,10 @@ since.
 
 - The agent mounts `/var/run/docker.sock`. Whoever controls that socket is
   root on the host, so the agent is as sensitive as the host itself. It runs
-  only in production, from the image `product:deploy` builds out of this
-  feature's `docker/` directory, and does only what this README describes.
+  only in production, in a container of its own, from the image
+  `product:deploy` builds out of this feature's `docker/` directory, and does
+  only what this README describes. No container of the app mounts the
+  socket: not the web container, not the migration job it creates.
 - It publishes no port. Only containers on the product network reach the API,
   and every call needs the shared secret. The secret never appears in the
   status, the history or the logs.
@@ -221,7 +281,8 @@ One per deploy, in the agent's `autodeploy_data` volume under
 so each file says which code its schema belongs to. The newest
 `AUTODEPLOY_KEEP_BACKUPS` are kept. They live on the same host as the
 database, so they protect against a bad release, not against losing the
-server. Copy them elsewhere if you need that.
+server: that is what `splent_feature_backup`'s off-host backup, taken just
+before, is for. Without that feature, copy them elsewhere if you need it.
 
 ## Restoring a backup by hand
 
@@ -248,7 +309,10 @@ To let the agent try an image it refused, pause it, delete the image id from
   feature that ships its own container), or needs a new setting without a
   default in `.env.deploy`, has to go through `splent product:deploy`; pause
   the agent while you do it. Left to the agent such a release fails its
-  health check and is rolled back, which is safe but not what you want.
+  readiness check and is rolled back, which is safe but not what you want.
+- It takes no database lock (`GET_LOCK`) around the migration: one agent
+  replaces one web container, which is stopped while the job runs, so the
+  job is the only migrator. Several replicas would need that lock.
 - It does not update itself. A new version of this feature reaches the
   server with `splent product:deploy`, which rebuilds the agent's image.
 - It replaces one web container, not a set of replicas.

{splent_feature_autodeploy-0.2.0 → splent_feature_autodeploy-0.3.0}/README.md

@@ -1,12 +1,14 @@
 # splent_feature_autodeploy
 
 Release-gated continuous delivery for a SPLENT product. A product that
-selects this feature runs a small agent next to its web container in
-production. The agent follows a registry image channel, for example
-`git.splent.io/splent-io/<product>:stable`, and when the channel points to a
-new image it redeploys the product. The database is backed up first, the
-new version has to pass its health check, and an automatic rollback of both the container and the database when
-the new version does not come up.
+selects this feature runs a small agent in a container of its own next to
+its web container in production. The agent follows a registry image
+channel, for example `git.splent.io/splent-io/<product>:stable`, and when
+the channel points to a new image it redeploys the product on that image's
+digest. It backs up first (off the host too, when the product has
+`splent_feature_backup`), migrates once, waits for the new version's
+`/readyz`, and rolls the container back, and the database when the deploy
+changed it, when the new version is not ready.
 
 The channel only moves when someone runs `splent product:release`, so nothing
 reaches production that nobody released. The agent never builds anything and
@@ -18,41 +20,92 @@ Every `AUTODEPLOY_INTERVAL` seconds, and whenever an administrator asks,
 the agent goes through these steps.
 
 1. Pull the channel image. If its id is the one the web container runs,
-   there is nothing to do.
+   there is nothing to do. Otherwise the image is named from now on by its
+   digest (`<registry>/<org>/<product>@sha256:...`), which no push can
+   move: that is what the new container runs and what the history records.
+   An image the registry knows no digest of runs by its id, just as fixed.
 2. Skip an image that failed before, until the channel points elsewhere.
    Honour the pause flag and the update window (`AUTODEPLOY_WINDOW`). A
    check somebody asks for (Check now) deploys outside the window too.
-3. Back up the database. `mariadb-dump --single-transaction --routines
-   --triggers` runs as root inside the database container, with the
-   credentials that container already holds, and the gzipped dump is kept
-   only if `mariadb-dump` exited cleanly and the dump ends with its
-   `Dump completed` trailer. No complete backup, no deploy.
-4. Record what the web container runs (`splent version --json`).
-5. Recreate the web container on the new image with everything it was
-   configured with, the way Watchtower does it. That is its name, its
-   environment (the container's own values on top of the new image's), its
-   labels, mounts and binds, port bindings, restart policy, healthcheck, and
-   its networks with their aliases. Cmd, Entrypoint and WorkingDir follow the new image when the old
-   container only inherited them, and are kept when it chose them (the
-   compose file's `command:`). The old container is renamed
-   `<name>-previous` and stopped, not removed.
-6. Wait for the healthcheck, up to `AUTODEPLOY_HEALTH_TIMEOUT` seconds. The
-   entrypoint runs every feature's migrations before gunicorn starts, so this
-   is generous.
-7. Once it is healthy, remove the previous container and write the history
-   entry (from and to image, product version, the features whose version
-   changed, duration).
-8. If it turns unhealthy, exits, restarts or runs out of time, keep the new
-   container's last 80 log lines, remove it, drop and recreate the database and import the
-   backup (so tables the new migrations created do not linger), put the
-   previous container back under its name, wait for it to be healthy, and
-   refuse that image from then on. The history entry says `rolled_back` and
-   carries the log lines.
+3. Back up.
+   - When the product has `splent_feature_backup`, run its off-host backup
+     in the running web container, as the app's user:
+     `splent feature:backup run --tag pre-deploy-<short id of the image
+     being replaced>` (`AUTODEPLOY_BACKUP_COMMAND`). It is encrypted to keys
+     the server does not hold, and it is the copy that survives the host.
+   - Then the agent's own dump, the one a rollback restores:
+     `mariadb-dump --single-transaction --routines --triggers` runs as root
+     inside the database container, with the credentials that container
+     already holds, and the gzipped dump is kept only if `mariadb-dump`
+     exited cleanly and the dump ends with its `Dump completed` trailer.
+
+   No complete backup, no deploy.
+4. Record what the web container runs (`splent version --json`), and a
+   fingerprint of the schema: every table's columns and every migration
+   revision (`alembic_*`, `splent_migrations`).
+5. Create the new web container on the image's digest with everything the
+   old one was configured with, the way Watchtower does it. That is its
+   name, its environment (the container's own values on top of the new
+   image's), its labels, mounts and binds, port bindings, restart policy,
+   healthcheck, and its networks with their aliases. Cmd, Entrypoint and
+   WorkingDir follow the new image when the old container only inherited
+   them, and are kept when it chose them (the compose file's `command:`).
+   The old container is renamed `<name>-previous` and stopped, not removed.
+6. Migrate, once: a one-off container of the new image, `<name>-migrate`,
+   with the web container's environment, mounts and networks, runs
+   `AUTODEPLOY_MIGRATE_COMMAND` (`splent db:upgrade`) as the app's user
+   while neither version serves. It answers to no alias, so nginx never
+   finds it, publishes no port and is removed once it exits. The new
+   container's entrypoint upgrades again as it starts and finds nothing
+   left to do.
+7. Start the new container and wait, up to `AUTODEPLOY_HEALTH_TIMEOUT`
+   seconds, for `/readyz` to answer 200 (`AUTODEPLOY_READY_PATH`, asked by
+   the container's name on `AUTODEPLOY_WEB_PORT`), while Docker's
+   healthcheck does not say unhealthy. `/readyz` is the framework's
+   (1.22.0 and later): 200 when the database and the services the product
+   declares answer. An image without it answers 404, and the healthcheck
+   decides for it.
+8. Once it is ready, remove the previous container and write the history
+   entry (from and to image, the digest, product version, the features
+   whose version changed, the off-host backup, duration).
+9. If the migration fails, or the new container turns unhealthy, exits,
+   restarts or is not ready in time, keep its last 80 log lines (the
+   migration's, when that is what failed), remove it, put the database
+   back when the rule below says so, put the previous container back under
+   its name, wait for it to be healthy, and refuse that image from then on.
+   The history entry says `rolled_back`, carries the log lines and says
+   whether the database went back and why.
+
+### When the database goes back
+
+A rollback restores the agent's pre-deploy dump (dropping the database
+first, so tables the new migrations created do not linger) when:
+
+- the migration failed, or the agent died during it. MariaDB commits DDL as
+  it goes, so a failed migration can leave half of its changes behind
+  without recording them;
+- the migration finished, and the schema fingerprint differs from the one
+  taken before the deploy. The previous version cannot run on the new
+  schema, and a down-migration is not something to trust with production
+  data;
+- the fingerprint cannot be read, which counts as changed;
+- `AUTODEPLOY_MIGRATE_COMMAND=off`: the new container migrated as it
+  started, and the agent cannot tell a failed migration from anything
+  else.
+
+It does not when the deploy stopped before anything ran against the
+database (a failed backup, a container that could not be created), and when
+the migration finished without changing the schema: the previous version
+comes back on the same database, and nothing written while the new one
+tried to start is lost. The dump is taken right before the old version
+stops, so a restore loses only what the new version wrote during its
+attempt.
 
 Every step is written to `/data/state.json` before it is taken. An agent
 restarted in the middle of a deploy (a reboot, a redeploy of the agent
-itself) finishes it if the new container turns healthy, and rolls it back
-otherwise.
+itself) finishes it if the new container turns ready, and rolls it back
+otherwise; one restarted during the migration rolls back with the
+database.
 
 | State                | Meaning                                                      |
 |----------------------|--------------------------------------------------------------|
@@ -126,11 +179,16 @@ with the product's names filled in.
 
 | Variable                     | Default                             | Meaning |
 |------------------------------|-------------------------------------|---------|
-| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE` |
+| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE`. A channel (`:stable`) or a version tag; `:latest`, or no tag at all, is refused |
 | `AUTODEPLOY_INTERVAL`        | `300`                               | Seconds between checks (at least 30) |
 | `AUTODEPLOY_WINDOW`          | empty                               | UTC span for deploys, like `02:00-05:00` or `22:00-04:00`. Empty means any time |
-| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the new container gets to turn healthy |
-| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Backups kept, newest first |
+| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the migrations get, and then the new container to turn ready |
+| `AUTODEPLOY_MIGRATE_COMMAND` | `splent db:upgrade`                 | What the one-off migration container runs. `off` leaves the migrations to the new container's start |
+| `AUTODEPLOY_READY_PATH`      | `/readyz`                           | The readiness probe. `off` waits for Docker's healthcheck alone |
+| `AUTODEPLOY_WEB_PORT`        | `5000`                              | The port the web container answers on, inside the product network |
+| `AUTODEPLOY_BACKUP_COMMAND`  | `auto`                              | The off-host backup before a deploy. `auto` is `splent_feature_backup`'s when the product has it, `off` none, anything else the command to run in the web container |
+| `AUTODEPLOY_APP_USER`        | `splent`                            | The user the migration and the backup command run as |
+| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Rollback dumps kept, newest first |
 | `AUTODEPLOY_WEB_CONTAINER`   | `<product>_web_deploy`              | The container that is replaced |
 | `AUTODEPLOY_DB_CONTAINER`    | `<product>_db_deploy`               | The MariaDB container that is backed up |
 | `AUTODEPLOY_AGENT_CONTAINER` | `<product>_autodeploy`              | The agent's container name |
@@ -191,8 +249,10 @@ since.
 
 - The agent mounts `/var/run/docker.sock`. Whoever controls that socket is
   root on the host, so the agent is as sensitive as the host itself. It runs
-  only in production, from the image `product:deploy` builds out of this
-  feature's `docker/` directory, and does only what this README describes.
+  only in production, in a container of its own, from the image
+  `product:deploy` builds out of this feature's `docker/` directory, and does
+  only what this README describes. No container of the app mounts the
+  socket: not the web container, not the migration job it creates.
 - It publishes no port. Only containers on the product network reach the API,
   and every call needs the shared secret. The secret never appears in the
   status, the history or the logs.
@@ -210,7 +270,8 @@ One per deploy, in the agent's `autodeploy_data` volume under
 so each file says which code its schema belongs to. The newest
 `AUTODEPLOY_KEEP_BACKUPS` are kept. They live on the same host as the
 database, so they protect against a bad release, not against losing the
-server. Copy them elsewhere if you need that.
+server: that is what `splent_feature_backup`'s off-host backup, taken just
+before, is for. Without that feature, copy them elsewhere if you need it.
 
 ## Restoring a backup by hand
 
@@ -237,7 +298,10 @@ To let the agent try an image it refused, pause it, delete the image id from
   feature that ships its own container), or needs a new setting without a
   default in `.env.deploy`, has to go through `splent product:deploy`; pause
   the agent while you do it. Left to the agent such a release fails its
-  health check and is rolled back, which is safe but not what you want.
+  readiness check and is rolled back, which is safe but not what you want.
+- It takes no database lock (`GET_LOCK`) around the migration: one agent
+  replaces one web container, which is stopped while the job runs, so the
+  job is the only migrator. Several replicas would need that lock.
 - It does not update itself. A new version of this feature reaches the
   server with `splent product:deploy`, which rebuilds the agent's image.
 - It replaces one web container, not a set of replicas.

{splent_feature_autodeploy-0.2.0 → splent_feature_autodeploy-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "splent_feature_autodeploy"
-version = "0.2.0"
+version = "0.3.0"
 readme = "README.md"
 requires-python = ">=3.13"
 license = "LGPL-3.0-or-later"
@@ -95,7 +95,9 @@ services = []
 # The registry image to follow. Empty means SPLENT_IMAGE, the image the web
 # service runs, which is the usual choice: the server follows the channel it
 # was deployed from, git.splent.io/splent-io/<product>:stable for example,
-# and that tag only moves when someone runs splent product:release.
+# and that tag only moves when someone runs splent product:release. Never
+# :latest, which the agent refuses; the container it creates runs the
+# image's digest, whatever tag it followed.
 AUTODEPLOY_IMAGE = ""
 # Seconds between two looks at the registry. A pull of an image already
 # present costs one manifest request.
@@ -103,10 +105,30 @@ AUTODEPLOY_INTERVAL = 300
 # Daily UTC span in which a deploy may start, like 02:00-05:00. It may wrap
 # past midnight (22:00-04:00). Empty means any time.
 AUTODEPLOY_WINDOW = ""
-# Seconds the new web container gets to report healthy. Its entrypoint runs
-# every feature's migrations before gunicorn starts, so this is generous.
+# Seconds the migrations get, and then the new web container to answer its
+# readiness probe.
 AUTODEPLOY_HEALTH_TIMEOUT = 600
-# Database backups kept in the agent's volume, one per deploy, newest first.
+# What a one-off container of the new image runs, once, between stopping the
+# old version and starting the new one. off leaves the migrations to the new
+# container's start, and then a failed deploy always restores the database.
+AUTODEPLOY_MIGRATE_COMMAND = "splent db:upgrade"
+# The new version is ready when this answers 200, asked by the web
+# container's name on its AUTODEPLOY_WEB_PORT. off waits for Docker's
+# healthcheck alone. An image whose framework has no /readyz (before 1.22.0)
+# answers 404, and the healthcheck decides for it.
+AUTODEPLOY_READY_PATH = "/readyz"
+AUTODEPLOY_WEB_PORT = 5000
+# The off-host backup taken in the web container before anything changes.
+# auto runs splent_feature_backup's (splent feature:backup run --tag
+# pre-deploy-<image>) when the product has that feature; off runs none; any
+# other value is the command to run. It failing stops the deploy.
+AUTODEPLOY_BACKUP_COMMAND = "auto"
+# The user the migrations and the backup command run as: the one the
+# production image runs the app as, so what they write belongs to the app.
+AUTODEPLOY_APP_USER = "splent"
+# Database dumps kept in the agent's volume, one per deploy, newest first.
+# They are the rollback's, on this host; the off-host copy is the backup
+# feature's.
 AUTODEPLOY_KEEP_BACKUPS = 5
 # The web container the agent replaces, as the production compose names it.
 AUTODEPLOY_WEB_CONTAINER = "__PRODUCT___web"

{splent_feature_autodeploy-0.2.0 → splent_feature_autodeploy-0.3.0/src/splent_feature_autodeploy.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: splent_feature_autodeploy
-Version: 0.2.0
+Version: 0.3.0
 License-Expression: LGPL-3.0-or-later
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
@@ -12,12 +12,14 @@ Dynamic: license-file
 # splent_feature_autodeploy
 
 Release-gated continuous delivery for a SPLENT product. A product that
-selects this feature runs a small agent next to its web container in
-production. The agent follows a registry image channel, for example
-`git.splent.io/splent-io/<product>:stable`, and when the channel points to a
-new image it redeploys the product. The database is backed up first, the
-new version has to pass its health check, and an automatic rollback of both the container and the database when
-the new version does not come up.
+selects this feature runs a small agent in a container of its own next to
+its web container in production. The agent follows a registry image
+channel, for example `git.splent.io/splent-io/<product>:stable`, and when
+the channel points to a new image it redeploys the product on that image's
+digest. It backs up first (off the host too, when the product has
+`splent_feature_backup`), migrates once, waits for the new version's
+`/readyz`, and rolls the container back, and the database when the deploy
+changed it, when the new version is not ready.
 
 The channel only moves when someone runs `splent product:release`, so nothing
 reaches production that nobody released. The agent never builds anything and
@@ -29,41 +31,92 @@ Every `AUTODEPLOY_INTERVAL` seconds, and whenever an administrator asks,
 the agent goes through these steps.
 
 1. Pull the channel image. If its id is the one the web container runs,
-   there is nothing to do.
+   there is nothing to do. Otherwise the image is named from now on by its
+   digest (`<registry>/<org>/<product>@sha256:...`), which no push can
+   move: that is what the new container runs and what the history records.
+   An image the registry knows no digest of runs by its id, just as fixed.
 2. Skip an image that failed before, until the channel points elsewhere.
    Honour the pause flag and the update window (`AUTODEPLOY_WINDOW`). A
    check somebody asks for (Check now) deploys outside the window too.
-3. Back up the database. `mariadb-dump --single-transaction --routines
-   --triggers` runs as root inside the database container, with the
-   credentials that container already holds, and the gzipped dump is kept
-   only if `mariadb-dump` exited cleanly and the dump ends with its
-   `Dump completed` trailer. No complete backup, no deploy.
-4. Record what the web container runs (`splent version --json`).
-5. Recreate the web container on the new image with everything it was
-   configured with, the way Watchtower does it. That is its name, its
-   environment (the container's own values on top of the new image's), its
-   labels, mounts and binds, port bindings, restart policy, healthcheck, and
-   its networks with their aliases. Cmd, Entrypoint and WorkingDir follow the new image when the old
-   container only inherited them, and are kept when it chose them (the
-   compose file's `command:`). The old container is renamed
-   `<name>-previous` and stopped, not removed.
-6. Wait for the healthcheck, up to `AUTODEPLOY_HEALTH_TIMEOUT` seconds. The
-   entrypoint runs every feature's migrations before gunicorn starts, so this
-   is generous.
-7. Once it is healthy, remove the previous container and write the history
-   entry (from and to image, product version, the features whose version
-   changed, duration).
-8. If it turns unhealthy, exits, restarts or runs out of time, keep the new
-   container's last 80 log lines, remove it, drop and recreate the database and import the
-   backup (so tables the new migrations created do not linger), put the
-   previous container back under its name, wait for it to be healthy, and
-   refuse that image from then on. The history entry says `rolled_back` and
-   carries the log lines.
+3. Back up.
+   - When the product has `splent_feature_backup`, run its off-host backup
+     in the running web container, as the app's user:
+     `splent feature:backup run --tag pre-deploy-<short id of the image
+     being replaced>` (`AUTODEPLOY_BACKUP_COMMAND`). It is encrypted to keys
+     the server does not hold, and it is the copy that survives the host.
+   - Then the agent's own dump, the one a rollback restores:
+     `mariadb-dump --single-transaction --routines --triggers` runs as root
+     inside the database container, with the credentials that container
+     already holds, and the gzipped dump is kept only if `mariadb-dump`
+     exited cleanly and the dump ends with its `Dump completed` trailer.
+
+   No complete backup, no deploy.
+4. Record what the web container runs (`splent version --json`), and a
+   fingerprint of the schema: every table's columns and every migration
+   revision (`alembic_*`, `splent_migrations`).
+5. Create the new web container on the image's digest with everything the
+   old one was configured with, the way Watchtower does it. That is its
+   name, its environment (the container's own values on top of the new
+   image's), its labels, mounts and binds, port bindings, restart policy,
+   healthcheck, and its networks with their aliases. Cmd, Entrypoint and
+   WorkingDir follow the new image when the old container only inherited
+   them, and are kept when it chose them (the compose file's `command:`).
+   The old container is renamed `<name>-previous` and stopped, not removed.
+6. Migrate, once: a one-off container of the new image, `<name>-migrate`,
+   with the web container's environment, mounts and networks, runs
+   `AUTODEPLOY_MIGRATE_COMMAND` (`splent db:upgrade`) as the app's user
+   while neither version serves. It answers to no alias, so nginx never
+   finds it, publishes no port and is removed once it exits. The new
+   container's entrypoint upgrades again as it starts and finds nothing
+   left to do.
+7. Start the new container and wait, up to `AUTODEPLOY_HEALTH_TIMEOUT`
+   seconds, for `/readyz` to answer 200 (`AUTODEPLOY_READY_PATH`, asked by
+   the container's name on `AUTODEPLOY_WEB_PORT`), while Docker's
+   healthcheck does not say unhealthy. `/readyz` is the framework's
+   (1.22.0 and later): 200 when the database and the services the product
+   declares answer. An image without it answers 404, and the healthcheck
+   decides for it.
+8. Once it is ready, remove the previous container and write the history
+   entry (from and to image, the digest, product version, the features
+   whose version changed, the off-host backup, duration).
+9. If the migration fails, or the new container turns unhealthy, exits,
+   restarts or is not ready in time, keep its last 80 log lines (the
+   migration's, when that is what failed), remove it, put the database
+   back when the rule below says so, put the previous container back under
+   its name, wait for it to be healthy, and refuse that image from then on.
+   The history entry says `rolled_back`, carries the log lines and says
+   whether the database went back and why.
+
+### When the database goes back
+
+A rollback restores the agent's pre-deploy dump (dropping the database
+first, so tables the new migrations created do not linger) when:
+
+- the migration failed, or the agent died during it. MariaDB commits DDL as
+  it goes, so a failed migration can leave half of its changes behind
+  without recording them;
+- the migration finished, and the schema fingerprint differs from the one
+  taken before the deploy. The previous version cannot run on the new
+  schema, and a down-migration is not something to trust with production
+  data;
+- the fingerprint cannot be read, which counts as changed;
+- `AUTODEPLOY_MIGRATE_COMMAND=off`: the new container migrated as it
+  started, and the agent cannot tell a failed migration from anything
+  else.
+
+It does not when the deploy stopped before anything ran against the
+database (a failed backup, a container that could not be created), and when
+the migration finished without changing the schema: the previous version
+comes back on the same database, and nothing written while the new one
+tried to start is lost. The dump is taken right before the old version
+stops, so a restore loses only what the new version wrote during its
+attempt.
 
 Every step is written to `/data/state.json` before it is taken. An agent
 restarted in the middle of a deploy (a reboot, a redeploy of the agent
-itself) finishes it if the new container turns healthy, and rolls it back
-otherwise.
+itself) finishes it if the new container turns ready, and rolls it back
+otherwise; one restarted during the migration rolls back with the
+database.
 
 | State                | Meaning                                                      |
 |----------------------|--------------------------------------------------------------|
@@ -137,11 +190,16 @@ with the product's names filled in.
 
 | Variable                     | Default                             | Meaning |
 |------------------------------|-------------------------------------|---------|
-| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE` |
+| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE`. A channel (`:stable`) or a version tag; `:latest`, or no tag at all, is refused |
 | `AUTODEPLOY_INTERVAL`        | `300`                               | Seconds between checks (at least 30) |
 | `AUTODEPLOY_WINDOW`          | empty                               | UTC span for deploys, like `02:00-05:00` or `22:00-04:00`. Empty means any time |
-| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the new container gets to turn healthy |
-| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Backups kept, newest first |
+| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the migrations get, and then the new container to turn ready |
+| `AUTODEPLOY_MIGRATE_COMMAND` | `splent db:upgrade`                 | What the one-off migration container runs. `off` leaves the migrations to the new container's start |
+| `AUTODEPLOY_READY_PATH`      | `/readyz`                           | The readiness probe. `off` waits for Docker's healthcheck alone |
+| `AUTODEPLOY_WEB_PORT`        | `5000`                              | The port the web container answers on, inside the product network |
+| `AUTODEPLOY_BACKUP_COMMAND`  | `auto`                              | The off-host backup before a deploy. `auto` is `splent_feature_backup`'s when the product has it, `off` none, anything else the command to run in the web container |
+| `AUTODEPLOY_APP_USER`        | `splent`                            | The user the migration and the backup command run as |
+| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Rollback dumps kept, newest first |
 | `AUTODEPLOY_WEB_CONTAINER`   | `<product>_web_deploy`              | The container that is replaced |
 | `AUTODEPLOY_DB_CONTAINER`    | `<product>_db_deploy`               | The MariaDB container that is backed up |
 | `AUTODEPLOY_AGENT_CONTAINER` | `<product>_autodeploy`              | The agent's container name |
@@ -202,8 +260,10 @@ since.
 
 - The agent mounts `/var/run/docker.sock`. Whoever controls that socket is
   root on the host, so the agent is as sensitive as the host itself. It runs
-  only in production, from the image `product:deploy` builds out of this
-  feature's `docker/` directory, and does only what this README describes.
+  only in production, in a container of its own, from the image
+  `product:deploy` builds out of this feature's `docker/` directory, and does
+  only what this README describes. No container of the app mounts the
+  socket: not the web container, not the migration job it creates.
 - It publishes no port. Only containers on the product network reach the API,
   and every call needs the shared secret. The secret never appears in the
   status, the history or the logs.
@@ -221,7 +281,8 @@ One per deploy, in the agent's `autodeploy_data` volume under
 so each file says which code its schema belongs to. The newest
 `AUTODEPLOY_KEEP_BACKUPS` are kept. They live on the same host as the
 database, so they protect against a bad release, not against losing the
-server. Copy them elsewhere if you need that.
+server: that is what `splent_feature_backup`'s off-host backup, taken just
+before, is for. Without that feature, copy them elsewhere if you need it.
 
 ## Restoring a backup by hand
 
@@ -248,7 +309,10 @@ To let the agent try an image it refused, pause it, delete the image id from
   feature that ships its own container), or needs a new setting without a
   default in `.env.deploy`, has to go through `splent product:deploy`; pause
   the agent while you do it. Left to the agent such a release fails its
-  health check and is rolled back, which is safe but not what you want.
+  readiness check and is rolled back, which is safe but not what you want.
+- It takes no database lock (`GET_LOCK`) around the migration: one agent
+  replaces one web container, which is stopped while the job runs, so the
+  job is the only migrator. Several replicas would need that lock.
 - It does not update itself. A new version of this feature reaches the
   server with `splent product:deploy`, which rebuilds the agent's image.
 - It replaces one web container, not a set of replicas.

{splent_feature_autodeploy-0.2.0 → splent_feature_autodeploy-0.3.0}/src/splent_io/splent_feature_autodeploy/commands.py

@@ -129,8 +129,15 @@ def history(limit, as_json):
                 f"      {change.get('name')} {change.get('from') or '(new)'} -> "
                 f"{change.get('to') or '(removed)'}"
             )
+        if entry.get("to_ref"):
+            click.echo(f"      image {entry['to_ref']}")
+        if entry.get("offsite_backup"):
+            click.echo(f"      off-host backup {entry['offsite_backup']}")
         if entry.get("reason"):
             click.echo(f"      reason {entry['reason']}")
+        if "database_restored" in entry:
+            state = "restored" if entry["database_restored"] else "kept"
+            click.echo(f"      database {state}: {entry.get('database_decision')}")
         for problem in entry.get("problems") or []:
             click.secho(f"      problem {problem}", fg="red")
     click.echo()