splent-feature-autodeploy 0.1.0__py3-none-any.whl

splent_feature_autodeploy-0.1.0.dist-info/METADATA

@@ -0,0 +1,265 @@
+Metadata-Version: 2.4
+Name: splent_feature_autodeploy
+Version: 0.1.0
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+
+# splent_feature_autodeploy
+
+Release-gated continuous delivery for a SPLENT product. A product that
+selects this feature runs a small agent next to its web container in
+production. The agent follows a registry image channel, for example
+`git.splent.io/splent-io/<product>:stable`, and when the channel points to a
+new image it redeploys the product. The database is backed up first, the
+new version has to pass its health check, and an automatic rollback of both the container and the database when
+the new version does not come up.
+
+The channel only moves when someone runs `splent product:release`, so nothing
+reaches production that nobody released. The agent never builds anything and
+never follows a branch.
+
+## How a check goes
+
+Every `AUTODEPLOY_INTERVAL` seconds, and whenever an administrator asks,
+the agent goes through these steps.
+
+1. Pull the channel image. If its id is the one the web container runs,
+   there is nothing to do.
+2. Skip an image that failed before, until the channel points elsewhere.
+   Honour the pause flag and the update window (`AUTODEPLOY_WINDOW`). A
+   check somebody asks for (Check now) deploys outside the window too.
+3. Back up the database. `mariadb-dump --single-transaction --routines
+   --triggers` runs as root inside the database container, with the
+   credentials that container already holds, and the gzipped dump is kept
+   only if `mariadb-dump` exited cleanly and the dump ends with its
+   `Dump completed` trailer. No complete backup, no deploy.
+4. Record what the web container runs (`splent version --json`).
+5. Recreate the web container on the new image with everything it was
+   configured with, the way Watchtower does it. That is its name, its
+   environment (the container's own values on top of the new image's), its
+   labels, mounts and binds, port bindings, restart policy, healthcheck, and
+   its networks with their aliases. Cmd, Entrypoint and WorkingDir follow the new image when the old
+   container only inherited them, and are kept when it chose them (the
+   compose file's `command:`). The old container is renamed
+   `<name>-previous` and stopped, not removed.
+6. Wait for the healthcheck, up to `AUTODEPLOY_HEALTH_TIMEOUT` seconds. The
+   entrypoint runs every feature's migrations before gunicorn starts, so this
+   is generous.
+7. Once it is healthy, remove the previous container and write the history
+   entry (from and to image, product version, the features whose version
+   changed, duration).
+8. If it turns unhealthy, exits, restarts or runs out of time, keep the new
+   container's last 80 log lines, remove it, drop and recreate the database and import the
+   backup (so tables the new migrations created do not linger), put the
+   previous container back under its name, wait for it to be healthy, and
+   refuse that image from then on. The history entry says `rolled_back` and
+   carries the log lines.
+
+Every step is written to `/data/state.json` before it is taken. An agent
+restarted in the middle of a deploy (a reboot, a redeploy of the agent
+itself) finishes it if the new container turns healthy, and rolls it back
+otherwise.
+
+| State                | Meaning                                                      |
+|----------------------|--------------------------------------------------------------|
+| `idle`               | The web container runs the image the channel points to       |
+| `waiting_for_window` | A new image is ready and waits for the update window         |
+| `deploying`          | Backup, recreate or health wait in progress                  |
+| `rolling_back`       | The new image failed and the previous version is coming back |
+| `paused`             | An administrator paused automatic deploys                    |
+| `held`               | The only image available failed before; waiting for the next release |
+| `error`              | Something needs a person, the detail says what               |
+| `not_configured`     | A required setting is missing, the detail lists it           |
+
+## Enable it in a product
+
+1. Add the feature to the product line as an optional feature, from a
+   detached workspace.
+
+   ```bash
+   splent spl:add-feature <spl> splent_feature_autodeploy
+   ```
+
+   In the UVL model it reads as below, with the constraint that keeps the
+   admin panel it lives in.
+
+   ```
+   optional
+       autodeploy {org 'splent-io', package 'splent_feature_autodeploy'}
+   constraints
+       autodeploy => admin
+   ```
+
+2. Attach it to the product and release the product, so the channel exists.
+
+   ```bash
+   splent product:select <product>
+   splent feature:attach splent-io/splent_feature_autodeploy v0.1.0
+   splent product:release
+   ```
+
+   A product release pushes `<registry>/<org>/<product>:<version>` and moves
+   `:stable` to it.
+
+3. On the server, set `SPLENT_IMAGE` in `docker/.env.deploy` to the channel
+   the server follows, then build and deploy. The web service of the
+   product's `docker-compose.prod.yml` must take its image from that
+   variable, as in `image: ${SPLENT_IMAGE:-<product>:latest}`.
+
+   ```bash
+   SPLENT_IMAGE=git.splent.io/splent-io/<product>:stable
+   ```
+
+   ```bash
+   splent product:build --no-image
+   splent product:deploy
+   ```
+
+   `product:build` adds the agent's service to `docker-compose.deploy.yml`
+   and copies `docker/` to `docker/features/splent_feature_autodeploy/`.
+   `product:deploy` offers a random value for `AUTODEPLOY_SECRET`, builds the
+   agent's image and starts it.
+
+4. For a private registry set `AUTODEPLOY_REGISTRY_USER` and
+   `AUTODEPLOY_REGISTRY_TOKEN` (a token that can read packages).
+
+From then on, `splent product:release` is the whole deploy.
+
+## Configuration
+
+Declared in `[tool.splent.config]`, so they appear in `.env.deploy.example`
+with the product's names filled in.
+
+| Variable                     | Default                             | Meaning |
+|------------------------------|-------------------------------------|---------|
+| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE` |
+| `AUTODEPLOY_INTERVAL`        | `300`                               | Seconds between checks (at least 30) |
+| `AUTODEPLOY_WINDOW`          | empty                               | UTC span for deploys, like `02:00-05:00` or `22:00-04:00`. Empty means any time |
+| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the new container gets to turn healthy |
+| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Backups kept, newest first |
+| `AUTODEPLOY_WEB_CONTAINER`   | `<product>_web_deploy`              | The container that is replaced |
+| `AUTODEPLOY_DB_CONTAINER`    | `<product>_db_deploy`               | The MariaDB container that is backed up |
+| `AUTODEPLOY_AGENT_CONTAINER` | `<product>_autodeploy`              | The agent's container name |
+| `AUTODEPLOY_URL`             | `http://<product>_autodeploy:8080`  | Where the web side finds the agent |
+| `AUTODEPLOY_REGISTRY_USER`   | empty                               | Registry account, for a private channel |
+| `AUTODEPLOY_REGISTRY_TOKEN`  | empty                               | Registry token, for a private channel |
+| `AUTODEPLOY_SECRET`          | `<SET>`                             | Shared by the web container and the agent |
+
+## The Deployments page
+
+`/admin/deployments`, in the admin panel's sidebar, for administrators only
+(the admin feature's `ADMIN_ROLES`). It shows the channel, the running image
+and product version, what the agent is doing and why, the last check, the
+update window, whether updates are paused, and the latest deploys with the
+features each one moved. A rolled back deploy shows its log lines in a
+collapsible block, and a banner says so until the next deploy. Check now,
+Pause and Resume are CSRF protected forms.
+
+When the agent is missing (development), stopped or refuses the secret the
+page explains what is going on. The site itself is never affected.
+
+## Commands
+
+```bash
+splent feature:autodeploy status [--json]
+splent feature:autodeploy history [--limit N] [--json]
+splent feature:autodeploy check
+splent feature:autodeploy pause
+splent feature:autodeploy resume
+```
+
+They talk to the agent's API, so they need the product network. On a
+production server run them in the web container.
+
+```bash
+docker exec <product>_web_deploy splent feature:autodeploy status
+```
+
+## The agent's API
+
+Port 8080 on the product network, never published on the host. Every request
+carries `Authorization: Bearer <AUTODEPLOY_SECRET>`, compared in constant
+time; anything else is a 401, whatever the path.
+
+| Method | Path               | Does |
+|--------|--------------------|------|
+| GET    | `/status`          | State, configuration, running versions, backups |
+| GET    | `/history?limit=N` | The newest N deploys, newest first (1 to 200) |
+| POST   | `/check`           | Look for a new image now |
+| POST   | `/pause`           | Stop deploying; a deploy in progress finishes |
+| POST   | `/resume`          | Deploy again, and check right away |
+
+There is no rollback endpoint on purpose. A rollback restores the backup taken
+before the deploy, so run later by hand it would throw away everything written
+since.
+
+## Security
+
+- The agent mounts `/var/run/docker.sock`. Whoever controls that socket is
+  root on the host, so the agent is as sensitive as the host itself. It runs
+  only in production, from the image `product:deploy` builds out of this
+  feature's `docker/` directory, and does only what this README describes.
+- It publishes no port. Only containers on the product network reach the API,
+  and every call needs the shared secret. The secret never appears in the
+  status, the history or the logs.
+- The web side calls the agent without any proxy, so an `HTTP_PROXY` in the
+  environment never sees the secret.
+- The database credentials stay in the database container. The dump and the
+  restore read them from that container's environment and pass them through
+  `MYSQL_PWD`, so they are not on any command line and are never printed.
+- Registry credentials are only sent to the registry, with the pull.
+
+## Backups
+
+One per deploy, in the agent's `autodeploy_data` volume under
+`/data/backups/`, named `<UTC time>-<short id of the image being replaced>.sql.gz`,
+so each file says which code its schema belongs to. The newest
+`AUTODEPLOY_KEEP_BACKUPS` are kept. They live on the same host as the
+database, so they protect against a bad release, not against losing the
+server. Copy them elsewhere if you need that.
+
+## Restoring a backup by hand
+
+Restoring a backup loses everything written after it was taken. Pause the
+agent first, so it does not deploy in the middle.
+
+```bash
+docker exec <product>_web_deploy splent feature:autodeploy pause
+docker cp <product>_autodeploy:/data/backups/<file>.sql.gz .
+docker stop <product>_web_deploy
+docker exec <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot -e "DROP DATABASE IF EXISTS \`$MARIADB_DATABASE\`; CREATE DATABASE \`$MARIADB_DATABASE\`"'
+gunzip -c <file>.sql.gz | docker exec -i <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot "$MARIADB_DATABASE"'
+```
+
+Then run a web container on the image named in the file name, so code and
+schema match, and resume the agent.
+
+To let the agent try an image it refused, pause it, delete the image id from
+`bad_images` in `/data/state.json` in the agent's volume, and resume.
+
+## What the agent does not do
+
+- It replaces the web container only. A release that adds a service (a
+  feature that ships its own container), or needs a new setting without a
+  default in `.env.deploy`, has to go through `splent product:deploy`; pause
+  the agent while you do it. Left to the agent such a release fails its
+  health check and is rolled back, which is safe but not what you want.
+- It does not update itself. A new version of this feature reaches the
+  server with `splent product:deploy`, which rebuilds the agent's image.
+- It replaces one web container, not a set of replicas.
+
+## Develop
+
+The agent is plain Python 3.13 with docker-py, in `docker/autodeploy_agent/`.
+It gets the Docker client handed in, so its tests run against a fake Engine.
+
+```bash
+splent feature:test splent_feature_autodeploy
+```
+
+From a bare checkout, with a Python that has the framework, Flask-Babel,
+Flask-Login and Flask-WTF installed, the same tests run without a product.
+
+```bash
+python -m pytest src/splent_io/splent_feature_autodeploy/tests
+```

splent_feature_autodeploy-0.1.0.dist-info/RECORD

@@ -0,0 +1,17 @@
+splent_io/splent_feature_autodeploy/__init__.py,sha256=-vCEeSoOSg1e5mxfP5RS6fJwFKa1zcT_laiyB-JvvDE,400
+splent_io/splent_feature_autodeploy/commands.py,sha256=z4sSiPRHDJFrZl6Yon5xHVJzx6u16_LcdWdfYVvN6P8,5395
+splent_io/splent_feature_autodeploy/config.py,sha256=VdFAiYX9N0F-FvYGbqTPpMEpwcP5vtBvD6baLoAMy6o,1420
+splent_io/splent_feature_autodeploy/forms.py,sha256=9yxlORGCXq_YsL_RyfTLC3gZKZGwr-IlE0g21bU6VKg,225
+splent_io/splent_feature_autodeploy/hooks.py,sha256=-d0GtGy_rCKqi1Y0z5fynKWt5iCvapk7HbSpSYvvJjU,1069
+splent_io/splent_feature_autodeploy/presenters.py,sha256=udGUF1A8aerGXd-8otIFh09RiFvZ9uus88NIDMkdD4k,6850
+splent_io/splent_feature_autodeploy/routes.py,sha256=V63KTraWrCZBee_e_AKVwLJhs57_fhpInM1jBmW-spo,4342
+splent_io/splent_feature_autodeploy/services.py,sha256=KxlGeVS4Nx6TqJSMyFUrF77PWBhM4P3r440NCIlbAdk,6244
+splent_io/splent_feature_autodeploy/templates/autodeploy/index.html,sha256=Jsu9tYhaj4DIcpx3tg6jnzbby5aUyXj2YIc_FG_e7_Y,12764
+splent_io/splent_feature_autodeploy/templates/hooks/sidebar_deployments.html,sha256=EhAAem2QBgeZJXTfJF3hPu2o02LKzTpwSeJBvDts3Q0,323
+splent_io/splent_feature_autodeploy/translations/messages.pot,sha256=tBXGk7NT7t45xeCBuOwHXv5q3lw700hRvydyfTh5ClA,13537
+splent_io/splent_feature_autodeploy/translations/es/LC_MESSAGES/messages.mo,sha256=Pl9FFXR6dG0LwPmf20Gw4VTp7N_nSiALPaHXL-V5acg,11033
+splent_io/splent_feature_autodeploy/translations/es/LC_MESSAGES/messages.po,sha256=QW4oWL7UBi70FLTGLPHR751PppLO6S2KBProgwuMrmU,18713
+splent_feature_autodeploy-0.1.0.dist-info/METADATA,sha256=Iw7j4-MyAy74m-vyS2N55SLuAmkAeHUbN1j3lXPnHqI,12642
+splent_feature_autodeploy-0.1.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+splent_feature_autodeploy-0.1.0.dist-info/top_level.txt,sha256=nksrfdr_edPhGPUsCqz08kyNRLBkNc2sInZYzmbZ4TA,10
+splent_feature_autodeploy-0.1.0.dist-info/RECORD,,

splent_feature_autodeploy-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

splent_feature_autodeploy-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+splent_io

splent_io/splent_feature_autodeploy/__init__.py

@@ -0,0 +1,14 @@
+from splent_framework.blueprints.base_blueprint import create_blueprint
+from splent_framework.services.service_locator import register_service
+
+from splent_io.splent_feature_autodeploy.services import AutodeployService
+
+autodeploy_bp = create_blueprint(__name__)
+
+
+def init_feature(app):
+    register_service(app, "AutodeployService", AutodeployService)
+
+
+def inject_context_vars(app):
+    return {}

splent_io/splent_feature_autodeploy/commands.py

@@ -0,0 +1,163 @@
+"""CLI commands contributed by splent_feature_autodeploy.
+
+Exposed under the ``feature:autodeploy`` group::
+
+    splent feature:autodeploy status
+    splent feature:autodeploy history --limit 5
+    splent feature:autodeploy check
+    splent feature:autodeploy pause
+    splent feature:autodeploy resume
+
+They talk to the agent's API at AUTODEPLOY_URL with AUTODEPLOY_SECRET, so
+they work wherever the product network is reachable. On a production server
+that is the web container itself::
+
+    docker exec <product>_web_deploy splent feature:autodeploy status
+"""
+
+import json
+
+import click
+
+from splent_io.splent_feature_autodeploy.services import (
+    DEFAULT_HISTORY_LIMIT,
+    AgentError,
+    AutodeployService,
+)
+
+HINTS = {
+    "not_configured": "Set AUTODEPLOY_URL and AUTODEPLOY_SECRET (product:deploy does both in production).",
+    "unreachable": "Is the agent container running, and is this command on the product network?",
+    "refused": "AUTODEPLOY_SECRET differs between here and the agent.",
+    "bad_response": "The agent and this feature may be of different versions.",
+}
+
+
+def _service() -> AutodeployService:
+    return AutodeployService()
+
+
+def _call(action: str, *args):
+    try:
+        return getattr(_service(), action)(*args)
+    except AgentError as exc:
+        hint = HINTS.get(exc.reason, "")
+        raise click.ClickException(f"{exc}. {hint}".strip())
+
+
+def _short(image_id) -> str:
+    return str(image_id or "").split(":", 1)[-1][:12] or "-"
+
+
+def _row(label: str, value) -> None:
+    click.echo(f"  {label:<18} {value if value not in (None, '') else '-'}")
+
+
+def _print_status(data: dict) -> None:
+    versions = data.get("running_versions") or {}
+    last = data.get("last_deploy") or {}
+    click.echo()
+    _row("state", data.get("status"))
+    if data.get("detail"):
+        _row("detail", data.get("detail"))
+    for problem in data.get("problems") or []:
+        _row("missing", problem)
+    _row("channel", data.get("image"))
+    _row("running image", _short(data.get("running_image")))
+    _row("product version", versions.get("product_version"))
+    if data.get("pending_image"):
+        _row("newest release", _short(data.get("pending_image")))
+    _row("last check", data.get("last_check"))
+    _row("window", f"{data['window']} UTC" if data.get("window") else "any time")
+    _row("paused", "yes" if data.get("paused") else "no")
+    if last:
+        _row(
+            "last deploy",
+            f"{last.get('result')} at {last.get('finished_at')} "
+            f"({_short(last.get('from_image'))} -> {_short(last.get('to_image'))})",
+        )
+    _row("refused images", len(data.get("bad_images") or []))
+    click.echo()
+
+
+@click.command("status")
+@click.option(
+    "--json", "as_json", is_flag=True, help="Print the agent's answer as JSON."
+)
+def status(as_json):
+    """What the deployment agent is doing and what the product runs."""
+    data = _call("status")
+    if as_json:
+        click.echo(json.dumps(data, indent=2, sort_keys=True))
+        return
+    _print_status(data)
+
+
+@click.command("history")
+@click.option(
+    "--limit",
+    default=DEFAULT_HISTORY_LIMIT,
+    show_default=True,
+    type=click.IntRange(1, 200),
+    help="How many deploys to show, newest first.",
+)
+@click.option("--json", "as_json", is_flag=True, help="Print the entries as JSON.")
+def history(limit, as_json):
+    """The latest deploys, newest first."""
+    entries = _call("history", limit)
+    if as_json:
+        click.echo(json.dumps(entries, indent=2, sort_keys=True))
+        return
+    if not entries:
+        click.echo("  No deploys yet.")
+        return
+    click.echo()
+    for entry in entries:
+        result = entry.get("result") or "?"
+        colour = {"success": "green", "rolled_back": "yellow"}.get(result, "red")
+        click.echo(
+            f"  {entry.get('finished_at') or entry.get('started_at')}  "
+            + click.style(f"{result:<15}", fg=colour)
+            + f" {_short(entry.get('from_image'))} -> {_short(entry.get('to_image'))}"
+            + f"  {entry.get('duration_seconds', '?')}s"
+        )
+        old, new = entry.get("product_version_from"), entry.get("product_version_to")
+        if old or new:
+            click.echo(f"      product {old or '?'} -> {new or '?'}")
+        for change in entry.get("features_changed") or []:
+            click.echo(
+                f"      {change.get('name')} {change.get('from') or '(new)'} -> "
+                f"{change.get('to') or '(removed)'}"
+            )
+        if entry.get("reason"):
+            click.echo(f"      reason {entry['reason']}")
+        for problem in entry.get("problems") or []:
+            click.secho(f"      problem {problem}", fg="red")
+    click.echo()
+
+
+@click.command("check")
+def check():
+    """Ask the agent to look for a new release now."""
+    _call("check")
+    click.secho(
+        "  The agent is checking now. It still deploys only inside its update window.",
+        fg="green",
+    )
+
+
+@click.command("pause")
+def pause():
+    """Stop automatic deploys until resumed. A deploy in progress finishes."""
+    _call("pause")
+    click.secho("  Automatic deploys are paused.", fg="green")
+
+
+@click.command("resume")
+def resume():
+    """Turn automatic deploys back on, and check right away."""
+    _call("resume")
+    click.secho("  Automatic deploys are back on.", fg="green")
+
+
+cli_commands = [status, history, check, pause, resume]

splent_io/splent_feature_autodeploy/config.py

@@ -0,0 +1,33 @@
+"""
+autodeploy feature configuration.
+
+Injects environment variables into Flask app.config. Only what the web side
+needs: where the agent listens and the secret it expects, plus the image and
+window to show while the agent cannot be reached. The rest of the AUTODEPLOY_*
+settings are read by the agent container itself (docker/autodeploy_agent).
+
+Every value has a working default, so the feature loads without a .env entry
+and the Deployments page explains what is missing.
+
+To regenerate from source code: splent feature:inject-config splent_feature_autodeploy
+"""
+
+import os
+
+
+def inject_config(app):
+    app.config.update(
+        {
+            # Base URL of the agent's API on the product network.
+            "AUTODEPLOY_URL": os.getenv("AUTODEPLOY_URL", ""),
+            # Shared with the agent; "<SET>" until product:deploy fills it in.
+            "AUTODEPLOY_SECRET": os.getenv("AUTODEPLOY_SECRET", ""),
+            # The channel the agent follows, shown while it is unreachable.
+            # Empty means the agent falls back to SPLENT_IMAGE.
+            "AUTODEPLOY_IMAGE": os.getenv("AUTODEPLOY_IMAGE", ""),
+            # Daily UTC span in which the agent may deploy, empty for any time.
+            "AUTODEPLOY_WINDOW": os.getenv("AUTODEPLOY_WINDOW", ""),
+            # Seconds between two checks of the registry.
+            "AUTODEPLOY_INTERVAL": os.getenv("AUTODEPLOY_INTERVAL", "300"),
+        }
+    )

splent_io/splent_feature_autodeploy/forms.py

@@ -0,0 +1,6 @@
+from flask_wtf import FlaskForm
+
+
+class DeploymentActionForm(FlaskForm):
+    """Nothing but the CSRF token, so Check now, Pause and Resume cannot be
+    triggered from another site. One instance renders all three buttons."""

splent_io/splent_feature_autodeploy/hooks.py

@@ -0,0 +1,29 @@
+"""Template hooks for splent_feature_autodeploy.
+
+One entry in the panel's sidebar, for the accounts that may open the page
+(the admin feature's ADMIN_ROLES), so nobody is shown a door that answers
+403. The sidebar never calls the agent: it renders on every panel page, and
+a slow or missing agent must not slow the panel down.
+
+The admin feature offers no dashboard card slot yet, so a rolled back deploy
+is reported on the Deployments page only.
+"""
+
+from flask import render_template
+from flask_login import current_user
+
+from splent_framework.hooks.template_hooks import register_template_hook
+from splent_io.splent_feature_admin.services import ADMIN_ROLES
+
+
+def deployments_sidebar_item():
+    if not getattr(current_user, "is_authenticated", False):
+        return ""
+    if not getattr(current_user, "active", False):
+        return ""
+    if getattr(current_user, "role", None) not in ADMIN_ROLES:
+        return ""
+    return render_template("hooks/sidebar_deployments.html")
+
+
+register_template_hook("layout.authenticated_sidebar", deployments_sidebar_item)