@lamalibre/create-lamaste 4.0.0

package/vendor/docs/04-api-reference/users.md

@@ -0,0 +1,333 @@
+# Users API
+
+> Manage Authelia users who authenticate to tunneled applications with TOTP two-factor authentication.
+
+## In Plain English
+
+When someone visits one of your tunneled apps (like `app.example.com`), they are prompted to log in through Authelia with a username, password, and a TOTP code from an authenticator app. The users API lets you create those login accounts, update their details, reset their TOTP secrets, and delete accounts you no longer need.
+
+These are not admin panel users — the panel itself is protected by mTLS client certificates with no login screen. These users are for the apps you expose through tunnels.
+
+## Authentication
+
+All user endpoints require a valid mTLS client certificate and a completed onboarding. See the [API Overview](./overview.md) for details.
+
+If onboarding is not complete, all endpoints return `503 Service Unavailable`.
+
+## Important Behavior
+
+- **Password hashing:** Passwords are hashed with bcrypt (not argon2id — argon2id uses ~93MB per hash and causes OOM kills on a 512MB droplet).
+- **Authelia restart:** After every user change that modifies `users.yml` (create, update, delete), Authelia is restarted via `systemctl restart authelia` to pick up the changes. If the restart fails, the operation still succeeds — a warning is logged but no error is returned to the client. Note: TOTP reset does **not** restart Authelia — it writes directly to Authelia's SQLite database via the `authelia storage user totp generate` CLI command, which takes effect immediately.
+- **Atomic writes:** The `users.yml` file is written atomically (write to temp file, then rename) to prevent Authelia from reading a partially written file.
+- **Last user protection:** You cannot delete the last remaining user. At least one user must always exist.
+
+## Endpoints
+
+### `GET /api/users`
+
+Returns all Authelia users, sorted alphabetically by username. Sensitive fields (password hash, TOTP secret) are excluded from the response.
+
+**Request:**
+
+No request body.
+
+```bash
+curl -s --cert client.p12:password \
+  https://203.0.113.42:9292/api/users | jq
+```
+
+**Response (200):**
+
+```json
+{
+  "users": [
+    {
+      "username": "admin",
+      "displayname": "Admin User",
+      "email": "admin@example.com",
+      "groups": []
+    },
+    {
+      "username": "alice",
+      "displayname": "Alice Smith",
+      "email": "alice@example.com",
+      "groups": ["developers"]
+    }
+  ]
+}
+```
+
+| Field         | Type       | Description                 |
+| ------------- | ---------- | --------------------------- |
+| `username`    | `string`   | Unique login identifier     |
+| `displayname` | `string`   | Human-readable display name |
+| `email`       | `string`   | User's email address        |
+| `groups`      | `string[]` | Authelia group memberships  |
+
+**Errors:**
+
+| Status | Body                                       | When                             |
+| ------ | ------------------------------------------ | -------------------------------- |
+| 500    | `{"error":"Failed to read user database"}` | Cannot read or parse `users.yml` |
+
+---
+
+### `POST /api/users`
+
+Creates a new Authelia user. The password is hashed with bcrypt before being stored in `users.yml`.
+
+**Request:**
+
+```json
+{
+  "username": "alice",
+  "displayname": "Alice Smith",
+  "email": "alice@example.com",
+  "password": "s3cur3p4ssw0rd!",
+  "groups": ["developers"]
+}
+```
+
+| Field         | Type       | Validation                                                                                  | Description                                 |
+| ------------- | ---------- | ------------------------------------------------------------------------------------------- | ------------------------------------------- |
+| `username`    | `string`   | 2-32 chars, lowercase alphanumeric + underscores + hyphens, starting with a letter or digit | Login identifier                            |
+| `displayname` | `string`   | 1-100 chars                                                                                 | Display name                                |
+| `email`       | `string`   | Valid email format                                                                          | Email address                               |
+| `password`    | `string`   | 8-128 chars                                                                                 | Plain-text password (hashed before storage) |
+| `groups`      | `string[]` | Optional, defaults to `[]`                                                                  | Authelia group memberships                  |
+
+**Username regex** (the same rule applies to invitations — a leading `-` would read as a flag to Authelia's CLI when a TOTP secret is stored):
+
+```
+^[a-z0-9][a-z0-9_-]*$
+```
+
+```bash
+curl -s --cert client.p12:password \
+  -X POST \
+  -H "Content-Type: application/json" \
+  -d '{"username":"alice","displayname":"Alice Smith","email":"alice@example.com","password":"s3cur3p4ssw0rd!"}' \
+  https://203.0.113.42:9292/api/users | jq
+```
+
+**Response (201):**
+
+```json
+{
+  "ok": true,
+  "user": {
+    "username": "alice",
+    "displayname": "Alice Smith",
+    "email": "alice@example.com",
+    "groups": []
+  }
+}
+```
+
+**Errors:**
+
+| Status | Body                                                       | When                                        |
+| ------ | ---------------------------------------------------------- | ------------------------------------------- |
+| 400    | `{"error":"Validation failed","details":{"issues":[...]}}` | Invalid username, email, or password length |
+| 409    | `{"error":"Username already exists"}`                      | A user with this username already exists    |
+| 500    | `{"error":"Failed to read user database"}`                 | Cannot read `users.yml`                     |
+| 500    | `{"error":"Failed to hash password"}`                      | bcrypt hashing failed                       |
+| 500    | `{"error":"Failed to update user database"}`               | Cannot write `users.yml`                    |
+
+---
+
+### `PUT /api/users/:username`
+
+Updates one or more fields of an existing user. At least one field must be provided. Only the fields included in the request body are updated — omitted fields remain unchanged.
+
+If a new password is provided, it is hashed with bcrypt before storage.
+
+**Request:**
+
+```json
+{
+  "displayname": "Alice J. Smith",
+  "email": "alice.smith@example.com",
+  "password": "n3wP4ssw0rd!",
+  "groups": ["developers", "admins"]
+}
+```
+
+| Field         | Type       | Validation            | Description                                  |
+| ------------- | ---------- | --------------------- | -------------------------------------------- |
+| `displayname` | `string`   | 1-100 chars, optional | New display name                             |
+| `email`       | `string`   | Valid email, optional | New email address                            |
+| `password`    | `string`   | 8-128 chars, optional | New password (hashed before storage)         |
+| `groups`      | `string[]` | Optional              | New group memberships (replaces entire list) |
+
+All fields are optional, but at least one must be provided (validated by a Zod `.refine()`).
+
+```bash
+curl -s --cert client.p12:password \
+  -X PUT \
+  -H "Content-Type: application/json" \
+  -d '{"displayname":"Alice J. Smith","groups":["developers","admins"]}' \
+  https://203.0.113.42:9292/api/users/alice | jq
+```
+
+**Response (200):**
+
+```json
+{
+  "ok": true,
+  "user": {
+    "username": "alice",
+    "displayname": "Alice J. Smith",
+    "email": "alice.smith@example.com",
+    "groups": ["developers", "admins"]
+  }
+}
+```
+
+**Errors:**
+
+| Status | Body                                                       | When                                             |
+| ------ | ---------------------------------------------------------- | ------------------------------------------------ |
+| 400    | `{"error":"Validation failed","details":{"issues":[...]}}` | No fields provided, or invalid field values      |
+| 404    | `{"error":"User not found"}`                               | No user with this username                       |
+| 500    | `{"error":"Failed to read user database"}`                 | Cannot read `users.yml`                          |
+| 500    | `{"error":"Failed to hash password"}`                      | bcrypt hashing failed (when password is updated) |
+| 500    | `{"error":"Failed to update user database"}`               | Cannot write `users.yml`                         |
+
+---
+
+### `DELETE /api/users/:username`
+
+Deletes an Authelia user. The user is removed from `users.yml` and Authelia is reloaded.
+
+You cannot delete the last remaining user. This safety check prevents accidentally locking out all users from tunneled applications.
+
+**Request:**
+
+No request body.
+
+```bash
+curl -s --cert client.p12:password \
+  -X DELETE \
+  https://203.0.113.42:9292/api/users/alice | jq
+```
+
+**Response (200):**
+
+```json
+{
+  "ok": true
+}
+```
+
+**Errors:**
+
+| Status | Body                                         | When                       |
+| ------ | -------------------------------------------- | -------------------------- |
+| 400    | `{"error":"Cannot delete the last user"}`    | Only one user remains      |
+| 404    | `{"error":"User not found"}`                 | No user with this username |
+| 500    | `{"error":"Failed to read user database"}`   | Cannot read `users.yml`    |
+| 500    | `{"error":"Failed to update user database"}` | Cannot write `users.yml`   |
+
+---
+
+### `POST /api/users/:username/reset-totp`
+
+Generates a new TOTP secret for a user and writes it to Authelia's SQLite database via the `authelia storage user totp generate` CLI command. Returns an `otpauth://` URI that can be used to enroll an authenticator app. The old TOTP secret is immediately replaced — the user's existing authenticator enrollment becomes invalid.
+
+**Request:**
+
+No request body.
+
+```bash
+curl -s --cert client.p12:password \
+  -X POST \
+  https://203.0.113.42:9292/api/users/alice/reset-totp | jq
+```
+
+**Response (200):**
+
+```json
+{
+  "ok": true,
+  "totpUri": "otpauth://totp/Lamaste:alice?secret=JBSWY3DPEHPK3PXP&issuer=Lamaste&algorithm=SHA1&digits=6&period=30"
+}
+```
+
+| Field     | Type      | Description                                                                     |
+| --------- | --------- | ------------------------------------------------------------------------------- |
+| `ok`      | `boolean` | Always `true` on success                                                        |
+| `totpUri` | `string`  | `otpauth://` URI for QR code generation or manual entry in an authenticator app |
+
+The `totpUri` follows the [Key Uri Format](https://github.com/google/google-authenticator/wiki/Key-Uri-Format) standard. The panel client renders this as a QR code for scanning with Google Authenticator, Authy, or similar apps.
+
+**Errors:**
+
+| Status | Body                                             | When                                                   |
+| ------ | ------------------------------------------------ | ------------------------------------------------------ |
+| 404    | `{"error":"User not found"}`                     | No user with this username                             |
+| 500    | `{"error":"Failed to read user database"}`       | Cannot read `users.yml`                                |
+| 500    | `{"error":"Failed to write TOTP configuration"}` | Cannot write TOTP secret to Authelia's SQLite database |
+
+## Quick Reference
+
+| Method | Path                              | Description                                        |
+| ------ | --------------------------------- | -------------------------------------------------- |
+| GET    | `/api/users`                      | List all users (alphabetical, no sensitive fields) |
+| POST   | `/api/users`                      | Create user (bcrypt hash + Authelia reload)        |
+| PUT    | `/api/users/:username`            | Update user fields (partial update)                |
+| DELETE | `/api/users/:username`            | Delete user (not the last one)                     |
+| POST   | `/api/users/:username/reset-totp` | Generate new TOTP secret                           |
+
+### User Object Shape
+
+Returned by list and mutation endpoints (sensitive fields excluded):
+
+```json
+{
+  "username": "alice",
+  "displayname": "Alice Smith",
+  "email": "alice@example.com",
+  "groups": ["developers"]
+}
+```
+
+### Validation Summary
+
+| Field         | Rules                                         |
+| ------------- | --------------------------------------------- |
+| `username`    | 2-32 chars, `/^[a-z0-9][a-z0-9_-]*$/`, unique |
+| `displayname` | 1-100 chars                                   |
+| `email`       | Valid email format                            |
+| `password`    | 8-128 chars                                   |
+| `groups`      | Array of strings, optional                    |
+
+### curl Cheat Sheet
+
+```bash
+# List users
+curl -s --cert client.p12:password \
+  https://203.0.113.42:9292/api/users | jq
+
+# Create user
+curl -s --cert client.p12:password \
+  -X POST -H "Content-Type: application/json" \
+  -d '{"username":"bob","displayname":"Bob","email":"bob@example.com","password":"mypassword"}' \
+  https://203.0.113.42:9292/api/users | jq
+
+# Update user (change display name and groups)
+curl -s --cert client.p12:password \
+  -X PUT -H "Content-Type: application/json" \
+  -d '{"displayname":"Robert","groups":["admins"]}' \
+  https://203.0.113.42:9292/api/users/bob | jq
+
+# Delete user
+curl -s --cert client.p12:password \
+  -X DELETE \
+  https://203.0.113.42:9292/api/users/bob | jq
+
+# Reset TOTP
+curl -s --cert client.p12:password \
+  -X POST \
+  https://203.0.113.42:9292/api/users/bob/reset-totp | jq
+```

package/vendor/docs/05-operations/backup-and-restore.md

@@ -0,0 +1,342 @@
+# Backup and Restore
+
+> Protect your Lamaste installation by backing up critical configuration, state, and certificates.
+
+## In Plain English
+
+A backup is a copy of everything that makes your Lamaste server work the way you set it up — your domain configuration, user accounts, encryption certificates, and tunnel definitions. If something goes wrong (accidental deletion, server failure, or provider issue), a backup lets you rebuild everything on a fresh server instead of starting from scratch.
+
+Lamaste stores all its important data in a handful of directories. There is no database to dump. Everything is plain files: JSON, YAML, and certificate files. Backing up means copying these directories to a safe location.
+
+## For Users
+
+### Critical Files to Back Up
+
+Lamaste's state is spread across four directories. All of them matter.
+
+#### `/etc/lamalibre/lamaste/` — Configuration, State, and PKI
+
+This is the most important directory. It contains:
+
+| File                      | Purpose                                                                                                                                                                                                   |
+| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `panel.json`              | Server configuration (IP, domain, email, onboarding status)                                                                                                                                               |
+| `tunnels.json`            | Tunnel definitions (subdomain, port, carrying agent, enabled state)                                                                                                                                       |
+| `sites.json`              | Static site definitions (including custom-domain aliases)                                                                                                                                                 |
+| `chisel-credentials.json` | Per-agent Chisel credentials. If lost, the panel mints a new credential for every active agent at its next start, and each agent's sync timer fetches it within about 30 seconds (the issue time changed) |
+| `chisel-users`            | Chisel authfile — derived; rewritten from the two files above, with mode `0640` and group `lamaste-chisel`, at every panel start                                                                          |
+| `chisel-sentinel`         | Password of the no-grants sentinel Chisel user                                                                                                                                                            |
+| `invitations.json`        | Pending user invitations                                                                                                                                                                                  |
+| `pki/revoked.json`        | Revoked certificate tracking                                                                                                                                                                              |
+| `pki/ca.key`              | Certificate Authority private key                                                                                                                                                                         |
+| `pki/ca.crt`              | Certificate Authority certificate                                                                                                                                                                         |
+| `pki/client.key`          | Client certificate private key                                                                                                                                                                            |
+| `pki/client.crt`          | Client certificate                                                                                                                                                                                        |
+| `pki/client.p12`          | PKCS12 bundle (imported into browsers)                                                                                                                                                                    |
+| `pki/.p12-password`       | Password for the PKCS12 bundle                                                                                                                                                                            |
+| `pki/self-signed.pem`     | Self-signed TLS cert for IP:9292 access                                                                                                                                                                   |
+| `pki/self-signed-key.pem` | Self-signed TLS key for IP:9292 access                                                                                                                                                                    |
+
+**If you lose the PKI files**, every browser that imported the client certificate will need a new one. This is the single most important directory to back up.
+
+#### `/etc/authelia/` — User Database and Authentication Config
+
+| File                | Purpose                                                   |
+| ------------------- | --------------------------------------------------------- |
+| `configuration.yml` | Authelia server configuration                             |
+| `users.yml`         | User accounts (usernames, bcrypt password hashes, groups) |
+| `.secrets.json`     | JWT secret, session secret, storage encryption key        |
+| `db.sqlite3`        | Authelia session/state database (TOTP registrations)      |
+
+**If you lose the users database**, all user accounts and TOTP secrets are gone. Users will need to be recreated and re-enroll their authenticator apps.
+
+#### `/etc/letsencrypt/` — TLS Certificates
+
+| Path                   | Purpose                                  |
+| ---------------------- | ---------------------------------------- |
+| `live/*/fullchain.pem` | Active certificate chain for each domain |
+| `live/*/privkey.pem`   | Private key for each domain              |
+| `renewal/*.conf`       | Auto-renewal configuration per domain    |
+| `accounts/`            | Let's Encrypt account credentials        |
+
+**If you lose Let's Encrypt certificates**, you can re-issue them (free, no limit for different domains), but it takes a few minutes per domain and is subject to rate limits (50 certificates per registered domain per week).
+
+#### `/etc/nginx/` — Reverse Proxy Configuration
+
+| Path                                   | Purpose                                       |
+| -------------------------------------- | --------------------------------------------- |
+| `sites-available/lamalibre-lamaste-*`  | Lamaste vhost configurations                  |
+| `sites-enabled/lamalibre-lamaste-*`    | Symlinks to enabled vhosts                    |
+| `snippets/lamalibre-lamaste-mtls.conf` | mTLS configuration snippet                    |
+| `nginx.conf`                           | Main nginx configuration (usually unmodified) |
+
+**If you lose nginx configs**, the panel can regenerate vhosts for tunnels and sites, but you will need to re-run the provisioning or manually recreate the base vhosts (panel, auth, tunnel).
+
+### Backup Commands
+
+#### Full Backup (Recommended)
+
+Run this on the droplet to create a single compressed archive of everything:
+
+```bash
+# Create a timestamped backup
+BACKUP_FILE="/root/lamaste-backup-$(date +%Y%m%d-%H%M%S).tar.gz"
+
+sudo tar czf "$BACKUP_FILE" \
+  /etc/lamalibre/lamaste/ \
+  /etc/authelia/ \
+  /etc/letsencrypt/ \
+  /etc/nginx/sites-available/lamalibre-lamaste-* \
+  /etc/nginx/sites-enabled/lamalibre-lamaste-* \
+  /etc/nginx/snippets/lamalibre-lamaste-mtls.conf \
+  /etc/systemd/system/lamalibre-lamaste-serverd.service \
+  /etc/systemd/system/chisel.service \
+  /etc/systemd/system/authelia.service \
+  /etc/fail2ban/jail.d/lamaste.conf \
+  /etc/sudoers.d/lamaste \
+  2>/dev/null
+
+echo "Backup created: $BACKUP_FILE"
+ls -lh "$BACKUP_FILE"
+```
+
+#### Download the Backup to Your Local Machine
+
+```bash
+# From your local machine
+scp root@203.0.113.42:/root/lamaste-backup-*.tar.gz ~/backups/
+```
+
+#### Minimal Backup (Essential Files Only)
+
+If you just want the irreplaceable data (certificates and state):
+
+```bash
+BACKUP_FILE="/root/lamaste-essential-$(date +%Y%m%d-%H%M%S).tar.gz"
+
+sudo tar czf "$BACKUP_FILE" \
+  /etc/lamalibre/lamaste/ \
+  /etc/authelia/users.yml \
+  /etc/authelia/.secrets.json \
+  /etc/authelia/configuration.yml
+
+echo "Essential backup created: $BACKUP_FILE"
+```
+
+#### Automated Daily Backup
+
+Create a cron job that runs daily and keeps the last 7 backups:
+
+```bash
+sudo tee /etc/cron.daily/lamaste-backup << 'SCRIPT'
+#!/bin/bash
+BACKUP_DIR="/root/lamaste-backups"
+mkdir -p "$BACKUP_DIR"
+
+tar czf "$BACKUP_DIR/lamaste-$(date +%Y%m%d).tar.gz" \
+  /etc/lamalibre/lamaste/ \
+  /etc/authelia/ \
+  /etc/letsencrypt/ \
+  /etc/nginx/sites-available/lamalibre-lamaste-* \
+  /etc/nginx/snippets/lamalibre-lamaste-mtls.conf \
+  2>/dev/null
+
+# Keep only the last 7 backups
+ls -t "$BACKUP_DIR"/lamaste-*.tar.gz | tail -n +8 | xargs -r rm --
+SCRIPT
+
+sudo chmod +x /etc/cron.daily/lamaste-backup
+```
+
+### Restore Procedure
+
+#### Restoring to the Same Server
+
+If you need to restore files on the same server (e.g., after accidental deletion):
+
+**Step 1: Stop all services**
+
+```bash
+sudo systemctl stop lamalibre-lamaste-serverd
+sudo systemctl stop chisel
+sudo systemctl stop authelia
+sudo systemctl stop nginx
+```
+
+**Step 2: Extract the backup**
+
+```bash
+# Preview the contents first
+tar tzf /root/lamaste-backup-20260313-103045.tar.gz | head -30
+
+# Extract (overwrites existing files)
+sudo tar xzf /root/lamaste-backup-20260313-103045.tar.gz -C /
+```
+
+**Step 3: Fix permissions**
+
+```bash
+sudo chown -R lamaste:lamaste /etc/lamalibre/lamaste/
+sudo chmod 700 /etc/lamalibre/lamaste/pki/
+sudo chmod 600 /etc/lamalibre/lamaste/pki/ca.key
+sudo chmod 600 /etc/lamalibre/lamaste/pki/client.key
+sudo chmod 600 /etc/lamalibre/lamaste/pki/client.p12
+sudo chmod 600 /etc/lamalibre/lamaste/pki/.p12-password
+sudo chmod 640 /etc/lamalibre/lamaste/panel.json
+# Files the Chisel server (User=nobody, Group=lamaste-chisel) must read
+sudo chgrp lamaste-chisel /etc/lamalibre/lamaste/chisel-users /etc/lamalibre/lamaste/chisel-server.key
+sudo chmod 640 /etc/lamalibre/lamaste/chisel-users /etc/lamalibre/lamaste/chisel-server.key
+# Authelia: the panel owns its configuration; Authelia (lamaste-authelia) reads it
+# through the group and owns its database and notification file
+sudo chown lamaste:lamaste-authelia /etc/authelia /etc/authelia/configuration.yml /etc/authelia/users.yml /etc/authelia/.secrets.json
+sudo chmod 2770 /etc/authelia
+sudo chmod 640 /etc/authelia/configuration.yml
+sudo chmod 660 /etc/authelia/users.yml
+sudo chmod 600 /etc/authelia/.secrets.json
+sudo chown lamaste-authelia:lamaste-authelia /etc/authelia/db.sqlite3* /etc/authelia/notifications.txt
+sudo chmod 600 /etc/authelia/db.sqlite3* /etc/authelia/notifications.txt
+```
+
+A backup taken on a version before the privilege boundary holds root-owned Authelia files and older unit files. After restoring one, re-run `npx @lamalibre/create-lamaste@4.0.0 --yes`: the redeploy rewrites the `chisel` and `authelia` units and converts the ownership of `/etc/authelia`, the Chisel key and the PKI directory (see [Upgrades](upgrades.md)).
+
+The panel re-applies the authfile's group and mode itself on every start, so a Chisel that fails to read it before the panel is up is started again by the panel's reconciliation.
+
+**Step 4: Validate nginx configuration**
+
+```bash
+sudo nginx -t
+```
+
+If the test fails, check for missing certificate files referenced in the vhosts.
+
+**Step 5: Restart services**
+
+```bash
+sudo systemctl start nginx
+sudo systemctl start authelia
+sudo systemctl start chisel
+sudo systemctl start lamalibre-lamaste-serverd
+```
+
+**Step 6: Verify**
+
+```bash
+sudo systemctl status nginx chisel authelia lamalibre-lamaste-serverd
+curl -s http://127.0.0.1:3100/api/health
+```
+
+#### Restoring to a New Server (Migration)
+
+To move Lamaste to a new droplet:
+
+**Step 1: Create a fresh droplet** (Ubuntu 24.04, same region if possible)
+
+**Step 2: Run the installer**
+
+```bash
+npx @lamalibre/create-lamaste@latest --yes
+```
+
+**Step 3: Stop services**
+
+```bash
+sudo systemctl stop lamalibre-lamaste-serverd chisel authelia nginx
+```
+
+**Step 4: Upload and extract the backup**
+
+```bash
+# From your local machine
+scp ~/backups/lamaste-backup-20260313-103045.tar.gz root@<new-ip>:/root/
+
+# On the new server
+sudo tar xzf /root/lamaste-backup-20260313-103045.tar.gz -C /
+```
+
+**Step 5: Update the IP address in panel.json**
+
+The new server has a different IP. Update it:
+
+```bash
+# Replace old IP with new IP
+sudo sed -i 's/OLD_IP/NEW_IP/g' /etc/lamalibre/lamaste/panel.json
+```
+
+**Step 6: Regenerate the self-signed certificate for the new IP**
+
+```bash
+sudo openssl req -x509 -nodes -days 3650 \
+  -newkey rsa:2048 \
+  -keyout /etc/lamalibre/lamaste/pki/self-signed-key.pem \
+  -out /etc/lamalibre/lamaste/pki/self-signed.pem \
+  -subj "/CN=NEW_IP/O=Lamaste" \
+  -addext "subjectAltName=IP:NEW_IP"
+```
+
+**Step 7: Update DNS records** to point to the new IP
+
+**Step 8: Fix permissions (same as Step 3 above)**
+
+**Step 9: Restart services**
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl start nginx authelia chisel lamalibre-lamaste-serverd
+```
+
+**Step 10: Re-issue Let's Encrypt certificates** (they are bound to the server that responded to the ACME challenge)
+
+If DNS is already pointing to the new server, the existing certificates will still work. When they are due for renewal, certbot will handle it automatically. If you need to force re-issuance:
+
+```bash
+sudo certbot renew --force-renewal
+```
+
+**Step 11: Verify everything works** through the browser at `https://<new-ip>:9292`
+
+The existing client certificate (`.p12` file) will still work because the CA key was restored from the backup.
+
+## For Developers
+
+### State File Locations
+
+All Lamaste state is stored as flat files with atomic writes:
+
+| File                                      | Format | Write Pattern                                                 |
+| ----------------------------------------- | ------ | ------------------------------------------------------------- |
+| `/etc/lamalibre/lamaste/panel.json`       | JSON   | Write `.tmp` then `rename()`                                  |
+| `/etc/lamalibre/lamaste/tunnels.json`     | JSON   | Write `.tmp`, `fsync()`, then `rename()`                      |
+| `/etc/lamalibre/lamaste/sites.json`       | JSON   | Write `.tmp`, `fsync()`, then `rename()`                      |
+| `/etc/lamalibre/lamaste/invitations.json` | JSON   | Write `.tmp`, `fsync()`, then `rename()`                      |
+| `/etc/authelia/users.yml`                 | YAML   | Temp file in `/etc/authelia`, `fsync()`, `rename()` (no sudo) |
+
+The atomic write pattern (write to temporary file, sync, rename) ensures that a crash during a write does not corrupt the primary file. The `rename()` system call is atomic on POSIX filesystems.
+
+### Backup Considerations
+
+- **`/etc/lamalibre/lamaste/pki/ca.key`** is the root of trust. All client certificates are signed by this CA. If the CA key changes, all previously issued client certificates become invalid.
+- **`/etc/authelia/.secrets.json`** contains the JWT secret. If this changes, all active Authelia sessions are invalidated and users must re-authenticate.
+- **`/etc/authelia/db.sqlite3`** contains Authelia session data and TOTP registration states. It can be regenerated (users re-enroll TOTP), but this is disruptive.
+- **Let's Encrypt account credentials** in `/etc/letsencrypt/accounts/` are tied to the email address used during registration. If lost, certbot creates a new account on next run.
+
+## Quick Reference
+
+| Directory                     | Contains                   | Criticality                       |
+| ----------------------------- | -------------------------- | --------------------------------- |
+| `/etc/lamalibre/lamaste/`     | Config, state, PKI         | Critical — irreplaceable PKI keys |
+| `/etc/authelia/`              | User database, auth config | High — user accounts and secrets  |
+| `/etc/letsencrypt/`           | TLS certificates           | Medium — can be re-issued         |
+| `/etc/nginx/sites-available/` | Vhost configs              | Low — can be regenerated          |
+
+| Backup Command                                                                   | What It Does        |
+| -------------------------------------------------------------------------------- | ------------------- |
+| `tar czf backup.tar.gz /etc/lamalibre/lamaste/ /etc/authelia/ /etc/letsencrypt/` | Full config backup  |
+| `scp root@IP:/root/backup.tar.gz ~/backups/`                                     | Download backup     |
+| `tar xzf backup.tar.gz -C /`                                                     | Restore from backup |
+
+| Post-Restore Step    | Command                                                                                          |
+| -------------------- | ------------------------------------------------------------------------------------------------ |
+| Fix PKI permissions  | `sudo chmod 700 /etc/lamalibre/lamaste/pki/ && sudo chmod 600 /etc/lamalibre/lamaste/pki/ca.key` |
+| Validate nginx       | `sudo nginx -t`                                                                                  |
+| Restart all services | `sudo systemctl restart nginx authelia chisel lamalibre-lamaste-serverd`                         |
+| Verify health        | `curl -s http://127.0.0.1:3100/api/health`                                                       |