spiderly 19.8.6 → 19.8.8

package/agent/docs/angular-customization/{SKILL.md → index.md}

@@ -182,6 +182,15 @@ cols: Column<ProductDTO>[] = [
 
 The `onClick` callback receives an `ActionClickEvent` — `{ id, row, element, originalEvent }`. Use `element` (or `originalEvent`) to anchor an overlay/popover to the clicked action; `row` gives you the full row object. It fires only for custom `field` values (never `'Details'` / `'Delete'`).
 
+For a click on the cell value itself (not an action icon), set a column's `onCellClick` callback — the mirror of `Action.onClick`, but for plain value cells:
+
+```typescript
+new Column({ field: 'total', name: 'Total', filterType: 'numeric',
+  onCellClick: (e) => this.itemsPopover.show(e.originalEvent, e.element) }),
+```
+
+It receives a `CellClickEvent` — `{ id, field, row, value, displayValue, element, originalEvent }`, where `value` is the raw cell value and `displayValue` is the formatted text shown. Only columns that set it become clickable (they get a `cursor`/hover affordance), and the click stops propagation so it does **not** also trigger `navigateOnRowClick`. Anchor overlays with `element` — it's the clicked `<td>`, captured synchronously, so it stays valid inside an async handler (`originalEvent.currentTarget` is null once dispatch ends). Not applied to editable cells.
+
 ### Key Inputs
 
 | Input                              | Type                     | Default | Purpose                 |

package/agent/docs/angular-customization/references/helper-functions.generated.md

@@ -32,6 +32,7 @@ Reusable helpers exported from `helper-functions.ts`. Import the one you need in
 | `nameof(key1: any, key2?: any): any` |  |
 | `parseDateOnlyLocal(s: string): Date \| null` |  |
 | `pushAction(cols: Column[], action: Action)` |  |
+| `saveResponseAsFile(res: HttpResponse<Blob>, fallbackName: string): void` |  |
 | `selectedTab(tabs: SpiderlyTab[]): number` |  |
 | `singleOrDefault<T>(array: T[], predicate: (item: T) => boolean): T \| undefined` |  |
 | `splitPascalCase(input: string)` |  |

package/agent/docs/mapper-customization/{SKILL.md → index.md}

@@ -49,6 +49,34 @@ public class Achievement : BusinessObject<long>
 
 The string is appended directly to the generated `.NewConfig<Entity, EntityDTO>()` chain. Use this for simple field mappings that the generator doesn't produce automatically.
 
+### `[ProjectToDTO]` only fills the value — the field must exist on the DTO
+
+`[ProjectToDTO]` adds a Mapster *mapping*; it does **not** create the property. If `dest.ProductId`
+is not a property on the DTO, the field **never appears in the generated Angular type**
+(`entities.generated.ts`) — `[ProjectToDTO]` fills a value that has nowhere to land. (A common
+false alarm: the value maps fine at runtime, so it looks like the frontend "didn't regenerate" —
+but a normal `dotnet build` *does* regenerate the Angular files; the property was just never on
+the DTO for the generator to emit.)
+
+To add a computed/projected field end-to-end, declare the property on a `partial class {Entity}DTO`
+extension — it merges into the generated DTO automatically, no attribute needed — **then** map its value:
+
+```csharp
+// 1. The property — a partial extension of the generated OrderItemDTO. No [SpiderlyDTO]
+//    needed: a partial that extends a generated DTO is merged in by name.
+public partial class OrderItemDTO
+{
+    public int? ProductId { get; set; }
+}
+
+// 2. The value — [ProjectToDTO] on the entity fills it during projection.
+[ProjectToDTO(".Map(dest => dest.ProductId, src => src.ProductVariant.ProductId)")]
+public class OrderItem : BusinessObject<long> { /* ... */ }
+```
+
+Then `dotnet build` the backend — the source generators run on build and the field appears
+in `entities.generated.ts`. There is no separate "regenerate" command; the build is it.
+
 ## Partial Method Override — Full Control
 
 For complex mapping logic, override the entire generated method. The generator **skips generation** for any method that already exists in the user's partial `Mapper` class (detected by method name match).

package/agent/manifest.json

@@ -5,10 +5,20 @@
       "surface": "skill",
       "description": "Scaffold a new Spiderly entity end-to-end (entity class, Angular pages, routes, menu, migration)"
     },
+    {
+      "name": "api-keys",
+      "surface": "skill",
+      "description": "Add the API-keys feature to a Spiderly app — per-key authentication via an X-Api-Key header, where each key is a first-class principal carrying its own roles. Use when a project needs machine/partner/agent access to its REST API (generate, scope-to-roles, expire, revoke). Opt-in; not part of the default app."
+    },
+    {
+      "name": "backups",
+      "surface": "skill",
+      "description": "Back up and restore a Spiderly app's Postgres database, and archive logs off the VPS. Use when setting up automated database backups, scheduling pg_dump to object storage (Cloudflare R2), restoring from a backup, running a restore drill, configuring backup retention, or archiving Serilog file logs. Covers the backup and restore scripts, cron scheduling, and the R2 bucket Terraform."
+    },
     {
       "name": "deployment",
       "surface": "skill",
-      "description": "Deploy a Spiderly project to your own infrastructure with Docker, Caddy, and Terraform. Use when deploying, redeploying, shipping, releasing, or rolling out the .NET backend or Angular admin to production — first-time setup or an ongoing deploy — and when diagnosing a down or erroring production origin (502/521, container crash-loop, failed deploy workflow). Also covers CI/CD pipelines, TLS with Cloudflare origin certificates, database backups and restores, and infrastructure-as-code layout for a Spiderly app."
+      "description": "Deploy a Spiderly project to production and keep it running. Use when deploying, redeploying, shipping, releasing, or rolling out the .NET backend or Angular admin — first-time setup or an ongoing deploy — and when diagnosing a down or erroring production origin (502/521, container crash-loop, failed deploy workflow). Covers the recommended VPS + Docker Compose + Caddy + Cloudflare + Terraform stack, CI/CD pipelines, TLS with Cloudflare origin certificates, and infrastructure-as-code layout."
     },
     {
       "name": "ef-migrations",

package/agent/skills/api-keys/SKILL.md

@@ -0,0 +1,267 @@
+---
+name: api-keys
+description: Add the API-keys feature to a Spiderly app — per-key authentication via an X-Api-Key header, where each key is a first-class principal carrying its own roles. Use when a project needs machine/partner/agent access to its REST API (generate, scope-to-roles, expire, revoke). Opt-in; not part of the default app.
+---
+
+# Add API Keys
+
+This builds the API-keys feature into a Spiderly consumer app: external clients authenticate with an `X-Api-Key` header, and each key is a **first-class principal** (`PrincipalKinds.ApiKey`) that carries its **own roles** — it is not an impersonation of a user. Authorization resolves a key's permissions through the same principal pipeline as a user, so `[HasPermission]` endpoints accept either a JWT or a key with no per-endpoint change.
+
+**This is opt-in.** API keys are not part of a default Spiderly app — nothing here exists until you run this skill, and an app without it has zero API-key surface. Don't add it unless the project actually needs machine/partner/agent API access.
+
+## Why a key is a principal (read once)
+
+An API key must be **individually revocable** and **separately expirable** — a stateless token (e.g. a long-lived JWT) can't be revoked without either rotating the signing key (logging out every user) or a per-request denylist. So a key is an opaque random secret, stored **hashed**, checked against a row on every request. The framework owns that mechanism (`Spiderly.Security`); this skill assembles the per-app pieces on top.
+
+Modeling the key as its own `ISecurityPrincipal` (rather than "resolve to the owning user + cap permissions") means:
+- The key's authority = the union of **its own** roles' permissions. A role-less key has **no** permissions.
+- The owning `User` is just **management metadata** (who created/lists/revokes it), decoupled from authority.
+- No runtime permission-cap machinery — authorization treats the key like any other principal.
+
+## Step 0 — Prerequisites
+
+1. **Spiderly version** — confirm the compiled mechanism is present. It must expose `Spiderly.Security.Authentication.IApiKeyAuthenticator`, `AddSpiderlyApiKeyAuthentication`, and `PrincipalKinds.ApiKey`. If a `using Spiderly.Security.Authentication;` doesn't resolve those, the app is on too old a Spiderly version — upgrade first (see the `spiderly-upgrade` skill).
+2. **The app must already use Spiderly auth** — it registers a human principal (`AddSpiderlyPrincipal<User>(PrincipalKinds.User)`) and calls `spiderly.AddAuthentication()`. API keys add a *second* principal kind alongside it.
+
+Decide with the user up front: **do they want an admin UI** to manage keys, or **API/agent-only** (drive generation via the REST endpoints / a management skill)? The backend is identical either way; the UI is the optional Step 8.
+
+## Step 1 — The `ApiKey` entity (+ junction + navs)
+
+Create the entity in the Business project's `Entities/`. Adapt the namespace and the `User`/`Role` types to the app.
+
+```csharp
+[Index(nameof(KeyHash), IsUnique = true)]
+[SpiderlyEntity]
+public class ApiKey : BusinessObject<long>, IApiKey
+{
+    [UIDoNotGenerate]
+    [Required]
+    [StringLength(64, MinimumLength = 1)]
+    public string KeyHash { get; set; }            // SHA-256 of the key; the plaintext is never stored
+
+    [Required]
+    [StringLength(100, MinimumLength = 1)]
+    [DisplayName]
+    public string Name { get; set; }
+
+    // Creator — the principal (any kind) that minted this key. Audit metadata, auto-stamped at creation.
+    // A soft reference (no FK): the creator can be any principal kind (User, ApiKey, ...), so it's stored
+    // as principal id + kind rather than a typed navigation.
+    [UIDoNotGenerate]
+    public long CreatedById { get; set; }
+
+    [UIDoNotGenerate]
+    [Required]
+    [StringLength(50, MinimumLength = 1)]
+    public string CreatedByPrincipalKind { get; set; }
+
+    public DateTime? ExpiresAt { get; set; }
+
+    public bool? IsRevoked { get; set; }
+
+    // ISecurityPrincipal: the key's authority is the union of its own roles' permissions (M2M, like User.Roles).
+    public virtual List<Role> Roles { get; } = new(); // M2M
+    IReadOnlyCollection<IRole> ISecurityPrincipal.Roles => Roles;
+
+    public bool? IsDisabled { get; set; }
+}
+```
+
+`IApiKey` (which extends `ISecurityPrincipal`) is `Spiderly.Security.Interfaces`; `IRole` is `Spiderly.Shared.Interfaces`. The entity implements `IApiKey` so the framework's default authenticator (Step 2) reads it generically.
+
+Add the M2M junction (mirrors the app's `UserRole`):
+
+```csharp
+[M2M]
+[SpiderlyEntity]
+public class ApiKeyRole
+{
+    [M2MWithMany(nameof(ApiKey.Roles))]
+    public virtual ApiKey ApiKey { get; set; }
+
+    [M2MWithMany(nameof(Role.ApiKeys))]
+    public virtual Role Role { get; set; }
+}
+```
+
+Add the inverse M2M nav `public virtual List<ApiKey> ApiKeys { get; } = new(); // M2M` on **`Role`**. There is **no** nav on `User` — the creator is a soft reference (id + kind), not a navigation.
+
+## Step 2 — The authenticator (you don't write one)
+
+The framework ships `DefaultApiKeyAuthenticator<TApiKey>`, which does the hash→active-id lookup over your `ApiKey` entity generically (it reads it via the `IApiKey` interface, so it stays decoupled from your namespace). You get it for free from the Step 4 registration — there is **no authenticator to hand-write**.
+
+Only implement your own `IApiKeyAuthenticator` for a non-standard lookup (e.g. an external key store). If you do, register it **before** `AddSpiderlyApiKeyAuthentication<ApiKey>()` — the framework adds the default with `TryAdd`, so your registration wins.
+
+## Step 3 — `ApiKeyService` hooks (generate + hash + show-once)
+
+Extend the generated `ApiKeyServiceGenerated` so the admin/CRUD save path mints the key, stores only its hash, and returns the plaintext **exactly once**:
+
+```csharp
+[SpiderlyService]
+public class ApiKeyService : ApiKeyServiceGenerated
+{
+    private readonly AuthenticationService _authenticationService;
+    private readonly AuthorizationService _authorizationService;   // the app's most-derived authorization service
+    private string _generatedPlainTextKey;
+
+    public ApiKeyService(EntityServiceDependencies deps, AuthenticationService authenticationService, AuthorizationService authorizationService) : base(deps)
+    {
+        _authenticationService = authenticationService;
+        _authorizationService = authorizationService;
+    }
+
+    protected override async Task OnBeforeSaveApiKeyAndReturnMainUIFormDTO(ApiKeySaveBodyDTO saveBodyDTO)
+    {
+        if (saveBodyDTO.ApiKeyDTO.Id <= 0)
+        {
+            string plainTextKey = ApiKeyHelper.GenerateRandomKey();
+            saveBodyDTO.ApiKeyDTO.KeyHash = ApiKeyHelper.ComputeSha256Hash(plainTextKey);
+            _generatedPlainTextKey = plainTextKey;
+            saveBodyDTO.ApiKeyDTO.CreatedById = _authenticationService.GetCurrentUserId();
+            saveBodyDTO.ApiKeyDTO.CreatedByPrincipalKind = _authenticationService.GetCurrentPrincipalKind();
+        }
+        else
+        {
+            saveBodyDTO.ApiKeyDTO.KeyHash = await _deps.Context.DbSet<ApiKey>()
+                .AsNoTracking().Where(x => x.Id == saveBodyDTO.ApiKeyDTO.Id).Select(x => x.KeyHash).FirstAsync();
+        }
+    }
+
+    protected override async Task OnAfterSaveApiKeyAndReturnMainUIFormDTO(ApiKeySaveBodyDTO saveBodyDTO, ApiKeyMainUIFormDTO mainUIFormDTO)
+    {
+        if (_generatedPlainTextKey != null)
+        {
+            mainUIFormDTO.PlainTextKey = _generatedPlainTextKey;   // add this property via a partial ApiKeyMainUIFormDTO (Step 6)
+            _generatedPlainTextKey = null;
+        }
+    }
+
+    // Guard EVERY role-assignment path, not just the Generate endpoint: an admin may only grant a key roles
+    // whose permissions they themselves hold. This is the M2M role-mutation method, so guarding it here covers
+    // the admin roles editor (and any future caller) by construction.
+    public override async Task UpdateRolesForApiKey(long id, List<int> selectedIds)
+    {
+        await _authorizationService.AuthorizeApiKeyRoleAssignmentAsync(selectedIds);
+        await base.UpdateRolesForApiKey(id, selectedIds);
+    }
+}
+```
+
+`ApiKeyHelper` is `Spiderly.Security.Authentication`. If the app's authorization service has a different class name, inject that.
+
+## Step 4 — Register the principal + the auth scheme
+
+In the app's service-registration (where `AddSpiderlyPrincipal<User>` is called, **after** `AddSpiderly(...)`):
+
+```csharp
+services.AddSpiderlyPrincipal<ApiKey>(PrincipalKinds.ApiKey);
+services.AddSpiderlyApiKeyAuthentication<ApiKey>();
+```
+
+`AddSpiderlyApiKeyAuthentication` registers the `ApiKey` scheme **and** a forwarding policy scheme set as the default — so `X-Api-Key` requests route to the key handler and everything else to JWT, with no per-endpoint `[Authorize(AuthenticationSchemes=...)]` churn. Registering a second principal kind makes the app multi-principal; the JWT login already stamps `PrincipalKinds.User`, the key handler stamps `PrincipalKinds.ApiKey`, so both satisfy the now-required `principal_kind` claim.
+
+## Step 5 — The issuance guard (authorization service)
+
+In the app's authorization service (the `[SpiderlyService]` extending `AuthorizationServiceGenerated`), add the guard the `UpdateRolesForApiKey` override and the Generate endpoint both call:
+
+```csharp
+/// <summary>
+/// Issuance guard: the principal creating a key may only grant it roles whose permissions that principal
+/// already holds, so a key can never be minted more powerful than its creator. The creator can be any
+/// principal kind (a user, a service account, another key), so the ceiling is resolved principal-agnostically
+/// via GetCurrentPrincipalPermissionCodesAsync() (not the User-typed GetCurrentUserPermissionCodes). No
+/// runtime cap — a key authorizes through its own roles — so this is enforced once, at assignment.
+/// </summary>
+public async Task AuthorizeApiKeyRoleAssignmentAsync(List<int> roleIds)
+{
+    if (roleIds == null || roleIds.Count == 0)
+        return;
+
+    HashSet<string> creatorPermissions = (await GetCurrentPrincipalPermissionCodesAsync()).ToHashSet();
+
+    List<string> targetRolePermissions = await _context.DbSet<Role>()
+        .AsNoTracking()
+        .Where(r => roleIds.Contains(r.Id))
+        .SelectMany(r => r.Permissions)
+        .Select(p => p.Code)
+        .Distinct()
+        .ToListAsync();
+
+    if (!targetRolePermissions.All(p => creatorPermissions.Contains(p)))
+        throw new BusinessException(_localizer["ApiKeyRoleExceedsYourPermissionsException"]);
+}
+```
+
+Add the `ApiKeyRoleExceedsYourPermissionsException` translation key (see the `backend-localization` doc).
+
+## Step 6 — Generate / Revoke endpoints + DTOs
+
+These are the machine-facing REST surface (also what a management skill drives). Add to a `[SpiderlyController]` (e.g. the security controller). DTOs go in the Admin DTO folder; the `PlainTextKey` extension is a **partial** of the generated `ApiKeyMainUIFormDTO` and must use the bare `...Business.DTO` namespace (see the `entity-design` doc on partial-DTO namespacing).
+
+```csharp
+// GenerateApiKeyRequestDTO:  string Name; int? ExpiresInDays; [Required] List<int> RoleIds = new();
+// GenerateApiKeyResponseDTO: string Key; string Name; DateTime? ExpiresAt; [Required] List<int> RoleIds = new(); [Required] List<string> RoleNames = new();
+// RevokeApiKeyRequestDTO:    [Required] long ApiKeyId;
+// partial ApiKeyMainUIFormDTO (bare DTO namespace): [UIDoNotGenerate][StringLength(64)] string PlainTextKey;
+
+[HttpPost, AuthGuard]
+public async Task<GenerateApiKeyResponseDTO> GenerateApiKey([FromBody] GenerateApiKeyRequestDTO request)
+{
+    await _authorizationService.AuthorizeAndThrowAsync(PermissionCodes.InsertApiKey);   // principal-agnostic — any principal kind can create a key
+    long currentUserId = _authenticationService.GetCurrentUserId();
+
+    List<Role> roles = new();
+    if (request.RoleIds != null && request.RoleIds.Count > 0)
+    {
+        roles = await _context.DbSet<Role>().Where(x => request.RoleIds.Contains(x.Id)).ToListAsync();
+        List<int> missing = request.RoleIds.Except(roles.Select(r => r.Id)).ToList();
+        if (missing.Count > 0) throw new BusinessException($"Role(s) not found: {string.Join(", ", missing)}.");
+        await _authorizationService.AuthorizeApiKeyRoleAssignmentAsync(request.RoleIds);
+    }
+
+    string plainTextKey = ApiKeyHelper.GenerateRandomKey();
+    DateTime? expiresAt = request.ExpiresInDays.HasValue ? DateTime.UtcNow.AddDays(request.ExpiresInDays.Value) : null;
+
+    ApiKey apiKey = new() { KeyHash = ApiKeyHelper.ComputeSha256Hash(plainTextKey), Name = request.Name,
+        CreatedById = currentUserId, CreatedByPrincipalKind = _authenticationService.GetCurrentPrincipalKind(),
+        ExpiresAt = expiresAt, IsRevoked = false };
+    apiKey.Roles.AddRange(roles);
+    _context.DbSet<ApiKey>().Add(apiKey);
+    await _context.SaveChangesAsync();
+
+    return new GenerateApiKeyResponseDTO { Key = plainTextKey, Name = request.Name, ExpiresAt = expiresAt,
+        RoleIds = roles.Select(r => r.Id).ToList(), RoleNames = roles.Select(r => r.Name).ToList() };
+}
+
+[HttpPost, AuthGuard]
+public async Task RevokeApiKey([FromBody] RevokeApiKeyRequestDTO request)
+{
+    await _authorizationService.AuthorizeAndThrowAsync(PermissionCodes.UpdateApiKey);
+    ApiKey apiKey = await _context.DbSet<ApiKey>().FirstOrDefaultAsync(x => x.Id == request.ApiKeyId)
+        ?? throw new BusinessException("API key not found.");
+    apiKey.IsRevoked = true;
+    await _context.SaveChangesAsync();
+}
+```
+
+## Step 7 — Seed the `ApiKey` permissions + migrate
+
+1. **Seed permissions** so the `PermissionCodes.{Read,Insert,Update,Delete}ApiKey` constants generate — add `Permission` seed rows (codes `ReadApiKey`/`InsertApiKey`/`UpdateApiKey`/`DeleteApiKey`) in the app's DbContext seed data, alongside the existing User/Role permission seeds, and grant them to the admin role.
+2. **Migration** — add + apply an EF migration for the new `ApiKey` + `ApiKeyRole` tables (see the `ef-migrations` skill). If you're converting an *existing* single-`Role` API-key table to this M2M model, make the migration **data-preserving**: create the junction first, `INSERT ... SELECT` the old `RoleId`s into it, then drop the old column.
+
+## Step 8 — (Optional) admin UI
+
+Only if the user wants a clickable admin surface (otherwise skip — keys are managed via the endpoints / a management skill). Scaffold the list + details pages for the `ApiKey` entity (see the `add-entity` skill for pages/routes/menu), then customize the details page to show the generated key **once**: intercept the save response, and if it carries `plainTextKey`, show it in a read-only "copy this — it won't be shown again" dialog and suppress the auto-reroute until the dialog closes. A roles editor on the form routes through `UpdateRolesForApiKey`, which Step 3 already guards.
+
+To keep the feature **headless** instead, add `[UIDoNotGenerate]` to the `ApiKey` class and don't create pages.
+
+## Step 9 — Verify
+
+Build the backend (`dotnet build` the Business project, then the WebAPI project). Then sanity-check the flow: generate a key (admin JWT) → call any `[HasPermission]` endpoint the key's roles cover with `X-Api-Key: <key>` → revoke it → confirm the same call now returns 401.
+
+## Invariants to preserve
+
+- **Store only the hash.** The plaintext is returned once at generation and never persisted.
+- **Guard every role-assignment path.** Both the Generate endpoint and `UpdateRolesForApiKey` call `AuthorizeApiKeyRoleAssignmentAsync` — a key can never be minted more powerful than its creator.
+- **A role-less key has no permissions** (deny), by design. Assign an admin role for a full-access key.
+- **Revocation/expiry/disabled are checked live** in `DefaultApiKeyAuthenticator` on every request — that's the whole point of opaque keys over JWTs.

package/agent/skills/backups/SKILL.md

@@ -0,0 +1,148 @@
+---
+name: backups
+description: Back up and restore a Spiderly app's Postgres database, and archive logs off the VPS. Use when setting up automated database backups, scheduling pg_dump to object storage (Cloudflare R2), restoring from a backup, running a restore drill, configuring backup retention, or archiving Serilog file logs. Covers the backup and restore scripts, cron scheduling, and the R2 bucket Terraform.
+---
+
+# Backups
+
+If you ship this stack to prod, you ship a backup with it. Postgres data lives in a Docker volume on a single VPS — disk failure, accidental `docker volume rm`, or `terraform destroy` all wipe it without a snapshot to restore from.
+
+## Database backups (required)
+
+Daily `pg_dump` to a Cloudflare R2 bucket, 7-day retention both local and remote. ~24h RPO; if you need tighter, layer WAL archiving on top (separate skill).
+
+**1. R2 bucket — Terraform-managed.** No chicken-and-egg here (only the *state* bucket has to be bootstrapped manually); manage data buckets like any other resource:
+
+```hcl
+# infrastructure/cloudflare-r2.tf
+resource "cloudflare_r2_bucket" "db_backups" {
+  account_id = var.cloudflare_account_id
+  name       = "<your-app>-db-backups"
+  location   = "EEUR"
+
+  lifecycle {
+    prevent_destroy = true
+  }
+}
+```
+
+**2. Backup script — `infrastructure/scripts/pg_backup_s3.sh`** (deployed to `/usr/local/bin/` on the VPS). Note the `trap` cleanup, the `aws s3api list-objects-v2 --query` for retention (structured + locale-safe vs parsing `aws s3 ls` with awk), and the per-iteration warn-on-failure inside the `while` subshell (where `set -e` does NOT propagate):
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+CONTAINER_NAME="<your-app>-postgres-1"
+DB_NAME="<your-db>"
+DB_USER="postgres"
+CLOUDFLARE_ACCOUNT_ID="<account-id>"
+S3_BUCKET="s3://<your-app>-db-backups"
+R2_ENDPOINT="https://${CLOUDFLARE_ACCOUNT_ID}.r2.cloudflarestorage.com"
+RETENTION_DAYS=7
+LOCAL_BACKUP_DIR="/var/backups/postgresql"
+LOG_FILE="/var/log/<your-app>_pg_backup.log"
+
+TIMESTAMP=$(date +%Y-%m-%d_%H%M%S)
+BACKUP_FILE="<your-app>_${TIMESTAMP}.sql.gz"
+LOCAL_PATH="${LOCAL_BACKUP_DIR}/${BACKUP_FILE}"
+
+log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"; }
+
+trap 'rc=$?; [[ $rc -ne 0 ]] && rm -f "$LOCAL_PATH" && log "Aborted, removed partial $LOCAL_PATH"; exit $rc' ERR INT TERM
+
+mkdir -p "$LOCAL_BACKUP_DIR"
+
+if ! docker exec "$CONTAINER_NAME" pg_dump -U "$DB_USER" "$DB_NAME" | gzip > "$LOCAL_PATH"; then
+    log "ERROR: pg_dump failed"; exit 1
+fi
+log "Backup created: ${LOCAL_PATH} ($(du -h "$LOCAL_PATH" | cut -f1))"
+
+if ! aws s3 --endpoint-url "$R2_ENDPOINT" cp "$LOCAL_PATH" "${S3_BUCKET}/${BACKUP_FILE}"; then
+    log "ERROR: S3 upload failed"; exit 1
+fi
+log "Uploaded to ${S3_BUCKET}/${BACKUP_FILE}"
+
+find "$LOCAL_BACKUP_DIR" -name "<your-app>_*.sql.gz" -mtime +"$RETENTION_DAYS" -delete
+
+CUTOFF_ISO=$(date -u -d "-${RETENTION_DAYS} days" +%Y-%m-%dT%H:%M:%SZ)
+BUCKET_NAME="${S3_BUCKET#s3://}"
+OLD_KEYS=$(aws s3api --endpoint-url "$R2_ENDPOINT" list-objects-v2 \
+    --bucket "$BUCKET_NAME" \
+    --query "Contents[?LastModified<'${CUTOFF_ISO}'].Key" \
+    --output text 2>/dev/null || echo "")
+if [[ -n "$OLD_KEYS" && "$OLD_KEYS" != "None" ]]; then
+    for KEY in $OLD_KEYS; do
+        aws s3 --endpoint-url "$R2_ENDPOINT" rm "${S3_BUCKET}/${KEY}" \
+            && log "Deleted remote: ${KEY}" \
+            || log "WARN: failed to delete remote ${KEY}"
+    done
+fi
+```
+
+**3. VPS prerequisites.** Add `awscli` and `cron` to `cloud-init` `packages:` so future server replacements have them. Configure R2 credentials in `/root/.aws/credentials` (same keys as the Terraform state backend — they're account-scoped):
+
+```ini
+[default]
+aws_access_key_id = <R2 access key>
+aws_secret_access_key = <R2 secret>
+```
+
+**4. Cron entry** (`/etc/cron.d/<your-app>-pg-backup`). Wrap with `flock -n` so a hung run doesn't get a second instance started 24h later:
+
+```
+0 2 * * * root /usr/bin/flock -n /var/lock/<your-app>-pg-backup.lock /usr/local/bin/pg_backup_s3.sh >> /var/log/<your-app>_pg_backup.log 2>&1
+```
+
+**5. Restore — `scripts/db-restore.sh`** (run from local). Lists server-side backups via SSH, prompts for selection, takes a safety dump first (with size check — refuses to proceed if pg_dump silently produced an empty file), then restores. Note `set -euo pipefail` inside the SSH heredocs (without it, a `pg_dump` failure inside a pipeline is masked by a successful `gzip`) and the `OVERWRITE <db>` confirmation phrase (a bare DB name is too easy to typo into):
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+SSH_ALIAS="<your-app>"
+CONTAINER="<your-app>-postgres-1"
+DB="<your-db>"
+REMOTE_DIR="/var/backups/postgresql"
+MIN_SAFETY_BYTES=1024
+
+trap 'echo "Interrupted — DB may be inconsistent. Latest safety snapshot is in $SSH_ALIAS:$REMOTE_DIR."' INT TERM
+
+ssh -o ConnectTimeout=5 "$SSH_ALIAS" "docker exec $CONTAINER pg_isready -U postgres -d $DB" >/dev/null \
+    || { echo "Postgres not ready"; exit 1; }
+
+mapfile -t BACKUPS < <(ssh "$SSH_ALIAS" "ls -1t $REMOTE_DIR/<your-app>_*.sql.gz 2>/dev/null | xargs -n1 basename")
+[[ ${#BACKUPS[@]} -gt 0 ]] || { echo "No backups found"; exit 1; }
+for i in "${!BACKUPS[@]}"; do printf "  %2d) %s\n" "$((i+1))" "${BACKUPS[$i]}"; done
+read -rp "Pick: " PICK
+[[ "$PICK" =~ ^[0-9]+$ ]] && (( PICK >= 1 && PICK <= ${#BACKUPS[@]} )) || { echo "Invalid"; exit 1; }
+CHOSEN="${BACKUPS[$((PICK-1))]}"
+
+read -rp "Type 'OVERWRITE $DB' to confirm: " CONFIRM
+[[ "$CONFIRM" == "OVERWRITE $DB" ]] || { echo "aborted"; exit 1; }
+
+SAFETY="<your-app>_pre_restore_$(date +%Y-%m-%d_%H%M%S).sql.gz"
+ssh "$SSH_ALIAS" "set -euo pipefail; docker exec $CONTAINER pg_dump -U postgres $DB | gzip > $REMOTE_DIR/$SAFETY"
+SIZE=$(ssh "$SSH_ALIAS" "stat -c%s $REMOTE_DIR/$SAFETY")
+(( SIZE >= MIN_SAFETY_BYTES )) || { echo "Safety snapshot suspiciously small ($SIZE B) — aborting"; exit 1; }
+
+ssh "$SSH_ALIAS" "
+    set -euo pipefail
+    docker exec $CONTAINER psql -U postgres -d postgres -c \"DROP DATABASE IF EXISTS \\\"$DB\\\" WITH (FORCE);\"
+    docker exec $CONTAINER psql -U postgres -d postgres -c \"CREATE DATABASE \\\"$DB\\\";\"
+    gunzip -c $REMOTE_DIR/$CHOSEN | docker exec -i $CONTAINER psql -U postgres -d $DB
+"
+echo "Restored. Safety snapshot at $SSH_ALIAS:$REMOTE_DIR/$SAFETY."
+```
+
+**6. Restore drill — quarterly.** A backup you've never restored from is not a backup. At least once a quarter, restore the latest dump into a throwaway local Postgres and verify schema + row counts. Calendar reminder.
+
+## Log archival (optional)
+
+When to use it: you need long-term log retention beyond Docker's `json-file` rotation buffer (default ~150 MB rolling per container, configured in compose).
+
+Pattern: a Hangfire recurring job watches `/app/logs/`, ships files older than N days to R2 once total > threshold, deletes locally.
+
+To make `/app/logs` writable under `USER app`:
+
+- **Bind mount with chowned host dir** (recommended): `volumes: - /var/log/<your-app>:/app/logs` in compose, one-time `mkdir -p /var/log/<your-app> && chown 1654:1654 /var/log/<your-app>` on the VPS. Bind mounts respect host ownership; named volumes overlay the path with root-owned storage which `app` can't write to.
+- **No File sink at all** (simpler): rely on Console + Docker rotation. Drop the `/app/logs` volume entirely. Right answer for greenfield deploys.
+