vibed-infra 0.5.0 → 0.7.0

package/README.md

@@ -1,100 +1,81 @@
 # vibed-infra
 
-Product-agnostic VPS deployment packager: **`package.sh`** turns four YAML configs into a committed **`dist/`** tree operators wget on the box — install scripts, generic start/update helpers, env examples, and gateway nginx.
+Product-agnostic VPS packager: **`package.sh`** builds committed **`dist/`** for wget install. One **host gateway** serves many apps; a **serial update agent** pulls images without overload; **persist-logs** enable event-sourced recovery; **`dist/DNS-SKILL.md`** configures DNS via an AU browser agent.
 
 Published on npm as **`vibed-infra`**.
 
 ## Product maintainer flow
 
-Author **`templates/`** in your product repo:
-
 | File | Purpose |
 |------|---------|
-| `vibed-infra-config.yml` | Product name, template refs, edge network, gateway `sites[]`, auto-update |
-| `api-config.yaml` | API image, port, opaque `config:` |
-| `ui-config.yaml` | UI image, port |
-| `nodes-config.yaml` | Worker image, opaque `config:` |
-
-Then package and commit **`dist/`**:
+| `vibed-infra-config.yml` | Name, templates, `network.edge`, `gateway.publicIp` / `tlsEmail` / `sites[]`, auto-update |
+| `api-config.yaml` / `ui-config.yaml` / `nodes-config.yaml` | Images + opaque config |
 
 ```bash
-./package.sh                  # exec vibed-infra packager → writes dist/
+./package.sh
 git add dist && git commit && git push
+# Copy dist/DNS-SKILL.md into AU agent (domains + publicIp already filled when configured)
 ```
 
-Resolve the packager from npm:
+## Operator flow (VPS)
 
 ```bash
-node -e "console.log(require('path').dirname(require.resolve('vibed-infra/package.json')))"
-```
-
-Example product: [`examples/vps-hello`](examples/vps-hello) (`app/`, `build-images.sh`, four YAMLs, committed `dist/`).
-
-## Operator flow (wget on VPS)
-
-```bash
-wget -qO- https://raw.githubusercontent.com/ORG/REPO/main/dist/install-api.sh | bash
-wget -qO- .../dist/install-nodes.sh | bash
+# 1) DNS — paste dist/DNS-SKILL.md into AU browser agent
+# 2) App roles
+wget -qO- .../dist/install-api.sh | bash
 wget -qO- .../dist/install-ui.sh | bash
+wget -qO- .../dist/install-nodes.sh | bash
+# 3) Host gateway + TLS (setup-tls.sh; re-run with --force if host/IP changes)
 wget -qO- .../dist/install-gateway.sh | bash
-# edit each install dir .env, then ./start-*.sh
 ```
 
 | Profile | Role |
 |---------|------|
-| `api` | Backend on shared edge network |
-| `ui` | UI container (separate install) |
-| `nodes` | Worker compose on edge network |
-| `gateway` | HTTPS nginx only (API + UI must already run) |
+| `api` / `ui` / `nodes` | Join shared `vps-edge` |
+| `gateway` | Host nginx + `apps/{name}/sites.conf` + HTTPS via `~/services/gateway/setup-tls.sh` |
+
+Multi-app: further products’ `install-gateway.sh` only add `apps/{other}/sites.conf`, refresh TLS SANs, and reload — no second 80/443 bind.
+
+## Machine services (installed once)
+
+| Path | Purpose |
+|------|---------|
+| `~/services/gateway` | Host nginx (`GATEWAY_HOME`) |
+| `~/services/vibed-infra/update-agent` | Serial pull queue + optional GHCR webhook |
+| `~/services/vibed-infra/persist-logs` | Per-app WALs + optional R2/S3 ship |
+
+Auto-update cron **enqueues** work; the agent processes one job at a time. After a GHCR push, the reusable image workflow mints a GitHub Actions OIDC JWT and notifies `https://{domain}/_vibed/hooks/ghcr` (fallback `http://{publicIp}/…`) so the pull does not wait for cron. Updates prune dangling images (`DOCKER_AUTO_PRUNE=1`).
 
 ## Layout
 
 | Path | Purpose |
 |------|---------|
-| `package.sh` / `lib/package.py` | Build product `dist/` from four YAMLs + generic templates |
-| `install.sh` | Single entrypoint (`INFRA_PROFILE` or `--profile`) |
-| `install-auto-update.sh` | Cron from profile flags |
-| `lib/` | fetch, env, tls, prompt, generate, product_config |
-| `templates/generic/` | Generic start/update/env/certs/gateway |
-| `schema/packageconfig.md` | Four-file product schema + compiled packageconfig |
-| `examples/vps-hello/` | Full example with dist e2e tests |
-| `skills/` | Cursor skills |
-| `github/workflows/` | Reusable GHCR build workflow |
+| `package.sh` / `lib/package.py` | Build `dist/` |
+| `templates/host-gateway/` | Shared gateway skeleton |
+| `templates/update-agent/` | Queue agent + webhook |
+| `templates/persist-logs/` | Shipper install |
+| `lib/persistlog/` | Python append / seal / replay |
+| `skills/` | system-gateway, infra-update-agent, persist-logs, dns-configure |
 
 ## Environment
 
 | Variable | Meaning |
 |----------|---------|
-| `PACKAGER_RAW` | Base URL or local path to this package root |
-| `PACKAGECONFIG_URL` | Product `packageconfig.yaml` or `dist/` URL (URL or path) |
-| `PRODUCT_RAW` | Product `dist/` base (templates + install scripts) |
-| `INFRA_PROFILE` | `api`, `ui`, `nodes`, or `gateway` |
-| `INSTALL_DIR` | Target directory (default `.`) |
-
-Auto-update cron runs profile `update-*.sh`. Each update **prunes dangling Docker images** by default (`DOCKER_AUTO_PRUNE=1`). Set `DOCKER_PRUNE_UNUSED=1` to also drop unused tagged images older than `DOCKER_PRUNE_UNTIL` (default `72h`).
+| `GATEWAY_HOME` | Host gateway dir (default `~/services/gateway`) |
+| `VIBED_HOME` | Machine vibed root (default `~/services/vibed-infra`) |
+| `GATEWAY_PUBLIC_IP` | VPS IPv4 (from `gateway.publicIp`) |
+| `TLS_EMAIL` / `TLS_MODE` | Let’s Encrypt email; `lab` or `letsencrypt` |
+| `PERSIST_LOG_DIR` | Per-service event log dir |
+| `DOCKER_AUTO_PRUNE` | Prune dangling images after update (default on) |
 
 ## Tests
 
 ```bash
-npm test                        # package + dry-run all dist/install-*.sh
-npm run test:dist               # build images + e2e api, ui, nodes (sequential)
-./examples/vps-hello/test-dist.sh --profile api   # single profile
+npm test
+npm run test:dist
+npm run test:e2e-multi   # two apps, localhost wget|bash, shared host gateway
 ```
 
-CI: `validate` job then parallel `dist-e2e` matrix (`api`, `ui`, `nodes`).
-
-## Publishing (npm)
-
-Every **merge to `main`** runs [Publish npm](.github/workflows/publish.yml):
-
-1. Bumps **minor** (`0.1.0` → `0.2.0`)
-2. Commits `chore: release vibed-infra v…`, tags `v…`, pushes
-3. Runs `npm publish` (needs repo secret `NPM_TOKEN`)
-
-**Major versions** — set `version` in `package.json` yourself (e.g. `1.0.0`) and include **`[major]`** in the commit message. That publishes the pinned version with **no** automatic minor bump.
-
-Set repository secret **`NPM_TOKEN`** before the first merge to `main`.
-
-Ordinary PRs only run CI (`npm test` + dist e2e); they do **not** publish.
+## Publishing
 
-Runbook: [`examples/vps-hello/README.md`](examples/vps-hello/README.md).
+Merge to `main` runs Publish npm (minor bump). Set `NPM_TOKEN`. Ordinary PRs run CI only.

package/github/notify-vps-pull.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""Notify a vibed-infra VPS to pull a GHCR image.
+
+Prefers GitHub Actions OIDC (no shared secret). Falls back to compiled
+config token / VIBED_WEBHOOK_SECRET for local curl.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+from pathlib import Path
+
+# Keep in sync with lib/webhook.py
+HOOK_PATH = "/_vibed/hooks/ghcr"
+TOKEN_VERSION = "vibed-webhook|v1|"
+
+
+def _derive_token(name: str, public_ip: str, host: str) -> str:
+    import hashlib
+
+    raw = f"{TOKEN_VERSION}{name}|{public_ip}|{host}"
+    return hashlib.sha256(raw.encode()).hexdigest()[:32]
+
+
+def _strip(s: str) -> str:
+    s = s.strip()
+    if (s.startswith('"') and s.endswith('"')) or (s.startswith("'") and s.endswith("'")):
+        return s[1:-1]
+    return s
+
+
+def _load_webhook(path: Path) -> dict[str, str]:
+    text = path.read_text(encoding="utf-8")
+    url = fallback = token = name = public_ip = host = ""
+    in_webhook = False
+    in_sites = False
+    for raw in text.splitlines():
+        line = raw.rstrip()
+        stripped = line.lstrip()
+        if stripped.startswith("webhook:"):
+            in_webhook = True
+            in_sites = False
+            continue
+        if in_webhook:
+            if stripped and not line.startswith(" ") and not line.startswith("\t"):
+                in_webhook = False
+            elif stripped.startswith("url:"):
+                url = _strip(stripped.split(":", 1)[1])
+            elif stripped.startswith("fallbackUrl:"):
+                fallback = _strip(stripped.split(":", 1)[1])
+            elif stripped.startswith("token:"):
+                token = _strip(stripped.split(":", 1)[1])
+        if stripped.startswith("name:") and not name:
+            name = _strip(stripped.split(":", 1)[1])
+        if stripped.startswith("publicIp:"):
+            public_ip = _strip(stripped.split(":", 1)[1])
+        if stripped.startswith("sites:"):
+            in_sites = True
+            continue
+        if in_sites and stripped.startswith("- host:"):
+            host = _strip(stripped.split(":", 1)[1])
+            in_sites = False
+        elif in_sites and stripped.startswith("host:"):
+            host = _strip(stripped.split(":", 1)[1])
+            in_sites = False
+    if not token and name and (public_ip or host):
+        token = _derive_token(name, public_ip, host)
+    if not url and host:
+        url = f"https://{host}{HOOK_PATH}"
+    if not fallback and public_ip:
+        fallback = f"http://{public_ip}{HOOK_PATH}"
+    if not url:
+        url = fallback
+    return {"url": url, "fallbackUrl": fallback, "token": token}
+
+
+def _package_from_image(image: str) -> str:
+    img = image.split("@", 1)[0]
+    if "/" in img:
+        return img.rsplit("/", 1)[-1].split(":")[0]
+    return img.split(":")[0]
+
+
+def fetch_oidc_token(audience: str) -> str:
+    url = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_URL") or ""
+    tok = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_TOKEN") or ""
+    if not url or not tok:
+        return ""
+    sep = "&" if "?" in url else "?"
+    req = urllib.request.Request(
+        f"{url}{sep}audience={urllib.parse.quote(audience, safe='')}",
+        headers={"Authorization": f"Bearer {tok}", "Accept": "application/json"},
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=20) as resp:
+            data = json.loads(resp.read().decode())
+        return str(data.get("value") or "")
+    except Exception as e:
+        print(f"notify-vps: OIDC mint failed: {e}", file=sys.stderr)
+        return ""
+
+
+def _post(url: str, body: dict, *, bearer: str = "", secret: str = "") -> tuple[int, str]:
+    headers = {"Content-Type": "application/json"}
+    if bearer:
+        headers["Authorization"] = f"Bearer {bearer}"
+    if secret:
+        headers["X-Vibed-Secret"] = secret
+    req = urllib.request.Request(
+        url,
+        data=json.dumps(body).encode(),
+        method="POST",
+        headers=headers,
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=15) as resp:
+            return resp.status, resp.read().decode()
+    except urllib.error.HTTPError as e:
+        return e.code, e.read().decode(errors="replace")
+    except Exception as e:
+        return 0, str(e)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--image", required=True)
+    ap.add_argument("--tag", default="main")
+    ap.add_argument("--config", default="")
+    ap.add_argument("--url", action="append", default=[], help="Override webhook URL (repeatable)")
+    ap.add_argument("--strict", action="store_true", help="Exit 1 if notify fails")
+    args = ap.parse_args()
+
+    spec = {"url": "", "fallbackUrl": "", "token": ""}
+    if not args.url:
+        roots = []
+        if args.config:
+            roots.append(Path(args.config))
+        cwd = Path.cwd()
+        roots.extend(
+            [
+                cwd / "dist" / "packageconfig.yaml",
+                cwd / "packageconfig.yaml",
+                cwd / "templates" / "vibed-infra-config.yml",
+            ]
+        )
+        conf_path = next((p for p in roots if p.is_file()), None)
+        if conf_path:
+            spec = _load_webhook(conf_path)
+        elif not args.url:
+            print("notify-vps: no packageconfig (set gateway.publicIp + sites[].host)", file=sys.stderr)
+            return 1 if args.strict else 0
+
+    token = os.environ.get("VIBED_WEBHOOK_SECRET") or spec.get("token") or ""
+    urls = list(args.url) if args.url else [u for u in (spec.get("url"), spec.get("fallbackUrl")) if u]
+    seen: set[str] = set()
+    urls = [u for u in urls if not (u in seen or seen.add(u))]
+    has_oidc = bool(os.environ.get("ACTIONS_ID_TOKEN_REQUEST_URL"))
+    if not urls:
+        print("notify-vps: skipped (no webhook url)", file=sys.stderr)
+        return 1 if args.strict else 0
+    if not has_oidc and not token:
+        print("notify-vps: skipped (no OIDC env and no token)", file=sys.stderr)
+        return 1 if args.strict else 0
+
+    pkg = _package_from_image(args.image)
+    payload = {"package": pkg, "tag": args.tag, "image": args.image}
+    last_err = ""
+    for url in urls:
+        bearer = fetch_oidc_token(url) if has_oidc else ""
+        secret = "" if bearer else token
+        if not bearer and not secret:
+            last_err = f"{url}: no OIDC token"
+            print(f"notify-vps: {last_err}", file=sys.stderr)
+            continue
+        try:
+            code, text = _post(url, payload, bearer=bearer, secret=secret)
+        except Exception as e:
+            code, text = 0, str(e)
+        if code == 200:
+            print(f"notify-vps: {url} → {text.strip()}")
+            return 0
+        last_err = f"{url} HTTP {code}: {text[:200]}"
+        print(f"notify-vps: {last_err}", file=sys.stderr)
+    print(f"notify-vps: failed ({last_err})", file=sys.stderr)
+    return 1 if args.strict else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/github/workflows/docker-build-reusable.yml

@@ -4,12 +4,19 @@
 #
 #   jobs:
 #     build-api:
-#       uses: ./infra/github/workflows/docker-build-reusable.yml
+#       uses: naiemk/vibed-infra/.github/workflows/docker-build-reusable.yml@main
 #       with:
 #         dockerfile: deploy/Dockerfile.api
 #         image: ghcr.io/${{ github.repository_owner }}/my-api
 #         context: .
+#       permissions:
+#         contents: read
+#         packages: write
+#         id-token: write
 #       secrets: inherit
+#
+# After push to the default branch, notifies the VPS using GitHub Actions OIDC
+# (audience = webhook URL). No GitHub Packages webhook and no extra secrets.
 
 name: Reusable docker build (single image)
 
@@ -26,14 +33,26 @@ on:
         required: false
         type: string
         default: .
+      notify:
+        required: false
+        type: boolean
+        default: true
+    secrets:
+      VIBED_WEBHOOK_SECRET:
+        required: false
 
 permissions:
   contents: read
   packages: write
+  id-token: write
 
 jobs:
   build-push:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+      id-token: write
     steps:
       - uses: actions/checkout@v4
       - uses: docker/setup-buildx-action@v3
@@ -59,3 +78,15 @@ jobs:
           labels: |
             org.opencontainers.image.revision=${{ github.sha }}
             org.opencontainers.image.source=https://github.com/${{ github.repository }}
+      - name: Notify VPS to pull
+        if: inputs.notify && github.event.repository.default_branch == github.ref_name
+        continue-on-error: true
+        env:
+          VIBED_WEBHOOK_SECRET: ${{ secrets.VIBED_WEBHOOK_SECRET }}
+        run: |
+          script=dist/notify-vps-pull.py
+          if [[ ! -f "$script" ]]; then
+            echo "notify-vps: dist/notify-vps-pull.py missing — re-package and commit dist/"
+            exit 0
+          fi
+          python3 "$script" --image "${{ inputs.image }}" --tag main

package/install-auto-update.sh

@@ -88,7 +88,23 @@ build_cron_minutes() {
 }
 CRON_MINUTES="$(build_cron_minutes "${CRON_OFFSET:-0}" "${INTERVAL_MIN:-30}")"
 CRON_SCHED="${CRON_MINUTES} * * * *"
-CRON_CMD="cd ${SCRIPT_DIR} && /bin/bash ${SCRIPT_DIR}/${UPDATE_SCRIPT} >/dev/null 2>&1"
+# Prefer shared update-agent queue when available; else direct update script.
+AGENT_ENQUEUE=""
+for cand in \
+  "${VIBED_HOME:-}/update-agent/enqueue.sh" \
+  "${HOME:-}/services/vibed-infra/update-agent/enqueue.sh" \
+  "/var/lib/vibed/update-agent/enqueue.sh"
+do
+  if [[ -n "$cand" && -x "$cand" ]]; then
+    AGENT_ENQUEUE="$cand"
+    break
+  fi
+done
+if [[ -n "$AGENT_ENQUEUE" ]]; then
+  CRON_CMD="cd ${SCRIPT_DIR} && UPDATE_REASON=cron /bin/bash ${AGENT_ENQUEUE} ${SCRIPT_DIR} >/dev/null 2>&1"
+else
+  CRON_CMD="cd ${SCRIPT_DIR} && /bin/bash ${SCRIPT_DIR}/${UPDATE_SCRIPT} >/dev/null 2>&1"
+fi
 dir_slug="$(basename "$SCRIPT_DIR" | tr -c 'A-Za-z0-9._-' '_')"
 CRON_D_FILE="/etc/cron.d/infra-${ROLE}-${dir_slug}"