vibed-infra 0.3.0 → 0.4.0

package/README.md

@@ -1,59 +1,63 @@
 # vibed-infra
 
-Product-agnostic VPS deployment packager: wget installer, Docker Compose layouts, digest-gated auto-update, TLS/nginx helpers, and CI workflow templates.
+Product-agnostic VPS deployment packager: **`package.sh`** turns four YAML configs into a committed **`dist/`** tree operators wget on the box — install scripts, generic start/update helpers, env examples, and gateway nginx.
 
 Published on npm as **`vibed-infra`**.
 
-## Install (npm)
+## Product maintainer flow
 
-```bash
-npm install vibed-infra
-# or in a monorepo sibling checkout:
-# "vibed-infra": "file:../vibed-infra"
-```
-
-Resolve the packager root from Node:
+Author **`templates/`** in your product repo:
 
-```bash
-node -e "console.log(require('path').dirname(require.resolve('vibed-infra/package.json')))"
-```
+| File | Purpose |
+|------|---------|
+| `vibed-infra-config.yml` | Product name, template refs, edge network, gateway `sites[]`, auto-update |
+| `api-config.yaml` | API image, port, opaque `config:` |
+| `ui-config.yaml` | UI image, port |
+| `nodes-config.yaml` | Worker image, opaque `config:` |
 
-Run install locally (product `packageconfig.yaml` + templates required):
+Then package and commit **`dist/`**:
 
 ```bash
-PACKAGER_RAW=/path/to/node_modules/vibed-infra \
-PACKAGECONFIG_URL=/path/to/product/deploy/packageconfig.yaml \
-PRODUCT_RAW=/path/to/product/deploy/templates \
-bash /path/to/node_modules/vibed-infra/install.sh --profile api
+./package.sh                  # exec vibed-infra packager → writes dist/
+git add dist && git commit && git push
 ```
 
-Or use the bin:
+Resolve the packager from npm:
 
 ```bash
-npx vibed-infra --profile api   # still needs PACKAGECONFIG_URL / PRODUCT_RAW (or packageconfig rawBase)
+node -e "console.log(require('path').dirname(require.resolve('vibed-infra/package.json')))"
 ```
 
-## Install (wget, operators)
+Example product: [`examples/vps-hello`](examples/vps-hello) (`app/`, `build-images.sh`, four YAMLs, committed `dist/`).
+
+## Operator flow (wget on VPS)
 
 ```bash
-wget -qO- https://raw.githubusercontent.com/naiemk/vibed-infra/main/install.sh | \
-  env INFRA_PROFILE=api \
-      PACKAGECONFIG_URL=https://raw.githubusercontent.com/ORG/REPO/main/deploy/packageconfig.yaml \
-      PRODUCT_RAW=https://raw.githubusercontent.com/ORG/REPO/main/deploy/templates \
-      bash
+wget -qO- https://raw.githubusercontent.com/ORG/REPO/main/dist/install-api.sh | bash
+wget -qO- .../dist/install-nodes.sh | bash
+wget -qO- .../dist/install-ui.sh | bash
+wget -qO- .../dist/install-gateway.sh | bash
+# edit each install dir .env, then ./start-*.sh
 ```
 
+| Profile | Role |
+|---------|------|
+| `api` | Backend on shared edge network |
+| `ui` | UI container (separate install) |
+| `nodes` | Worker compose on edge network |
+| `gateway` | HTTPS nginx only (API + UI must already run) |
+
 ## Layout
 
 | Path | Purpose |
 |------|---------|
+| `package.sh` / `lib/package.py` | Build product `dist/` from four YAMLs + generic templates |
 | `install.sh` | Single entrypoint (`INFRA_PROFILE` or `--profile`) |
-| `start.sh` / `update.sh` | Generic lifecycle in install dir |
 | `install-auto-update.sh` | Cron from profile flags |
-| `lib/` | fetch, env, tls, prompt, generate |
-| `templates/` | Generic compose/nginx skeletons |
-| `schema/packageconfig.md` | Schema reference |
-| `examples/vps-hello/` | Full product that installs API + worker + HTTPS gateway on a VPS |
+| `lib/` | fetch, env, tls, prompt, generate, product_config |
+| `templates/generic/` | Generic start/update/env/certs/gateway |
+| `schema/packageconfig.md` | Four-file product schema + compiled packageconfig |
+| `examples/vps-hello/` | Full example with dist e2e tests |
 | `skills/` | Cursor skills |
 | `github/workflows/` | Reusable GHCR build workflow |
 
@@ -62,11 +66,21 @@ wget -qO- https://raw.githubusercontent.com/naiemk/vibed-infra/main/install.sh |
 | Variable | Meaning |
 |----------|---------|
 | `PACKAGER_RAW` | Base URL or local path to this package root |
-| `PACKAGECONFIG_URL` | Product `packageconfig.yaml` (URL or path) |
-| `PRODUCT_RAW` | Product templates base (overrides `rawBase`; opaque to infra) |
-| `INFRA_PROFILE` | `api`, `nodes`, or `gateway` |
+| `PACKAGECONFIG_URL` | Product `packageconfig.yaml` or `dist/` URL (URL or path) |
+| `PRODUCT_RAW` | Product `dist/` base (templates + install scripts) |
+| `INFRA_PROFILE` | `api`, `ui`, `nodes`, or `gateway` |
 | `INSTALL_DIR` | Target directory (default `.`) |
 
+## Tests
+
+```bash
+npm test                        # package + dry-run all dist/install-*.sh
+npm run test:dist               # build images + e2e api, ui, nodes (sequential)
+./examples/vps-hello/test-dist.sh --profile api   # single profile
+```
+
+CI: `validate` job then parallel `dist-e2e` matrix (`api`, `ui`, `nodes`).
+
 ## Publishing (npm)
 
 Every **merge to `main`** runs [Publish npm](.github/workflows/publish.yml):
@@ -75,25 +89,10 @@ Every **merge to `main`** runs [Publish npm](.github/workflows/publish.yml):
 2. Commits `chore: release vibed-infra v…`, tags `v…`, pushes
 3. Runs `npm publish` (needs repo secret `NPM_TOKEN`)
 
-**Major versions** — set `version` in `package.json` yourself (e.g. `1.0.0`) and include **`[major]`** in the commit message (or PR merge commit). That publishes the pinned version with **no** automatic minor bump.
-
-Manual run: Actions → Publish npm → `workflow_dispatch` (`minor` / `major` / `none`).
+**Major versions** — set `version` in `package.json` yourself (e.g. `1.0.0`) and include **`[major]`** in the commit message. That publishes the pinned version with **no** automatic minor bump.
 
-Set repository secret **`NPM_TOKEN`** (npm automation token with publish access) before the first merge to `main`.
+Set repository secret **`NPM_TOKEN`** before the first merge to `main`.
 
-Ordinary PRs only run CI (`npm test`); they do **not** publish.
-
-## Example (VPS)
-
-[`examples/vps-hello`](examples/vps-hello) is a complete product that uses this packager: `packageconfig.yaml`, install wrappers, start/update scripts, and a tiny API/UI/worker you can build on the box.
-
-```bash
-git clone https://github.com/naiemk/vibed-infra.git
-cd vibed-infra
-./examples/vps-hello/scripts/build-images.sh
-mkdir -p ~/hello-vps/api && cd ~/hello-vps/api
-bash /path/to/vibed-infra/examples/vps-hello/install/install-api.sh
-# edit .env, then ./start-hello-api.sh
-```
+Ordinary PRs only run CI (`npm test` + dist e2e); they do **not** publish.
 
 Runbook: [`examples/vps-hello/README.md`](examples/vps-hello/README.md).

package/install-auto-update.sh

@@ -19,14 +19,17 @@ fi
 if [[ -z "$ROLE" && -n "$PROFILE" ]]; then
   case "$PROFILE" in
     api) ROLE=backend ;;
+    ui) ROLE=ui ;;
     nodes) ROLE=workers ;;
     gateway) ROLE=gateway ;;
   esac
 fi
 if [[ -z "$ROLE" ]]; then
-  if compgen -G "start-*-gateway.sh" >/dev/null || [[ -f docker-compose.gateway.yml ]]; then
+  if [[ -f start-gateway.sh ]] || compgen -G "start-*-gateway.sh" >/dev/null || [[ -f docker-compose.gateway.yml ]]; then
     ROLE=gateway
-  elif compgen -G "start-*-nodes.sh" >/dev/null || [[ -f docker-compose.workers.yml ]]; then
+  elif [[ -f start-ui.sh ]] || compgen -G "start-*-ui.sh" >/dev/null; then
+    ROLE=ui
+  elif [[ -f start-nodes.sh ]] || compgen -G "start-*-nodes.sh" >/dev/null || [[ -f docker-compose.workers.yml ]]; then
     ROLE=workers
   else
     ROLE=backend
@@ -47,6 +50,11 @@ case "$ROLE" in
     INTERVAL_MIN="${API_AUTO_UPDATE_INTERVAL_MIN:-30}"
     CRON_OFFSET=0
     ;;
+  ui)
+    if role_auto_update_on UI_AUTO_UPDATE; then cron_enabled=1; fi
+    INTERVAL_MIN="${UI_AUTO_UPDATE_INTERVAL_MIN:-20}"
+    CRON_OFFSET=15
+    ;;
   workers|nodes)
     ROLE=nodes
     if role_auto_update_on NODES_AUTO_UPDATE; then cron_enabled=1; fi

package/install.sh

@@ -18,6 +18,7 @@ while [[ $# -gt 0 ]]; do
       for c in "${_comps[@]}"; do
         case "$c" in
           app|backend|api) PROFILE=api; break ;;
+          ui) PROFILE=ui; break ;;
           workers|nodes) PROFILE=nodes; break ;;
           gateway) PROFILE=gateway; break ;;
         esac
@@ -26,7 +27,7 @@ while [[ $# -gt 0 ]]; do
       ;;
     --dest) DEST="$2"; shift 2 ;;
     -h|--help)
-      echo "usage: install.sh [--profile api|nodes|gateway] [--dest DIR]"
+      echo "usage: install.sh [--profile api|ui|nodes|gateway] [--dest DIR]"
       echo "env: PACKAGECONFIG_URL PRODUCT_RAW PACKAGER_RAW INFRA_PROFILE INSTALL_DIR"
       exit 0
       ;;
@@ -160,16 +161,24 @@ if [[ -n "$COMPOSE_MAINNET" ]]; then
   infra_write_template "${PRODUCT_RAW}/${COMPOSE_MAINNET}" "$DEST/${COMPOSE_MAINNET}" 0
 fi
 
-# Generic infra compose templates (optional USE_COMPOSE=1)
+# Generic infra compose templates — only when product did not supply compose in packageconfig.
 case "$ROLE" in
   backend)
-    infra_write_template "${PACKAGER_RAW}/templates/docker-compose.backend.yml" "$DEST/docker-compose.backend.yml" 0
+    if [[ -z "$COMPOSE_TPL" ]]; then
+      infra_write_template "${PACKAGER_RAW}/templates/docker-compose.backend.yml" "$DEST/docker-compose.backend.yml" 0
+    fi
+    ;;
+  ui)
     ;;
   workers)
-    infra_write_template "${PACKAGER_RAW}/templates/docker-compose.workers.yml" "$DEST/docker-compose.workers.yml" 0
+    if [[ -z "$COMPOSE_TPL" ]]; then
+      infra_write_template "${PACKAGER_RAW}/templates/docker-compose.workers.yml" "$DEST/docker-compose.workers.yml" 0
+    fi
     ;;
   gateway)
-    infra_write_template "${PACKAGER_RAW}/templates/docker-compose.gateway.yml" "$DEST/docker-compose.gateway.yml" 0
+    if [[ -z "$COMPOSE_MAINNET" && -z "$COMPOSE_TPL" ]]; then
+      infra_write_template "${PACKAGER_RAW}/templates/docker-compose.gateway.yml" "$DEST/docker-compose.gateway.yml" 0
+    fi
     ;;
 esac
 

package/lib/package.py

@@ -0,0 +1,259 @@
+#!/usr/bin/env python3
+"""Build product dist/ from templates/*.yaml — operators wget install-*.sh from dist."""
+from __future__ import annotations
+
+import argparse
+import shutil
+import stat
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from product_config import (  # noqa: E402
+    compile_packageconfig,
+    dump_yaml,
+    load_product,
+    write_app_config,
+)
+
+
+def _chmod_x(path: Path) -> None:
+    path.chmod(path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
+
+
+def _write(path: Path, text: str, executable: bool = False) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(text, encoding="utf-8")
+    if executable:
+        _chmod_x(path)
+
+
+def _copy_generic(packager: Path, out: Path, name: str) -> None:
+    src = packager / "templates" / "generic" / name
+    if not src.is_file():
+        raise FileNotFoundError(f"missing generic template {src}")
+    shutil.copy2(src, out / name)
+    _chmod_x(out / name)
+
+
+def _env_api(meta: dict, conf: dict, api: dict) -> str:
+    au = conf["profiles"]["api"]["autoUpdate"]
+    return f"""# Generated by vibed-infra package — edit .env on the VPS, not in git dist.
+API_TOKEN=change-me-api-token
+DOCKER_NETWORK={meta['network']}
+DOCKER_NAME={meta['apiContainer']}
+BACKEND_IMAGE={conf['images']['backend']}
+HOST_PORT={meta['apiPort']}
+DATA_DIR=./data
+CONFIG_FILE=api-app.yaml
+API_MEMORY_LIMIT=256m
+{au['flag']}=0
+{au['intervalEnv']}=30
+{au['stopTimeoutEnv']}=60
+"""
+
+
+def _env_ui(meta: dict, conf: dict, ui: dict) -> str:
+    au = conf["profiles"]["ui"]["autoUpdate"]
+    return f"""# Generated by vibed-infra package
+DOCKER_NETWORK={meta['network']}
+UI_NAME={meta['uiContainer']}
+UI_IMAGE={conf['images']['ui']}
+UI_MEMORY_LIMIT=32m
+{au['flag']}=0
+{au['intervalEnv']}=20
+{au['stopTimeoutEnv']}=30
+"""
+
+
+def _env_nodes(meta: dict, conf: dict, nodes: dict) -> str:
+    au = conf["profiles"]["nodes"]["autoUpdate"]
+    api_host = meta["apiContainer"]
+    return f"""# Generated by vibed-infra package
+API_URL=http://{api_host}:{meta['apiPort']}
+API_TOKEN=change-me-api-token
+DOCKER_NETWORK={meta['network']}
+WORKER_IMAGE={conf['images']['worker']}
+WORKER_CONTAINER_NAME={meta['workerContainer']}
+WORKER_ROLE=heartbeat
+INTERVAL_SEC=60
+WORKER_MEMORY_LIMIT=64m
+{au['flag']}=0
+{au['intervalEnv']}=30
+{au['stopTimeoutEnv']}=30
+"""
+
+
+def _env_gateway(meta: dict, conf: dict, sites: list) -> str:
+    au = conf["profiles"]["gateway"]["autoUpdate"]
+    host = sites[0]["host"] if sites else "example.com"
+    cert_dir = sites[0].get("tlsCertDir", f"/etc/letsencrypt/live/{host}") if sites else "/etc/letsencrypt/live/example.com"
+    flags = au.get("flags") or ["GATEWAY_AUTO_UPDATE"]
+    flag_lines = "\n".join(f"{f}=0" for f in flags)
+    return f"""# Generated by vibed-infra package
+DOCKER_NETWORK={meta['network']}
+GATEWAY_NAME={meta['gatewayContainer']}
+NGINX_IMAGE={conf['images']['nginx']}
+TLS_FULLCHAIN={cert_dir}/fullchain.pem
+TLS_PRIVKEY={cert_dir}/privkey.pem
+CERTBOT_WWW=./certbot-www
+HTTP_PORT=80
+HTTPS_PORT=443
+GATEWAY_MEMORY_LIMIT=64m
+{flag_lines}
+{au.get('intervalEnv', 'GATEWAY_AUTO_UPDATE_INTERVAL_MIN')}=20
+{au.get('stopTimeoutEnv', 'GATEWAY_STOP_TIMEOUT')}=30
+"""
+
+
+def _compose_workers(meta: dict, conf: dict) -> str:
+    return f"""# Generated by vibed-infra package
+networks:
+  edge:
+    external: true
+    name: ${{DOCKER_NETWORK:-{meta['network']}}}
+
+services:
+  worker:
+    image: ${{WORKER_IMAGE:-{conf['images']['worker']}}}
+    container_name: ${{WORKER_CONTAINER_NAME:-{meta['workerContainer']}}}
+    restart: unless-stopped
+    mem_limit: ${{WORKER_MEMORY_LIMIT:-64m}}
+    env_file: .env
+    environment:
+      WORKER_ROLE: ${{WORKER_ROLE:-heartbeat}}
+      WORKER_CONFIG: /config/workers.yaml
+      API_URL: ${{API_URL:-http://{meta['apiContainer']}:{meta['apiPort']}}}
+      INTERVAL_SEC: ${{INTERVAL_SEC:-60}}
+    volumes:
+      - ./logs:/data/logs
+      - ./nodes-workers.yaml:/config/workers.yaml:ro
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    networks:
+      - edge
+"""
+
+
+def _install_wrapper(profile: str, packager_raw: str) -> str:
+    return f"""#!/usr/bin/env bash
+# wget install — profile {profile}. Generated by vibed-infra package.
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${{BASH_SOURCE[0]:-$0}}")" && pwd)"
+PACKAGER_RAW="${{PACKAGER_RAW:-{packager_raw}}}"
+export PACKAGECONFIG_URL="${{PACKAGECONFIG_URL:-$SCRIPT_DIR/packageconfig.yaml}}"
+export PRODUCT_RAW="${{PRODUCT_RAW:-$SCRIPT_DIR}}"
+export INFRA_PROFILE={profile}
+export INSTALL_DIR="${{INSTALL_DIR:-.}}"
+if [[ "$PACKAGER_RAW" =~ ^/ ]]; then
+  exec bash "$PACKAGER_RAW/install.sh" --profile {profile}
+fi
+if command -v curl >/dev/null 2>&1; then
+  exec bash <(curl -fsSL "${{PACKAGER_RAW}}/install.sh") --profile {profile}
+else
+  exec bash <(wget -qO- "${{PACKAGER_RAW}}/install.sh") --profile {profile}
+fi
+"""
+
+
+def build_dist(product_dir: Path, out_dir: Path, packager_root: Path) -> None:
+    product_dir = product_dir.resolve()
+    out_dir = out_dir.resolve()
+    packager_root = packager_root.resolve()
+
+    if out_dir.exists():
+        shutil.rmtree(out_dir)
+    out_dir.mkdir(parents=True)
+
+    product = load_product(product_dir)
+    tpl = product["templates_dir"]
+    raw_base = str(out_dir)  # local dist path; operators override PRODUCT_RAW via wget URL
+
+    conf = compile_packageconfig(
+        product,
+        raw_base=raw_base,
+        packager_raw=str(packager_root),
+    )
+    meta = conf["_meta"]
+    comps = product["components"]
+    sites = conf["profiles"]["gateway"].get("sites") or []
+
+    # Copy source configs into dist
+    shutil.copy2(tpl / "vibed-infra-config.yml", out_dir / "vibed-infra-config.yml")
+    refs = product["infra"].get("templates") or {}
+    for dst, key in (
+        ("api-config.yaml", "api"),
+        ("ui-config.yaml", "ui"),
+        ("nodes-config.yaml", "nodes"),
+    ):
+        src_name = refs.get(key) or f"{key}-config.yaml"
+        shutil.copy2(tpl / src_name, out_dir / dst)
+
+    _write(out_dir / "packageconfig.yaml", dump_yaml(conf))
+
+    write_app_config(comps["api"], out_dir / "api-app.yaml")
+    write_app_config(comps["nodes"], out_dir / "nodes-workers.yaml")
+
+    _write(out_dir / ".env.api.example", _env_api(meta, conf, comps["api"]))
+    _write(out_dir / ".env.ui.example", _env_ui(meta, conf, comps["ui"]))
+    _write(out_dir / ".env.nodes.example", _env_nodes(meta, conf, comps["nodes"]))
+    _write(out_dir / ".env.gateway.example", _env_gateway(meta, conf, sites))
+    _write(out_dir / "docker-compose.workers.yml", _compose_workers(meta, conf))
+
+    gw_dir = out_dir / "gateway"
+    gw_dir.mkdir()
+    shutil.copy2(packager_root / "templates" / "generic" / "gateway" / "nginx.conf", gw_dir / "nginx.conf")
+
+    for script in (
+        "start-api.sh",
+        "start-ui.sh",
+        "start-nodes.sh",
+        "start-gateway.sh",
+        "update-api.sh",
+        "update-ui.sh",
+        "update-nodes.sh",
+        "update-gateway.sh",
+        "gen-dev-certs.sh",
+    ):
+        _copy_generic(packager_root, out_dir, script)
+
+    packager_raw_url = "https://raw.githubusercontent.com/naiemk/vibed-infra/main"
+    for profile in ("api", "ui", "nodes", "gateway"):
+        name = f"install-{profile}.sh"
+        _write(out_dir / name, _install_wrapper(profile, packager_raw_url), executable=True)
+
+    readme = f"""# dist — generated by vibed-infra package
+
+Do not edit by hand. Run `../package.sh` from the product root and commit.
+
+## VPS install
+
+```bash
+wget -qO- https://raw.githubusercontent.com/ORG/REPO/main/dist/install-api.sh | bash
+wget -qO- https://raw.githubusercontent.com/ORG/REPO/main/dist/install-ui.sh | bash
+wget -qO- https://raw.githubusercontent.com/ORG/REPO/main/dist/install-nodes.sh | bash
+wget -qO- https://raw.githubusercontent.com/ORG/REPO/main/dist/install-gateway.sh | bash
+```
+
+Network: `{meta['network']}`. API container: `{meta['apiContainer']}`.
+"""
+    _write(out_dir / "README.md", readme)
+    print(f"packaged: {out_dir}")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description="Build product dist/ for VPS wget install")
+    ap.add_argument("--product", required=True, help="Product root (contains templates/)")
+    ap.add_argument("--out", required=True, help="Output dist directory")
+    ap.add_argument(
+        "--packager",
+        default=str(Path(__file__).resolve().parent.parent),
+        help="vibed-infra root",
+    )
+    args = ap.parse_args()
+    build_dist(Path(args.product), Path(args.out), Path(args.packager))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())