@aifabrix/server-setup 1.4.0 → 1.5.2

package/README.md

@@ -42,72 +42,93 @@ Do the steps in [What you must do before running af-server](#what-you-must-do-be
 
 Complete the [manual prerequisites](#what-you-must-do-before-running-af-server) first (DNS, SSL directory, certificate and key on the server).
 
-### Option A: From your PC (remote install over SSH)
+**Flow summary:** Only **step 1** runs over SSH from your PC. Steps **5** and **7** run **on the server** after you log in, so errors and output are visible directly there.
 
-1. **Install CLIs** (one-time):
+| Step | Where     | Action |
+| ---- | --------- | ------ |
+| 1    | From PC   | `af-server install-init $SSH` — only command over SSH; installs on server: SSH (if needed), Node 18+, npm, and `af-server` CLI. |
+| 2    | One-time  | Log in to the server once (e.g. with password) to approve passwordless SSH; then from PC: `af-server ssh-cert install $SSH`. |
+| 3    | From PC   | Copy SSL certificate and key to the server (see [SSL directory and certificates](#ssl-directory-and-certificates)); example commands below. |
+| 4    | From PC   | Log in to the server via SSH (passwordless). |
+| 5    | **On server** | `sudo af-server install` — install all services (Docker, nginx package, Mutagen, cron, data dir); no builder-server container yet. |
+| 6    | On server | Get the builder-server image (e.g. `az login` / `docker pull`). |
+| 7    | **On server** | `sudo af-server install-server --dev-domain $DOMAIN` — nginx vhost, builder-server container, Docker TLS. |
+| 8    | —         | Done. |
 
-   ```bash
-   npm install -g @aifabrix/builder
-   npm install -g @aifabrix/server-setup
-   ```
+### Step 1: Bootstrap the server (from PC)
 
-2. **Set your server target and local path** (replace with your host, user, and the folder on your PC where your cert and key files are):
+Set your target and run the only command that uses SSH from your PC:
 
-   ```bash
-   export SSH=serveradmin@builder02.aifabrix.dev
-   export HDD=/c/git/esystemsdev/aifabrix-setup/certificates
-   export DOMAIN=builder02.aifabrix.dev
-   ```
+```bash
+export SSH=serveradmin@builder02.aifabrix.dev
+export DOMAIN=builder02.aifabrix.dev
+af-server install-init $SSH
+```
 
-3. **If the server has no SSH yet** (e.g. fresh VM with console only), enable SSH first (you need another way to run one command, e.g. cloud console):
+This installs on the server: openssh-server (if needed), Node 18+, npm, and `@aifabrix/builder` + `@aifabrix/server-setup` so `af-server` is available there. No Docker, nginx, or builder-server yet.
 
-   ```bash
-   af-server install-ssh $SSH
-   ```
+### Step 2: Passwordless SSH (from PC)
 
-4. **Add your SSH key** for passwordless auth (so install can run over SSH):
+Log in to the server once (e.g. with password) to accept the host key and/or approve auth. Then from your PC:
 
-   ```bash
-   af-server ssh-cert install $SSH
-   ```
+```bash
+af-server ssh-cert install $SSH
+```
 
-5. **Put SSL cert and key on the server** in `/opt/aifabrix/ssl` as `wildcard.crt` and `wildcard.key`. See [SSL directory and certificates](#ssl-directory-and-certificates).
+### Step 3: Copy SSL (from PC)
 
-   Example: copy from your PC to the server (create the directory on the server first, then copy and fix key permissions):
+Put `wildcard.crt` and `wildcard.key` in `/opt/aifabrix/ssl` on the server. Example (replace `$HDD` with the folder on your PC that has the cert and key):
 
-   ```bash
-   ssh $SSH "sudo mkdir -p /opt/aifabrix/ssl"
-   scp $HDD/wildcard.crt $SSH:/tmp/wildcard.crt
-   scp $HDD/wildcard.key $SSH:/tmp/wildcard.key
-   ssh $SSH "sudo mv /tmp/wildcard.crt /tmp/wildcard.key /opt/aifabrix/ssl/ && sudo chmod 600 /opt/aifabrix/ssl/wildcard.key"
-   ```
+```bash
+export HDD=/workspace/aifabrix-setup/certificates
+ssh $SSH "sudo mkdir -p /opt/aifabrix/ssl"
+scp $HDD/wildcard.crt $SSH:/tmp/wildcard.crt
+scp $HDD/wildcard.key $SSH:/tmp/wildcard.key
+ssh $SSH "sudo mv /tmp/wildcard.crt /tmp/wildcard.key /opt/aifabrix/ssl/ && sudo chmod 600 /opt/aifabrix/ssl/wildcard.key"
+```
 
-6. **Run install** (uses sudo on the server):
+### Step 4: Log in to the server
 
-   ```bash
-   af-server install $SSH
-   ```
+```bash
+ssh $SSH
+```
 
-   To use a different domain or SSL path:
+### Step 5: Install services (on server)
 
-   ```bash
-   af-server install $SSH --dev-domain $DOMAIN --ssl-dir /opt/aifabrix/ssl
-   ```
+```bash
+sudo af-server install
+```
 
-   - Space between `$DOMAIN` and `--ssl-dir` is required.
-   - **Git Bash (Windows):** quote server paths so the shell doesn't expand `/opt` (e.g. `--ssl-dir '/opt/aifabrix/ssl'`, `--data-dir '/opt/aifabrix/builder-server/data'`).
+This installs Docker, nginx (package only), Mutagen, data dir, apply-dev-users script and cron. It does **not** write the builder nginx vhost or start the builder-server container.
 
-### Option B: On the server itself (Ubuntu, no target)
+### Step 6: Get the builder-server image (on server)
 
-If you are already on the server (e.g. over console or SSH), run install locally. Prerequisites (DNS, SSL dir with `wildcard.crt` and `wildcard.key`) must still be done first.
+The image is not on a public registry. Use your platform’s method (e.g. Azure CLI or Docker login), then pull. Example:
 
 ```bash
-npm install -g @aifabrix/builder
-npm install -g @aifabrix/server-setup
-sudo af-server install
+az login
+az acr login --name aifabrixdevacr
+docker pull aifabrixdevacr.azurecr.io/aifabrix/builder-server:latest
 ```
 
-After install, your builder-server is up. Use the **AI Fabrix Builder CLI** for all usage (users, secrets, certs, etc.)—see [Builder documentation](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/developer-isolation.md).
+Or with Docker login (username/password from your registry):
+
+```bash
+docker login <registry> -u <user> -p <password>
+docker pull <registry>/aifabrix/builder-server:latest
+```
+
+### Step 7: Install server (nginx vhost + container) (on server)
+
+```bash
+sudo af-server install-server --dev-domain $DOMAIN
+```
+
+Use the same domain as your DNS and SSL. Optional: `--ssl-dir /opt/aifabrix/ssl`, `--data-dir /opt/aifabrix/builder-server/data`, `--builder-port 3000`.
+
+### Step 8: Done
+
+Your builder-server is up. Use the **AI Fabrix Builder CLI** for users, secrets, certs, etc.—see [Builder documentation](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/developer-isolation.md).
 
 ---
 
@@ -115,7 +136,9 @@ After install, your builder-server is up. Use the **AI Fabrix Builder CLI** for
 
 | Command | Description |
 | -------- | ----------- |
-| `af-server install [ user@host ] [ -d DATA_DIR ] [ --dev-domain DOMAIN ] [ --ssl-dir PATH ] [ -i SSH_KEY ]` | Install or update: Docker, nginx (builder vhost always updated from template), SSL proxy, sync user, cron. Re-run to apply latest config. |
+| `af-server install-init <user@host> [ -i SSH_KEY ]` | **From PC only.** One-time bootstrap over SSH: install on server SSH (if needed), Node 18+, npm, and af-server CLI. |
+| `af-server install [ user@host ] [ -d DATA_DIR ] [ --dev-domain DOMAIN ] [ --ssl-dir PATH ] [ -i SSH_KEY ]` | **Run on server** (omit target): `sudo af-server install`. Infra only: Docker, nginx pkg, Mutagen, data dir, cron. No builder vhost or container. With target: same infra over SSH. |
+| `af-server install-server --dev-domain DOMAIN [ -d DATA_DIR ] [ --ssl-dir PATH ] [ --builder-port PORT ]` | **On server only.** Nginx vhost, builder-server container, Docker TLS. Run after `sudo af-server install`. |
 | `af-server backup [ user@host ] [ -d DATA_DIR ] [ -o output.zip ] [ -i SSH_KEY ]` | On-demand backup (config + DB + keys). |
 | `af-server backup [ user@host ] --schedule [ --backup-dir PATH ] [ --keep-days N ] [ -i SSH_KEY ]` | Cron backup (daily 02:00, keep last N, default 7). |
 | `af-server restore backup.zip [ user@host ] [ -d DATA_DIR ] [ --force ] [ -i SSH_KEY ]` | Restore backup to DATA_DIR. |
@@ -171,18 +194,23 @@ If you SSH as a non-root user, that user must be able to sudo. The script will a
 
 ---
 
-## High level: what the install does on your Ubuntu server
+## High level: what install vs install-server does
+
+**`af-server install`** (step 5 — run on the server) does **infra only**:
+
+- **System** — `apt update` and `apt upgrade`; optional hostname if `SETUP_HOSTNAME` is set.
+- **Docker** — Installs Docker if missing; enables and starts it.
+- **Admin user** — Adds the admin user (default `serveradmin`) to the `docker` group and grants passwordless sudo.
+- **Nginx** — Installs the nginx **package** only; enables and starts it. Does **not** write the builder vhost yet.
+- **Data dir** — Creates the data directory (default `/opt/aifabrix/builder-server/data`), workspace and ssh-keys subdirs, ownership for the container.
+- **Apply-dev-users** — Installs the script and cron job (every 2 minutes) that sync per-developer OS users from builder-server state.
+- **Mutagen** — Downloads and installs Mutagen; systemd service and daemon.
+- **Optional** — If `INSTALL_PORTAINER=1`, installs the Portainer container.
 
-When you run `af-server install`, the script (as root) does the following on the server:
+**`af-server install-server`** (step 7 — run on the server) does the **server phase**:
 
-- **System** — `apt update` and `apt upgrade`; optional hostname change if `SETUP_HOSTNAME` is set.
-- **Docker** — Installs Docker if missing; enables and starts the Docker service.
-- **Admin user** — Adds the admin user (default `serveradmin`) to the `docker` group and grants passwordless sudo (`/etc/sudoers.d/<admin>`).
-- **Nginx** — Installs nginx if missing; enables and starts it. Generates a site config for your domain that proxies HTTPS to the builder-server container (using `wildcard.crt` and `wildcard.key` from your SSL dir).
-- **Builder-server data dir** — Creates the data directory (default `/opt/aifabrix/builder-server/data`), sets ownership for the container. Starts the `builder-server` container if the image already exists on the server; otherwise prints how to build and run it (you get the image from the AI Fabrix platform).
-- **Sync user** — Creates a system user (default `aifabrix-sync`) for Mutagen SSH sync; home under the data dir; creates `.ssh` and `authorized_keys`.
-- **Cron job** — Installs a cron job (every 2 minutes) that copies the managed `authorized_keys` file into the sync user’s `.ssh/authorized_keys`.
-- **Mutagen** — Downloads and installs the Mutagen binary; creates a systemd service and starts the daemon.
-- **Optional** — If `INSTALL_PORTAINER=1`, installs the Portainer container. If Docker TLS is not skipped, writes `/etc/docker/daemon.json`; you can use the **same certificate** from `/opt/aifabrix/ssl` (e.g. symlink or copy `wildcard.crt` and `wildcard.key` to the paths Docker expects: `/etc/docker/server-cert.pem`, `/etc/docker/server-key.pem`, and if needed `ca.pem` for the CA), or provide separate Docker TLS certs.
+- **Nginx vhost** — Writes the builder site config from template (domain, SSL dir, proxy to builder-server), reloads nginx.
+- **Builder-server container** — Creates data dir (if needed), starts the builder-server container (if the image is present).
+- **Docker TLS** — Copies certs and configures `/etc/docker/daemon.json` for TLS (using website cert and builder-server CA).
 
-Result: your Ubuntu server has Docker, nginx (HTTPS for your domain), the builder-server container (if image present), the sync user and key sync, and Mutagen—ready for the Builder CLI to use.
+Result: after both steps, the server has Docker, nginx (HTTPS for your domain), the builder-server container (if image present), and Mutagen—ready for the Builder CLI to use.

package/assets/setup-dev-server-no-node.sh

@@ -3,9 +3,12 @@
 # Run via af-server install user@host. REPO_ROOT must be set to the dir containing builder/builder-server/nginx-builder-server.conf.template.
 # On empty server: installs Docker, nginx, admin user, optional Portainer/Mutagen; then updates nginx config and ensures builder-server container.
 # Optional env: DEV_DOMAIN, SSL_DIR, DATA_DIR, SETUP_ADMIN_USER, SYNC_USER, BUILDER_SERVER_PORT, NGINX_CONF_DIR, SETUP_HOSTNAME, INSTALL_PORTAINER=1, SKIP_DOCKER_TLS=1.
+# INSTALL_PHASE: infra = Docker, nginx pkg, Mutagen, data dir, apply-dev-users (no vhost, no container, no Docker TLS). server = nginx vhost, builder-server container, Docker TLS. full = both.
 
 set -e
 export DEBIAN_FRONTEND=noninteractive
+# Cache sudo so one password covers the whole script (avoids repeated "[sudo] password for ...")
+sudo -v
 
 REPO_ROOT="${REPO_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"
 DATA_DIR="${DATA_DIR:-/opt/aifabrix/builder-server/data}"
@@ -15,6 +18,7 @@ SETUP_ADMIN_USER="${SETUP_ADMIN_USER:-serveradmin}"
 SYNC_USER="${SYNC_USER:-aifabrix-sync}"
 BUILDER_SERVER_PORT="${BUILDER_SERVER_PORT:-3000}"
 NGINX_CONF_DIR="${NGINX_CONF_DIR:-/etc/nginx/conf.d}"
+INSTALL_PHASE="${INSTALL_PHASE:-full}"
 
 # Sanitize user-controlled env to prevent path/command injection
 sanitize_domain() {
@@ -52,6 +56,9 @@ require_sudo() {
 }
 require_sudo
 
+# --- Infra phase: hostname, system, Docker, nginx package, Mutagen, data dir, apply-dev-users ---
+if [ "$INSTALL_PHASE" = "infra" ] || [ "$INSTALL_PHASE" = "full" ]; then
+
 # --- Hostname (optional) ---
 if [ -n "$SETUP_HOSTNAME" ]; then
   current=$(hostname 2>/dev/null || true)
@@ -76,6 +83,13 @@ apt-get install -y \
 systemctl enable ssh 2>/dev/null || systemctl enable sshd 2>/dev/null || true
 systemctl start ssh 2>/dev/null || systemctl start sshd 2>/dev/null || true
 
+# --- Azure CLI (az) ---
+# Manual one-liner (sudo must apply to bash): curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
+echo "=== Azure CLI (optional - remove block if not needed) ==="
+if ! command -v az >/dev/null 2>&1; then
+  curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
+fi
+
 # --- Docker ---
 if ! command -v docker >/dev/null 2>&1; then
   apt-get install -y docker.io
@@ -112,30 +126,12 @@ if [ "$INSTALL_PORTAINER" = "1" ] && ! docker ps -a --format '{{.Names}}' 2>/dev
     portainer/portainer-ce:latest 2>/dev/null || true
 fi
 
-# --- Nginx ---
+# --- Nginx (package only in infra; vhost + reload in server phase) ---
 if ! command -v nginx >/dev/null 2>&1; then
   apt-get install -y nginx
   systemctl enable nginx
   systemctl start nginx
 fi
-NGINX_CONF="$NGINX_CONF_DIR/$DEV_DOMAIN.conf"
-NGINX_TEMPLATE="$REPO_ROOT/builder/builder-server/nginx-builder-server.conf.template"
-# Always update builder vhost from template so re-running install applies latest config.
-if [ -f "$NGINX_TEMPLATE" ]; then
-  sed -e "s|DEV_DOMAIN_PLACEHOLDER|$DEV_DOMAIN|g" \
-      -e "s|SSL_DIR_PLACEHOLDER|$SSL_DIR|g" \
-      -e "s|BUILDER_SERVER_PORT_PLACEHOLDER|$BUILDER_SERVER_PORT|g" \
-      -e "s|DATA_DIR_PLACEHOLDER|$DATA_DIR|g" \
-      "$NGINX_TEMPLATE" > "$NGINX_CONF"
-elif [ ! -f "$NGINX_CONF" ]; then
-  echo "Warning: SSL prereqs required. Place $NGINX_CONF (see SETUP.md) and ensure $SSL_DIR/wildcard.crt and $SSL_DIR/wildcard.key exist."
-fi
-if command -v nginx >/dev/null 2>&1 && nginx -t 2>/dev/null; then
-  systemctl reload nginx
-elif [ -f "$NGINX_TEMPLATE" ] && command -v nginx >/dev/null 2>&1; then
-  echo "Warning: nginx -t failed (config was still written to $NGINX_CONF). Nginx was NOT reloaded."
-  nginx -t 2>&1 || true
-fi
 
 # --- Mutagen ---
 if ! command -v mutagen >/dev/null 2>&1; then
@@ -168,6 +164,112 @@ MUTAGEN_EOF
   fi
 fi
 
+# --- Data dir and workspace/ssh-keys (for apply-dev-users) ---
+mkdir -p "$DATA_DIR" "${DATA_DIR}/workspace" "${DATA_DIR}/ssh-keys"
+chown -R 1001:65533 "$DATA_DIR"
+chmod 755 "$DATA_DIR"
+DATA_DIR_ABS=$(cd "$DATA_DIR" && pwd)
+
+# --- Host job: apply per-developer OS users from builder-server state ---
+APPLY_DEV_USERS_SCRIPT="/usr/local/bin/aifabrix-apply-dev-users.sh"
+SETUP_ASSETS_DIR="$(cd "$(dirname "$0")" && pwd)"
+if [ ! -f "$APPLY_DEV_USERS_SCRIPT" ]; then
+  if [ -f "$SETUP_ASSETS_DIR/aifabrix-apply-dev-users.sh" ]; then
+    cp "$SETUP_ASSETS_DIR/aifabrix-apply-dev-users.sh" "$APPLY_DEV_USERS_SCRIPT"
+  else
+    # Inline fallback when run from builder/builder-server context
+    cat > "$APPLY_DEV_USERS_SCRIPT" << 'APPLY_DEV_EOF'
+#!/bin/sh
+DATA_DIR="${DATA_DIR:-/opt/aifabrix/builder-server/data}"
+SSH_KEYS_DIR="${DATA_DIR}/ssh-keys"
+PENDING_REMOVALS="${DATA_DIR}/pending-removals"
+[ ! -d "$DATA_DIR" ] && exit 0
+if [ -f "$PENDING_REMOVALS" ]; then
+  while IFS= read -r dev_id || [ -n "$dev_id" ]; do
+    dev_id=$(echo "$dev_id" | tr -d '\r\n ')
+    [ -z "$dev_id" ] && continue
+    user_name="dev${dev_id}"
+    getent passwd "$user_name" >/dev/null 2>&1 && ( userdel -r "$user_name" 2>/dev/null || userdel "$user_name" 2>/dev/null || true )
+  done < "$PENDING_REMOVALS"
+  : > "$PENDING_REMOVALS"
+fi
+for key_file in "${SSH_KEYS_DIR}"/*/authorized_keys; do
+  [ -f "$key_file" ] || continue
+  dev_id=$(dirname "$key_file" | xargs basename)
+  [ -z "$dev_id" ] && continue
+  user_name="dev${dev_id}"
+  home_dir="/home/${user_name}"
+  workspace_target="${DATA_DIR}/workspace/${user_name}"
+  getent passwd "$user_name" >/dev/null 2>&1 || useradd -m -s /bin/bash "$user_name"
+  mkdir -p "${home_dir}/.ssh"
+  cp "$key_file" "${home_dir}/.ssh/authorized_keys"
+  chown -R "${user_name}:${user_name}" "${home_dir}/.ssh"
+  chmod 700 "${home_dir}/.ssh"
+  chmod 600 "${home_dir}/.ssh/authorized_keys"
+  config_file="${home_dir}/.aifabrix/config.yaml"
+  if [ ! -f "$config_file" ]; then
+    mkdir -p "${home_dir}/.aifabrix"
+    hostname_val=$(hostname -f 2>/dev/null || hostname 2>/dev/null || echo "localhost")
+    printf 'user-mutagen-folder: %s\nsecrets-encryption: ""\naifabrix-secrets: ""\naifabrix-env-config: ""\nremote-server: "http://localhost:3000"\ndocker-endpoint: "tcp://%s:2376"\nsync-ssh-user: "%s"\nsync-ssh-host: "%s"\n' "$workspace_target" "$hostname_val" "$user_name" "$hostname_val" > "$config_file"
+    chown -R "${user_name}:${user_name}" "${home_dir}/.aifabrix"
+    chmod 700 "${home_dir}/.aifabrix"
+    chmod 600 "$config_file"
+  fi
+  mkdir -p "$workspace_target"
+  chown -R "${user_name}:${user_name}" "$workspace_target"
+  [ -L "${home_dir}/workspace" ] && [ "$(readlink -f "${home_dir}/workspace" 2>/dev/null)" != "$(readlink -f "$workspace_target" 2>/dev/null)" ] && rm -f "${home_dir}/workspace"
+  [ ! -e "${home_dir}/workspace" ] && ln -s "$workspace_target" "${home_dir}/workspace" && chown -h "${user_name}:${user_name}" "${home_dir}/workspace"
+  profile="${home_dir}/.profile"
+  [ ! -f "$profile" ] && touch "$profile" && chown "${user_name}:${user_name}" "$profile"
+  grep -q 'cd.*workspace' "$profile" 2>/dev/null || echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$profile"
+  bashrc="${home_dir}/.bashrc"
+  [ ! -f "$bashrc" ] && touch "$bashrc" && chown "${user_name}:${user_name}" "$bashrc"
+  grep -q 'cd.*workspace' "$bashrc" 2>/dev/null || echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$bashrc"
+done
+APPLY_DEV_EOF
+  fi
+  chmod 755 "$APPLY_DEV_USERS_SCRIPT"
+fi
+# Defaults for apply script (aifabrix-secrets, aifabrix-env-config in generated config); match builder-server env.template
+APPLY_DEFAULTS="$DATA_DIR_ABS/apply-dev-users-defaults"
+if [ ! -f "$APPLY_DEFAULTS" ]; then
+  {
+    echo "# Sourced by cron before aifabrix-apply-dev-users.sh; override to match builder-server .env"
+    echo 'export AIFABRIX_SECRETS="${AIFABRIX_SECRETS:-/aifabrix-miso/builder/secrets.local.yaml}"'
+    echo 'export AIFABRIX_ENV_CONFIG="${AIFABRIX_ENV_CONFIG:-aifabrix-miso/builder/env-config.yaml}"'
+  } > "$APPLY_DEFAULTS"
+  chmod 644 "$APPLY_DEFAULTS"
+fi
+if [ -d /etc/cron.d ] && [ ! -f /etc/cron.d/aifabrix-apply-dev-users ]; then
+  echo "*/2 * * * * root /bin/sh -c '. $APPLY_DEFAULTS 2>/dev/null; export DATA_DIR=$DATA_DIR_ABS; exec $APPLY_DEV_USERS_SCRIPT'" > /etc/cron.d/aifabrix-apply-dev-users
+  chmod 644 /etc/cron.d/aifabrix-apply-dev-users
+fi
+
+fi
+# --- End infra phase ---
+
+# --- Server phase: nginx vhost, builder-server container, Docker TLS ---
+if [ "$INSTALL_PHASE" = "server" ] || [ "$INSTALL_PHASE" = "full" ]; then
+
+# --- Nginx builder vhost from template ---
+NGINX_CONF="$NGINX_CONF_DIR/$DEV_DOMAIN.conf"
+NGINX_TEMPLATE="$REPO_ROOT/builder/builder-server/nginx-builder-server.conf.template"
+if [ -f "$NGINX_TEMPLATE" ]; then
+  sed -e "s|DEV_DOMAIN_PLACEHOLDER|$DEV_DOMAIN|g" \
+      -e "s|SSL_DIR_PLACEHOLDER|$SSL_DIR|g" \
+      -e "s|BUILDER_SERVER_PORT_PLACEHOLDER|$BUILDER_SERVER_PORT|g" \
+      -e "s|DATA_DIR_PLACEHOLDER|$DATA_DIR|g" \
+      "$NGINX_TEMPLATE" > "$NGINX_CONF"
+elif [ ! -f "$NGINX_CONF" ]; then
+  echo "Warning: SSL prereqs required. Place $NGINX_CONF (see SETUP.md) and ensure $SSL_DIR/wildcard.crt and $SSL_DIR/wildcard.key exist."
+fi
+if command -v nginx >/dev/null 2>&1 && nginx -t 2>/dev/null; then
+  systemctl reload nginx
+elif [ -f "$NGINX_TEMPLATE" ] && command -v nginx >/dev/null 2>&1; then
+  echo "Warning: nginx -t failed (config was still written to $NGINX_CONF). Nginx was NOT reloaded."
+  nginx -t 2>&1 || true
+fi
+
 # --- Builder-server data dir and container ---
 # Paths match AI Fabrix Builder: aifabrix build + resolve use DATA_DIR=/mnt/data in container; host DATA_DIR (e.g. /opt/aifabrix/builder-server/data) is the HDD/mount. See builder/builder-server/README.md and env.template.
 # Nginx uses DATA_DIR/ca.crt for ssl_client_certificate; container must use the same host path so CA matches.
@@ -308,83 +410,7 @@ DOCKER_EOF
   fi
 fi
 
-# --- Workspace and ssh-keys dirs (per-developer users; apply script manages OS users) ---
-mkdir -p "${DATA_DIR}/workspace"
-mkdir -p "${DATA_DIR}/ssh-keys"
-
-# --- Host job: apply per-developer OS users from builder-server state ---
-APPLY_DEV_USERS_SCRIPT="/usr/local/bin/aifabrix-apply-dev-users.sh"
-SETUP_ASSETS_DIR="$(cd "$(dirname "$0")" && pwd)"
-if [ ! -f "$APPLY_DEV_USERS_SCRIPT" ]; then
-  if [ -f "$SETUP_ASSETS_DIR/aifabrix-apply-dev-users.sh" ]; then
-    cp "$SETUP_ASSETS_DIR/aifabrix-apply-dev-users.sh" "$APPLY_DEV_USERS_SCRIPT"
-  else
-    # Inline fallback when run from builder/builder-server context
-    cat > "$APPLY_DEV_USERS_SCRIPT" << 'APPLY_DEV_EOF'
-#!/bin/sh
-DATA_DIR="${DATA_DIR:-/opt/aifabrix/builder-server/data}"
-SSH_KEYS_DIR="${DATA_DIR}/ssh-keys"
-PENDING_REMOVALS="${DATA_DIR}/pending-removals"
-[ ! -d "$DATA_DIR" ] && exit 0
-if [ -f "$PENDING_REMOVALS" ]; then
-  while IFS= read -r dev_id || [ -n "$dev_id" ]; do
-    dev_id=$(echo "$dev_id" | tr -d '\r\n ')
-    [ -z "$dev_id" ] && continue
-    user_name="dev${dev_id}"
-    getent passwd "$user_name" >/dev/null 2>&1 && ( userdel -r "$user_name" 2>/dev/null || userdel "$user_name" 2>/dev/null || true )
-  done < "$PENDING_REMOVALS"
-  : > "$PENDING_REMOVALS"
-fi
-for key_file in "${SSH_KEYS_DIR}"/*/authorized_keys; do
-  [ -f "$key_file" ] || continue
-  dev_id=$(dirname "$key_file" | xargs basename)
-  [ -z "$dev_id" ] && continue
-  user_name="dev${dev_id}"
-  home_dir="/home/${user_name}"
-  workspace_target="${DATA_DIR}/workspace/${user_name}"
-  getent passwd "$user_name" >/dev/null 2>&1 || useradd -m -s /bin/bash "$user_name"
-  mkdir -p "${home_dir}/.ssh"
-  cp "$key_file" "${home_dir}/.ssh/authorized_keys"
-  chown -R "${user_name}:${user_name}" "${home_dir}/.ssh"
-  chmod 700 "${home_dir}/.ssh"
-  chmod 600 "${home_dir}/.ssh/authorized_keys"
-  config_file="${home_dir}/.aifabrix/config.yaml"
-  if [ ! -f "$config_file" ]; then
-    mkdir -p "${home_dir}/.aifabrix"
-    hostname_val=$(hostname -f 2>/dev/null || hostname 2>/dev/null || echo "localhost")
-    printf 'user-mutagen-folder: %s\nsecrets-encryption: ""\naifabrix-secrets: ""\naifabrix-env-config: ""\nremote-server: "http://localhost:3000"\ndocker-endpoint: "tcp://%s:2376"\nsync-ssh-user: "%s"\nsync-ssh-host: "%s"\n' "$workspace_target" "$hostname_val" "$user_name" "$hostname_val" > "$config_file"
-    chown -R "${user_name}:${user_name}" "${home_dir}/.aifabrix"
-    chmod 700 "${home_dir}/.aifabrix"
-    chmod 600 "$config_file"
-  fi
-  mkdir -p "$workspace_target"
-  chown -R "${user_name}:${user_name}" "$workspace_target"
-  [ -L "${home_dir}/workspace" ] && [ "$(readlink -f "${home_dir}/workspace" 2>/dev/null)" != "$(readlink -f "$workspace_target" 2>/dev/null)" ] && rm -f "${home_dir}/workspace"
-  [ ! -e "${home_dir}/workspace" ] && ln -s "$workspace_target" "${home_dir}/workspace" && chown -h "${user_name}:${user_name}" "${home_dir}/workspace"
-  profile="${home_dir}/.profile"
-  [ ! -f "$profile" ] && touch "$profile" && chown "${user_name}:${user_name}" "$profile"
-  grep -q 'cd.*workspace' "$profile" 2>/dev/null || echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$profile"
-  bashrc="${home_dir}/.bashrc"
-  [ ! -f "$bashrc" ] && touch "$bashrc" && chown "${user_name}:${user_name}" "$bashrc"
-  grep -q 'cd.*workspace' "$bashrc" 2>/dev/null || echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$bashrc"
-done
-APPLY_DEV_EOF
-  fi
-  chmod 755 "$APPLY_DEV_USERS_SCRIPT"
-fi
-# Defaults for apply script (aifabrix-secrets, aifabrix-env-config in generated config); match builder-server env.template
-APPLY_DEFAULTS="$DATA_DIR_ABS/apply-dev-users-defaults"
-if [ ! -f "$APPLY_DEFAULTS" ]; then
-  {
-    echo "# Sourced by cron before aifabrix-apply-dev-users.sh; override to match builder-server .env"
-    echo 'export AIFABRIX_SECRETS="${AIFABRIX_SECRETS:-/aifabrix-miso/builder/secrets.local.yaml}"'
-    echo 'export AIFABRIX_ENV_CONFIG="${AIFABRIX_ENV_CONFIG:-aifabrix-miso/builder/env-config.yaml}"'
-  } > "$APPLY_DEFAULTS"
-  chmod 644 "$APPLY_DEFAULTS"
-fi
-if [ -d /etc/cron.d ] && [ ! -f /etc/cron.d/aifabrix-apply-dev-users ]; then
-  echo "*/2 * * * * root /bin/sh -c '. $APPLY_DEFAULTS 2>/dev/null; export DATA_DIR=$DATA_DIR_ABS; exec $APPLY_DEV_USERS_SCRIPT'" > /etc/cron.d/aifabrix-apply-dev-users
-  chmod 644 /etc/cron.d/aifabrix-apply-dev-users
 fi
+# --- End server phase ---
 
 echo "Setup complete. Ensure manual prerequisites (SSL at $SSL_DIR: wildcard.crt, wildcard.key; DNS $DEV_DOMAIN) are done; see SETUP.md."

package/assets/setup-install-init.sh

@@ -0,0 +1,42 @@
+#!/bin/sh
+# One-time bootstrap: ensure SSH, install Node 18+ and npm, then af-server CLI on the server.
+# Run via: af-server install-init user@host (from PC over SSH). No Docker, nginx, or builder-server.
+# After this, user logs in to the server and runs: sudo af-server install, then sudo af-server install-server --dev-domain DOMAIN.
+
+set -e
+export DEBIAN_FRONTEND=noninteractive
+
+# --- SSH server (so install-init and later ssh-cert install can connect) ---
+wait_for_apt() {
+  i=0
+  while [ $i -lt 20 ]; do
+    apt-get update -qq 2>/dev/null && return 0
+    echo "Waiting for apt lock..."; sleep 6
+    i=$((i + 1))
+  done
+  return 1
+}
+wait_for_apt
+apt-get install -y openssh-server
+systemctl enable ssh 2>/dev/null || systemctl enable sshd 2>/dev/null || true
+systemctl start ssh 2>/dev/null || systemctl start sshd 2>/dev/null || true
+
+# --- Node.js 18+ and npm (NodeSource) ---
+need_node=0
+if ! command -v node >/dev/null 2>&1; then
+  need_node=1
+else
+  case "$(node -v 2>/dev/null)" in v1[89].*|v[2-9]*) ;; *) need_node=1 ;; esac
+fi
+if [ "$need_node" = "1" ]; then
+  curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
+  apt-get install -y nodejs
+fi
+node -v
+npm -v
+
+# --- af-server CLI (same as on PC) ---
+npm install -g @aifabrix/builder @aifabrix/server-setup
+command -v af-server >/dev/null 2>&1 && af-server --version || true
+
+echo "Bootstrap complete. Log in to the server and run: sudo af-server install, then sudo af-server install-server --dev-domain YOUR_DOMAIN"

package/dist/cli.js

@@ -3,22 +3,42 @@
  * af-server — Install, backup, and restore AI Fabrix builder-server (config + DB) over SSH.
  * Usage: af-server <command> [options] [user@host]
  */
+import { readFileSync } from 'fs';
+import { dirname, join } from 'path';
+import { fileURLToPath } from 'url';
 import { Command } from 'commander';
-import { runInstall, runInstallLocal } from './install.js';
+import { runInstall, runInstallLocal, runInstallServerLocal } from './install.js';
 import { runBackup, runBackupLocal } from './backup.js';
 import { runBackupScheduleInstall, runBackupScheduleInstallLocal } from './backup-schedule.js';
 import { runRestore, runRestoreLocal } from './restore.js';
 import { runSshCertRequest, runSshCertInstall, runSshCertInstallLocal } from './ssh-cert.js';
 import { runInstallSsh, runInstallSshLocal } from './install-ssh.js';
+import { runInstallInit } from './install-init.js';
 import { requireUbuntu } from './ubuntu.js';
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const pkg = JSON.parse(readFileSync(join(__dirname, '..', 'package.json'), 'utf8'));
 const program = new Command();
 program
     .name('af-server')
     .description('Install, backup, and restore AI Fabrix builder-server over SSH (config + DB only)')
-    .version('0.1.0');
+    .version(pkg.version);
+program
+    .command('install-init <user@host>')
+    .description('One-time bootstrap over SSH: install on server SSH (if needed), Node 18+, npm, and af-server CLI. Run from PC only.')
+    .option('-i, --identity <path>', 'SSH private key path')
+    .action(async (target, opts) => {
+    try {
+        await runInstallInit({ target: target.trim(), privateKeyPath: opts.identity });
+        console.log('Install-init complete. Log in to the server and run: sudo af-server install, then sudo af-server install-server --dev-domain YOUR_DOMAIN');
+    }
+    catch (err) {
+        console.error(err instanceof Error ? err.message : err);
+        process.exit(1);
+    }
+});
 program
     .command('install [user@host]')
-    .description('Run server setup on host or locally (omit target on server). Docker, nginx, SSL, cron.')
+    .description('Run server setup on host or locally (omit target on server). Docker, nginx pkg, Mutagen, cron; no builder-server. Run on server: sudo af-server install.')
     .option('-d, --data-dir <path>', 'DATA_DIR', '/opt/aifabrix/builder-server/data')
     .option('--dev-domain <domain>', 'DEV_DOMAIN for nginx', 'builder01.aifabrix.dev')
     .option('--ssl-dir <path>', 'SSL_DIR', '/opt/aifabrix/ssl')
@@ -52,6 +72,29 @@ program
         process.exit(1);
     }
 });
+program
+    .command('install-server')
+    .description('On server only: nginx vhost, builder-server container, Docker TLS. Run after: sudo af-server install. Requires --dev-domain.')
+    .option('-d, --data-dir <path>', 'DATA_DIR', '/opt/aifabrix/builder-server/data')
+    .requiredOption('--dev-domain <domain>', 'DEV_DOMAIN for nginx (e.g. builder01.aifabrix.dev)')
+    .option('--ssl-dir <path>', 'SSL_DIR', '/opt/aifabrix/ssl')
+    .option('--builder-port <port>', 'BUILDER_SERVER_PORT', '3000')
+    .action(async (opts) => {
+    try {
+        requireUbuntu();
+        runInstallServerLocal({
+            dataDir: opts.dataDir,
+            devDomain: opts.devDomain,
+            sslDir: opts.sslDir,
+            builderServerPort: opts.builderPort ? parseInt(opts.builderPort, 10) : undefined,
+        });
+        console.log('Install-server complete.');
+    }
+    catch (err) {
+        console.error(err instanceof Error ? err.message : err);
+        process.exit(1);
+    }
+});
 program
     .command('backup [user@host]')
     .description('On-demand backup, or --schedule to install cron (omit target for local).')

package/dist/install-init.d.ts

@@ -0,0 +1,6 @@
+/**
+ * install-init: One-time bootstrap over SSH. Installs on the server: SSH (if needed), Node 18+, npm, and af-server CLI.
+ * No Docker, nginx, or builder-server. After this, the user logs in to the server and runs install + install-server locally.
+ */
+import { type SSHConnectionOptions } from './ssh.js';
+export declare function runInstallInit(options: SSHConnectionOptions): Promise<void>;

package/dist/install-init.js

@@ -0,0 +1,41 @@
+/**
+ * install-init: One-time bootstrap over SSH. Installs on the server: SSH (if needed), Node 18+, npm, and af-server CLI.
+ * No Docker, nginx, or builder-server. After this, the user logs in to the server and runs install + install-server locally.
+ */
+import * as path from 'path';
+import * as fs from 'fs';
+import { fileURLToPath } from 'url';
+import { createSSHClient, exec, writeFile, close } from './ssh.js';
+const scriptDir = path.dirname(fileURLToPath(import.meta.url));
+const ASSETS_DIR = path.resolve(scriptDir, '..', 'assets');
+function toUnixLf(s) {
+    return s.replace(/\r\n/g, '\n').replace(/\r/g, '\n');
+}
+function getInitScript() {
+    const p = path.join(ASSETS_DIR, 'setup-install-init.sh');
+    return toUnixLf(fs.readFileSync(p, 'utf8'));
+}
+export async function runInstallInit(options) {
+    const conn = await createSSHClient(options);
+    try {
+        const tmpDir = `/tmp/aifabrix-init-${Date.now()}`;
+        await exec(conn, `mkdir -p ${tmpDir}`);
+        await exec(conn, `chmod 755 ${tmpDir}`);
+        const script = getInitScript();
+        await writeFile(conn, `${tmpDir}/setup-install-init.sh`, script);
+        await exec(conn, `chmod +x ${tmpDir}/setup-install-init.sh`);
+        const cmd = `sudo ${tmpDir}/setup-install-init.sh`;
+        const result = await exec(conn, cmd);
+        if (result.stderr)
+            process.stderr.write(result.stderr);
+        if (result.stdout)
+            process.stdout.write(result.stdout);
+        if (result.code !== 0) {
+            throw new Error(`install-init script exited with code ${result.code}`);
+        }
+        await exec(conn, `rm -rf ${tmpDir}`);
+    }
+    finally {
+        close(conn);
+    }
+}

package/dist/install-init.spec.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/install-init.spec.js

@@ -0,0 +1,10 @@
+/**
+ * Tests for install-init: bootstrap over SSH (mocked).
+ * install-init.ts uses import.meta.url; Jest (CJS) fails to load it. Skipped until ESM/import.meta is supported in tests.
+ */
+describe.skip('install-init', () => {
+    it('placeholder until Jest supports import.meta in transformed modules', () => {
+        expect(true).toBe(true);
+    });
+});
+export {};

package/dist/install.d.ts

@@ -17,3 +17,6 @@ export interface InstallLocalOptions {
 export declare function runInstall(options: InstallOptions): Promise<void>;
 /** Run setup script locally (no SSH). Call requireUbuntu() before this. */
 export declare function runInstallLocal(options?: InstallLocalOptions): void;
+export declare function runInstallServer(options: InstallOptions): Promise<void>;
+/** Run server phase locally (nginx vhost, builder-server container, Docker TLS). Call requireUbuntu() before this. */
+export declare function runInstallServerLocal(options?: InstallLocalOptions): void;

package/dist/install.js

@@ -39,6 +39,7 @@ export async function runInstall(options) {
         const q = (v) => `'${String(v).replace(/'/g, "'\\''")}'`;
         const env = [
             `REPO_ROOT=${q(tmpDir)}`,
+            `INSTALL_PHASE=infra`,
             `DATA_DIR=${q(dataDir)}`,
             `DEV_DOMAIN=${q(devDomain)}`,
             `SSL_DIR=${q(sslDir)}`,
@@ -71,7 +72,65 @@ export function runInstallLocal(options = {}) {
     try {
         fs.writeFileSync(path.join(tmpDir, 'setup.sh'), getSetupScript(), { mode: 0o755 });
         fs.writeFileSync(path.join(builderSubdir, 'nginx-builder-server.conf.template'), getNginxTemplate());
-        const env = [`REPO_ROOT=${tmpDir}`, `DATA_DIR=${dataDir}`, `DEV_DOMAIN=${devDomain}`, `SSL_DIR=${sslDir}`, `BUILDER_SERVER_PORT=${builderServerPort}`].join(' ');
+        const env = [`REPO_ROOT=${tmpDir}`, `INSTALL_PHASE=infra`, `DATA_DIR=${dataDir}`, `DEV_DOMAIN=${devDomain}`, `SSL_DIR=${sslDir}`, `BUILDER_SERVER_PORT=${builderServerPort}`].join(' ');
+        execSync(`sudo ${env} ${tmpDir}/setup.sh`, { stdio: 'inherit' });
+    }
+    finally {
+        fs.rmSync(tmpDir, { recursive: true, force: true });
+    }
+}
+export async function runInstallServer(options) {
+    const dataDir = options.dataDir ?? '/opt/aifabrix/builder-server/data';
+    const devDomain = options.devDomain ?? 'builder01.aifabrix.dev';
+    const sslDir = options.sslDir ?? '/opt/aifabrix/ssl';
+    const builderServerPort = options.builderServerPort ?? 3000;
+    const conn = await createSSHClient(options);
+    try {
+        const tmpDir = `/tmp/aifabrix-setup-${Date.now()}`;
+        await exec(conn, `mkdir -p ${tmpDir}/builder/builder-server`);
+        await exec(conn, `chmod 755 ${tmpDir}`);
+        const setupScript = getSetupScript();
+        const nginxTemplate = getNginxTemplate();
+        await writeFile(conn, `${tmpDir}/setup.sh`, setupScript);
+        await writeFile(conn, `${tmpDir}/builder/builder-server/nginx-builder-server.conf.template`, nginxTemplate);
+        await exec(conn, `chmod +x ${tmpDir}/setup.sh`);
+        const q = (v) => `'${String(v).replace(/'/g, "'\\''")}'`;
+        const env = [
+            `REPO_ROOT=${q(tmpDir)}`,
+            `INSTALL_PHASE=server`,
+            `DATA_DIR=${q(dataDir)}`,
+            `DEV_DOMAIN=${q(devDomain)}`,
+            `SSL_DIR=${q(sslDir)}`,
+            `BUILDER_SERVER_PORT=${q(String(builderServerPort))}`,
+        ].join(' ');
+        const cmd = `sudo ${env} ${tmpDir}/setup.sh`;
+        const result = await exec(conn, cmd);
+        if (result.stderr)
+            process.stderr.write(result.stderr);
+        if (result.stdout)
+            process.stdout.write(result.stdout);
+        if (result.code !== 0) {
+            throw new Error(`Install-server script exited with code ${result.code}`);
+        }
+        await exec(conn, `rm -rf ${tmpDir}`);
+    }
+    finally {
+        close(conn);
+    }
+}
+/** Run server phase locally (nginx vhost, builder-server container, Docker TLS). Call requireUbuntu() before this. */
+export function runInstallServerLocal(options = {}) {
+    const dataDir = options.dataDir ?? '/opt/aifabrix/builder-server/data';
+    const devDomain = options.devDomain ?? 'builder01.aifabrix.dev';
+    const sslDir = options.sslDir ?? '/opt/aifabrix/ssl';
+    const builderServerPort = options.builderServerPort ?? 3000;
+    const tmpDir = `/tmp/aifabrix-setup-${Date.now()}`;
+    const builderSubdir = path.join(tmpDir, 'builder', 'builder-server');
+    fs.mkdirSync(builderSubdir, { recursive: true });
+    try {
+        fs.writeFileSync(path.join(tmpDir, 'setup.sh'), getSetupScript(), { mode: 0o755 });
+        fs.writeFileSync(path.join(builderSubdir, 'nginx-builder-server.conf.template'), getNginxTemplate());
+        const env = [`REPO_ROOT=${tmpDir}`, `INSTALL_PHASE=server`, `DATA_DIR=${dataDir}`, `DEV_DOMAIN=${devDomain}`, `SSL_DIR=${sslDir}`, `BUILDER_SERVER_PORT=${builderServerPort}`].join(' ');
         execSync(`sudo ${env} ${tmpDir}/setup.sh`, { stdio: 'inherit' });
     }
     finally {

package/dist/install.spec.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/install.spec.js

@@ -0,0 +1,23 @@
+/**
+ * Tests for install: runInstall (infra phase) and runInstallServer / runInstallServerLocal (server phase).
+ * install.ts uses import.meta.url; Jest (CJS) fails to load it. Unit tests for runInstall* are skipped; we test script content.
+ */
+import * as fs from 'fs';
+import * as path from 'path';
+const ASSETS_DIR = path.resolve(__dirname, '..', 'assets');
+const SETUP_SCRIPT = path.join(ASSETS_DIR, 'setup-dev-server-no-node.sh');
+describe('install script (setup-dev-server-no-node.sh)', () => {
+    it('defines INSTALL_PHASE and infra/server phase blocks', () => {
+        const content = fs.readFileSync(SETUP_SCRIPT, 'utf8');
+        expect(content).toContain('INSTALL_PHASE="${INSTALL_PHASE:-full}"');
+        expect(content).toContain('"$INSTALL_PHASE" = "infra"');
+        expect(content).toContain('"$INSTALL_PHASE" = "server"');
+        expect(content).toContain('End infra phase');
+        expect(content).toContain('End server phase');
+    });
+});
+describe.skip('install (runInstall* unit tests)', () => {
+    it('placeholder until Jest supports import.meta in transformed modules', () => {
+        expect(true).toBe(true);
+    });
+});

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aifabrix/server-setup",
-  "version": "1.4.0",
+  "version": "1.5.2",
   "description": "CLI to install, backup, and restore AI Fabrix builder-server (config + DB) over SSH",
   "type": "module",
   "main": "dist/cli.js",
@@ -46,5 +46,10 @@
   "files": [
     "dist",
     "assets"
-  ]
+  ],
+  "pnpm": {
+    "onlyBuiltDependencies": [
+      "better-sqlite3"
+    ]
+  }
 }