@aifabrix/server-setup 1.3.0 → 1.4.0

package/README.md

@@ -17,7 +17,8 @@ This is the **one document** you need to get your builder-server **from zero to
 **The builder-server Docker image is not on a public registry.** You get it with the **AI Fabrix platform**.
 
 1. Install the **AI Fabrix Builder** CLI: `npm install -g @aifabrix/builder`
-2. Use the CLI to run or deploy the platform (including builder-server). See [AI Fabrix Builder](https://github.com/esystemsdev/aifabrix-builder) and [docs/README.md](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/README.md).
+2. Install the **AI Fabrix server-setup** CLI (af-server): `npm install -g @aifabrix/server-setup`
+3. Use the Builder CLI to run or deploy the platform (including builder-server). See [AI Fabrix Builder](https://github.com/esystemsdev/aifabrix-builder) and [docs/README.md](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/README.md).
 
 Once you have the platform (and the builder-server image), use **af-server** to install that server on your own host.
 
@@ -39,16 +40,71 @@ Do the steps in [What you must do before running af-server](#what-you-must-do-be
 
 ## Install: from zero to running
 
-**From your PC** (SSH to server):
+Complete the [manual prerequisites](#what-you-must-do-before-running-af-server) first (DNS, SSL directory, certificate and key on the server).
 
-```bash
-af-server install user@host
-```
+### Option A: From your PC (remote install over SSH)
+
+1. **Install CLIs** (one-time):
+
+   ```bash
+   npm install -g @aifabrix/builder
+   npm install -g @aifabrix/server-setup
+   ```
+
+2. **Set your server target and local path** (replace with your host, user, and the folder on your PC where your cert and key files are):
+
+   ```bash
+   export SSH=serveradmin@builder02.aifabrix.dev
+   export HDD=/c/git/esystemsdev/aifabrix-setup/certificates
+   export DOMAIN=builder02.aifabrix.dev
+   ```
+
+3. **If the server has no SSH yet** (e.g. fresh VM with console only), enable SSH first (you need another way to run one command, e.g. cloud console):
+
+   ```bash
+   af-server install-ssh $SSH
+   ```
+
+4. **Add your SSH key** for passwordless auth (so install can run over SSH):
+
+   ```bash
+   af-server ssh-cert install $SSH
+   ```
+
+5. **Put SSL cert and key on the server** in `/opt/aifabrix/ssl` as `wildcard.crt` and `wildcard.key`. See [SSL directory and certificates](#ssl-directory-and-certificates).
+
+   Example: copy from your PC to the server (create the directory on the server first, then copy and fix key permissions):
+
+   ```bash
+   ssh $SSH "sudo mkdir -p /opt/aifabrix/ssl"
+   scp $HDD/wildcard.crt $SSH:/tmp/wildcard.crt
+   scp $HDD/wildcard.key $SSH:/tmp/wildcard.key
+   ssh $SSH "sudo mv /tmp/wildcard.crt /tmp/wildcard.key /opt/aifabrix/ssl/ && sudo chmod 600 /opt/aifabrix/ssl/wildcard.key"
+   ```
+
+6. **Run install** (uses sudo on the server):
+
+   ```bash
+   af-server install $SSH
+   ```
+
+   To use a different domain or SSL path:
+
+   ```bash
+   af-server install $SSH --dev-domain $DOMAIN --ssl-dir /opt/aifabrix/ssl
+   ```
+
+   - Space between `$DOMAIN` and `--ssl-dir` is required.
+   - **Git Bash (Windows):** quote server paths so the shell doesn't expand `/opt` (e.g. `--ssl-dir '/opt/aifabrix/ssl'`, `--data-dir '/opt/aifabrix/builder-server/data'`).
+
+### Option B: On the server itself (Ubuntu, no target)
 
-**On the server itself** (Ubuntu, no target):
+If you are already on the server (e.g. over console or SSH), run install locally. Prerequisites (DNS, SSL dir with `wildcard.crt` and `wildcard.key`) must still be done first.
 
 ```bash
-af-server install
+npm install -g @aifabrix/builder
+npm install -g @aifabrix/server-setup
+sudo af-server install
 ```
 
 After install, your builder-server is up. Use the **AI Fabrix Builder CLI** for all usage (users, secrets, certs, etc.)—see [Builder documentation](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/developer-isolation.md).
@@ -64,6 +120,7 @@ After install, your builder-server is up. Use the **AI Fabrix Builder CLI** for
 | `af-server backup [ user@host ] --schedule [ --backup-dir PATH ] [ --keep-days N ] [ -i SSH_KEY ]` | Cron backup (daily 02:00, keep last N, default 7). |
 | `af-server restore backup.zip [ user@host ] [ -d DATA_DIR ] [ --force ] [ -i SSH_KEY ]` | Restore backup to DATA_DIR. |
 | `af-server ssh-cert install [ user@host ] [ -i SSH_KEY ]` | Add your SSH public key to server (passwordless auth). |
+| `af-server install-ssh [ user@host ] [ -i SSH_KEY ]` | Activate SSH server (install openssh-server, enable and start ssh) without login. Omit target for local. |
 
 Backups contain secrets; store encrypted. Cron backup needs SQLite (`builder.db`) and `zip` on the server; default backup dir: `/opt/aifabrix/backups`.
 

package/assets/aifabrix-apply-dev-users.sh

@@ -0,0 +1,123 @@
+#!/bin/sh
+# Apply per-developer OS users from builder-server state (DATA_DIR/ssh-keys, pending-removals).
+# Run via cron every 2 minutes. Creates/updates dev<id> users, .ssh/authorized_keys, workspace symlink,
+# and optional ~/.aifabrix/config.yaml when missing. Processes pending-removals then applies keys.
+# Requires root. DATA_DIR must match builder-server (e.g. /opt/aifabrix/builder-server/data).
+
+set -e
+
+DATA_DIR="${DATA_DIR:-/opt/aifabrix/builder-server/data}"
+SSH_KEYS_DIR="${DATA_DIR}/ssh-keys"
+PENDING_REMOVALS="${DATA_DIR}/pending-removals"
+# Defaults matching builder-server env.template (override via env or apply-dev-users-defaults)
+AIFABRIX_SECRETS="${AIFABRIX_SECRETS:-/aifabrix-miso/builder/secrets.local.yaml}"
+AIFABRIX_ENV_CONFIG="${AIFABRIX_ENV_CONFIG:-aifabrix-miso/builder/env-config.yaml}"
+
+if [ ! -d "$DATA_DIR" ]; then
+  echo "DATA_DIR not found: $DATA_DIR"
+  exit 0
+fi
+
+# Read secrets-encryption key from server data dir (same as builder-server ENCRYPTION_KEY_PATH); do not log.
+secrets_encryption_value=""
+if [ -f "${DATA_DIR}/secrets-encryption.key" ]; then
+  secrets_encryption_value=$(cat "${DATA_DIR}/secrets-encryption.key" | tr -d '\n\r' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
+fi
+
+# --- 1. Process removals ---
+if [ -f "$PENDING_REMOVALS" ]; then
+  while IFS= read -r dev_id || [ -n "$dev_id" ]; do
+    dev_id=$(echo "$dev_id" | tr -d '\r\n ')
+    [ -z "$dev_id" ] && continue
+    user_name="dev${dev_id}"
+    if getent passwd "$user_name" >/dev/null 2>&1; then
+      userdel -r "$user_name" 2>/dev/null || userdel "$user_name" 2>/dev/null || true
+    fi
+  done < "$PENDING_REMOVALS"
+  : > "$PENDING_REMOVALS"
+fi
+
+# --- 2. Process each developer that has keys ---
+for key_file in "${SSH_KEYS_DIR}"/*/authorized_keys; do
+  [ -f "$key_file" ] || continue
+  dev_id=$(dirname "$key_file" | xargs basename)
+  [ -z "$dev_id" ] && continue
+  user_name="dev${dev_id}"
+  home_dir="/home/${user_name}"
+  workspace_target="${DATA_DIR}/workspace/${user_name}"
+
+  if ! getent passwd "$user_name" >/dev/null 2>&1; then
+    useradd -m -s /bin/bash "$user_name"
+  else
+    # Ensure shell is bash (e.g. after manual changes)
+    current_shell=$(getent passwd "$user_name" | cut -d: -f7)
+    if [ "$current_shell" != "/bin/bash" ]; then
+      usermod -s /bin/bash "$user_name" 2>/dev/null || true
+    fi
+  fi
+
+  mkdir -p "${home_dir}/.ssh"
+  cp "$key_file" "${home_dir}/.ssh/authorized_keys"
+  chown -R "${user_name}:${user_name}" "${home_dir}/.ssh"
+  chmod 700 "${home_dir}/.ssh"
+  chmod 600 "${home_dir}/.ssh/authorized_keys"
+
+  # .aifabrix/config.yaml only if missing (do not overwrite)
+  config_dir="${home_dir}/.aifabrix"
+  config_file="${config_dir}/config.yaml"
+  if [ ! -f "$config_file" ]; then
+    mkdir -p "$config_dir"
+    hostname_val=$(hostname -f 2>/dev/null || hostname 2>/dev/null || echo "localhost")
+    # Values from builder-server: secrets-encryption from DATA_DIR, paths from env (apply-dev-users-defaults or env.template)
+    yaml_escape() { echo "$1" | sed 's/\\/\\\\/g;s/"/\\"/g'; }
+    {
+      echo "user-mutagen-folder: ${workspace_target}"
+      printf "secrets-encryption: \"%s\"\n" "$(yaml_escape "$secrets_encryption_value")"
+      printf "aifabrix-secrets: \"%s\"\n" "$(yaml_escape "$AIFABRIX_SECRETS")"
+      printf "aifabrix-env-config: \"%s\"\n" "$(yaml_escape "$AIFABRIX_ENV_CONFIG")"
+      echo "remote-server: \"http://localhost:3000\""
+      echo "docker-endpoint: \"tcp://${hostname_val}:2376\""
+      echo "sync-ssh-user: \"${user_name}\""
+      echo "sync-ssh-host: \"${hostname_val}\""
+    } > "$config_file"
+    chown -R "${user_name}:${user_name}" "$config_dir"
+    chmod 700 "$config_dir"
+    chmod 600 "$config_file"
+  fi
+
+  # Workspace: ensure dir exists, owned by user; symlink from home
+  mkdir -p "$workspace_target"
+  chown -R "${user_name}:${user_name}" "$workspace_target"
+  if [ -L "${home_dir}/workspace" ]; then
+    current_target=$(readlink -f "${home_dir}/workspace" 2>/dev/null || true)
+    want_target=$(readlink -f "$workspace_target" 2>/dev/null || echo "$workspace_target")
+    if [ -n "$current_target" ] && [ -n "$want_target" ] && [ "$current_target" != "$want_target" ]; then
+      rm -f "${home_dir}/workspace"
+      ln -s "$workspace_target" "${home_dir}/workspace"
+      chown -h "${user_name}:${user_name}" "${home_dir}/workspace"
+    fi
+  elif [ ! -e "${home_dir}/workspace" ]; then
+    ln -s "$workspace_target" "${home_dir}/workspace"
+    chown -h "${user_name}:${user_name}" "${home_dir}/workspace"
+  fi
+
+  # Default directory on SSH login: cd to workspace
+  profile="${home_dir}/.profile"
+  if [ ! -f "$profile" ]; then
+    touch "$profile"
+    chown "${user_name}:${user_name}" "$profile"
+  fi
+  if ! grep -q 'cd.*workspace' "$profile" 2>/dev/null; then
+    echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$profile"
+    chown "${user_name}:${user_name}" "$profile"
+  fi
+  bashrc="${home_dir}/.bashrc"
+  if [ ! -f "$bashrc" ]; then
+    touch "$bashrc"
+    chown "${user_name}:${user_name}" "$bashrc"
+  fi
+  if ! grep -q 'cd.*workspace' "$bashrc" 2>/dev/null; then
+    echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$bashrc"
+    chown "${user_name}:${user_name}" "$bashrc"
+  fi
+done

package/assets/setup-dev-server-no-node.sh

@@ -5,6 +5,7 @@
 # Optional env: DEV_DOMAIN, SSL_DIR, DATA_DIR, SETUP_ADMIN_USER, SYNC_USER, BUILDER_SERVER_PORT, NGINX_CONF_DIR, SETUP_HOSTNAME, INSTALL_PORTAINER=1, SKIP_DOCKER_TLS=1.
 
 set -e
+export DEBIAN_FRONTEND=noninteractive
 
 REPO_ROOT="${REPO_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"
 DATA_DIR="${DATA_DIR:-/opt/aifabrix/builder-server/data}"
@@ -25,6 +26,9 @@ sanitize_path() {
     *) return 0 ;;
   esac
 }
+case "$DEV_DOMAIN" in
+  *'--'*) echo "Invalid DEV_DOMAIN: '$DEV_DOMAIN' looks like a typo (e.g. missing space before --ssl-dir). Use: af-server install user@host --dev-domain DOMAIN --ssl-dir PATH"; exit 1 ;;
+esac
 if ! sanitize_domain "$DEV_DOMAIN"; then
   echo "Invalid DEV_DOMAIN (use only letters, digits, dots, hyphens)."
   exit 1
@@ -58,7 +62,19 @@ fi
 
 # --- System updates ---
 apt-get update -qq
-DEBIAN_FRONTEND=noninteractive apt-get upgrade -y -qq
+apt-get upgrade -y -qq
+
+# --- Packages for SSH users (same base as builder-server/wsl/install-wsl-ubuntu-dev.sh; no Node/Ruby) ---
+apt-get install -y \
+  openssh-server sudo curl git ca-certificates gnupg software-properties-common apt-transport-https \
+  build-essential unzip zip jq wget vim nano less locales \
+  python3 python3-venv python3-dev python3-pip \
+  openjdk-17-jdk \
+  autoconf bison libssl-dev libyaml-dev libreadline-dev zlib1g-dev libffi-dev libgdbm-dev libncurses-dev \
+  redis-tools dbus-x11 imagemagick \
+  openssh-client rsync sshfs
+systemctl enable ssh 2>/dev/null || systemctl enable sshd 2>/dev/null || true
+systemctl start ssh 2>/dev/null || systemctl start sshd 2>/dev/null || true
 
 # --- Docker ---
 if ! command -v docker >/dev/null 2>&1; then
@@ -152,23 +168,6 @@ MUTAGEN_EOF
   fi
 fi
 
-# --- Docker TLS (daemon.json) ---
-if [ "$SKIP_DOCKER_TLS" != "1" ] && [ -d /etc/docker ]; then
-  if [ ! -f /etc/docker/daemon.json ]; then
-    cat > /etc/docker/daemon.json << 'DOCKER_EOF'
-{
-  "tls": true,
-  "tlsverify": true,
-  "tlscacert": "/etc/docker/ca.pem",
-  "tlscert": "/etc/docker/server-cert.pem",
-  "tlskey": "/etc/docker/server-key.pem",
-  "hosts": ["tcp://0.0.0.0:2376", "unix:///var/run/docker.sock"]
-}
-DOCKER_EOF
-    echo "Docker TLS daemon.json created. Ensure /etc/docker/ca.pem, server-cert.pem, server-key.pem exist (see SETUP.md Docker API TLS)."
-  fi
-fi
-
 # --- Builder-server data dir and container ---
 # Paths match AI Fabrix Builder: aifabrix build + resolve use DATA_DIR=/mnt/data in container; host DATA_DIR (e.g. /opt/aifabrix/builder-server/data) is the HDD/mount. See builder/builder-server/README.md and env.template.
 # Nginx uses DATA_DIR/ca.crt for ssl_client_certificate; container must use the same host path so CA matches.
@@ -217,6 +216,7 @@ if command -v docker >/dev/null 2>&1; then
         -e PORT=3000 \
         -e DATA_DIR="$CONTAINER_DATA_PATH" \
         -e ENCRYPTION_KEY_PATH="${CONTAINER_DATA_PATH}/secrets-encryption.key" \
+        -e DOCKER_ENDPOINT="tcp://${DEV_DOMAIN}:2376" \
         "$IMG_TO_USE"
     else
       echo "Builder-server image not found. Get the image from the AI Fabrix Builder (aifabrix build builder-server; then push/deploy to this host). No source or docker build on server. See builder/builder-server/README.md."
@@ -227,50 +227,164 @@ if command -v docker >/dev/null 2>&1; then
   fi
 fi
 
-# --- Sync user for Mutagen SSH ---
-SYNC_USER_HOME="${DATA_DIR}/.sync-home"
-MANAGED_KEYS_FILE="${DATA_DIR}/sync-authorized-keys"
-if ! getent passwd "$SYNC_USER" >/dev/null 2>&1; then
-  useradd --system --home-dir "$SYNC_USER_HOME" --create-home --shell /bin/bash "$SYNC_USER"
-fi
-mkdir -p "$SYNC_USER_HOME/.ssh"
-chown -R "$SYNC_USER:$SYNC_USER" "$SYNC_USER_HOME"
-chmod 700 "$SYNC_USER_HOME/.ssh"
-if [ ! -f "$SYNC_USER_HOME/.ssh/authorized_keys" ]; then
-  touch "$SYNC_USER_HOME/.ssh/authorized_keys"
-  chown "$SYNC_USER:$SYNC_USER" "$SYNC_USER_HOME/.ssh/authorized_keys"
-  chmod 600 "$SYNC_USER_HOME/.ssh/authorized_keys"
+# --- Docker TLS (daemon.json + certs): use website cert for server, builder-server CA for client verification ---
+if [ "$SKIP_DOCKER_TLS" != "1" ] && [ -d /etc/docker ]; then
+  DATA_DIR_ABS=$(cd "$DATA_DIR" 2>/dev/null && pwd) || true
+  if [ -n "$DATA_DIR_ABS" ]; then
+    # Wait for builder-server to create ca.crt (up to 30s)
+    for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
+      [ -f "$DATA_DIR_ABS/ca.crt" ] && break
+      sleep 2
+    done
+    # Server cert/key: use existing website cert (same as nginx) so no extra cert to manage
+    if [ -f "$SSL_DIR/wildcard.crt" ] && [ -f "$SSL_DIR/wildcard.key" ]; then
+      cp "$SSL_DIR/wildcard.crt" /etc/docker/server-cert.pem
+      cp "$SSL_DIR/wildcard.key" /etc/docker/server-key.pem
+      chmod 600 /etc/docker/server-key.pem
+    fi
+    # Client verification: CA that signed developer certs (builder-server creates ca.crt on first run)
+    if [ -f "$DATA_DIR_ABS/ca.crt" ]; then
+      cp "$DATA_DIR_ABS/ca.crt" /etc/docker/ca.pem
+    fi
+    if [ -f /etc/docker/ca.pem ] && [ -f /etc/docker/server-cert.pem ] && [ -f /etc/docker/server-key.pem ]; then
+      # Allow daemon.json to set "hosts" (avoid conflict with systemd -H fd://)
+      DOCKER_DROPIN="/etc/systemd/system/docker.service.d"
+      if [ -d /etc/systemd/system ]; then
+        mkdir -p "$DOCKER_DROPIN"
+        if [ ! -f "$DOCKER_DROPIN/override.conf" ]; then
+          cat > "$DOCKER_DROPIN/override.conf" << 'DROPIN_EOF'
+[Service]
+ExecStart=
+ExecStart=/usr/bin/dockerd
+DROPIN_EOF
+          systemctl daemon-reload
+        fi
+      fi
+      [ -f /etc/docker/daemon.json ] && cp -a /etc/docker/daemon.json /etc/docker/daemon.json.bak
+      DATA_ROOT=''
+      if [ -f /etc/docker/daemon.json.bak ]; then
+        DATA_ROOT=$(grep -o '"data-root"[[:space:]]*:[[:space:]]*"[^"]*"' /etc/docker/daemon.json.bak 2>/dev/null | sed 's/.*:[[:space:]]*"\([^"]*\)".*/\1/' | head -1)
+      fi
+      if [ -n "$DATA_ROOT" ] && [ -d "$DATA_ROOT" ]; then
+        cat > /etc/docker/daemon.json << DOCKER_EOF
+{
+  "data-root": "$DATA_ROOT",
+  "tls": true,
+  "tlsverify": true,
+  "tlscacert": "/etc/docker/ca.pem",
+  "tlscert": "/etc/docker/server-cert.pem",
+  "tlskey": "/etc/docker/server-key.pem",
+  "hosts": ["tcp://0.0.0.0:2376", "unix:///var/run/docker.sock"]
+}
+DOCKER_EOF
+      else
+        cat > /etc/docker/daemon.json << 'DOCKER_EOF'
+{
+  "tls": true,
+  "tlsverify": true,
+  "tlscacert": "/etc/docker/ca.pem",
+  "tlscert": "/etc/docker/server-cert.pem",
+  "tlskey": "/etc/docker/server-key.pem",
+  "hosts": ["tcp://0.0.0.0:2376", "unix:///var/run/docker.sock"]
+}
+DOCKER_EOF
+      fi
+      systemctl restart docker 2>/dev/null || true
+      if ! docker info >/dev/null 2>&1; then
+        echo "Docker failed to start with TLS config. Restoring previous daemon.json so Docker can start."
+        if [ -f /etc/docker/daemon.json.bak ]; then
+          mv /etc/docker/daemon.json.bak /etc/docker/daemon.json
+        else
+          rm -f /etc/docker/daemon.json
+        fi
+        systemctl start docker 2>/dev/null || true
+        echo "Run scripts/fix-docker-daemon.sh if Docker still does not start. See SETUP.md Docker API TLS."
+      else
+        docker start aifabrix-builder-server 2>/dev/null || docker start builder-server 2>/dev/null || true
+      fi
+    else
+      echo "Docker TLS skipped: need $SSL_DIR/wildcard.crt, wildcard.key and $DATA_DIR/ca.crt (ca.crt appears after builder-server first run). Re-run af-server install after container has started. See SETUP.md Docker API TLS."
+    fi
+  fi
 fi
 
-# --- Workspace dir ---
+# --- Workspace and ssh-keys dirs (per-developer users; apply script manages OS users) ---
 mkdir -p "${DATA_DIR}/workspace"
-chown -R "$SYNC_USER:$SYNC_USER" "${DATA_DIR}/workspace" 2>/dev/null || true
+mkdir -p "${DATA_DIR}/ssh-keys"
 
-# --- Host job: apply managed SSH keys to sync user ---
-APPLY_KEYS_SCRIPT="/usr/local/bin/aifabrix-apply-sync-keys.sh"
-if [ ! -f "$APPLY_KEYS_SCRIPT" ]; then
-  cat > "$APPLY_KEYS_SCRIPT" << APPLY_EOF
+# --- Host job: apply per-developer OS users from builder-server state ---
+APPLY_DEV_USERS_SCRIPT="/usr/local/bin/aifabrix-apply-dev-users.sh"
+SETUP_ASSETS_DIR="$(cd "$(dirname "$0")" && pwd)"
+if [ ! -f "$APPLY_DEV_USERS_SCRIPT" ]; then
+  if [ -f "$SETUP_ASSETS_DIR/aifabrix-apply-dev-users.sh" ]; then
+    cp "$SETUP_ASSETS_DIR/aifabrix-apply-dev-users.sh" "$APPLY_DEV_USERS_SCRIPT"
+  else
+    # Inline fallback when run from builder/builder-server context
+    cat > "$APPLY_DEV_USERS_SCRIPT" << 'APPLY_DEV_EOF'
 #!/bin/sh
-SYNC_USER="$SYNC_USER"
-DATA_DIR="$DATA_DIR"
-SYNC_HOME="\${DATA_DIR}/.sync-home"
-MANAGED="\${DATA_DIR}/sync-authorized-keys"
-if [ -f "\$MANAGED" ]; then
-  cp "\$MANAGED" "\$SYNC_HOME/.ssh/authorized_keys"
-else
-  touch "\$SYNC_HOME/.ssh/authorized_keys"
+DATA_DIR="${DATA_DIR:-/opt/aifabrix/builder-server/data}"
+SSH_KEYS_DIR="${DATA_DIR}/ssh-keys"
+PENDING_REMOVALS="${DATA_DIR}/pending-removals"
+[ ! -d "$DATA_DIR" ] && exit 0
+if [ -f "$PENDING_REMOVALS" ]; then
+  while IFS= read -r dev_id || [ -n "$dev_id" ]; do
+    dev_id=$(echo "$dev_id" | tr -d '\r\n ')
+    [ -z "$dev_id" ] && continue
+    user_name="dev${dev_id}"
+    getent passwd "$user_name" >/dev/null 2>&1 && ( userdel -r "$user_name" 2>/dev/null || userdel "$user_name" 2>/dev/null || true )
+  done < "$PENDING_REMOVALS"
+  : > "$PENDING_REMOVALS"
 fi
-chown "\$SYNC_USER:\$SYNC_USER" "\$SYNC_HOME/.ssh/authorized_keys"
-chmod 600 "\$SYNC_HOME/.ssh/authorized_keys"
-if [ -d "\${DATA_DIR}/workspace" ]; then
-  chown -R "\$SYNC_USER:\$SYNC_USER" "\${DATA_DIR}/workspace"
+for key_file in "${SSH_KEYS_DIR}"/*/authorized_keys; do
+  [ -f "$key_file" ] || continue
+  dev_id=$(dirname "$key_file" | xargs basename)
+  [ -z "$dev_id" ] && continue
+  user_name="dev${dev_id}"
+  home_dir="/home/${user_name}"
+  workspace_target="${DATA_DIR}/workspace/${user_name}"
+  getent passwd "$user_name" >/dev/null 2>&1 || useradd -m -s /bin/bash "$user_name"
+  mkdir -p "${home_dir}/.ssh"
+  cp "$key_file" "${home_dir}/.ssh/authorized_keys"
+  chown -R "${user_name}:${user_name}" "${home_dir}/.ssh"
+  chmod 700 "${home_dir}/.ssh"
+  chmod 600 "${home_dir}/.ssh/authorized_keys"
+  config_file="${home_dir}/.aifabrix/config.yaml"
+  if [ ! -f "$config_file" ]; then
+    mkdir -p "${home_dir}/.aifabrix"
+    hostname_val=$(hostname -f 2>/dev/null || hostname 2>/dev/null || echo "localhost")
+    printf 'user-mutagen-folder: %s\nsecrets-encryption: ""\naifabrix-secrets: ""\naifabrix-env-config: ""\nremote-server: "http://localhost:3000"\ndocker-endpoint: "tcp://%s:2376"\nsync-ssh-user: "%s"\nsync-ssh-host: "%s"\n' "$workspace_target" "$hostname_val" "$user_name" "$hostname_val" > "$config_file"
+    chown -R "${user_name}:${user_name}" "${home_dir}/.aifabrix"
+    chmod 700 "${home_dir}/.aifabrix"
+    chmod 600 "$config_file"
+  fi
+  mkdir -p "$workspace_target"
+  chown -R "${user_name}:${user_name}" "$workspace_target"
+  [ -L "${home_dir}/workspace" ] && [ "$(readlink -f "${home_dir}/workspace" 2>/dev/null)" != "$(readlink -f "$workspace_target" 2>/dev/null)" ] && rm -f "${home_dir}/workspace"
+  [ ! -e "${home_dir}/workspace" ] && ln -s "$workspace_target" "${home_dir}/workspace" && chown -h "${user_name}:${user_name}" "${home_dir}/workspace"
+  profile="${home_dir}/.profile"
+  [ ! -f "$profile" ] && touch "$profile" && chown "${user_name}:${user_name}" "$profile"
+  grep -q 'cd.*workspace' "$profile" 2>/dev/null || echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$profile"
+  bashrc="${home_dir}/.bashrc"
+  [ ! -f "$bashrc" ] && touch "$bashrc" && chown "${user_name}:${user_name}" "$bashrc"
+  grep -q 'cd.*workspace' "$bashrc" 2>/dev/null || echo '[ -d "$HOME/workspace" ] && cd "$HOME/workspace"' >> "$bashrc"
+done
+APPLY_DEV_EOF
+  fi
+  chmod 755 "$APPLY_DEV_USERS_SCRIPT"
 fi
-APPLY_EOF
-  chmod 755 "$APPLY_KEYS_SCRIPT"
+# Defaults for apply script (aifabrix-secrets, aifabrix-env-config in generated config); match builder-server env.template
+APPLY_DEFAULTS="$DATA_DIR_ABS/apply-dev-users-defaults"
+if [ ! -f "$APPLY_DEFAULTS" ]; then
+  {
+    echo "# Sourced by cron before aifabrix-apply-dev-users.sh; override to match builder-server .env"
+    echo 'export AIFABRIX_SECRETS="${AIFABRIX_SECRETS:-/aifabrix-miso/builder/secrets.local.yaml}"'
+    echo 'export AIFABRIX_ENV_CONFIG="${AIFABRIX_ENV_CONFIG:-aifabrix-miso/builder/env-config.yaml}"'
+  } > "$APPLY_DEFAULTS"
+  chmod 644 "$APPLY_DEFAULTS"
 fi
-if [ -d /etc/cron.d ] && [ ! -f /etc/cron.d/aifabrix-sync-keys ]; then
-  echo "*/2 * * * * root $APPLY_KEYS_SCRIPT" > /etc/cron.d/aifabrix-sync-keys
-  chmod 644 /etc/cron.d/aifabrix-sync-keys
+if [ -d /etc/cron.d ] && [ ! -f /etc/cron.d/aifabrix-apply-dev-users ]; then
+  echo "*/2 * * * * root /bin/sh -c '. $APPLY_DEFAULTS 2>/dev/null; export DATA_DIR=$DATA_DIR_ABS; exec $APPLY_DEV_USERS_SCRIPT'" > /etc/cron.d/aifabrix-apply-dev-users
+  chmod 644 /etc/cron.d/aifabrix-apply-dev-users
 fi
 
 echo "Setup complete. Ensure manual prerequisites (SSL at $SSL_DIR: wildcard.crt, wildcard.key; DNS $DEV_DOMAIN) are done; see SETUP.md."

package/dist/backup.spec.js

@@ -100,7 +100,7 @@ describe('runBackup', () => {
         mockExec.mockResolvedValue({ stdout: '1\n', stderr: '', code: 0 });
         mockReadFile.mockResolvedValue(Buffer.from(''));
         const result = await runBackup({ target: 'user@host', dataDir: '/data' });
-        expect(result).toMatch(/\/aifabrix-backup-\d{8}-\d{4}\.zip$/);
+        expect(result).toMatch(/aifabrix-backup-\d{8}-\d{4}\.zip$/);
         expect(fs.existsSync(result)).toBe(true);
         try {
             fs.unlinkSync(result);

package/dist/cli.js

@@ -9,6 +9,7 @@ import { runBackup, runBackupLocal } from './backup.js';
 import { runBackupScheduleInstall, runBackupScheduleInstallLocal } from './backup-schedule.js';
 import { runRestore, runRestoreLocal } from './restore.js';
 import { runSshCertRequest, runSshCertInstall, runSshCertInstallLocal } from './ssh-cert.js';
+import { runInstallSsh, runInstallSshLocal } from './install-ssh.js';
 import { requireUbuntu } from './ubuntu.js';
 const program = new Command();
 program
@@ -133,6 +134,25 @@ program
         process.exit(1);
     }
 });
+program
+    .command('install-ssh [user@host]')
+    .description('Activate SSH server (install openssh-server, enable and start ssh) without login. Omit target for local.')
+    .option('-i, --identity <path>', 'SSH private key path')
+    .action(async (target, opts) => {
+    try {
+        if (!target || !target.trim()) {
+            requireUbuntu();
+            runInstallSshLocal();
+        }
+        else {
+            await runInstallSsh({ target: target.trim(), privateKeyPath: opts.identity });
+        }
+    }
+    catch (err) {
+        console.error(err instanceof Error ? err.message : err);
+        process.exit(1);
+    }
+});
 const sshCert = program.command('ssh-cert').description('Passwordless auth: install = append your SSH public key; request = stub for future SSH CA.');
 sshCert
     .command('request')

package/dist/install-ssh.d.ts

@@ -0,0 +1,10 @@
+/**
+ * install-ssh: Activate SSH server (openssh-server) without login.
+ * Installs openssh-server if needed, enables and starts the ssh service so the server accepts SSH connections.
+ */
+import { type SSHConnectionOptions } from './ssh.js';
+export declare function runInstallSsh(options: SSHConnectionOptions): Promise<void>;
+/**
+ * Enable and start SSH server locally (no SSH). Call requireUbuntu() before this.
+ */
+export declare function runInstallSshLocal(): void;

package/dist/install-ssh.js

@@ -0,0 +1,39 @@
+/**
+ * install-ssh: Activate SSH server (openssh-server) without login.
+ * Installs openssh-server if needed, enables and starts the ssh service so the server accepts SSH connections.
+ */
+import { execSync } from 'child_process';
+import { createSSHClient, exec, close } from './ssh.js';
+const INSTALL_SSH_SCRIPT = [
+    'wait_for_apt() { i=0; while [ $i -lt 20 ]; do',
+    '  apt-get update -qq 2>/dev/null && return 0;',
+    '  echo "Waiting for apt lock (another process is using apt)..."; sleep 6; i=$((i+1));',
+    '  done; return 1;',
+    '};',
+    'wait_for_apt && DEBIAN_FRONTEND=noninteractive apt-get install -y openssh-server && systemctl enable ssh && systemctl start ssh',
+].join(' ');
+export async function runInstallSsh(options) {
+    const conn = await createSSHClient(options);
+    try {
+        const cmd = `sudo bash -c '${INSTALL_SSH_SCRIPT.replace(/'/g, "'\"'\"'")}'`;
+        const result = await exec(conn, cmd);
+        if (result.stderr)
+            process.stderr.write(result.stderr);
+        if (result.stdout)
+            process.stdout.write(result.stdout);
+        if (result.code !== 0) {
+            throw new Error(`install-ssh exited with code ${result.code}`);
+        }
+        console.log('SSH server is enabled and running. You can connect with key-based or password auth.');
+    }
+    finally {
+        close(conn);
+    }
+}
+/**
+ * Enable and start SSH server locally (no SSH). Call requireUbuntu() before this.
+ */
+export function runInstallSshLocal() {
+    execSync(`sudo bash -c '${INSTALL_SSH_SCRIPT.replace(/'/g, "'\"'\"'")}'`, { stdio: 'inherit' });
+    console.log('SSH server is enabled and running. You can connect with key-based or password auth.');
+}

package/dist/install-ssh.spec.d.ts

@@ -0,0 +1,4 @@
+/**
+ * Tests for install-ssh: remote (mocked SSH) and local (mocked execSync).
+ */
+export {};

package/dist/install-ssh.spec.js

@@ -0,0 +1,55 @@
+/**
+ * Tests for install-ssh: remote (mocked SSH) and local (mocked execSync).
+ */
+const mockConn = {};
+const mockExec = jest.fn();
+const mockClose = jest.fn();
+jest.mock('child_process', () => ({ execSync: jest.fn(() => Buffer.from('')) }));
+jest.mock('./ssh.js', () => ({
+    createSSHClient: jest.fn(() => Promise.resolve(mockConn)),
+    exec: jest.fn((...args) => mockExec(...args)),
+    close: jest.fn((...args) => mockClose(...args)),
+}));
+import { execSync } from 'child_process';
+import { runInstallSsh, runInstallSshLocal } from './install-ssh.js';
+describe('install-ssh', () => {
+    beforeEach(() => {
+        jest.clearAllMocks();
+        mockExec.mockResolvedValue({ stdout: '', stderr: '', code: 0 });
+        jest.spyOn(console, 'log').mockImplementation(() => { });
+    });
+    afterEach(() => {
+        console.log.mockRestore();
+    });
+    describe('runInstallSsh', () => {
+        it('connects, runs sudo apt install openssh-server and systemctl enable/start ssh, then closes', async () => {
+            await runInstallSsh({ target: 'user@host' });
+            expect(mockExec).toHaveBeenCalledTimes(1);
+            const cmd = mockExec.mock.calls[0][1];
+            expect(cmd).toContain('sudo');
+            expect(cmd).toContain('openssh-server');
+            expect(cmd).toContain('systemctl enable ssh');
+            expect(cmd).toContain('systemctl start ssh');
+            expect(mockClose).toHaveBeenCalledWith(mockConn);
+            expect(console.log).toHaveBeenCalledWith(expect.stringContaining('SSH server is enabled and running'));
+        });
+        it('throws when script exits non-zero', async () => {
+            mockExec.mockResolvedValueOnce({ stdout: '', stderr: 'E: Unable to locate package', code: 100 });
+            await expect(runInstallSsh({ target: 'user@host' })).rejects.toThrow(/install-ssh exited with code 100/);
+            expect(mockClose).toHaveBeenCalledWith(mockConn);
+        });
+    });
+    describe('runInstallSshLocal', () => {
+        it('calls execSync with sudo and install script', () => {
+            execSync.mockClear();
+            runInstallSshLocal();
+            expect(execSync).toHaveBeenCalledTimes(1);
+            const cmd = execSync.mock.calls[0][0];
+            expect(cmd).toContain('sudo');
+            expect(cmd).toContain('openssh-server');
+            expect(cmd).toContain('systemctl enable ssh');
+            expect(cmd).toContain('systemctl start ssh');
+            expect(console.log).toHaveBeenCalledWith(expect.stringContaining('SSH server is enabled and running'));
+        });
+    });
+});

package/dist/install.js

@@ -8,13 +8,17 @@ import { fileURLToPath } from 'url';
 import { createSSHClient, exec, writeFile, close, } from './ssh.js';
 const scriptDir = path.dirname(fileURLToPath(import.meta.url));
 const ASSETS_DIR = path.resolve(scriptDir, '..', 'assets');
+/** Normalize to Unix LF so shebang and scripts run on Linux (avoids "No such file or directory" when CRLF). */
+function toUnixLf(s) {
+    return s.replace(/\r\n/g, '\n').replace(/\r/g, '\n');
+}
 function getSetupScript() {
     const p = path.join(ASSETS_DIR, 'setup-dev-server-no-node.sh');
-    return fs.readFileSync(p, 'utf8');
+    return toUnixLf(fs.readFileSync(p, 'utf8'));
 }
 function getNginxTemplate() {
     const p = path.join(ASSETS_DIR, 'builder', 'builder-server', 'nginx-builder-server.conf.template');
-    return fs.readFileSync(p, 'utf8');
+    return toUnixLf(fs.readFileSync(p, 'utf8'));
 }
 export async function runInstall(options) {
     const dataDir = options.dataDir ?? '/opt/aifabrix/builder-server/data';
@@ -31,12 +35,14 @@ export async function runInstall(options) {
         await writeFile(conn, `${tmpDir}/setup.sh`, setupScript);
         await writeFile(conn, `${tmpDir}/builder/builder-server/nginx-builder-server.conf.template`, nginxTemplate);
         await exec(conn, `chmod +x ${tmpDir}/setup.sh`);
+        /** Quote for remote shell so paths with spaces (e.g. Git Bash expanding /opt on Windows) don't break sudo. */
+        const q = (v) => `'${String(v).replace(/'/g, "'\\''")}'`;
         const env = [
-            `REPO_ROOT=${tmpDir}`,
-            `DATA_DIR=${dataDir}`,
-            `DEV_DOMAIN=${devDomain}`,
-            `SSL_DIR=${sslDir}`,
-            `BUILDER_SERVER_PORT=${builderServerPort}`,
+            `REPO_ROOT=${q(tmpDir)}`,
+            `DATA_DIR=${q(dataDir)}`,
+            `DEV_DOMAIN=${q(devDomain)}`,
+            `SSL_DIR=${q(sslDir)}`,
+            `BUILDER_SERVER_PORT=${q(String(builderServerPort))}`,
         ].join(' ');
         const cmd = `sudo ${env} ${tmpDir}/setup.sh`;
         const result = await exec(conn, cmd);

package/dist/ssh.d.ts

@@ -1,6 +1,7 @@
 /**
  * SSH client wrapper using ssh2. Used for install, backup, restore.
  * Never log private keys or sensitive data.
+ * When no private key is given, prompts for password via keyboard-interactive.
  */
 import { Client } from 'ssh2';
 export interface SSHConnectionOptions {

package/dist/ssh.js

@@ -1,10 +1,14 @@
 /**
  * SSH client wrapper using ssh2. Used for install, backup, restore.
  * Never log private keys or sensitive data.
+ * When no private key is given, prompts for password via keyboard-interactive.
  */
 import { Client } from 'ssh2';
 import * as fs from 'fs';
+import * as os from 'os';
 import * as path from 'path';
+import read from 'read';
+const DEFAULT_KEY_NAMES = ['id_ed25519', 'id_rsa'];
 export function parseTarget(target) {
     const at = target.lastIndexOf('@');
     if (at <= 0 || at === target.length - 1) {
@@ -26,18 +30,107 @@ export function createSSHClient(options) {
         }
         privateKey = fs.readFileSync(resolved, 'utf8');
     }
+    else {
+        const sshDir = path.join(os.homedir(), '.ssh');
+        for (const name of DEFAULT_KEY_NAMES) {
+            const keyPath = path.join(sshDir, name);
+            if (fs.existsSync(keyPath)) {
+                try {
+                    privateKey = fs.readFileSync(keyPath, 'utf8');
+                    break;
+                }
+                catch {
+                    // skip unreadable key, try next
+                }
+            }
+        }
+    }
+    const usedDefaultKey = !options.privateKeyPath && !options.privateKey && !!privateKey;
+    function isAuthError(err) {
+        const msg = err.message || '';
+        return (msg.includes('All configured authentication methods failed') ||
+            msg.includes('authentication') ||
+            err.level === 'client-authentication');
+    }
     return new Promise((resolve, reject) => {
-        const conn = new Client();
-        conn
-            .on('ready', () => resolve(conn))
-            .on('error', (err) => reject(err))
-            .connect({
-            host,
-            port,
-            username: user,
-            privateKey: privateKey || undefined,
-            tryKeyboard: !privateKey,
-        });
+        const doConnect = (password, useKey = true) => {
+            const keyForThisConnection = useKey ? privateKey : undefined;
+            const conn = new Client();
+            if (!keyForThisConnection && password !== undefined) {
+                conn.on('keyboard-interactive', (_name, _instructions, _lang, prompts, finish) => {
+                    if (prompts.length === 0) {
+                        finish([]);
+                        return;
+                    }
+                    finish(prompts.map(() => password));
+                });
+            }
+            else if (!keyForThisConnection) {
+                const onKeyboardInteractive = (_name, _instructions, _lang, prompts, finish) => {
+                    if (prompts.length === 0) {
+                        finish([]);
+                        return;
+                    }
+                    const next = (index, answers) => {
+                        if (index >= prompts.length) {
+                            finish(answers);
+                            return;
+                        }
+                        const p = prompts[index];
+                        read({
+                            prompt: p.prompt || (p.echo ? 'Response: ' : 'Password: '),
+                            silent: !p.echo,
+                        }, (err, value) => {
+                            if (err) {
+                                finish(answers);
+                                return;
+                            }
+                            next(index + 1, answers.concat(value || ''));
+                        });
+                    };
+                    next(0, []);
+                };
+                conn.on('keyboard-interactive', onKeyboardInteractive);
+            }
+            conn
+                .on('ready', () => resolve(conn))
+                .on('error', (err) => {
+                if (usedDefaultKey &&
+                    useKey &&
+                    isAuthError(err)) {
+                    read({ prompt: 'Password: ', silent: true }, (errRead, value) => {
+                        if (errRead) {
+                            reject(errRead);
+                            return;
+                        }
+                        doConnect(value || '', false);
+                    });
+                }
+                else {
+                    reject(err);
+                }
+            })
+                .connect({
+                host,
+                port,
+                username: user,
+                privateKey: keyForThisConnection || undefined,
+                password: !keyForThisConnection ? password : undefined,
+                tryKeyboard: !keyForThisConnection,
+            });
+        };
+        if (!privateKey) {
+            read({ prompt: 'Password: ', silent: true }, (err, value) => {
+                if (err) {
+                    reject(err);
+                    return;
+                }
+                doConnect(value || '');
+            });
+        }
+        else {
+            doConnect(undefined);
+        }
     });
 }
 export function exec(conn, command) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aifabrix/server-setup",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "description": "CLI to install, backup, and restore AI Fabrix builder-server (config + DB) over SSH",
   "type": "module",
   "main": "dist/cli.js",
@@ -24,6 +24,7 @@
     "better-sqlite3": "^11.6.0",
     "commander": "^12.1.0",
     "extract-zip": "^2.0.1",
+    "read": "^1.0.7",
     "ssh2": "^1.15.0"
   },
   "devDependencies": {
@@ -32,6 +33,7 @@
     "@types/better-sqlite3": "^7.6.11",
     "@types/jest": "^29.5.14",
     "@types/node": "^22.10.2",
+    "@types/read": "^0.0.32",
     "@types/ssh2": "^1.15.5",
     "@typescript-eslint/eslint-plugin": "^8.17.0",
     "@typescript-eslint/parser": "^8.17.0",