@aifabrix/server-setup 1.1.0 → 1.2.0

package/README.md

@@ -1,68 +1,131 @@
-# @aifabrix/server-setup (af-server)
+# AI Fabrix Builder Server Setup (af-server)
 
-CLI to install, backup, and restore AI Fabrix builder-server over SSH. Runs on your PC or CI; the server does **not** need Node.js or the aifabrix Builder.
+This is the **one document** you need to get your builder-server **from zero to up and running** in your own Docker. How to use the server (users, secrets, onboarding, etc.) is in the **AI Fabrix Builder CLI** documentation—full open source: [github.com/esystemsdev/aifabrix-builder](https://github.com/esystemsdev/aifabrix-builder).
 
-- **Backup** = configuration + database only (no workspace or developer code).
-- **Restore** = push backup zip (builder.db + keys) back to server DATA_DIR.
+---
 
-## Commands
+## Why AI Fabrix Builder server?
 
-Omit `user@host` to run **locally** on the current machine (Ubuntu only). With `user@host`, commands run over SSH.
+- **Full AI Fabrix platform, one server, multi-developer:** You run the full AI Fabrix platform and a **separate builder-server** where multiple developers each get their **own dedicated, isolated environment** (users, certificates, secrets, workspaces). One installer (af-server) sets up Docker, nginx, SSL proxy, sync user, and backup on that server.
+- **One tool for install and management:** The same **AI Fabrix Builder** CLI gets you the platform and manages users, secrets, and onboarding. All how-to in [Builder CLI docs](https://github.com/esystemsdev/aifabrix-builder).
+- **Security and standards:** TLS, client certificates, no secrets in version control. Backups are explicit (config + DB + keys); store them encrypted.
 
-```bash
-# Install server (Docker, nginx, SSL proxy, sync user, cron).
-af-server install [ user@host ] [ -d DATA_DIR ] [ --dev-domain DOMAIN ] [ --ssl-dir PATH ] [ -i SSH_KEY ]
+---
 
-# On-demand backup: fetch config + DB + keys from server (or local DATA_DIR), save zip.
-af-server backup [ user@host ] [ -d DATA_DIR ] [ -o output.zip ] [ -i SSH_KEY ]
+## Getting the builder-server Docker image
 
-# Install cron backup (daily 02:00, keep last 7).
-af-server backup [ user@host ] --schedule [ --backup-dir PATH ] [ --keep-days N ] [ -i SSH_KEY ]
+**The builder-server Docker image is not on a public registry.** You get it with the **AI Fabrix platform**.
 
-# Restore backup zip to DATA_DIR.
-af-server restore backup.zip [ user@host ] [ -d DATA_DIR ] [ --force ] [ -i SSH_KEY ]
+1. Install the **AI Fabrix Builder** CLI: `npm install -g @aifabrix/builder`
+2. Use the CLI to run or deploy the platform (including builder-server). See [AI Fabrix Builder](https://github.com/esystemsdev/aifabrix-builder) and [docs/README.md](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/README.md).
 
-# ssh-cert: install = append your local SSH public key for passwordless auth; request = stub for future SSH CA.
-af-server ssh-cert request [ --user ID ] [ -i SSH_KEY ]
-af-server ssh-cert install [ user@host ] [ -i SSH_KEY ]
-```
+Once you have the platform (and the builder-server image), use **af-server** to install that server on your own host.
 
-**ssh-cert install:** Appends your **local SSH public key** (from `~/.ssh/id_ed25519.pub` or `id_rsa.pub`, or from `-i key` → `key.pub`) to the server user's `~/.ssh/authorized_keys`. After that, install/backup/restore work without password when using that key or ssh-agent. First run may use password or `-i`. **request** remains a stub for future SSH CA integration.
+---
 
 ## Prerequisites
 
-- Node.js >= 18
-- SSH access to the server (key-based auth recommended; use `-i path/to/key`)
+- **Node.js ≥ 18** (where you run af-server)
+- **SSH access** to the server (key-based auth). af-server uses `~/.ssh/id_ed25519.pub` or `~/.ssh/id_rsa.pub` by default; use `-i path/to/key` if your key is elsewhere.
+- **Server:** Ubuntu. For remote install the script can install Docker; you provide domain, SSL directory, and certificates (next section).
 
-The server should have **builder-server** data in either:
+---
 
-- **SQLite** (`DATA_DIR/builder.db`) — after JSON→SQLite migration (see repo plan 006); backup/restore and cron backup use this.
-- **JSON files** — legacy; on-demand backup still works (exports to SQLite inside the zip).
+## Manual prerequisites (before install)
 
-## Backup contents
+Do the steps in [What you must do before running af-server](#what-you-must-do-before-running-af-server) (below): Ubuntu version, root/sudo, SSH, DNS (CNAME or A), SSL directory, and certificate + key. The install script does **not** create or obtain certificates; it configures nginx to use the files you provide.
 
-- `config.json` — DATA_DIR, createdAt, source (builder.db or json)
-- `backup.db` — SQLite (either copy of builder.db or export from JSON)
-- `ca.crt`, `ca.key`, `secrets-encryption.key` — for restore
+---
 
-**Security:** Backups contain secrets. Store them encrypted at rest and restrict filesystem access. Never log key material.
+## Install: from zero to running
 
-## Cron backup (--schedule)
+**From your PC** (SSH to server):
 
-Installs a shell script on the server that runs daily at 02:00. Requires `builder.db` (SQLite) and `zip` on the server. Keeps the last N backups (default 7). Backup dir default: `/opt/aifabrix/backups`.
+```bash
+af-server install user@host
+```
 
-## Build and run locally
+**On the server itself** (Ubuntu, no target):
 
 ```bash
-cd server-setup
-npm install
-npm run build
-node dist/cli.js --help
-# or: npm link && af-server --help
+af-server install
 ```
 
-## References
+After install, your builder-server is up. Use the **AI Fabrix Builder CLI** for all usage (users, secrets, certs, etc.)—see [Builder documentation](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/developer-isolation.md).
+
+---
+
+## Commands
+
+| Command | Description |
+| -------- | ----------- |
+| `af-server install [ user@host ] [ -d DATA_DIR ] [ --dev-domain DOMAIN ] [ --ssl-dir PATH ] [ -i SSH_KEY ]` | Install: Docker, nginx, SSL proxy, sync user, cron. |
+| `af-server backup [ user@host ] [ -d DATA_DIR ] [ -o output.zip ] [ -i SSH_KEY ]` | On-demand backup (config + DB + keys). |
+| `af-server backup [ user@host ] --schedule [ --backup-dir PATH ] [ --keep-days N ] [ -i SSH_KEY ]` | Cron backup (daily 02:00, keep last N, default 7). |
+| `af-server restore backup.zip [ user@host ] [ -d DATA_DIR ] [ --force ] [ -i SSH_KEY ]` | Restore backup to DATA_DIR. |
+| `af-server ssh-cert install [ user@host ] [ -i SSH_KEY ]` | Add your SSH public key to server (passwordless auth). |
+
+Backups contain secrets; store encrypted. Cron backup needs SQLite (`builder.db`) and `zip` on the server; default backup dir: `/opt/aifabrix/backups`.
+
+---
+
+## Documentation
+
+All **how to use** the builder-server (users, secrets, onboarding, developer isolation, etc.) is in the open-source **AI Fabrix Builder** docs:
+
+- [AI Fabrix Builder](https://github.com/esystemsdev/aifabrix-builder) — repo and README  
+- [docs/README.md](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/README.md) — full documentation index  
+
+This README is only: **zero to up and running your builder server**.
+
+---
+
+## What you must do before running af-server
+
+Complete these **manually** before `af-server install`. The install script does not create DNS, certificates, or the admin account; it expects them to exist.
+
+**Server and access**
+
+- **Ubuntu** — 22.04 LTS or later (recommended). Other Debian-based systems may work but are not guaranteed.
+- **Root or sudo** — The install script must run with root (or sudo). From your PC you SSH as a user that can `sudo` to root; on the server you run `sudo af-server install` (or as root).
+- **SSH access** — You need key-based or password SSH to the server so you can run `af-server install user@host`, or you run install locally on the server.
+
+**DNS (CNAME or A record)**
+
+- Create a **DNS A record** or **CNAME** so your chosen domain points to the server’s IP.
+- Default domain used by the script: `builder01.aifabrix.dev`. To use another domain, set `--dev-domain` or env `DEV_DOMAIN`.
+- Ensure DNS has propagated before install (nginx will use this domain).
+
+**SSL directory and certificates**
+
+- Create the SSL directory (default `/opt/aifabrix/ssl`):  
+  `sudo mkdir -p /opt/aifabrix/ssl`  
+  To use another path, set `--ssl-dir` or env `SSL_DIR`.
+- **Obtain** a TLS certificate and private key (e.g. Let’s Encrypt, internal CA, or purchased). Full chain if applicable.
+- **Place** in the SSL directory:
+  - `wildcard.crt` — certificate (full chain if applicable)
+  - `wildcard.key` — private key  
+  Set permissions: `chmod 600 /opt/aifabrix/ssl/wildcard.key`  
+  If your files have other names (e.g. `fullchain.pem` / `privkey.pem`), symlink them to `wildcard.crt` and `wildcard.key`.
+- The install script **does not** create or obtain certificates; it only configures nginx to use these files.
+
+**Admin user (optional but typical)**  
+If you SSH as a non-root user, that user must be able to sudo. The script will add an admin user (default `serveradmin`) to the `docker` group and grant it passwordless sudo. Ensure that user exists on the server if you rely on it, or set `SETUP_ADMIN_USER` to your SSH user.
+
+---
+
+## High level: what the install does on your Ubuntu server
+
+When you run `af-server install`, the script (as root) does the following on the server:
+
+- **System** — `apt update` and `apt upgrade`; optional hostname change if `SETUP_HOSTNAME` is set.
+- **Docker** — Installs Docker if missing; enables and starts the Docker service.
+- **Admin user** — Adds the admin user (default `serveradmin`) to the `docker` group and grants passwordless sudo (`/etc/sudoers.d/<admin>`).
+- **Nginx** — Installs nginx if missing; enables and starts it. Generates a site config for your domain that proxies HTTPS to the builder-server container (using `wildcard.crt` and `wildcard.key` from your SSL dir).
+- **Builder-server data dir** — Creates the data directory (default `/opt/aifabrix/builder-server/data`), sets ownership for the container. Starts the `builder-server` container if the image already exists on the server; otherwise prints how to build and run it (you get the image from the AI Fabrix platform).
+- **Sync user** — Creates a system user (default `aifabrix-sync`) for Mutagen SSH sync; home under the data dir; creates `.ssh` and `authorized_keys`.
+- **Cron job** — Installs a cron job (every 2 minutes) that copies the managed `authorized_keys` file into the sync user’s `.ssh/authorized_keys`.
+- **Mutagen** — Downloads and installs the Mutagen binary; creates a systemd service and starts the daemon.
+- **Optional** — If `INSTALL_PORTAINER=1`, installs the Portainer container. If Docker TLS is not skipped, writes `/etc/docker/daemon.json`; you can use the **same certificate** from `/opt/aifabrix/ssl` (e.g. symlink or copy `wildcard.crt` and `wildcard.key` to the paths Docker expects: `/etc/docker/server-cert.pem`, `/etc/docker/server-key.pem`, and if needed `ca.pem` for the CA), or provide separate Docker TLS certs.
 
-- [SETUP.md](../SETUP.md) — server installation and manual prerequisites
-- [builder/builder-server/README.md](../builder/builder-server/README.md) — builder-server data dir and mounts
-- [AI Fabrix Builder CLI](https://github.com/esystemsdev/aifabrix-builder/blob/main/docs/CLI-REFERENCE.md) — for onboarding and certs (talks to builder-server API)
+Result: your Ubuntu server has Docker, nginx (HTTPS for your domain), the builder-server container (if image present), the sync user and key sync, and Mutagen—ready for the Builder CLI to use.

package/assets/builder/builder-server/nginx-builder-server.conf.template

@@ -2,6 +2,10 @@
 # Generated from template by af-server install (substitutes DEV_DOMAIN, SSL_DIR, BUILDER_SERVER_PORT, DATA_DIR).
 # SSL cert and key from SSL_DIR_PLACEHOLDER (wildcard.crt, wildcard.key).
 # Client cert: ssl_client_certificate uses DATA_DIR_PLACEHOLDER/ca.crt (Builder CA; created by builder-server on first run).
+# If the backend returns 400 Bad Request for requests with client cert, nginx may be sending $ssl_client_cert with
+# literal newlines (header folding). Use njs to send cert on one line: load_module modules/ngx_http_js_module.so;
+# then js_set $client_cert_escaped cert.oneline; js_include cert-escaped.js; and proxy_set_header X-Client-Cert $client_cert_escaped;
+# (cert-escaped.js: function cert(r){return r.variables.ssl_client_cert?r.variables.ssl_client_cert.replace(/\n/g,'\\n'):'';}).
 # Reload nginx after placing: sudo nginx -t && sudo systemctl reload nginx.
 
 server {

package/assets/setup-dev-server-no-node.sh

@@ -116,6 +116,11 @@ if [ ! -f "$NGINX_CONF" ] && [ -f "$NGINX_TEMPLATE" ]; then
 elif [ ! -f "$NGINX_CONF" ]; then
   echo "Warning: SSL prereqs required. Place $NGINX_CONF (see SETUP.md) and ensure $SSL_DIR/wildcard.crt and $SSL_DIR/wildcard.key exist."
 fi
+if command -v nginx >/dev/null 2>&1; then
+  if nginx -t 2>/dev/null; then
+    systemctl reload nginx
+  fi
+fi
 
 # --- Mutagen ---
 if ! command -v mutagen >/dev/null 2>&1; then
@@ -166,8 +171,10 @@ DOCKER_EOF
 fi
 
 # --- Builder-server data dir and container ---
+# Container runs as uid 1001 (nodejs); data dir must be writable for bootstrap (key, CA, DB).
 mkdir -p "$DATA_DIR"
-chown 1001:65533 "$DATA_DIR"
+chown -R 1001:65533 "$DATA_DIR"
+chmod 755 "$DATA_DIR"
 if command -v docker >/dev/null 2>&1; then
   if ! docker ps -a --format '{{.Names}}' 2>/dev/null | grep -q '^builder-server$'; then
     echo "Builder-server container not found. Build and run manually (see builder/builder-server/README.md):"

package/dist/backup-db.spec.js

@@ -77,6 +77,14 @@ describe('backup-db', () => {
             expect(row).toEqual({ id: '03', name: 'C' });
             db.close();
         });
+        it('imports no users when users.json is empty object (no .users key, not array)', () => {
+            const outPath = path.join(tmpDir, 'empty-users.db');
+            createBackupDbFromJson(outPath, { users: '{}' }, config);
+            const db = new Database(outPath, { readonly: true });
+            const count = db.prepare('SELECT COUNT(*) as n FROM users').get();
+            expect(count.n).toBe(0);
+            db.close();
+        });
         it('imports pins from tokens.json shape', () => {
             const outPath = path.join(tmpDir, 'out.db');
             const tokensJson = JSON.stringify({
@@ -129,6 +137,14 @@ describe('backup-db', () => {
             expect(row.user_id).toBe('03');
             db.close();
         });
+        it('imports no pins when tokens.json is empty object (no .pins key, not array)', () => {
+            const outPath = path.join(tmpDir, 'empty-pins.db');
+            createBackupDbFromJson(outPath, { tokens: '{}' }, config);
+            const db = new Database(outPath, { readonly: true });
+            const count = db.prepare('SELECT COUNT(*) as n FROM pin_tokens').get();
+            expect(count.n).toBe(0);
+            db.close();
+        });
         it('imports secrets from secrets.json shape (camelCase)', () => {
             const outPath = path.join(tmpDir, 'out.db');
             const secretsJson = JSON.stringify({
@@ -159,6 +175,21 @@ describe('backup-db', () => {
             expect(row.auth_tag).toBe('tag2');
             db.close();
         });
+        it('skips secret entries with null or invalid shape', () => {
+            const outPath = path.join(tmpDir, 'secrets-skip.db');
+            const secretsJson = JSON.stringify({
+                secrets: {
+                    valid: { iv: 'i', auth_tag: 't', cipher: 'c' },
+                    nullEntry: null,
+                    invalid: { iv: 'i' },
+                },
+            });
+            createBackupDbFromJson(outPath, { secrets: secretsJson }, config);
+            const db = new Database(outPath, { readonly: true });
+            const rows = db.prepare('SELECT key FROM secrets').all();
+            expect(rows.map((r) => r.key)).toEqual(['valid']);
+            db.close();
+        });
         it('imports ssh_keys from byUser shape', () => {
             const outPath = path.join(tmpDir, 'out.db');
             const sshKeysJson = JSON.stringify({
@@ -213,6 +244,20 @@ describe('backup-db', () => {
             expect(count.n).toBe(0);
             db.close();
         });
+        it('imports ssh_keys with createdAt (camelCase) only', () => {
+            const outPath = path.join(tmpDir, 'ssh-createdAt.db');
+            const sshKeysJson = JSON.stringify({
+                byUser: {
+                    '01': [{ publicKey: 'ssh-rsa AAAA', label: 'l', fingerprint: 'f', createdAt: '2025-06-01T00:00:00Z' }],
+                },
+            });
+            createBackupDbFromJson(outPath, { sshKeys: sshKeysJson }, config);
+            const db = new Database(outPath, { readonly: true });
+            const row = db.prepare('SELECT user_id, created_at FROM ssh_keys').get();
+            expect(row.user_id).toBe('01');
+            expect(row.created_at).toBe('2025-06-01T00:00:00Z');
+            db.close();
+        });
     });
     describe('copyBuilderDbAsBackup', () => {
         it('copies users and pin_tokens from source db to backup', () => {

package/dist/backup.spec.js

@@ -183,6 +183,23 @@ describe('runBackupLocal', () => {
         const config = JSON.parse(fs.readFileSync(path.join(tmpExtract, CONFIG_JSON), 'utf8'));
         expect(config.source).toBe('json');
     });
+    it('uses fallback JSON when a local JSON file is missing (catch path)', async () => {
+        fs.writeFileSync(path.join(dataDir, 'users.json'), '{"users":[]}');
+        fs.writeFileSync(path.join(dataDir, 'tokens.json'), '{"pins":[]}');
+        // omit secrets.json and ssh-public-keys.json so readFileSync throws and fallback is used
+        const outPath = path.join(dataDir, 'fallback.zip');
+        const result = await runBackupLocal({ dataDir, outputPath: outPath });
+        expect(fs.existsSync(result)).toBe(true);
+        const tmpExtract = path.join(dataDir, 'extract-fallback');
+        fs.mkdirSync(tmpExtract, { recursive: true });
+        await extract(result, { dir: tmpExtract });
+        const db = new Database(path.join(tmpExtract, BACKUP_DB), { readonly: true });
+        const userCount = db.prepare('SELECT COUNT(*) as n FROM users').get();
+        const secretCount = db.prepare('SELECT COUNT(*) as n FROM secrets').get();
+        expect(userCount.n).toBe(0);
+        expect(secretCount.n).toBe(0);
+        db.close();
+    });
     it('includes key files when present in dataDir', async () => {
         const dbPath = path.join(dataDir, BUILDER_DB);
         const db = new Database(dbPath);

package/dist/restore.spec.js

@@ -212,4 +212,36 @@ describe('runRestoreLocal', () => {
         fs.writeFileSync(path.join(dataDir, BUILDER_DB), 'existing');
         await expect(runRestoreLocal({ zipPath, dataDir })).rejects.toThrow('DATA_DIR already has builder.db');
     });
+    it('uses config.dataDir from zip when options.dataDir not provided', async () => {
+        const zipContentDir = path.join(workDir, 'content');
+        fs.mkdirSync(zipContentDir, { recursive: true });
+        const dbPath = path.join(zipContentDir, BACKUP_DB);
+        const db = new Database(dbPath);
+        db.exec('CREATE TABLE users (id TEXT PRIMARY KEY); INSERT INTO users (id) VALUES (\'1\');');
+        db.close();
+        const configFromZip = { dataDir: path.join(workDir, 'restore-from-config'), createdAt: new Date().toISOString(), source: 'builder.db' };
+        fs.writeFileSync(path.join(zipContentDir, CONFIG_JSON), JSON.stringify(configFromZip));
+        for (const k of KEY_FILES) {
+            fs.writeFileSync(path.join(zipContentDir, k), `content-${k}`);
+        }
+        await new Promise((resolve, reject) => {
+            const archive = archiver('zip', { zlib: { level: 6 } });
+            const out = fs.createWriteStream(zipPath);
+            out.on('close', () => resolve());
+            archive.on('error', reject);
+            archive.pipe(out);
+            archive.file(dbPath, { name: BACKUP_DB });
+            archive.file(path.join(zipContentDir, CONFIG_JSON), { name: CONFIG_JSON });
+            for (const k of KEY_FILES) {
+                archive.file(path.join(zipContentDir, k), { name: k });
+            }
+            archive.finalize();
+        });
+        await runRestoreLocal({ zipPath, force: true });
+        expect(fs.existsSync(path.join(configFromZip.dataDir, BUILDER_DB))).toBe(true);
+        const restoredDb = new Database(path.join(configFromZip.dataDir, BUILDER_DB), { readonly: true });
+        const user = restoredDb.prepare('SELECT id FROM users').get();
+        expect(user.id).toBe('1');
+        restoredDb.close();
+    });
 });

package/dist/ssh-cert.spec.js

@@ -64,6 +64,13 @@ describe('ssh-cert', () => {
             await expect(runSshCertInstall({ target: 'user@host' })).rejects.toThrow('Could not read authorized_keys');
             expect(mockClose).toHaveBeenCalledWith(mockConn);
         });
+        it('appends newline before new key when existing content does not end with newline', async () => {
+            mockReadFile.mockResolvedValueOnce(Buffer.from('existing key line', 'utf8'));
+            await runSshCertInstall({ target: 'user@host' });
+            const written = mockWriteFile.mock.calls[0][2].toString('utf8');
+            expect(written).toMatch(/existing key line\n.*# af-server/);
+            expect(written).toContain(KEY_LINE);
+        });
     });
     describe('runSshCertInstallLocal', () => {
         let fakeHome;
@@ -97,5 +104,15 @@ describe('ssh-cert', () => {
             expect(afterSecond).toBe(afterFirst);
             expect(console.log).toHaveBeenCalledWith('Key already installed.');
         });
+        it('appends newline before new key when authorized_keys exists and does not end with newline', () => {
+            const sshDir = path.join(fakeHome, '.ssh');
+            fs.mkdirSync(sshDir, { recursive: true });
+            const authKeysPath = path.join(sshDir, 'authorized_keys');
+            fs.writeFileSync(authKeysPath, 'existing key line', { mode: 0o600 });
+            runSshCertInstallLocal(undefined, { homedir: fakeHome });
+            const content = fs.readFileSync(authKeysPath, 'utf8');
+            expect(content).toMatch(/existing key line\n.*# af-server/);
+            expect(content).toContain(KEY_LINE);
+        });
     });
 });

package/dist/ubuntu.spec.js

@@ -53,4 +53,13 @@ describe('ubuntu', () => {
         Object.defineProperty(process, 'platform', { value: 'linux', configurable: true });
         expect(() => requireUbuntu({ osReleasePath: path.join(TEST_TMP, 'nonexistent') })).toThrow(/Local mode is supported only on Ubuntu/);
     });
+    it('uses default os-release path when options is empty object', () => {
+        Object.defineProperty(process, 'platform', { value: 'linux', configurable: true });
+        try {
+            requireUbuntu({});
+        }
+        catch (e) {
+            expect(e.message).toMatch(/Local mode is supported only on Ubuntu/);
+        }
+    });
 });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aifabrix/server-setup",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "description": "CLI to install, backup, and restore AI Fabrix builder-server (config + DB) over SSH",
   "type": "module",
   "main": "dist/cli.js",