gitops-ai 1.1.0 → 1.2.0

package/README.md

@@ -1,5 +1,7 @@
 # GitOps AI - Bootstrapper
 
+[![Website](https://img.shields.io/badge/website-gitops--ai.vercel.app-blue)](https://gitops-ai.vercel.app) [![Docs](https://img.shields.io/badge/docs-gitops--ai.vercel.app-orange)](https://gitops-ai.vercel.app/#docs/prerequisites)
+
 GitOps-managed Kubernetes infrastructure for AI-powered applications powered by the [Flux Operator](https://fluxoperator.dev/) and [Flux CD](https://fluxcd.io/). A single bootstrap application provisions a Kubernetes cluster, installs all infrastructure components, and enables continuous delivery from Git.
 
 ## Why GitOps for your infrastructure
@@ -14,7 +16,8 @@ GitOps-managed Kubernetes infrastructure for AI-powered applications powered by
 
 ## Quick Start
 
-Run on our macOS machine:
+On your Mac or Linux machine:
+
 ```bash
 npx gitops-ai bootstrap
 ```
@@ -31,17 +34,17 @@ Or, if you already have Node.js >= 18:
 npx gitops-ai bootstrap
 ```
 
-The interactive wizard will prompt for your GitLab PAT, fork the template into your namespace, and run the full bootstrap.
+The interactive wizard will prompt for your Git provider (GitHub or GitLab), create or use a repository from the [GitOps AI Template](https://gitlab.com/everythings-gonna-be-alright/gitops_ai_template), and run the full bootstrap.
 
 ## Requirements
 
-| Resource       | Minimum                |
-|----------------|------------------------|
-| **CPU**        | 2+ cores               |
-| **Memory**     | 4+ GB                  |
-| **Disk**       | 20+ GB free            |
-| **OS**         | Ubuntu 25.04+ or macOS |
-| **Node.js**    | 18+ (installed automatically by `install.sh`) |
+| Resource    | Minimum                                       |
+|-------------|-----------------------------------------------|
+| **CPU**     | 2+ cores                                      |
+| **Memory**  | 4+ GB                                         |
+| **Disk**    | 20+ GB free                                   |
+| **OS**      | Ubuntu 25.04+ or macOS                        |
+| **Node.js** | 18+ (installed automatically by `install.sh`) |
 
 You will also need a [GitLab PAT](docs/prerequisites.md#1-gitlab-personal-access-token), a [Cloudflare API Token](docs/prerequisites.md#2-cloudflare-api-token) (if using automatic DNS/TLS), and an [OpenAI API Key](docs/prerequisites.md#3-openai-api-key) (if using OpenClaw). See [Prerequisites](docs/prerequisites.md) for full details.
 
@@ -67,9 +70,21 @@ Keeping the template in a separate repository means:
 - **Clean separation** -- the bootstrapper CLI handles provisioning logic; the template holds pure infrastructure declarations. Each can be versioned and tested independently.
 - **Customisation without lock-in** -- after the fork you own the repo. Add namespaces, swap Helm charts, or restructure directories to fit your needs.
 
+### Repository layout (template → your repo)
+
+The upstream template (and your bootstrapped repo) is organised roughly as:
+
+| Path                     | Role                                                                                                                                                                                        |
+|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `templates/<category>/…` | Shared Helm bases and component manifests (e.g. `templates/system/`, `templates/ai/`, `templates/monitoring/`). |
+| `clusters/_template/`    | Prototype cluster layout; the CLI copies this to `clusters/<your-cluster-name>/` during bootstrap.            |
+| `clusters/<name>/`       | Your live cluster overlay (`cluster-sync.yaml`, `components/`, encrypted secrets).                                                                                                          |
+
+See [Architecture](docs/architecture.md) for diagrams and a fuller tree.
+
 ## CLI Commands
 
-The CLI provides three commands:
+The CLI provides these commands:
 
 ### `bootstrap` (alias: `install`)
 
@@ -89,16 +104,16 @@ SOPS secret encryption management. Run without arguments for an interactive menu
 npx gitops-ai sops [subcommand] [file]
 ```
 
-| Subcommand       | Description                                              |
-|------------------|----------------------------------------------------------|
+| Subcommand       | Description                                                            |
+|------------------|------------------------------------------------------------------------|
 | `init`           | First-time setup: generate age key, create `.sops.yaml` and K8s secret |
-| `encrypt`        | Encrypt all unencrypted secret files                     |
-| `encrypt <file>` | Encrypt a specific file                                  |
-| `decrypt <file>` | Decrypt a file for viewing (re-encrypt before commit)    |
-| `edit <file>`    | Open encrypted file in `$EDITOR` (auto re-encrypts on save) |
-| `status`         | Show encryption status of all secret files               |
-| `import`         | Import an existing age key into a new cluster            |
-| `rotate`         | Rotate to a new age key and re-encrypt everything        |
+| `encrypt`        | Encrypt all unencrypted secret files                                   |
+| `encrypt <file>` | Encrypt a specific file                                                |
+| `decrypt <file>` | Decrypt a file for viewing (re-encrypt before commit)                  |
+| `edit <file>`    | Open encrypted file in `$EDITOR` (auto re-encrypts on save)            |
+| `status`         | Show encryption status of all secret files                             |
+| `import`         | Import an existing age key into a new cluster                          |
+| `rotate`         | Rotate to a new age key and re-encrypt everything                      |
 
 ### `openclaw-pair`
 
@@ -108,6 +123,18 @@ Pair an OpenClaw device with the cluster after bootstrap:
 npx gitops-ai openclaw-pair
 ```
 
+### `template sync`
+
+Fetch the upstream GitOps template and merge changes into your current branch. Run without flags for an **interactive wizard** (tag picker, diff preview with risk classification, merge/dry-run/cancel), or pass flags for non-interactive use:
+
+```bash
+npx gitops-ai template sync                        # interactive wizard
+npx gitops-ai template sync --ref v1.0.0           # non-interactive merge
+npx gitops-ai template sync --ref main --dry-run   # non-interactive preview
+```
+
+See [Template synchronization](docs/template-sync.md).
+
 ## Components
 
 The bootstrap wizard lets you select which components to install:
@@ -128,23 +155,27 @@ Components marked **DNS/TLS** are automatically enabled when you opt into automa
 
 ## Documentation
 
-| Document | Description |
-|----------|-------------|
-| [Prerequisites](docs/prerequisites.md) | API tokens, Docker runtime, network requirements |
-| [Bootstrap](docs/bootstrap.md) | What the bootstrap does, wizard walkthrough, resume capability |
-| [Architecture](docs/architecture.md) | Repository structure, Flux Operator, GitOps workflow |
-| [Configuration](docs/configuration.md) | Cluster variables, environment variables, post-bootstrap changes |
+| Document                                          | Description                                                         |
+|---------------------------------------------------|---------------------------------------------------------------------|
+| [Prerequisites](docs/prerequisites.md)            | Node.js, Docker (macOS), Git provider, optional Cloudflare / OpenAI |
+| [Bootstrap](docs/bootstrap.md)                    | What the bootstrap does, wizard walkthrough, resume capability      |
+| [Architecture](docs/architecture.md)              | Repositories, bootstrap flow, Flux Operator & Instance, repo tree   |
+| [Configuration](docs/configuration.md)            | Cluster variables, SOPS defaults, post-bootstrap changes            |
+| [Template synchronization](docs/template-sync.md) | Upstream merges, `template sync`, CI parity, risk tiers             |
+| [Scaling](docs/scaling.md)                        | Adding k3s worker and server nodes (Linux)                          |
+| [Security](docs/security.md)                      | SOPS, Git auth, hardening, network                                  |
 
 ## Development
 
 ```bash
-git clone <repo-url> && cd gitops-ai
+git clone https://gitlab.com/everythings-gonna-be-alright/gitops_ai_bootstrapper.git
+cd gitops_ai_bootstrapper
 npm install
 
 npm run dev              # Run CLI locally via tsx
 npm run build            # Compile TypeScript to dist/
 npm run typecheck        # Type-check without emitting
-npm run test:validate    # Validate Flux build against template
+npm run test:sync        # Unit tests for template sync logic
 npm run test:integration # Full k3d + Flux integration test (requires Docker)
 ```
 

package/dist/commands/bootstrap.js

@@ -1,5 +1,6 @@
 import { existsSync } from "node:fs";
 import { execSync } from "node:child_process";
+import { networkInterfaces } from "node:os";
 import { resolve } from "node:path";
 import * as p from "@clack/prompts";
 import pc from "picocolors";
@@ -13,6 +14,7 @@ import * as k8s from "../core/kubernetes.js";
 import * as flux from "../core/flux.js";
 import { getProvider, } from "../core/git-provider.js";
 import { COMPONENTS, REQUIRED_COMPONENT_IDS, DNS_TLS_COMPONENT_IDS, MONITORING_COMPONENT_IDS, OPTIONAL_COMPONENTS, SOURCE_GITLAB_HOST, SOURCE_PROJECT_PATH, } from "../schemas.js";
+import { fetchTemplateTags, readPackageVersion } from "../core/template-sync.js";
 import { stepWizard, back, maskSecret, } from "../utils/wizard.js";
 import { loginAndCreateCloudflareToken } from "../core/cloudflare-oauth.js";
 // ---------------------------------------------------------------------------
@@ -40,20 +42,7 @@ function openclawEnabled(state) {
 function componentLabel(id) {
     return COMPONENTS.find((c) => c.id === id)?.label ?? id;
 }
-async function fetchTemplateTags() {
-    const encoded = encodeURIComponent(SOURCE_PROJECT_PATH);
-    const url = `https://${SOURCE_GITLAB_HOST}/api/v4/projects/${encoded}/repository/tags?per_page=50&order_by=version&sort=desc`;
-    try {
-        const res = await fetch(url);
-        if (!res.ok)
-            return [];
-        const tags = (await res.json());
-        return tags.map((t) => t.name);
-    }
-    catch {
-        return [];
-    }
-}
+// fetchTemplateTags is imported from core/template-sync.ts
 function providerLabel(type) {
     return type === "gitlab" ? "GitLab" : "GitHub";
 }
@@ -447,6 +436,8 @@ function buildFields(detectedIp, hasSavedPlan) {
                     hint: "Victoria Metrics + Grafana Operator (dashboards, alerting, metrics)",
                 };
                 const hasMonitoring = MONITORING_COMPONENT_IDS.every((id) => state.selectedComponents.includes(id));
+                const monitoringExplicitlyRemoved = !hasMonitoring
+                    && state.selectedComponents.some((id) => !REQUIRED_COMPONENT_IDS.includes(id));
                 const selected = await p.multiselect({
                     message: pc.bold("Optional components to install"),
                     options: [
@@ -458,7 +449,7 @@ function buildFields(detectedIp, hasSavedPlan) {
                         })),
                     ],
                     initialValues: [
-                        ...(hasMonitoring ? [MONITORING_GROUP_ID] : []),
+                        ...((hasMonitoring || !monitoringExplicitlyRemoved) ? [MONITORING_GROUP_ID] : []),
                         ...state.selectedComponents.filter((id) => OPTIONAL_COMPONENTS.some((c) => c.id === id)),
                     ],
                     required: false,
@@ -640,7 +631,10 @@ function buildFields(detectedIp, hasSavedPlan) {
                         `  3. Name it (e.g. ${pc.cyan("gitops-ai")})\n` +
                         `  4. Copy the key value\n\n` +
                         pc.dim("The key starts with sk-… and is only shown once."), "OpenAI API Key");
-                    p.log.info(pc.dim("Opening browser…"));
+                    await p.text({
+                        message: pc.dim("Press ") + pc.bold(pc.yellow("Enter")) + pc.dim(" to open browser…"),
+                        defaultValue: "",
+                    });
                     openUrl(OPENAI_API_KEYS_URL);
                 }
                 const v = await p.password({
@@ -661,45 +655,143 @@ function buildFields(detectedIp, hasSavedPlan) {
         },
         // ── Network ──────────────────────────────────────────────────────────
         {
-            id: "ingressAllowedIps",
+            id: "networkAccessMode",
             section: "Network",
-            skip: (state) => saved(state, "ingressAllowedIps"),
+            skip: (state) => saved(state, "ingressAllowedIps") && saved(state, "clusterPublicIp"),
             run: async (state) => {
-                const v = await p.text({
-                    message: pc.bold("IPs allowed to access your cluster (CIDR, comma-separated)"),
-                    placeholder: "0.0.0.0/0",
-                    defaultValue: state.ingressAllowedIps,
+                const mode = await p.select({
+                    message: pc.bold("How will you access the cluster?"),
+                    options: [
+                        {
+                            value: "public",
+                            label: "Public",
+                            hint: "auto-detects your public IP",
+                        },
+                        {
+                            value: "local",
+                            label: "Local only (localhost / LAN)",
+                            hint: "uses 127.0.0.1 or a private IP",
+                        },
+                    ],
                 });
-                if (p.isCancel(v))
+                if (p.isCancel(mode))
                     return back();
-                return { ...state, ingressAllowedIps: v };
-            },
-            review: (state) => ["Allowed IPs", state.ingressAllowedIps],
-        },
-        {
-            id: "clusterPublicIp",
-            section: "Network",
-            skip: (state) => dnsAndTlsEnabled(state) && saved(state, "clusterPublicIp"),
-            run: async (state) => {
-                const useLocal = !dnsAndTlsEnabled(state);
-                const fallback = useLocal ? "127.0.0.1" : detectedIp;
-                const defaultIp = useLocal ? fallback : (state.clusterPublicIp || fallback);
-                const v = await p.text({
-                    message: useLocal
-                        ? pc.bold("Cluster IP") + pc.dim("  (local because DNS management is disabled. Rewrite if it necessary)")
-                        : pc.bold("Public IP of your cluster"),
-                    defaultValue: defaultIp,
-                    placeholder: fallback || "x.x.x.x",
-                    validate: (v) => {
-                        if (!v && !defaultIp)
-                            return "Required";
-                    },
+                const m = mode;
+                async function resolvePublicIp() {
+                    if (detectedIp)
+                        return detectedIp;
+                    const services = ["ifconfig.me", "api.ipify.org", "icanhazip.com"];
+                    await withSpinner("Detecting public IP", async () => {
+                        for (const svc of services) {
+                            try {
+                                const ip = (await execAsync(`curl -s --max-time 4 ${svc}`)).trim();
+                                if (/^\d{1,3}(\.\d{1,3}){3}$/.test(ip)) {
+                                    detectedIp = ip;
+                                    return;
+                                }
+                            }
+                            catch { /* try next */ }
+                        }
+                    });
+                    return detectedIp;
+                }
+                if (m === "public") {
+                    const publicIp = await resolvePublicIp();
+                    const confirmIp = await p.text({
+                        message: pc.bold("Public IP of your cluster"),
+                        ...(publicIp
+                            ? { initialValue: publicIp }
+                            : { placeholder: "x.x.x.x" }),
+                        validate: (v) => { if (!v)
+                            return "Required"; },
+                    });
+                    if (p.isCancel(confirmIp))
+                        return back();
+                    const restriction = await p.select({
+                        message: pc.bold("Restrict ingress access?"),
+                        options: [
+                            {
+                                value: "open",
+                                label: "Open to everyone (0.0.0.0/0)",
+                                hint: "any IP can reach the cluster",
+                            },
+                            {
+                                value: "restrict",
+                                label: "Restrict to specific IPs",
+                                hint: "only listed CIDRs can reach the cluster",
+                            },
+                        ],
+                    });
+                    if (p.isCancel(restriction))
+                        return back();
+                    if (restriction === "open") {
+                        return { ...state, clusterPublicIp: confirmIp, ingressAllowedIps: "0.0.0.0/0" };
+                    }
+                    const allowedCidrs = await p.text({
+                        message: pc.bold("Allowed CIDRs (comma-separated)"),
+                        placeholder: "203.0.113.0/24,198.51.100.5/32",
+                        validate: (v) => { if (!v)
+                            return "At least one CIDR is required"; },
+                    });
+                    if (p.isCancel(allowedCidrs))
+                        return back();
+                    return { ...state, clusterPublicIp: confirmIp, ingressAllowedIps: allowedCidrs };
+                }
+                // local — detect LAN IPs from network interfaces
+                const lanIps = [];
+                const ifaces = networkInterfaces();
+                for (const [name, addrs] of Object.entries(ifaces)) {
+                    for (const addr of addrs ?? []) {
+                        if (addr.family === "IPv4" && !addr.internal) {
+                            lanIps.push({ ip: addr.address, iface: name });
+                        }
+                    }
+                }
+                let localIp;
+                if (lanIps.length > 0) {
+                    localIp = await p.select({
+                        message: pc.bold("Cluster IP"),
+                        options: [
+                            ...lanIps.map((l) => ({
+                                value: l.ip,
+                                label: l.ip,
+                                hint: l.iface,
+                            })),
+                            { value: "127.0.0.1", label: "127.0.0.1", hint: "localhost" },
+                            { value: "__custom__", label: "Enter manually" },
+                        ],
+                    });
+                    if (p.isCancel(localIp))
+                        return back();
+                    if (localIp === "__custom__") {
+                        localIp = await p.text({
+                            message: pc.bold("Cluster IP"),
+                            placeholder: "192.168.x.x",
+                            validate: (v) => { if (!v)
+                                return "Required"; },
+                        });
+                        if (p.isCancel(localIp))
+                            return back();
+                    }
+                }
+                else {
+                    localIp = await p.text({
+                        message: pc.bold("Cluster IP") + pc.dim("  (127.0.0.1 for localhost, or your LAN IP)"),
+                        defaultValue: "127.0.0.1",
+                    });
+                }
+                if (p.isCancel(localIp))
+                    return back();
+                const allowedIps = await p.text({
+                    message: pc.bold("IPs allowed to access the cluster (CIDR)"),
+                    defaultValue: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
+                    placeholder: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
                 });
-                if (p.isCancel(v))
+                if (p.isCancel(allowedIps))
                     return back();
-                return { ...state, clusterPublicIp: v };
+                return { ...state, clusterPublicIp: localIp, ingressAllowedIps: allowedIps };
             },
-            review: (state) => ["Public IP", state.clusterPublicIp],
+            review: (state) => ["Network", `${state.clusterPublicIp}  allowed: ${state.ingressAllowedIps}`],
         },
     ];
 }
@@ -837,10 +929,26 @@ export async function bootstrap() {
         }
     });
     console.log();
-    p.box(`💅 Secure, isolated and flexible GitOps infrastructure for modern requirements\n` +
-        `🤖 You can manage it yourself — or delegate to AI.\n` +
-        `🔐 Encrypted secrets, hardened containers, continuous delivery.`, pc.bold("Welcome to GitOps AI Bootstrapper"), {
-        contentAlign: "center",
+    const version = readPackageVersion();
+    const logo = [
+        "  ▄█████▄ ",
+        "  ██ ◆ ██ ",
+        "  ██   ██ ",
+        "  ▀██ ██▀ ",
+        "    ▀█▀   ",
+    ];
+    const taglines = [
+        "💅 Secure, isolated & flexible GitOps infrastructure",
+        "🤖 Manage it yourself — or delegate to AI",
+        "🔐 Encrypted secrets, hardened containers,",
+        "   continuous delivery",
+        pc.dim(`v${version}`),
+    ];
+    const banner = logo
+        .map((l, i) => pc.cyan(l) + "  " + (taglines[i] ?? ""))
+        .join("\n");
+    p.box(banner, pc.bold("Welcome to GitOps AI Bootstrapper"), {
+        contentAlign: "left",
         titleAlign: "center",
         rounded: true,
         formatBorder: (text) => pc.cyan(text),
@@ -879,14 +987,18 @@ export async function bootstrap() {
             return;
         }
     }
-    // ── Detect public IP (silent) ────────────────────────────────────────
-    let detectedIp = prev.clusterPublicIp ?? "";
-    if (!detectedIp) {
-        try {
-            detectedIp = await execAsync("curl -s --max-time 5 ifconfig.me");
-        }
-        catch {
-            detectedIp = "";
+    // ── Detect public IP (silent, try multiple services) ─────────────────
+    let detectedIp = "";
+    if (!prev.clusterPublicIp) {
+        for (const svc of ["ifconfig.me", "api.ipify.org", "icanhazip.com"]) {
+            try {
+                const ip = (await execAsync(`curl -s --max-time 4 ${svc}`)).trim();
+                if (/^\d{1,3}(\.\d{1,3}){3}$/.test(ip)) {
+                    detectedIp = ip;
+                    break;
+                }
+            }
+            catch { /* try next */ }
         }
     }
     // ── Run interactive wizard ───────────────────────────────────────────
@@ -898,6 +1010,7 @@ export async function bootstrap() {
         : [
             ...REQUIRED_COMPONENT_IDS,
             ...(savedDnsTls ? DNS_TLS_COMPONENT_IDS : []),
+            ...MONITORING_COMPONENT_IDS,
             ...OPTIONAL_COMPONENTS.map((c) => c.id),
         ];
     const initialState = {
@@ -1035,6 +1148,8 @@ export async function bootstrap() {
             ? wizard.openclawGatewayToken
             : undefined,
         selectedComponents,
+        templateRef: wizard.templateTag?.trim() ||
+            (isNewRepo(wizard) ? "main" : undefined),
     };
     // ── Check macOS prerequisites ────────────────────────────────────────
     if (isMacOS()) {
@@ -1055,20 +1170,25 @@ export async function bootstrap() {
         log.error(`Bootstrap failed\n${formatError(err)}`);
         return process.exit(1);
     }
-    // ── /etc/hosts suggestion (no DNS management) ───────────────────────
-    if (!wizard.manageDnsAndTls) {
+    // ── /etc/hosts suggestion (local IP or no DNS management) ───────────
+    const isLocalIp = /^(127\.|10\.|172\.(1[6-9]|2\d|3[01])\.|192\.168\.)/.test(fullConfig.clusterPublicIp)
+        || fullConfig.clusterPublicIp === "localhost";
+    if (isLocalIp || !wizard.manageDnsAndTls) {
         const hostsEntries = selectedComponents
             .map((id) => COMPONENTS.find((c) => c.id === id))
             .filter((c) => !!c?.subdomain)
             .map((c) => `${fullConfig.clusterPublicIp}  ${c.subdomain}.${fullConfig.clusterDomain}`);
         if (hostsEntries.length > 0) {
             const hostsBlock = hostsEntries.join("\n");
-            p.note(`${pc.dim("Since automatic DNS is disabled, add these to")} ${pc.bold("/etc/hosts")}${pc.dim(":")}\n\n` +
+            const reason = isLocalIp
+                ? "Your cluster uses a local/private IP, so DNS won't resolve publicly."
+                : "Automatic DNS is disabled.";
+            p.note(`${pc.dim(reason + " Add these to")} ${pc.bold("/etc/hosts")}${pc.dim(":")}\n\n` +
                 hostsEntries.map((e) => pc.cyan(e)).join("\n"), "Local DNS");
             const addHosts = await p.confirm({
                 message: pc.bold("Append these entries to /etc/hosts now?") +
                     pc.dim("  (requires sudo — macOS will prompt for your password)"),
-                initialValue: false,
+                initialValue: true,
             });
             if (!p.isCancel(addHosts) && addHosts) {
                 try {