@openpalm/lib 0.9.8 → 0.10.1

package/README.md

@@ -1,83 +1,43 @@
 # @openpalm/lib
 
-Shared control-plane library consumed by both the CLI and the admin SvelteKit app. All portable logic for managing an OpenPalm stack lives here -- paths, secrets, staging, lifecycle, Docker Compose operations, channel/connection management, scheduling, and setup orchestration.
+Shared control-plane library for OpenPalm.
+CLI, admin, and scheduler use this package so stack behavior stays consistent.
 
-Admin is a thin UI layer that re-exports from this package. The CLI calls these functions directly.
+The current model is direct-write over `~/.openpalm/` plus native Docker Compose.
+Compose files in `stack/` and env files in `vault/` are the live runtime inputs.
 
-## Modules
+## What lives here
 
-| Module | Purpose |
-|---|---|
-| `types` | Core type definitions -- `ControlPlaneState`, `CoreServiceName`, `ChannelInfo`, `CanonicalConnectionProfile`, etc. |
-| `paths` | XDG directory resolution (`resolveConfigHome`, `resolveDataHome`, `resolveStateHome`, `ensureXdgDirs`) |
-| `env` | `.env` file parsing and merging (`parseEnvContent`, `parseEnvFile`, `mergeEnvContent`) |
-| `secrets` | Secrets management -- ensure, read, patch, mask `secrets.env` |
-| `setup-status` | First-boot detection (`isSetupComplete`, `readSecretsKeys`, `detectUserId`) |
-| `setup` | Setup wizard backend (`performSetup`, `detectProviders`, `validateSetupInput`) |
-| `staging` | Artifact staging pipeline (`stageArtifacts`, `persistArtifacts`, `buildEnvFiles`) |
-| `lifecycle` | Install/update/uninstall/upgrade orchestration (`applyInstall`, `applyUpdate`, `createState`) |
-| `docker` | Docker Compose CLI wrapper (`composeUp`, `composeDown`, `composePs`, `composeLogs`, etc.) |
-| `channels` | Channel discovery, install, uninstall from registry or filesystem |
-| `connection-profiles` | CRUD for connection profiles (LLM providers, embedding, etc.) |
-| `connection-mapping` | Build OpenCode and mem0 config from connection profiles |
-| `memory-config` | Read/write memory service config, push to running memory container |
-| `scheduler` | Automation YAML parsing, Croner-based scheduler, execution log |
-| `model-runner` | Local provider detection (Ollama, Docker Model Runner, LM Studio) |
-| `core-assets` | Seed and read core infrastructure files (compose, Caddyfile, schemas) |
-| `connection-migration-flags` | Migration compatibility detection (`readConnectionMigrationFlags`, `detectConnectionCompatibilityMode`) |
-| `audit` | Append to the audit log |
-| `logger` | Structured logger factory (`createLogger`) |
-| `provider-constants` | LLM provider metadata (`LLM_PROVIDERS`, `PROVIDER_DEFAULT_URLS`, `EMBEDDING_DIMS`) |
-
-## Dependency Injection
-
-Asset loading differs between CLI and admin. Two interfaces abstract this:
-
-### CoreAssetProvider
-
-Returns the content of bundled infrastructure files (compose, Caddyfile, schemas, automations). Functions that need these files accept a `CoreAssetProvider` parameter.
-
-| Implementation | Consumer | Source |
-|---|---|---|
-| `FilesystemAssetProvider` | CLI | Reads from `DATA_HOME` on disk |
-| `ViteAssetProvider` | Admin | Reads from Vite `$assets` imports (defined in `packages/admin`, not in lib) |
+- OpenPalm home/path helpers
+- Env parsing and secret management
+- Addon install/uninstall and registry helpers
+- Compose lifecycle wrappers
+- Memory and connection profile helpers
+- Automation parsing used by the scheduler
+- Shared structured logging
 
-### RegistryProvider
+## Important context
 
-Returns channel and automation definitions from the registry catalog.
+- Some filenames still use legacy names like `staging`; those modules now support the direct-write compose model
+- `config/` is user-owned, `vault/stack/stack.env` is system-managed, `registry/` is catalog-only, and `stack/addons/` contains enabled runtime overlays
+- New reusable control-plane logic belongs here, not duplicated in consumers
 
-| Implementation | Consumer | Source |
-|---|---|---|
-| `FilesystemRegistryProvider` | CLI | Reads from `registry/` directory |
-| `ViteRegistryProvider` | Admin | Reads via `import.meta.glob` (defined in `packages/admin`, not in lib) |
+## Main module areas
 
-## Usage
-
-```ts
-import {
-  createState,
-  stageArtifacts,
-  persistArtifacts,
-  applyInstall,
-  FilesystemAssetProvider,
-} from "@openpalm/lib";
-
-const assets = new FilesystemAssetProvider(dataHome);
-const state = createState(configHome, dataHome, stateHome);
-
-stageArtifacts(state, assets);
-persistArtifacts(state, assets);
-await applyInstall(state, assets, "cli");
-```
-
-Sub-path imports are also available:
+| Module area | Purpose |
+|---|---|
+| `control-plane/home` and `control-plane/paths` | Resolve the OpenPalm home layout |
+| `control-plane/env` and `control-plane/secrets` | Read, merge, and patch env files |
+| `control-plane/lifecycle` and `control-plane/docker` | Compose operations and stack lifecycle helpers |
+| `control-plane/channels` and `control-plane/components` | Addon discovery and install/uninstall logic |
+| `control-plane/memory-config` | Memory service configuration helpers |
+| `control-plane/scheduler` | Automation parsing and scheduler helpers |
+| `logger` | Shared structured logger |
 
-```ts
-import { resolveConfigHome } from "@openpalm/lib/control-plane/paths";
-import { createLogger } from "@openpalm/lib/shared/logger";
-import { LLM_PROVIDERS } from "@openpalm/lib/provider-constants";
-```
+## Consumer model
 
-## Architecture
+- CLI: direct host-side orchestrator
+- Admin: optional UI/API wrapper
+- Scheduler: automation runner without Docker socket access
 
-See [`docs/technical/core-principles.md`](../../docs/technical/core-principles.md) for the filesystem contract and security invariants that govern this library.
+See `docs/technical/core-principles.md` for the authoritative filesystem contract and security rules.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@openpalm/lib",
-  "version": "0.9.8",
+  "version": "0.10.1",
   "license": "MPL-2.0",
   "type": "module",
   "description": "Shared control-plane library for OpenPalm — lifecycle, staging, secrets, channels, connections, scheduler",

package/src/control-plane/audit.ts

@@ -3,7 +3,8 @@
  */
 import { mkdirSync, appendFileSync } from "node:fs";
 import type { ControlPlaneState, AuditEntry, CallerType } from "./types.js";
-import { MAX_AUDIT_MEMORY } from "./types.js";
+
+const MAX_AUDIT_MEMORY = 1000;
 
 export function appendAudit(
   state: ControlPlaneState,
@@ -28,10 +29,9 @@ export function appendAudit(
     state.audit = state.audit.slice(-MAX_AUDIT_MEMORY);
   }
   try {
-    const auditDir = `${state.stateDir}/audit`;
-    mkdirSync(auditDir, { recursive: true });
+    mkdirSync(state.logsDir, { recursive: true });
     appendFileSync(
-      `${auditDir}/admin-audit.jsonl`,
+      `${state.logsDir}/admin-audit.jsonl`,
       JSON.stringify(entry) + "\n"
     );
   } catch {

package/src/control-plane/backup.ts

@@ -0,0 +1,31 @@
+import { cpSync, existsSync, mkdirSync, readdirSync } from "node:fs";
+import { join } from "node:path";
+
+function timestampDirName(now = new Date()): string {
+  return now.toISOString().replace(/[:.]/g, "-");
+}
+
+/**
+ * Create a durable backup snapshot of the current OP_HOME contents.
+ *
+ * The backup is written under OP_HOME/backups/<timestamp>/ and excludes the
+ * backups directory itself to avoid recursive copies.
+ */
+export function backupOpenPalmHome(homeDir: string): string | null {
+  if (!existsSync(homeDir)) return null;
+
+  const backupDir = join(homeDir, "backups", timestampDirName());
+  mkdirSync(backupDir, { recursive: true });
+
+  let copiedAny = false;
+  for (const entry of readdirSync(homeDir, { withFileTypes: true })) {
+    if (entry.name === "backups") continue;
+
+    const sourcePath = join(homeDir, entry.name);
+    const targetPath = join(backupDir, entry.name);
+    cpSync(sourcePath, targetPath, { recursive: true });
+    copiedAny = true;
+  }
+
+  return copiedAny ? backupDir : null;
+}

package/src/control-plane/channels.ts

@@ -1,13 +1,11 @@
 /**
- * Channel validation, discovery, install, and uninstall for the OpenPalm control plane.
- *
- * Install/uninstall functions accept a RegistryProvider for catalog access,
- * decoupling from Vite-specific imports.
+ * Channel validation, discovery, and allowlist checks for the OpenPalm control plane.
  */
-import { mkdirSync, writeFileSync, existsSync, readdirSync, rmSync } from "node:fs";
+import { existsSync, readdirSync, readFileSync } from "node:fs";
+import { dirname } from "node:path";
+import { parse as yamlParse } from "yaml";
 import type { ChannelInfo } from "./types.js";
 import { CORE_SERVICES } from "./types.js";
-import type { RegistryProvider } from "./registry-provider.js";
 
 // ── Channel Name Validation ───────────────────────────────────────────
 
@@ -21,32 +19,61 @@ function isValidChannelName(name: string): boolean {
 // ── Channel Discovery ─────────────────────────────────────────────────
 
 /**
- * Discover installed channels by scanning CONFIG_HOME/channels/.
+ * Check if a compose file defines a channel service (has CHANNEL_NAME or GUARDIAN_URL).
+ * This is compose-derived: we parse the actual compose content rather than
+ * relying on filename patterns or directory naming conventions.
+ */
+export function isChannelAddon(composePath: string): boolean {
+  try {
+    const content = readFileSync(composePath, "utf-8");
+    const doc = yamlParse(content);
+    if (typeof doc !== "object" || doc === null) return false;
+    const services = (doc as Record<string, unknown>).services;
+    if (typeof services !== "object" || services === null) return false;
+
+    for (const svcDef of Object.values(services as Record<string, unknown>)) {
+      if (typeof svcDef !== "object" || svcDef === null) continue;
+      const env = (svcDef as Record<string, unknown>).environment;
+      if (typeof env === "object" && env !== null) {
+        if (Array.isArray(env)) {
+          if (env.some((e: unknown) => typeof e === "string" && (e.startsWith("CHANNEL_NAME=") || e.startsWith("GUARDIAN_URL=")))) return true;
+        } else {
+          if ("CHANNEL_NAME" in (env as Record<string, unknown>) || "GUARDIAN_URL" in (env as Record<string, unknown>)) return true;
+        }
+      }
+    }
+    return false;
+  } catch {
+    return false;
+  }
+}
+
+/**
+ * Discover installed channels by scanning stack/addons/ for channel addons.
+ * A channel addon is identified by compose-derived truth: its compose.yml
+ * defines services with CHANNEL_NAME or GUARDIAN_URL environment variables.
  *
- * A channel is any .yml file in the channels directory.
- * A .caddy file is optional — if present, the channel gets Caddy HTTP routing.
- * If absent, the channel is only accessible on the Docker network (host + containers).
+ * Non-channel addons (admin, ollama, etc.) are excluded.
+ *
+ * @param configDir - The config directory (~/.openpalm/config). The stack
+ *   directory is derived from the parent (homeDir).
  */
 export function discoverChannels(configDir: string): ChannelInfo[] {
-  const channelsDir = `${configDir}/channels`;
-  if (!existsSync(channelsDir)) return [];
-
-  const files = readdirSync(channelsDir);
-  const ymlFiles = files.filter((f) => f.endsWith(".yml"));
-  const caddyFiles = new Set(files.filter((f) => f.endsWith(".caddy")));
-
-  return ymlFiles
-    .map((ymlFile) => {
-      const name = ymlFile.replace(/\.yml$/, "");
-      const caddyFile = `${name}.caddy`;
-      const hasCaddy = caddyFiles.has(caddyFile);
-      return {
-        name,
-        hasRoute: hasCaddy,
-        ymlPath: `${channelsDir}/${ymlFile}`,
-        caddyPath: hasCaddy ? `${channelsDir}/${caddyFile}` : null
-      };
+  const homeDir = dirname(configDir);
+  const addonsDir = `${homeDir}/stack/addons`;
+  if (!existsSync(addonsDir)) return [];
+
+  const entries = readdirSync(addonsDir, { withFileTypes: true });
+  return entries
+    .filter((entry) => {
+      if (!entry.isDirectory()) return false;
+      const composePath = `${addonsDir}/${entry.name}/compose.yml`;
+      return existsSync(composePath) && isChannelAddon(composePath);
     })
+    .map((entry) => ({
+      name: entry.name,
+      ymlPath: `${addonsDir}/${entry.name}/compose.yml`,
+    }))
     .filter((ch) => isValidChannelName(ch.name));
 }
 
@@ -54,146 +81,51 @@ export function discoverChannels(configDir: string): ChannelInfo[] {
 
 /**
  * Check if a service name is allowed. Core services are always allowed.
- * Ollama is allowed when its compose overlay is staged.
- * Channel services (channel-*) are allowed if a corresponding staged .yml exists
- * in STATE_HOME/artifacts/channels/.
+ * Addon services are allowed if they appear as a compose service defined in
+ * any addon compose file under stack/addons/. This is compose-derived: the
+ * actual compose content is checked, not directory naming conventions.
  */
-export function isAllowedService(value: string, stateDir?: string): boolean {
+export function isAllowedService(value: string, configDir?: string): boolean {
   if (!value || !value.trim() || value !== value.toLowerCase()) return false;
   if ((CORE_SERVICES as string[]).includes(value)) return true;
-  if (value === "ollama" && stateDir) {
-    return existsSync(`${stateDir}/artifacts/ollama.yml`);
-  }
-  if ((value === "admin" || value === "caddy" || value === "docker-socket-proxy") && stateDir) {
-    return existsSync(`${stateDir}/artifacts/admin.yml`);
-  }
-  if (value.startsWith("channel-")) {
-    const ch = value.slice("channel-".length);
-    if (!isValidChannelName(ch)) return false;
-    if (stateDir) {
-      return existsSync(`${stateDir}/artifacts/channels/${ch}.yml`);
+
+  if (configDir) {
+    const homeDir = dirname(configDir);
+    const addonsDir = `${homeDir}/stack/addons`;
+    if (!existsSync(addonsDir)) return false;
+
+    // Check if any addon compose.yml defines this service name (YAML-parsed)
+    for (const entry of readdirSync(addonsDir, { withFileTypes: true })) {
+      if (!entry.isDirectory()) continue;
+      const composePath = `${addonsDir}/${entry.name}/compose.yml`;
+      if (!existsSync(composePath)) continue;
+      try {
+        const content = readFileSync(composePath, "utf-8");
+        const doc = yamlParse(content);
+        if (typeof doc === "object" && doc !== null) {
+          const services = (doc as Record<string, unknown>).services;
+          if (typeof services === "object" && services !== null && value in (services as Record<string, unknown>)) {
+            return true;
+          }
+        }
+      } catch {
+        continue;
+      }
     }
   }
   return false;
 }
 
 /**
- * Check if a channel name is valid. Accepts any channel with a staged
- * .yml file in STATE_HOME/artifacts/channels/.
+ * Check if a channel name is valid and installed.
+ * Accepts any channel with a compose.yml in stack/addons/<name>/.
  */
-export function isValidChannel(value: string, stateDir?: string): boolean {
+export function isValidChannel(value: string, configDir?: string): boolean {
   if (!value || !value.trim()) return false;
   if (!isValidChannelName(value)) return false;
-  if (stateDir) {
-    return existsSync(`${stateDir}/artifacts/channels/${value}.yml`);
+  if (configDir) {
+    const homeDir = dirname(configDir);
+    return existsSync(`${homeDir}/stack/addons/${value}/compose.yml`);
   }
   return false;
 }
-
-// ── Channel Install / Uninstall ─────────────────────────────────────────
-
-/**
- * Install a channel from the registry catalog into CONFIG_HOME/channels/.
- * Copies the .yml (and optional .caddy) from the registry provider.
- * Refuses if the channel is already installed (files already exist).
- */
-export function installChannelFromRegistry(
-  name: string,
-  configDir: string,
-  registry: RegistryProvider
-): { ok: true } | { ok: false; error: string } {
-  if (!isValidChannelName(name)) {
-    return { ok: false, error: `Invalid channel name: ${name}` };
-  }
-  const channelYml = registry.channelYml();
-  if (!(name in channelYml)) {
-    return { ok: false, error: `Channel "${name}" not found in registry` };
-  }
-  const channelsDir = `${configDir}/channels`;
-  mkdirSync(channelsDir, { recursive: true });
-
-  const ymlPath = `${channelsDir}/${name}.yml`;
-  if (existsSync(ymlPath)) {
-    return { ok: false, error: `Channel "${name}" is already installed` };
-  }
-
-  writeFileSync(ymlPath, channelYml[name]);
-  const channelCaddy = registry.channelCaddy();
-  if (name in channelCaddy) {
-    writeFileSync(`${channelsDir}/${name}.caddy`, channelCaddy[name]);
-  }
-  return { ok: true };
-}
-
-/**
- * Uninstall a channel by removing its .yml (and .caddy) from CONFIG_HOME/channels/.
- */
-export function uninstallChannel(
-  name: string,
-  configDir: string
-): { ok: true } | { ok: false; error: string } {
-  if (!isValidChannelName(name)) {
-    return { ok: false, error: `Invalid channel name: ${name}` };
-  }
-  const channelsDir = `${configDir}/channels`;
-  const ymlPath = `${channelsDir}/${name}.yml`;
-  if (!existsSync(ymlPath)) {
-    return { ok: false, error: `Channel "${name}" is not installed` };
-  }
-
-  rmSync(ymlPath, { force: true });
-  rmSync(`${channelsDir}/${name}.caddy`, { force: true });
-  return { ok: true };
-}
-
-// ── Automation Install / Uninstall ──────────────────────────────────────
-
-/** Strict automation name: same rules as channel names */
-const AUTOMATION_NAME_RE = /^[a-z0-9][a-z0-9-]{0,62}$/;
-
-/**
- * Install an automation from the registry catalog into CONFIG_HOME/automations/.
- */
-export function installAutomationFromRegistry(
-  name: string,
-  configDir: string,
-  registry: RegistryProvider
-): { ok: true } | { ok: false; error: string } {
-  if (!AUTOMATION_NAME_RE.test(name)) {
-    return { ok: false, error: `Invalid automation name: ${name}` };
-  }
-  const automationYml = registry.automationYml();
-  if (!(name in automationYml)) {
-    return { ok: false, error: `Automation "${name}" not found in registry` };
-  }
-  const automationsDir = `${configDir}/automations`;
-  mkdirSync(automationsDir, { recursive: true });
-
-  const ymlPath = `${automationsDir}/${name}.yml`;
-  if (existsSync(ymlPath)) {
-    return { ok: false, error: `Automation "${name}" is already installed` };
-  }
-
-  writeFileSync(ymlPath, automationYml[name]);
-  return { ok: true };
-}
-
-/**
- * Uninstall an automation by removing its .yml from CONFIG_HOME/automations/.
- */
-export function uninstallAutomation(
-  name: string,
-  configDir: string
-): { ok: true } | { ok: false; error: string } {
-  if (!AUTOMATION_NAME_RE.test(name)) {
-    return { ok: false, error: `Invalid automation name: ${name}` };
-  }
-  const automationsDir = `${configDir}/automations`;
-  const ymlPath = `${automationsDir}/${name}.yml`;
-  if (!existsSync(ymlPath)) {
-    return { ok: false, error: `Automation "${name}" is not installed` };
-  }
-
-  rmSync(ymlPath, { force: true });
-  return { ok: true };
-}