orkestrator 0.0.0-stage → 2.23.0

package/README.md

@@ -1,3 +1,155 @@
-# Temporary Holding Version
+# Orkestrator backend and client
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+The `orkestrator` executable has two modes:
+
+- **Service** — `orkestrator`, `orkestrator serve …`, or the historical
+  `orkestrator --host … --port …` forms run the backend in the foreground.
+- **Client** — `orkestrator <group> <command> …` talks to a backend that is
+  already running. Client commands never start a backend.
+
+Run the Orkestrator backend service without installing the desktop application:
+
+```bash
+bunx orkestrator
+```
+
+The service stays in the foreground and stops cleanly when it receives Ctrl-C,
+SIGINT, or SIGTERM. It stores persistent state in the normal Orkestrator data
+directory.
+
+For the hosted web client, publish the backend through Tailscale Serve and allow
+the Orkestrator origins:
+
+```bash
+bunx orkestrator \
+  --tailscale-serve \
+  --allowed-origins https://orkestrator.dev,https://www.orkestrator.dev
+```
+
+For a loopback-only service:
+
+```bash
+bunx orkestrator \
+  --host 127.0.0.1 \
+  --port 34121 \
+  --allow-non-tailscale-bind
+```
+
+Or install Bun when needed and start the service in one command:
+
+```bash
+curl -fsSL https://orkestrator.dev/install.sh | bash
+```
+
+Pass backend arguments through the installer with `bash -s --`:
+
+```bash
+curl -fsSL https://orkestrator.dev/install.sh | \
+  bash -s -- --tailscale-serve \
+  --allowed-origins https://orkestrator.dev,https://www.orkestrator.dev
+```
+
+macOS and Linux are supported. Docker is required for container environments;
+Tailscale is required only for tailnet access and Tailscale Serve.
+
+## Agent binaries
+
+The desktop app downloads the pinned Claude, Codex, OpenCode, Grok and Pi
+binaries when it starts. A backend run this way does not, and without them it
+falls back to whatever is on `PATH`. Install the exact pinned versions once, and
+again after each upgrade:
+
+```bash
+orkestrator toolchain install             # what this data directory enables
+orkestrator toolchain install --dry-run   # show what would be fetched
+```
+
+Every download is checked against its pinned size and SHA-256 before use, into
+`<data-dir>/toolchains`. Which tools are installed follows the data directory's
+enabled agent platforms (all of them when nothing is configured yet); `--tool`
+installs exactly the tools you name instead. On success `toolchains/bin/current`
+points at the installed set and the service uses it by default for that data
+directory — restart a running service to pick up a new one. Pass `--data-dir`
+when the service does, and see `orkestrator toolchain install --help`.
+
+## Client commands
+
+Select the backend explicitly. A running backend publishes a private
+`backend-instance.json` in its data directory; save it once as the default:
+
+```bash
+orkestrator connection add local --data-dir "$HOME/.config/orkestrator-v2" --default
+orkestrator connection check
+```
+
+Remote backends use a private token file or stdin — never a token argument:
+
+```bash
+orkestrator connection add laptop --url https://laptop.example.ts.net --credential-file ~/.secrets/token
+```
+
+A complete, owned workflow (IDs and request keys are explicit so a script can
+recover after a lost response or restart):
+
+```bash
+set -euo pipefail
+PROJECT=$(orkestrator project add --path /srv/repos/app --request-id "$RUN:add" --output id)
+ENV=$(orkestrator environment create --project "$PROJECT" --type local \
+  --name "$RUN" --request-id "$RUN:create" --output id)
+trap 'orkestrator environment delete "$ENV" --wait deleted --timeout 5m --json >/dev/null' EXIT
+orkestrator environment start "$ENV" --wait ready --timeout 10m --json
+orkestrator environment config set "$ENV" --set agent.claude.model=opus
+orkestrator session start --environment "$ENV" --agent claude \
+  --prompt-file task.txt --request-id "$RUN:prompt" --json > launch.json
+RUN_ID=$(jq -er '.result.runId' launch.json)
+SESSION=$(jq -er '.result.sessionId' launch.json)
+orkestrator run wait "$RUN_ID" --timeout 30m --json
+orkestrator session transcript "$SESSION" --limit 20 --json > transcript.json
+orkestrator environment exec "$ENV" --wait -- bun test
+```
+
+- `--json` prints exactly one envelope on stdout; `--output id` prints only
+  the documented ID. Exit codes: 0 ok, 1 failed, 2 invalid input, 3 not
+  found/ambiguous/expired, 4 connection/auth, 5 observer deadline,
+  6 interaction required, 7 unknown dispatch/outcome, 8 conflict/unsupported,
+  130/143 observation interrupted.
+- Paths given to `project` commands are on the backend host; prompt, patch and
+  credential files are read locally.
+- `project create` creates a **private GitHub repository** and pushes to it;
+  it requires `--github-private`.
+- A timeout or Ctrl+C only stops observing. Resume with `orkestrator run wait
+  OP`; recover a lost response with `orkestrator run get --request-id KEY
+  --action ACTION`. The client never resubmits under a new key.
+
+Run `orkestrator help` and `orkestrator help <group> <command>` for the full
+command reference. The contract, retention rules and provider completion
+matrix are documented in the repository's `docs/architecture/public-cli.md`.
+
+## Publishing
+
+`dependencies` here lists only the modules `scripts/build.ts` leaves unbundled;
+everything else is inlined into `dist/` and `resources/`. `tests/cli.test.ts`
+derives that list from the built artifacts, so adding a dependency the bundles do
+not resolve — or bumping one in `apps/backend` or `bridges/` without mirroring it
+here — fails the suite.
+
+Publish from the repository root:
+
+```bash
+mise run publish:cli
+```
+
+That runs `smoke:cli` first, which packs the tarball, installs it into a scratch
+project, checks the unbundled dependencies resolve from the installed layout,
+runs client help/version/errors with no backend, starts the backend, drives a
+fixture project/environment lifecycle through the installed client, confirms
+only one backend ran, and stops it. It needs network access, which is why it is
+a publish gate rather than part of `mise run test`.
+
+The npm dist-tag follows the version's update channel: `2.23.0` publishes as
+`latest`, `2.23.0-beta` as `beta` and `2.23.0-alpha` as `alpha`, so
+`bunx orkestrator` only ever gets a stable release. Try a prerelease with
+`bunx orkestrator@beta` or `bunx orkestrator@alpha`.
+
+`dist/client.js` is the client bundle the launcher imports first; it must stay
+free of backend imports so client commands never initialize a service.

package/bin/orkestrator-launcher.js

@@ -0,0 +1,216 @@
+#!/usr/bin/env bun
+/**
+ * The process launchd or systemd runs for a standalone backend installed with
+ * `orkestrator service install`.
+ *
+ * It starts `<dataDir>/backend/current`, with the Bun and serve arguments in
+ * `service.json`, and supervises it:
+ *
+ * - A release that has not reached readiness (a backend-instance.json naming
+ *   its process and version) gets two boots. After that the launcher points
+ *   `current` back at `previous` and records the rollback, which the restored
+ *   backend reports as the outcome of its update.
+ * - When the backend exits after writing `restart-requested` (its self-update
+ *   just switched `current`), the launcher starts the new release at once.
+ * - SIGTERM and SIGINT are forwarded; the launcher then exits with the
+ *   backend's status, and the service manager decides what happens next.
+ *
+ * This file is copied out of the release into `<dataDir>/backend/launcher.js`,
+ * so it must not import anything outside Node's and Bun's built-ins. It
+ * repeats the layout in `packages/protocol/src/node/standalone-service.ts`.
+ */
+import { spawn, spawnSync } from "node:child_process";
+import {
+  existsSync,
+  readFileSync,
+  readlinkSync,
+  renameSync,
+  rmSync,
+  symlinkSync,
+  writeFileSync,
+} from "node:fs";
+import path from "node:path";
+
+const MAX_BOOT_ATTEMPTS = 2;
+const READY_TIMEOUT_MS = Number(process.env.ORKESTRATOR_LAUNCHER_READY_TIMEOUT_MS ?? 120_000);
+const RETRY_DELAY_MS = Number(process.env.ORKESTRATOR_LAUNCHER_RETRY_DELAY_MS ?? 2_000);
+const CRASH_DELAY_MS = Number(process.env.ORKESTRATOR_LAUNCHER_CRASH_DELAY_MS ?? 5_000);
+const STOP_TIMEOUT_MS = Number(process.env.ORKESTRATOR_LAUNCHER_STOP_TIMEOUT_MS ?? 10_000);
+const TOOLCHAIN_TIMEOUT_MS = 20 * 60 * 1000;
+
+const dataDir = path.resolve(process.argv[2] ?? "");
+if (!process.argv[2]) {
+  console.error("usage: launcher.js <data-dir>");
+  process.exit(2);
+}
+const root = path.join(dataDir, "backend");
+const layout = {
+  current: path.join(root, "current"),
+  previous: path.join(root, "previous"),
+  serviceFile: path.join(root, "service.json"),
+  stateFile: path.join(root, "launcher-state.json"),
+  restartMarker: path.join(root, "restart-requested"),
+  instance: path.join(dataDir, "backend-instance.json"),
+};
+
+function log(message) {
+  console.error(`[launcher] ${new Date().toISOString()} ${message}`);
+}
+
+function readJson(file) {
+  try {
+    return JSON.parse(readFileSync(file, "utf8"));
+  } catch {
+    return null;
+  }
+}
+
+function readState() {
+  const value = readJson(layout.stateFile) ?? {};
+  return {
+    version: 1,
+    lastReadyVersion: typeof value.lastReadyVersion === "string" ? value.lastReadyVersion : null,
+    bootAttempts:
+      value.bootAttempts && typeof value.bootAttempts === "object" ? value.bootAttempts : {},
+    rollback: value.rollback ?? null,
+  };
+}
+
+function writeState(state) {
+  const temporary = `${layout.stateFile}.${process.pid}.tmp`;
+  writeFileSync(temporary, `${JSON.stringify(state, null, 2)}\n`, { mode: 0o600 });
+  renameSync(temporary, layout.stateFile);
+}
+
+function linked(link) {
+  try {
+    return readlinkSync(link);
+  } catch {
+    return null;
+  }
+}
+
+function pointLink(link, target) {
+  const temporary = `${link}.${process.pid}.tmp`;
+  rmSync(temporary, { force: true });
+  symlinkSync(target, temporary);
+  renameSync(temporary, link);
+}
+
+function entryOf(release) {
+  return path.join(release, "node_modules", "orkestrator", "bin", "orkestrator.js");
+}
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+
+const service = readJson(layout.serviceFile);
+if (!service || typeof service.bun !== "string" || !Array.isArray(service.args)) {
+  log(`${layout.serviceFile} is missing or invalid; run \`orkestrator service install\``);
+  process.exit(1);
+}
+
+let child = null;
+let stopping = false;
+for (const signal of ["SIGTERM", "SIGINT"]) {
+  process.on(signal, () => {
+    stopping = true;
+    if (child && child.exitCode === null) child.kill(signal);
+    else process.exit(0);
+  });
+}
+
+async function waitForReady(pid, version, exited) {
+  const deadline = Date.now() + READY_TIMEOUT_MS;
+  while (Date.now() < deadline && !exited()) {
+    const instance = readJson(layout.instance);
+    if (instance && instance.pid === pid && instance.appVersion === version) return true;
+    await sleep(500);
+  }
+  return false;
+}
+
+while (!stopping) {
+  const target = linked(layout.current);
+  if (!target) {
+    log(`${layout.current} does not point at a release`);
+    process.exit(1);
+  }
+  const release = path.resolve(root, target);
+  const version = path.basename(release);
+  const state = readState();
+
+  if (version !== state.lastReadyVersion) {
+    const attempts = (state.bootAttempts[version] ?? 0) + 1;
+    const previousTarget = linked(layout.previous);
+    const previousVersion = previousTarget ? path.basename(previousTarget) : null;
+    if (attempts > MAX_BOOT_ATTEMPTS && previousTarget && previousVersion !== version) {
+      log(
+        `${version} did not start after ${MAX_BOOT_ATTEMPTS} attempts; rolling back to ${previousVersion}`,
+      );
+      pointLink(layout.current, previousTarget);
+      state.bootAttempts[version] = 0;
+      state.rollback = {
+        failedVersion: version,
+        restoredVersion: previousVersion,
+        at: new Date().toISOString(),
+      };
+      writeState(state);
+      continue;
+    }
+    state.bootAttempts[version] = attempts;
+    writeState(state);
+  }
+
+  const entry = entryOf(release);
+  // Idempotent and quick when nothing changed; a new release installs its pins.
+  const toolchain = spawnSync(service.bun, [entry, "toolchain", "install", "--data-dir", dataDir], {
+    stdio: ["ignore", "ignore", "inherit"],
+    timeout: TOOLCHAIN_TIMEOUT_MS,
+  });
+  if (toolchain.status !== 0) log(`toolchain install for ${version} failed; starting anyway`);
+
+  rmSync(layout.restartMarker, { force: true });
+  log(`starting ${version}`);
+  let exited = false;
+  child = spawn(
+    service.bun,
+    [entry, "serve", ...service.args, "--data-dir", dataDir, "--standalone-updater", root],
+    { stdio: "inherit", env: process.env },
+  );
+  const exit = new Promise((resolve) =>
+    child.once("exit", (code, signal) => {
+      exited = true;
+      resolve({ code, signal });
+    }),
+  );
+  const ready = await waitForReady(child.pid, version, () => exited);
+  if (ready) {
+    const next = readState();
+    next.lastReadyVersion = version;
+    next.bootAttempts[version] = 0;
+    writeState(next);
+    log(`${version} is ready`);
+  }
+  if (!ready && !exited) {
+    log(`${version} missed its readiness deadline; stopping it`);
+    child.kill("SIGTERM");
+    let timer;
+    await Promise.race([
+      exit,
+      new Promise((resolve) => {
+        timer = setTimeout(resolve, STOP_TIMEOUT_MS);
+      }),
+    ]);
+    clearTimeout(timer);
+    if (!exited) child.kill("SIGKILL");
+  }
+  const { code, signal } = await exit;
+  child = null;
+  if (stopping) process.exit(code ?? 0);
+  if (existsSync(layout.restartMarker)) {
+    log(`${version} exited to restart into ${path.basename(linked(layout.current) ?? "")}`);
+    continue;
+  }
+  log(`${version} exited (code ${code ?? "none"}, signal ${signal ?? "none"})`);
+  await sleep(ready ? CRASH_DELAY_MS : RETRY_DELAY_MS);
+}

package/bin/orkestrator.js

@@ -0,0 +1,32 @@
+#!/usr/bin/env bun
+
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const packageRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+const manifest = await Bun.file(path.join(packageRoot, "package.json")).json();
+const version = typeof manifest.version === "string" ? manifest.version : "0.0.0";
+
+// Decide service versus client before the backend is imported: help, version
+// and client commands must never initialize storage, listeners, or bridges.
+const client = await import("../dist/client.js");
+const argv = process.argv.slice(2);
+const invocation = client.classifyInvocation(argv);
+
+if (invocation.mode === "serve") {
+  // `serve` is an explicit spelling of the historical no-subcommand form; the
+  // backend's own parser reads the remaining service options from argv.
+  process.argv.splice(2, process.argv.length - 2, ...invocation.args);
+  // The standalone bundle is deliberately laid out like the desktop resources,
+  // but it lives in Bun's package cache instead of an application bundle. Make
+  // those roots authoritative before main.ts calculates any path defaults.
+  process.env.NODE_ENV ??= "production";
+  process.env.ORKESTRATOR_APP_ROOT ??= packageRoot;
+  process.env.ORKESTRATOR_RESOURCE_ROOT ??= path.join(packageRoot, "resources");
+  process.env.ORKESTRATOR_VERSION ??= version;
+  await import("../dist/main.js");
+} else if (invocation.mode === "client") {
+  process.exitCode = await client.runClientProcess(invocation.argv, version);
+} else {
+  process.exitCode = client.renderInvocationError(argv, invocation.message);
+}