subconscious-cli 6.0.1 → 6.1.0

package/README.md

@@ -161,6 +161,125 @@ The packaged integrations live in `bin/runbook`.
 | `subc pi` | Refresh the Pi provider from the live catalog, then launch |
 | `subc dsh` | Launch the DeepSeek Harness Web UI with a temporary provider populated from the live catalog |
 
+## Headless runs
+
+`subc <agent> headless PROMPT [args...]` runs one task without a terminal UI and exits with the agent's status. It works for `claude`, `codex`, `opencode`, `pi`, `marathon`, and `dsh`, and uses the same setup as an interactive launch. It never prompts: there is no update check, a missing agent fails with its install command, stdin is not read, and stdout carries only the agent's own output. Remaining arguments go to the agent, for example `subc claude headless "fix the failing test" --output-format stream-json --verbose`. subc reads `-p`/`--profile` and `--model` itself; put agent options with those names after `--`.
+
+The agents keep their own permission defaults, so a run that should edit files usually needs the agent's permission flags in `[args...]`. For example, `codex exec` uses a read-only sandbox and requires a git repository unless given `--skip-git-repo-check`.
+
+Model, endpoint, and key resolve as for any launch, so a logged-in user needs nothing extra. To point a run somewhere else:
+
+| Input | How to set it |
+| --- | --- |
+| Model | `--model`, or `SUBCONSCIOUS_MODEL` |
+| Endpoint | `SUBCONSCIOUS_BASE_URL`. In a headless run every agent uses it, including Claude Code (`CLAUDE_GATEWAY_URL` is ignored). |
+| Key | `SUBCONSCIOUS_API_KEY`, otherwise the `subc login` key |
+
+```bash
+SUBCONSCIOUS_BASE_URL=https://gateway.example SUBCONSCIOUS_API_KEY=... \
+  subc codex headless "fix the failing test" --model subconscious/deepseek-v4.1-flash-marathon
+```
+
+The endpoint must speak the same APIs as the Subconscious gateway: Anthropic Messages for Claude Code, OpenAI Responses for Codex, and OpenAI Chat Completions for the others.
+
+| Agent | Runs |
+| --- | --- |
+| `claude` | `claude -p -- PROMPT` |
+| `codex` | `codex exec -- PROMPT` |
+| `opencode` | `opencode run`, with the prompt on stdin |
+| `pi` | `pi --print`, with the prompt on stdin |
+| `marathon` | `marathon --print=PROMPT` |
+| `dsh` | `dsh --profile headless`, with the prompt on stdin |
+
+The prompt is always the argument after `headless`, including for `dsh`, which no longer reads a task piped into subc. Where the table says stdin, subc passes the prompt that way so it reaches the agent unchanged even when it starts with `-` or `@`. On Windows only `dsh` supports headless runs.
+
+The exit status is the agent's own. Pi 1.0.1 exits 1 when the model request fails in its default text output, but 0 with `--mode json`, so check its JSON output for errors. Pi also trims whitespace around a prompt it reads from stdin.
+
+A blank prompt, and `-h` or `--help` anywhere in a headless run, are refused so a scripted run never reports success without doing the task.
+
+Stopping subc with SIGTERM or SIGHUP stops the agent too. SIGKILL cannot be forwarded, so a runner that may kill subc should start it in its own process group and kill the group.
+
+## Harness manifest
+
+`subc harness-manifest` prints how this version of subc installs, configures, and launches each agent above, so other tools can copy the setup exactly instead of re-implementing it.
+
+```bash
+subc harness-manifest --json   # full manifest
+subc harness-manifest          # summary table in a terminal, JSON when piped
+```
+
+The manifest is built when the command runs, from the agent files subc ships (`bin/runbook/<id>/agent.json`). Those files are what the launcher itself executes: launch and headless argv, flags, input defaults, env, and Codex `-c` overrides come from them, so the manifest describes what runs on macOS and Linux. Windows builds its own Codex catalog and `-c` list. The command adds `cli_version` from `package.json`.
+
+| Field | Meaning |
+| --- | --- |
+| `schema_version` | Raised when a field is renamed, removed, or changes meaning. New fields do not raise it. |
+| `cli_version` | The installed `subconscious-cli` version. |
+| `tokens` | Placeholders used in values (listed below). |
+| `runbook_env` | What subc passes to every launch: `order` of the layers, `inherited` sources (profile values, then `process.env`), `fixed` values set last, `runbook_scripts` extras, and `per_harness` extras (Claude's model picker and `SUBC_CLAUDE_SETTINGS`). |
+| `harnesses.<id>` | One entry per agent. |
+
+Each `harnesses.<id>` entry has:
+
+| Field | Meaning |
+| --- | --- |
+| `install` | `method`, `package` or `repository`, `version` or `channel`, `minimum_version`, release `targets`, and the per-OS `commands`. |
+| `prerequisites` | Commands the runbook needs, such as `jq`, with what needs them and whether they are required. `a\|b` means either one. |
+| `binary` | Executable name, or `null` for IDE integrations. |
+| `launch` | `argv` template, `headless_argv` for `subc <agent> headless` (absent when unsupported), `headless_stdin` when the prompt is passed on stdin instead, `headless_platforms` (`darwin`, `linux`, `win32`) where it works, `resume` and `handoff` argv for `subc sessions`, Codex's `config_flag`, and whether a launch also writes persistent files. |
+| `inputs` | Profile or environment variables subc reads for the agent, with `default`, `flag`, `aliases`, and profile metadata. `export: true` inputs are also passed to the agent and listed in `env`. |
+| `env` | Variables the agent receives: `value`, what it is (`description`), and the `override` variables that win when set. |
+| `config` | Files, directories, `-c` overrides (with `condition`), flags, and JSON-in-env. |
+| `capabilities.compaction` | `default`, plus `on`, `off`, and `threshold`, each with the exact knob and whether subc sets it (`set_by_subc`). |
+| `capabilities.mcp` | Whether subc configures MCP, and which transports. |
+| `capabilities.headers` | Headers sent and on which requests. |
+| `capabilities.hooks` | Hook events installed, what each posts, and where. |
+
+A compaction knob has `kind`, `name`, `value`, and `override`. When `kind` is `config-field` or `model-catalog-field`, `config` names the `config` entry (its `path` or `name`) that holds it. `threshold.semantics` says what the number means: `window` is the context size the agent compacts against (Claude Code, OpenCode, Pi, DeepSeek Harness, Copilot); `trigger` is the token count that starts compaction (Codex).
+
+Enumerated values:
+
+| Field | Values |
+| --- | --- |
+| `install.method` | `npm`, `script`, `github-release`, `user`, `null` |
+| `compaction.default` | `on`, `unknown` |
+| knob `kind` | `env`, `cli-config`, `config-field`, `model-catalog-field`, `null` |
+| `threshold.semantics` | `window`, `trigger` |
+| `config[].kind` | `file`, `dir`, `cli-flag`, `cli-config`, `env-json` |
+| `config[].lifetime` | `launch` (removed or not written to disk), `persistent`, `legacy` (left by older setups) |
+
+Placeholders:
+
+| Token | Meaning |
+| --- | --- |
+| `{baseUrl}`, `{baseUrlV1}` | Gateway origin, and the origin followed by `/v1` |
+| `{apiKey}`, `{model}` | The agent's API key (its own key input, else `API_KEY`) and the launch model |
+| `${NAME}` | The value of env var `NAME` at launch; left in place only where it has no default |
+| `{catalog}`, `{catalog[N]}` | All live catalog models (newline-separated), or the one at index N (clamped to the last) |
+| `{args}`, `{config}` | Passed-through arguments, and Codex's `-c` overrides |
+| `{prompt}` | The task given to `subc <agent> headless` |
+| `{sessionId}`, `{sessionFile}` | A local session's ID or file, for `launch.resume` |
+| `{tempFile}`, `{tmp}` | A temporary file the runbook writes (its `config` entry says whether it is removed), the system temp directory |
+| `{opencodeConfig}`, `{claudeSettings}` | Documents shown in the matching `config` entries |
+| `{target}` | A release target triple from `install.targets` or `install.windows_targets` |
+| `{binDir}`, `{PATH}`, `<install dirs>` | Parts of the `PATH` subc builds |
+| `<agent key>` | The agent-specific API key input, such as `CODEX_API_KEY` |
+| `<id>`, `models[]` | Any catalog model ID, and every element of a models array |
+| `<runbook>` | The installed `bin/runbook` directory |
+| `<VS Code user dir>` | The VS Code user settings directory |
+
+`verified: false` means subc does not set the value, or cannot prove its effect; a `note` says which. Entries describing files a runbook writes (the Codex catalog, Pi's `models.json`, the DeepSeek overlay, Copilot's provider, hooks) are descriptions of that script; values they take from inputs are filled in from the input defaults.
+
+## Long screenshot sessions
+
+Coding agents resend every screenshot on every turn, so a long computer-use or screenshot session grows until the gateway refuses it. For vision models, subc applies one rule in Pi and OpenCode before a request leaves the machine: keep the newest screenshots, up to 30 and about 25 MiB in total, and put the text `image` where older ones were. It never removes more than one screenshot per turn, because Subconscious Cache reuses around one removed screenshot, not several; a big screenshot can take the total past 25 MiB for a few turns while it catches up, and only past 30 MiB does it drop straight back to 25. The gateway applies the same rule, so it receives exactly what its own window would produce, and Subconscious Cache still reuses the rest of the conversation: each turn only reads the new screenshot.
+
+| Agent | How subc applies it |
+| --- | --- |
+| OpenCode | A plugin loaded from the launch config (`bin/runbook/opencode/subconscious-image-window.ts`) that gives the provider a trimming `fetch`. |
+| Pi | An extension installed with the provider (`bin/runbook/image-window/index.ts`, on `before_provider_request`). |
+
+The rule is in `bin/runbook/image-window/window.js`. `SUBCONSCIOUS_IMAGE_WINDOW_SOFT_MIB` and `SUBCONSCIOUS_IMAGE_WINDOW_HARD_MIB` change the two limits (defaults 25 and 30). Text-only models are left alone, and a request the integrations do not recognize is sent unchanged.
+
 ## Sessions and cross-harness handoff
 
 Run `subc` and choose **Coding sessions**, or list the same local catalog from

package/bin/agent-data.js

@@ -0,0 +1,242 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+export const RUNBOOK_DIR = fileURLToPath(
+  new URL('./runbook/', import.meta.url),
+);
+
+const ENV_NAME = /^[A-Z_][A-Z0-9_]*$/;
+const PLATFORMS = new Set(['darwin', 'linux', 'win32']);
+const MODES = new Set(['launch', 'setup']);
+const SETUP_ACTIONS = new Set(['install', 'status', 'uninstall']);
+const INPUT_TYPES = new Set([
+  'secret',
+  'integer',
+  'text',
+  'choice',
+  'url',
+  'model',
+]);
+
+function readJson(relative) {
+  const file = new URL(`./runbook/${relative}`, import.meta.url);
+  try {
+    return JSON.parse(readFileSync(file, 'utf-8'));
+  } catch (error) {
+    throw new Error(`Cannot read bin/runbook/${relative}: ${error.message}`);
+  }
+}
+
+function check(condition, where, message) {
+  if (!condition) throw new Error(`Invalid ${where}: ${message}`);
+}
+
+const isObject = (value) =>
+  Boolean(value) && typeof value === 'object' && !Array.isArray(value);
+const isStringList = (value) =>
+  Array.isArray(value) && value.every((item) => typeof item === 'string');
+
+function validateEntry(entry, where, { needsValue = false } = {}) {
+  check(isObject(entry), where, 'expected an object');
+  check(ENV_NAME.test(entry.name ?? ''), where, `bad name ${entry.name}`);
+  if (needsValue) check('value' in entry, where, `${entry.name} has no value`);
+  if (entry.type !== undefined)
+    check(INPUT_TYPES.has(entry.type), where, `bad type ${entry.type}`);
+  if (entry.override !== undefined)
+    check(isStringList(entry.override), where, `${entry.name} override`);
+  if (entry.aliases !== undefined)
+    check(isStringList(entry.aliases), where, `${entry.name} aliases`);
+  if (entry.flag !== undefined)
+    check(/^--[a-z][a-z-]*$/.test(entry.flag), where, `bad flag ${entry.flag}`);
+  if (entry.strict)
+    check(isStringList(entry.choices), where, `${entry.name} needs choices`);
+}
+
+function validateLaunch(launch, where) {
+  check(isObject(launch), where, 'launch agents need a launch block');
+  for (const key of ['argv', 'headless_argv', 'resume', 'handoff']) {
+    if (launch[key] === undefined) continue;
+    check(
+      isStringList(launch[key]) && launch[key].length > 0,
+      where,
+      `launch.${key} must be a non-empty list of strings`,
+    );
+  }
+  check(launch.argv !== undefined, where, 'launch.argv is required');
+  for (const key of ['mode_word', 'headless_stdin', 'config_flag'])
+    if (launch[key] !== undefined)
+      check(typeof launch[key] === 'string', where, `launch.${key}`);
+  if (launch.headless_argv) {
+    const platforms = launch.headless_platforms;
+    check(
+      isStringList(platforms) && platforms.every((p) => PLATFORMS.has(p)),
+      where,
+      'launch.headless_platforms must list darwin, linux, or win32',
+    );
+  }
+}
+
+function validateRunbook(runbook, where) {
+  check(isObject(runbook), where, 'runbook block is required');
+  check(MODES.has(runbook.mode), where, `bad runbook.mode ${runbook.mode}`);
+  const actions = runbook.setup_actions ?? [];
+  check(
+    isStringList(actions) && actions.every((a) => SETUP_ACTIONS.has(a)),
+    where,
+    'runbook.setup_actions may only list install, status, uninstall',
+  );
+  check(
+    actions.length === 0 || typeof runbook.setup_script === 'string',
+    where,
+    'setup actions need runbook.setup_script',
+  );
+}
+
+/** Check one agent file; throws with the file and the first problem found. */
+export function validateAgent(agent, id) {
+  const where = `bin/runbook/${id}/agent.json`;
+  check(isObject(agent), where, 'expected an object');
+  check(agent.id === id, where, `id must be ${id}`);
+  for (const key of ['command', 'name', 'description', 'protocol'])
+    check(typeof agent[key] === 'string' && agent[key], where, `${key}`);
+  check(isStringList(agent.aliases), where, 'aliases must be a list');
+  check(isObject(agent.install), where, 'install block is required');
+  check(isObject(agent.help), where, 'help block is required');
+  check(Array.isArray(agent.help.options), where, 'help.options');
+  validateRunbook(agent.runbook, where);
+  if (agent.runbook.mode === 'launch') validateLaunch(agent.launch, where);
+  for (const key of ['prerequisites', 'inputs', 'env', 'config'])
+    check(Array.isArray(agent[key]), where, `${key} must be a list`);
+  for (const input of agent.inputs) validateEntry(input, `${where} inputs`);
+  check(
+    !agent.config.some((entry) => entry.kind === 'cli-config') ||
+      typeof agent.launch?.config_flag === 'string',
+    where,
+    'cli-config entries need launch.config_flag',
+  );
+  for (const entry of agent.env)
+    validateEntry(entry, `${where} env`, { needsValue: true });
+  const caps = agent.capabilities;
+  check(isObject(caps) && isObject(caps.compaction), where, 'capabilities');
+  return agent;
+}
+
+function loadShared() {
+  const shared = readJson('shared.json');
+  check(isStringList(shared.agents), 'shared.json', 'agents must be ids');
+  check(isObject(shared.defaults), 'shared.json', 'defaults is required');
+  for (const input of shared.inputs ?? [])
+    validateEntry(input, 'shared.json inputs');
+  return shared;
+}
+
+const shared = loadShared();
+
+export const DEFAULTS = Object.freeze(shared.defaults);
+export const PACKAGED_MODELS = Object.freeze(
+  DEFAULTS.models?.length ? [...DEFAULTS.models] : [DEFAULTS.model],
+);
+export const MODEL_CAPABILITIES = Object.freeze(
+  shared.model_capabilities ?? {},
+);
+export const SHARED_INPUTS = Object.freeze(shared.inputs ?? []);
+
+export const AGENTS = Object.freeze(
+  shared.agents.map((id) => validateAgent(readJson(`${id}/agent.json`), id)),
+);
+
+const SHARED_BY_NAME = new Map(SHARED_INPUTS.map((i) => [i.name, i]));
+
+export function agentById(id) {
+  return AGENTS.find((agent) => agent.id === id) ?? null;
+}
+
+/** An agent's inputs, each merged over the shared input of the same name. */
+export function agentInputs(agent) {
+  return agent.inputs.map((input) => ({
+    ...(SHARED_BY_NAME.get(input.name) || {}),
+    ...input,
+  }));
+}
+
+export function agentBinary(agent) {
+  return agent.launch?.argv?.[0] ?? null;
+}
+
+/** Per-OS install commands with `{package}` filled in, or null. */
+export function installCommands(agent) {
+  const { commands, package: name } = agent.install;
+  if (!commands) return null;
+  return Object.fromEntries(
+    Object.entries(commands).map(([os, command]) => [
+      os,
+      name ? command.replaceAll('{package}', name) : command,
+    ]),
+  );
+}
+
+/** The API key input that overrides API_KEY for this agent, if any. */
+export function agentKeyName(agent) {
+  return (
+    agent.inputs.find(
+      (input) => input.type === 'secret' && input.name !== 'API_KEY',
+    )?.name ?? null
+  );
+}
+
+export function agentCommandName(agent) {
+  return agent.command || agent.id;
+}
+
+export function agentSetupActions(agent) {
+  return agent.runbook?.setup_actions ?? [];
+}
+
+/** Where the agent's own installer puts its binary, if it says. */
+export function agentInstallDirs(agent, environment, home) {
+  const { install_dir: dir, install_dir_env: envName } = agent.install;
+  const configured = envName ? environment[envName]?.trim() : '';
+  if (configured) return [path.resolve(configured)];
+  return dir ? [path.resolve(home, dir.replace(/^~[/\\]/, ''))] : [];
+}
+
+const WHOLE_REF = /^\$\{([A-Z_][A-Z0-9_]*)\}$/;
+const REF = /\$\{([A-Z_][A-Z0-9_]*)\}/g;
+
+/**
+ * Replace `${NAME}` references with `lookup(NAME)`, keeping the referenced
+ * value's type when the reference is the whole string. Unknown names stay.
+ */
+export function resolveRefs(value, lookup) {
+  if (typeof value === 'string') {
+    const whole = WHOLE_REF.exec(value);
+    if (whole) {
+      const target = lookup(whole[1]);
+      return target === undefined ? value : resolveRefs(target, lookup);
+    }
+    return value.replace(REF, (match, name) => {
+      const target = lookup(name);
+      return target === undefined ? match : String(resolveRefs(target, lookup));
+    });
+  }
+  if (Array.isArray(value))
+    return value.map((item) => resolveRefs(item, lookup));
+  if (isObject(value))
+    return Object.fromEntries(
+      Object.entries(value).map(([key, item]) => [
+        key,
+        resolveRefs(item, lookup),
+      ]),
+    );
+  return value;
+}
+
+/** Look up an input's default or an env entry's value by name. */
+export function staticLookup(agent) {
+  const values = new Map();
+  for (const entry of agent.env) values.set(entry.name, entry.value);
+  for (const input of agentInputs(agent))
+    if (input.default !== undefined) values.set(input.name, input.default);
+  return (name) => values.get(name);
+}

package/bin/agent-help.js

@@ -0,0 +1,106 @@
+import {
+  agentCommandName,
+  agentInputs,
+  agentSetupActions,
+} from './agent-data.js';
+import { c } from './colors.js';
+import { profileSettingsForAgent, resolvedProfileValues } from './profiles.js';
+
+export function isAgentHelpRequest(argv = []) {
+  return ['help', '-h', '--help'].includes(argv[0]);
+}
+
+function displayProfileValue(setting, value, values) {
+  if (setting.type === 'secret') {
+    if (value) return '(set)';
+    return setting.key !== 'API_KEY' && values.API_KEY
+      ? '(shared key)'
+      : '(not set)';
+  }
+  if (!value && setting.model) return 'UNSET';
+  return value || '(auto)';
+}
+
+function optionText(description) {
+  if (typeof description === 'string') return description;
+  return description[process.platform] ?? description.default;
+}
+
+/** Static options, then documented flags, then the pass-through argument. */
+function helpOptions(agent) {
+  const { options, args } = agent.help;
+  const flags = agentInputs(agent)
+    .filter((input) => input.flag && input.help)
+    .map((input) => [
+      input.metavar ? `${input.flag} ${input.metavar}` : input.flag,
+      input.help,
+    ]);
+  const listed = [...options, ...flags, ...(args ? [args] : [])].map(
+    ([option, description]) => [option, optionText(description)],
+  );
+  const headless =
+    agent.launch?.headless_argv &&
+    !listed.some(([option]) => option.startsWith('headless'));
+  return headless
+    ? [
+        ...listed,
+        ['headless PROMPT', 'Run one task without prompting, then exit'],
+      ]
+    : listed;
+}
+
+function printSettings(agent, profile) {
+  const settings = profileSettingsForAgent(agent.id);
+  const values = resolvedProfileValues(profile);
+  const width = Math.max(0, ...settings.map((setting) => setting.key.length));
+  console.log(
+    `  ${c.bold}Profile settings${c.reset} ${c.dim}(${profile?.name || 'default'})${c.reset}`,
+  );
+  for (const setting of settings) {
+    const value = displayProfileValue(setting, values[setting.key], values);
+    console.log(
+      `    ${c.cyan}${setting.key.padEnd(width)}${c.reset}  ${value}`,
+    );
+    console.log(
+      `    ${' '.repeat(width)}  ${c.dim}${setting.description}${c.reset}`,
+    );
+  }
+}
+
+function printSetupHints(agent, profile) {
+  const command = agentCommandName(agent);
+  console.log(
+    `\n  Edit the env file with ${c.cyan}subc -p ${profile?.name || 'default'} config edit${c.reset}.`,
+  );
+  const actions = agentSetupActions(agent);
+  if (actions.includes('install')) {
+    const what = agent.runbook.binary_install_script
+      ? 'Install the agent binary with'
+      : 'Install the persistent integration with';
+    console.log(`  ${what} ${c.cyan}subc ${command} install${c.reset}.`);
+  }
+  if (actions.includes('uninstall')) {
+    console.log(
+      `  Remove it with ${c.cyan}subc ${command} uninstall${c.reset}.`,
+    );
+  }
+}
+
+export function printAgentHelp(agent, profile) {
+  const options = helpOptions(agent);
+  const width = Math.max(0, ...options.map(([option]) => option.length));
+  console.log(`\n  ${c.bold}${agent.name} + Subconscious${c.reset}\n`);
+  console.log(`  ${agent.help.behavior}\n`);
+  console.log(`  ${c.bold}Usage${c.reset}\n    ${agent.help.usage}\n`);
+  if (options.length) {
+    console.log(`  ${c.bold}Commands and options${c.reset}`);
+    for (const [option, description] of options)
+      console.log(
+        `    ${c.cyan}${option.padEnd(width)}${c.reset}  ${description}`,
+      );
+    console.log();
+  }
+  printSettings(agent, profile);
+  printSetupHints(agent, profile);
+  console.log();
+}