@agentprojectcontext/apx 1.25.0 → 1.27.1

package/skills/apx-voice/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: apx-voice
+scope: optional
+description: How APX handles voice — TTS engines (Piper local, ElevenLabs / OpenAI / Gemini cloud), the unified /voice/turn channel, and apx voice CLI. Load when the user wants to speak with APX, configure a voice engine, or troubleshoot silent output.
+---
+
+# apx-voice
+
+APX has a Text-to-Speech (TTS) facade in `core/voice/` with five engines. STT (speech-to-text) lives separately in `host/daemon/transcription.js` (Whisper). The "voice channel" combines both for a full mic→agent→speaker turn.
+
+## Engines
+
+| id | Local? | Needs key? | Quality | Notes |
+|---|---|---|---|---|
+| `piper`      | yes | no  | Good (es_AR-daniela-high recommended) | Local, offline. Requires `piper` CLI + `.onnx` voice model. |
+| `elevenlabs` | no  | yes | Excellent | Free tier 10k chars/mo. `eleven_multilingual_v2`. |
+| `openai`     | no  | yes | Good | Reuses `engines.openai.api_key`. `tts-1`. |
+| `gemini`     | no  | yes | Good | Returns raw L16 PCM — APX wraps in WAV automatically. |
+| `mock`       | yes | no  | Silent | Silent WAV; placeholder for tests. |
+
+`auto` provider probes in order: piper → elevenlabs → openai → gemini → mock.
+
+## Concrete CLI calls
+
+```bash
+# Inspect what's configured + available
+apx voice providers
+
+# Synthesize and play
+apx voice say "Hola Manú" --provider piper
+apx voice say "Hola Manú" --provider gemini
+apx voice say "..." --no-play          # generate WAV, don't play
+
+# Listen (mic → STT)
+apx voice listen                       # records until silence (sox) or Ctrl+C
+apx voice listen --seconds 5           # fixed-duration capture
+```
+
+Playback uses system binaries (`afplay`, `paplay`, `aplay`, `play`, `ffplay`) — APX doesn't bundle an audio runtime. If none is found, you get the file path and no playback.
+
+## Configuration
+
+`~/.apx/config.json → voice.tts.<engine>`:
+
+```json
+{
+  "voice": {
+    "tts": {
+      "provider": "gemini",
+      "piper":      { "bin": "piper", "model": "/Users/.../es_AR-daniela-high.onnx" },
+      "elevenlabs": { "api_key": "...", "model": "eleven_multilingual_v2", "voice_id": "..." },
+      "openai":     { "api_key": "...", "model": "tts-1", "voice": "alloy", "format": "mp3" },
+      "gemini":     { "api_key": "...", "model": "gemini-2.5-flash-preview-tts", "voice": "Aoede" }
+    }
+  }
+}
+```
+
+`apx config set voice.tts.provider <name>` to switch.
+
+## Quick setup paths
+
+### Piper local (recommended, no internet)
+
+```bash
+# 1. Install binary (macOS arm64)
+curl -L https://github.com/rhasspy/piper/releases/latest/download/piper_macos_aarch64.tar.gz \
+  -o /tmp/piper.tar.gz
+sudo tar xzf /tmp/piper.tar.gz -C /usr/local/bin --strip-components=1
+
+# 2. Voice model (es_AR — Argentine Spanish, daughter)
+mkdir -p ~/.apx/voices
+cd ~/.apx/voices
+curl -LO https://huggingface.co/rhasspy/piper-voices/resolve/main/es/es_AR/daniela/high/es_AR-daniela-high.onnx
+curl -LO https://huggingface.co/rhasspy/piper-voices/resolve/main/es/es_AR/daniela/high/es_AR-daniela-high.onnx.json
+
+# 3. APX config
+apx config set voice.tts.provider piper
+apx config set voice.tts.piper.model "$HOME/.apx/voices/es_AR-daniela-high.onnx"
+
+# 4. Test
+apx voice say "hola Manú" --provider piper
+```
+
+### Gemini cloud (quick if you already have a key)
+
+```bash
+apx config set voice.tts.provider gemini
+apx config set voice.tts.gemini.api_key '<GEMINI_KEY>'
+apx config set engines.gemini.api_key   '<GEMINI_KEY>'    # reuse for LLM router
+apx voice say "hola Manú" --provider gemini
+```
+
+## The unified voice channel
+
+`POST /voice/turn` is one round-trip: send audio (or text), get back `{ user_text, reply_text, reply_audio_path }`. STT in, agent loop, TTS out. Surface for the overlay and any future "voice room" client.
+
+```bash
+# Drive from curl with already-transcribed text (skip STT)
+curl -X POST http://127.0.0.1:7430/voice/turn \
+  -H "Authorization: Bearer $(cat ~/.apx/daemon.token)" \
+  -H "Content-Type: application/json" \
+  -d '{"text":"Hola APX","channel":"voice"}'
+```
+
+Telegram voice messages and the overlay mascot still have their own STT pipelines today — they don't go through `/voice/turn` (yet). The endpoint exists for callers that want one-shot bidirectional voice.
+
+## Anti-examples
+
+```bash
+# DON'T trust `apx voice providers` saying "mock available" as a green light.
+# Mock is silence; useful for tests, useless for talking to humans.
+# If only mock is "available", configure a real provider.
+
+# DON'T set voice.tts.provider to a provider with no key.
+# It will fall through `auto` to the next, but that's not what you asked for.
+
+# DON'T expect Gemini TTS to give you an MP3.
+# Today it returns raw L16 PCM. APX wraps it in a 44-byte WAV header so afplay
+# accepts it. Files are .wav, mime "audio/wav". If you need MP3, convert with ffmpeg.
+```
+
+## Troubleshooting silent output
+
+1. `apx voice providers` — what's actually available?
+2. `apx voice say "test" --provider <engine> --no-play` — does the file exist?
+3. `file <path>` — is it a valid container? Gemini's output should be `RIFF WAVE Microsoft PCM`.
+4. `afplay <path>` directly — does the OS player open it?
+5. If 3 fails for Gemini, you may be on an older APX before the PCM-wrap fix (commit `ba5c416` or later).
+
+## Don't
+
+- Don't paste base64 audio into chat. Use file paths or upload via `send_voice` / `send_audio`.
+- Don't switch providers mid-routine without testing — voice quality varies a lot between Piper voices and cloud engines.
+- Don't expect TTS streaming yet — `apx voice say` returns a complete file. A `/tts/stream` endpoint with chunked audio is open work.

package/src/core/agent/constants.js

@@ -0,0 +1,3 @@
+export const MAX_TOOL_ITERS = 6;
+export const ACK_ONLY_TOOLS = new Set(["send_telegram"]);
+export const MAX_CONSECUTIVE_ACKS = 2;

package/src/core/agent/ghost-guard.js

@@ -0,0 +1,24 @@
+export function isShortConfirmation(text) {
+  return /^(yes|y|si|si dale|dale|ok|okay|confirm|confirmed|go|proceed|do it)\b/i
+    .test(String(text || "").trim());
+}
+
+export function lastAssistantAskedForConfirmation(messages) {
+  for (let i = messages.length - 1; i >= 0; i--) {
+    if (messages[i]?.role !== "assistant") continue;
+    return /\b(confirm|confirmation|ok|okay|permission|allowed|proceed|do it|dale)\b/i.test(messages[i].content || "");
+  }
+  return false;
+}
+
+export function isGhostResponse(text) {
+  const t = String(text || "").trim();
+  if (t.length > 200) return false;
+  return /^(ok|okay|got it|understood|sure|of course|on it|dale|entendido|claro|voy|ya lo hago|dame un (segundo|momento)|un momento|let me|i (will|can|shall)|i'm (going|about)|give me a|ahora lo|enseguida|checking|looking|fetching|working on|stand by|please wait|un seg|dame sec)[\s.,!]*/i
+    .test(t);
+}
+
+export function looksLikeActionRequest(text) {
+  const t = String(text || "").trim().toLowerCase();
+  return /\b(list|show|find|get|fetch|search|run|execute|create|add|make|start|stop|delete|update|send|check|read|write|look|tell me|dame|mostra|busca|ejecuta|crea|agrega|mandá|revisá|corré|borrá|arrancá)\b/.test(t);
+}

package/src/core/agent/index.js

@@ -0,0 +1,35 @@
+export {
+  MAX_TOOL_ITERS,
+  ACK_ONLY_TOOLS,
+  MAX_CONSECUTIVE_ACKS,
+} from "./constants.js";
+export {
+  buildIdentityBlock,
+  buildUserContextBlock,
+  buildChannelContextBlock,
+  buildRelationshipBlock,
+  buildSuperAgentSystem,
+  isSuperAgentEnabled,
+  loadDefaultSystemPrompt,
+  loadPrompt,
+  renderPromptTemplate,
+  DEFAULT_SYSTEM,
+} from "./prompt-builder.js";
+export {
+  parseModelId,
+  resolveActiveModel,
+  checkProviderHealth,
+  probeAllProviders,
+  fallbackOrder,
+  fallbackModels,
+  modelForProvider,
+  isFallbackEnabled,
+  DEFAULT_FALLBACK_ORDER,
+  DEFAULT_FALLBACK_MODELS,
+} from "./model-router.js";
+export { runAgent } from "./run-agent.js";
+export {
+  POSTCMD_TOOL_OVERLAP,
+  computeSuppressedTools,
+  filterToolSchemas,
+} from "./tools-overlap.js";

package/src/core/agent/model-router.js

@@ -0,0 +1,257 @@
+// Model router: thin orchestrator over engine adapters.
+//
+// What used to be a long per-provider switch (health checks, env-var maps,
+// default fallback models, base URLs) now lives in each adapter under
+// src/core/engines/<provider>.js. The router only knows the grammar of model
+// ids and the algorithm of trying the chain in order. Per-provider details
+// are owned by the provider.
+import { getAdapter } from "../engines/index.js";
+
+export function parseModelId(modelId) {
+  if (typeof modelId !== "string" || !modelId) {
+    throw new Error("model id is empty");
+  }
+  if (modelId.includes(":")) {
+    const [provider, ...rest] = modelId.split(":");
+    return { provider: provider.toLowerCase(), model: rest.join(":") };
+  }
+  if (/^claude/i.test(modelId)) return { provider: "anthropic", model: modelId };
+  if (/^gpt|^o1|^o3|^o4/i.test(modelId)) return { provider: "openai", model: modelId };
+  if (/^gemini/i.test(modelId)) return { provider: "gemini", model: modelId };
+  if (modelId === "mock") return { provider: "mock", model: "mock" };
+  throw new Error(`cannot infer provider for model "${modelId}"`);
+}
+
+// Default chain when no super_agent.model_fallback is configured. Order: the
+// cloud providers that have a published default model id. Ollama isn't here
+// because it depends on the user's pulled models. Anthropic/Gemini neither
+// (key-gated, no canonical free fallback).
+export const DEFAULT_FALLBACK_ORDER = ["openrouter", "groq"];
+
+// Built on demand from each adapter's `defaultFallbackModel`. Kept as a
+// getter so adding an engine doesn't require updating this file.
+function getDefaultFallbackModels() {
+  const map = {};
+  for (const id of DEFAULT_FALLBACK_ORDER) {
+    try {
+      const a = getAdapter(id);
+      if (a?.defaultFallbackModel) map[id] = a.defaultFallbackModel;
+    } catch { /* missing adapter — skip */ }
+  }
+  return map;
+}
+export const DEFAULT_FALLBACK_MODELS = getDefaultFallbackModels();
+
+function engineCfg(config, provider) {
+  return (config?.engines && config.engines[provider]) || {};
+}
+
+/**
+ * Delegate to the provider's adapter for the actual probe. The router used to
+ * have a per-provider switch here; that's now adapter responsibility.
+ *
+ * `candidateModel` is the bare model id (no "<provider>:" prefix). Adapters
+ * that care (Ollama strict mode) use it; others ignore it.
+ */
+export async function checkProviderHealth(provider, config, timeoutMs = 800, opts = {}) {
+  const p = String(provider || "").toLowerCase();
+  let adapter;
+  try {
+    adapter = getAdapter(p);
+  } catch {
+    return { ok: false, provider: p, reason: "unknown provider" };
+  }
+  if (typeof adapter.health !== "function") {
+    return { ok: false, provider: p, reason: "adapter has no health()" };
+  }
+  return adapter.health(engineCfg(config, p), { timeoutMs, candidateModel: opts.candidateModel || null });
+}
+
+/**
+ * Return the fallback chain as a flat list of model ids in the order they
+ * should be attempted. Each item is a fully-qualified `<provider>:<model>`
+ * string — no separate `order` array, no `models{provider}` map.
+ *
+ * Reads three formats, in priority order:
+ *
+ *  1. **New (preferred)**: `model_fallback.models` is an array of strings.
+ *     Order = array order. Each string carries its own provider prefix.
+ *
+ *       "model_fallback": { "models": ["openrouter:foo", "groq:bar"] }
+ *
+ *  2. **Legacy**: `model_fallback.order` is a provider list +
+ *     `model_fallback.models` is `{ <provider>: "<provider>:<model>" }`.
+ *     Walked in `order`; missing entries fill from DEFAULT_FALLBACK_MODELS or
+ *     (for Ollama) from `super_agent.model`.
+ *
+ *  3. **None**: no fallback configured → defaults derived from
+ *     DEFAULT_FALLBACK_ORDER + DEFAULT_FALLBACK_MODELS.
+ *
+ * The primary model (`super_agent.model`) is NOT included here — it's tried
+ * first by `resolveActiveModel`. This function returns only the alternates.
+ */
+export function fallbackModels(globalConfig) {
+  const sa = globalConfig?.super_agent || {};
+  const fb = sa.model_fallback || {};
+
+  // Format 1 — new
+  if (Array.isArray(fb.models)) {
+    return fb.models
+      .filter((m) => typeof m === "string" && m.includes(":"))
+      .map(String);
+  }
+
+  // Format 2 — legacy
+  const legacyMap = fb.models && typeof fb.models === "object" ? fb.models : null;
+  const order = Array.isArray(fb.order) ? fb.order.map(String) : null;
+  if (legacyMap || order) {
+    const chain = order || DEFAULT_FALLBACK_ORDER;
+    const out = [];
+    for (const provider of chain) {
+      const p = String(provider).toLowerCase();
+      let model = legacyMap?.[p];
+      if (!model) {
+        if (p === "ollama" && typeof sa.model === "string" && /^ollama:/i.test(sa.model)) {
+          model = sa.model;
+        } else if (DEFAULT_FALLBACK_MODELS[p]) {
+          model = DEFAULT_FALLBACK_MODELS[p];
+        }
+      }
+      if (model && typeof model === "string" && model.includes(":")) out.push(model);
+    }
+    return out;
+  }
+
+  // Format 3 — empty config, derive from defaults
+  return DEFAULT_FALLBACK_ORDER
+    .map((p) => DEFAULT_FALLBACK_MODELS[p])
+    .filter((m) => typeof m === "string" && m.includes(":"));
+}
+
+/**
+ * @deprecated use fallbackModels(). Kept for tests / external callers that
+ * still ask "what provider to try after Ollama?". Derives the answer from the
+ * resolved model chain.
+ */
+export function fallbackOrder(globalConfig) {
+  const models = fallbackModels(globalConfig);
+  const providers = [];
+  for (const m of models) {
+    try {
+      const p = parseModelId(m).provider;
+      if (!providers.includes(p)) providers.push(p);
+    } catch { /* skip malformed entries */ }
+  }
+  return providers.length ? providers : [...DEFAULT_FALLBACK_ORDER];
+}
+
+/**
+ * @deprecated use fallbackModels(). Looks up a single provider's model in
+ * the resolved chain. Returns "" if the provider isn't in the fallback list.
+ */
+export function modelForProvider(globalConfig, provider) {
+  const p = String(provider).toLowerCase();
+  const sa = globalConfig?.super_agent || {};
+  const models = fallbackModels(globalConfig);
+  const match = models.find((m) => {
+    try { return parseModelId(m).provider === p; } catch { return false; }
+  });
+  if (match) return match;
+  // Ollama gets a special fallback to the primary model (legacy behavior).
+  if (p === "ollama" && typeof sa.model === "string" && /^ollama:/i.test(sa.model)) {
+    return sa.model;
+  }
+  return DEFAULT_FALLBACK_MODELS[p] || "";
+}
+
+export function isFallbackEnabled(globalConfig) {
+  const fb = globalConfig?.super_agent?.model_fallback || {};
+  return fb.enabled !== false;
+}
+
+/**
+ * Pick first healthy model following configured provider order.
+ */
+export async function resolveActiveModel(globalConfig, { overrideModel = null, timeoutMs } = {}) {
+  if (overrideModel) {
+    const { provider } = parseModelId(overrideModel);
+    return {
+      modelId: overrideModel,
+      provider,
+      fromFallback: false,
+      forced: true,
+      tried: [{ provider, modelId: overrideModel, healthy: true, reason: "override" }],
+    };
+  }
+
+  const sa = globalConfig?.super_agent || {};
+  const fb = sa.model_fallback || {};
+  const healthMs = timeoutMs ?? fb.health_timeout_ms ?? 800;
+  const tried = [];
+
+  if (isFallbackEnabled(globalConfig)) {
+    // Build the full chain: primary first, then the fallback list, deduped.
+    // Each entry is a fully-qualified "<provider>:<model>" string.
+    const chain = [];
+    if (typeof sa.model === "string" && sa.model.includes(":")) chain.push(sa.model);
+    for (const m of fallbackModels(globalConfig)) {
+      if (!chain.includes(m)) chain.push(m);
+    }
+
+    for (const modelId of chain) {
+      let provider;
+      try {
+        provider = parseModelId(modelId).provider;
+      } catch (e) {
+        tried.push({ provider: null, modelId, healthy: false, reason: e.message });
+        continue;
+      }
+      const candidateModel = parseModelId(modelId).model;
+      const health = await checkProviderHealth(provider, globalConfig, healthMs, { candidateModel });
+      tried.push({
+        provider,
+        modelId,
+        healthy: health.ok,
+        reason: health.reason || (health.ok ? "ok" : "unhealthy"),
+        soft: health.soft,
+        available: health.available, // for "model not loaded" diagnostics
+      });
+      if (health.ok) {
+        const isPrimary = modelId === sa.model;
+        return {
+          modelId,
+          provider,
+          fromFallback: !isPrimary,
+          tried,
+        };
+      }
+    }
+  }
+
+  if (sa.model) {
+    const { provider } = parseModelId(sa.model);
+    return {
+      modelId: sa.model,
+      provider,
+      fromFallback: false,
+      forced: true,
+      tried,
+    };
+  }
+
+  throw new Error("no model configured (set super_agent.model in ~/.apx/config.json)");
+}
+
+export async function probeAllProviders(globalConfig, timeoutMs) {
+  const order = fallbackOrder(globalConfig);
+  const out = [];
+  for (const provider of order) {
+    const health = await checkProviderHealth(provider, globalConfig, timeoutMs);
+    out.push({
+      provider,
+      model: modelForProvider(globalConfig, provider) || "(not set)",
+      ...health,
+    });
+  }
+  return out;
+}