@andreprado/agentkit 0.1.0-alpha.15 → 0.1.0-alpha.16

package/src/templates/skills/agentkit-channels/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: agentkit-channels
-description: Use when adding, connecting, testing, buffering, or debugging AgentKit website, Telegram, or WhatsApp channels, including channel config helpers, provider secrets, webhook setup, channel tests, delivery logs, and burst-message buffers.
+description: Use when adding, connecting, testing, buffering, transcribing audio, or debugging AgentKit website, Telegram, or WhatsApp channels, including channel config helpers, provider secrets, webhook setup, channel tests, delivery logs, burst-message buffers, and transcription provider secrets.
 ---
 
 # AgentKit Channels
@@ -16,6 +16,39 @@ Channels receive user messages. Tools let the agent call external systems. Keep
 5. Connect channel resources through the CLI.
 6. Test, doctor, and inspect delivery logs.
 
+## Audio Transcription
+
+Enable transcription at the agent level and opt in per channel with `audio.mode: "transcribe"`.
+
+```ts
+export default defineAgent({
+  // ...
+  transcription: {
+    provider: "groq",
+    model: "whisper-large-v3-turbo",
+    secret: "GROQ_API_KEY",
+    language: "pt",
+    limits: {
+      maxDurationSeconds: 180,
+      maxBytes: 20_000_000,
+    },
+  },
+  channels: [
+    telegramChannel({
+      name: "support-telegram",
+      audio: { mode: "transcribe" },
+    }),
+  ],
+});
+```
+
+V1 providers:
+
+- `openai`: `gpt-4o-mini-transcribe`, `gpt-4o-transcribe`, `whisper-1`; default secret `OPENAI_API_KEY`.
+- `groq`: `whisper-large-v3-turbo`, `whisper-large-v3`, `distil-whisper-large-v3-en`; default secret `GROQ_API_KEY`.
+
+Telegram voice notes are usually OGG/Opus, so use Groq for the default Telegram voice-note path in V1. Zapster audio needs a usable HTTPS Zapster media download URL in the webhook payload; arbitrary hosts are rejected before bearer auth is sent. Hosted channel creation requires the transcription secret automatically when the channel enables transcription. Webhooks only enqueue audio jobs; download and transcription run in the retryable channel worker before the agent run.
+
 ## Buffering
 
 Enable `buffer.mode: "debounce"` when clients send several short messages in a row and the agent should answer once.

package/src/templates/skills/agentkit-channels/references/channel-debugging.md

@@ -22,6 +22,10 @@ Common states:
 webhook_received
 validated
 duplicate
+audio_received
+audio_downloaded
+transcribing
+transcribed
 buffered
 queued
 running
@@ -43,6 +47,15 @@ Common errors:
 - `channel_signature_invalid`: webhook secret, token, or origin header mismatch.
 - `channel_payload_invalid`: malformed or unsupported provider payload.
 - `channel_event_duplicate`: provider retry; do not create a second run.
+- `audio_received`: audio message was accepted and normalized.
+- `audio_downloaded`: retryable channel worker downloaded provider media into memory.
+- `transcribing`: AgentKit is calling the configured transcription provider.
+- `transcribed`: transcript text was queued for the agent.
+- `channel_audio_download_unavailable`: provider audio payload did not include a usable download URL, or Zapster sent a non-HTTPS/non-Zapster media host.
+- `transcription_secret_missing`: managed transcription secret is missing.
+- `transcription_audio_too_large` or `transcription_audio_too_long`: audio exceeded configured limits.
+- `transcription_audio_format_unsupported`: provider does not accept this audio MIME type or extension.
+- `transcription_provider_unavailable`: retryable transcription provider failure.
 - `channel_limit_exceeded`: backpressure skipped the message.
 - `synthetic_expected_failure`: a synthetic test reached AgentKit, but the provider correctly rejected a fake test recipient.
 - `buffered` delivery state: message is waiting for the channel quiet window or max wait before one coalesced agent run is queued.

package/src/templates/skills/agentkit-channels/references/telegram.md

@@ -7,6 +7,12 @@ TELEGRAM_BOT_TOKEN
 TELEGRAM_WEBHOOK_SECRET
 ```
 
+Audio transcription also needs the configured transcription secret, usually:
+
+```txt
+GROQ_API_KEY
+```
+
 Commands:
 
 ```sh
@@ -36,3 +42,29 @@ telegramChannel({
   },
 })
 ```
+
+Transcribe Telegram voice notes:
+
+```ts
+export default defineAgent({
+  // ...
+  transcription: {
+    provider: "groq",
+    model: "whisper-large-v3-turbo",
+    secret: "GROQ_API_KEY",
+    language: "pt",
+    limits: {
+      maxDurationSeconds: 180,
+      maxBytes: 20_000_000,
+    },
+  },
+  channels: [
+    telegramChannel({
+      name: "support-telegram",
+      audio: { mode: "transcribe" },
+    }),
+  ],
+});
+```
+
+AgentKit validates the Telegram webhook, normalizes `voice` and `audio` payloads, enqueues an audio job, then the retryable channel worker calls Telegram `getFile`, downloads the media with `TELEGRAM_BOT_TOKEN`, sends the bytes to the configured transcription provider, and runs the agent with transcript text. Telegram voice notes are usually OGG/Opus; use Groq in V1 for that path.

package/src/templates/skills/agentkit-channels/references/whatsapp-zapster.md

@@ -14,6 +14,8 @@ Optional hardening secret:
 ZAPSTER_WEBHOOK_TOKEN
 ```
 
+Audio transcription also needs the configured transcription secret, usually `OPENAI_API_KEY` or `GROQ_API_KEY`.
+
 Commands:
 
 ```sh
@@ -46,3 +48,30 @@ whatsappChannel({
   },
 })
 ```
+
+Transcribe WhatsApp audio:
+
+```ts
+export default defineAgent({
+  // ...
+  transcription: {
+    provider: "openai",
+    model: "gpt-4o-mini-transcribe",
+    secret: "OPENAI_API_KEY",
+    language: "pt",
+    limits: {
+      maxDurationSeconds: 180,
+      maxBytes: 20_000_000,
+    },
+  },
+  channels: [
+    whatsappChannel({
+      name: "support-whatsapp",
+      provider: "zapster",
+      audio: { mode: "transcribe" },
+    }),
+  ],
+});
+```
+
+Zapster audio payloads must include a usable HTTPS Zapster media download URL such as `audio.downloadUrl`, `audio.url`, `audio.mediaUrl`, or the snake_case equivalents. AgentKit rejects arbitrary hosts before sending `ZAPSTER_API_KEY`. The retryable channel worker downloads the media, transcribes it through the configured provider secret, and runs the agent with transcript text. If Zapster sends only a media ID in V1, AgentKit records `channel_audio_download_unavailable`.