@agent-native/core 0.14.8 → 0.15.0

package/src/templates/workspace-core/.agents/skills/voice-transcription/SKILL.md

@@ -0,0 +1,124 @@
+---
+name: voice-transcription
+description: >-
+  Framework-wide voice dictation in the agent sidebar composer. Use when
+  changing composer microphone UX, the transcribe-voice route, or the
+  Voice Transcription settings section. Covers transcription-source routing,
+  cleanup routing, Google realtime gating, and the voice transcription
+  application-state keys.
+---
+
+# Voice Transcription
+
+Click-to-toggle microphone inside the sidebar composer turns speech into
+text. Users configure live transcription separately from AI cleanup in
+Settings → Voice Transcription. The feature is available in every template
+that renders `TiptapComposer`.
+
+## UX rules
+
+- **Always show the mic alongside the send button.** Cursor replaces send
+  with mic when the composer is empty; their users complain. We keep both
+  visible — Lovable does the same.
+- **Click-to-toggle, not push-to-talk.** More forgiving in a sidebar, avoids
+  host-app hotkey clashes. Keyboard shortcut is `Cmd/Ctrl+Shift+M` and
+  `Escape` cancels mid-recording.
+- **Transcript lands in the composer, editable, never auto-sent.** Insert at
+  the caret via `editor.chain().focus().insertContent(text).run()`.
+- **No CSS transitions for the recording state.** Framework rule; use static
+  brand color (`#625DF5`) instead of pulses.
+- **Icon:** Tabler `IconMicrophone` (idle) / `IconPlayerStopFilled` (recording).
+  Never use a sparkle or robot icon.
+- **Errors via inline alert or toast, never `window.alert`.**
+
+## Source And Cleanup
+
+Settings must keep these as separate choices:
+
+- **Live transcription source**: `mac-native`, `google-realtime`, or `batch`.
+- **AI cleanup**: independent off/on toggle. Cleanup uses Builder Gemini first
+  when hosted Gemini is configured, then BYOK Gemini (`GEMINI_API_KEY`).
+  Gemini cleanup/title/summary generation is not a live STT source.
+
+`application_state["voice-transcription-prefs"]` stores
+`{ transcriptionMode, provider, instructions }`. The legacy `provider` field
+is still written for old clients and batch provider preferences:
+
+| Value             | Meaning                                                        | Needs key                    |
+| ----------------- | -------------------------------------------------------------- | ---------------------------- |
+| `mac-native`      | Native macOS/Tauri speech path; web clients normalize to browser-native where needed | No                           |
+| `google-realtime` | Dedicated WebSocket → Google Speech-to-Text gRPC `StreamingRecognize` path | `GOOGLE_APPLICATION_CREDENTIALS` |
+| `batch`           | Upload audio after stop through the existing batch route       | Builder/Gemini/Groq/OpenAI depending on fallback |
+| `auto` provider   | Existing batch fallback chain                                  | Any configured batch provider |
+| `builder-gemini`  | Builder Gemini Flash-Lite batch/cleanup preference             | hosted Gemini provider connected |
+| `gemini`          | Direct Google Gemini BYOK batch/cleanup preference             | `GEMINI_API_KEY`             |
+| `groq`            | Groq Whisper batch preference                                  | `GROQ_API_KEY`               |
+| `openai`          | OpenAI Whisper batch preference                                | `OPENAI_API_KEY`             |
+| `browser`         | Legacy native/browser live speech preference                   | No                           |
+
+Default behavior:
+
+- The shared web settings/composer default to Batch / `auto`.
+- Dedicated macOS Tauri-native surfaces may save `mac-native`, but do not
+  assume the shared React settings default to it.
+- Old stored `builder` values are treated as `builder-gemini`.
+- Old stored `browser` values are treated as `mac-native`.
+- Saved `google-realtime` preferences must never hit `/_agent-native/transcribe-voice`. They go through the dedicated session bridge `POST /_agent-native/transcribe-stream/session`, which mints an opaque ai-services websocket session and keeps the Google service-account JSON off the client.
+- In the current bridge, the Google option is only actually ready when both the user's `GOOGLE_APPLICATION_CREDENTIALS` secret exists and Builder is connected, because the framework mints the managed ai-services session with Builder auth before streaming begins.
+
+## Where the pieces live
+
+| File                                                                  | Purpose                                             |
+| --------------------------------------------------------------------- | --------------------------------------------------- |
+| `packages/core/src/client/composer/useVoiceDictation.ts`              | Provider-routing hook (MediaRecorder / Web Speech)  |
+| `packages/core/src/client/composer/VoiceButton.tsx`                   | Mic button + live amplitude + cancel overlay        |
+| `packages/core/src/client/composer/TiptapComposer.tsx`                | Wires the hook, insertion, and keyboard shortcut    |
+| `packages/core/src/client/settings/VoiceTranscriptionSection.tsx`     | Live source + cleanup controls in sidebar settings  |
+| `packages/core/src/client/transcription/BuilderTranscriptionCta.tsx`  | CTA shown when Builder account isn't connected      |
+| `packages/core/src/client/transcription/use-live-transcription.ts`    | Web Speech live-transcription hook for recordings   |
+| `packages/core/src/server/transcribe-voice.ts`                        | Route handler (routes to Builder/Gemini/Groq/Whisper) |
+| `packages/core/src/transcription/builder-transcription.ts`            | Builder proxy transcription client                  |
+| `packages/core/src/secrets/register-framework-secrets.ts`             | Framework-level provider key registration           |
+
+## Key resolution (server)
+
+`transcribe-voice.ts` is batch-only. Do not add realtime streaming to this
+route. Google Speech-to-Text realtime uses a dedicated audio-frame protocol:
+client audio frames → `/_agent-native/transcribe-stream/session` →
+ai-services WebSocket → Google gRPC `StreamingRecognize` → partial / final
+transcript events. Use the canonical docs URL:
+https://cloud.google.com/speech-to-text/v2/docs/streaming-recognize
+
+Batch routing is based on the user's provider preference:
+
+1. If `builder-gemini` and `resolveHasBuilderPrivateKey()` → calls `transcribeWithBuilder({ model: "gemini-3-1-flash-lite" })` via Builder proxy, or uses Builder Gemini Flash-Lite to clean up a live native/browser transcript when the desktop client sends text instead of audio.
+2. If `builder` and `resolveHasBuilderPrivateKey()` → legacy alias; prefer `builder-gemini`.
+3. If `gemini` → resolves `GEMINI_API_KEY` and calls the direct Google Gemini path.
+4. If `groq` → resolves `GROQ_API_KEY` and calls Groq's Whisper-compatible endpoint.
+5. If `openai` → resolves `OPENAI_API_KEY`:
+   - `readAppSecret({ key: "OPENAI_API_KEY", scope: "user", scopeId: session.email })` — user's encrypted secret.
+   - `resolveCredential("OPENAI_API_KEY")` — env var + SQL settings fallback.
+
+In auto mode / no preference, the route tries Builder Gemini Flash-Lite first
+when Builder is connected, then Gemini BYOK, Groq, and OpenAI.
+When a request includes `instructions`, pass them through to the selected LLM
+provider. Gemini uses them in the transcription prompt, Builder receives them
+as transcription/cleanup instructions, and Whisper-compatible providers receive
+them as provider prompt/context.
+
+Never hardcode a shared key. Never log the value. Never echo it back to the
+client.
+
+## Overriding per-template
+
+Templates can:
+- **Disable the mic**: pass `voiceEnabled={false}` to `TiptapComposer`.
+- **Replace the button**: wrap `TiptapComposer` and render your own `extraActionButton` (the framework mic sits between `extraActionButton` and the send button).
+- **Pre-register provider keys as `required: true`**: call `registerRequiredSecret(...)` from your own server plugin when a template needs a specific BYOK provider in onboarding.
+
+## Don'ts
+
+- Don't call transcription providers from the client — go through `/_agent-native/transcribe-voice` so the user's secret stays server-side.
+- Don't remove the cancel affordance — mic permission abuse paranoia is real.
+- Don't auto-submit the transcript — users always edit before sending.
+- Don't copy Cursor's "hide send when empty" pattern — it confuses users.

package/src/templates/workspace-core/AGENTS.md

@@ -11,12 +11,22 @@ agent should know.
 
 ## Shared Conventions
 
+- All AI/LLM behavior goes through the app's agent chat. UI and server code
+  must not call model providers, AI SDK `generateText()` / `streamText()`, or
+  other inline LLM APIs directly. Use `sendToAgentChat()` for local app-agent
+  work, and read `.agents/skills/delegate-to-agent/SKILL.md` before building
+  agent-driven UI or "AI" features.
 - Put shared code in `packages/shared` only when multiple apps need it.
 - Keep app-specific screens, actions, state, and skills inside `apps/<app>`.
 - Store shared runtime configuration in the workspace root `.env`; use
   `apps/<app>/.env` only for app-specific overrides.
 - Prefer framework defaults until the workspace has a real custom rule,
   component, plugin, action, or skill to share.
+- Keep the Workspace files view for user-authored or user-requested resources.
+  Agents may create hidden `agent_scratch` resources for temporary working
+  notes, scripts, task plans, or intermediate outputs, but should promote them
+  to normal workspace visibility only when the user explicitly asks to keep or
+  manage the file.
 
 ## Adding Apps
 
@@ -33,6 +43,10 @@ Do not implement a new app by adding a route, page, component, or file to
 `apps/starter` or another existing app unless the user explicitly asks to modify
 that existing app.
 
+Dispatch vault access is workspace-wide by default: every saved vault key is
+available to every workspace app. Only create or request per-app vault grants
+when Dispatch's vault access setting is switched to manual mode.
+
 Workspace apps are discovered from `apps/<app-name>/package.json`. There is no
 separate workspace app registry to edit for Dispatch to list the app. Use
 relative workspace links like `/<app-name>` and never hardcode `localhost`,
@@ -42,7 +56,8 @@ port. React Router apps must preserve `APP_BASE_PATH` / `VITE_APP_BASE_PATH` in
 `app/entry.client.tsx` via `appBasePath()` so the app hydrates correctly when
 mounted at `/<app-name>`. Use the framework/template UI stack for standard UI:
 shadcn/ui components and `@tabler/icons-react`. Do not add `lucide-react` or
-another icon library.
+another icon library. Read `.agents/skills/shadcn-ui/SKILL.md` before adding,
+updating, or debugging shadcn components.
 
 In local development, run
 `pnpm exec agent-native create <app-name> --template=<template>` from the

package/src/templates/workspace-root/AGENTS.md

@@ -3,6 +3,27 @@
 These instructions apply at the workspace root. App-specific behavior belongs
 in `apps/<app>/AGENTS.md`; shared cross-app behavior belongs in
 `packages/shared/AGENTS.md` or `packages/shared/.agents/skills/`.
+The root `.agents/skills` path points at the shared package's skills so local
+coding agents can discover the same workspace-wide guidance from the root.
+
+## Core Agent Rule
+
+- All AI/LLM behavior goes through the app's agent chat. UI and server code
+  must not call model providers, AI SDK `generateText()` / `streamText()`, or
+  other inline LLM APIs directly. Use `sendToAgentChat()` for local app-agent
+  work, and read `packages/shared/.agents/skills/delegate-to-agent/SKILL.md`
+  before building agent-driven UI or "AI" features.
+
+## Workspace Resources
+
+- The Workspace files view is for user-authored or user-requested resources
+  they intentionally add, edit, or manage.
+- Agents may use hidden `agent_scratch` resources for temporary working notes,
+  scripts, task plans, or intermediate outputs. Keep those scratch files hidden
+  by default and promote them only when the user explicitly asks to keep or
+  manage the file.
+- Durable instructions, skills, jobs, memories, custom agents, and files the
+  user explicitly asked to save belong in normal workspace visibility.
 
 ## Workspace Scope
 
@@ -26,6 +47,9 @@ in `apps/<app>/AGENTS.md`; shared cross-app behavior belongs in
   `/<app-id>`.
 - When a user explicitly asks for a new app or workspace app, create the
   separate workspace app.
+- Dispatch vault access is workspace-wide by default: every saved vault key is
+  available to every workspace app. Only create or request per-app vault grants
+  when Dispatch's vault access setting is switched to manual mode.
 - Do not satisfy a new-app request by adding a route, page, component, or file
   to `apps/starter` or another existing app unless the user explicitly asks to
   modify that existing app.
@@ -44,6 +68,15 @@ in `apps/<app>/AGENTS.md`; shared cross-app behavior belongs in
   base template improvements continue to flow automatically.
 - Workspace apps are discovered from `apps/<app-id>/package.json`. There is no
   separate workspace app registry to edit for Dispatch to list the app.
+- Always save a concise, human-readable `description` in the generated app's
+  `apps/<app-id>/package.json`. Dispatch lists and A2A connected-agent context
+  use the app name plus this description so other agents understand what the app
+  does. Dispatch users can later edit the displayed name/description from the
+  Apps page without changing source.
+- All sibling workspace apps are accessible by default over A2A through
+  `call-agent`. Agents receive a compact list of available app names and
+  descriptions in prompt context; use tool search or app-specific actions only
+  when more detail is needed.
 - Use relative workspace links like `/<app-id>`. Never hardcode
   `localhost`, `127.0.0.1`, `8080`, `8100`, or any dev port in app cards,
   instructions, redirects, or navigation; the active workspace gateway/browser
@@ -53,6 +86,8 @@ in `apps/<app>/AGENTS.md`; shared cross-app behavior belongs in
   when mounted at `/<app-id>`.
 - Use the framework/template UI stack for standard UI: shadcn/ui components and
   `@tabler/icons-react`. Do not add `lucide-react` or another icon library.
+  Read `packages/shared/.agents/skills/shadcn-ui/SKILL.md` before adding,
+  updating, or debugging shadcn components.
 - In local development, scaffold the app from the workspace root with
   `pnpm exec agent-native create <app-id> --template=<template>`. In production
   Dispatch posts the request to Builder branch creation; the Builder branch

package/src/templates/workspace-root/README.md

@@ -33,6 +33,9 @@ Apps don't need any configuration to opt in. Discovery happens via the
 `agent-native.workspaceCore` field in this root `package.json`, which names
 the shared package (`@{{APP_NAME}}/shared`).
 
+The workspace root also links `.agents/skills` to the shared package so coding
+agents launched from the root can discover the same workspace-wide skills.
+
 ## Getting started
 
 ```bash
@@ -49,6 +52,10 @@ watches `apps/`, so newly-created apps are detected without restarting
 links should stay relative, such as `/starter` or `/<app-id>`; do not hardcode
 localhost or dev ports because the active gateway origin owns the port.
 
+Dispatch vault keys are workspace-wide by default: every saved vault key is
+available to every workspace app and can be synced from Dispatch. Switch the
+Vault page to manual access only when you need explicit per-app key grants.
+
 ## Workspace org identity
 
 Set these root `.env` values before production deploys or when repairing