dsh-agora 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +88 -0
- package/assets/agora/SKILL.md +113 -0
- package/assets/agora/references/cli/README.md +161 -0
- package/assets/agora/references/cli/automation.md +189 -0
- package/assets/agora/references/cli/doctor.md +129 -0
- package/assets/agora/references/cli/env.md +158 -0
- package/assets/agora/references/cli/install-auth.md +152 -0
- package/assets/agora/references/cli/projects.md +116 -0
- package/assets/agora/references/cli/quickstarts.md +117 -0
- package/assets/agora/references/cloud-recording/README.md +86 -0
- package/assets/agora/references/conversational-ai/README.md +285 -0
- package/assets/agora/references/conversational-ai/agent-client-toolkit-react.md +182 -0
- package/assets/agora/references/conversational-ai/agent-samples.md +101 -0
- package/assets/agora/references/conversational-ai/agent-toolkit-android.md +209 -0
- package/assets/agora/references/conversational-ai/agent-toolkit-ios.md +208 -0
- package/assets/agora/references/conversational-ai/agent-toolkit.md +201 -0
- package/assets/agora/references/conversational-ai/agent-ui-kit.md +63 -0
- package/assets/agora/references/conversational-ai/architecture.md +221 -0
- package/assets/agora/references/conversational-ai/auth-flow.md +154 -0
- package/assets/agora/references/conversational-ai/conversational-ai-studio.md +173 -0
- package/assets/agora/references/conversational-ai/go-sdk.md +184 -0
- package/assets/agora/references/conversational-ai/integration-from-quickstart.md +203 -0
- package/assets/agora/references/conversational-ai/python-sdk.md +122 -0
- package/assets/agora/references/conversational-ai/quickstarts.md +710 -0
- package/assets/agora/references/conversational-ai/server-custom-llm.md +45 -0
- package/assets/agora/references/conversational-ai/server-mcp.md +40 -0
- package/assets/agora/references/conversational-ai/server-sdk-rename.md +78 -0
- package/assets/agora/references/conversational-ai/server-sdks.md +128 -0
- package/assets/agora/references/doc-fetching.md +67 -0
- package/assets/agora/references/integration-patterns.md +201 -0
- package/assets/agora/references/mcp-tools.md +49 -0
- package/assets/agora/references/rtc/README.md +104 -0
- package/assets/agora/references/rtc/android.md +344 -0
- package/assets/agora/references/rtc/cross-platform-coordination.md +61 -0
- package/assets/agora/references/rtc/flutter.md +282 -0
- package/assets/agora/references/rtc/ios.md +306 -0
- package/assets/agora/references/rtc/nextjs.md +87 -0
- package/assets/agora/references/rtc/react-native.md +266 -0
- package/assets/agora/references/rtc/react.md +186 -0
- package/assets/agora/references/rtc/web.md +506 -0
- package/assets/agora/references/rtm/README.md +80 -0
- package/assets/agora/references/rtm/android.md +277 -0
- package/assets/agora/references/rtm/ios.md +231 -0
- package/assets/agora/references/rtm/web.md +348 -0
- package/assets/agora/references/server/README.md +22 -0
- package/assets/agora/references/server/tokens.md +74 -0
- package/assets/agora/references/server-gateway/README.md +80 -0
- package/assets/agora/references/server-gateway/linux-cpp.md +251 -0
- package/assets/agora/references/testing-guidance/SKILL.md +65 -0
- package/assets/agora/references/testing-guidance/completeness-gate.md +28 -0
- package/assets/agora/references/testing-guidance/convoai-rest.md +83 -0
- package/assets/agora/references/testing-guidance/mobile-rtm-and-renewal.md +109 -0
- package/assets/agora/references/testing-guidance/rtc-android.md +70 -0
- package/assets/agora/references/testing-guidance/rtc-ios.md +73 -0
- package/assets/agora/references/testing-guidance/rtc-react.md +51 -0
- package/assets/agora/references/testing-guidance/rtc-web.md +94 -0
- package/cordis.patch.yml +5 -0
- package/index.js +53 -0
- package/package.json +49 -0
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Custom LLM Server
|
|
2
|
+
|
|
3
|
+
OpenAI-compatible LLM proxy for Agora Conversational AI. Intercepts LLM requests for RAG, tool calling, and conversation memory.
|
|
4
|
+
|
|
5
|
+
**Repo:** <https://github.com/AgoraIO-Conversational-AI/server-custom-llm>
|
|
6
|
+
|
|
7
|
+
## Implementations
|
|
8
|
+
|
|
9
|
+
| Language | Framework | Port | Notes |
|
|
10
|
+
|----------|-----------|------|-------|
|
|
11
|
+
| Python | FastAPI | 8100 | |
|
|
12
|
+
| Node.js | Express | 8101 | Advanced — exclusive features below |
|
|
13
|
+
| Go | Gin | 8102 | |
|
|
14
|
+
|
|
15
|
+
## Endpoints
|
|
16
|
+
|
|
17
|
+
- `/chat/completions` — LLM proxy with server-side tool execution (up to 5 passes)
|
|
18
|
+
- `/rag/chat/completions` — RAG-enhanced with retrieval step
|
|
19
|
+
- `/audio/chat/completions` — Multimodal audio responses
|
|
20
|
+
|
|
21
|
+
> **[README — Endpoints](https://github.com/AgoraIO-Conversational-AI/server-custom-llm#endpoints)**
|
|
22
|
+
|
|
23
|
+
## Integration with ConvoAI
|
|
24
|
+
|
|
25
|
+
Set `LLM_URL` to your server endpoint, `LLM_VENDOR=custom` in agent-samples `.env`.
|
|
26
|
+
|
|
27
|
+
> **[README — Integration](https://github.com/AgoraIO-Conversational-AI/server-custom-llm#integration-with-agora-conversational-ai)**
|
|
28
|
+
|
|
29
|
+
## Features
|
|
30
|
+
|
|
31
|
+
- Server-side tool execution (get_weather, calculate examples)
|
|
32
|
+
- Conversation memory (per appId:userId:channel, 100 message limit, 24h cleanup)
|
|
33
|
+
- RTM integration for text messaging (Node.js)
|
|
34
|
+
- Streaming SSE responses
|
|
35
|
+
|
|
36
|
+
> **[README — Features](https://github.com/AgoraIO-Conversational-AI/server-custom-llm#features)**
|
|
37
|
+
|
|
38
|
+
## Node.js-Exclusive Features
|
|
39
|
+
|
|
40
|
+
- `/register-agent` and `/unregister-agent` endpoints — agent lifecycle hooks
|
|
41
|
+
- Pluggable module system: `init()`, `getToolDefinitions()`, `onRequest`/`onResponse` hooks
|
|
42
|
+
- Dynamic RTM initialization via request headers (`X-Agora-Customllm-*`)
|
|
43
|
+
- Go audio subscriber for RTC audio capture (spawned as child process)
|
|
44
|
+
|
|
45
|
+
> **[node/integrations/README.md](https://github.com/AgoraIO-Conversational-AI/server-custom-llm/blob/main/node/integrations/README.md)** — Module system and integrations
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# MCP Memory Server
|
|
2
|
+
|
|
3
|
+
Multi-user MCP server with persistent memory and full-text search for Agora Conversational AI agents.
|
|
4
|
+
|
|
5
|
+
**Repo:** <https://github.com/AgoraIO-Conversational-AI/server-mcp-memory>
|
|
6
|
+
|
|
7
|
+
## Implementations
|
|
8
|
+
|
|
9
|
+
All 3 implementations are production-ready with feature parity. Test suite: `test/run_all.sh` (comprehensive happy + failure path tests).
|
|
10
|
+
|
|
11
|
+
| Language | Framework | Port | Notes |
|
|
12
|
+
|----------|-----------|------|-------|
|
|
13
|
+
| Python | Starlette | 8090 | |
|
|
14
|
+
| Node.js | Express | 8091 | |
|
|
15
|
+
| Go | Gin | 8092 | Requires `CGO_ENABLED=1` build flag (SQLite) |
|
|
16
|
+
|
|
17
|
+
## MCP Tools
|
|
18
|
+
|
|
19
|
+
- `save_memory` — store a memory with category and tags
|
|
20
|
+
- `search_memory` — BM25 full-text search
|
|
21
|
+
- `list_memories` — list by category
|
|
22
|
+
- `delete_memory` — delete by ID
|
|
23
|
+
- `compact_memories` — merge related memories
|
|
24
|
+
- `log_message` — append to conversation log
|
|
25
|
+
|
|
26
|
+
> **[README — Tools](https://github.com/AgoraIO-Conversational-AI/server-mcp-memory#mcp-tools)**
|
|
27
|
+
|
|
28
|
+
## Integration with ConvoAI
|
|
29
|
+
|
|
30
|
+
Configure `MCP_SERVERS` JSON array in agent-samples `.env`. Uses `build_mcp_servers` function.
|
|
31
|
+
|
|
32
|
+
> **[README — Integration](https://github.com/AgoraIO-Conversational-AI/server-mcp-memory#integration-with-agora-conversational-ai)**
|
|
33
|
+
|
|
34
|
+
## Architecture
|
|
35
|
+
|
|
36
|
+
- SQLite with FTS5 full-text search
|
|
37
|
+
- Per-user memory isolation via URL path (`/mcp/{user_id}`)
|
|
38
|
+
- MCP Streamable HTTP protocol
|
|
39
|
+
|
|
40
|
+
> **[README — Architecture](https://github.com/AgoraIO-Conversational-AI/server-mcp-memory#architecture)**
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agora-server-sdk-rename
|
|
3
|
+
description: |
|
|
4
|
+
Opt-in only. Load when the user's project uses outdated ConvoAI server SDK package or
|
|
5
|
+
module names and they need to migrate to current names. Do not load for greenfield work.
|
|
6
|
+
Triggers on: agora-agent-server-sdk, agora-agent (PyPI), agora-agent-server-sdk-go,
|
|
7
|
+
AgoraIO-Community/agora-agent-server-sdk-go, migrate agora sdk, upgrade server sdk package.
|
|
8
|
+
license: MIT
|
|
9
|
+
metadata:
|
|
10
|
+
author: agora
|
|
11
|
+
version: '1.0.0'
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# ConvoAI Server SDK — package rename migration
|
|
15
|
+
|
|
16
|
+
> **Opt-in only.** Load this file when workspace detection or the user shows outdated server SDK names. Default ConvoAI routing uses [server-sdks.md](server-sdks.md), [python-sdk.md](python-sdk.md), and [go-sdk.md](go-sdk.md) with current names.
|
|
17
|
+
|
|
18
|
+
Apply the renames below in dependency manifests and import statements. The public API is unchanged — only install paths and import strings move.
|
|
19
|
+
|
|
20
|
+
## Detection signals
|
|
21
|
+
|
|
22
|
+
Scan the user's project (read-only) for any of:
|
|
23
|
+
|
|
24
|
+
| Signal | Where to look |
|
|
25
|
+
|--------|----------------|
|
|
26
|
+
| `agora-agent-server-sdk` | `package.json`, lockfiles, `import` / `require` strings |
|
|
27
|
+
| `agora-agent` (not `agora-agents`) | `pyproject.toml`, `requirements.txt`, `Pipfile`, `poetry.lock` |
|
|
28
|
+
| `github.com/AgoraIO-Community/agora-agent-server-sdk-go` | `go.mod`, Go import paths |
|
|
29
|
+
| `agent-server-sdk-go` repo URL | README, CI config, docs |
|
|
30
|
+
|
|
31
|
+
If none match, do not load this file.
|
|
32
|
+
|
|
33
|
+
## Rename map
|
|
34
|
+
|
|
35
|
+
### TypeScript / npm
|
|
36
|
+
|
|
37
|
+
| Update | From | To |
|
|
38
|
+
|--------|------|-----|
|
|
39
|
+
| Dependency | `agora-agent-server-sdk` | `agora-agents` |
|
|
40
|
+
| Import | `from 'agora-agent-server-sdk'` | `from 'agora-agents'` |
|
|
41
|
+
|
|
42
|
+
The npm package `agora-agent-server-sdk` still resolves via a compat shim, but new projects and migrations should use `agora-agents`.
|
|
43
|
+
|
|
44
|
+
### Python / PyPI
|
|
45
|
+
|
|
46
|
+
| Update | From | To |
|
|
47
|
+
|--------|------|-----|
|
|
48
|
+
| Dependency | `agora-agent` or `agora-agent-server-sdk` | `agora-agents` |
|
|
49
|
+
| Import | `from agora_agent import ...` | **No change** |
|
|
50
|
+
|
|
51
|
+
Only the PyPI install name changes. Python import paths stay `agora_agent`.
|
|
52
|
+
|
|
53
|
+
### Go
|
|
54
|
+
|
|
55
|
+
| Update | From | To |
|
|
56
|
+
|--------|------|-----|
|
|
57
|
+
| Module | `github.com/AgoraIO-Community/agora-agent-server-sdk-go` | `github.com/AgoraIO/agora-agents-go` |
|
|
58
|
+
| Imports | `.../agora-agent-server-sdk-go/...` | `github.com/AgoraIO/agora-agents-go/...` |
|
|
59
|
+
| Repo | `AgoraIO-Conversational-AI/agent-server-sdk-go` | `AgoraIO/agora-agents-go` |
|
|
60
|
+
|
|
61
|
+
Go has no module alias — update `go.mod` and all import paths together, then `go mod tidy`.
|
|
62
|
+
|
|
63
|
+
## Migration steps
|
|
64
|
+
|
|
65
|
+
1. Update the manifest (`package.json`, `pyproject.toml` / `requirements.txt`, or `go.mod`).
|
|
66
|
+
2. Replace import strings in source files per the table above.
|
|
67
|
+
3. Reinstall / tidy (`npm install`, `pip install -r ...`, or `go mod tidy`).
|
|
68
|
+
4. Run the project's test suite or a minimal start/stop agent smoke test.
|
|
69
|
+
|
|
70
|
+
Do not change application logic, env var names, or AgentKit builder code — the API surface is the same.
|
|
71
|
+
|
|
72
|
+
## After migration
|
|
73
|
+
|
|
74
|
+
Route back to the current SDK references for ongoing work:
|
|
75
|
+
|
|
76
|
+
- TypeScript: [server-sdks.md](server-sdks.md)
|
|
77
|
+
- Python: [python-sdk.md](python-sdk.md)
|
|
78
|
+
- Go: [go-sdk.md](go-sdk.md)
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agora-convoai-server-sdks
|
|
3
|
+
description: |
|
|
4
|
+
Server-side SDKs for Agora Conversational AI: TypeScript, Python, and Go wrappers around the
|
|
5
|
+
ConvoAI REST API. Use when the user is building a backend to start/stop/manage ConvoAI agents.
|
|
6
|
+
Triggers on: agora-agents, AgoraClient, AgentSession, session.start, session.stop,
|
|
7
|
+
agent server SDK, ConvoAI backend, ConvoAI server, withStt, withLlm, withTts.
|
|
8
|
+
license: MIT
|
|
9
|
+
metadata:
|
|
10
|
+
author: agora
|
|
11
|
+
version: '1.0.0'
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# ConvoAI Server SDKs
|
|
15
|
+
|
|
16
|
+
> **Architecture first:** For call sequence and lifecycle overview before SDK details, read [architecture.md](architecture.md).
|
|
17
|
+
|
|
18
|
+
TypeScript, Go, and Python SDKs — convenience wrappers around the ConvoAI REST API. For any other backend language, call the REST API directly. Fetch the live OpenAPI spec for the full schema: `https://docs-md.agora.io/api/conversational-ai-api-v2.x.yaml`
|
|
19
|
+
|
|
20
|
+
## TypeScript — `agora-agents`
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
npm install agora-agents
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Builder pattern — configure the AI pipeline then create sessions:
|
|
27
|
+
|
|
28
|
+
```typescript
|
|
29
|
+
import { AgoraClient, Agent, Area } from 'agora-agents';
|
|
30
|
+
|
|
31
|
+
const client = new AgoraClient({
|
|
32
|
+
area: Area.US,
|
|
33
|
+
appId: process.env.AGORA_APP_ID,
|
|
34
|
+
appCertificate: process.env.AGORA_APP_CERTIFICATE,
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
const agent = new Agent({
|
|
38
|
+
name: `agent_${crypto.randomUUID().slice(0, 8)}`, // must be unique per project
|
|
39
|
+
instructions: 'You are a helpful voice assistant.',
|
|
40
|
+
greeting: 'Hello! How can I help you today?',
|
|
41
|
+
})
|
|
42
|
+
.withStt(new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY }))
|
|
43
|
+
.withLlm(new OpenAI({ apiKey: process.env.OPENAI_API_KEY }))
|
|
44
|
+
.withTts(new ElevenLabsTTS({ apiKey: process.env.ELEVENLABS_API_KEY }));
|
|
45
|
+
|
|
46
|
+
// Start a session (joins the agent to a channel)
|
|
47
|
+
const session = agent.createSession({ channel: 'my-channel', agentUid: 0 });
|
|
48
|
+
const sessionId = await session.start();
|
|
49
|
+
|
|
50
|
+
// Stop from the same process
|
|
51
|
+
await session.stop();
|
|
52
|
+
|
|
53
|
+
// Stop from a stateless server (e.g. a different request handler)
|
|
54
|
+
await client.stopAgent(sessionId);
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Token auth is handled automatically when `appCertificate` is provided. For vendor-specific STT/LLM/TTS import paths and MLLM (OpenAI Realtime, Gemini Live) config, see the [SDK README](https://github.com/AgoraIO-Conversational-AI/agent-server-sdk-ts).
|
|
58
|
+
|
|
59
|
+
## Auth Modes
|
|
60
|
+
|
|
61
|
+
Three modes, in order of recommendation:
|
|
62
|
+
|
|
63
|
+
- **App Credentials** (`appId` + `appCertificate`): SDK generates a fresh ConvoAI token per REST call. No token management needed. The App Certificate never leaves your server. Recommended for production.
|
|
64
|
+
- **Token Auth** (`authToken`): A pre-built combined RTC+RTM token you supply. This token is **reused for every request** until you replace the SDK instance — the SDK does not refresh it. You are responsible for refreshing before expiry. Max token validity is 24 hours.
|
|
65
|
+
- **Basic Auth** (`customerId` + `customerSecret`): Credentials never expire but are long-lived secrets that grant access to every project on your account. Use for local testing only; do not ship to production.
|
|
66
|
+
|
|
67
|
+
## Session State Machine
|
|
68
|
+
|
|
69
|
+
Sessions follow a strict state sequence. Calling methods outside the valid states throws an error:
|
|
70
|
+
|
|
71
|
+
| State | `start()` | `stop()` | `say()` / `interrupt()` / `update()` |
|
|
72
|
+
|-------|-----------|----------|---------------------------------------|
|
|
73
|
+
| `idle` | ✅ | ❌ | ❌ |
|
|
74
|
+
| `starting` | ❌ | ❌ | ❌ |
|
|
75
|
+
| `running` | ❌ | ✅ | ✅ |
|
|
76
|
+
| `stopping` | ❌ | ❌ | ❌ |
|
|
77
|
+
| `stopped` | ✅ | ❌ | ❌ |
|
|
78
|
+
| `error` | ✅ | ❌ | ❌ |
|
|
79
|
+
|
|
80
|
+
`stop()` on a 404 (agent already stopped on the platform) resolves without throwing — the SDK treats it as already stopped.
|
|
81
|
+
|
|
82
|
+
## In-Process Events (TypeScript)
|
|
83
|
+
|
|
84
|
+
`AgentSession` emits events within your Node.js process — these are not HTTP webhooks:
|
|
85
|
+
|
|
86
|
+
```typescript
|
|
87
|
+
session.on('started', () => {
|
|
88
|
+
// Agent has connected to the RTC channel and is ready
|
|
89
|
+
// session.id is now set; session.status === 'running'
|
|
90
|
+
});
|
|
91
|
+
|
|
92
|
+
session.on('stopped', () => {
|
|
93
|
+
// Agent has left the channel
|
|
94
|
+
});
|
|
95
|
+
|
|
96
|
+
session.on('error', (err) => {
|
|
97
|
+
// Non-recoverable error; session.status === 'error'
|
|
98
|
+
// Safe to call session.start() again from this state
|
|
99
|
+
});
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
## Avatar + TTS Sample Rate
|
|
103
|
+
|
|
104
|
+
HeyGen and Akool avatars require a specific TTS sample rate. The SDK validates this at `session.start()` and throws if mismatched. The error message identifies the avatar config as the problem — the root cause is actually the TTS sample rate.
|
|
105
|
+
|
|
106
|
+
| Avatar vendor | Required TTS sample rate |
|
|
107
|
+
|--------------|--------------------------|
|
|
108
|
+
| HeyGen | **24000 Hz** |
|
|
109
|
+
| Akool | **16000 Hz** |
|
|
110
|
+
|
|
111
|
+
Whenever an avatar vendor is set, explicitly configure the TTS sample rate to match. Do not rely on defaults.
|
|
112
|
+
|
|
113
|
+
```typescript
|
|
114
|
+
// HeyGen — must pair with 24 kHz TTS
|
|
115
|
+
const agent = new Agent({ ... })
|
|
116
|
+
.withAvatar(new HeyGen({ ... }))
|
|
117
|
+
.withTts(new ElevenLabsTTS({ sampleRate: 24000, ... }));
|
|
118
|
+
|
|
119
|
+
// Akool — must pair with 16 kHz TTS
|
|
120
|
+
const agent = new Agent({ ... })
|
|
121
|
+
.withAvatar(new Akool({ ... }))
|
|
122
|
+
.withTts(new ElevenLabsTTS({ sampleRate: 16000, ... }));
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
## Python and Go SDKs
|
|
126
|
+
|
|
127
|
+
- **[python-sdk.md](python-sdk.md)** — Python SDK: sync vs async, deprecation warnings, debug logging.
|
|
128
|
+
- **[go-sdk.md](go-sdk.md)** — Go SDK: context.Context pattern, builder syntax, SessionStatus constants, token helpers.
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Documentation Lookup
|
|
2
|
+
|
|
3
|
+
**Level 1 — Bundled references (always try first)**
|
|
4
|
+
|
|
5
|
+
Check the relevant file under `skills/agora/references/`. These are inline-stable:
|
|
6
|
+
RTC init patterns, RTM messaging, token generation, ConvoAI gotchas and generation
|
|
7
|
+
rules. If the answer is here, stop — no fetch needed.
|
|
8
|
+
|
|
9
|
+
**Level 2 — Live docs (when Level 1 is insufficient)**
|
|
10
|
+
|
|
11
|
+
When bundled references don't cover the detail needed (full request/response schemas,
|
|
12
|
+
vendor-specific configs, language-specific quick-start code):
|
|
13
|
+
|
|
14
|
+
If the Agora Docs MCP tool (`agora-docs-mcp`) is available in the current tool/runtime, prefer it for Level 2 documentation lookup. Otherwise use the HTTP fetch flow below. If MCP returns no useful result, fall back to HTTP fetch.
|
|
15
|
+
|
|
16
|
+
1. Fetch the Agora docs sitemap:
|
|
17
|
+
```
|
|
18
|
+
GET https://docs.agora.io/en/llms.txt
|
|
19
|
+
```
|
|
20
|
+
2. Scan the response for a URL matching the product and topic.
|
|
21
|
+
3. Fetch that URL and use its content to answer.
|
|
22
|
+
|
|
23
|
+
## Fallback
|
|
24
|
+
|
|
25
|
+
If `llms.txt` is unreachable or the fetched URL returns no useful content, try these
|
|
26
|
+
known markdown entry points directly:
|
|
27
|
+
|
|
28
|
+
| Product / Language | Markdown URL |
|
|
29
|
+
|---|---|
|
|
30
|
+
| RTC (Web/general) | https://docs-md.agora.io/en/video-calling/get-started/get-started-sdk.md |
|
|
31
|
+
| RTC (voice-only) | https://docs-md.agora.io/en/voice-calling/get-started/get-started-sdk.md |
|
|
32
|
+
| RTM (Web/general) | https://docs-md.agora.io/en/signaling/get-started/sdk-quickstart.md |
|
|
33
|
+
| RTM (iOS) | https://docs-md.agora.io/en/signaling/get-started/sdk-quickstart?platform=ios.md |
|
|
34
|
+
| RTM (Android) | https://docs-md.agora.io/en/signaling/get-started/sdk-quickstart?platform=android.md |
|
|
35
|
+
| ConvoAI | https://docs-md.agora.io/en/conversational-ai/get-started/quickstart.md |
|
|
36
|
+
| ConvoAI (TypeScript SDK) | https://docs-md.agora.io/en/conversational-ai/develop/integrate-sdk.md |
|
|
37
|
+
| ConvoAI (Python SDK) | https://docs-md.agora.io/en/conversational-ai/develop/integrate-sdk?platform=python.md |
|
|
38
|
+
| Cloud Recording | https://docs-md.agora.io/en/cloud-recording/get-started/getstarted.md |
|
|
39
|
+
| Server Gateway (C++) | https://docs-md.agora.io/en/server-gateway/get-started/integrate-sdk.md |
|
|
40
|
+
| Server Gateway (Java) | https://docs-md.agora.io/en/server-gateway/get-started/integrate-sdk?platform=java.md |
|
|
41
|
+
| Server Gateway (Python) | https://docs-md.agora.io/en/server-gateway/get-started/integrate-sdk?platform=python.md |
|
|
42
|
+
| Server Gateway (Go) | https://docs-md.agora.io/en/server-gateway/get-started/integrate-sdk?platform=go.md |
|
|
43
|
+
| Tokens | https://docs-md.agora.io/en/video-calling/token-authentication/deploy-token-server.md |
|
|
44
|
+
|
|
45
|
+
## Agora Docs MCP Server (optional)
|
|
46
|
+
|
|
47
|
+
Agora also provides `agora-docs-mcp`, an MCP server that gives AI assistants
|
|
48
|
+
direct tool-call access to documentation — an alternative to the Level 2 HTTP
|
|
49
|
+
fetch above. It traverses docs only; it does not use Agora backend APIs, log in,
|
|
50
|
+
create projects, or inspect account state. If a user asks about installing or
|
|
51
|
+
using the Agora Docs MCP server, see [mcp-tools.md](mcp-tools.md).
|
|
52
|
+
|
|
53
|
+
## Agora CLI
|
|
54
|
+
|
|
55
|
+
For local `agora` command-line usage, check [cli/README.md](cli/README.md) first. Treat that file as the source of truth for the currently verified CLI baseline and command surface.
|
|
56
|
+
|
|
57
|
+
If the bundled CLI references do not cover a CLI-only detail, use the canonical CLI repository instead of general Agora product docs:
|
|
58
|
+
|
|
59
|
+
- <https://github.com/AgoraIO/cli>
|
|
60
|
+
- `README.md` — install, first run, command model, common workflows
|
|
61
|
+
- `docs/commands.md` — generated command reference
|
|
62
|
+
- `docs/automation.md` — JSON envelopes and automation contract
|
|
63
|
+
- `docs/error-codes.md` — stable error codes and recovery decisions
|
|
64
|
+
- `docs/telemetry.md` — telemetry preferences and environment controls
|
|
65
|
+
- `CHANGELOG.md` and GitHub Releases — version-specific changes
|
|
66
|
+
|
|
67
|
+
Do not invent CLI commands, flags, environment variables, or JSON fields. If the installed CLI exposes a command that is not in the bundled references, prefer `agora introspect --json` or `agora --help --all`, then cross-check the CLI repository docs before answering.
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
# Multi-Product Integration Patterns
|
|
2
|
+
|
|
3
|
+
This file covers how RTC, RTM, and Conversational AI work together. For SDK-specific code, follow the Level 2 fetch links in each product's reference file.
|
|
4
|
+
|
|
5
|
+
## When to Use Multiple Products
|
|
6
|
+
|
|
7
|
+
| Scenario | Products |
|
|
8
|
+
|---|---|
|
|
9
|
+
| Video/voice call only | RTC only |
|
|
10
|
+
| Video call + chat or presence | RTC + RTM |
|
|
11
|
+
| AI agent in a call | RTC + ConvoAI REST API (server-side) |
|
|
12
|
+
| AI agent + live transcripts or state | RTC + RTM + ConvoAI REST API |
|
|
13
|
+
| Text-only signaling (no media) | RTM only |
|
|
14
|
+
|
|
15
|
+
## RTC + RTM Together
|
|
16
|
+
|
|
17
|
+
### Initialization Order
|
|
18
|
+
|
|
19
|
+
Always initialize both SDKs independently — they share no initialization path.
|
|
20
|
+
|
|
21
|
+
RTC UID and RTM user ID do **not** need to match. Using a shared identity is a coordination convention, not a platform requirement. The right initialization order depends on which identity strategy your app uses.
|
|
22
|
+
|
|
23
|
+
#### Default pattern — auto-assigned RTC UID
|
|
24
|
+
|
|
25
|
+
Use this when the RTC client joins with `uid: 0` / `null` and the SDK assigns the final RTC UID:
|
|
26
|
+
|
|
27
|
+
1. Create and initialize the RTC engine/client
|
|
28
|
+
2. Join the RTC channel
|
|
29
|
+
3. Read the assigned RTC UID from the join callback / resolved promise
|
|
30
|
+
4. Create or log in to the RTM client with `String(rtcUid)`
|
|
31
|
+
5. Subscribe to the RTM channel
|
|
32
|
+
|
|
33
|
+
This is the safest default because it avoids guessing the RTM identity before the RTC UID is known.
|
|
34
|
+
|
|
35
|
+
#### Advanced pattern — app-level stable identity
|
|
36
|
+
|
|
37
|
+
Use this when your app already has a stable user identity independent of the RTC UID:
|
|
38
|
+
|
|
39
|
+
1. Create and initialize the RTC engine/client
|
|
40
|
+
2. Create and log in to the RTM client with your app-level user identity
|
|
41
|
+
3. Subscribe to the RTM channel
|
|
42
|
+
4. Join the RTC channel
|
|
43
|
+
|
|
44
|
+
This pattern is useful when you want signaling, presence, or pre-call state to arrive before media starts. It requires an explicit RTC↔RTM mapping in your app or backend.
|
|
45
|
+
|
|
46
|
+
### UID Strategy
|
|
47
|
+
|
|
48
|
+
RTC and RTM use different UID types:
|
|
49
|
+
|
|
50
|
+
| SDK | Platform | UID type |
|
|
51
|
+
|---|---|---|
|
|
52
|
+
| RTC | Web | `number` |
|
|
53
|
+
| RTC | iOS | `UInt` (unsigned 32-bit) |
|
|
54
|
+
| RTC | Android | `Int` (signed 32-bit) |
|
|
55
|
+
| RTM | All platforms | `String` |
|
|
56
|
+
|
|
57
|
+
Recommended default: use `String(rtcUid)` as the RTM `userId` when your RTC UID is auto-assigned and you want a simple 1:1 mapping between the two systems.
|
|
58
|
+
|
|
59
|
+
Alternative: use a stable app-level string identity for RTM and maintain an explicit mapping from RTC UID → RTM user ID in your app/backend.
|
|
60
|
+
|
|
61
|
+
The RTM token subject and RTM login `userId` must match exactly. If your server minted the RTM token for `String(rtcUid)`, the client must log in RTM with that same string — not a different random or app-generated identity.
|
|
62
|
+
|
|
63
|
+
UIDs greater than 2,147,483,647 wrap to negative on Android RTC — avoid them if Android clients are present.
|
|
64
|
+
|
|
65
|
+
### Channel Name Convention
|
|
66
|
+
|
|
67
|
+
Use the **same channel name** for both RTC and RTM. The RTM channel is the coordination layer for the RTC session — same name makes routing unambiguous and is required by the ConvoAI toolkit.
|
|
68
|
+
|
|
69
|
+
### Token Matrix
|
|
70
|
+
|
|
71
|
+
Both SDKs require separate tokens in production. Generate them independently on your server:
|
|
72
|
+
|
|
73
|
+
| Token | Scope | Renew on |
|
|
74
|
+
|---|---|---|
|
|
75
|
+
| RTC token | `appId` + `channelName` + `uid` | `token-privilege-will-expire` / `onTokenPrivilegeWillExpire` |
|
|
76
|
+
| RTM token | `appId` + `userId` | RTM connection state `TOKEN_EXPIRED` or equivalent |
|
|
77
|
+
|
|
78
|
+
See [server/tokens.md](server/tokens.md) for token generation details.
|
|
79
|
+
|
|
80
|
+
For Level 2 fetch: fetch `https://docs.agora.io/en/llms.txt`, find the token management guide for your platform, then fetch it.
|
|
81
|
+
|
|
82
|
+
## RTC + RTM + ConvoAI
|
|
83
|
+
|
|
84
|
+
Before wiring ConvoAI into an existing multi-product app, read [conversational-ai/architecture.md](conversational-ai/architecture.md) for the end-to-end call sequence, then use the official quickstart as the source of truth for code shape and commands. For existing-app work, use [conversational-ai/integration-from-quickstart.md](conversational-ai/integration-from-quickstart.md) after the quickstart source is inspected; runtime proof is required before claiming the integrated flow works.
|
|
85
|
+
|
|
86
|
+
### How the Products Connect
|
|
87
|
+
|
|
88
|
+
```
|
|
89
|
+
Client Server
|
|
90
|
+
──────────────────────────────────────────────────────
|
|
91
|
+
RTC Engine/Client ──── media ────► Agora RTC
|
|
92
|
+
RTM Client ────── messages ──────► Agora RTM
|
|
93
|
+
│
|
|
94
|
+
[ConvoAI agent joins
|
|
95
|
+
same RTC channel,
|
|
96
|
+
publishes transcripts
|
|
97
|
+
and state to same
|
|
98
|
+
RTM channel]
|
|
99
|
+
▲
|
|
100
|
+
App Server ── REST POST /join ──────► Agora ConvoAI API
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
### Initialization Order for ConvoAI
|
|
104
|
+
|
|
105
|
+
1. Initialize RTC (but do not join the channel yet)
|
|
106
|
+
2. Initialize RTM and log in
|
|
107
|
+
3. Subscribe to the RTM channel (to receive agent events)
|
|
108
|
+
4. Call ConvoAI REST `POST /join` from your **app server** with `channelName`, `uid` (your user's RTC uid), and the RTM flags (see below)
|
|
109
|
+
5. Join the RTC channel — the agent will already be there or will join shortly
|
|
110
|
+
|
|
111
|
+
If your client uses auto-assigned RTC UIDs, adapt the order slightly:
|
|
112
|
+
|
|
113
|
+
1. Initialize RTC
|
|
114
|
+
2. Join RTC and wait for the assigned user UID
|
|
115
|
+
3. Log in to RTM with `String(rtcUid)` and subscribe to the RTM channel
|
|
116
|
+
4. Call `POST /join` from your app server
|
|
117
|
+
|
|
118
|
+
The agent can publish transcripts/state through RTM only after the RTM client is logged in and subscribed.
|
|
119
|
+
|
|
120
|
+
If RTM was just enabled on the project, allow bounded wait/retry before treating an RTM startup failure as a persistent project misconfiguration. Control-plane state can update before the RTM runtime path is actually usable.
|
|
121
|
+
|
|
122
|
+
### Required ConvoAI Flags for RTM Delivery
|
|
123
|
+
|
|
124
|
+
When calling `POST /join`, include both of these in the request body:
|
|
125
|
+
|
|
126
|
+
```json
|
|
127
|
+
{
|
|
128
|
+
"advanced_features": {
|
|
129
|
+
"enable_rtm": true
|
|
130
|
+
},
|
|
131
|
+
"parameters": {
|
|
132
|
+
"data_channel": "rtm"
|
|
133
|
+
}
|
|
134
|
+
}
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Both flags are required. Omitting either one means the agent publishes transcripts via RTC data channel instead of RTM, and RTM message events will not fire.
|
|
138
|
+
|
|
139
|
+
### RTM Channel Name = RTC Channel Name
|
|
140
|
+
|
|
141
|
+
The ConvoAI agent joins the RTM channel with the **same name** as the RTC channel it joined. When handling `onMessageEvent` / `didReceiveMessageEvent`, filter by `channelName` matching your RTC channel if you are subscribed to multiple channels.
|
|
142
|
+
|
|
143
|
+
### Agent Events via RTM
|
|
144
|
+
|
|
145
|
+
The agent publishes JSON messages to the RTM channel. Parse `event.message.data` (Android) or `event.message.stringData` (iOS) as JSON. The `customType` field indicates the event type (transcript, interruption, state change, etc.).
|
|
146
|
+
|
|
147
|
+
For the full event schema and `customType` values, use Level 2 fetch: fetch `https://docs.agora.io/en/llms.txt`, locate the Conversational AI event reference, then fetch it.
|
|
148
|
+
|
|
149
|
+
### Token Matrix for ConvoAI
|
|
150
|
+
|
|
151
|
+
ConvoAI sessions require three separate tokens:
|
|
152
|
+
|
|
153
|
+
| Token | Purpose | Who generates |
|
|
154
|
+
|---|---|---|
|
|
155
|
+
| RTC token (user) | User joins RTC channel | Your app server |
|
|
156
|
+
| RTM token (user) | User logs into RTM | Your app server |
|
|
157
|
+
| RTC token (agent) | ConvoAI agent joins RTC | Your app server, passed in POST /join |
|
|
158
|
+
|
|
159
|
+
See [conversational-ai/auth-flow.md](conversational-ai/auth-flow.md) for the full three-token flow (REST API implementors) or [conversational-ai/README.md](conversational-ai/README.md) for SDK-based flows.
|
|
160
|
+
|
|
161
|
+
## Codec Selection for Mixed Platforms
|
|
162
|
+
|
|
163
|
+
When Web clients share a channel with iOS or Android native clients:
|
|
164
|
+
|
|
165
|
+
- Web defaults to `codec: 'vp8'` — this scales well for multi-user calls and is the recommended choice
|
|
166
|
+
- `'vp9'` is also a good option with better compression
|
|
167
|
+
- `'h264'` does not scale well beyond small groups — avoid it for multi-user scenarios
|
|
168
|
+
- If codecs differ between Web and native clients, Agora transcodes server-side — this works but adds latency and is billed separately
|
|
169
|
+
|
|
170
|
+
See [rtc/cross-platform-coordination.md](rtc/cross-platform-coordination.md) for full cross-platform interop notes.
|
|
171
|
+
|
|
172
|
+
## Cleanup Order
|
|
173
|
+
|
|
174
|
+
Reverse of initialization:
|
|
175
|
+
|
|
176
|
+
1. Leave the RTC channel
|
|
177
|
+
2. Unsubscribe from RTM channels
|
|
178
|
+
3. Log out of RTM
|
|
179
|
+
4. Release both engines/clients
|
|
180
|
+
|
|
181
|
+
For ConvoAI: call `DELETE /leave` (or the SDK equivalent) from your app server before leaving the RTC channel, to give the agent time to exit gracefully.
|
|
182
|
+
|
|
183
|
+
## Reference Files by Product
|
|
184
|
+
|
|
185
|
+
| Product | Platform | File |
|
|
186
|
+
|---|---|---|
|
|
187
|
+
| RTC | Web | [rtc/web.md](rtc/web.md) |
|
|
188
|
+
| RTC | React | [rtc/react.md](rtc/react.md) |
|
|
189
|
+
| RTC | Next.js | [rtc/nextjs.md](rtc/nextjs.md) |
|
|
190
|
+
| RTC | iOS | [rtc/ios.md](rtc/ios.md) |
|
|
191
|
+
| RTC | Android | [rtc/android.md](rtc/android.md) |
|
|
192
|
+
| RTC | React Native | [rtc/react-native.md](rtc/react-native.md) |
|
|
193
|
+
| RTC | Flutter | [rtc/flutter.md](rtc/flutter.md) |
|
|
194
|
+
| RTM | Web | [rtm/web.md](rtm/web.md) |
|
|
195
|
+
| RTM | iOS | [rtm/ios.md](rtm/ios.md) |
|
|
196
|
+
| RTM | Android | [rtm/android.md](rtm/android.md) |
|
|
197
|
+
| ConvoAI | All | [conversational-ai/README.md](conversational-ai/README.md) |
|
|
198
|
+
| ConvoAI toolkit | iOS | [conversational-ai/agent-toolkit-ios.md](conversational-ai/agent-toolkit-ios.md) |
|
|
199
|
+
| ConvoAI toolkit | Android | [conversational-ai/agent-toolkit-android.md](conversational-ai/agent-toolkit-android.md) |
|
|
200
|
+
| Tokens | Server | [server/tokens.md](server/tokens.md) |
|
|
201
|
+
| Cross-platform | All | [rtc/cross-platform-coordination.md](rtc/cross-platform-coordination.md) |
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Agora Docs MCP Server
|
|
2
|
+
|
|
3
|
+
<!-- applies-from: v0.2.0 -->
|
|
4
|
+
|
|
5
|
+
The Agora Docs MCP server (`agora-docs-mcp`) gives AI assistants direct tool-call access to Agora documentation. It is an
|
|
6
|
+
optional enhancement — the skill works without it using the two-tier fetch
|
|
7
|
+
approach in [doc-fetching.md](doc-fetching.md).
|
|
8
|
+
|
|
9
|
+
Agora Docs MCP is for traversing docs only. It does not operate Agora backends,
|
|
10
|
+
log in to accounts, create projects, write env files, or check project readiness;
|
|
11
|
+
the local `agora` CLI handles those workflows.
|
|
12
|
+
|
|
13
|
+
**Only use MCP when the user explicitly asks for it.** The default documentation
|
|
14
|
+
lookup is the two-tier fetch approach in [doc-fetching.md](doc-fetching.md) — use
|
|
15
|
+
that regardless of whether MCP is installed.
|
|
16
|
+
|
|
17
|
+
**MCP endpoint:** `https://mcp.agora.io`
|
|
18
|
+
|
|
19
|
+
## Tools
|
|
20
|
+
|
|
21
|
+
| Tool | Input | Returns |
|
|
22
|
+
|---|---|---|
|
|
23
|
+
| `get-doc-content` | `{"uri": "docs://..."}` | Full markdown content |
|
|
24
|
+
| `search-docs` | `{"query": "keyword"}` | List of matching doc URIs |
|
|
25
|
+
| `list-docs` | `{"category": "...", "limit": 20}` | All docs in a category |
|
|
26
|
+
|
|
27
|
+
Use `search-docs` when the topic is known but the URI isn't. Use `get-doc-content`
|
|
28
|
+
directly when the URI is known.
|
|
29
|
+
|
|
30
|
+
## Installation
|
|
31
|
+
|
|
32
|
+
**Claude Code:**
|
|
33
|
+
```bash
|
|
34
|
+
claude mcp add agora-docs-mcp --transport http https://mcp.agora.io
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
**Cursor / Windsurf / other MCP-compatible tools:** Add `https://mcp.agora.io` as
|
|
38
|
+
an HTTP MCP server in your tool's MCP settings. See your tool's documentation for
|
|
39
|
+
the exact configuration format.
|
|
40
|
+
|
|
41
|
+
For the latest setup instructions and any changes to the endpoint, see:
|
|
42
|
+
<https://docs.agora.io/en/mcp>
|
|
43
|
+
|
|
44
|
+
## Usage Note
|
|
45
|
+
|
|
46
|
+
After fetching quick-start docs via MCP, use the content for API structure and field
|
|
47
|
+
names only. Do NOT copy sample code verbatim — quick-start examples typically hardcode
|
|
48
|
+
credentials and omit production requirements. Apply the gotchas and generation rules
|
|
49
|
+
in `references/conversational-ai/README.md` to any generated code.
|