dsh-agora 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +88 -0
- package/assets/agora/SKILL.md +113 -0
- package/assets/agora/references/cli/README.md +161 -0
- package/assets/agora/references/cli/automation.md +189 -0
- package/assets/agora/references/cli/doctor.md +129 -0
- package/assets/agora/references/cli/env.md +158 -0
- package/assets/agora/references/cli/install-auth.md +152 -0
- package/assets/agora/references/cli/projects.md +116 -0
- package/assets/agora/references/cli/quickstarts.md +117 -0
- package/assets/agora/references/cloud-recording/README.md +86 -0
- package/assets/agora/references/conversational-ai/README.md +285 -0
- package/assets/agora/references/conversational-ai/agent-client-toolkit-react.md +182 -0
- package/assets/agora/references/conversational-ai/agent-samples.md +101 -0
- package/assets/agora/references/conversational-ai/agent-toolkit-android.md +209 -0
- package/assets/agora/references/conversational-ai/agent-toolkit-ios.md +208 -0
- package/assets/agora/references/conversational-ai/agent-toolkit.md +201 -0
- package/assets/agora/references/conversational-ai/agent-ui-kit.md +63 -0
- package/assets/agora/references/conversational-ai/architecture.md +221 -0
- package/assets/agora/references/conversational-ai/auth-flow.md +154 -0
- package/assets/agora/references/conversational-ai/conversational-ai-studio.md +173 -0
- package/assets/agora/references/conversational-ai/go-sdk.md +184 -0
- package/assets/agora/references/conversational-ai/integration-from-quickstart.md +203 -0
- package/assets/agora/references/conversational-ai/python-sdk.md +122 -0
- package/assets/agora/references/conversational-ai/quickstarts.md +710 -0
- package/assets/agora/references/conversational-ai/server-custom-llm.md +45 -0
- package/assets/agora/references/conversational-ai/server-mcp.md +40 -0
- package/assets/agora/references/conversational-ai/server-sdk-rename.md +78 -0
- package/assets/agora/references/conversational-ai/server-sdks.md +128 -0
- package/assets/agora/references/doc-fetching.md +67 -0
- package/assets/agora/references/integration-patterns.md +201 -0
- package/assets/agora/references/mcp-tools.md +49 -0
- package/assets/agora/references/rtc/README.md +104 -0
- package/assets/agora/references/rtc/android.md +344 -0
- package/assets/agora/references/rtc/cross-platform-coordination.md +61 -0
- package/assets/agora/references/rtc/flutter.md +282 -0
- package/assets/agora/references/rtc/ios.md +306 -0
- package/assets/agora/references/rtc/nextjs.md +87 -0
- package/assets/agora/references/rtc/react-native.md +266 -0
- package/assets/agora/references/rtc/react.md +186 -0
- package/assets/agora/references/rtc/web.md +506 -0
- package/assets/agora/references/rtm/README.md +80 -0
- package/assets/agora/references/rtm/android.md +277 -0
- package/assets/agora/references/rtm/ios.md +231 -0
- package/assets/agora/references/rtm/web.md +348 -0
- package/assets/agora/references/server/README.md +22 -0
- package/assets/agora/references/server/tokens.md +74 -0
- package/assets/agora/references/server-gateway/README.md +80 -0
- package/assets/agora/references/server-gateway/linux-cpp.md +251 -0
- package/assets/agora/references/testing-guidance/SKILL.md +65 -0
- package/assets/agora/references/testing-guidance/completeness-gate.md +28 -0
- package/assets/agora/references/testing-guidance/convoai-rest.md +83 -0
- package/assets/agora/references/testing-guidance/mobile-rtm-and-renewal.md +109 -0
- package/assets/agora/references/testing-guidance/rtc-android.md +70 -0
- package/assets/agora/references/testing-guidance/rtc-ios.md +73 -0
- package/assets/agora/references/testing-guidance/rtc-react.md +51 -0
- package/assets/agora/references/testing-guidance/rtc-web.md +94 -0
- package/cordis.patch.yml +5 -0
- package/index.js +53 -0
- package/package.json +49 -0
|
@@ -0,0 +1,221 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: conversational-ai-architecture
|
|
3
|
+
description: |
|
|
4
|
+
System architecture and call sequence for Agora Conversational AI agents. Use when the user
|
|
5
|
+
asks how ConvoAI works, what components are involved, initialization order, token flow,
|
|
6
|
+
agent lifecycle, start/stop sequence, or how server, client, RTC, RTM, and the ConvoAI
|
|
7
|
+
platform fit together.
|
|
8
|
+
license: MIT
|
|
9
|
+
metadata:
|
|
10
|
+
author: agora
|
|
11
|
+
version: '1.0.0'
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# ConvoAI Architecture and Call Sequence
|
|
15
|
+
|
|
16
|
+
Read this file first when the user asks how a ConvoAI voice agent is wired end to end, what calls happen in what order, or how to manage agent lifecycle across server and client.
|
|
17
|
+
|
|
18
|
+
For implementation details after the architecture is clear:
|
|
19
|
+
|
|
20
|
+
| Topic | File |
|
|
21
|
+
| ------------------------------------------ | -------------------------------------------------------- |
|
|
22
|
+
| Token types and auth headers | [auth-flow.md](auth-flow.md) |
|
|
23
|
+
| Server SDK session management | [server-sdks.md](server-sdks.md) |
|
|
24
|
+
| RTC + RTM + ConvoAI init order and cleanup | [../integration-patterns.md](../integration-patterns.md) |
|
|
25
|
+
| Client transcripts, state, sendText | [agent-toolkit.md](agent-toolkit.md) |
|
|
26
|
+
| REST endpoint schemas | [README.md](README.md) + OpenAPI spec |
|
|
27
|
+
| Official quickstart source code | [quickstarts.md](quickstarts.md) |
|
|
28
|
+
|
|
29
|
+
## System Components
|
|
30
|
+
|
|
31
|
+
```text
|
|
32
|
+
┌──────────────────────┐ start/stop session ┌─────────────────┐ REST /join,/leave,/update ┌──────────────────────┐
|
|
33
|
+
│ Browser/Mobile App │ ──────────────────────────► │ Your Server │ ─────────────────────────────► │ Agora ConvoAI Engine │
|
|
34
|
+
│ UI + app state │ ◄──── channel + tokens ──── │ (app backend) │ ◄────── agent_id,status ────── │ manages agent │
|
|
35
|
+
└──────────┬───────────┘ └─────────────────┘ └──────────┬───────────┘
|
|
36
|
+
│ uses RTC/RTM SDKs creates/updates/stops
|
|
37
|
+
▼ ▼
|
|
38
|
+
┌──────────────────────┐ ┌──────────────────────┐
|
|
39
|
+
│ Agora RTC + RTM SDKs │ │ ConvoAI Agent │
|
|
40
|
+
│ client-side runtime │ │ managed participant │
|
|
41
|
+
└──────────┬───────────┘ └──────────┬───────────┘
|
|
42
|
+
│ joins/publishes/subscribes │ joins/publishes/subscribes
|
|
43
|
+
▼ ▼
|
|
44
|
+
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
|
|
45
|
+
│ Agora Realtime Channels (SDRTN) │
|
|
46
|
+
│ RTC channel: client mic audio flows to the agent; agent TTS audio flows back to the client │
|
|
47
|
+
│ RTM channel: transcripts, agent state, metrics, errors, and control messages use the same channel name │
|
|
48
|
+
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Roles:
|
|
52
|
+
|
|
53
|
+
- **Your server** starts and stops agents, mints client tokens, and owns session state in your app.
|
|
54
|
+
- **ConvoAI Engine** manages the agent lifecycle and pipeline (speech in → ASR → LLM → TTS → speech out).
|
|
55
|
+
- **ConvoAI agent** is the managed realtime participant created by the engine. It joins the same RTC channel as the client and can publish RTM events to the matching RTM channel when enabled.
|
|
56
|
+
- **Client app** triggers the app backend to start/stop the agent. The app itself sits above the Agora RTC/RTM SDKs.
|
|
57
|
+
- **Agora RTC/RTM SDKs and the ConvoAI agent communicate through Agora realtime channels.** The channels are the lowest-level transport layer in this architecture.
|
|
58
|
+
|
|
59
|
+
The client toolkit does **not** start agents. The server does.
|
|
60
|
+
|
|
61
|
+
## End-to-End Start Sequence
|
|
62
|
+
|
|
63
|
+
Typical production flow when a user clicks "Start conversation":
|
|
64
|
+
|
|
65
|
+
```text
|
|
66
|
+
1. Client ──► Your server: "start session" (channel, user identity)
|
|
67
|
+
2. Your server ──► Token builder: mint RTC token + RTM token for the user
|
|
68
|
+
3. Your server ──► ConvoAI REST POST /join (or SDK session.start())
|
|
69
|
+
4. ConvoAI Engine ──► creates agent, agent joins RTC channel (async)
|
|
70
|
+
5. Your server ──► Client: { channel, rtcToken, rtmToken, agentId }
|
|
71
|
+
6. Client ──► RTM login + subscribe to channel
|
|
72
|
+
7. Client ──► RTC join(channel, rtcToken, uid)
|
|
73
|
+
8. Client waits for agent RTC user-joined / agent audio before treating session as live
|
|
74
|
+
9. User speaks ──► agent hears via RTC ──► agent responds via RTC
|
|
75
|
+
10. Transcripts/state arrive via RTM (if enabled) or RTC data channel
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Important timing rules:
|
|
79
|
+
|
|
80
|
+
- `POST /join` success means the request was accepted, not that the agent is already in the channel. Wait for RTC `user-joined` or SDK/client agent-state events before expecting audio.
|
|
81
|
+
- RTM channel name must match the RTC channel name passed in the join payload.
|
|
82
|
+
- RTM login identity must match the RTM token subject.
|
|
83
|
+
|
|
84
|
+
For auto-assigned RTC UIDs, join RTC first, read the assigned UID, then log into RTM with `String(rtcUid)`. See [../integration-patterns.md](../integration-patterns.md).
|
|
85
|
+
|
|
86
|
+
## Initialization Order
|
|
87
|
+
|
|
88
|
+
Default client-side order when the user's RTC UID is known up front:
|
|
89
|
+
|
|
90
|
+
1. Initialize RTC engine/client (do not join yet)
|
|
91
|
+
2. Initialize RTM and log in
|
|
92
|
+
3. Subscribe to the RTM channel
|
|
93
|
+
4. Call `POST /join` from your server
|
|
94
|
+
5. Join the RTC channel
|
|
95
|
+
|
|
96
|
+
Auto-assigned UID variant:
|
|
97
|
+
|
|
98
|
+
1. Initialize RTC
|
|
99
|
+
2. Join RTC and wait for assigned UID
|
|
100
|
+
3. Log into RTM with `String(rtcUid)` and subscribe
|
|
101
|
+
4. Call `POST /join` from your server
|
|
102
|
+
|
|
103
|
+
## Tokens
|
|
104
|
+
|
|
105
|
+
Three distinct tokens appear in most integrations:
|
|
106
|
+
|
|
107
|
+
| Token | Purpose | Used by |
|
|
108
|
+
| -------------------- | ----------------------------------- | ---------------------------------------------- |
|
|
109
|
+
| RTC client token | User joins RTC channel | Browser/mobile RTC SDK |
|
|
110
|
+
| RTM client token | User logs into RTM | Browser/mobile RTM SDK |
|
|
111
|
+
| ConvoAI server token | Authorize REST calls to ConvoAI API | Your server → `Authorization: agora token=...` |
|
|
112
|
+
|
|
113
|
+
The agent also needs an RTC token in the join payload (`properties.token`). Your server generates it and passes it to ConvoAI when starting the agent.
|
|
114
|
+
|
|
115
|
+
SDK users in App Credentials mode: pass `appId + appCertificate` to the server SDK and it generates the ConvoAI server token per request. You still mint the two client tokens yourself.
|
|
116
|
+
|
|
117
|
+
See [auth-flow.md](auth-flow.md) and [../server/tokens.md](../server/tokens.md).
|
|
118
|
+
|
|
119
|
+
## Agent Lifecycle
|
|
120
|
+
|
|
121
|
+
### Platform agent status (REST)
|
|
122
|
+
|
|
123
|
+
| Status | Code | Meaning |
|
|
124
|
+
| ---------- | ---- | -------------------------- |
|
|
125
|
+
| IDLE | 0 | Ready, not active |
|
|
126
|
+
| STARTING | 1 | Initialization in progress |
|
|
127
|
+
| RUNNING | 2 | Active, processing audio |
|
|
128
|
+
| STOPPING | 3 | Shutdown in progress |
|
|
129
|
+
| STOPPED | 4 | Exited channel |
|
|
130
|
+
| RECOVERING | 5 | Error recovery |
|
|
131
|
+
| FAILED | 6 | Execution failure |
|
|
132
|
+
|
|
133
|
+
### Server SDK session states
|
|
134
|
+
|
|
135
|
+
When using `agora-agents`, map platform lifecycle to SDK session states:
|
|
136
|
+
|
|
137
|
+
| SDK state | Typical action |
|
|
138
|
+
| ------------------- | -------------------------------------------- |
|
|
139
|
+
| `idle` | call `start()` |
|
|
140
|
+
| `starting` | wait |
|
|
141
|
+
| `running` | `stop()`, `say()`, `interrupt()`, `update()` |
|
|
142
|
+
| `stopping` | wait |
|
|
143
|
+
| `stopped` / `error` | may call `start()` again |
|
|
144
|
+
|
|
145
|
+
See [server-sdks.md](server-sdks.md).
|
|
146
|
+
|
|
147
|
+
## Managing Agents
|
|
148
|
+
|
|
149
|
+
Your server owns agent lifecycle. Common operations:
|
|
150
|
+
|
|
151
|
+
| Operation | REST | SDK equivalent |
|
|
152
|
+
| ------------------- | ----------------------------- | ------------------------------------------ |
|
|
153
|
+
| Start agent | `POST /join` | `session.start()` |
|
|
154
|
+
| Stop agent | `POST /agents/{id}/leave` | `session.stop()` or `client.stopAgent(id)` |
|
|
155
|
+
| Update config/token | `POST /agents/{id}/update` | `session.update()` |
|
|
156
|
+
| Query status | `GET /agents/{id}` | platform query |
|
|
157
|
+
| List agents | `GET /agents` | platform query |
|
|
158
|
+
| Broadcast speech | `POST /agents/{id}/speak` | `session.say()` |
|
|
159
|
+
| Interrupt speech | `POST /agents/{id}/interrupt` | `session.interrupt()` |
|
|
160
|
+
|
|
161
|
+
Production patterns:
|
|
162
|
+
|
|
163
|
+
- **Stateful server**: hold `AgentSession` in memory; use in-process SDK events.
|
|
164
|
+
- **Stateless server**: store `agent_id` in your DB; use platform webhooks or poll `GET /agents/{id}` for state changes.
|
|
165
|
+
- **Client UI**: subscribe to RTM agent events for transcripts and live state in the browser.
|
|
166
|
+
|
|
167
|
+
Fetch full request/response schemas from the OpenAPI spec — do not invent fields:
|
|
168
|
+
|
|
169
|
+
```text
|
|
170
|
+
GET https://docs-md.agora.io/api/conversational-ai-api-v2.x.yaml
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
## RTM Event Delivery
|
|
174
|
+
|
|
175
|
+
To receive transcripts and agent state in the client via RTM, both join flags are required:
|
|
176
|
+
|
|
177
|
+
```json
|
|
178
|
+
{
|
|
179
|
+
"advanced_features": { "enable_rtm": true },
|
|
180
|
+
"parameters": { "data_channel": "rtm" }
|
|
181
|
+
}
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Without both, events may arrive via RTC data channel instead, and RTM toolkit handlers will not fire.
|
|
185
|
+
|
|
186
|
+
Optional flags:
|
|
187
|
+
|
|
188
|
+
- `parameters.enable_metrics: true` for metrics events
|
|
189
|
+
- `parameters.enable_error_message: true` for error events
|
|
190
|
+
|
|
191
|
+
## Stop and Cleanup Sequence
|
|
192
|
+
|
|
193
|
+
Reverse initialization on session end:
|
|
194
|
+
|
|
195
|
+
1. Stop the agent from your server (`POST /leave` or SDK `stop()`)
|
|
196
|
+
2. Client leaves RTC channel
|
|
197
|
+
3. Client unsubscribes from RTM channel
|
|
198
|
+
4. Client logs out of RTM
|
|
199
|
+
5. Release RTC/RTM engines
|
|
200
|
+
|
|
201
|
+
Stopping the agent before the client leaves gives the platform time to exit the channel cleanly.
|
|
202
|
+
|
|
203
|
+
## Where Code Lives
|
|
204
|
+
|
|
205
|
+
| Layer | Responsibility | Typical source |
|
|
206
|
+
| ------------------------ | ----------------------------------- | --------------------------------------------------------------- |
|
|
207
|
+
| Server start/stop/update | Agent lifecycle, token minting | Official quickstart backend or [server-sdks.md](server-sdks.md) |
|
|
208
|
+
| Client RTC join | Audio in/out | Official quickstart frontend or RTC refs |
|
|
209
|
+
| Client RTM + toolkit | Transcripts, state, sendText | [agent-toolkit.md](agent-toolkit.md) |
|
|
210
|
+
| Join payload shape | LLM/TTS/ASR config, channel, tokens | Official quickstart or OpenAPI spec |
|
|
211
|
+
|
|
212
|
+
When adapting into an existing app, use [integration-from-quickstart.md](integration-from-quickstart.md) to map quickstart source files to your app paths before editing.
|
|
213
|
+
|
|
214
|
+
## Common Mistakes
|
|
215
|
+
|
|
216
|
+
- Generating `/join` payloads from memory instead of copying from the official quickstart source.
|
|
217
|
+
- Treating `POST /join` response as proof the agent is already audible in RTC.
|
|
218
|
+
- Using one RTC token for both the client and the ConvoAI `Authorization` header.
|
|
219
|
+
- Mismatching RTM login identity and RTM token subject.
|
|
220
|
+
- Enabling RTM in the project but omitting both RTM join flags.
|
|
221
|
+
- Calling client toolkit init before the server has started an agent.
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agora-convoai-auth-flow
|
|
3
|
+
description: |
|
|
4
|
+
End-to-end token and authentication flow for Agora Conversational AI — for implementors
|
|
5
|
+
calling the REST API directly (Java, Ruby, PHP, C#, or any language without an SDK).
|
|
6
|
+
The TypeScript/Python/Go SDKs handle ConvoAI token generation automatically in App
|
|
7
|
+
Credentials mode — use server-sdks.md for those.
|
|
8
|
+
Use this file when: the user needs to understand which token goes where, is building a
|
|
9
|
+
non-SDK backend, or explicitly asks about token types / buildTokenWithRtm / auth flow.
|
|
10
|
+
Triggers on: RTC token, RTM token, ConvoAI token, token flow, auth flow, buildTokenWithRtm,
|
|
11
|
+
three tokens, REST API auth, Java backend, direct REST.
|
|
12
|
+
license: MIT
|
|
13
|
+
metadata:
|
|
14
|
+
author: agora
|
|
15
|
+
version: '1.0.0'
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# ConvoAI Auth Flow — End to End
|
|
19
|
+
|
|
20
|
+
> **Architecture first:** For the full system diagram, start/stop sequence, and component roles, read [architecture.md](architecture.md).
|
|
21
|
+
|
|
22
|
+
> **SDK users:** If you are using the TypeScript, Python, or Go SDK, you do not need to implement this flow manually. Pass `appId + appCertificate` to the SDK client and it generates the ConvoAI token per request automatically. See [server-sdks.md](server-sdks.md). This file is for backends calling the REST API directly.
|
|
23
|
+
|
|
24
|
+
Three separate tokens exist in a ConvoAI integration. Developers routinely confuse them because they all use the same App ID + App Certificate as inputs.
|
|
25
|
+
|
|
26
|
+
## The Three Tokens
|
|
27
|
+
|
|
28
|
+
| Token | Generated by | Used by | Where |
|
|
29
|
+
|-------|-------------|---------|-------|
|
|
30
|
+
| **RTC client token** | Your server | Browser / mobile RTC client | `rtcClient.join(appId, channel, token, uid)` |
|
|
31
|
+
| **RTM client token** | Your server | Browser / mobile RTM client | `rtmClient.login({ token })` |
|
|
32
|
+
| **ConvoAI server token** | SDK (App Credentials mode) or you (Token Auth mode) | Your server → ConvoAI REST API | `Authorization: agora token=<value>` |
|
|
33
|
+
|
|
34
|
+
The **ConvoAI server token** is a combined RTC + RTM token generated with `RtcTokenBuilder.buildTokenWithRtm()`. It is **not** the same as the RTC token the client uses to join the channel — even though both use the same builder.
|
|
35
|
+
|
|
36
|
+
## Sequence Diagram
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
Your Server ConvoAI Platform Browser Client
|
|
40
|
+
| | |
|
|
41
|
+
|-- POST /join -------->| |
|
|
42
|
+
| Authorization: | |
|
|
43
|
+
| agora token=<convoAI token> |
|
|
44
|
+
| agent starts |
|
|
45
|
+
|<-- agentId -----------| |
|
|
46
|
+
| | |
|
|
47
|
+
|-- RTC token + RTM token ---------------------->|
|
|
48
|
+
| | |
|
|
49
|
+
| |<-- rtcClient.join() ---|
|
|
50
|
+
| |<-- rtmClient.login() --|
|
|
51
|
+
| | |
|
|
52
|
+
| |<==== RTC audio =======>|
|
|
53
|
+
| |<==== RTM messages ====>|
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Worked Example
|
|
57
|
+
|
|
58
|
+
### 1. Your server generates client tokens
|
|
59
|
+
|
|
60
|
+
```typescript
|
|
61
|
+
import { RtcTokenBuilder, RtmTokenBuilder, RtcRole } from 'agora-token';
|
|
62
|
+
|
|
63
|
+
const uid = 12345; // user's numeric UID
|
|
64
|
+
const channel = 'my-channel';
|
|
65
|
+
const expiry = 3600; // 1 hour
|
|
66
|
+
|
|
67
|
+
// RTC token — for the client to join the RTC channel
|
|
68
|
+
const rtcToken = RtcTokenBuilder.buildTokenWithUid(
|
|
69
|
+
appId, appCertificate, channel, uid, RtcRole.PUBLISHER,
|
|
70
|
+
expiry, expiry
|
|
71
|
+
);
|
|
72
|
+
|
|
73
|
+
// RTM token — for the client to log into RTM (required for sendText/interrupt)
|
|
74
|
+
const rtmToken = RtmTokenBuilder.buildToken(
|
|
75
|
+
appId, appCertificate, String(uid), expiry
|
|
76
|
+
);
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
### 2. Your server starts the agent
|
|
80
|
+
|
|
81
|
+
Using App Credentials mode — the SDK generates the ConvoAI token automatically:
|
|
82
|
+
|
|
83
|
+
```typescript
|
|
84
|
+
import { AgoraClient, Agent } from 'agora-agents';
|
|
85
|
+
|
|
86
|
+
const client = new AgoraClient({
|
|
87
|
+
appId: process.env.AGORA_APP_ID,
|
|
88
|
+
appCertificate: process.env.AGORA_APP_CERTIFICATE, // SDK handles ConvoAI token
|
|
89
|
+
});
|
|
90
|
+
|
|
91
|
+
const session = agent.createSession({ channel, agentUid: 0 });
|
|
92
|
+
const agentId = await session.start(); // ConvoAI token generated internally per request
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Using Token Auth mode — you generate the ConvoAI token yourself:
|
|
96
|
+
|
|
97
|
+
```typescript
|
|
98
|
+
// ConvoAI token = combined RTC + RTM token, bound to the agent's uid on the channel
|
|
99
|
+
const convoAIToken = RtcTokenBuilder.buildTokenWithRtm(
|
|
100
|
+
appId, appCertificate, channel,
|
|
101
|
+
'agent-account', // use account (string) not numeric uid for the agent
|
|
102
|
+
RtcRole.PUBLISHER,
|
|
103
|
+
expiry, expiry
|
|
104
|
+
);
|
|
105
|
+
|
|
106
|
+
const client = new AgoraClient({
|
|
107
|
+
appId: process.env.AGORA_APP_ID,
|
|
108
|
+
authToken: convoAIToken, // reused for every request until you replace it
|
|
109
|
+
});
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### 3. Your server returns client tokens to the browser
|
|
113
|
+
|
|
114
|
+
```typescript
|
|
115
|
+
// Express handler example
|
|
116
|
+
app.post('/start-session', async (req, res) => {
|
|
117
|
+
const { uid, channel } = req.body;
|
|
118
|
+
|
|
119
|
+
const rtcToken = /* generate as above */;
|
|
120
|
+
const rtmToken = /* generate as above */;
|
|
121
|
+
const agentId = await session.start();
|
|
122
|
+
|
|
123
|
+
res.json({ rtcToken, rtmToken, agentId, channel });
|
|
124
|
+
});
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
### 4. Browser initializes with both tokens
|
|
128
|
+
|
|
129
|
+
```typescript
|
|
130
|
+
import AgoraRTC from 'agora-rtc-sdk-ng';
|
|
131
|
+
import AgoraRTM from 'agora-rtm';
|
|
132
|
+
import { AgoraVoiceAI } from 'agora-agent-client-toolkit';
|
|
133
|
+
|
|
134
|
+
const { rtcToken, rtmToken, channel } = await fetch('/start-session', { ... }).then(r => r.json());
|
|
135
|
+
|
|
136
|
+
const rtcClient = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' });
|
|
137
|
+
const rtmClient = new AgoraRTM.RTM(appId, String(uid));
|
|
138
|
+
await rtmClient.login({ token: rtmToken }); // RTM token here
|
|
139
|
+
|
|
140
|
+
const ai = await AgoraVoiceAI.init({
|
|
141
|
+
rtcEngine: rtcClient,
|
|
142
|
+
rtmConfig: { rtmEngine: rtmClient },
|
|
143
|
+
});
|
|
144
|
+
|
|
145
|
+
await rtcClient.join(appId, channel, rtcToken, uid); // RTC token here
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
**Identity rule:** the RTM client identity must match the identity the RTM token was minted for. If your backend minted `rtmToken` for `String(uid)`, do not create the RTM client with a different random user ID. In first-success paths, this kind of mismatch can surface as generic startup failures rather than a clear token error.
|
|
149
|
+
|
|
150
|
+
## App Credentials Shortcut
|
|
151
|
+
|
|
152
|
+
With `appId + appCertificate` passed to `AgoraClient`, the SDK generates the ConvoAI token per request. The developer only needs to manage the two client-side tokens (RTC + RTM). This is the recommended path for production.
|
|
153
|
+
|
|
154
|
+
App Certificate never leaves your server in either case — the browser only ever receives the scoped RTC and RTM tokens.
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
# ConvoAI Studio Agent ID
|
|
2
|
+
|
|
3
|
+
Use this file when the user already has an **Agora Studio Agent ID** from:
|
|
4
|
+
|
|
5
|
+
<https://console.agora.io/studio/agents>
|
|
6
|
+
|
|
7
|
+
This is the Agora-version analogue of a preconfigured agent path: the Studio-managed agent
|
|
8
|
+
configuration already exists, so quickstart should avoid rebuilding the provider stack from
|
|
9
|
+
scratch unless the user explicitly asks to replace it.
|
|
10
|
+
|
|
11
|
+
## What It Is
|
|
12
|
+
|
|
13
|
+
- **Studio Agent ID**: identifies an agent configuration created or managed in Agora Studio.
|
|
14
|
+
- **Runtime `agent_id`**: identifies a live started agent session returned by the ConvoAI REST API.
|
|
15
|
+
- **Request field mapping**: when reusing an Agora Studio-managed agent in the start flow, pass the Studio Agent ID via the request field `pipeline_id`.
|
|
16
|
+
|
|
17
|
+
These are **not** interchangeable.
|
|
18
|
+
|
|
19
|
+
## When to Use This Path
|
|
20
|
+
|
|
21
|
+
Use the Studio Agent ID path when:
|
|
22
|
+
|
|
23
|
+
- the user explicitly says they already have an Agent ID from the Studio Agents page
|
|
24
|
+
- the user wants to reuse an agent configured in Studio
|
|
25
|
+
- the user does not want to re-enter STT / LLM / TTS provider details during quickstart
|
|
26
|
+
|
|
27
|
+
Do **not** use this path when:
|
|
28
|
+
|
|
29
|
+
- the user only has the runtime `agent_id` returned by `/join`
|
|
30
|
+
- the user still needs to choose or build the provider stack from scratch
|
|
31
|
+
|
|
32
|
+
## Quickstart Rules
|
|
33
|
+
|
|
34
|
+
If the Studio Agent ID path is chosen:
|
|
35
|
+
|
|
36
|
+
1. Treat Agora Studio as the source of truth for the agent configuration.
|
|
37
|
+
2. Do not re-ask provider-vendor questions unless the user explicitly wants to replace the Studio-managed config.
|
|
38
|
+
3. Keep the client and auth path aligned with the chosen quickstart baseline (`full-stack-nextjs`, `separate-backend-frontend`, or `existing-app-integration`).
|
|
39
|
+
4. Use the Studio Agent ID as `pipeline_id` in the request body.
|
|
40
|
+
5. Before generating exact request code, still verify the current official ConvoAI docs for any other request-shape changes. Do not fabricate undocumented fields beyond the confirmed `pipeline_id` mapping.
|
|
41
|
+
|
|
42
|
+
For an existing app, still use the official quickstart as the Studio-managed source path first. After the source is inspected, use [integration-from-quickstart.md](integration-from-quickstart.md) to detect the app shape and create a copy map before editing the existing app. Runtime proof is required before claiming the integrated app works.
|
|
43
|
+
|
|
44
|
+
## Goal
|
|
45
|
+
|
|
46
|
+
Reuse the official sample repo as the structural baseline, but replace the default provider-selection path with the user's existing Studio-managed agent path.
|
|
47
|
+
|
|
48
|
+
This flow is for implementation after quickstart confirms the user already has a Studio Agent ID. Do not send the user back to provider selection unless they explicitly want to replace the Studio-managed config.
|
|
49
|
+
|
|
50
|
+
## User-Facing Guidance
|
|
51
|
+
|
|
52
|
+
Suggested explanation:
|
|
53
|
+
|
|
54
|
+
```text
|
|
55
|
+
If you already configured the agent in Agora Studio, we can treat Studio as the source of truth for the agent configuration and avoid rebuilding the provider stack from scratch here.
|
|
56
|
+
|
|
57
|
+
Open `https://console.agora.io/studio/agents`, find the agent you want to reuse, and copy its Agent ID.
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## Request Shape Rule
|
|
61
|
+
|
|
62
|
+
For this Studio path, use the same request-field convention as the parallel preconfigured-agent flow:
|
|
63
|
+
|
|
64
|
+
- copy the **Agent ID** from `https://console.agora.io/studio/agents`
|
|
65
|
+
- pass that value as `pipeline_id` in the request body
|
|
66
|
+
|
|
67
|
+
In other words:
|
|
68
|
+
|
|
69
|
+
- **Studio UI name**: `Agent ID`
|
|
70
|
+
- **Request field name**: `pipeline_id`
|
|
71
|
+
|
|
72
|
+
## Current Request Shape
|
|
73
|
+
|
|
74
|
+
Current fixed request shape for the Studio Agent ID path:
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
POST https://api.agora.io/api/conversational-ai-agent/v2/projects/{AGORA_APP_ID}/join
|
|
78
|
+
Authorization: agora token={RTC_HEADER_TOKEN}
|
|
79
|
+
Content-Type: application/json
|
|
80
|
+
|
|
81
|
+
{
|
|
82
|
+
"name": "{channel}",
|
|
83
|
+
"pipeline_id": "{AGORA_STUDIO_AGENT_ID}",
|
|
84
|
+
"properties": {
|
|
85
|
+
"agent_rtc_uid": "{agent_rtc_uid}",
|
|
86
|
+
"channel": "{channel}",
|
|
87
|
+
"remote_rtc_uids": ["*"],
|
|
88
|
+
"token": "{RTC_AGENT_TOKEN}"
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Field mapping rules:
|
|
94
|
+
|
|
95
|
+
| Request field | Source |
|
|
96
|
+
|---|---|
|
|
97
|
+
| URL project segment | Existing `AGORA_APP_ID` |
|
|
98
|
+
| `Authorization` header token | RTC token generated with the caller/user UID using the sample's existing token-generation path |
|
|
99
|
+
| `name` | Same value as `channel` |
|
|
100
|
+
| `pipeline_id` | `AGORA_STUDIO_AGENT_ID` copied from `https://console.agora.io/studio/agents` |
|
|
101
|
+
| `properties.agent_rtc_uid` | Runtime RTC UID string |
|
|
102
|
+
| `properties.channel` | Runtime channel value |
|
|
103
|
+
| `properties.remote_rtc_uids` | `["*"]` unless the user asks for specific UIDs |
|
|
104
|
+
| `properties.token` | Separate RTC token generated with the agent UID using the sample's existing token-generation path |
|
|
105
|
+
|
|
106
|
+
Do not reintroduce the old provider-based `llm`, `tts`, or `asr` request blocks in this Studio-managed path.
|
|
107
|
+
Do not reuse one RTC token for both the `Authorization` header and `properties.token`.
|
|
108
|
+
|
|
109
|
+
## Expected Success Response
|
|
110
|
+
|
|
111
|
+
Code generated for this flow should expect and preserve the standard live-agent response fields:
|
|
112
|
+
|
|
113
|
+
- `agent_id`
|
|
114
|
+
- `create_ts`
|
|
115
|
+
- `status`
|
|
116
|
+
|
|
117
|
+
## Minimum Contract
|
|
118
|
+
|
|
119
|
+
For the Studio-managed path, the skill may assume:
|
|
120
|
+
|
|
121
|
+
- the Studio Agent ID value is supplied by the user
|
|
122
|
+
- that value maps to `pipeline_id`
|
|
123
|
+
- the runtime `agent_id` is still returned by the live start/join flow and must not be confused with the Studio Agent ID
|
|
124
|
+
|
|
125
|
+
## Env and Config Rules
|
|
126
|
+
|
|
127
|
+
- Prefer `AGORA_STUDIO_AGENT_ID` as the config key / placeholder name in code or env templates.
|
|
128
|
+
- Reuse the sample's existing `AGORA_APP_ID` and token-generation path.
|
|
129
|
+
- Remove or bypass provider-only config when it is only used for the old three-stage selection flow.
|
|
130
|
+
- Keep the sample's existing config style; do not invent a second config-loading layer just for `AGORA_STUDIO_AGENT_ID`.
|
|
131
|
+
- Treat `AGORA_STUDIO_AGENT_ID` as a user-filled config value. Add the placeholder, but do not ask the user to paste the live value into the conversation after it has already been identified.
|
|
132
|
+
|
|
133
|
+
## Implementation Guardrails
|
|
134
|
+
|
|
135
|
+
- Do not confuse the Studio Agent ID with the runtime `agent_id`.
|
|
136
|
+
- Do not replace `pipeline_id` with `agent_id` in request generation.
|
|
137
|
+
- Do not hardcode the `Authorization` token or the RTC `properties.token`; both must come from the runtime token path.
|
|
138
|
+
- Before code generation, fetch the current official ConvoAI docs / OpenAPI and verify:
|
|
139
|
+
- any other required body fields around `pipeline_id`
|
|
140
|
+
- whether Studio-managed agents require additional prerequisites or restrictions
|
|
141
|
+
- the current response shape
|
|
142
|
+
|
|
143
|
+
## Implementation Workflow
|
|
144
|
+
|
|
145
|
+
1. Keep the chosen quickstart baseline (`full-stack-nextjs`, `separate-backend-frontend`, or `existing-app-integration`) as the structural baseline.
|
|
146
|
+
2. Inspect the repo's actual env/config files and the current request path that starts the agent.
|
|
147
|
+
3. Replace only the provider-selection-specific request/config path with the fixed Studio Agent ID request shape in this file.
|
|
148
|
+
4. Generate the new request code from the fixed shape above, preserving:
|
|
149
|
+
- `POST /api/conversational-ai-agent/v2/projects/{appId}/join`
|
|
150
|
+
- `Authorization: agora token=...`
|
|
151
|
+
- JSON body with `name`, `pipeline_id`, and `properties`
|
|
152
|
+
5. Map dynamic fields to runtime/config sources:
|
|
153
|
+
- `name` → same value as `channel`
|
|
154
|
+
- `pipeline_id` → `AGORA_STUDIO_AGENT_ID`
|
|
155
|
+
- URL project segment → `AGORA_APP_ID`
|
|
156
|
+
- `Authorization` header token → RTC token generated with the caller/user UID
|
|
157
|
+
- `properties.token` → separate RTC token generated with the agent UID
|
|
158
|
+
- `channel` / `agent_rtc_uid` → runtime values
|
|
159
|
+
6. Parse and preserve `agent_id`, `create_ts`, and `status` from the response.
|
|
160
|
+
7. Keep the rest of the repo structure and RTC/UI flow as close to the sample as possible.
|
|
161
|
+
|
|
162
|
+
## After This Step
|
|
163
|
+
|
|
164
|
+
Once the Studio Agent ID is collected:
|
|
165
|
+
|
|
166
|
+
- keep quickstart in the selected baseline path
|
|
167
|
+
- for existing apps, continue through [integration-from-quickstart.md](integration-from-quickstart.md) after the Studio quickstart source is inspected
|
|
168
|
+
- use the official current ConvoAI docs to verify the exact start flow
|
|
169
|
+
- then continue with the appropriate backend/client reference:
|
|
170
|
+
- [server-sdks.md](server-sdks.md)
|
|
171
|
+
- [python-sdk.md](python-sdk.md)
|
|
172
|
+
- [go-sdk.md](go-sdk.md)
|
|
173
|
+
- [agent-samples.md](agent-samples.md)
|