@aituber-onair/core 0.7.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +639 -7
- package/dist/constants/gemini.d.ts +1 -1
- package/dist/constants/gemini.js +1 -1
- package/dist/constants/gemini.js.map +1 -1
- package/dist/core/AITuberOnAirCore.d.ts +32 -1
- package/dist/core/AITuberOnAirCore.js +38 -2
- package/dist/core/AITuberOnAirCore.js.map +1 -1
- package/dist/core/ChatProcessor.d.ts +9 -1
- package/dist/core/ChatProcessor.js +116 -51
- package/dist/core/ChatProcessor.js.map +1 -1
- package/dist/core/MemoryManager.js +10 -9
- package/dist/core/MemoryManager.js.map +1 -1
- package/dist/core/ToolExecutor.d.ts +9 -0
- package/dist/core/ToolExecutor.js +39 -0
- package/dist/core/ToolExecutor.js.map +1 -0
- package/dist/services/chat/ChatService.d.ts +13 -0
- package/dist/services/chat/providers/claude/ClaudeChatService.d.ts +45 -4
- package/dist/services/chat/providers/claude/ClaudeChatService.js +227 -180
- package/dist/services/chat/providers/claude/ClaudeChatService.js.map +1 -1
- package/dist/services/chat/providers/claude/ClaudeChatServiceProvider.js +1 -1
- package/dist/services/chat/providers/claude/ClaudeChatServiceProvider.js.map +1 -1
- package/dist/services/chat/providers/gemini/GeminiChatService.d.ts +19 -16
- package/dist/services/chat/providers/gemini/GeminiChatService.js +376 -157
- package/dist/services/chat/providers/gemini/GeminiChatService.js.map +1 -1
- package/dist/services/chat/providers/gemini/GeminiChatServiceProvider.js +1 -1
- package/dist/services/chat/providers/gemini/GeminiChatServiceProvider.js.map +1 -1
- package/dist/services/chat/providers/openai/OpenAIChatService.d.ts +21 -3
- package/dist/services/chat/providers/openai/OpenAIChatService.js +205 -114
- package/dist/services/chat/providers/openai/OpenAIChatService.js.map +1 -1
- package/dist/services/chat/providers/openai/OpenAIChatServiceProvider.js +3 -1
- package/dist/services/chat/providers/openai/OpenAIChatServiceProvider.js.map +1 -1
- package/dist/services/voice/VoiceEngineAdapter.d.ts +1 -1
- package/dist/services/voice/VoiceEngineAdapter.js +1 -1
- package/dist/services/voice/VoiceService.d.ts +1 -1
- package/dist/types/chat.d.ts +2 -2
- package/dist/types/index.d.ts +1 -0
- package/dist/types/index.js +2 -0
- package/dist/types/index.js.map +1 -1
- package/dist/types/toolChat.d.ts +46 -0
- package/dist/types/toolChat.js +2 -0
- package/dist/types/toolChat.js.map +1 -0
- package/package.json +1 -1
- package/dist/constants/api.d.ts +0 -4
- package/dist/constants/api.js +0 -13
- package/dist/constants/api.js.map +0 -1
- package/dist/constants/openaiApi.d.ts +0 -15
- package/dist/constants/openaiApi.js +0 -15
- package/dist/constants/openaiApi.js.map +0 -1
- package/dist/services/chat/ClaudeChatService.d.ts +0 -64
- package/dist/services/chat/ClaudeChatService.js +0 -237
- package/dist/services/chat/ClaudeChatService.js.map +0 -1
- package/dist/services/chat/GeminiChatService.d.ts +0 -63
- package/dist/services/chat/GeminiChatService.js +0 -314
- package/dist/services/chat/GeminiChatService.js.map +0 -1
- package/dist/services/chat/OpenAIChatService.d.ts +0 -39
- package/dist/services/chat/OpenAIChatService.js +0 -171
- package/dist/services/chat/OpenAIChatService.js.map +0 -1
- package/dist/services/chat/OpenAISummarizer.d.ts +0 -25
- package/dist/services/chat/OpenAISummarizer.js +0 -70
- package/dist/services/chat/OpenAISummarizer.js.map +0 -1
- package/dist/services/chat/providers/ClaudeChatServiceProvider.d.ts +0 -39
- package/dist/services/chat/providers/ClaudeChatServiceProvider.js +0 -57
- package/dist/services/chat/providers/ClaudeChatServiceProvider.js.map +0 -1
- package/dist/services/chat/providers/GeminiChatServiceProvider.d.ts +0 -39
- package/dist/services/chat/providers/GeminiChatServiceProvider.js +0 -57
- package/dist/services/chat/providers/GeminiChatServiceProvider.js.map +0 -1
- package/dist/services/chat/providers/OpenAIChatServiceProvider.d.ts +0 -39
- package/dist/services/chat/providers/OpenAIChatServiceProvider.js +0 -57
- package/dist/services/chat/providers/OpenAIChatServiceProvider.js.map +0 -1
package/README.md
CHANGED
|
@@ -16,6 +16,9 @@ It specializes in generating response text and audio from text or image inputs,
|
|
|
16
16
|
- [Installation](#installation)
|
|
17
17
|
- [Main Features](#main-features)
|
|
18
18
|
- [Basic Usage](#basic-usage)
|
|
19
|
+
- [Tool System](#tool-system)
|
|
20
|
+
- [Function Calling Differences](#function-calling-differences)
|
|
21
|
+
- [Using MCP](#using-mcp)
|
|
19
22
|
- [Architecture](#architecture)
|
|
20
23
|
- [Main Components](#main-components)
|
|
21
24
|
- [Event System](#event-system)
|
|
@@ -25,6 +28,7 @@ It specializes in generating response text and audio from text or image inputs,
|
|
|
25
28
|
- [Examples](#examples)
|
|
26
29
|
- [Integration with Existing Applications](#integration-with-existing-applications)
|
|
27
30
|
- [Testing & Development](#testing--development)
|
|
31
|
+
- [Migration Guide for Memory Events](#migration-guide-for-memory-events)
|
|
28
32
|
|
|
29
33
|
## Overview
|
|
30
34
|
|
|
@@ -68,9 +72,14 @@ pnpm install @aituber-onair/core
|
|
|
68
72
|
Allows customization of prompts for vision processing and conversation summarization.
|
|
69
73
|
- **Pluggable Persistence**
|
|
70
74
|
Memory features can be persisted via LocalStorage, IndexedDB, or other customizable methods.
|
|
75
|
+
- **Function Calling with Tools Support**
|
|
76
|
+
Enables AI to use tools for performing actions beyond text generation, such as calculations, API calls, or data retrieval.
|
|
71
77
|
|
|
72
78
|
## Basic Usage
|
|
73
79
|
|
|
80
|
+
For more practical usage examples, please see [Simple AI Chat App with AITuber OnAir Core]
|
|
81
|
+
(https://github.com/shinshin86/simple-aichat-app-with-aituber-onair-core).
|
|
82
|
+
|
|
74
83
|
Below is a simplified example of how to use **AITuber OnAir Core**:
|
|
75
84
|
|
|
76
85
|
```typescript
|
|
@@ -153,10 +162,35 @@ aituber.on(AITuberOnAirCoreEvent.SPEECH_END, () => {
|
|
|
153
162
|
console.log('Speech playback finished');
|
|
154
163
|
});
|
|
155
164
|
|
|
165
|
+
aituber.on(AITuberOnAirCoreEvent.TOOL_USE, (toolBlock) =>
|
|
166
|
+
console.log(`Tool use -> ${toolBlock.name}`, toolBlock.input));
|
|
167
|
+
|
|
168
|
+
aituber.on(AITuberOnAirCoreEvent.TOOL_RESULT, (resultBlock) =>
|
|
169
|
+
console.log(`Tool result ->`, resultBlock.content));
|
|
170
|
+
|
|
156
171
|
aituber.on(AITuberOnAirCoreEvent.ERROR, (error) => {
|
|
157
172
|
console.error('Error occurred:', error);
|
|
158
173
|
});
|
|
159
174
|
|
|
175
|
+
// Memory and chat history related events
|
|
176
|
+
aituber.on(AITuberOnAirCoreEvent.CHAT_HISTORY_SET, (messages) =>
|
|
177
|
+
console.log('Chat history set:', messages.length));
|
|
178
|
+
|
|
179
|
+
aituber.on(AITuberOnAirCoreEvent.CHAT_HISTORY_CLEARED, () =>
|
|
180
|
+
console.log('Chat history cleared'));
|
|
181
|
+
|
|
182
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_CREATED, (memory) =>
|
|
183
|
+
console.log(`New memory created: ${memory.type}`));
|
|
184
|
+
|
|
185
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_REMOVED, (memoryIds) =>
|
|
186
|
+
console.log('Memory removed:', memoryIds));
|
|
187
|
+
|
|
188
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_LOADED, (memories) =>
|
|
189
|
+
console.log('Memory loaded:', memories.length));
|
|
190
|
+
|
|
191
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_SAVED, (memories) =>
|
|
192
|
+
console.log('Memory saved:', memories.length));
|
|
193
|
+
|
|
160
194
|
// 4. Process text input
|
|
161
195
|
await aituber.processChat('Hello, how is the weather today?');
|
|
162
196
|
|
|
@@ -164,6 +198,488 @@ await aituber.processChat('Hello, how is the weather today?');
|
|
|
164
198
|
aituber.offAll();
|
|
165
199
|
```
|
|
166
200
|
|
|
201
|
+
## Tool System
|
|
202
|
+
|
|
203
|
+
AITuber OnAir Core includes a powerful tool system that allows AI to perform actions beyond text generation, such as retrieving data or making calculations. This is particularly useful for creating interactive AITuber experiences.
|
|
204
|
+
|
|
205
|
+
### Tool Definition Structure
|
|
206
|
+
|
|
207
|
+
Tools are defined using the `ToolDefinition` interface, which conforms to the function calling specification used by LLM providers:
|
|
208
|
+
|
|
209
|
+
```typescript
|
|
210
|
+
type ToolDefinition = {
|
|
211
|
+
name: string; // The name of the tool
|
|
212
|
+
description?: string; // Optional description of what the tool does
|
|
213
|
+
parameters: {
|
|
214
|
+
type: 'object'; // Must be 'object' (strictly typed)
|
|
215
|
+
properties?: Record<string, {
|
|
216
|
+
type?: string; // Parameter type (e.g. 'string', 'integer')
|
|
217
|
+
description?: string; // Parameter description
|
|
218
|
+
enum?: any[]; // For enumerated values
|
|
219
|
+
items?: any; // For array types
|
|
220
|
+
required?: string[]; // Required nested properties
|
|
221
|
+
[key: string]: any; // Other JSON Schema properties
|
|
222
|
+
}>;
|
|
223
|
+
required?: string[]; // Names of required parameters
|
|
224
|
+
[key: string]: any; // Other JSON Schema properties
|
|
225
|
+
};
|
|
226
|
+
config?: { timeoutMs?: number }; // Optional configuration
|
|
227
|
+
};
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
Note that the `parameters.type` property is strictly typed as `'object'` to conform to function calling standards used by LLM providers.
|
|
231
|
+
|
|
232
|
+
### Registering and Using Tools
|
|
233
|
+
|
|
234
|
+
Tools are registered when initializing AITuberOnAirCore:
|
|
235
|
+
|
|
236
|
+
```typescript
|
|
237
|
+
// Define a tool
|
|
238
|
+
const randomIntTool: ToolDefinition = {
|
|
239
|
+
name: 'randomInt',
|
|
240
|
+
description: 'Return a random integer from 0 to (max - 1)',
|
|
241
|
+
parameters: {
|
|
242
|
+
type: 'object', // This must be 'object'
|
|
243
|
+
properties: {
|
|
244
|
+
max: {
|
|
245
|
+
type: 'integer',
|
|
246
|
+
description: 'Upper bound (exclusive). Defaults to 100.',
|
|
247
|
+
minimum: 1,
|
|
248
|
+
},
|
|
249
|
+
},
|
|
250
|
+
},
|
|
251
|
+
};
|
|
252
|
+
|
|
253
|
+
// Create a handler for the tool
|
|
254
|
+
async function randomIntHandler({ max = 100 }: { max?: number }) {
|
|
255
|
+
return Math.floor(Math.random() * max).toString();
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
// Register the tool with AITuberOnAirCore
|
|
259
|
+
const aituber = new AITuberOnAirCore({
|
|
260
|
+
// ... other options ...
|
|
261
|
+
tools: [{ definition: randomIntTool, handler: randomIntHandler }],
|
|
262
|
+
});
|
|
263
|
+
|
|
264
|
+
// Set up event listeners for tool use
|
|
265
|
+
aituber.on(AITuberOnAirCoreEvent.TOOL_USE, (toolBlock) =>
|
|
266
|
+
console.log(`Tool use -> ${toolBlock.name}`, toolBlock.input));
|
|
267
|
+
|
|
268
|
+
aituber.on(AITuberOnAirCoreEvent.TOOL_RESULT, (resultBlock) =>
|
|
269
|
+
console.log(`Tool result ->`, resultBlock.content));
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
### Tool Iteration Control
|
|
273
|
+
|
|
274
|
+
You can limit the number of tool call iterations using the `maxHops` option:
|
|
275
|
+
|
|
276
|
+
```typescript
|
|
277
|
+
const aituber = new AITuberOnAirCore({
|
|
278
|
+
// ... other options ...
|
|
279
|
+
chatOptions: {
|
|
280
|
+
systemPrompt: 'Your system prompt',
|
|
281
|
+
// ... other chat options ...
|
|
282
|
+
maxHops: 10, // Maximum number of tool call iterations (default: 6)
|
|
283
|
+
},
|
|
284
|
+
tools: [/* your tools */],
|
|
285
|
+
});
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
### Function Calling Differences
|
|
289
|
+
|
|
290
|
+
AITuber OnAir Core supports three major AI providers: OpenAI, Claude, and Gemini. Each provider has a different implementation of function calling (tool invocation). These differences are abstracted by AITuber OnAir Core, allowing developers to use a unified interface, but understanding the background is important.
|
|
291
|
+
|
|
292
|
+
> Note: This explanation covers the API versions as of May 2025. APIs are frequently updated, so please refer to the official documentation for the latest information.
|
|
293
|
+
|
|
294
|
+
#### OpenAI Function Calling Implementation
|
|
295
|
+
|
|
296
|
+
OpenAI's function calling has the following characteristics:
|
|
297
|
+
|
|
298
|
+
- **Tool Definition Format**: Uses an array of `functions` (deprecated) or `tools` (recommended from 2023-12-01) based on JSON Schema
|
|
299
|
+
- **Response Format**: Returns a response object containing a `tool_calls` array when using tools
|
|
300
|
+
- **Tool Result Submission**: Tool results are sent as messages with `role: 'tool'`
|
|
301
|
+
- **Multiple Tool Support**: Can call multiple tools simultaneously (Parallel function calling)
|
|
302
|
+
|
|
303
|
+
```typescript
|
|
304
|
+
// OpenAI tool definition example (minimal form)
|
|
305
|
+
const tools = [
|
|
306
|
+
{
|
|
307
|
+
type: "function",
|
|
308
|
+
function: {
|
|
309
|
+
name: "randomInt",
|
|
310
|
+
description: "Return a random integer from 0 to (max - 1)",
|
|
311
|
+
parameters: {
|
|
312
|
+
type: "object",
|
|
313
|
+
properties: {
|
|
314
|
+
max: {
|
|
315
|
+
type: "integer",
|
|
316
|
+
description: "Upper bound (exclusive). Defaults to 100."
|
|
317
|
+
}
|
|
318
|
+
},
|
|
319
|
+
required: [] // Explicitly specifying even when empty improves schema validity
|
|
320
|
+
}
|
|
321
|
+
}
|
|
322
|
+
}
|
|
323
|
+
];
|
|
324
|
+
|
|
325
|
+
// OpenAI tool call response example
|
|
326
|
+
{
|
|
327
|
+
role: "assistant",
|
|
328
|
+
content: null,
|
|
329
|
+
tool_calls: [
|
|
330
|
+
{
|
|
331
|
+
id: "call_abc123",
|
|
332
|
+
type: "function",
|
|
333
|
+
function: {
|
|
334
|
+
name: "randomInt",
|
|
335
|
+
arguments: "{\"max\":10}" // Note that this is returned as a stringified JSON
|
|
336
|
+
}
|
|
337
|
+
}
|
|
338
|
+
]
|
|
339
|
+
}
|
|
340
|
+
|
|
341
|
+
// Multiple tool calls example (Parallel function calling)
|
|
342
|
+
{
|
|
343
|
+
role: "assistant",
|
|
344
|
+
content: null,
|
|
345
|
+
tool_calls: [
|
|
346
|
+
{
|
|
347
|
+
id: "call_abc123",
|
|
348
|
+
type: "function",
|
|
349
|
+
function: {
|
|
350
|
+
name: "randomInt",
|
|
351
|
+
arguments: "{\"max\":10}"
|
|
352
|
+
}
|
|
353
|
+
},
|
|
354
|
+
{
|
|
355
|
+
id: "call_def456",
|
|
356
|
+
type: "function",
|
|
357
|
+
function: {
|
|
358
|
+
name: "getCurrentTime",
|
|
359
|
+
arguments: "{\"timezone\":\"JST\"}"
|
|
360
|
+
}
|
|
361
|
+
}
|
|
362
|
+
]
|
|
363
|
+
}
|
|
364
|
+
|
|
365
|
+
// OpenAI tool result submission example
|
|
366
|
+
{
|
|
367
|
+
role: "tool",
|
|
368
|
+
tool_call_id: "call_abc123",
|
|
369
|
+
content: "7"
|
|
370
|
+
}
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
When handling OpenAI's function calling, AITuber OnAir Core converts tool definitions to OpenAI's format and processes tool calls and results. The `transformToolToFunction` method in the class performs this conversion.
|
|
374
|
+
|
|
375
|
+
#### Claude's Tool Calling Implementation
|
|
376
|
+
|
|
377
|
+
Claude's tool calling has the following characteristics:
|
|
378
|
+
|
|
379
|
+
- **Tool Definition Format**: Specifies `name`, `description`, and `input_schema` for each tool in the `tools` array
|
|
380
|
+
- **Response Format**: Returned as a special block with `type: 'tool_use'` and stops with `stop_reason: 'tool_use'`
|
|
381
|
+
- **Tool Result Submission**: Included in user role messages as `type: 'tool_result'`
|
|
382
|
+
- **Special Streaming Handling**: Requires special logic to handle tool calls in streaming responses
|
|
383
|
+
|
|
384
|
+
```typescript
|
|
385
|
+
// Claude tool definition example
|
|
386
|
+
const tools = [
|
|
387
|
+
{
|
|
388
|
+
name: "randomInt",
|
|
389
|
+
description: "Return a random integer from 0 to (max - 1)",
|
|
390
|
+
input_schema: {
|
|
391
|
+
type: "object",
|
|
392
|
+
properties: {
|
|
393
|
+
max: {
|
|
394
|
+
type: "integer",
|
|
395
|
+
description: "Upper bound (exclusive). Defaults to 100."
|
|
396
|
+
}
|
|
397
|
+
}
|
|
398
|
+
}
|
|
399
|
+
}
|
|
400
|
+
];
|
|
401
|
+
|
|
402
|
+
// Claude tool call response example
|
|
403
|
+
{
|
|
404
|
+
id: "msg_abc123",
|
|
405
|
+
model: "claude-3-haiku-20240307",
|
|
406
|
+
role: "assistant",
|
|
407
|
+
content: [
|
|
408
|
+
{ type: "text", text: "I'll generate a random number for you." },
|
|
409
|
+
{
|
|
410
|
+
type: "tool_use",
|
|
411
|
+
id: "tu_abc123",
|
|
412
|
+
name: "randomInt",
|
|
413
|
+
input: { max: 10 }
|
|
414
|
+
}
|
|
415
|
+
],
|
|
416
|
+
stop_reason: "tool_use"
|
|
417
|
+
}
|
|
418
|
+
|
|
419
|
+
// Example with only tool use, no text content
|
|
420
|
+
{
|
|
421
|
+
id: "msg_xyz789",
|
|
422
|
+
model: "claude-3-haiku-20240307",
|
|
423
|
+
role: "assistant",
|
|
424
|
+
content: [
|
|
425
|
+
{
|
|
426
|
+
type: "tool_use",
|
|
427
|
+
id: "tu_xyz789",
|
|
428
|
+
name: "randomInt",
|
|
429
|
+
input: { max: 100 }
|
|
430
|
+
}
|
|
431
|
+
],
|
|
432
|
+
stop_reason: "tool_use"
|
|
433
|
+
}
|
|
434
|
+
|
|
435
|
+
// Claude tool result submission example
|
|
436
|
+
{
|
|
437
|
+
role: "user",
|
|
438
|
+
content: [
|
|
439
|
+
{
|
|
440
|
+
type: "tool_result",
|
|
441
|
+
tool_use_id: "tu_abc123",
|
|
442
|
+
content: "7"
|
|
443
|
+
}
|
|
444
|
+
]
|
|
445
|
+
}
|
|
446
|
+
```
|
|
447
|
+
|
|
448
|
+
When handling Claude's tool calls, AITuber OnAir Core processes Claude's unique format and abstracts the complex processing, especially during streaming responses. Special handling is included in the `runToolLoop` method.
|
|
449
|
+
|
|
450
|
+
#### Gemini's Tool Calling Implementation
|
|
451
|
+
|
|
452
|
+
Gemini's tool calling has the following characteristics:
|
|
453
|
+
|
|
454
|
+
- **Tool Definition Format**: Describes definitions in `functionDeclarations` within the `tools` array
|
|
455
|
+
- **Response Format**: Returned as content objects containing `functionCall` parts
|
|
456
|
+
- **Tool Result Submission**: Sent as `functionResponse` objects included in content parts
|
|
457
|
+
- **Compositional Calling**: Supports Compositional Function Calling
|
|
458
|
+
|
|
459
|
+
```typescript
|
|
460
|
+
// Gemini tool definition example
|
|
461
|
+
const tools = [
|
|
462
|
+
{
|
|
463
|
+
functionDeclarations: [
|
|
464
|
+
{
|
|
465
|
+
name: "randomInt",
|
|
466
|
+
description: "Return a random integer from 0 to (max - 1)",
|
|
467
|
+
parameters: {
|
|
468
|
+
type: "object",
|
|
469
|
+
properties: {
|
|
470
|
+
max: {
|
|
471
|
+
type: "integer",
|
|
472
|
+
description: "Upper bound (exclusive). Defaults to 100."
|
|
473
|
+
}
|
|
474
|
+
}
|
|
475
|
+
}
|
|
476
|
+
}
|
|
477
|
+
]
|
|
478
|
+
}
|
|
479
|
+
];
|
|
480
|
+
|
|
481
|
+
// Gemini tool call response example (note the deep structure)
|
|
482
|
+
{
|
|
483
|
+
candidates: [
|
|
484
|
+
{
|
|
485
|
+
content: {
|
|
486
|
+
parts: [
|
|
487
|
+
{
|
|
488
|
+
functionCall: {
|
|
489
|
+
name: "randomInt",
|
|
490
|
+
args: {
|
|
491
|
+
max: 10
|
|
492
|
+
}
|
|
493
|
+
}
|
|
494
|
+
}
|
|
495
|
+
]
|
|
496
|
+
}
|
|
497
|
+
}
|
|
498
|
+
]
|
|
499
|
+
}
|
|
500
|
+
|
|
501
|
+
// Compositional function calling example
|
|
502
|
+
{
|
|
503
|
+
candidates: [
|
|
504
|
+
{
|
|
505
|
+
content: {
|
|
506
|
+
parts: [
|
|
507
|
+
{
|
|
508
|
+
functionCall: {
|
|
509
|
+
name: "randomInt",
|
|
510
|
+
args: {
|
|
511
|
+
max: 10
|
|
512
|
+
}
|
|
513
|
+
}
|
|
514
|
+
},
|
|
515
|
+
{
|
|
516
|
+
functionCall: {
|
|
517
|
+
name: "formatResult",
|
|
518
|
+
args: {
|
|
519
|
+
prefix: "Random number:",
|
|
520
|
+
value: "<function_response:randomInt>"
|
|
521
|
+
}
|
|
522
|
+
}
|
|
523
|
+
}
|
|
524
|
+
]
|
|
525
|
+
}
|
|
526
|
+
}
|
|
527
|
+
]
|
|
528
|
+
}
|
|
529
|
+
|
|
530
|
+
// Gemini tool result submission example
|
|
531
|
+
// Include functionResponse directly in content parts (SDK automatically sets the role)
|
|
532
|
+
{
|
|
533
|
+
parts: [
|
|
534
|
+
{
|
|
535
|
+
functionResponse: {
|
|
536
|
+
name: "randomInt",
|
|
537
|
+
response: {
|
|
538
|
+
value: "7"
|
|
539
|
+
}
|
|
540
|
+
}
|
|
541
|
+
}
|
|
542
|
+
]
|
|
543
|
+
}
|
|
544
|
+
|
|
545
|
+
// When directly calling REST API, you might include role like this
|
|
546
|
+
{
|
|
547
|
+
role: "function",
|
|
548
|
+
parts: [
|
|
549
|
+
{
|
|
550
|
+
functionResponse: {
|
|
551
|
+
name: "randomInt",
|
|
552
|
+
response: {
|
|
553
|
+
value: "7"
|
|
554
|
+
}
|
|
555
|
+
}
|
|
556
|
+
}
|
|
557
|
+
]
|
|
558
|
+
}
|
|
559
|
+
```
|
|
560
|
+
|
|
561
|
+
When handling Gemini's tool calls, AITuber OnAir Core processes Gemini's complex response structure and tool result format. Special logic is needed to convert tool responses to the appropriate JSON format.
|
|
562
|
+
|
|
563
|
+
#### Streaming Implementation Differences
|
|
564
|
+
|
|
565
|
+
Each provider also has differences in how tool calls are processed during streaming responses:
|
|
566
|
+
|
|
567
|
+
1. **OpenAI**:
|
|
568
|
+
- During streaming, delta updates are sent as `delta.tool_calls`
|
|
569
|
+
- Requires accumulation to reconstruct complete tool call data
|
|
570
|
+
|
|
571
|
+
2. **Claude**:
|
|
572
|
+
- SSE streaming uses special event types `content_block_delta` and `content_block_stop`
|
|
573
|
+
- Sends `stop_reason: "tool_use"` when a tool call is completed
|
|
574
|
+
- Requires a special parser to detect tool calls
|
|
575
|
+
|
|
576
|
+
3. **Gemini**:
|
|
577
|
+
- During streaming, `functionCall` may be split across chunks
|
|
578
|
+
- Requires buffering to reconstruct complete JSON structures
|
|
579
|
+
|
|
580
|
+
AITuber OnAir Core abstracts these streaming processing differences, allowing you to process tool calls and results with the same interface regardless of which provider you use.
|
|
581
|
+
|
|
582
|
+
### Key Differences and Abstraction Between Providers
|
|
583
|
+
|
|
584
|
+
AITuber OnAir Core abstracts the differences between these three providers and provides a unified interface:
|
|
585
|
+
|
|
586
|
+
1. **Input Format Differences**:
|
|
587
|
+
- Each provider uses its own tool definition format
|
|
588
|
+
- AITuber OnAir Core performs appropriate conversions internally and provides a common `ToolDefinition` interface
|
|
589
|
+
|
|
590
|
+
2. **Response Processing Differences**:
|
|
591
|
+
- OpenAI uses `tool_calls` objects
|
|
592
|
+
- Claude uses `tool_use` blocks
|
|
593
|
+
- Gemini uses `functionCall` objects
|
|
594
|
+
- AITuber OnAir Core processes each format and converts to unified `TOOL_USE` events
|
|
595
|
+
|
|
596
|
+
3. **Tool Result Submission Format Differences**:
|
|
597
|
+
- Each provider accepts tool results in different formats
|
|
598
|
+
- AITuber OnAir Core converts and sends in the appropriate format
|
|
599
|
+
|
|
600
|
+
4. **Streaming Processing Differences**:
|
|
601
|
+
- Claude in particular requires special handling for tool calls during streaming
|
|
602
|
+
- AITuber OnAir Core abstracts this and provides a consistent streaming experience across all providers
|
|
603
|
+
|
|
604
|
+
5. **Tool Call Iteration**:
|
|
605
|
+
- The `runToolLoop` method is implemented according to each provider's characteristics, providing consistent tool iteration
|
|
606
|
+
|
|
607
|
+
Through these abstractions, developers can use tool functionality through AITuber OnAir Core's unified interface without worrying about the details of provider implementations. Even when switching providers, there's no need to change tool definition and processing code.
|
|
608
|
+
|
|
609
|
+
## Using MCP
|
|
610
|
+
AITuber OnAir Core allows you to integrate [MCP](https://modelcontextprotocol.io/introduction) using tool calls.
|
|
611
|
+
|
|
612
|
+
Here's an example of integration.
|
|
613
|
+
The following is a simple sample that integrates an `MCP` that returns a random number.
|
|
614
|
+
|
|
615
|
+
```typescript
|
|
616
|
+
// mcpClient.ts
|
|
617
|
+
import { Client as MCPClient } from "@modelcontextprotocol/sdk/client/index.js";
|
|
618
|
+
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";
|
|
619
|
+
|
|
620
|
+
let clientPromise: Promise<MCPClient> | null = null;
|
|
621
|
+
|
|
622
|
+
async function getMcpClient(): Promise<MCPClient> {
|
|
623
|
+
if (clientPromise) return clientPromise;
|
|
624
|
+
|
|
625
|
+
const client = new MCPClient({
|
|
626
|
+
name: "random-int-server",
|
|
627
|
+
version: "0.0.1",
|
|
628
|
+
});
|
|
629
|
+
const endpoint = import.meta.env.VITE_MCP_ENDPOINT as string;
|
|
630
|
+
if (!endpoint) throw new Error("VITE_MCP_ENDPOINT is not defined");
|
|
631
|
+
|
|
632
|
+
const transport = new StreamableHTTPClientTransport(new URL(endpoint));
|
|
633
|
+
clientPromise = client.connect(transport).then(() => client);
|
|
634
|
+
return clientPromise;
|
|
635
|
+
}
|
|
636
|
+
|
|
637
|
+
export function createMcpToolHandler<T extends { [key: string]: unknown } = any>(toolName: string) {
|
|
638
|
+
return async (args: T): Promise<string> => {
|
|
639
|
+
const client = await getMcpClient();
|
|
640
|
+
const out = await client.callTool({ name: toolName, arguments: args });
|
|
641
|
+
return (out.content as { text: string }[] | undefined)?.[0]?.text ?? "";
|
|
642
|
+
};
|
|
643
|
+
}
|
|
644
|
+
```
|
|
645
|
+
|
|
646
|
+
```typescript
|
|
647
|
+
import { createMcpToolHandler } from './mcpClient';
|
|
648
|
+
|
|
649
|
+
// tool definition
|
|
650
|
+
const randomIntTool: ToolDefinition<{ max: number }> = {
|
|
651
|
+
name: 'randomInt',
|
|
652
|
+
description:
|
|
653
|
+
"Return a random integer from 0 (inclusive) up to, but not including, `max`. If `max` is omitted the default upper‑bound is 100.",
|
|
654
|
+
parameters: {
|
|
655
|
+
type: 'object',
|
|
656
|
+
properties: {
|
|
657
|
+
max: { type: 'integer', description: 'Exclusive upper bound for the random integer', minimum: 1 },
|
|
658
|
+
},
|
|
659
|
+
required: ['max'],
|
|
660
|
+
},
|
|
661
|
+
};
|
|
662
|
+
|
|
663
|
+
// mcp tool handler
|
|
664
|
+
const randomIntHandler = createMcpToolHandler<{ max: number }>('randomInt');
|
|
665
|
+
|
|
666
|
+
// create options
|
|
667
|
+
const aituberOptions: AITuberOnAirCoreOptions = {
|
|
668
|
+
chatProvider,
|
|
669
|
+
apiKey: apiKey.trim(),
|
|
670
|
+
model,
|
|
671
|
+
chatOptions: {
|
|
672
|
+
systemPrompt: systemPrompt.trim() || DEFAULT_SYSTEM_PROMPT,
|
|
673
|
+
visionPrompt: visionPrompt.trim() || DEFAULT_VISION_PROMPT,
|
|
674
|
+
},
|
|
675
|
+
tools: [{ definition: randomIntTool, handler: randomIntHandler }],
|
|
676
|
+
debug: true,
|
|
677
|
+
};
|
|
678
|
+
|
|
679
|
+
// create new instance
|
|
680
|
+
const newAITuber = new AITuberOnAirCore(aituberOptions);
|
|
681
|
+
```
|
|
682
|
+
|
|
167
683
|
## Architecture
|
|
168
684
|
|
|
169
685
|
**AITuberOnAirCore** is designed with the following layered structure:
|
|
@@ -298,7 +814,16 @@ await aituberOnairCore.speakTextWithOptions('[happy] Hello, everyone watching!',
|
|
|
298
814
|
- `ASSISTANT_RESPONSE`: Upon receiving a complete response (includes a screenplay object and rawText with emotion tags)
|
|
299
815
|
- `SPEECH_START`: When speech playback starts (includes a screenplay object with emotion and rawText with emotion tags)
|
|
300
816
|
- `SPEECH_END`: When speech playback ends
|
|
817
|
+
- `TOOL_USE`: When the AI calls a tool (includes the name of the tool and its input parameters)
|
|
818
|
+
- `TOOL_RESULT`: When a tool execution completes and returns a result
|
|
301
819
|
- `ERROR`: When an error occurs
|
|
820
|
+
- `CHAT_HISTORY_SET`: When chat history is set
|
|
821
|
+
- `CHAT_HISTORY_CLEARED`: When chat history is cleared
|
|
822
|
+
- `MEMORY_CREATED`: When a new memory is created
|
|
823
|
+
- `MEMORY_REMOVED`: When memory is removed
|
|
824
|
+
- `MEMORY_LOADED`: When memory is loaded from storage
|
|
825
|
+
- `MEMORY_SAVED`: When memory is saved to storage
|
|
826
|
+
- `STORAGE_CLEARED`: When storage is cleared
|
|
302
827
|
|
|
303
828
|
### Safely Handling Event Data
|
|
304
829
|
|
|
@@ -632,13 +1157,55 @@ const aiTuber = new AITuberOnAirCore({
|
|
|
632
1157
|
|
|
633
1158
|
The memory feature triggers the following events:
|
|
634
1159
|
|
|
635
|
-
- `memoriesLoaded`: When memory is loaded from storage
|
|
636
|
-
- `memoryCreated`: When a new memory record is created
|
|
637
|
-
- `memoriesRemoved`: When a memory record is deleted
|
|
638
|
-
- `memoriesSaved`: When memory records are saved to storage
|
|
639
|
-
- `storageCleared`: When the storage is cleared
|
|
1160
|
+
- `memoriesLoaded`: When memory is loaded from storage (corresponds to AITuberOnAirCoreEvent.MEMORY_LOADED)
|
|
1161
|
+
- `memoryCreated`: When a new memory record is created (corresponds to AITuberOnAirCoreEvent.MEMORY_CREATED)
|
|
1162
|
+
- `memoriesRemoved`: When a memory record is deleted (corresponds to AITuberOnAirCoreEvent.MEMORY_REMOVED)
|
|
1163
|
+
- `memoriesSaved`: When memory records are saved to storage (corresponds to AITuberOnAirCoreEvent.MEMORY_SAVED)
|
|
1164
|
+
- `storageCleared`: When the storage is cleared (corresponds to AITuberOnAirCoreEvent.STORAGE_CLEARED)
|
|
1165
|
+
|
|
1166
|
+
These events are emitted by the `MemoryManager` instance internally, but in the latest version, the same events are also emitted by AITuberOnAirCore, so it's recommended to use the AITuberOnAirCore events.
|
|
640
1167
|
|
|
641
|
-
|
|
1168
|
+
#### Using AITuberOnAirCore Events
|
|
1169
|
+
|
|
1170
|
+
AITuberOnAirCore emits memory-related events that you can use directly:
|
|
1171
|
+
|
|
1172
|
+
```typescript
|
|
1173
|
+
// Example of setting up memory and chat history related event listeners
|
|
1174
|
+
aituber.on(AITuberOnAirCoreEvent.CHAT_HISTORY_SET, (messages) => {
|
|
1175
|
+
console.log('Chat history set:', messages.length);
|
|
1176
|
+
// Update UI, etc.
|
|
1177
|
+
});
|
|
1178
|
+
|
|
1179
|
+
aituber.on(AITuberOnAirCoreEvent.CHAT_HISTORY_CLEARED, () => {
|
|
1180
|
+
console.log('Chat history cleared');
|
|
1181
|
+
// Clear UI, etc.
|
|
1182
|
+
});
|
|
1183
|
+
|
|
1184
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_CREATED, (memory) => {
|
|
1185
|
+
console.log(`New memory created: ${memory.type}`);
|
|
1186
|
+
// Display memory creation notification, etc.
|
|
1187
|
+
});
|
|
1188
|
+
|
|
1189
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_REMOVED, (memoryIds) => {
|
|
1190
|
+
console.log('Memory removed:', memoryIds);
|
|
1191
|
+
});
|
|
1192
|
+
|
|
1193
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_LOADED, (memories) => {
|
|
1194
|
+
console.log('Memory loaded:', memories.length);
|
|
1195
|
+
// Display loaded memory information, etc.
|
|
1196
|
+
});
|
|
1197
|
+
|
|
1198
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_SAVED, (memories) => {
|
|
1199
|
+
console.log('Memory saved:', memories.length);
|
|
1200
|
+
// Display save confirmation, etc.
|
|
1201
|
+
});
|
|
1202
|
+
|
|
1203
|
+
aituber.on(AITuberOnAirCoreEvent.STORAGE_CLEARED, () => {
|
|
1204
|
+
console.log('Storage cleared');
|
|
1205
|
+
});
|
|
1206
|
+
```
|
|
1207
|
+
|
|
1208
|
+
You can use these events to reflect memory state changes in your UI, display debug information, or trigger other actions.
|
|
642
1209
|
|
|
643
1210
|
### Memory Cleanup
|
|
644
1211
|
|
|
@@ -839,4 +1406,69 @@ npm install
|
|
|
839
1406
|
|
|
840
1407
|
# Run the test suite
|
|
841
1408
|
npm test
|
|
842
|
-
```
|
|
1409
|
+
```
|
|
1410
|
+
|
|
1411
|
+
## Migration Guide for Memory Events
|
|
1412
|
+
|
|
1413
|
+
### Changes in v0.8.0
|
|
1414
|
+
|
|
1415
|
+
In version 0.8.0, we've unified the event system by forwarding all memory-related events from the internal `MemoryManager` component to the main `AITuberOnAirCore` instance. This creates a more consistent API where all events can be listened to in one place.
|
|
1416
|
+
|
|
1417
|
+
### What Changed
|
|
1418
|
+
|
|
1419
|
+
- Previously: Memory-related events (`memoriesLoaded`, `memoryCreated`, etc.) were only emitted by the internal `MemoryManager` instance, requiring direct access to this component.
|
|
1420
|
+
- Now: These events are also emitted as standardized `AITuberOnAirCoreEvent` enum values from the main `AITuberOnAirCore` instance.
|
|
1421
|
+
|
|
1422
|
+
### How to Migrate
|
|
1423
|
+
|
|
1424
|
+
If you were previously accessing the `MemoryManager` directly to listen for memory events:
|
|
1425
|
+
|
|
1426
|
+
```typescript
|
|
1427
|
+
// Old approach (deprecated)
|
|
1428
|
+
const memoryManager = aituber['memoryManager'];
|
|
1429
|
+
if (memoryManager) {
|
|
1430
|
+
memoryManager.on('memoriesLoaded', (memories) => {
|
|
1431
|
+
console.log('Memories loaded:', memories.length);
|
|
1432
|
+
});
|
|
1433
|
+
}
|
|
1434
|
+
```
|
|
1435
|
+
|
|
1436
|
+
Update your code to use the main AITuberOnAirCore instance:
|
|
1437
|
+
|
|
1438
|
+
```typescript
|
|
1439
|
+
// New approach (recommended)
|
|
1440
|
+
aituber.on(AITuberOnAirCoreEvent.MEMORY_LOADED, (memories) => {
|
|
1441
|
+
console.log('Memories loaded:', memories.length);
|
|
1442
|
+
});
|
|
1443
|
+
```
|
|
1444
|
+
|
|
1445
|
+
### Event Mapping
|
|
1446
|
+
|
|
1447
|
+
Here's how the internal MemoryManager events map to AITuberOnAirCoreEvent values:
|
|
1448
|
+
|
|
1449
|
+
| MemoryManager Event | AITuberOnAirCoreEvent |
|
|
1450
|
+
|----------------------|-------------------------|
|
|
1451
|
+
| `memoriesLoaded` | `MEMORY_LOADED` |
|
|
1452
|
+
| `memoryCreated` | `MEMORY_CREATED` |
|
|
1453
|
+
| `memoriesRemoved` | `MEMORY_REMOVED` |
|
|
1454
|
+
| `memoriesSaved` | `MEMORY_SAVED` |
|
|
1455
|
+
| `storageCleared` | `STORAGE_CLEARED` |
|
|
1456
|
+
|
|
1457
|
+
Additionally, these new events were added:
|
|
1458
|
+
|
|
1459
|
+
| New Event | Description |
|
|
1460
|
+
|---------------------------|--------------------------|
|
|
1461
|
+
| `CHAT_HISTORY_SET` | When chat history is set |
|
|
1462
|
+
| `CHAT_HISTORY_CLEARED` | When chat history is cleared |
|
|
1463
|
+
|
|
1464
|
+
### Benefits of the New Approach
|
|
1465
|
+
|
|
1466
|
+
- **Simplified API**: All events are available through a single entry point
|
|
1467
|
+
- **Type Safety**: Using enum values provides better TypeScript support
|
|
1468
|
+
- **Abstraction**: Internal implementation details are properly hidden
|
|
1469
|
+
- **Consistency**: All events follow the same pattern
|
|
1470
|
+
- **Documentation**: Events are clearly documented with the enum values
|
|
1471
|
+
|
|
1472
|
+
### Will This Break My Code?
|
|
1473
|
+
|
|
1474
|
+
If you were directly accessing the `MemoryManager` instance to subscribe to events, your code will continue to function but is now considered deprecated. We recommend migrating to the new approach for future compatibility.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export declare const ENDPOINT_GEMINI_API = "https://generativelanguage.googleapis.com
|
|
1
|
+
export declare const ENDPOINT_GEMINI_API = "https://generativelanguage.googleapis.com";
|
|
2
2
|
export declare const MODEL_GEMINI_2_0_FLASH = "gemini-2.0-flash";
|
|
3
3
|
export declare const MODEL_GEMINI_2_0_FLASH_LITE = "gemini-2.0-flash-lite";
|
|
4
4
|
export declare const MODEL_GEMINI_1_5_FLASH = "gemini-1.5-flash";
|