@lobehub/chat 1.2.12 → 1.2.14

package/CHANGELOG.md

@@ -2,6 +2,56 @@
 
 # Changelog
 
+### [Version 1.2.14](https://github.com/lobehub/lobe-chat/compare/v1.2.13...v1.2.14)
+
+<sup>Released on **2024-07-08**</sup>
+
+#### 💄 Styles
+
+- **misc**: Provider changes with model in model settings.
+
+<br/>
+
+<details>
+<summary><kbd>Improvements and Fixes</kbd></summary>
+
+#### Styles
+
+- **misc**: Provider changes with model in model settings, closes [#3146](https://github.com/lobehub/lobe-chat/issues/3146) ([e53bb5a](https://github.com/lobehub/lobe-chat/commit/e53bb5a))
+
+</details>
+
+<div align="right">
+
+[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)
+
+</div>
+
+### [Version 1.2.13](https://github.com/lobehub/lobe-chat/compare/v1.2.12...v1.2.13)
+
+<sup>Released on **2024-07-07**</sup>
+
+#### 🐛 Bug Fixes
+
+- **misc**: Fix tool message order.
+
+<br/>
+
+<details>
+<summary><kbd>Improvements and Fixes</kbd></summary>
+
+#### What's fixed
+
+- **misc**: Fix tool message order, closes [#3155](https://github.com/lobehub/lobe-chat/issues/3155) ([6171b2a](https://github.com/lobehub/lobe-chat/commit/6171b2a))
+
+</details>
+
+<div align="right">
+
+[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)
+
+</div>
+
 ### [Version 1.2.12](https://github.com/lobehub/lobe-chat/compare/v1.2.11...v1.2.12)
 
 <sup>Released on **2024-07-07**</sup>

package/docs/usage/tools-calling/anthropic.mdx

@@ -1 +1,185 @@
-TODO
+---
+title: Anthropic Claude 系列 Tools Calling 评测
+description: >-
+  使用 LobeChat 测试 Anthropic Claude 系列模型（Claude 3.5 sonnet / Claude 3 Opus /
+  Claude 3 haiku） 的工具调用（Function Calling）能力，并展现评测结果
+tags:
+  - Tools Calling
+  - Benchmark
+  - Function Calling 评测
+  - 工具调用
+  - 插件
+---
+
+# Anthropic Claude Series Tools Calling
+
+Overview of Anthropic Claude Series model Tools Calling capabilities:
+
+| Model | Support Tools Calling | Stream | Parallel | Simple Instruction Score | Complex Instruction |
+| --- | --- | --- | --- | --- | --- |
+| Claude 3.5 Sonnet | ✅ | ✅ | ✅ | 🌟🌟🌟 | 🌟🌟 |
+| Claude 3 Opus | ✅ | ✅ | ❌ | 🌟 | ⛔️ |
+| Claude 3 Sonnet | ✅ | ✅ | ❌ | 🌟🌟 | ⛔️ |
+| Claude 3 Haiku | ✅ | ✅ | ❌ | 🌟🌟 | ⛔️ |
+
+## Claude 3.5 Sonnet
+
+### Simple Instruction Call: Weather Query
+
+Test Instruction: Instruction ①
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/42a6980c-ea2a-44fd-b61f-a7989827f5a5" />
+
+<Image
+  alt="Claude 3.5 Sonnet Tools Calling for Simple Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/71146b75-2c73-48c3-9688-1d8814d2a791"
+/>
+
+<details>
+  <summary>Tools Calling Raw Output:</summary>
+
+```yml
+
+```
+
+</details>
+
+### Complex Instruction Call: Literary Map
+
+Test Instruction: Instruction ②
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/a9a40899-d5f3-4ef2-aa08-922751b05ca6" />
+
+From the above video:
+
+1. Sonnet 3.5 supports Stream Tools Calling and Parallel Tools Calling;
+2. In Stream Tools Calling, it is observed that creating long sentences will cause a delay (as seen in the Tools Calling raw output `[chunk 40]` and `[chunk 41]` with a delay of 6s). Therefore, there will be a relatively long waiting time at the beginning stage of Tools Calling.
+
+<Image
+  alt="Claude 3.5 Sonnet Tools Calling for Complex Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/23e2d7e5-a6f3-4f4c-9c6a-5651f35a5910"
+/>
+
+<details>
+  <summary>Tools Calling Raw Output:</summary>
+
+```yml
+
+```
+
+</details>
+
+## Claude 3 Opus
+
+### Simple Instruction Call: Weather Query
+
+Test Instruction: Instruction ①
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/0e120fa2-8410-4552-a947-5ab7a91d994d" />
+
+From the above video:
+
+1. Claude 3 Opus outputs a `<thinking>` tag at the beginning of Tools Calling, which is not very helpful for users and consumes more tokens;
+2. Opus triggers Tools Calling twice, indicating that it does not support Parallel Tools Calling;
+3. The raw output of Tools Calling shows that Opus also supports Stream Tools Calling.
+
+<Image
+  alt="Claude 3 Opus Tools Calling for Simple Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/fa2f89bc-b9d5-43e3-a15e-1e79174d002c"
+/>
+
+<details>
+  <summary>Tools Calling Raw Output:</summary>
+
+</details>
+
+### Complex Instruction Call: Literary Map
+
+Test Instruction: Instruction ②
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/b2dc8cd9-2582-43fe-9121-29c20a1cdc7b" />
+
+From the above video:
+
+1. Combining with simple tasks, Opus will always output a `<thinking>` tag, which significantly impacts the user experience;
+2. Opus outputs the prompts field as a string instead of an array, causing an error and preventing the plugin from being called correctly.
+
+<Image
+  alt="Claude 3 Opus Tools Calling for Complex Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/1eee785d-932f-4320-845e-eed0bee4b1ae"
+/>
+
+<details>
+  <summary>Tools Calling Raw Output:</summary>
+
+</details>
+
+## Claude 3 Sonnet
+
+### Simple Instruction Call: Weather Query
+
+Test Instruction: Instruction ①
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/600becd5-7f12-4a9a-86c7-e5cca0db6b1b" />
+
+From the above video, it can be seen that Claude 3 Sonnet triggers Tools Calling twice, indicating that it does not support Parallel Tools Calling.
+
+<Image
+  alt="Claude 3 Sonnet Tools Calling for Simple Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/e82f5c69-7607-488f-8c10-0482fb380c6c"
+/>
+
+<details>
+  <summary>Tools Calling Raw Output:</summary>
+
+</details>
+
+### Complex Instruction Call: Literary Map
+
+Test Instruction: Instruction ②
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/c150aa5f-36bc-40f2-a779-9c4fdcf2cd4c" />
+
+From the above video, it can be seen that Sonnet 3 fails in the complex instruction call. The error is due to prompts being expected as an array but generated as a string.
+
+<Image
+  alt="Claude 3.5 Sonnet Tools Calling for Complex Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/b7d84e26-920d-4a82-8798-1b1060ebb341"
+/>
+
+<details>
+  <summary>Tools Calling Raw Output:</summary>
+
+</details>
+
+## Claude 3 Haiku
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/02b3e872-735a-4928-8245-a90786acea8b" />
+
+From the above video:
+
+1. Claude 3 Haiku triggers Tools Calling twice, indicating that it also does not support Parallel Tools Calling;
+2. Haiku does not provide a good response and directly calls the tool;
+
+<Image
+  alt="Claude 3 Haiku Tools Calling for Simple Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/9081b586-cf43-440f-8ef8-1de5d8658694"
+/>
+
+### Complex Instruction Call: Literary Map
+
+Test Instruction: Instruction ②
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/d1e3f804-0b89-4b90-9d78-69aee0db1c4d" />
+
+From the above video, it can be seen that Haiku 3 also fails in the complex instruction call. The error is the same as prompts generating a string instead of an array.
+
+<Image
+  alt="Claude 3 Haiku Tools Calling for Complex Instruction"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/cde80220-4615-43bb-934f-35fe0de88754"
+/>
+
+<details>
+  <summary>Tools Calling Raw Output:</summary>
+
+</details>

package/docs/usage/tools-calling/anthropic.zh-CN.mdx

@@ -13,6 +13,15 @@ tags:
 
 # Anthropic Claude 系列 Tools Calling
 
+Anthropic Claude 系列模型 Tools Calling 能力一览：
+
+| 模型 | 支持 Tools Calling | 流式 （Stream） | 并发（Parallel） | 简单指令得分 | 复杂指令 |
+| --- | --- | --- | --- | --- | --- |
+| Claude 3.5 Sonnet | ✅ | ✅ | ✅ | 🌟🌟🌟 | 🌟🌟 |
+| Claude 3 Opus | ✅ | ✅ | ❌ | 🌟 | ⛔️ |
+| Claude 3 Sonnet | ✅ | ✅ | ❌ | 🌟🌟 | ⛔️ |
+| Claude 3 Haiku | ✅ | ✅ | ❌ | 🌟🌟 | ⛔️ |
+
 ## Claude 3.5 Sonnet
 
 ### 简单调用指令：天气查询
@@ -42,6 +51,7 @@ tags:
 <Video src="https://github.com/lobehub/lobe-chat/assets/28616219/a9a40899-d5f3-4ef2-aa08-922751b05ca6" />
 
 从上述视频中可以看到：
+
 1.  Sonnet 3.5 支持流式 Tools Calling 和 Parallel Tools Calling；
 2.  在流式 Tools Calling 时，表现出来的特征是在创建长句会等待住（详见 Tools Calling 原始输出 `[chunk 40]` 和 `[chunk 41]` 中间的耗时达到 6s）。所以相对来说会在 Tools Calling 的起始阶段有一个较长的等待时间。
 
@@ -65,11 +75,11 @@ tags:
 
 测试指令：指令 ①
 
-
 <Video src="https://github.com/lobehub/lobe-chat/assets/28616219/0e120fa2-8410-4552-a947-5ab7a91d994d" />
 
 从上述视频中看到：
-1. Claude 3 Opus 在调用 Tools 的起点会输出一段 <thinking> 标签的内容，这段内容对于用户来说几乎没有什么帮助，反而带来了较多的 Token 消耗；
+
+1. Claude 3 Opus 在调用 Tools 的起点会输出一段 `<thinking>` 标签的内容，这段内容对于用户来说几乎没有什么帮助，反而带来了较多的 Token 消耗；
 2. Opus 会触发两次 Tools Calling，说明它并不支持 Parallel Tools Calling；
 3. 从 Tools Calling 的原始输出来看， Opus 也是支持流式 Tools Calling 的
 
@@ -78,15 +88,11 @@ tags:
   src="https://github.com/lobehub/lobe-chat/assets/28616219/fa2f89bc-b9d5-43e3-a15e-1e79174d002c"
 />
 
-
-
   <details>
     <summary>Tools Calling 原始输出：</summary>
 
-
 </details>
 
-
 ### 复杂调用指令：文生图
 
 测试指令：指令 ②
@@ -94,7 +100,8 @@ tags:
 <Video src="https://github.com/lobehub/lobe-chat/assets/28616219/b2dc8cd9-2582-43fe-9121-29c20a1cdc7b" />
 
 从上述视频中看到：
-1. 结合简单任务， Opus 的工具调用一定会输出 <thinking> 标签，这其实对体验影响非常大
+
+1. 结合简单任务， Opus 的工具调用一定会输出 `<thinking>` 标签，这其实对体验影响非常大
 2. Opus 输出的 prompts 字段是字符串，而不是数组，导致报错，无法正常调用插件。
 
 <Image
@@ -105,7 +112,6 @@ tags:
 <details>
   <summary>Tools Calling 原始输出：</summary>
 
-
 </details>
 
 ## Claude 3 Sonnet
@@ -123,18 +129,15 @@ tags:
   src="https://github.com/lobehub/lobe-chat/assets/28616219/e82f5c69-7607-488f-8c10-0482fb380c6c"
 />
 
-
 <details>
   <summary>Tools Calling 原始输出：</summary>
 
-
 </details>
 
 ### 复杂调用指令：文生图
 
 测试指令：指令 ②
 
-
 <Video src="https://github.com/lobehub/lobe-chat/assets/28616219/c150aa5f-36bc-40f2-a779-9c4fdcf2cd4c" />
 
 从上述视频中可以看到， Sonnet 3 在复杂指令调用下就失败了。报错原因是 prompts 原本预期为一个数组，但是生成的却是一个字符串。
@@ -147,13 +150,10 @@ tags:
 <details>
   <summary>Tools Calling 原始输出：</summary>
 
-
-
 </details>
 
 ## Claude 3 Haiku
 
-
 <Video src="https://github.com/lobehub/lobe-chat/assets/28616219/02b3e872-735a-4928-8245-a90786acea8b" />
 
 从上述视频中可以看出：
@@ -161,18 +161,15 @@ tags:
 1. Claude 3 Haiku 会调用两次 Tools Calling，说明它也不支持 Parallel Tools Calling；
 2. Haiku 并没有回答好的，也是直接调用的工具；
 
-
 <Image
   alt="Claude 3 Haiku 简单指令的 Tools Calling"
   src="https://github.com/lobehub/lobe-chat/assets/28616219/9081b586-cf43-440f-8ef8-1de5d8658694"
 />
 
-
 ### 复杂调用指令：文生图
 
 测试指令：指令 ②
 
-
 <Video src="https://github.com/lobehub/lobe-chat/assets/28616219/d1e3f804-0b89-4b90-9d78-69aee0db1c4d" />
 
 从上述视频中可以看到， Haiku 3 在复杂指令调用下也是失败的。报错原因同样是 prompts 生成了字符串而不是数组。
@@ -185,6 +182,4 @@ tags:
 <details>
   <summary>Tools Calling 原始输出：</summary>
 
-
-
 </details>

package/docs/usage/tools-calling/google.mdx

@@ -1 +1,116 @@
-TODO
+---
+title: Google Gemini 系列 Tool Calling  评测
+description: >-
+  使用 LobeChat 测试 Google Gemini 系列模型（Gemini 1.5 Pro / Gemini 1.5 Flash）
+  的工具调用（Function Calling）能力，并展现评测结果
+tags:
+  - Tools Calling
+  - Benchmark
+  - Function Calling 评测
+  - 工具调用
+  - 插件
+---
+
+# Google Gemini Series Tool Calling
+
+Overview of Google Gemini series model Tools Calling capabilities:
+
+| Model | Tools Calling Support | Streaming | Parallel | Simple Instruction Score | Complex Instruction |
+| --- | --- | --- | --- | --- | --- |
+| Gemini 1.5 Pro | ✅ | ❌ | ✅ | ⛔ | ⛔ |
+| Gemini 1.5 Flash | ❌ | ❌ | ❌ | ⛔ | ⛔ |
+
+<Callout type={'important'}>
+  Based on our actual tests, we strongly recommend not enabling plugins for Gemini because as of
+  July 7, 2024, its Tools Calling capability is extremely poor.
+</Callout>
+
+## Gemini 1.5 Pro
+
+### Simple Instruction Call: Weather Query
+
+Test Instruction: Instruction ①
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/a5a35431-2a15-4e79-97d5-502637f829bc" />
+
+In the json output from Gemini, the name is incorrect, so LobeChat cannot recognize which plugin it called. (In the input, the name of the weather plugin is `realtime-weather____fetchCurrentWeather`, while Gemini returns `weather____fetchCurrentWeather`).
+
+<Image
+  alt="Tools Calling for Simple Instruction in Gemini 1.5 Pro"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/1e077799-c25e-43c7-8492-c5c0bb9aed9b"
+/>
+
+<details>
+  <summary>Original Tools Calling Output:</summary>
+
+```yml
+[stream start] 2024-7-7 17:53:25.647
+[chunk 0] 2024-7-7 17:53:25.654
+{"candidates":[{"content":{"parts":[{"text":"好的"}],"role":"model"},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":95,"candidatesTokenCount":1,"totalTokenCount":96}}
+
+[chunk 1] 2024-7-7 17:53:26.288
+{"candidates":[{"content":{"parts":[{"text":"\n\n"}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":95,"candidatesTokenCount":1,"totalTokenCount":96}}
+
+[chunk 2] 2024-7-7 17:53:26.336
+{"candidates":[{"content":{"parts":[{"functionCall":{"name":"weather____fetchCurrentWeather","args":{"city":"Hangzhou"}}},{"functionCall":{"name":"weather____fetchCurrentWeather","args":{"city":"Beijing"}}}],"role":"model"},"finishReasoSTOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":95,"candidatesTokenCount":79,"totalTokenCount":174}}
+
+[stream finished] total chunks: 3
+```
+
+</details>
+
+### Complex Instruction Call: Image Generation
+
+Test Instruction: Instruction ②
+
+<Image
+  alt="Tools Calling for Complex Instruction in Gemini 1.5 Pro"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/a2454a60-3271-4786-861f-d49ceac1316e"
+/>
+
+When testing a set of complex instructions, Google throws an error directly:
+
+```json
+{
+  "message": "[400 Bad Request] Invalid JSON payload received. Unknown name \"maxItems\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"minItems\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[3].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[4].value': Cannot find field. [{\"@type\":\"type.googleapis.com/google.rpc.BadRequest\",\"fieldViolations\":[{\"field\":\"tools[0].function_declarations[0].parameters.properties[0].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"maxItems\\\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[0].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"minItems\\\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[1].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"default\\\" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[3].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"default\\\" at 'tools[0].function_declarations[0].parameters.properties[3].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[4].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"default\\\" at 'tools[0].function_declarations[0].parameters.properties[4].value': Cannot find field.\"}]}]"
+}
+```
+
+The error above mentions that it does not support a schema containing `maxItems`, so Gemini 1.5 Pro is essentially unable to use the DallE plugin.
+
+Related issues:
+
+- [Support for minItems and maxItems for FunctionDeclarationSchemaType.ARRAY?](https://github.com/google-gemini/generative-ai-js/issues/200)
+- [Gemini Models unusable when dalle plugin is enabled](https://github.com/lobehub/lobe-chat/issues/2537)
+
+Based on the above two tests, Google's Tool Calling capability seems to be supported, but it is almost unusable in daily use. I personally think it is equivalent to false advertising.
+
+## Gemini 1.5 Flash
+
+### Simple Command: Weather Query
+
+Test Command: Command ①
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/6cab77e8-d761-4a91-8325-a61748cebac1" />
+
+Gemini 1.5 Flash is more abstract, and the call ends as soon as it is made. Combining the original output below, it can be seen that Gemini 1.5 Flash does not output Tool Calling data, so it can be considered completely unusable.
+
+```yml
+stream start] 2024-7-7 19:4:50.936
+[chunk 0] 2024-7-7 19:4:50.943
+{"candidates":[{"content":{"parts":[{"text":"Okay"}],"role":"model"},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":96,"candidatesTokenCount":1,"totalTokenCount":97}}
+
+[chunk 1] 2024-7-7 19:4:52.209
+{"candidates":[{"content":{"parts":[{"text":", please wait, I am checking the weather information for Hangzhou and Beijing."}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":96,"candidatesTokenCount":16,"totalTokenCount":112}}
+
+[chunk 2] 2024-7-7 19:4:53.288
+{"candidates":[{"content":{"parts":[{"text":"\n"}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":96,"candidatesTokenCount":16,"totalTokenCount":112}}
+
+[stream finished] total chunks: 3
+```
+
+### Complex Command: Wenshengtu
+
+Test Command: Command ②
+
+This command, like the complex commands of Gemini 1.5 Pro, throws an error directly, so it will not be further elaborated.

package/docs/usage/tools-calling/google.zh-CN.mdx

@@ -1,13 +1,114 @@
 ---
-title: Google Gemini 系列 Tool Calling
+title: Google Gemini 系列 Tool Calling  评测
+description: 使用 LobeChat 测试 Google Gemini 系列模型（Gemini 1.5 Pro / Gemini 1.5 Flash）的工具调用（Function Calling）能力，并展现评测结果
+tags:
+  - Tools Calling
+  - Benchmark
+  - Function Calling 评测
+  - 工具调用
+  - 插件
 ---
 
 # Google Gemini 系列 Tool Calling
 
+Google Gemini 系列模型 Tools Calling 能力一览：
+
+| 模型 | 支持 Tools Calling | 流式 （Stream） | 并发（Parallel） | 简单指令得分 | 复杂指令 |
+| --- | --- | --- | --- | --- | --- |
+| Gemini 1.5 Pro | ✅ | ❌ | ✅ | ⛔ | ⛔ |
+| Gemini 1.5 Flash | ❌ | ❌ | ❌ | ⛔ | ⛔ |
+
+<Callout type={'important'}>
+  根据我们的的实际测试，强烈建议不要给 Gemini 开启插件，因为目前（截止2024.07.07）它的 Tools Calling
+  能力实在太烂了。
+</Callout>
+
 ## Gemini 1.5 Pro
 
-TODO
+### 简单调用指令：天气查询
+
+测试指令：指令 ①
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/a5a35431-2a15-4e79-97d5-502637f829bc" />
+
+Gemini 输出的 json 中，name 是错误的，因此 LobeChat 无法识别到它调用了什么插件。（入参中，天气插件的 name 为 `realtime-weather____fetchCurrentWeather`，而 Gemini 返回的是 `weather____fetchCurrentWeather`）。
+
+<Image
+  alt="Gemini 1.5 Pro 简单指令的 Tools Calling"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/1e077799-c25e-43c7-8492-c5c0bb9aed9b"
+/>
+
+<details>
+  <summary>Tools Calling 原始输出：</summary>
+
+```yml
+[stream start] 2024-7-7 17:53:25.647
+[chunk 0] 2024-7-7 17:53:25.654
+{"candidates":[{"content":{"parts":[{"text":"好的"}],"role":"model"},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":95,"candidatesTokenCount":1,"totalTokenCount":96}}
+
+[chunk 1] 2024-7-7 17:53:26.288
+{"candidates":[{"content":{"parts":[{"text":"\n\n"}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":95,"candidatesTokenCount":1,"totalTokenCount":96}}
+
+[chunk 2] 2024-7-7 17:53:26.336
+{"candidates":[{"content":{"parts":[{"functionCall":{"name":"weather____fetchCurrentWeather","args":{"city":"杭州"}}},{"functionCall":{"name":"weather____fetchCurrentWeather","args":{"city":"北京"}}}],"role":"model"},"finishReasoSTOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":95,"candidatesTokenCount":79,"totalTokenCount":174}}
+
+[stream finished] total chunks: 3
+```
+
+</details>
+
+### 复杂调用指令：文生图
+
+测试指令：指令 ②
+
+<Image
+  alt="Gemini 1.5 Pro 复杂指令的 Tools Calling"
+  src="https://github.com/lobehub/lobe-chat/assets/28616219/a2454a60-3271-4786-861f-d49ceac1316e"
+/>
+
+在测试复杂指令集时，Google 直接抛错：
+
+```json
+{
+  "message": "[400 Bad Request] Invalid JSON payload received. Unknown name \"maxItems\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"minItems\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[3].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[4].value': Cannot find field. [{\"@type\":\"type.googleapis.com/google.rpc.BadRequest\",\"fieldViolations\":[{\"field\":\"tools[0].function_declarations[0].parameters.properties[0].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"maxItems\\\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[0].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"minItems\\\" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[1].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"default\\\" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[3].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"default\\\" at 'tools[0].function_declarations[0].parameters.properties[3].value': Cannot find field.\"},{\"field\":\"tools[0].function_declarations[0].parameters.properties[4].value\",\"description\":\"Invalid JSON payload received. Unknown name \\\"default\\\" at 'tools[0].function_declarations[0].parameters.properties[4].value': Cannot find field.\"}]}]"
+}
+```
+
+上述抛错中提到并不支持包含 `maxItems` 的 schema，因此 Gemini 1.5 Pro 相当于无法使用 DallE 插件。
+
+相关 issue:
+
+- [Support for minItems and maxItems for FunctionDeclarationSchemaType.ARRAY?](https://github.com/google-gemini/generative-ai-js/issues/200)
+- [Gemini Models unusable when dalle plugin is enabled](https://github.com/lobehub/lobe-chat/issues/2537)
+
+综合以上两个测试来看，Google 的 Tool Calling 能力似乎是支持了，但是几乎没法在日常中使用，我个人认为已经等于虚假宣传了。
 
 ## Gemini 1.5 Flash
 
-TODO
+### 简单调用指令：天气查询
+
+测试指令：指令 ①
+
+<Video src="https://github.com/lobehub/lobe-chat/assets/28616219/6cab77e8-d761-4a91-8325-a61748cebac1" />
+
+而 Gemini 1.5 flash 更为抽象，说完调用就结束了。结合以下原始输出可以看到，Gemini 1.5 Flash 并没有输出 Tool Calling 的数据，因此可以说是完全不可用。
+
+```yml
+stream start] 2024-7-7 19:4:50.936
+[chunk 0] 2024-7-7 19:4:50.943
+{"candidates":[{"content":{"parts":[{"text":"好的"}],"role":"model"},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":96,"candidatesTokenCount":1,"totalTokenCount":97}}
+
+[chunk 1] 2024-7-7 19:4:52.209
+{"candidates":[{"content":{"parts":[{"text":"，请稍等，我正在查询杭州和北京的天气信息。 "}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"ATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":96,"candidatesTokenCount":16,"totalTokenCount":112}}
+
+[chunk 2] 2024-7-7 19:4:53.288
+{"candidates":[{"content":{"parts":[{"text":"\n"}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":96,"candidatesTokenCount":16,"totalTokenCount":112}}
+
+[stream finished] total chunks: 3
+```
+
+### 复杂调用指令：文生图
+
+测试指令：指令 ②
+
+该指令和 Gemini 1.5 Pro 的复杂指令一样，直接抛错，因此不再详细展开。

package/docs/usage/tools-calling/moonshot.mdx

@@ -0,0 +1 @@
+TODO

package/docs/usage/tools-calling/moonshot.zh-CN.mdx

@@ -0,0 +1,24 @@
+---
+title: Moonshot 系列 Tools Calling 评测
+description: 使用 LobeChat 测试 Moonshot 系列模型（Moonshot-1） 的工具调用（Function Calling）能力，并展现评测结果
+tags:
+  - Tools Calling
+  - Benchmark
+  - Function Calling
+  - 工具调用
+  - 插件
+---
+
+# Moonshot 系列工具调用（Tools Calling）
+
+### 简单调用指令：天气查询
+
+测试指令：指令 ①
+
+TODO
+
+### 复杂调用指令：文生图
+
+测试指令：指令 ②
+
+TODO