@agentica/benchmark 0.12.1 → 0.12.2-dev.20250314

package/README.md

@@ -1,327 +1,327 @@
-# `@agentica/benchmark`
-![agentica-conceptual-diagram](https://github.com/user-attachments/assets/d7ebbd1f-04d3-4b0d-9e2a-234e29dd6c57)
-
-[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/wrtnlabs/agentica/blob/master/LICENSE)
-[![npm version](https://img.shields.io/npm/v/@agentica/benchmark.svg)](https://www.npmjs.com/package/@agentica/benchmark)
-[![Downloads](https://img.shields.io/npm/dm/@agentica/benchmark.svg)](https://www.npmjs.com/package/@agentica/benchmark)
-[![Build Status](https://github.com/wrtnlabs/agentica/workflows/build/badge.svg)](https://github.com/wrtnlabs/agentica/actions?query=workflow%3Abuild)
-
-Benchmark program of `Agentica`.
-
-`Agentica` is the simplest Agentic AI library specialized in **LLM Function Calling**, and `@agentica/benchmark` is the benchmark tool of such Agentic AI library. It supports two quantitive benchmark tools `AgenticaSelectBenchmark` and `AgenticaCallBenchmark` which can measure function calling's selecting and calling qualities.
-
-Here is an example report generated by `@agentica/benchmark` measuring function calling quality of "Shopping Mall" scenario. Below measured benchmark scenario is exactly same with the recorded video, and you can find that every function calling has succeeded without any error.
-
-> https://github.com/user-attachments/assets/01604b53-aca4-41cb-91aa-3faf63549ea6
->
-> Benchmark of Shopping Mall Scenario
->
-> - [Benchmark Report](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/select)
-> - Swagger Document: https://shopping-be.wrtn.ai/editor
-> - Repository: https://github.com/wrtnlabs/shopping-backend
-
-
-
-
-## How to use
-### Setup
-```bash
-npm install @agentica/core @agentica/benchmark @samchon/openapi typia
-npx typia setup
-```
-
-Install `@agentica/benchmark` with its dependent libraries.
-
-Note that, you have to install not only `@agentica/core` or `@agentica/benchmark` libraries, but also [`@samchon/openapi`](https://github.com/samchon/openapi) and [`typia`](https://github.com/samchon/typia) too. 
-
-`@samchon/openapi` is an OpenAPI specification library which can convert Swagger/OpenAPI document to LLM function calling schema. And `typia` is a transformer (compiler) library which can compose LLM function calling schema from a TypeScript class type.
-
-By the way, as `typia` is a transformer library analyzing TypeScript source code in the compilation level, it needs additional setup command `npx typia setup`.
-
-### Function selecting Benchmark
-```typescript
-import { AgenticaSelectBenchmark } from "@agentica/benchmark";
-import { Agentica, IAgenticaOperation } from "@agentica/core";
-import { HttpLlm, IHttpConnection, OpenApi } from "@samchon/openapi";
-import fs from "fs";
-import OpenAI from "openai";
-import path from "path";
-
-const main = async (): Promise<void> => {
-  // CREATE AI AGENT
-  const agent: Agentica<"chatgpt"> = new Agentica({
-    model: "chatgpt",
-    vendor: {
-      api: new OpenAI({
-        apiKey: "YOUR_OPENAI_API_KEY",
-      }),
-      model: "gpt-4o-mini",
-    },
-    controllers: [
-      {
-        protocol: "http",
-        name: "shopping",
-        application: HttpLlm.application({
-          model: "chatgpt",
-          document: await fetch(
-            "https://shopping-be.wrtn.ai/editor/swagger.json",
-          ).then((res) => res.json()),
-        }),
-        connection: {
-          host: "https://shopping-be.wrtn.ai",
-        },
-      },
-    ],
-  });
-
-  // DO BENCHMARK
-  const find = (method: OpenApi.Method, path: string): IAgenticaOperation => {
-    const found = agent
-      .getOperations()
-      .find(
-        (op) =>
-          op.protocol === "http" &&
-          op.function.method === method &&
-          op.function.path === path,
-      );
-    if (!found) throw new Error(`Operation not found: ${method} ${path}`);
-    return found;
-  };
-  const benchmark: AgenticaSelectBenchmark<"chatgpt"> = 
-    new AgenticaSelectBenchmark({
-      agent,
-      config: {
-        repeat: 4,
-      },
-      scenarios: [
-        {
-          name: "order",
-          text: [
-            "I wanna see every sales in the shopping mall",
-            "",
-            "And then show me the detailed information about the Macbook.",
-            "",
-            "After that, select the most expensive stock",
-            "from the Macbook, and put it into my shopping cart.",
-            "And take the shopping cart to the order.",
-            "",
-            "At last, I'll publish it by cash payment, and my address is",
-            "",
-            "  - country: South Korea",
-            "  - city/province: Seoul",
-            "  - department: Wrtn Apartment",
-            "  - Possession: 101-1411",
-          ].join("\n"),
-          expected: {
-            type: "array",
-            items: [
-              {
-                type: "standalone",
-                operation: find("patch", "/shoppings/customers/sales"),
-              },
-              {
-                type: "standalone",
-                operation: find("get", "/shoppings/customers/sales/{id}"),
-              },
-              {
-                type: "anyOf",
-                anyOf: [
-                  {
-                    type: "standalone",
-                    operation: find("post", "/shoppings/customers/orders"),
-                  },
-                  {
-                    type: "standalone",
-                    operation: find("post", "/shoppings/customers/orders/direct"),
-                  },
-                ],
-              },
-              {
-                type: "standalone",
-                operation: find(
-                  "post",
-                  "/shoppings/customers/orders/{orderId}/publish",
-                ),
-              },
-            ],
-          },
-        },
-      ],
-    });
-  await benchmark.execute();
-
-  // REPORT
-  const docs: Record<string, string> = benchmark.report();
-  const root: string = `docs/benchmarks/call`;
-
-  await rmdir(root);
-  for (const [key, value] of Object.entries(docs)) {
-    await mkdir(path.join(root, key.split("/").slice(0, -1).join("/")));
-    await fs.promises.writeFile(path.join(root, key), value, "utf8");
-  }
-};
-```
-
-> Benchmark of Shopping Mall Scenario
->
-> - [Benchmark Report](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/select)
-> - Swagger Document: https://shopping-be.wrtn.ai/editor
-> - Repository: https://github.com/wrtnlabs/shopping-backend
-
-Benchmark function selecting quality.
-
-You can measure a benchmark that AI agent can select proper functions from the user's conversations by the LLM (Large Language Model) function calling feature. Create `Agentica` and `AgenticaSelectBenchmark` typed instances, and execute the benchmark with your specific scenarios like above.
-
-If you have written enough and proper descriptions to the functions (or API operations) and DTO schema types, success ratio of `AgenticaSelectBenchmark` would be higher. Otherwise descriptions are not enough or have bad quality, you may get a threatening benchmark report. If you wanna see how the `AgenticaSelectBenchmark` reports, click above [benchmark report link](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/select) please.
-
-### Function Calling Benchmark
-> Benchmark of Shopping Mall Scenario
->
-> - [Benchmark Report](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/call)
-> - Swagger Document: https://shopping-be.wrtn.ai/editor
-> - Repository: https://github.com/wrtnlabs/shopping-backend
-
-```typescript
-import { AgenticaCallBenchmark } from "@agentica/benchmark";
-import { Agentica, IAgenticaOperation } from "@agentica/core";
-import { HttpLlm, IHttpConnection, OpenApi } from "@samchon/openapi";
-import fs from "fs";
-import OpenAI from "openai";
-import path from "path";
-
-const main = async (): Promise<void> => {
-  // CREATE AI AGENT
-  const agent: Agentica<"chatgpt"> = new Agentica({
-    model: "chatgpt",
-    vendor: {
-      api: new OpenAI({
-        apiKey: "YOUR_OPENAI_API_KEY",
-      }),
-      model: "gpt-4o-mini",
-    },
-    controllers: [
-      {
-        protocol: "http",
-        name: "shopping",
-        application: HttpLlm.application({
-          model: "chatgpt",
-          document: await fetch(
-            "https://shopping-be.wrtn.ai/editor/swagger.json",
-          ).then((res) => res.json()),
-        }),
-        connection: {
-          host: "https://shopping-be.wrtn.ai",
-        },
-      },
-    ],
-  });
-
-  // DO BENCHMARK
-  const find = (method: OpenApi.Method, path: string): IAgenticaOperation => {
-    const found = agent
-      .getOperations()
-      .find(
-        (op) =>
-          op.protocol === "http" &&
-          op.function.method === method &&
-          op.function.path === path,
-      );
-    if (!found) throw new Error(`Operation not found: ${method} ${path}`);
-    return found;
-  };
-  const benchmark: AgenticaSelectBenchmark<"chatgpt"> = 
-    new AgenticaSelectBenchmark({
-      agent,
-      config: {
-        repeat: 4,
-      },
-      scenarios: [
-        {
-          name: "order",
-          text: [
-            "I wanna see every sales in the shopping mall",
-            "",
-            "And then show me the detailed information about the Macbook.",
-            "",
-            "After that, select the most expensive stock",
-            "from the Macbook, and put it into my shopping cart.",
-            "And take the shopping cart to the order.",
-            "",
-            "At last, I'll publish it by cash payment, and my address is",
-            "",
-            "  - country: South Korea",
-            "  - city/province: Seoul",
-            "  - department: Wrtn Apartment",
-            "  - Possession: 101-1411",
-          ].join("\n"),
-          expected: {
-            type: "array",
-            items: [
-              {
-                type: "standalone",
-                operation: find("patch", "/shoppings/customers/sales"),
-              },
-              {
-                type: "standalone",
-                operation: find("get", "/shoppings/customers/sales/{id}"),
-              },
-              {
-                type: "anyOf",
-                anyOf: [
-                  {
-                    type: "standalone",
-                    operation: find("post", "/shoppings/customers/orders"),
-                  },
-                  {
-                    type: "standalone",
-                    operation: find("post", "/shoppings/customers/orders/direct"),
-                  },
-                ],
-              },
-              {
-                type: "standalone",
-                operation: find(
-                  "post",
-                  "/shoppings/customers/orders/{orderId}/publish",
-                ),
-              },
-            ],
-          },
-        },
-      ],
-    });
-  await benchmark.execute();
-
-  // REPORT
-  const docs: Record<string, string> = benchmark.report();
-  const root: string = `docs/benchmarks/call`;
-
-  await rmdir(root);
-  for (const [key, value] of Object.entries(docs)) {
-    await mkdir(path.join(root, key.split("/").slice(0, -1).join("/")));
-    await fs.promises.writeFile(path.join(root, key), value, "utf8");
-  }
-};
-```
-
-Benchmark function calling quality.
-
-You can measure a benchmark that AI agent can call proper functions from the user's conversations by the LLM (Large Language Model) function calling feature. Create `Agentica` and `AgenticaCallBenchmark` typed instances, and execute the benchmark with your specific scenarios like above.
-
-If you have written enough and proper descriptions to the functions (or API operations) and DTO schema types, success ratio of `AgenticaCallBenchmark` would be higher. Otherwise descriptions are not enough or have bad quality, you may get a threatening benchmark report. If you wanna see how the `AgenticaCallBenchmark` reports, click above [benchmark report link](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/call) please.
-
-For reference, `@agentica/core` tends not to failed on arguments filling of LLM function calling. So it is okay that ending up with [`AgenticaSelectBenchmark`](#function-selecting-benchmark) stage, because function calling with arguments filling spends much more times and LLM tokens. 
-
-Also, current `AgenticaCallBenchmark` has been designed to perform multiple LLM function callings just by one conversation text. However, the multiple LLM function calling benchmark actually requires the [#Multi Turn Benchmark](#multi-turn-benchmark) feature of [#Roadmap](#rodmap). Therefore, [`AgenticaSelectBenchmark`](#function-selecting-benchmark) is economic than `AgenticaCallBenchmark`.
-
-> In the above "Shopping Mall" scenario, function selecting benchmark ends in 4 seconds, but function calling benchmark consumes about 3 minutes.
-
-
-
-
-## Roadmap
-### Multi Turn Benchmark
-Will support multi-turn benchmark for [#Function Calling Benchmark](#function-calling-benchmark).
-
-### Estimator Agent
+# `@agentica/benchmark`
+![agentica-conceptual-diagram](https://github.com/user-attachments/assets/d7ebbd1f-04d3-4b0d-9e2a-234e29dd6c57)
+
+[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/wrtnlabs/agentica/blob/master/LICENSE)
+[![npm version](https://img.shields.io/npm/v/@agentica/benchmark.svg)](https://www.npmjs.com/package/@agentica/benchmark)
+[![Downloads](https://img.shields.io/npm/dm/@agentica/benchmark.svg)](https://www.npmjs.com/package/@agentica/benchmark)
+[![Build Status](https://github.com/wrtnlabs/agentica/workflows/build/badge.svg)](https://github.com/wrtnlabs/agentica/actions?query=workflow%3Abuild)
+
+Benchmark program of `Agentica`.
+
+`Agentica` is the simplest Agentic AI library specialized in **LLM Function Calling**, and `@agentica/benchmark` is the benchmark tool of such Agentic AI library. It supports two quantitive benchmark tools `AgenticaSelectBenchmark` and `AgenticaCallBenchmark` which can measure function calling's selecting and calling qualities.
+
+Here is an example report generated by `@agentica/benchmark` measuring function calling quality of "Shopping Mall" scenario. Below measured benchmark scenario is exactly same with the recorded video, and you can find that every function calling has succeeded without any error.
+
+> https://github.com/user-attachments/assets/01604b53-aca4-41cb-91aa-3faf63549ea6
+>
+> Benchmark of Shopping Mall Scenario
+>
+> - [Benchmark Report](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/select)
+> - Swagger Document: https://shopping-be.wrtn.ai/editor
+> - Repository: https://github.com/wrtnlabs/shopping-backend
+
+
+
+
+## How to use
+### Setup
+```bash
+npm install @agentica/core @agentica/benchmark @samchon/openapi typia
+npx typia setup
+```
+
+Install `@agentica/benchmark` with its dependent libraries.
+
+Note that, you have to install not only `@agentica/core` or `@agentica/benchmark` libraries, but also [`@samchon/openapi`](https://github.com/samchon/openapi) and [`typia`](https://github.com/samchon/typia) too. 
+
+`@samchon/openapi` is an OpenAPI specification library which can convert Swagger/OpenAPI document to LLM function calling schema. And `typia` is a transformer (compiler) library which can compose LLM function calling schema from a TypeScript class type.
+
+By the way, as `typia` is a transformer library analyzing TypeScript source code in the compilation level, it needs additional setup command `npx typia setup`.
+
+### Function selecting Benchmark
+```typescript
+import { AgenticaSelectBenchmark } from "@agentica/benchmark";
+import { Agentica, IAgenticaOperation } from "@agentica/core";
+import { HttpLlm, IHttpConnection, OpenApi } from "@samchon/openapi";
+import fs from "fs";
+import OpenAI from "openai";
+import path from "path";
+
+const main = async (): Promise<void> => {
+  // CREATE AI AGENT
+  const agent: Agentica<"chatgpt"> = new Agentica({
+    model: "chatgpt",
+    vendor: {
+      api: new OpenAI({
+        apiKey: "YOUR_OPENAI_API_KEY",
+      }),
+      model: "gpt-4o-mini",
+    },
+    controllers: [
+      {
+        protocol: "http",
+        name: "shopping",
+        application: HttpLlm.application({
+          model: "chatgpt",
+          document: await fetch(
+            "https://shopping-be.wrtn.ai/editor/swagger.json",
+          ).then((res) => res.json()),
+        }),
+        connection: {
+          host: "https://shopping-be.wrtn.ai",
+        },
+      },
+    ],
+  });
+
+  // DO BENCHMARK
+  const find = (method: OpenApi.Method, path: string): IAgenticaOperation => {
+    const found = agent
+      .getOperations()
+      .find(
+        (op) =>
+          op.protocol === "http" &&
+          op.function.method === method &&
+          op.function.path === path,
+      );
+    if (!found) throw new Error(`Operation not found: ${method} ${path}`);
+    return found;
+  };
+  const benchmark: AgenticaSelectBenchmark<"chatgpt"> = 
+    new AgenticaSelectBenchmark({
+      agent,
+      config: {
+        repeat: 4,
+      },
+      scenarios: [
+        {
+          name: "order",
+          text: [
+            "I wanna see every sales in the shopping mall",
+            "",
+            "And then show me the detailed information about the Macbook.",
+            "",
+            "After that, select the most expensive stock",
+            "from the Macbook, and put it into my shopping cart.",
+            "And take the shopping cart to the order.",
+            "",
+            "At last, I'll publish it by cash payment, and my address is",
+            "",
+            "  - country: South Korea",
+            "  - city/province: Seoul",
+            "  - department: Wrtn Apartment",
+            "  - Possession: 101-1411",
+          ].join("\n"),
+          expected: {
+            type: "array",
+            items: [
+              {
+                type: "standalone",
+                operation: find("patch", "/shoppings/customers/sales"),
+              },
+              {
+                type: "standalone",
+                operation: find("get", "/shoppings/customers/sales/{id}"),
+              },
+              {
+                type: "anyOf",
+                anyOf: [
+                  {
+                    type: "standalone",
+                    operation: find("post", "/shoppings/customers/orders"),
+                  },
+                  {
+                    type: "standalone",
+                    operation: find("post", "/shoppings/customers/orders/direct"),
+                  },
+                ],
+              },
+              {
+                type: "standalone",
+                operation: find(
+                  "post",
+                  "/shoppings/customers/orders/{orderId}/publish",
+                ),
+              },
+            ],
+          },
+        },
+      ],
+    });
+  await benchmark.execute();
+
+  // REPORT
+  const docs: Record<string, string> = benchmark.report();
+  const root: string = `docs/benchmarks/call`;
+
+  await rmdir(root);
+  for (const [key, value] of Object.entries(docs)) {
+    await mkdir(path.join(root, key.split("/").slice(0, -1).join("/")));
+    await fs.promises.writeFile(path.join(root, key), value, "utf8");
+  }
+};
+```
+
+> Benchmark of Shopping Mall Scenario
+>
+> - [Benchmark Report](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/select)
+> - Swagger Document: https://shopping-be.wrtn.ai/editor
+> - Repository: https://github.com/wrtnlabs/shopping-backend
+
+Benchmark function selecting quality.
+
+You can measure a benchmark that AI agent can select proper functions from the user's conversations by the LLM (Large Language Model) function calling feature. Create `Agentica` and `AgenticaSelectBenchmark` typed instances, and execute the benchmark with your specific scenarios like above.
+
+If you have written enough and proper descriptions to the functions (or API operations) and DTO schema types, success ratio of `AgenticaSelectBenchmark` would be higher. Otherwise descriptions are not enough or have bad quality, you may get a threatening benchmark report. If you wanna see how the `AgenticaSelectBenchmark` reports, click above [benchmark report link](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/select) please.
+
+### Function Calling Benchmark
+> Benchmark of Shopping Mall Scenario
+>
+> - [Benchmark Report](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/call)
+> - Swagger Document: https://shopping-be.wrtn.ai/editor
+> - Repository: https://github.com/wrtnlabs/shopping-backend
+
+```typescript
+import { AgenticaCallBenchmark } from "@agentica/benchmark";
+import { Agentica, IAgenticaOperation } from "@agentica/core";
+import { HttpLlm, IHttpConnection, OpenApi } from "@samchon/openapi";
+import fs from "fs";
+import OpenAI from "openai";
+import path from "path";
+
+const main = async (): Promise<void> => {
+  // CREATE AI AGENT
+  const agent: Agentica<"chatgpt"> = new Agentica({
+    model: "chatgpt",
+    vendor: {
+      api: new OpenAI({
+        apiKey: "YOUR_OPENAI_API_KEY",
+      }),
+      model: "gpt-4o-mini",
+    },
+    controllers: [
+      {
+        protocol: "http",
+        name: "shopping",
+        application: HttpLlm.application({
+          model: "chatgpt",
+          document: await fetch(
+            "https://shopping-be.wrtn.ai/editor/swagger.json",
+          ).then((res) => res.json()),
+        }),
+        connection: {
+          host: "https://shopping-be.wrtn.ai",
+        },
+      },
+    ],
+  });
+
+  // DO BENCHMARK
+  const find = (method: OpenApi.Method, path: string): IAgenticaOperation => {
+    const found = agent
+      .getOperations()
+      .find(
+        (op) =>
+          op.protocol === "http" &&
+          op.function.method === method &&
+          op.function.path === path,
+      );
+    if (!found) throw new Error(`Operation not found: ${method} ${path}`);
+    return found;
+  };
+  const benchmark: AgenticaSelectBenchmark<"chatgpt"> = 
+    new AgenticaSelectBenchmark({
+      agent,
+      config: {
+        repeat: 4,
+      },
+      scenarios: [
+        {
+          name: "order",
+          text: [
+            "I wanna see every sales in the shopping mall",
+            "",
+            "And then show me the detailed information about the Macbook.",
+            "",
+            "After that, select the most expensive stock",
+            "from the Macbook, and put it into my shopping cart.",
+            "And take the shopping cart to the order.",
+            "",
+            "At last, I'll publish it by cash payment, and my address is",
+            "",
+            "  - country: South Korea",
+            "  - city/province: Seoul",
+            "  - department: Wrtn Apartment",
+            "  - Possession: 101-1411",
+          ].join("\n"),
+          expected: {
+            type: "array",
+            items: [
+              {
+                type: "standalone",
+                operation: find("patch", "/shoppings/customers/sales"),
+              },
+              {
+                type: "standalone",
+                operation: find("get", "/shoppings/customers/sales/{id}"),
+              },
+              {
+                type: "anyOf",
+                anyOf: [
+                  {
+                    type: "standalone",
+                    operation: find("post", "/shoppings/customers/orders"),
+                  },
+                  {
+                    type: "standalone",
+                    operation: find("post", "/shoppings/customers/orders/direct"),
+                  },
+                ],
+              },
+              {
+                type: "standalone",
+                operation: find(
+                  "post",
+                  "/shoppings/customers/orders/{orderId}/publish",
+                ),
+              },
+            ],
+          },
+        },
+      ],
+    });
+  await benchmark.execute();
+
+  // REPORT
+  const docs: Record<string, string> = benchmark.report();
+  const root: string = `docs/benchmarks/call`;
+
+  await rmdir(root);
+  for (const [key, value] of Object.entries(docs)) {
+    await mkdir(path.join(root, key.split("/").slice(0, -1).join("/")));
+    await fs.promises.writeFile(path.join(root, key), value, "utf8");
+  }
+};
+```
+
+Benchmark function calling quality.
+
+You can measure a benchmark that AI agent can call proper functions from the user's conversations by the LLM (Large Language Model) function calling feature. Create `Agentica` and `AgenticaCallBenchmark` typed instances, and execute the benchmark with your specific scenarios like above.
+
+If you have written enough and proper descriptions to the functions (or API operations) and DTO schema types, success ratio of `AgenticaCallBenchmark` would be higher. Otherwise descriptions are not enough or have bad quality, you may get a threatening benchmark report. If you wanna see how the `AgenticaCallBenchmark` reports, click above [benchmark report link](https://github.com/wrtnlabs/agentica/tree/main/test/examples/benchmarks/call) please.
+
+For reference, `@agentica/core` tends not to failed on arguments filling of LLM function calling. So it is okay that ending up with [`AgenticaSelectBenchmark`](#function-selecting-benchmark) stage, because function calling with arguments filling spends much more times and LLM tokens. 
+
+Also, current `AgenticaCallBenchmark` has been designed to perform multiple LLM function callings just by one conversation text. However, the multiple LLM function calling benchmark actually requires the [#Multi Turn Benchmark](#multi-turn-benchmark) feature of [#Roadmap](#rodmap). Therefore, [`AgenticaSelectBenchmark`](#function-selecting-benchmark) is economic than `AgenticaCallBenchmark`.
+
+> In the above "Shopping Mall" scenario, function selecting benchmark ends in 4 seconds, but function calling benchmark consumes about 3 minutes.
+
+
+
+
+## Roadmap
+### Multi Turn Benchmark
+Will support multi-turn benchmark for [#Function Calling Benchmark](#function-calling-benchmark).
+
+### Estimator Agent
 We will create some benchmark features that can analyze conversation context and issue summary reports or provide quantitative evaluations.