@azure/ai-agents 1.0.0-alpha.20250502.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +907 -0
- package/dist/browser/agentsClient.d.ts +110 -0
- package/dist/browser/agentsClient.d.ts.map +1 -0
- package/dist/browser/agentsClient.js +203 -0
- package/dist/browser/agentsClient.js.map +1 -0
- package/dist/browser/api/agentsContext.d.ts +15 -0
- package/dist/browser/api/agentsContext.d.ts.map +1 -0
- package/dist/browser/api/agentsContext.js +55 -0
- package/dist/browser/api/agentsContext.js.map +1 -0
- package/dist/browser/api/index.d.ts +4 -0
- package/dist/browser/api/index.d.ts.map +1 -0
- package/dist/browser/api/index.js +5 -0
- package/dist/browser/api/index.js.map +1 -0
- package/dist/browser/api/operations.d.ts +212 -0
- package/dist/browser/api/operations.d.ts.map +1 -0
- package/dist/browser/api/operations.js +1348 -0
- package/dist/browser/api/operations.js.map +1 -0
- package/dist/browser/api/options.d.ts +456 -0
- package/dist/browser/api/options.d.ts.map +1 -0
- package/dist/browser/api/options.js +4 -0
- package/dist/browser/api/options.js.map +1 -0
- package/dist/browser/api/poller.d.ts +9 -0
- package/dist/browser/api/poller.d.ts.map +1 -0
- package/dist/browser/api/poller.js +137 -0
- package/dist/browser/api/poller.js.map +1 -0
- package/dist/browser/constants.d.ts +9 -0
- package/dist/browser/constants.d.ts.map +1 -0
- package/dist/browser/constants.js +11 -0
- package/dist/browser/constants.js.map +1 -0
- package/dist/browser/index.d.ts +8 -0
- package/dist/browser/index.d.ts.map +1 -0
- package/dist/browser/index.js +7 -0
- package/dist/browser/index.js.map +1 -0
- package/dist/browser/logger.d.ts +2 -0
- package/dist/browser/logger.d.ts.map +1 -0
- package/dist/browser/logger.js +5 -0
- package/dist/browser/logger.js.map +1 -0
- package/dist/browser/models/index.d.ts +3 -0
- package/dist/browser/models/index.d.ts.map +1 -0
- package/dist/browser/models/index.js +5 -0
- package/dist/browser/models/index.js.map +1 -0
- package/dist/browser/models/models.d.ts +1977 -0
- package/dist/browser/models/models.d.ts.map +1 -0
- package/dist/browser/models/models.js +2053 -0
- package/dist/browser/models/models.js.map +1 -0
- package/dist/browser/models/streamingModels.d.ts +108 -0
- package/dist/browser/models/streamingModels.d.ts.map +1 -0
- package/dist/browser/models/streamingModels.js +75 -0
- package/dist/browser/models/streamingModels.js.map +1 -0
- package/dist/browser/package.json +3 -0
- package/dist/browser/static-helpers/multipartHelpers.d.ts +6 -0
- package/dist/browser/static-helpers/multipartHelpers.d.ts.map +1 -0
- package/dist/browser/static-helpers/multipartHelpers.js +23 -0
- package/dist/browser/static-helpers/multipartHelpers.js.map +1 -0
- package/dist/browser/static-helpers/urlTemplate.d.ts +5 -0
- package/dist/browser/static-helpers/urlTemplate.d.ts.map +1 -0
- package/dist/browser/static-helpers/urlTemplate.js +169 -0
- package/dist/browser/static-helpers/urlTemplate.js.map +1 -0
- package/dist/browser/utils.d.ts +214 -0
- package/dist/browser/utils.d.ts.map +1 -0
- package/dist/browser/utils.js +311 -0
- package/dist/browser/utils.js.map +1 -0
- package/dist/commonjs/agentsClient.d.ts +110 -0
- package/dist/commonjs/agentsClient.d.ts.map +1 -0
- package/dist/commonjs/agentsClient.js +207 -0
- package/dist/commonjs/agentsClient.js.map +1 -0
- package/dist/commonjs/api/agentsContext.d.ts +15 -0
- package/dist/commonjs/api/agentsContext.d.ts.map +1 -0
- package/dist/commonjs/api/agentsContext.js +58 -0
- package/dist/commonjs/api/agentsContext.js.map +1 -0
- package/dist/commonjs/api/index.d.ts +4 -0
- package/dist/commonjs/api/index.d.ts.map +1 -0
- package/dist/commonjs/api/index.js +51 -0
- package/dist/commonjs/api/index.js.map +1 -0
- package/dist/commonjs/api/operations.d.ts +212 -0
- package/dist/commonjs/api/operations.d.ts.map +1 -0
- package/dist/commonjs/api/operations.js +1484 -0
- package/dist/commonjs/api/operations.js.map +1 -0
- package/dist/commonjs/api/options.d.ts +456 -0
- package/dist/commonjs/api/options.d.ts.map +1 -0
- package/dist/commonjs/api/options.js +5 -0
- package/dist/commonjs/api/options.js.map +1 -0
- package/dist/commonjs/api/poller.d.ts +9 -0
- package/dist/commonjs/api/poller.d.ts.map +1 -0
- package/dist/commonjs/api/poller.js +140 -0
- package/dist/commonjs/api/poller.js.map +1 -0
- package/dist/commonjs/constants.d.ts +9 -0
- package/dist/commonjs/constants.d.ts.map +1 -0
- package/dist/commonjs/constants.js +14 -0
- package/dist/commonjs/constants.js.map +1 -0
- package/dist/commonjs/index.d.ts +8 -0
- package/dist/commonjs/index.d.ts.map +1 -0
- package/dist/commonjs/index.js +13 -0
- package/dist/commonjs/index.js.map +1 -0
- package/dist/commonjs/logger.d.ts +2 -0
- package/dist/commonjs/logger.d.ts.map +1 -0
- package/dist/commonjs/logger.js +8 -0
- package/dist/commonjs/logger.js.map +1 -0
- package/dist/commonjs/models/index.d.ts +3 -0
- package/dist/commonjs/models/index.d.ts.map +1 -0
- package/dist/commonjs/models/index.js +10 -0
- package/dist/commonjs/models/index.js.map +1 -0
- package/dist/commonjs/models/models.d.ts +1977 -0
- package/dist/commonjs/models/models.d.ts.map +1 -0
- package/dist/commonjs/models/models.js +2340 -0
- package/dist/commonjs/models/models.js.map +1 -0
- package/dist/commonjs/models/streamingModels.d.ts +108 -0
- package/dist/commonjs/models/streamingModels.d.ts.map +1 -0
- package/dist/commonjs/models/streamingModels.js +78 -0
- package/dist/commonjs/models/streamingModels.js.map +1 -0
- package/dist/commonjs/package.json +3 -0
- package/dist/commonjs/static-helpers/multipartHelpers.d.ts +6 -0
- package/dist/commonjs/static-helpers/multipartHelpers.d.ts.map +1 -0
- package/dist/commonjs/static-helpers/multipartHelpers.js +26 -0
- package/dist/commonjs/static-helpers/multipartHelpers.js.map +1 -0
- package/dist/commonjs/static-helpers/urlTemplate.d.ts +5 -0
- package/dist/commonjs/static-helpers/urlTemplate.d.ts.map +1 -0
- package/dist/commonjs/static-helpers/urlTemplate.js +172 -0
- package/dist/commonjs/static-helpers/urlTemplate.js.map +1 -0
- package/dist/commonjs/tsdoc-metadata.json +11 -0
- package/dist/commonjs/utils.d.ts +214 -0
- package/dist/commonjs/utils.d.ts.map +1 -0
- package/dist/commonjs/utils.js +317 -0
- package/dist/commonjs/utils.js.map +1 -0
- package/dist/esm/agentsClient.d.ts +110 -0
- package/dist/esm/agentsClient.d.ts.map +1 -0
- package/dist/esm/agentsClient.js +203 -0
- package/dist/esm/agentsClient.js.map +1 -0
- package/dist/esm/api/agentsContext.d.ts +15 -0
- package/dist/esm/api/agentsContext.d.ts.map +1 -0
- package/dist/esm/api/agentsContext.js +55 -0
- package/dist/esm/api/agentsContext.js.map +1 -0
- package/dist/esm/api/index.d.ts +4 -0
- package/dist/esm/api/index.d.ts.map +1 -0
- package/dist/esm/api/index.js +5 -0
- package/dist/esm/api/index.js.map +1 -0
- package/dist/esm/api/operations.d.ts +212 -0
- package/dist/esm/api/operations.d.ts.map +1 -0
- package/dist/esm/api/operations.js +1348 -0
- package/dist/esm/api/operations.js.map +1 -0
- package/dist/esm/api/options.d.ts +456 -0
- package/dist/esm/api/options.d.ts.map +1 -0
- package/dist/esm/api/options.js +4 -0
- package/dist/esm/api/options.js.map +1 -0
- package/dist/esm/api/poller.d.ts +9 -0
- package/dist/esm/api/poller.d.ts.map +1 -0
- package/dist/esm/api/poller.js +137 -0
- package/dist/esm/api/poller.js.map +1 -0
- package/dist/esm/constants.d.ts +9 -0
- package/dist/esm/constants.d.ts.map +1 -0
- package/dist/esm/constants.js +11 -0
- package/dist/esm/constants.js.map +1 -0
- package/dist/esm/index.d.ts +8 -0
- package/dist/esm/index.d.ts.map +1 -0
- package/dist/esm/index.js +7 -0
- package/dist/esm/index.js.map +1 -0
- package/dist/esm/logger.d.ts +2 -0
- package/dist/esm/logger.d.ts.map +1 -0
- package/dist/esm/logger.js +5 -0
- package/dist/esm/logger.js.map +1 -0
- package/dist/esm/models/index.d.ts +3 -0
- package/dist/esm/models/index.d.ts.map +1 -0
- package/dist/esm/models/index.js +5 -0
- package/dist/esm/models/index.js.map +1 -0
- package/dist/esm/models/models.d.ts +1977 -0
- package/dist/esm/models/models.d.ts.map +1 -0
- package/dist/esm/models/models.js +2053 -0
- package/dist/esm/models/models.js.map +1 -0
- package/dist/esm/models/streamingModels.d.ts +108 -0
- package/dist/esm/models/streamingModels.d.ts.map +1 -0
- package/dist/esm/models/streamingModels.js +75 -0
- package/dist/esm/models/streamingModels.js.map +1 -0
- package/dist/esm/package.json +3 -0
- package/dist/esm/static-helpers/multipartHelpers.d.ts +6 -0
- package/dist/esm/static-helpers/multipartHelpers.d.ts.map +1 -0
- package/dist/esm/static-helpers/multipartHelpers.js +23 -0
- package/dist/esm/static-helpers/multipartHelpers.js.map +1 -0
- package/dist/esm/static-helpers/urlTemplate.d.ts +5 -0
- package/dist/esm/static-helpers/urlTemplate.d.ts.map +1 -0
- package/dist/esm/static-helpers/urlTemplate.js +169 -0
- package/dist/esm/static-helpers/urlTemplate.js.map +1 -0
- package/dist/esm/utils.d.ts +214 -0
- package/dist/esm/utils.d.ts.map +1 -0
- package/dist/esm/utils.js +311 -0
- package/dist/esm/utils.js.map +1 -0
- package/dist/react-native/agentsClient.d.ts +110 -0
- package/dist/react-native/agentsClient.d.ts.map +1 -0
- package/dist/react-native/agentsClient.js +203 -0
- package/dist/react-native/agentsClient.js.map +1 -0
- package/dist/react-native/api/agentsContext.d.ts +15 -0
- package/dist/react-native/api/agentsContext.d.ts.map +1 -0
- package/dist/react-native/api/agentsContext.js +55 -0
- package/dist/react-native/api/agentsContext.js.map +1 -0
- package/dist/react-native/api/index.d.ts +4 -0
- package/dist/react-native/api/index.d.ts.map +1 -0
- package/dist/react-native/api/index.js +5 -0
- package/dist/react-native/api/index.js.map +1 -0
- package/dist/react-native/api/operations.d.ts +212 -0
- package/dist/react-native/api/operations.d.ts.map +1 -0
- package/dist/react-native/api/operations.js +1348 -0
- package/dist/react-native/api/operations.js.map +1 -0
- package/dist/react-native/api/options.d.ts +456 -0
- package/dist/react-native/api/options.d.ts.map +1 -0
- package/dist/react-native/api/options.js +4 -0
- package/dist/react-native/api/options.js.map +1 -0
- package/dist/react-native/api/poller.d.ts +9 -0
- package/dist/react-native/api/poller.d.ts.map +1 -0
- package/dist/react-native/api/poller.js +137 -0
- package/dist/react-native/api/poller.js.map +1 -0
- package/dist/react-native/constants.d.ts +9 -0
- package/dist/react-native/constants.d.ts.map +1 -0
- package/dist/react-native/constants.js +11 -0
- package/dist/react-native/constants.js.map +1 -0
- package/dist/react-native/index.d.ts +8 -0
- package/dist/react-native/index.d.ts.map +1 -0
- package/dist/react-native/index.js +7 -0
- package/dist/react-native/index.js.map +1 -0
- package/dist/react-native/logger.d.ts +2 -0
- package/dist/react-native/logger.d.ts.map +1 -0
- package/dist/react-native/logger.js +5 -0
- package/dist/react-native/logger.js.map +1 -0
- package/dist/react-native/models/index.d.ts +3 -0
- package/dist/react-native/models/index.d.ts.map +1 -0
- package/dist/react-native/models/index.js +5 -0
- package/dist/react-native/models/index.js.map +1 -0
- package/dist/react-native/models/models.d.ts +1977 -0
- package/dist/react-native/models/models.d.ts.map +1 -0
- package/dist/react-native/models/models.js +2053 -0
- package/dist/react-native/models/models.js.map +1 -0
- package/dist/react-native/models/streamingModels.d.ts +108 -0
- package/dist/react-native/models/streamingModels.d.ts.map +1 -0
- package/dist/react-native/models/streamingModels.js +75 -0
- package/dist/react-native/models/streamingModels.js.map +1 -0
- package/dist/react-native/package.json +3 -0
- package/dist/react-native/static-helpers/multipartHelpers.d.ts +6 -0
- package/dist/react-native/static-helpers/multipartHelpers.d.ts.map +1 -0
- package/dist/react-native/static-helpers/multipartHelpers.js +23 -0
- package/dist/react-native/static-helpers/multipartHelpers.js.map +1 -0
- package/dist/react-native/static-helpers/urlTemplate.d.ts +5 -0
- package/dist/react-native/static-helpers/urlTemplate.d.ts.map +1 -0
- package/dist/react-native/static-helpers/urlTemplate.js +169 -0
- package/dist/react-native/static-helpers/urlTemplate.js.map +1 -0
- package/dist/react-native/utils.d.ts +214 -0
- package/dist/react-native/utils.d.ts.map +1 -0
- package/dist/react-native/utils.js +311 -0
- package/dist/react-native/utils.js.map +1 -0
- package/package.json +159 -0
package/README.md
ADDED
|
@@ -0,0 +1,907 @@
|
|
|
1
|
+
# Azure AI Projects client library for JavaScript
|
|
2
|
+
|
|
3
|
+
Use the AI Projects client library (in preview) to:
|
|
4
|
+
|
|
5
|
+
- **Enumerate connections** in your Azure AI Foundry project and get connection properties.
|
|
6
|
+
For example, get the inference endpoint URL and credentials associated with your Azure OpenAI connection.
|
|
7
|
+
- **Develop Agents using the Azure AI Agent Service**, leveraging an extensive ecosystem of models, tools, and capabilities from OpenAI, Microsoft, and other LLM providers. The Azure AI Agent Service enables the building of Agents for a wide range of generative AI use cases. The package is currently in private preview.
|
|
8
|
+
- **Enable OpenTelemetry tracing**.
|
|
9
|
+
|
|
10
|
+
[Product documentation](https://aka.ms/azsdk/azure-ai-projects/product-doc)
|
|
11
|
+
|
|
12
|
+
| [Samples](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-projects/samples)
|
|
13
|
+
| [Package (npm)](https://www.npmjs.com/package/@azure/ai-projects)
|
|
14
|
+
| [API reference documentation](https://learn.microsoft.com/javascript/api/overview/azure/ai-projects-readme?view=azure-node-preview)
|
|
15
|
+
|
|
16
|
+
## Table of contents
|
|
17
|
+
|
|
18
|
+
- [Getting started](#getting-started)
|
|
19
|
+
- [Prerequisite](#prerequisite)
|
|
20
|
+
- [Install the package](#install-the-package)
|
|
21
|
+
- [Key concepts](#key-concepts)
|
|
22
|
+
- [Create and authenticate the client](#create-and-authenticate-the-client)
|
|
23
|
+
- [Examples](#examples)
|
|
24
|
+
- [Enumerate connections](#enumerate-connections)
|
|
25
|
+
- [Get properties of all connections](#get-properties-of-all-connections)
|
|
26
|
+
- [Get properties of all connections of a particular type](#get-properties-of-all-connections-of-a-particular-type)
|
|
27
|
+
- [Get properties of a default connection](#get-properties-of-a-default-connection)
|
|
28
|
+
- [Get properties of a connection by its connection name](#get-properties-of-a-connection-by-its-connection-name)
|
|
29
|
+
- [Agents (Preview)](#agents-private-preview)
|
|
30
|
+
- [Create an Agent](#create-agent) with:
|
|
31
|
+
- [File Search](#create-agent-with-file-search)
|
|
32
|
+
- [Code interpreter](#create-agent-with-code-interpreter)
|
|
33
|
+
- [Bing grounding](#create-agent-with-bing-grounding)
|
|
34
|
+
- [Azure AI Search](#create-agent-with-azure-ai-search)
|
|
35
|
+
- [Function call](#create-agent-with-function-call)
|
|
36
|
+
- [Fabric Data](#create-an-agent-with-fabric)
|
|
37
|
+
- [Create thread](#create-thread) with
|
|
38
|
+
- [Tool resource](#create-thread-with-tool-resource)
|
|
39
|
+
- [Create message](#create-message) with:
|
|
40
|
+
- [File search attachment](#create-message-with-file-search-attachment)
|
|
41
|
+
- [Code interpreter attachment](#create-message-with-code-interpreter-attachment)
|
|
42
|
+
- [Execute Run, Create Thread and Run, or Stream](#create-run-run_and_process-or-stream)
|
|
43
|
+
- [Retrieve message](#retrieve-message)
|
|
44
|
+
- [Retrieve file](#retrieve-file)
|
|
45
|
+
- [Tear down by deleting resource](#teardown)
|
|
46
|
+
- [Tracing](#tracing)
|
|
47
|
+
- [Tracing](#tracing)
|
|
48
|
+
- [Installation](#installation)
|
|
49
|
+
- [Tracing example](#tracing-example)
|
|
50
|
+
- [Troubleshooting](#troubleshooting)
|
|
51
|
+
- [Exceptions](#exceptions)
|
|
52
|
+
- [Reporting issues](#reporting-issues)
|
|
53
|
+
- [Next steps](#next-steps)
|
|
54
|
+
- [Contributing](#contributing)
|
|
55
|
+
|
|
56
|
+
## Getting started
|
|
57
|
+
|
|
58
|
+
### Prerequisite
|
|
59
|
+
|
|
60
|
+
- [LTS versions of Node.js](https://github.com/nodejs/release#release-schedule)
|
|
61
|
+
- An [Azure subscription][azure_sub].
|
|
62
|
+
- A [project in Azure AI Foundry](https://learn.microsoft.com/azure/ai-studio/how-to/create-projects?tabs=ai-studio).
|
|
63
|
+
- The project connection string. It can be found in your Azure AI Foundry project overview page, under "Project details". Below we will assume the environment variable `PROJECT_ENDPOINT` was defined to hold this value.
|
|
64
|
+
- Entra ID is needed to authenticate the client. Your application needs an object that implements the [TokenCredential](https://learn.microsoft.com/javascript/api/@azure/core-auth/tokencredential) interface. Code samples here use [DefaultAzureCredential](https://learn.microsoft.com/javascript/api/@azure/identity/defaultazurecredential?view=azure-node-latest). To get that working, you will need:
|
|
65
|
+
- The `Contributor` role. Role assigned can be done via the "Access Control (IAM)" tab of your Azure AI Project resource in the Azure portal.
|
|
66
|
+
- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) installed.
|
|
67
|
+
- You are logged into your Azure account by running `az login`.
|
|
68
|
+
- Note that if you have multiple Azure subscriptions, the subscription that contains your Azure AI Project resource must be your default subscription. Run `az account list --output table` to list all your subscription and see which one is the default. Run `az account set --subscription "Your Subscription ID or Name"` to change your default subscription.
|
|
69
|
+
|
|
70
|
+
### Install the package
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
npm install @azure/ai-agents @azure/identity
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Key concepts
|
|
77
|
+
|
|
78
|
+
### Create and authenticate the client
|
|
79
|
+
|
|
80
|
+
The class factory method `fromConnectionString` is used to construct the client. To construct a client:
|
|
81
|
+
|
|
82
|
+
```ts snippet:setup
|
|
83
|
+
import { AIProjectsClient } from "@azure/ai-projects";
|
|
84
|
+
import { DefaultAzureCredential } from "@azure/identity";
|
|
85
|
+
|
|
86
|
+
const connectionString = process.env.PROJECT_ENDPOINT ?? "<connectionString>";
|
|
87
|
+
const client = AIProjectsClient.fromConnectionString(
|
|
88
|
+
connectionString,
|
|
89
|
+
new DefaultAzureCredential(),
|
|
90
|
+
);
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
## Examples
|
|
94
|
+
|
|
95
|
+
### Agents (Preview)
|
|
96
|
+
|
|
97
|
+
Agents in the Azure AI Projects client library are designed to facilitate various interactions and operations within your AI projects. They serve as the core components that manage and execute tasks, leveraging different tools and resources to achieve specific goals. The following steps outline the typical sequence for interacting with Agents. See the "agents" folder in the [package samples](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-projects/samples) for additional Agent samples.
|
|
98
|
+
|
|
99
|
+
Agents are actively being developed. A sign-up form for private preview is coming soon.
|
|
100
|
+
|
|
101
|
+
#### Create Agent
|
|
102
|
+
|
|
103
|
+
Here is an example of how to create an Agent:
|
|
104
|
+
|
|
105
|
+
```ts snippet:createAgent
|
|
106
|
+
const agent = await client.agents.createAgent("gpt-4o", {
|
|
107
|
+
name: "my-agent",
|
|
108
|
+
instructions: "You are a helpful assistant",
|
|
109
|
+
});
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
To allow Agents to access your resources or custom functions, you need tools. You can pass tools to `createAgent` through the `tools` and `toolResources` arguments.
|
|
113
|
+
|
|
114
|
+
You can use `ToolSet` to do this:
|
|
115
|
+
|
|
116
|
+
```ts snippet:toolSet
|
|
117
|
+
import { ToolSet } from "@azure/ai-projects";
|
|
118
|
+
|
|
119
|
+
const toolSet = new ToolSet();
|
|
120
|
+
toolSet.addFileSearchTool([vectorStore.id]);
|
|
121
|
+
toolSet.addCodeInterpreterTool([codeInterpreterFile.id]);
|
|
122
|
+
const agent = await client.agents.createAgent("gpt-4o", {
|
|
123
|
+
name: "my-agent",
|
|
124
|
+
instructions: "You are a helpful agent",
|
|
125
|
+
tools: toolSet.toolDefinitions,
|
|
126
|
+
toolResources: toolSet.toolResources,
|
|
127
|
+
});
|
|
128
|
+
console.log(`Created agent, agent ID: ${agent.id}`);
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
#### Create Agent with File Search
|
|
132
|
+
|
|
133
|
+
To perform file search by an Agent, we first need to upload a file, create a vector store, and associate the file to the vector store. Here is an example:
|
|
134
|
+
|
|
135
|
+
```ts snippet:fileSearch
|
|
136
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
137
|
+
|
|
138
|
+
const localFileStream = fs.createReadStream(filePath);
|
|
139
|
+
const file = await client.agents.uploadFile(localFileStream, "assistants", {
|
|
140
|
+
fileName: "sample_file_for_upload.txt",
|
|
141
|
+
});
|
|
142
|
+
console.log(`Uploaded file, ID: ${file.id}`);
|
|
143
|
+
const vectorStore = await client.agents.createVectorStore({
|
|
144
|
+
fileIds: [file.id],
|
|
145
|
+
name: "my_vector_store",
|
|
146
|
+
});
|
|
147
|
+
console.log(`Created vector store, ID: ${vectorStore.id}`);
|
|
148
|
+
const fileSearchTool = ToolUtility.createFileSearchTool([vectorStore.id]);
|
|
149
|
+
const agent = await client.agents.createAgent("gpt-4o", {
|
|
150
|
+
name: "SDK Test Agent - Retrieval",
|
|
151
|
+
instructions: "You are helpful agent that can help fetch data from files you know about.",
|
|
152
|
+
tools: [fileSearchTool.definition],
|
|
153
|
+
toolResources: fileSearchTool.resources,
|
|
154
|
+
});
|
|
155
|
+
console.log(`Created agent, agent ID : ${agent.id}`);
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
#### Create Agent with Code Interpreter
|
|
159
|
+
|
|
160
|
+
Here is an example to upload a file and use it for code interpreter by an Agent:
|
|
161
|
+
|
|
162
|
+
```ts snippet:codeInterpreter
|
|
163
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
164
|
+
|
|
165
|
+
const localFileStream = fs.createReadStream(filePath);
|
|
166
|
+
const localFile = await client.agents.uploadFile(localFileStream, "assistants", {
|
|
167
|
+
fileName: "localFile",
|
|
168
|
+
});
|
|
169
|
+
console.log(`Uploaded local file, file ID : ${localFile.id}`);
|
|
170
|
+
const codeInterpreterTool = ToolUtility.createCodeInterpreterTool([localFile.id]);
|
|
171
|
+
// Notice that CodeInterpreter must be enabled in the agent creation, otherwise the agent will not be able to see the file attachment
|
|
172
|
+
const agent = await client.agents.createAgent("gpt-4o-mini", {
|
|
173
|
+
name: "my-agent",
|
|
174
|
+
instructions: "You are a helpful agent",
|
|
175
|
+
tools: [codeInterpreterTool.definition],
|
|
176
|
+
toolResources: codeInterpreterTool.resources,
|
|
177
|
+
});
|
|
178
|
+
console.log(`Created agent, agent ID: ${agent.id}`);
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
#### Create Agent with Bing Grounding
|
|
182
|
+
|
|
183
|
+
To enable your Agent to perform search through Bing search API, you use `ToolUtility.createConnectionTool()` along with a connection.
|
|
184
|
+
|
|
185
|
+
Here is an example:
|
|
186
|
+
|
|
187
|
+
```ts snippet:bingGrounding
|
|
188
|
+
import { ToolUtility, connectionToolType } from "@azure/ai-projects";
|
|
189
|
+
|
|
190
|
+
const bingConnection = await client.connections.getConnection(
|
|
191
|
+
process.env.BING_CONNECTION_NAME ?? "<connection-name>",
|
|
192
|
+
);
|
|
193
|
+
const connectionId = bingConnection.id;
|
|
194
|
+
const bingTool = ToolUtility.createConnectionTool(connectionToolType.BingGrounding, [connectionId]);
|
|
195
|
+
const agent = await client.agents.createAgent("gpt-4o", {
|
|
196
|
+
name: "my-agent",
|
|
197
|
+
instructions: "You are a helpful agent",
|
|
198
|
+
tools: [bingTool.definition],
|
|
199
|
+
});
|
|
200
|
+
console.log(`Created agent, agent ID : ${agent.id}`);
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
#### Create Agent with Azure AI Search
|
|
204
|
+
|
|
205
|
+
Azure AI Search is an enterprise search system for high-performance applications. It integrates with Azure OpenAI Service and Azure Machine Learning, offering advanced search technologies like vector search and full-text search. Ideal for knowledge base insights, information discovery, and automation
|
|
206
|
+
|
|
207
|
+
Here is an example to integrate Azure AI Search:
|
|
208
|
+
|
|
209
|
+
```ts snippet:AISearch
|
|
210
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
211
|
+
|
|
212
|
+
const connectionName =
|
|
213
|
+
process.env.AZURE_AI_SEARCH_CONNECTION_NAME ?? "<AzureAISearchConnectionName>";
|
|
214
|
+
const connection = await client.connections.getConnection(connectionName);
|
|
215
|
+
const azureAISearchTool = ToolUtility.createAzureAISearchTool(connection.id, connection.name);
|
|
216
|
+
const agent = await client.agents.createAgent("gpt-4-0125-preview", {
|
|
217
|
+
name: "my-agent",
|
|
218
|
+
instructions: "You are a helpful agent",
|
|
219
|
+
tools: [azureAISearchTool.definition],
|
|
220
|
+
toolResources: azureAISearchTool.resources,
|
|
221
|
+
});
|
|
222
|
+
console.log(`Created agent, agent ID : ${agent.id}`);
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
#### Create Agent with Function Call
|
|
226
|
+
|
|
227
|
+
You can enhance your Agents by defining callback functions as function tools. These can be provided to `createAgent` via the combination of `tools` and `toolResources`. Only the function definitions and descriptions are provided to `createAgent`, without the implementations. The `Run` or `event handler of stream` will raise a `requires_action` status based on the function definitions. Your code must handle this status and call the appropriate functions.
|
|
228
|
+
|
|
229
|
+
Here is an example:
|
|
230
|
+
|
|
231
|
+
```ts snippet:functionTools
|
|
232
|
+
import {
|
|
233
|
+
FunctionToolDefinition,
|
|
234
|
+
ToolUtility,
|
|
235
|
+
RequiredToolCallOutput,
|
|
236
|
+
FunctionToolDefinitionOutput,
|
|
237
|
+
ToolOutput,
|
|
238
|
+
} from "@azure/ai-projects";
|
|
239
|
+
|
|
240
|
+
class FunctionToolExecutor {
|
|
241
|
+
private functionTools: {
|
|
242
|
+
func: Function;
|
|
243
|
+
definition: FunctionToolDefinition;
|
|
244
|
+
}[];
|
|
245
|
+
constructor() {
|
|
246
|
+
this.functionTools = [
|
|
247
|
+
{
|
|
248
|
+
func: this.getUserFavoriteCity,
|
|
249
|
+
...ToolUtility.createFunctionTool({
|
|
250
|
+
name: "getUserFavoriteCity",
|
|
251
|
+
description: "Gets the user's favorite city.",
|
|
252
|
+
parameters: {},
|
|
253
|
+
}),
|
|
254
|
+
},
|
|
255
|
+
{
|
|
256
|
+
func: this.getCityNickname,
|
|
257
|
+
...ToolUtility.createFunctionTool({
|
|
258
|
+
name: "getCityNickname",
|
|
259
|
+
description: "Gets the nickname of a city, e.g. 'LA' for 'Los Angeles, CA'.",
|
|
260
|
+
parameters: {
|
|
261
|
+
type: "object",
|
|
262
|
+
properties: {
|
|
263
|
+
location: { type: "string", description: "The city and state, e.g. Seattle, Wa" },
|
|
264
|
+
},
|
|
265
|
+
},
|
|
266
|
+
}),
|
|
267
|
+
},
|
|
268
|
+
{
|
|
269
|
+
func: this.getWeather,
|
|
270
|
+
...ToolUtility.createFunctionTool({
|
|
271
|
+
name: "getWeather",
|
|
272
|
+
description: "Gets the weather for a location.",
|
|
273
|
+
parameters: {
|
|
274
|
+
type: "object",
|
|
275
|
+
properties: {
|
|
276
|
+
location: { type: "string", description: "The city and state, e.g. Seattle, Wa" },
|
|
277
|
+
unit: { type: "string", enum: ["c", "f"] },
|
|
278
|
+
},
|
|
279
|
+
},
|
|
280
|
+
}),
|
|
281
|
+
},
|
|
282
|
+
];
|
|
283
|
+
}
|
|
284
|
+
private getUserFavoriteCity(): {} {
|
|
285
|
+
return { location: "Seattle, WA" };
|
|
286
|
+
}
|
|
287
|
+
private getCityNickname(location: string): {} {
|
|
288
|
+
return { nickname: "The Emerald City" };
|
|
289
|
+
}
|
|
290
|
+
private getWeather(location: string, unit: string): {} {
|
|
291
|
+
return { weather: unit === "f" ? "72f" : "22c" };
|
|
292
|
+
}
|
|
293
|
+
public invokeTool(
|
|
294
|
+
toolCall: RequiredToolCallOutput & FunctionToolDefinitionOutput,
|
|
295
|
+
): ToolOutput | undefined {
|
|
296
|
+
console.log(`Function tool call - ${toolCall.function.name}`);
|
|
297
|
+
const args = [];
|
|
298
|
+
if (toolCall.function.parameters) {
|
|
299
|
+
try {
|
|
300
|
+
const params = JSON.parse(toolCall.function.parameters);
|
|
301
|
+
for (const key in params) {
|
|
302
|
+
if (Object.prototype.hasOwnProperty.call(params, key)) {
|
|
303
|
+
args.push(params[key]);
|
|
304
|
+
}
|
|
305
|
+
}
|
|
306
|
+
} catch (error) {
|
|
307
|
+
console.error(`Failed to parse parameters: ${toolCall.function.parameters}`, error);
|
|
308
|
+
return undefined;
|
|
309
|
+
}
|
|
310
|
+
}
|
|
311
|
+
const result = this.functionTools
|
|
312
|
+
.find((tool) => tool.definition.function.name === toolCall.function.name)
|
|
313
|
+
?.func(...args);
|
|
314
|
+
return result
|
|
315
|
+
? {
|
|
316
|
+
toolCallId: toolCall.id,
|
|
317
|
+
output: JSON.stringify(result),
|
|
318
|
+
}
|
|
319
|
+
: undefined;
|
|
320
|
+
}
|
|
321
|
+
public getFunctionDefinitions(): FunctionToolDefinition[] {
|
|
322
|
+
return this.functionTools.map((tool) => {
|
|
323
|
+
return tool.definition;
|
|
324
|
+
});
|
|
325
|
+
}
|
|
326
|
+
}
|
|
327
|
+
const functionToolExecutor = new FunctionToolExecutor();
|
|
328
|
+
const functionTools = functionToolExecutor.getFunctionDefinitions();
|
|
329
|
+
const agent = await client.agents.createAgent("gpt-4o", {
|
|
330
|
+
name: "my-agent",
|
|
331
|
+
instructions:
|
|
332
|
+
"You are a weather bot. Use the provided functions to help answer questions. Customize your responses to the user's preferences as much as possible and use friendly nicknames for cities whenever possible.",
|
|
333
|
+
tools: functionTools,
|
|
334
|
+
});
|
|
335
|
+
console.log(`Created agent, agent ID: ${agent.id}`);
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
#### Create Agent With OpenAPI
|
|
339
|
+
|
|
340
|
+
OpenAPI specifications describe REST operations against a specific endpoint. Agents SDK can read an OpenAPI spec, create a function from it, and call that function against the REST endpoint without additional client-side execution.
|
|
341
|
+
Here is an example creating an OpenAPI tool (using anonymous authentication):
|
|
342
|
+
|
|
343
|
+
```ts snippet:createAgentWithOpenApi
|
|
344
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
345
|
+
|
|
346
|
+
// Read in OpenApi spec
|
|
347
|
+
const filePath = "./data/weatherOpenApi.json";
|
|
348
|
+
const openApiSpec = JSON.parse(fs.readFileSync(filePath, "utf-8"));
|
|
349
|
+
// Define OpenApi function
|
|
350
|
+
const openApiFunction = {
|
|
351
|
+
name: "getWeather",
|
|
352
|
+
spec: openApiSpec,
|
|
353
|
+
description: "Retrieve weather information for a location",
|
|
354
|
+
auth: {
|
|
355
|
+
type: "anonymous",
|
|
356
|
+
},
|
|
357
|
+
default_params: ["format"], // optional
|
|
358
|
+
};
|
|
359
|
+
// Create OpenApi tool
|
|
360
|
+
const openApiTool = ToolUtility.createOpenApiTool(openApiFunction);
|
|
361
|
+
// Create agent with OpenApi tool
|
|
362
|
+
const agent = await client.agents.createAgent("gpt-4o-mini", {
|
|
363
|
+
name: "myAgent",
|
|
364
|
+
instructions: "You are a helpful agent",
|
|
365
|
+
tools: [openApiTool.definition],
|
|
366
|
+
});
|
|
367
|
+
console.log(`Created agent, agent ID: ${agent.id}`);
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
#### Create an Agent with Fabric
|
|
371
|
+
|
|
372
|
+
To enable your Agent to answer queries using Fabric data, use `FabricTool` along with a connection to the Fabric resource.
|
|
373
|
+
|
|
374
|
+
Here is an example:
|
|
375
|
+
|
|
376
|
+
```ts snippet:createAgentWithFabric
|
|
377
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
378
|
+
|
|
379
|
+
const fabricConnection = await client.connections.getConnection(
|
|
380
|
+
process.env["FABRIC_CONNECTION_NAME"] || "<connection-name>",
|
|
381
|
+
);
|
|
382
|
+
const connectionId = fabricConnection.id;
|
|
383
|
+
// Initialize agent Microsoft Fabric tool with the connection id
|
|
384
|
+
const fabricTool = ToolUtility.createFabricTool(connectionId);
|
|
385
|
+
// Create agent with the Microsoft Fabric tool and process assistant run
|
|
386
|
+
const agent = await client.agents.createAgent("gpt-4o", {
|
|
387
|
+
name: "my-agent",
|
|
388
|
+
instructions: "You are a helpful agent",
|
|
389
|
+
tools: [fabricTool.definition],
|
|
390
|
+
});
|
|
391
|
+
console.log(`Created agent, agent ID : ${agent.id}`);
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
#### Create Thread
|
|
395
|
+
|
|
396
|
+
For each session or conversation, a thread is required. Here is an example:
|
|
397
|
+
|
|
398
|
+
```ts snippet:createThread
|
|
399
|
+
const thread = await client.agents.createThread();
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
#### Create Thread with Tool Resource
|
|
403
|
+
|
|
404
|
+
In some scenarios, you might need to assign specific resources to individual threads. To achieve this, you provide the `toolResources` argument to `createThread`. In the following example, you create a vector store and upload a file, enable an Agent for file search using the `tools` argument, and then associate the file with the thread using the `toolResources` argument.
|
|
405
|
+
|
|
406
|
+
```ts snippet:threadWithTool
|
|
407
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
408
|
+
|
|
409
|
+
const localFileStream = fs.createReadStream(filePath);
|
|
410
|
+
const file = await client.agents.uploadFile(localFileStream, "assistants", {
|
|
411
|
+
fileName: "sample_file_for_upload.csv",
|
|
412
|
+
});
|
|
413
|
+
console.log(`Uploaded file, ID: ${file.id}`);
|
|
414
|
+
const vectorStore = await client.agents.createVectorStore({
|
|
415
|
+
fileIds: [file.id],
|
|
416
|
+
});
|
|
417
|
+
console.log(`Created vector store, ID: ${vectorStore.id}`);
|
|
418
|
+
const fileSearchTool = ToolUtility.createFileSearchTool([vectorStore.id]);
|
|
419
|
+
const agent = await client.agents.createAgent("gpt-4o", {
|
|
420
|
+
name: "myAgent",
|
|
421
|
+
instructions: "You are helpful agent that can help fetch data from files you know about.",
|
|
422
|
+
tools: [fileSearchTool.definition],
|
|
423
|
+
});
|
|
424
|
+
console.log(`Created agent, agent ID : ${agent.id}`);
|
|
425
|
+
// Create thread with file resources.
|
|
426
|
+
// If the agent has multiple threads, only this thread can search this file.
|
|
427
|
+
const thread = await client.agents.createThread({ toolResources: fileSearchTool.resources });
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
#### List Threads
|
|
431
|
+
|
|
432
|
+
To list all threads attached to a given agent, use the list_threads API:
|
|
433
|
+
|
|
434
|
+
```ts snippet:listThreads
|
|
435
|
+
const threads = await client.agents.listThreads();
|
|
436
|
+
console.log(`Threads for agent ${agent.id}:`);
|
|
437
|
+
for await (const t of (await threads).data) {
|
|
438
|
+
console.log(`Thread ID: ${t.id}`);
|
|
439
|
+
console.log(`Created at: ${t.createdAt}`);
|
|
440
|
+
console.log(`Metadata: ${t.metadata}`);
|
|
441
|
+
console.log(`---- `);
|
|
442
|
+
}
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
#### Create Message
|
|
446
|
+
|
|
447
|
+
To create a message for assistant to process, you pass `user` as `role` and a question as `content`:
|
|
448
|
+
|
|
449
|
+
```ts snippet:createMessage
|
|
450
|
+
const message = await client.agents.createMessage(thread.id, {
|
|
451
|
+
role: "user",
|
|
452
|
+
content: "hello, world!",
|
|
453
|
+
});
|
|
454
|
+
console.log(`Created message, message ID: ${message.id}`);
|
|
455
|
+
```
|
|
456
|
+
|
|
457
|
+
#### Create Message with File Search Attachment
|
|
458
|
+
|
|
459
|
+
To attach a file to a message for content searching, you use `ToolUtility.createFileSearchTool()` and the `attachments` argument:
|
|
460
|
+
|
|
461
|
+
```ts snippet:messageWithFileSearch
|
|
462
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
463
|
+
|
|
464
|
+
const fileSearchTool = ToolUtility.createFileSearchTool();
|
|
465
|
+
const message = await client.agents.createMessage(thread.id, {
|
|
466
|
+
role: "user",
|
|
467
|
+
content: "What feature does Smart Eyewear offer?",
|
|
468
|
+
attachments: {
|
|
469
|
+
fileId: file.id,
|
|
470
|
+
tools: [fileSearchTool.definition],
|
|
471
|
+
},
|
|
472
|
+
});
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
#### Create Message with Code Interpreter Attachment
|
|
476
|
+
|
|
477
|
+
To attach a file to a message for data analysis, you use `ToolUtility.createCodeInterpreterTool()` and the `attachment` argument.
|
|
478
|
+
|
|
479
|
+
Here is an example:
|
|
480
|
+
|
|
481
|
+
```ts snippet:messageWithCodeInterpreter
|
|
482
|
+
import { ToolUtility } from "@azure/ai-projects";
|
|
483
|
+
|
|
484
|
+
// notice that CodeInterpreter must be enabled in the agent creation,
|
|
485
|
+
// otherwise the agent will not be able to see the file attachment for code interpretation
|
|
486
|
+
const codeInterpreterTool = ToolUtility.createCodeInterpreterTool();
|
|
487
|
+
const agent = await client.agents.createAgent("gpt-4-1106-preview", {
|
|
488
|
+
name: "my-assistant",
|
|
489
|
+
instructions: "You are helpful assistant",
|
|
490
|
+
tools: [codeInterpreterTool.definition],
|
|
491
|
+
});
|
|
492
|
+
console.log(`Created agent, agent ID: ${agent.id}`);
|
|
493
|
+
const thread = await client.agents.createThread();
|
|
494
|
+
console.log(`Created thread, thread ID: ${thread.id}`);
|
|
495
|
+
const message = await client.agents.createMessage(thread.id, {
|
|
496
|
+
role: "user",
|
|
497
|
+
content:
|
|
498
|
+
"Could you please create bar chart in TRANSPORTATION sector for the operating profit from the uploaded csv file and provide file to me?",
|
|
499
|
+
attachments: {
|
|
500
|
+
fileId: file.id,
|
|
501
|
+
tools: [codeInterpreterTool.definition],
|
|
502
|
+
},
|
|
503
|
+
});
|
|
504
|
+
console.log(`Created message, message ID: ${message.id}`);
|
|
505
|
+
```
|
|
506
|
+
|
|
507
|
+
#### Create Message with Image Inputs
|
|
508
|
+
|
|
509
|
+
You can send messages to Azure agents with image inputs in following ways:
|
|
510
|
+
|
|
511
|
+
- **Using an image stored as a uploaded file**
|
|
512
|
+
- **Using a public image accessible via URL**
|
|
513
|
+
- **Using a base64 encoded image string**
|
|
514
|
+
|
|
515
|
+
The following examples demonstrate each method:
|
|
516
|
+
|
|
517
|
+
##### Create message using uploaded image file
|
|
518
|
+
|
|
519
|
+
```ts snippet:imageInputWithFile
|
|
520
|
+
// Upload the local image file
|
|
521
|
+
const fileStream = fs.createReadStream(imagePath);
|
|
522
|
+
const imageFile = await client.agents.uploadFile(fileStream, "assistants", {
|
|
523
|
+
fileName: "image_file.png",
|
|
524
|
+
});
|
|
525
|
+
console.log(`Uploaded file, file ID: ${imageFile.id}`);
|
|
526
|
+
// Create a message with both text and image content
|
|
527
|
+
console.log("Creating message with image content...");
|
|
528
|
+
const inputMessage = "Hello, what is in the image?";
|
|
529
|
+
const content = [
|
|
530
|
+
{
|
|
531
|
+
type: "text",
|
|
532
|
+
text: inputMessage,
|
|
533
|
+
},
|
|
534
|
+
{
|
|
535
|
+
type: "image_file",
|
|
536
|
+
image_file: {
|
|
537
|
+
file_id: imageFile.id,
|
|
538
|
+
detail: "high",
|
|
539
|
+
},
|
|
540
|
+
},
|
|
541
|
+
];
|
|
542
|
+
const message = await client.agents.createMessage(thread.id, {
|
|
543
|
+
role: "user",
|
|
544
|
+
content: content,
|
|
545
|
+
});
|
|
546
|
+
console.log(`Created message, message ID: ${message.id}`);
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
##### Create message with an image URL input
|
|
550
|
+
|
|
551
|
+
```ts snippet:imageInputWithUrl
|
|
552
|
+
// Specify the public image URL
|
|
553
|
+
const imageUrl =
|
|
554
|
+
"https://github.com/Azure/azure-sdk-for-js/blob/0aa88ceb18d865726d423f73b8393134e783aea6/sdk/ai/ai-projects/data/image_file.png?raw=true";
|
|
555
|
+
// Create content directly referencing image URL
|
|
556
|
+
const inputMessage = "Hello, what is in the image?";
|
|
557
|
+
const content = [
|
|
558
|
+
{
|
|
559
|
+
type: "text",
|
|
560
|
+
text: inputMessage,
|
|
561
|
+
},
|
|
562
|
+
{
|
|
563
|
+
type: "image_url",
|
|
564
|
+
image_url: {
|
|
565
|
+
url: imageUrl,
|
|
566
|
+
detail: "high",
|
|
567
|
+
},
|
|
568
|
+
},
|
|
569
|
+
];
|
|
570
|
+
const message = await client.agents.createMessage(thread.id, {
|
|
571
|
+
role: "user",
|
|
572
|
+
content: content,
|
|
573
|
+
});
|
|
574
|
+
console.log(`Created message, message ID: ${message.id}`);
|
|
575
|
+
```
|
|
576
|
+
|
|
577
|
+
##### Create message with base64-encoded image input
|
|
578
|
+
|
|
579
|
+
```ts snippet:imageInputWithBase64
|
|
580
|
+
function imageToBase64DataUrl(imagePath: string, mimeType: string): string {
|
|
581
|
+
try {
|
|
582
|
+
// Read the image file as binary
|
|
583
|
+
const imageBuffer = fs.readFileSync(imagePath);
|
|
584
|
+
// Convert to base64
|
|
585
|
+
const base64Data = imageBuffer.toString("base64");
|
|
586
|
+
// Format as a data URL
|
|
587
|
+
return `data:${mimeType};base64,${base64Data}`;
|
|
588
|
+
} catch (error) {
|
|
589
|
+
console.error(`Error reading image file at ${imagePath}:`, error);
|
|
590
|
+
throw error;
|
|
591
|
+
}
|
|
592
|
+
}
|
|
593
|
+
// Convert your image file to base64 format
|
|
594
|
+
const imageDataUrl = imageToBase64DataUrl(filePath, "image/png");
|
|
595
|
+
// Create a message with both text and image content
|
|
596
|
+
const inputMessage = "Hello, what is in the image?";
|
|
597
|
+
const content = [
|
|
598
|
+
{
|
|
599
|
+
type: "text",
|
|
600
|
+
text: inputMessage,
|
|
601
|
+
},
|
|
602
|
+
{
|
|
603
|
+
type: "image_url",
|
|
604
|
+
image_url: {
|
|
605
|
+
url: imageDataUrl,
|
|
606
|
+
detail: "high",
|
|
607
|
+
},
|
|
608
|
+
},
|
|
609
|
+
];
|
|
610
|
+
const message = await client.agents.createMessage(thread.id, {
|
|
611
|
+
role: "user",
|
|
612
|
+
content: content,
|
|
613
|
+
});
|
|
614
|
+
console.log(`Created message, message ID: ${message.id}`);
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
#### Create Run, Run_and_Process, or Stream
|
|
618
|
+
|
|
619
|
+
Here is an example of `createRun` and poll until the run is completed:
|
|
620
|
+
|
|
621
|
+
```ts snippet:createRun
|
|
622
|
+
let run = await client.agents.createRun(thread.id, agent.id);
|
|
623
|
+
// Poll the run as long as run status is queued or in progress
|
|
624
|
+
while (
|
|
625
|
+
run.status === "queued" ||
|
|
626
|
+
run.status === "in_progress" ||
|
|
627
|
+
run.status === "requires_action"
|
|
628
|
+
) {
|
|
629
|
+
// Wait for a second
|
|
630
|
+
await new Promise((resolve) => setTimeout(resolve, 1000));
|
|
631
|
+
run = await client.agents.getRun(thread.id, run.id);
|
|
632
|
+
}
|
|
633
|
+
```
|
|
634
|
+
|
|
635
|
+
To have the SDK poll on your behalf, use the `createThreadAndRun` method.
|
|
636
|
+
|
|
637
|
+
Here is an example:
|
|
638
|
+
|
|
639
|
+
```ts snippet:createThreadAndRun
|
|
640
|
+
const run = await client.agents.createThreadAndRun(agent.id, {
|
|
641
|
+
thread: {
|
|
642
|
+
messages: [
|
|
643
|
+
{
|
|
644
|
+
role: "user",
|
|
645
|
+
content: "hello, world!",
|
|
646
|
+
},
|
|
647
|
+
],
|
|
648
|
+
},
|
|
649
|
+
});
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
With streaming, polling also need not be considered.
|
|
653
|
+
|
|
654
|
+
Here is an example:
|
|
655
|
+
|
|
656
|
+
```ts snippet:createRunStream
|
|
657
|
+
const streamEventMessages = await client.agents.createRun(thread.id, agent.id).stream();
|
|
658
|
+
```
|
|
659
|
+
|
|
660
|
+
Event handling can be done as follows:
|
|
661
|
+
|
|
662
|
+
```ts snippet:eventHandling
|
|
663
|
+
import {
|
|
664
|
+
RunStreamEvent,
|
|
665
|
+
ThreadRunOutput,
|
|
666
|
+
MessageStreamEvent,
|
|
667
|
+
MessageDeltaChunk,
|
|
668
|
+
MessageDeltaTextContent,
|
|
669
|
+
DoneEvent,
|
|
670
|
+
} from "@azure/ai-projects";
|
|
671
|
+
|
|
672
|
+
const streamEventMessages = await client.agents.createRun(thread.id, agent.id).stream();
|
|
673
|
+
for await (const eventMessage of streamEventMessages) {
|
|
674
|
+
switch (eventMessage.event) {
|
|
675
|
+
case RunStreamEvent.ThreadRunCreated:
|
|
676
|
+
console.log(`ThreadRun status: ${(eventMessage.data as ThreadRunOutput).status}`);
|
|
677
|
+
break;
|
|
678
|
+
case MessageStreamEvent.ThreadMessageDelta:
|
|
679
|
+
{
|
|
680
|
+
const messageDelta = eventMessage.data as MessageDeltaChunk;
|
|
681
|
+
messageDelta.delta.content.forEach((contentPart) => {
|
|
682
|
+
if (contentPart.type === "text") {
|
|
683
|
+
const textContent = contentPart as MessageDeltaTextContent;
|
|
684
|
+
const textValue = textContent.text?.value || "No text";
|
|
685
|
+
console.log(`Text delta received:: ${textValue}`);
|
|
686
|
+
}
|
|
687
|
+
});
|
|
688
|
+
}
|
|
689
|
+
break;
|
|
690
|
+
case RunStreamEvent.ThreadRunCompleted:
|
|
691
|
+
console.log("Thread Run Completed");
|
|
692
|
+
break;
|
|
693
|
+
case ErrorEvent.Error:
|
|
694
|
+
console.log(`An error occurred. Data ${eventMessage.data}`);
|
|
695
|
+
break;
|
|
696
|
+
case DoneEvent.Done:
|
|
697
|
+
console.log("Stream completed.");
|
|
698
|
+
break;
|
|
699
|
+
}
|
|
700
|
+
}
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
#### Retrieve Message
|
|
704
|
+
|
|
705
|
+
To retrieve messages from agents, use the following example:
|
|
706
|
+
|
|
707
|
+
```ts snippet:listMessages
|
|
708
|
+
import { MessageContentOutput, isOutputOfType, MessageTextContentOutput } from "@azure/ai-projects";
|
|
709
|
+
|
|
710
|
+
const messages = await client.agents.listMessages(thread.id);
|
|
711
|
+
while (messages.hasMore) {
|
|
712
|
+
const nextMessages = await client.agents.listMessages(currentRun.threadId, {
|
|
713
|
+
after: messages.lastId,
|
|
714
|
+
});
|
|
715
|
+
messages.data = messages.data.concat(nextMessages.data);
|
|
716
|
+
messages.hasMore = nextMessages.hasMore;
|
|
717
|
+
messages.lastId = nextMessages.lastId;
|
|
718
|
+
}
|
|
719
|
+
// The messages are following in the reverse order,
|
|
720
|
+
// we will iterate them and output only text contents.
|
|
721
|
+
for (const dataPoint of messages.data.reverse()) {
|
|
722
|
+
const lastMessageContent: MessageContentOutput = dataPoint.content[dataPoint.content.length - 1];
|
|
723
|
+
console.log(lastMessageContent);
|
|
724
|
+
if (isOutputOfType<MessageTextContentOutput>(lastMessageContent, "text")) {
|
|
725
|
+
console.log(
|
|
726
|
+
`${dataPoint.role}: ${(lastMessageContent as MessageTextContentOutput).text.value}`,
|
|
727
|
+
);
|
|
728
|
+
}
|
|
729
|
+
}
|
|
730
|
+
```
|
|
731
|
+
|
|
732
|
+
### Retrieve File
|
|
733
|
+
|
|
734
|
+
Files uploaded by Agents cannot be retrieved back. If your use case needs to access the file content uploaded by the Agents, you are advised to keep an additional copy accessible by your application. However, files generated by Agents are retrievable by `getFileContent`.
|
|
735
|
+
|
|
736
|
+
Here is an example retrieving file ids from messages:
|
|
737
|
+
|
|
738
|
+
```ts snippet:retrieveFile
|
|
739
|
+
import {
|
|
740
|
+
isOutputOfType,
|
|
741
|
+
MessageTextContentOutput,
|
|
742
|
+
MessageImageFileContentOutput,
|
|
743
|
+
} from "@azure/ai-projects";
|
|
744
|
+
|
|
745
|
+
const messages = await client.agents.listMessages(thread.id);
|
|
746
|
+
// Get most recent message from the assistant
|
|
747
|
+
const assistantMessage = messages.data.find((msg) => msg.role === "assistant");
|
|
748
|
+
if (assistantMessage) {
|
|
749
|
+
const textContent = assistantMessage.content.find((content) =>
|
|
750
|
+
isOutputOfType<MessageTextContentOutput>(content, "text"),
|
|
751
|
+
) as MessageTextContentOutput;
|
|
752
|
+
if (textContent) {
|
|
753
|
+
console.log(`Last message: ${textContent.text.value}`);
|
|
754
|
+
}
|
|
755
|
+
}
|
|
756
|
+
const imageFile = (messages.data[0].content[0] as MessageImageFileContentOutput).imageFile;
|
|
757
|
+
const imageFileName = (await client.agents.getFile(imageFile.fileId)).filename;
|
|
758
|
+
const fileContent = await (
|
|
759
|
+
await client.agents.getFileContent(imageFile.fileId).asNodeStream()
|
|
760
|
+
).body;
|
|
761
|
+
if (fileContent) {
|
|
762
|
+
const chunks: Buffer[] = [];
|
|
763
|
+
for await (const chunk of fileContent) {
|
|
764
|
+
chunks.push(Buffer.from(chunk));
|
|
765
|
+
}
|
|
766
|
+
const buffer = Buffer.concat(chunks);
|
|
767
|
+
fs.writeFileSync(imageFileName, buffer);
|
|
768
|
+
} else {
|
|
769
|
+
console.error("Failed to retrieve file content: fileContent is undefined");
|
|
770
|
+
}
|
|
771
|
+
console.log(`Saved image file to: ${imageFileName}`);
|
|
772
|
+
```
|
|
773
|
+
|
|
774
|
+
#### Teardown
|
|
775
|
+
|
|
776
|
+
To remove resources after completing tasks, use the following functions:
|
|
777
|
+
|
|
778
|
+
```ts snippet:teardown
|
|
779
|
+
await client.agents.deleteVectorStore(vectorStore.id);
|
|
780
|
+
console.log(`Deleted vector store, vector store ID: ${vectorStore.id}`);
|
|
781
|
+
await client.agents.deleteFile(file.id);
|
|
782
|
+
console.log(`Deleted file, file ID: ${file.id}`);
|
|
783
|
+
client.agents.deleteAgent(agent.id);
|
|
784
|
+
console.log(`Deleted agent, agent ID: ${agent.id}`);
|
|
785
|
+
```
|
|
786
|
+
|
|
787
|
+
### Tracing
|
|
788
|
+
|
|
789
|
+
You can add an Application Insights Azure resource to your Azure AI Foundry project. See the Tracing tab in your studio. If one was enabled, you can get the Application Insights connection string, configure your Agents, and observe the full execution path through Azure Monitor. Typically, you might want to start tracing before you create an Agent.
|
|
790
|
+
|
|
791
|
+
#### Installation
|
|
792
|
+
|
|
793
|
+
Make sure to install OpenTelemetry and the Azure SDK tracing plugin via
|
|
794
|
+
|
|
795
|
+
```bash
|
|
796
|
+
npm install @opentelemetry/api \
|
|
797
|
+
@opentelemetry/instrumentation \
|
|
798
|
+
@opentelemetry/sdk-trace-node \
|
|
799
|
+
@azure/opentelemetry-instrumentation-azure-sdk \
|
|
800
|
+
@azure/monitor-opentelemetry-exporter
|
|
801
|
+
```
|
|
802
|
+
|
|
803
|
+
You will also need an exporter to send telemetry to your observability backend. You can print traces to the console or use a local viewer such as [Aspire Dashboard](https://learn.microsoft.com/dotnet/aspire/fundamentals/dashboard/standalone?tabs=bash).
|
|
804
|
+
|
|
805
|
+
To connect to Aspire Dashboard or another OpenTelemetry compatible backend, install OTLP exporter:
|
|
806
|
+
|
|
807
|
+
```bash
|
|
808
|
+
npm install @opentelemetry/exporter-trace-otlp-proto \
|
|
809
|
+
@opentelemetry/exporter-metrics-otlp-proto
|
|
810
|
+
```
|
|
811
|
+
|
|
812
|
+
#### Tracing example
|
|
813
|
+
|
|
814
|
+
Here is a code sample to be included above `createAgent`:
|
|
815
|
+
|
|
816
|
+
```ts snippet:tracing
|
|
817
|
+
import {
|
|
818
|
+
NodeTracerProvider,
|
|
819
|
+
SimpleSpanProcessor,
|
|
820
|
+
ConsoleSpanExporter,
|
|
821
|
+
} from "@opentelemetry/sdk-trace-node";
|
|
822
|
+
import { trace } from "@opentelemetry/api";
|
|
823
|
+
import { AzureMonitorTraceExporter } from "@azure/monitor-opentelemetry-exporter";
|
|
824
|
+
|
|
825
|
+
const provider = new NodeTracerProvider();
|
|
826
|
+
provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
|
|
827
|
+
provider.register();
|
|
828
|
+
const tracer = trace.getTracer("Agents Sample", "1.0.0");
|
|
829
|
+
let appInsightsConnectionString =
|
|
830
|
+
process.env.APP_INSIGHTS_CONNECTION_STRING ?? "<appInsightsConnectionString>";
|
|
831
|
+
if (appInsightsConnectionString == "<appInsightsConnectionString>") {
|
|
832
|
+
appInsightsConnectionString = await client.telemetry.getConnectionString();
|
|
833
|
+
}
|
|
834
|
+
if (appInsightsConnectionString) {
|
|
835
|
+
const exporter = new AzureMonitorTraceExporter({
|
|
836
|
+
connectionString: appInsightsConnectionString,
|
|
837
|
+
});
|
|
838
|
+
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
|
|
839
|
+
}
|
|
840
|
+
await tracer.startActiveSpan("main", async (span) => {
|
|
841
|
+
client.telemetry.updateSettings({ enableContentRecording: true });
|
|
842
|
+
// ...
|
|
843
|
+
});
|
|
844
|
+
```
|
|
845
|
+
|
|
846
|
+
## Troubleshooting
|
|
847
|
+
|
|
848
|
+
### Exceptions
|
|
849
|
+
|
|
850
|
+
Client methods that make service calls raise an [RestError](https://learn.microsoft.com/javascript/api/%40azure/core-rest-pipeline/resterror) for a non-success HTTP status code response from the service. The exception's `code` will hold the HTTP response status code. The exception's `error.message` contains a detailed message that may be helpful in diagnosing the issue:
|
|
851
|
+
|
|
852
|
+
```ts snippet:exceptions
|
|
853
|
+
import { RestError } from "@azure/core-rest-pipeline";
|
|
854
|
+
|
|
855
|
+
try {
|
|
856
|
+
const result = await client.connections.listConnections();
|
|
857
|
+
} catch (e) {
|
|
858
|
+
if (e instanceof RestError) {
|
|
859
|
+
console.log(`Status code: ${e.code}`);
|
|
860
|
+
console.log(e.message);
|
|
861
|
+
} else {
|
|
862
|
+
console.error(e);
|
|
863
|
+
}
|
|
864
|
+
}
|
|
865
|
+
```
|
|
866
|
+
|
|
867
|
+
For example, when you provide wrong credentials:
|
|
868
|
+
|
|
869
|
+
```text
|
|
870
|
+
Status code: 401 (Unauthorized)
|
|
871
|
+
Operation returned an invalid status 'Unauthorized'
|
|
872
|
+
```
|
|
873
|
+
|
|
874
|
+
### Reporting issues
|
|
875
|
+
|
|
876
|
+
To report issues with the client library, or request additional features, please open a GitHub issue [here](https://github.com/Azure/azure-sdk-for-js/issues)
|
|
877
|
+
|
|
878
|
+
## Next steps
|
|
879
|
+
|
|
880
|
+
Have a look at the [package samples](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-projects/samples) folder, containing fully runnable code.
|
|
881
|
+
|
|
882
|
+
## Contributing
|
|
883
|
+
|
|
884
|
+
This project welcomes contributions and suggestions. Most contributions require
|
|
885
|
+
you to agree to a Contributor License Agreement (CLA) declaring that you have
|
|
886
|
+
the right to, and actually do, grant us the rights to use your contribution.
|
|
887
|
+
For details, visit https://cla.microsoft.com.
|
|
888
|
+
|
|
889
|
+
When you submit a pull request, a CLA-bot will automatically determine whether
|
|
890
|
+
you need to provide a CLA and decorate the PR appropriately (e.g., label,
|
|
891
|
+
comment). Simply follow the instructions provided by the bot. You will only
|
|
892
|
+
need to do this once across all repos using our CLA.
|
|
893
|
+
|
|
894
|
+
This project has adopted the
|
|
895
|
+
[Microsoft Open Source Code of Conduct][code_of_conduct]. For more information,
|
|
896
|
+
see the Code of Conduct FAQ or contact opencode@microsoft.com with any
|
|
897
|
+
additional questions or comments.
|
|
898
|
+
|
|
899
|
+
<!-- LINKS -->
|
|
900
|
+
|
|
901
|
+
[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/
|
|
902
|
+
[entra_id]: https://learn.microsoft.com/azure/ai-services/authentication?tabs=powershell#authenticate-with-microsoft-entra-id
|
|
903
|
+
[azure_identity_npm]: https://www.npmjs.com/package/@azure/identity
|
|
904
|
+
[default_azure_credential]: https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/identity/identity#defaultazurecredential
|
|
905
|
+
[azure_sub]: https://azure.microsoft.com/free/
|
|
906
|
+
[evaluators]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk
|
|
907
|
+
[evaluator_library]: https://learn.microsoft.com/azure/ai-studio/how-to/evaluate-generative-ai-app#view-and-manage-the-evaluators-in-the-evaluator-library
|