zuplo 7.7.8 → 7.7.10

package/docs/ai-gateway/azure-ai.mdx

@@ -0,0 +1,466 @@
+---
+title: Using Azure AI
+sidebar_label: Azure AI
+description:
+  Serve Azure OpenAI and Microsoft Foundry models through one provider
+  configuration. Map each deployment to the model it serves so the gateway
+  prices usage.
+---
+
+**Azure AI** serves models from your own Azure resource through one provider
+configuration—one resource endpoint and one resource key. Adding it gives your
+[apps](./apps.mdx) the models you have deployed, through the
+[Universal API](./universal-api.mdx), on your Azure subscription and Azure
+billing.
+
+One provider type covers both kinds of Azure resource:
+
+- An **Azure OpenAI** resource deploys OpenAI models and serves them on an
+  OpenAI-compatible API.
+- A **Microsoft Foundry** resource serves that same API for every Foundry model
+  family—Grok, DeepSeek, Llama, Mistral, Phi, Kimi and more—_and_ serves Claude
+  on the native Anthropic Messages API.
+
+Apps reference models as `providerName/model`, where `providerName` is the name
+you give the provider configuration. A provider named `azureai` serves
+`azureai/my-gpt`.
+
+## Azure serves deployments, not model names
+
+Of the ways Azure differs from other providers, this is the one that changes
+what you put in the `model` field.
+
+Other providers accept a published model ID. Azure resolves the `model` field
+against the **deployments** in your resource, and you choose deployment names
+when you create them. If you deploy `gpt-4.1-mini` under the name `my-gpt`, then
+`my-gpt` is the only name Azure accepts for it.
+
+The gateway forwards your deployment name to Azure unchanged, because it's the
+only address Azure answers to. What the gateway needs to know separately is
+which catalog model each deployment serves—that's what prices your usage.
+
+:::caution{title="An unmapped deployment is rejected"}
+
+The gateway prices a request by looking up the model name in its catalog. A
+deployment named after something outside the catalog, such as `my-gpt`, matches
+nothing—so the gateway returns **400** rather than serving a request it can't
+bill. Azure would have served it, and the usage would have recorded no cost.
+
+Map every deployment whose name differs from the model it serves. You do this
+when you add or edit the provider. The error names the deployment and lists the
+names that do work.
+
+:::
+
+Mapping is optional when it isn't needed. Deploy `gpt-4.1-mini` under the name
+`gpt-4.1-mini`—the Azure portal's own default—and it already matches the
+catalog.
+
+### Deployment name rules
+
+Azure accepts 2 to 64 characters of letters, digits, hyphens, underscores and
+periods, and won't accept a name ending in a period. Leading hyphens and
+periods, doubled periods, trailing hyphens and uppercase letters are all fine.
+
+Deployment names match case-insensitively, so a deployment named `MyGpt` answers
+to `mygpt`, and one named `my-gpt` answers to `MY-GPT`. Map each deployment once
+using whichever spelling you prefer; requests in any casing resolve to it.
+
+## How the gateway routes Azure AI models
+
+A Foundry resource serves two API formats on the same host with the same
+resource key:
+
+- An **OpenAI-compatible API** at `/openai/v1`, serving every non-Claude model.
+- The **native Anthropic Messages API** at `/anthropic/v1/messages`, serving the
+  Claude models.
+
+<Diagram height="h-64">
+  <DiagramNode id="app">Your app</DiagramNode>
+  <DiagramNode id="gateway" variant="zuplo">
+    AI Gateway
+  </DiagramNode>
+  <DiagramGroup id="azure" label="Your Azure resource">
+    <DiagramNode id="openai-surface" variant="blue">
+      OpenAI-compatible API
+    </DiagramNode>
+    <DiagramNode id="messages-surface" variant="green">
+      Anthropic Messages API
+    </DiagramNode>
+  </DiagramGroup>
+  <DiagramEdge from="app" to="gateway" label="Universal API" />
+  <DiagramEdge
+    from="gateway"
+    to="openai-surface"
+    label="OpenAI-compatible models"
+  />
+  <DiagramEdge from="gateway" to="messages-surface" label="Claude models" />
+</Diagram>
+
+The gateway's model catalog records which API serves each model, and routes
+every request accordingly. Your clients always call your app's URL and never see
+the Azure endpoint.
+
+Claude needs a Foundry resource, because an Azure OpenAI resource can't deploy
+Claude at all. Which endpoint host you configure doesn't affect this—a Foundry
+resource serves Claude on either host family the provider accepts.
+
+## Supported endpoints
+
+| Endpoint               | OpenAI-compatible models | Claude models |
+| ---------------------- | ------------------------ | ------------- |
+| `/v1/chat/completions` | ✅ Forwarded             | ✅ Translated |
+| `/v1/embeddings`       | ✅ Embedding models      | ❌            |
+| `/v1/messages`         | ❌                       | ✅ Forwarded  |
+| `/v1/responses`        | ✅ Models that serve it  | ❌            |
+
+**Forwarded** means the gateway sends your request on in the shape you wrote it.
+**Translated** means it converts between two API shapes, which limits you to the
+parameters listed below.
+
+Streaming (`stream: true`) works on chat completions, messages, and responses.
+
+The catalog carries three embedding models—`text-embedding-3-small`,
+`text-embedding-3-large` and `text-embedding-ada-002`—so `/v1/embeddings` works
+once you deploy one of them and map your deployment to it.
+
+`/v1/responses` serves the OpenAI Responses API, buffered and streaming. Support
+is **per deployed model**, not per resource: Azure serves Responses for
+particular model and version combinations, and a deployment outside that set
+answers `400` with `The requested operation is unsupported`—which is what an
+embedding deployment on this endpoint returns. Check Azure's own Responses API
+model list for your model and version before relying on it.
+
+The management operations on a stored response are forwarded too:
+
+| Operation                            | Does                                  |
+| ------------------------------------ | ------------------------------------- |
+| `GET /v1/responses/{id}`             | Retrieves the response                |
+| `GET /v1/responses/{id}/input_items` | Lists the input items of one response |
+| `DELETE /v1/responses/{id}`          | Deletes the response                  |
+
+Two things they need. Send **`store: true`** on the create, since your resource
+serves them only for a response it kept. And because these requests carry no
+body, there is no `model` in them for the gateway to route on—it uses the app's
+first configured completions model instead, so the app needs a
+[Model Filtering](../policies/ai-gateway-model-filtering-v2-inbound.mdx) policy
+whose completions allow list **starts with an OpenAI-compatible Azure
+deployment**. Without one the gateway answers `400`, asking for preselected
+routing, before reaching Azure.
+
+A Claude deployment doesn't work in that first slot, even though the allow list
+accepts it—Claude doesn't serve Responses at all, so the request still fails.
+
+Azure also takes a moment to make a new response readable. A `GET` immediately
+after a create can return `404` before it settles, so retry rather than treating
+the first `404` as final.
+
+Claude deployments don't serve Responses, so a Claude model on this endpoint
+fails with a `400` error—use
+[`/v1/messages`](#call-claude-models-on-the-messages-api) for those.
+
+Claude models work on `/v1/chat/completions` through the gateway's translation
+to the Messages API, which supports the core chat parameters: `messages`,
+`max_tokens`, `temperature`, `top_p`, `stop`, and `stream`. For tool use or
+other Anthropic-specific features, call
+[`/v1/messages`](#call-claude-models-on-the-messages-api) instead.
+
+This is the gateway's standard handling for Claude, not something specific to
+Azure—[Bedrock Mantle](./bedrock-mantle.mdx#supported-endpoints-by-model-family)
+serves Claude the same way. You get an ordinary `chat.completion` object back,
+with `choices` and `prompt_tokens`/`completion_tokens`. The one visible trace of
+the conversion is the response `id`, which keeps Anthropic's `msg_` prefix.
+Azure's OpenAI-compatible models are forwarded instead of translated, so their
+responses carry Azure's own fields, such as `content_filter_results`.
+
+## Before you begin
+
+You need:
+
+- An Azure subscription with an Azure OpenAI or Microsoft Foundry resource. Note
+  the resource name—it's the first label of the resource's endpoint host.
+- **At least one model deployed in that resource**, and its deployment name. The
+  gateway addresses deployments, so a resource with no deployments serves
+  nothing. Deploy models in the Azure portal under your resource's **Model
+  deployments**.
+- One of the resource's API keys, from the resource's **Keys and Endpoint** page
+  in the Azure portal.
+- An AI Gateway project in the Zuplo Portal.
+- An AI Gateway [app](./apps.mdx) to call the models from. The app page shows
+  the app's API URL, and its API key lives on the app's **API Key** tab.
+
+:::note
+
+Use a resource key, not a Microsoft Entra ID token. Azure accepts both on its
+own API, but the gateway stores long-lived resource keys, and the provider
+dialog rejects a pasted Entra token—those expire within hours.
+
+:::
+
+### Deploying Claude on Foundry
+
+Claude deployments carry extra requirements that Azure applies only to them, and
+the failures are easy to misread:
+
+- Azure asks for your **industry, organization name and country** when you
+  create an Anthropic-format deployment. Give a plain organization name;
+  punctuation in it has been enough to fail the deployment.
+- Azure validates none of it up front. A deployment can report success, spend a
+  few minutes provisioning, and then land in a **Failed** state with an
+  internal-error message. Check the deployment's provisioning state before you
+  configure the provider, and delete a failed deployment before reusing its
+  name.
+- If you deploy through the Azure REST API or a template rather than the portal,
+  use API version `2025-12-01` or later. Earlier versions silently ignore those
+  fields and then reject the request for not providing them.
+
+## Add the provider
+
+Adding or editing providers requires the **Edit** permission, granted to Zuplo
+account and project **Admins**—see
+[Managing Providers](./managing-providers.mdx).
+
+<Stepper>
+
+1. Open
+   [**Settings → AI Providers**](https://portal.zuplo.com/+/account/project/ai/settings/data-models)
+   in your AI Gateway project in the Zuplo Portal.
+
+1. Click the **Add Provider** button.
+
+1. In the **AI Provider** list, select **Azure AI** from the Default Providers
+   group.
+
+1. Review the **Provider Name**, which fills in as `azureai` (a second
+   configuration becomes `azureai-2`). You can replace it with your own name,
+   but only now—the name is permanent after creation, and it's the prefix in
+   every model reference: a provider named `azureai` serves `azureai/my-gpt`.
+
+1. In **Azure Resource Name**, enter your resource's name, such as
+   `my-resource`. Use the host family selector beside the field to pick
+   `openai.azure.com` or `services.ai.azure.com`, matching your resource's
+   endpoint. The gateway sends this provider's requests to
+   `https://<resource>.<host family>`.
+
+1. In **API Key**, paste one of the resource's keys.
+
+1. Select the models to enable, or click **Select All**. The picker shows
+   per-token prices, which the gateway uses to track cost per app. You can
+   change the selection later.
+
+1. Under **Deployment Names**, click **Add deployment** for each deployment
+   whose name differs from the model it serves. Enter the deployment name, then
+   pick the model it serves from the list. Skip this for deployments already
+   named after their model.
+
+1. Click **Create**.
+
+</Stepper>
+
+:::note
+
+Saving provider settings triggers an automatic production deployment of your
+gateway, because provider credentials are part of the deployed gateway. The
+change is live once the deployment completes.
+
+:::
+
+When you edit the provider later—see
+[Managing Providers](./managing-providers.mdx)—the **Azure Resource Name**, host
+family and deployment mappings all stay editable, and you can replace the API
+key. The **Provider Name** doesn't change.
+
+:::caution{title="Which endpoint form to enter"}
+
+Enter the resource name and pick a host family; don't paste a full URL. The
+provider accepts the `openai.azure.com` and `services.ai.azure.com` families
+only.
+
+A Foundry resource also has a `cognitiveservices.azure.com` address, and that's
+the one the Azure portal shows as the resource's endpoint. It isn't accepted
+here—use the resource name with one of the two families above instead. The
+resource is the same either way.
+
+:::
+
+## Verify the provider
+
+Once the deployment completes, send a chat completions request to your app's
+[Universal API](./universal-api.mdx) URL—shown at the top of the
+[app page](./apps.mdx)—with the app's API key as the bearer token and one of
+your deployments:
+
+```bash
+curl https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e/v1/chat/completions \
+  -H "Authorization: Bearer $ZUPLO_APP_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "azureai/my-gpt",
+    "messages": [{ "role": "user", "content": "Say hi" }]
+  }'
+```
+
+The URL is a sample—replace it with your app's API URL plus
+`/v1/chat/completions`, set `ZUPLO_APP_API_KEY` to the app's API key, and
+substitute your own deployment name for `my-gpt`.
+
+A `200` response confirms the provider works. If the request fails immediately
+after you save the provider, the deployment may not have finished—retry before
+debugging further.
+
+## Call OpenAI-compatible models
+
+Use any OpenAI client with your app's URL plus `/v1` as the base URL and the
+app's API key:
+
+```ts
+import OpenAI from "openai";
+
+const client = new OpenAI({
+  apiKey: process.env.ZUPLO_APP_API_KEY,
+  baseURL:
+    "https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e/v1",
+});
+
+const response = await client.chat.completions.create({
+  model: "azureai/my-gpt",
+  messages: [{ role: "user", content: "Summarize this ticket." }],
+});
+```
+
+The same call works with a Claude deployment—the gateway translates it to the
+Messages API—within the [translation's parameter subset](#supported-endpoints).
+
+## Call Claude models on the Messages API
+
+Claude deployments on a Foundry resource serve the native
+[Anthropic Messages API](./universal-api.mdx#supported-endpoints) at
+`/v1/messages`. With the Anthropic SDK, set `baseURL` to the app's URL _without_
+`/v1`—the SDK appends `/v1/messages` itself—and pass the app's API key as
+`authToken`, not `apiKey`:
+
+```ts
+import Anthropic from "@anthropic-ai/sdk";
+
+const client = new Anthropic({
+  baseURL:
+    "https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e",
+  authToken: process.env.ZUPLO_APP_API_KEY, // apiKey would send x-api-key, which the gateway ignores
+});
+
+const message = await client.messages.create({
+  model: "azureai/my-claude",
+  max_tokens: 1024,
+  messages: [{ role: "user", content: "Say hi" }],
+});
+```
+
+The gateway forwards the request body to Azure verbatim, so tool use, system
+prompts and streaming work as they do against Anthropic directly.
+
+`anthropic-beta` headers are the exception. The gateway forwards the
+long-context and prompt-caching betas and drops every other value, because Azure
+rejects an unrecognized beta by failing the whole request. A client asking for a
+different beta still gets an answer, without that feature.
+
+## What the model field reports in responses
+
+The `model` field in a response doesn't always say what you might expect, and
+what it says depends on the model you called.
+
+**Claude models and `/v1/responses` report your deployment name.** Everything
+else reports Azure's own model ID, which usually carries a version date the
+short name doesn't.
+
+| Endpoint               | Deployment `my-gpt` / `my-embed` / `my-claude` serves | `model` in the response   |
+| ---------------------- | ----------------------------------------------------- | ------------------------- |
+| `/v1/chat/completions` | `gpt-4.1-mini`                                        | `gpt-4.1-mini-2025-04-14` |
+| `/v1/responses`        | `gpt-4.1-mini`                                        | `my-gpt`                  |
+| `/v1/embeddings`       | `text-embedding-3-small`                              | `text-embedding-3-small`  |
+| `/v1/messages`         | `claude-haiku-4-5`                                    | `my-claude`               |
+| `/v1/chat/completions` | `claude-haiku-4-5`                                    | `my-claude`               |
+
+The same deployment reports two different values depending on the endpoint you
+call it on: chat completions answers with Azure's dated ID, Responses with the
+name you gave the deployment. That's Azure's own behavior on each surface, not
+something the gateway changes.
+
+A Claude model reports your deployment name on both endpoints—the native
+Messages API and the chat completions translation—but for a different reason:
+there the gateway builds the response itself and echoes back the model you asked
+for.
+
+One exception, if you stream Claude: the `message_start` event reports Azure's
+ID (`claude-haiku-4-5-20251001`) rather than your deployment name, because the
+gateway forwards each event as it arrives instead of rebuilding the response. So
+the same request reports two different values depending on whether you streamed
+it.
+
+Treat this field as informational either way. If you need to know which
+deployment served a request, use your own request metadata rather than parsing
+this.
+
+## Troubleshooting
+
+**A request fails with a deployment-not-found error.** The `model` value after
+the provider prefix must be a deployment name in your Azure resource, not a
+published model ID. Check the deployment list in the Azure portal under your
+resource's **Model deployments**, and confirm the deployment finished
+provisioning.
+
+**A request fails saying the deployment isn't mapped.** The deployment exists in
+Azure, but the gateway can't tell which model it serves, so it can't price the
+request. Edit the provider, add the deployment under **Deployment Names**, and
+pick its model. If the deployment is named after its model, selecting that model
+is enough. The error lists the names that currently work.
+
+**The dialog rejects your endpoint.** Enter the resource name, not a URL, and
+pick `openai.azure.com` or `services.ai.azure.com`. A Foundry resource's
+`cognitiveservices.azure.com` address isn't accepted—use the resource name with
+one of those two families. Resource names are 2 to 63 characters of lowercase
+letters, digits and hyphens, starting and ending with a letter or digit.
+
+**The dialog rejects your API key.** The gateway takes a resource key from the
+resource's **Keys and Endpoint** page. A Microsoft Entra ID access token is
+rejected—it expires within hours, so the connection would break the same day.
+
+**A Claude model returns an error on an Azure OpenAI resource.** Only Foundry
+resources deploy Claude. Confirm the deployment exists and is Anthropic-format,
+and see [Deploying Claude on Foundry](#deploying-claude-on-foundry) for the
+requirements Azure applies to those deployments.
+
+**A `400` error names `/v1/responses`.** Two different causes, and the message
+tells you which. If it names the provider or the endpoint, the deployment is a
+**Claude** one—Claude doesn't serve Responses, so call
+[`/v1/messages`](#call-claude-models-on-the-messages-api) instead, or
+`/v1/chat/completions` for the translated form. If it says
+`The requested operation is unsupported`, that came from Azure: the deployed
+model doesn't serve Responses. Support is per model and version, so check
+Azure's Responses API model list—an embedding deployment always fails this way.
+
+**A `400` asks for preselected routing.** You called `GET` or `DELETE` on a
+response, or listed its input items. Those requests carry no body, so there is
+no `model` for the gateway to route on and it falls back to the app's first
+configured completions model. Add a
+[Model Filtering](../policies/ai-gateway-model-filtering-v2-inbound.mdx) policy
+to the app with an **OpenAI-compatible** Azure deployment first in its
+completions allow list. A Claude deployment in that slot still fails, since
+Claude doesn't serve Responses.
+
+**A model in the picker fails at request time.** The picker lists the full Azure
+catalog, but a model only works once you deploy it in your resource, and model
+availability varies by region and resource kind. Deploy the model, then map your
+deployment name to it.
+
+## Next steps
+
+- [AI Providers](./providers.mdx)—the capability matrix across every supported
+  provider.
+- [Universal API](./universal-api.mdx)—the endpoints every app serves and how
+  model references work.
+- [Managing Providers](./managing-providers.mdx)—edit models, keys, and the
+  endpoint, and understand when changes deploy.
+- [AI Gateway Apps](./apps.mdx)—create the apps that call your Azure-backed
+  models.
+- [Model Filtering policy](../policies/ai-gateway-model-filtering-v2-inbound.mdx)—control
+  which models each app can call.

package/docs/ai-gateway/bedrock-mantle.mdx

@@ -72,9 +72,13 @@ either way.
 | ---------------------- | ---------------------------- | ------------- |
 | `/v1/chat/completions` | ✅ Forwarded                 | ✅ Translated |
 | `/v1/responses`        | ✅ Models that serve it      | ❌            |
-| `/v1/messages`         | ❌                           | ✅ Native     |
+| `/v1/messages`         | ❌                           | ✅ Forwarded  |
 | `/v1/embeddings`       | ❌ Mantle has no such models | ❌            |
 
+**Forwarded** means the gateway sends your request on in the shape you wrote it.
+**Translated** means it converts between two API shapes, which limits you to the
+parameters listed below.
+
 Streaming (`stream: true`) works on all three serving endpoints.
 
 On the OpenAI-compatible side, endpoint support is per model, decided by AWS: