zuplo 7.1.1 → 7.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/ai-gateway/apps.mdx +28 -10
- package/docs/ai-gateway/cookbooks/custom-fallback.mdx +96 -0
- package/docs/ai-gateway/cookbooks/dynamic-model-routing.mdx +119 -0
- package/docs/ai-gateway/custom-policies.mdx +213 -0
- package/docs/ai-gateway/custom-providers.mdx +10 -2
- package/docs/ai-gateway/fallback.mdx +48 -75
- package/docs/ai-gateway/getting-started.mdx +200 -132
- package/docs/ai-gateway/integrations/ai-sdk.mdx +71 -28
- package/docs/ai-gateway/integrations/claude-code.mdx +32 -8
- package/docs/ai-gateway/integrations/codex.mdx +28 -10
- package/docs/ai-gateway/integrations/goose.mdx +26 -19
- package/docs/ai-gateway/integrations/langchain.mdx +31 -15
- package/docs/ai-gateway/integrations/openai.mdx +28 -16
- package/docs/ai-gateway/introduction.mdx +73 -32
- package/docs/ai-gateway/managing-apps.mdx +62 -17
- package/docs/ai-gateway/managing-providers.mdx +23 -7
- package/docs/ai-gateway/managing-teams.mdx +29 -12
- package/docs/ai-gateway/policies/akamai-ai-firewall.mdx +92 -96
- package/docs/ai-gateway/policies/authentication.mdx +44 -0
- package/docs/ai-gateway/policies/comet-opik-tracing.mdx +40 -2
- package/docs/ai-gateway/policies/fallback-model.mdx +40 -0
- package/docs/ai-gateway/policies/galileo-tracing.mdx +36 -22
- package/docs/ai-gateway/policies/metering.mdx +47 -0
- package/docs/ai-gateway/policies/model-filtering.mdx +49 -0
- package/docs/ai-gateway/policies/overview.mdx +28 -0
- package/docs/ai-gateway/policies/semantic-cache.mdx +48 -0
- package/docs/ai-gateway/policy-chains.mdx +178 -0
- package/docs/ai-gateway/policy-templates.mdx +68 -0
- package/docs/ai-gateway/providers.mdx +23 -9
- package/docs/ai-gateway/source-control.mdx +88 -0
- package/docs/ai-gateway/teams.mdx +16 -10
- package/docs/ai-gateway/universal-api.mdx +39 -12
- package/docs/ai-gateway/usage-limits.mdx +87 -55
- package/docs/policies/cdn-cache-control-outbound/doc.md +13 -13
- package/package.json +5 -5
- package/docs/ai-gateway/guardrails.mdx +0 -65
|
@@ -1,6 +1,9 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: Galileo
|
|
3
|
-
sidebar_label: Galileo
|
|
2
|
+
title: Galileo tracing
|
|
3
|
+
sidebar_label: Galileo tracing
|
|
4
|
+
description:
|
|
5
|
+
Send LLM traces from the Zuplo AI Gateway to Galileo for monitoring,
|
|
6
|
+
debugging, and quality evaluation of your AI applications.
|
|
4
7
|
---
|
|
5
8
|
|
|
6
9
|
The Galileo Tracing policy integrates [Galileo AI](https://www.galileo.ai/) with
|
|
@@ -36,7 +39,7 @@ For each request, the policy automatically captures:
|
|
|
36
39
|
**Request Information**
|
|
37
40
|
|
|
38
41
|
- User prompts and messages
|
|
39
|
-
- Model parameters (temperature, max_tokens
|
|
42
|
+
- Model parameters (temperature, `max_tokens`, etc.)
|
|
40
43
|
- Request metadata (route, request ID)
|
|
41
44
|
- Timestamp and duration
|
|
42
45
|
|
|
@@ -64,23 +67,6 @@ For each request, the policy automatically captures:
|
|
|
64
67
|
2. Create a new project in your Galileo dashboard
|
|
65
68
|
3. Generate an API key specifically for use with Zuplo
|
|
66
69
|
|
|
67
|
-
2. ### Add the policy to your app
|
|
68
|
-
|
|
69
|
-
You can add the policy to any AI Gateway app by clicking on **Policies**,
|
|
70
|
-
then on **Add Policy** and select **Galileo Tracing**.
|
|
71
|
-
|
|
72
|
-

|
|
73
|
-
|
|
74
|
-
3. ### Configure the policy
|
|
75
|
-
|
|
76
|
-
You will need to enter the following information from your Galileo account
|
|
77
|
-
to configure the policy:
|
|
78
|
-
- **`apiKey`**: Your Galileo API key for authentication
|
|
79
|
-
- **`projectId`**: The Galileo project ID to send traces to
|
|
80
|
-
- **`logStreamId`**: The specific log stream within your project
|
|
81
|
-
- **`baseUrl`** (optional): Custom Galileo API endpoint (defaults to
|
|
82
|
-
`https://api.galileo.ai`)
|
|
83
|
-
|
|
84
70
|
:::note
|
|
85
71
|
|
|
86
72
|
The `projectId` and `logStreamId` are both found in the URL of the Galileo
|
|
@@ -92,6 +78,34 @@ For each request, the policy automatically captures:
|
|
|
92
78
|
|
|
93
79
|
:::
|
|
94
80
|
|
|
81
|
+
2. ### Configure the declaration in your gateway's repository
|
|
82
|
+
|
|
83
|
+
The scaffolded gateway declares the Galileo Tracing policy in
|
|
84
|
+
`config/policies.json`. Set its settings and push:
|
|
85
|
+
|
|
86
|
+
```json title="config/policies.json"
|
|
87
|
+
{
|
|
88
|
+
"name": "galileo-tracing-v2-inbound",
|
|
89
|
+
"policyType": "galileo-tracing-v2-inbound",
|
|
90
|
+
"handler": {
|
|
91
|
+
"export": "GalileoTracingV2InboundPolicy",
|
|
92
|
+
"module": "$import(@zuplo/runtime)",
|
|
93
|
+
"options": {
|
|
94
|
+
"apiKey": "your-galileo-api-key",
|
|
95
|
+
"projectId": "3e71c65e-48b6-4f5d-842d-0851c4704f95",
|
|
96
|
+
"logStreamId": "f8c71402-1f6b-4f5b-b073-1de999e6a8ea"
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
3. ### Add the policy to your app
|
|
103
|
+
|
|
104
|
+
Open the app's **Policies** tab, click **Add Policy**, and select **Galileo
|
|
105
|
+
Tracing**. Place it at the end of the chain so traces reflect the request
|
|
106
|
+
the provider actually receives. The app inherits the configuration from the
|
|
107
|
+
declaration.
|
|
108
|
+
|
|
95
109
|
</Stepper>
|
|
96
110
|
|
|
97
111
|
### Key Metrics
|
|
@@ -139,8 +153,8 @@ Common issues and solutions:
|
|
|
139
153
|
|
|
140
154
|
- **Authentication Errors**: Check that your Galileo API key is valid and has
|
|
141
155
|
proper permissions
|
|
142
|
-
- **Configuration Issues**: Ensure the
|
|
143
|
-
|
|
156
|
+
- **Configuration Issues**: Ensure the policy declaration's environment
|
|
157
|
+
variables are set in your project settings
|
|
144
158
|
|
|
145
159
|
## Additional Resources
|
|
146
160
|
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Budgets and Costs
|
|
3
|
+
sidebar_label: Budgets and Costs
|
|
4
|
+
description:
|
|
5
|
+
Record an app's usage and enforce its budgets for spend, tokens, and requests.
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
`ai-gateway-metering-v2`
|
|
9
|
+
|
|
10
|
+
The **Budgets and Costs** policy records what an app uses and enforces its
|
|
11
|
+
budgets. It meters three things—spend, tokens, and requests—each with
|
|
12
|
+
independent daily and monthly periods, and each with an optional warning
|
|
13
|
+
threshold that notifies without blocking.
|
|
14
|
+
|
|
15
|
+
When a limit is exceeded, the policy either routes the request to the model
|
|
16
|
+
selection's `quotaFallback` if [Fallback Model](./fallback-model.mdx) supplied
|
|
17
|
+
one, or rejects it with `429`.
|
|
18
|
+
|
|
19
|
+
An app's own limits aren't the only ones that apply. The policy also checks the
|
|
20
|
+
parent team's and the gateway's limits, which are enforced centrally rather than
|
|
21
|
+
from this policy's options—so a request can be blocked by a limit that isn't
|
|
22
|
+
visible in the app's chain.
|
|
23
|
+
|
|
24
|
+
Place Budgets and Costs after [Model Filtering](./model-filtering.mdx) and
|
|
25
|
+
[Fallback Model](./fallback-model.mdx), so an exceeded budget can activate the
|
|
26
|
+
quota fallback.
|
|
27
|
+
|
|
28
|
+
:::caution{title="Budgets fail open by default"}
|
|
29
|
+
|
|
30
|
+
`throwOnFailure` defaults to `false`, so if the metering service is unavailable
|
|
31
|
+
the request proceeds unmetered—no limit is checked, even for an app that's
|
|
32
|
+
already over budget. Set it to `true` to reject the request instead.
|
|
33
|
+
|
|
34
|
+
:::
|
|
35
|
+
|
|
36
|
+
:::note
|
|
37
|
+
|
|
38
|
+
Each limit period carries an `enabled` flag next to its number, and so does each
|
|
39
|
+
warning threshold. A period enforces nothing until `enabled` is `true` _and_
|
|
40
|
+
`limit` is set. Watch for this when editing options directly, such as in
|
|
41
|
+
`config/policies.json` or a [policy template](../policy-templates.mdx).
|
|
42
|
+
|
|
43
|
+
:::
|
|
44
|
+
|
|
45
|
+
For the portal workflow and the gateway and team levels see
|
|
46
|
+
[Usage Limits](../usage-limits.mdx); for every option see the
|
|
47
|
+
[generated reference](/policies/ai-gateway-metering-v2-inbound).
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Model Filtering
|
|
3
|
+
sidebar_label: Model Filtering
|
|
4
|
+
description:
|
|
5
|
+
Restrict an app to specific models with an allow list or block list, and set
|
|
6
|
+
the default model.
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
`ai-gateway-model-filtering-v2`
|
|
10
|
+
|
|
11
|
+
Model Filtering decides which models an app may use and creates the model
|
|
12
|
+
selection the rest of the chain works from. Without it, an app can reach any
|
|
13
|
+
model offered by the providers assigned to its [team](../teams.mdx).
|
|
14
|
+
|
|
15
|
+
Configure each capability—completions, embeddings—in one of two modes. The modes
|
|
16
|
+
are mutually exclusive per capability.
|
|
17
|
+
|
|
18
|
+
- An **allow list** exposes a curated set of models. The first entry is the
|
|
19
|
+
default when a request names no model.
|
|
20
|
+
- A **block list** permits every available model except the ones named. Each
|
|
21
|
+
request must name its model, because a block list supplies no default.
|
|
22
|
+
|
|
23
|
+
Models are written as `providerName/model`, where `providerName` is the provider
|
|
24
|
+
name configured in your gateway—for example `openai/gpt-5`.
|
|
25
|
+
|
|
26
|
+
## What a rejected request gets
|
|
27
|
+
|
|
28
|
+
| Request | Response |
|
|
29
|
+
| ------------------------------------------------ | ----------------------------- |
|
|
30
|
+
| A model the rules don't allow | `403`, listing what's allowed |
|
|
31
|
+
| No `model` named, in block list mode | `400` |
|
|
32
|
+
| A capability the policy doesn't configure at all | `403` |
|
|
33
|
+
| A provider name that isn't in the catalog | `400` |
|
|
34
|
+
|
|
35
|
+
Rejections are returned as problem details, and the `detail` names what the
|
|
36
|
+
caller should have sent—the allowed models for a blocked model, or the gateway's
|
|
37
|
+
configured provider names for an unrecognized one. The
|
|
38
|
+
[Akamai AI Firewall](./akamai-ai-firewall.mdx) is the exception among the
|
|
39
|
+
policies here: it answers in the provider's own error format instead.
|
|
40
|
+
|
|
41
|
+
Place Model Filtering first in the chain—it creates the model selection that
|
|
42
|
+
[Fallback Model](./fallback-model.mdx) and [Budgets and Costs](./metering.mdx)
|
|
43
|
+
build on.
|
|
44
|
+
|
|
45
|
+
For the portal workflow see
|
|
46
|
+
[Restricting models](../managing-apps.mdx#restricting-models); for every option
|
|
47
|
+
see the [generated reference](/policies/ai-gateway-model-filtering-v2-inbound).
|
|
48
|
+
When model selection needs logic instead of a list, see
|
|
49
|
+
[Dynamic model routing](../cookbooks/dynamic-model-routing.mdx).
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: AI Gateway policies
|
|
3
|
+
sidebar_label: Overview
|
|
4
|
+
description: Every policy an AI Gateway app can run, and what each one does.
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
The gateway's `config/policies.json` declares which policies apps may use, and
|
|
8
|
+
each app picks from that menu in the order it wants. See
|
|
9
|
+
[Policy Chains](../policy-chains.mdx) for how a chain executes and how options
|
|
10
|
+
and secrets are inherited.
|
|
11
|
+
|
|
12
|
+
| Policy | What it does |
|
|
13
|
+
| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
|
14
|
+
| [API key authentication](./authentication.mdx) | Requires an app API key and resolves the calling app from it |
|
|
15
|
+
| [Model Filtering](./model-filtering.mdx) | Restricts the app to an allow list or block list of models, and supplies the default model |
|
|
16
|
+
| [Fallback Model](./fallback-model.mdx) | Adds a backup model for provider errors and timeouts, and a cheaper model for when a budget runs out |
|
|
17
|
+
| [Budgets and Costs](./metering.mdx) | Records usage and enforces the app's budgets for spend, tokens, and requests |
|
|
18
|
+
| [Semantic cache](./semantic-cache.mdx) | Returns a cached response when a new prompt is close enough to one already seen |
|
|
19
|
+
| [Akamai AI Firewall](./akamai-ai-firewall.mdx) | Scans prompts and completions with your Akamai configuration and blocks what it denies |
|
|
20
|
+
| [Comet Opik tracing](./comet-opik-tracing.mdx) | Sends a trace of each request and response to Comet Opik |
|
|
21
|
+
| [Galileo tracing](./galileo-tracing.mdx) | Sends a trace of each request and response to Galileo |
|
|
22
|
+
| [Configuration Executor](../policy-chains.mdx#configuration-executor) | Loads the app's configuration and runs its chain—this is what makes app chains work |
|
|
23
|
+
|
|
24
|
+
The Configuration Executor sits on the gateway's route rather than in an app's
|
|
25
|
+
chain. API key authentication works in either spot: in an app's chain it
|
|
26
|
+
protects that app alone, and on the route it covers every app. Any
|
|
27
|
+
[custom policy](../custom-policies.mdx) you declare joins the menu alongside
|
|
28
|
+
these.
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Semantic cache
|
|
3
|
+
sidebar_label: Semantic cache
|
|
4
|
+
description:
|
|
5
|
+
Return a cached response when a new prompt is close enough to one the app has
|
|
6
|
+
already answered.
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
`ai-gateway-semantic-cache-v2-inbound`
|
|
10
|
+
|
|
11
|
+
The semantic cache answers a request from a previous response when the new
|
|
12
|
+
prompt is semantically similar to one already seen, skipping the provider
|
|
13
|
+
entirely. One policy handles both halves: it checks the cache on the way in and,
|
|
14
|
+
on a miss, stores the provider's response on the way out. There's no separate
|
|
15
|
+
outbound policy to add.
|
|
16
|
+
|
|
17
|
+
A cache hit answers the request, so entries after it in the chain don't run.
|
|
18
|
+
Place it after [Budgets and Costs](./metering.mdx) so hits still count toward
|
|
19
|
+
the app's request limits.
|
|
20
|
+
|
|
21
|
+
## Options
|
|
22
|
+
|
|
23
|
+
| Option | Default | What it does |
|
|
24
|
+
| ----------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------- |
|
|
25
|
+
| `semanticTolerance` | `0.2` | How loose a match to accept. A response is reused at similarity ≥ 1 − tolerance, so `0.2` needs 0.8 |
|
|
26
|
+
| `expirationSecondsTtl` | `3600` | How long a cached response stays valid, up to 30 days |
|
|
27
|
+
| `maxConversationLength` | `3` | Longest conversation to cache, in messages |
|
|
28
|
+
| `endpoints` | Chat Completions, Anthropic Messages | Which API shapes to cache |
|
|
29
|
+
| `namespace` | The app's ID | Cache partition. Only applies when the request has no app configuration |
|
|
30
|
+
|
|
31
|
+
## What isn't cached
|
|
32
|
+
|
|
33
|
+
- Conversations longer than `maxConversationLength`—three messages by
|
|
34
|
+
default—aren't cached at all.
|
|
35
|
+
- The Responses API isn't supported. The cache covers Chat Completions and
|
|
36
|
+
Anthropic Messages.
|
|
37
|
+
- The cache always fails open: if the cache is unavailable, the request goes to
|
|
38
|
+
the provider as normal.
|
|
39
|
+
|
|
40
|
+
One app can never read another's cached responses. The namespace is fixed to the
|
|
41
|
+
app's ID whenever the request resolves to an app, so the `namespace` option only
|
|
42
|
+
matters on a gateway that doesn't identify apps.
|
|
43
|
+
|
|
44
|
+
## Reading cache outcomes
|
|
45
|
+
|
|
46
|
+
Each response carries the outcome as a `Cache-Status` header under the cache
|
|
47
|
+
name `zp-aigw-sem-cache`, alongside `x-ai-gateway-cache` (`HIT` or `MISS`) and
|
|
48
|
+
`x-ai-gateway-cache-similarity`.
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: AI Gateway Policy Chains
|
|
3
|
+
sidebar_label: Policy Chains
|
|
4
|
+
description:
|
|
5
|
+
Each AI Gateway app runs an ordered chain of policies selected from the
|
|
6
|
+
gateway's policies.json. Learn how the chain executes, how options and secrets
|
|
7
|
+
are inherited, and the recommended policy order.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
Every request to an AI Gateway app runs down through the app's **policy chain**:
|
|
11
|
+
an ordered list of policies that control model access, budgets, caching,
|
|
12
|
+
guardrails, and tracing for that app. Different apps on the same gateway can run
|
|
13
|
+
completely different chains.
|
|
14
|
+
|
|
15
|
+
The chain works in two layers:
|
|
16
|
+
|
|
17
|
+
- **The gateway declares the menu.** `config/policies.json` in the gateway's
|
|
18
|
+
[repository](./source-control.mdx) declares every policy that apps may use
|
|
19
|
+
—built-in AI Gateway policies and any [custom policies](./custom-policies.mdx)
|
|
20
|
+
you write. Changing the menu is a repository change that takes effect on the
|
|
21
|
+
next production deploy.
|
|
22
|
+
- **Each app selects from the menu.** An app's chain—which policies run, in what
|
|
23
|
+
order, with what settings—is edited on the app's **Policies** tab in the
|
|
24
|
+
portal. Chain changes apply within about a minute, with no deploy.
|
|
25
|
+
|
|
26
|
+
No policy is required. An app whose chain is empty still serves requests: each
|
|
27
|
+
request names its own model, and no model restrictions, budgets, guardrails, or
|
|
28
|
+
caching apply. Every control is something you add.
|
|
29
|
+
|
|
30
|
+
A request passes through whatever the app selected before the AI Gateway handler
|
|
31
|
+
calls the provider:
|
|
32
|
+
|
|
33
|
+
```text
|
|
34
|
+
Request
|
|
35
|
+
→ Model Filtering ─┐
|
|
36
|
+
→ Budgets and Costs ├─ the app's policy chain
|
|
37
|
+
→ Custom policy ─┘
|
|
38
|
+
→ AI Gateway handler
|
|
39
|
+
→ LLM provider
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Editing a chain
|
|
43
|
+
|
|
44
|
+
The app's **Policies** tab shows the chain in execution order. Drag entries to
|
|
45
|
+
reorder them, and click **Add Policy** to add one from the gateway's menu.
|
|
46
|
+
|
|
47
|
+
Each declared policy has a single **Add** button. For a policy with settings,
|
|
48
|
+
Add opens its settings screen, where you confirm or change the values before the
|
|
49
|
+
policy joins the chain. For a policy with no settings screen, Add puts it
|
|
50
|
+
straight into the chain with the options declared in `policies.json`.
|
|
51
|
+
|
|
52
|
+
A policy that's already in the chain shows **Already added** instead of a
|
|
53
|
+
button—edit it in the chain instead.
|
|
54
|
+
|
|
55
|
+
Entries seeded by a [team policy template](./policy-templates.mdx) may be
|
|
56
|
+
locked: a lock icon means the template controls whether this app may edit or
|
|
57
|
+
remove that entry.
|
|
58
|
+
|
|
59
|
+
## How the chain executes
|
|
60
|
+
|
|
61
|
+
- Entries run in the listed order, after the gateway's route-level policies and
|
|
62
|
+
before the AI Gateway handler calls the provider.
|
|
63
|
+
- A policy can answer the request itself—for example, a semantic cache hit or a
|
|
64
|
+
guardrail block. Later entries don't run in that case.
|
|
65
|
+
- An entry can be disabled to keep its configuration without running it.
|
|
66
|
+
Disabled entries are still validated, so a disabled entry can't reference a
|
|
67
|
+
policy that's no longer declared.
|
|
68
|
+
- An app with an empty chain runs no app-selected policies; requests go straight
|
|
69
|
+
to the AI Gateway handler.
|
|
70
|
+
- The chain is inbound-only. An app can't select an outbound policy, and the
|
|
71
|
+
gateway rejects a chain entry whose declaration is outbound-only. Policies
|
|
72
|
+
that need the response—Semantic Cache and the Akamai AI Firewall—register a
|
|
73
|
+
response hook from their inbound position, so one entry covers both
|
|
74
|
+
directions.
|
|
75
|
+
- There's no allow-list of policy types. Any policy declared in `policies.json`
|
|
76
|
+
that works as an inbound policy is a valid chain entry, including a
|
|
77
|
+
[custom code policy](./custom-policies.mdx). The only exclusions are the
|
|
78
|
+
[Configuration Executor](#configuration-executor) itself and outbound-only
|
|
79
|
+
policies.
|
|
80
|
+
- Chain validation is all-or-nothing: if any entry is invalid—for example, it
|
|
81
|
+
references a policy that isn't declared in the gateway's `policies.json`—the
|
|
82
|
+
request fails with an error identifying the entry to fix, and no entries run.
|
|
83
|
+
The gateway never guesses.
|
|
84
|
+
|
|
85
|
+
### Options and secrets
|
|
86
|
+
|
|
87
|
+
An entry either **inherits** the options declared in `policies.json` or
|
|
88
|
+
**replaces them completely** with its own—options are never merged
|
|
89
|
+
field-by-field. What matters is whether the entry has an `options` key at all:
|
|
90
|
+
an entry with empty options replaces the declared options with an empty object,
|
|
91
|
+
which breaks a policy that has required settings. Omit options entirely to
|
|
92
|
+
inherit.
|
|
93
|
+
|
|
94
|
+
Keep credentials in the declaration's options and let entries inherit them, so
|
|
95
|
+
the gateway's repository—not app configuration—holds them. See
|
|
96
|
+
[Configure credentials](./custom-policies.mdx#configure-credentials).
|
|
97
|
+
|
|
98
|
+
## Recommended order
|
|
99
|
+
|
|
100
|
+
```text
|
|
101
|
+
Model Filtering → Fallback Model → Budgets and Costs → Semantic Cache
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
- **Model Filtering** first: it accepts or rejects the request and creates the
|
|
105
|
+
model selection.
|
|
106
|
+
- **Fallback Model** second: it only enriches an existing model selection and
|
|
107
|
+
never creates one. A chain with Fallback Model but no Model Filtering does
|
|
108
|
+
nothing at all—the policy logs a warning and passes the request through
|
|
109
|
+
unchanged. That's also why a fallback can never bypass the filter.
|
|
110
|
+
- **Budgets and Costs** third: it needs the resolved selection so an exceeded
|
|
111
|
+
budget can activate the quota fallback.
|
|
112
|
+
- **Semantic Cache** after Budgets and Costs: cache hits still count toward
|
|
113
|
+
request limits.
|
|
114
|
+
|
|
115
|
+
Tracing policies work well at the end of the chain, so traces reflect the
|
|
116
|
+
request the earlier policies produced.
|
|
117
|
+
|
|
118
|
+
## Built-in policies
|
|
119
|
+
|
|
120
|
+
See the [policies overview](./policies/overview.mdx) for every policy an app can
|
|
121
|
+
run, what each one does, and the identifier it needs in `config/policies.json`.
|
|
122
|
+
Any [custom policy](./custom-policies.mdx) declared in the repository appears in
|
|
123
|
+
the Add Policy dialog alongside the built-in ones.
|
|
124
|
+
|
|
125
|
+
## Configuration Executor
|
|
126
|
+
|
|
127
|
+
`ai-gateway-configuration-executor-v2-inbound` is the policy that makes the
|
|
128
|
+
two-layer model work. It loads the app's configuration and then runs the app's
|
|
129
|
+
stored inbound chain, instantiating only policies already declared in
|
|
130
|
+
`config/policies.json`. Every scaffolded gateway declares it and puts it on the
|
|
131
|
+
AI Gateway route—that route entry is what gives an app's chain somewhere to run.
|
|
132
|
+
|
|
133
|
+
Keep it on the **route**, not in an app's chain: it refuses to run itself
|
|
134
|
+
recursively.
|
|
135
|
+
|
|
136
|
+
Its only option is `cacheTtlSeconds`, which sets how long the gateway caches a
|
|
137
|
+
loaded app configuration. The default is `10` seconds, `10` is also the minimum,
|
|
138
|
+
and a lower value throws a configuration error—as does any other option key. The
|
|
139
|
+
cache is skipped when the [authentication policy](#authentication) already
|
|
140
|
+
loaded the configuration for the request.
|
|
141
|
+
|
|
142
|
+
## Authentication
|
|
143
|
+
|
|
144
|
+
The `ai-gateway-auth-v2-inbound` policy—**API Key Authentication** in the
|
|
145
|
+
portal—requires callers to present the app's API key, and it applies per app:
|
|
146
|
+
add it to an app's chain to require a key for that app alone. A new team's
|
|
147
|
+
[policy template](./policy-templates.mdx) includes it as a locked entry, so apps
|
|
148
|
+
created in the team require keys from the start. To require keys for every app
|
|
149
|
+
on a route regardless of what each app's chain selects, put the policy on the
|
|
150
|
+
route in the gateway's repository, before the Configuration Executor.
|
|
151
|
+
|
|
152
|
+
Where the policy applies, clients send the app's API key as a bearer token and
|
|
153
|
+
the gateway resolves the calling app from the key; for a missing or invalid key
|
|
154
|
+
the gateway returns a `401 Unauthorized`. Its `authHeader` and `authScheme`
|
|
155
|
+
options let the gateway read the key from somewhere other than
|
|
156
|
+
`Authorization: Bearer`—`x-api-key`, for example. The gateway caches key checks
|
|
157
|
+
for `cacheTtlSeconds`, which defaults to 10 seconds, so a revoked key can keep
|
|
158
|
+
working until the cache expires. Each app's key appears on its app page—see
|
|
159
|
+
[Apps](./apps.mdx).
|
|
160
|
+
|
|
161
|
+
:::caution{title="A gateway without the authentication policy is open"}
|
|
162
|
+
|
|
163
|
+
When the policy sits neither on the route nor in the app's chain, the gateway
|
|
164
|
+
resolves the app from the `{app_id}` segment of the request URL and serves the
|
|
165
|
+
request with no key of any kind. Anyone who knows the URL can spend against the
|
|
166
|
+
app's providers and budget. Only run an open gateway when something else, such
|
|
167
|
+
as network isolation, controls who can reach it.
|
|
168
|
+
|
|
169
|
+
:::
|
|
170
|
+
|
|
171
|
+
## Next steps
|
|
172
|
+
|
|
173
|
+
- [Custom Policies](./custom-policies.mdx): write your own policy and add it to
|
|
174
|
+
a chain
|
|
175
|
+
- [Policy Templates](./policy-templates.mdx): seed consistent chains across a
|
|
176
|
+
team's apps
|
|
177
|
+
- [Usage Limits](./usage-limits.mdx): budgets at the gateway, team, and app
|
|
178
|
+
levels
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: AI Gateway Policy Templates
|
|
3
|
+
sidebar_label: Policy Templates
|
|
4
|
+
description:
|
|
5
|
+
Team policy templates give every new app in a team a consistent starting
|
|
6
|
+
policy chain, with per-policy controls over what each app may change or
|
|
7
|
+
remove.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
A team's **policy template** is the policy chain that apps created in that team
|
|
11
|
+
start with. Use templates to make sure every app in a team begins with your
|
|
12
|
+
required policies—Budgets and Costs with a sensible budget, a guardrail, your
|
|
13
|
+
custom compliance policy—while controlling what individual apps may change.
|
|
14
|
+
|
|
15
|
+
## How templates apply
|
|
16
|
+
|
|
17
|
+
When an app is created in a team, the team's template is copied into the app as
|
|
18
|
+
its starting [policy chain](./policy-chains.mdx). From that point on, the chain
|
|
19
|
+
belongs to the app.
|
|
20
|
+
|
|
21
|
+
A new team's template starts with the **API Key Authentication** policy as a
|
|
22
|
+
locked entry, so every app created in the team requires its API key from the
|
|
23
|
+
start. A new sub-team starts by
|
|
24
|
+
[inheriting its parent's template](#inheriting-from-a-parent-team) instead of
|
|
25
|
+
defining its own.
|
|
26
|
+
|
|
27
|
+
:::caution{title="Template edits don't change existing apps"}
|
|
28
|
+
|
|
29
|
+
A template is a starting point, not a live link. Editing a team's template
|
|
30
|
+
affects apps created afterwards—it never changes the chains of the team's
|
|
31
|
+
existing apps. To change a running app's behavior, edit that app's chain on its
|
|
32
|
+
Policies tab.
|
|
33
|
+
|
|
34
|
+
:::
|
|
35
|
+
|
|
36
|
+
## Editing a template
|
|
37
|
+
|
|
38
|
+
Open a team and select its **Policies** tab. The template editor works like the
|
|
39
|
+
app chain editor: add policies from the gateway's declared menu, order them, and
|
|
40
|
+
configure their options.
|
|
41
|
+
|
|
42
|
+
Each policy in the template carries two checkboxes that control what an app
|
|
43
|
+
starting from the template may do with that entry. Both start checked—a policy
|
|
44
|
+
you add to a template stays locked until you uncheck the boxes:
|
|
45
|
+
|
|
46
|
+
| Checkbox | When checked |
|
|
47
|
+
| ------------------------------------------------------- | ------------------------------------------------------------------- |
|
|
48
|
+
| **Cannot edit** (`canEdit: false` in configuration) | Apps can't change the entry's options—the template's settings apply |
|
|
49
|
+
| **Cannot delete** (`canRemove: false` in configuration) | Apps can't remove the entry from their chain |
|
|
50
|
+
|
|
51
|
+
Locked entries appear in the app's chain with a lock icon and the note "Locked
|
|
52
|
+
by the team policy template".
|
|
53
|
+
|
|
54
|
+
## Inheriting from a parent team
|
|
55
|
+
|
|
56
|
+
A sub-team can use its parent team's template instead of defining its own,
|
|
57
|
+
controlled by the **Inherit from parent** switch on the sub-team's Policies tab.
|
|
58
|
+
While inheriting, the sub-team's template is read-only—it shows the parent's
|
|
59
|
+
effective template. Turn the switch off to give the sub-team its own template:
|
|
60
|
+
the editor starts from a copy of the parent's effective template, which you then
|
|
61
|
+
adjust. Turning inheritance back on discards the sub-team's own template in
|
|
62
|
+
favor of the parent's.
|
|
63
|
+
|
|
64
|
+
## Next steps
|
|
65
|
+
|
|
66
|
+
- [Policy Chains](./policy-chains.mdx): how an app's chain executes
|
|
67
|
+
- [Managing Teams](./managing-teams.mdx): creating teams and sub-teams
|
|
68
|
+
- [Custom Policies](./custom-policies.mdx): put your own policy in a template
|
|
@@ -1,10 +1,14 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: AI Providers
|
|
3
3
|
sidebar_label: Overview
|
|
4
|
+
description:
|
|
5
|
+
The AI providers and capabilities the Zuplo AI Gateway supports, including
|
|
6
|
+
OpenAI, Anthropic, Google, Mistral, xAI, and OpenAI-compatible custom
|
|
7
|
+
providers.
|
|
4
8
|
---
|
|
5
9
|
|
|
6
10
|
Zuplo's AI Gateway supports integration with various AI providers, allowing you
|
|
7
|
-
to leverage different models and services for your
|
|
11
|
+
to leverage different models and services for your apps.
|
|
8
12
|
|
|
9
13
|
## Supported Providers
|
|
10
14
|
|
|
@@ -20,14 +24,24 @@ Zuplo currently supports the following AI providers:
|
|
|
20
24
|
|
|
21
25
|
The following capabilities are supported across providers:
|
|
22
26
|
|
|
23
|
-
| Provider | Chat Completions |
|
|
24
|
-
| -------------------------- | ---------------- |
|
|
25
|
-
| OpenAI | ✅ | ✅
|
|
26
|
-
| Anthropic | ✅ |
|
|
27
|
-
| Google | ✅ | ✅
|
|
28
|
-
| Mistral | ✅ | ✅
|
|
29
|
-
| xAI | ✅ | ✅
|
|
30
|
-
| OpenAI-compatible (Custom) | ✅ | ✅
|
|
27
|
+
| Provider | Chat Completions | Embeddings | Responses | Messages |
|
|
28
|
+
| -------------------------- | ---------------- | ---------- | --------- | -------- |
|
|
29
|
+
| OpenAI | ✅ | ✅ | ✅ | ❌ |
|
|
30
|
+
| Anthropic | ✅ | ❌ | ❌ | ✅ |
|
|
31
|
+
| Google | ✅ | ✅ | ❌ | ❌ |
|
|
32
|
+
| Mistral | ✅ | ✅ | ❌ | ❌ |
|
|
33
|
+
| xAI | ✅ | ✅ | ❌ | ❌ |
|
|
34
|
+
| OpenAI-compatible (Custom) | ✅ | ✅ | ❌ | ❌ |
|
|
35
|
+
|
|
36
|
+
**Responses** is the OpenAI Responses API (`/v1/responses`), and **Messages** is
|
|
37
|
+
the native Anthropic Messages API (`/v1/messages`). See the
|
|
38
|
+
[Universal API](./universal-api.mdx#supported-endpoints) for the endpoint each
|
|
39
|
+
capability maps to.
|
|
40
|
+
|
|
41
|
+
Apps reference a provider's models as `providerName/model`—for example
|
|
42
|
+
`openai/gpt-5-mini` or `anthropic/claude-sonnet-4-6`—where `providerName` is the
|
|
43
|
+
name you give the provider configuration. See the
|
|
44
|
+
[Universal API](./universal-api.mdx).
|
|
31
45
|
|
|
32
46
|
If you need support for additional providers or capabilities, please contact us
|
|
33
47
|
at [support@zuplo.com](mailto:support@zuplo.com). We're continually working to
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: AI Gateway Source Control
|
|
3
|
+
sidebar_label: Source Control
|
|
4
|
+
description:
|
|
5
|
+
Every AI Gateway project deploys from a Git repository you own. Learn what the
|
|
6
|
+
scaffolded repository contains, how pushes deploy to production, and which
|
|
7
|
+
changes require a deploy versus applying live.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
Every AI Gateway project is backed by a Git repository that you own. The
|
|
11
|
+
repository contains the gateway's routes and policy declarations, so gateway
|
|
12
|
+
changes go through your normal review workflow, and custom policies are plain
|
|
13
|
+
TypeScript files you commit alongside them.
|
|
14
|
+
|
|
15
|
+
## Connecting a repository
|
|
16
|
+
|
|
17
|
+
A new AI Gateway project must be connected to a repository before it can deploy.
|
|
18
|
+
Zuplo supports GitHub, GitLab, Bitbucket, and Azure DevOps—see
|
|
19
|
+
[Source Control and Deployment](../concepts/source-control-and-deployment.mdx)
|
|
20
|
+
for what each provider supports.
|
|
21
|
+
|
|
22
|
+
The project shows a **Let's Get You Deployed** screen with two options:
|
|
23
|
+
|
|
24
|
+
- **Create New Repo** starts a new repository with the name prefilled. With
|
|
25
|
+
GitHub, this opens GitHub directly; create the repository, then return to the
|
|
26
|
+
portal to connect it.
|
|
27
|
+
- Connect an existing repository. Use an empty one so the gateway's source has
|
|
28
|
+
the repository to itself.
|
|
29
|
+
|
|
30
|
+
When you connect, Zuplo adds the gateway's source to the repository, pushes it,
|
|
31
|
+
and starts the first production deployment.
|
|
32
|
+
|
|
33
|
+
:::note
|
|
34
|
+
|
|
35
|
+
If your default branch requires pull requests, Zuplo pushes the gateway source
|
|
36
|
+
to a setup branch and surfaces a pull request for you to merge. The first
|
|
37
|
+
deployment starts once the pull request lands on the default branch.
|
|
38
|
+
|
|
39
|
+
:::
|
|
40
|
+
|
|
41
|
+
## What the repository contains
|
|
42
|
+
|
|
43
|
+
The scaffolded gateway is a small, readable Zuplo project:
|
|
44
|
+
|
|
45
|
+
| File | Purpose |
|
|
46
|
+
| ---------------------- | ------------------------------------------------------------------------------- |
|
|
47
|
+
| `config/ai.oas.json` | The gateway's route, which the AI Gateway handler serves |
|
|
48
|
+
| `config/policies.json` | The menu of policies apps can add to their [policy chains](./policy-chains.mdx) |
|
|
49
|
+
| `zuplo.jsonc` | Project configuration |
|
|
50
|
+
| `env.example` | Example environment variables |
|
|
51
|
+
| `tsconfig.json` | TypeScript configuration for custom policy modules |
|
|
52
|
+
|
|
53
|
+
Add custom policy modules under `modules/` and declare them in
|
|
54
|
+
`config/policies.json`—see [Custom Policies](./custom-policies.mdx).
|
|
55
|
+
|
|
56
|
+
## How deployments work
|
|
57
|
+
|
|
58
|
+
The repository's default branch is what production runs. With GitHub, every push
|
|
59
|
+
to it deploys automatically. With GitLab, Bitbucket, and Azure DevOps, your own
|
|
60
|
+
CI/CD pipeline deploys by calling `zuplo deploy`—see
|
|
61
|
+
[Source Control and Deployment](../concepts/source-control-and-deployment.mdx).
|
|
62
|
+
|
|
63
|
+
There is no in-portal code editor for AI Gateway projects—the repository is the
|
|
64
|
+
source of truth, and you edit it with your normal tools.
|
|
65
|
+
|
|
66
|
+
Three categories of changes take effect differently:
|
|
67
|
+
|
|
68
|
+
| Change | Takes effect |
|
|
69
|
+
| ----------------------------------------------------------------- | ---------------------------------------------------- |
|
|
70
|
+
| Repository changes (routes, `policies.json`, custom policy code) | On the next deploy of the default branch |
|
|
71
|
+
| App policy chains, teams, budgets, and templates (portal changes) | Within about a minute, no deploy needed |
|
|
72
|
+
| Provider settings and environment variables | Automatically, via a rebuild and deploy Zuplo starts |
|
|
73
|
+
|
|
74
|
+
:::note
|
|
75
|
+
|
|
76
|
+
Provider API keys are stored as
|
|
77
|
+
[environment variables](../articles/environment-variables.mdx), so saving
|
|
78
|
+
provider settings or environment variables triggers an automatic production
|
|
79
|
+
deployment.
|
|
80
|
+
|
|
81
|
+
:::
|
|
82
|
+
|
|
83
|
+
## Next steps
|
|
84
|
+
|
|
85
|
+
- [Policy Chains](./policy-chains.mdx): how apps select policies from
|
|
86
|
+
`config/policies.json`
|
|
87
|
+
- [Custom Policies](./custom-policies.mdx): add your own policy to the
|
|
88
|
+
repository
|