zuplo 7.9.5 → 7.9.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/ai-gateway/apps.mdx +28 -16
- package/docs/ai-gateway/custom-policies.mdx +7 -7
- package/docs/ai-gateway/custom-providers.mdx +1 -1
- package/docs/ai-gateway/fallback.mdx +5 -5
- package/docs/ai-gateway/integrations/ai-sdk.mdx +2 -2
- package/docs/ai-gateway/integrations/claude-code.mdx +41 -33
- package/docs/ai-gateway/integrations/claude-desktop.mdx +37 -34
- package/docs/ai-gateway/integrations/codex.mdx +42 -31
- package/docs/ai-gateway/integrations/github-copilot.mdx +39 -37
- package/docs/ai-gateway/integrations/goose.mdx +27 -30
- package/docs/ai-gateway/integrations/langchain.mdx +2 -2
- package/docs/ai-gateway/integrations/openai.mdx +2 -2
- package/docs/ai-gateway/jev.mdx +344 -0
- package/docs/ai-gateway/managing-apps.mdx +22 -22
- package/docs/ai-gateway/managing-pools.mdx +87 -0
- package/docs/ai-gateway/managing-providers.mdx +4 -4
- package/docs/ai-gateway/overview.mdx +38 -26
- package/docs/ai-gateway/policy-chains.mdx +7 -7
- package/docs/ai-gateway/policy-templates.mdx +22 -22
- package/docs/ai-gateway/pools.mdx +53 -0
- package/docs/ai-gateway/providers.mdx +32 -2
- package/docs/ai-gateway/source-control.mdx +2 -2
- package/docs/ai-gateway/universal-api.mdx +4 -0
- package/docs/ai-gateway/usage-limits.mdx +49 -47
- package/docs/ai-gateway/user-apps.mdx +166 -0
- package/docs/articles/accounts/roles-and-permissions.mdx +92 -63
- package/docs/concepts/ai-gateway.mdx +24 -20
- package/docs/policies/ai-gateway-akamai-firewall-inbound/doc.md +1 -1
- package/docs/policies/ai-gateway-metering-inbound/schema.json +16 -2
- package/docs/policies/ai-gateway-smart-router-inbound/doc.md +172 -29
- package/docs/policies/ai-gateway-smart-router-inbound/intro.md +4 -3
- package/docs/policies/ai-gateway-smart-router-inbound/schema.json +177 -30
- package/package.json +5 -5
- package/docs/ai-gateway/managing-teams.mdx +0 -87
- package/docs/ai-gateway/teams.mdx +0 -49
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
title: Managing Apps
|
|
3
3
|
description:
|
|
4
4
|
Create, edit, and delete AI Gateway apps. Apps are created with just a name
|
|
5
|
-
and a
|
|
5
|
+
and a pool; models, app-specific budgets, and other behavior live on the app's
|
|
6
6
|
policy chain.
|
|
7
7
|
---
|
|
8
8
|
|
|
@@ -11,16 +11,16 @@ feature in a larger codebase. Each app has its own API key, which the gateway
|
|
|
11
11
|
checks once you add the
|
|
12
12
|
[authentication policy](./policy-chains.mdx#authentication); usage is tracked
|
|
13
13
|
per app whether or not that policy applies. Each app belongs to a
|
|
14
|
-
[
|
|
14
|
+
[pool](./pools.mdx), which controls who can access and manage the app. Provider
|
|
15
15
|
access is governed separately by the user's Zuplo account and Zuplo project
|
|
16
16
|
roles. A Zuplo account contains one or more projects; this AI Gateway and its
|
|
17
|
-
providers,
|
|
17
|
+
providers, pools, and apps belong to one Zuplo project. See
|
|
18
18
|
[Role Permissions](../articles/accounts/roles-and-permissions.mdx).
|
|
19
19
|
|
|
20
20
|
:::note{title="App permissions"}
|
|
21
21
|
|
|
22
|
-
A
|
|
23
|
-
**Member** can access the
|
|
22
|
+
A pool **Admin** can create, edit, and delete apps owned by that pool. A pool
|
|
23
|
+
**Member** can access the pool's apps but can't manage them. Both roles require
|
|
24
24
|
permission to view the Zuplo project that contains the AI Gateway.
|
|
25
25
|
|
|
26
26
|
:::
|
|
@@ -29,25 +29,25 @@ permission to view the Zuplo project that contains the AI Gateway.
|
|
|
29
29
|
|
|
30
30
|
<Stepper>
|
|
31
31
|
|
|
32
|
-
1. Open the [Apps
|
|
33
|
-
|
|
32
|
+
1. Open the [Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of
|
|
33
|
+
your AI Gateway project in the Zuplo Portal.
|
|
34
34
|
|
|
35
|
-
1. Select the
|
|
36
|
-
|
|
37
|
-
|
|
35
|
+
1. Select the pool that will own the app, then click **Create App** in the
|
|
36
|
+
pool's header. You can also choose **New App** from the menu next to **New
|
|
37
|
+
Pool** above the tree.
|
|
38
38
|
|
|
39
39
|
1. Enter the **App name**.
|
|
40
40
|
|
|
41
|
-
1. Confirm the **
|
|
41
|
+
1. Confirm the **Pool**—it's prefilled with the pool you started from.
|
|
42
42
|
|
|
43
43
|
1. Click **Create app**
|
|
44
44
|
|
|
45
45
|
</Stepper>
|
|
46
46
|
|
|
47
47
|
That's the whole dialog. The portal then takes you to the app's **Policies**
|
|
48
|
-
tab. If the app's
|
|
48
|
+
tab. If the app's pool defines a [policy template](./policy-templates.mdx), the
|
|
49
49
|
new app starts with that chain; otherwise the chain is empty and the app is
|
|
50
|
-
subject to no app-specific controls until you add policies.
|
|
50
|
+
subject to no app-specific controls until you add policies. Pool and gateway
|
|
51
51
|
usage limits still apply.
|
|
52
52
|
|
|
53
53
|
## Restricting models
|
|
@@ -88,24 +88,24 @@ over—see [Dynamic model routing](./cookbooks/dynamic-model-routing.mdx).
|
|
|
88
88
|
App budgets live on the **Budgets and Costs** policy in the app's chain. Open
|
|
89
89
|
the app's **Policies** tab and add budget rules capping spending, tokens, or
|
|
90
90
|
requests over an hourly, daily, weekly, or monthly period—either for the app as
|
|
91
|
-
a whole or per metadata value, so each caller gets its own budget.
|
|
91
|
+
a whole or per metadata value, so each caller gets its own budget. Pool and
|
|
92
92
|
gateway limits apply to every app automatically, even when the app's chain
|
|
93
93
|
doesn't include Budgets and Costs. See [Usage Limits](./usage-limits.mdx).
|
|
94
94
|
|
|
95
95
|
## Editing an App
|
|
96
96
|
|
|
97
97
|
To edit an app, open the
|
|
98
|
-
[Apps
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
98
|
+
[Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of your AI
|
|
99
|
+
Gateway project in the Zuplo Portal. Select the app you want to edit. The app's
|
|
100
|
+
behavior—models, app-specific budgets, caching, guardrails—is edited on the
|
|
101
|
+
**Policies** tab; the app's name lives on the **Settings** tab. Make your
|
|
102
102
|
changes and click the **Save** button. Policy chain changes apply within about a
|
|
103
103
|
minute.
|
|
104
104
|
|
|
105
105
|
## Deleting an App
|
|
106
106
|
|
|
107
107
|
To delete an app, open the
|
|
108
|
-
[Apps
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
108
|
+
[Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of your AI
|
|
109
|
+
Gateway project in the Zuplo Portal. Select the app you want to delete, open its
|
|
110
|
+
**Settings** tab, and click the **Delete App** button. You will be prompted to
|
|
111
|
+
confirm the deletion.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Managing Pools
|
|
3
|
+
description:
|
|
4
|
+
Create pools and sub-pools, add members, and set pool-level policy templates
|
|
5
|
+
and usage limits in your AI Gateway project.
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Pools in the Zuplo AI Gateway grant users access to apps. Pools are
|
|
9
|
+
hierarchical, so access propagates to sub-pools only when the user also has
|
|
10
|
+
permission to view the Zuplo project that contains the AI Gateway. Pool or
|
|
11
|
+
sub-pool membership doesn't grant Zuplo project access. Pools are also where you
|
|
12
|
+
define usage limits and [policy templates](./policy-templates.mdx).
|
|
13
|
+
|
|
14
|
+
## Creating a Pool
|
|
15
|
+
|
|
16
|
+
Creating a pool requires Admin rights on the Zuplo project or Zuplo account for
|
|
17
|
+
a top-level pool, or on the parent pool for a sub-pool. As a convenience, the
|
|
18
|
+
Portal disables the **New Pool** button when you don't have the required rights.
|
|
19
|
+
|
|
20
|
+
<Stepper>
|
|
21
|
+
|
|
22
|
+
1. Open the [Apps](https://portal.zuplo.com/+/account/project/ai/pools) tab of
|
|
23
|
+
your AI Gateway project in the Zuplo Portal.
|
|
24
|
+
|
|
25
|
+
1. Click **New Pool** above the tree.
|
|
26
|
+
|
|
27
|
+
1. Choose where the pool sits. Once the gateway has at least one pool, the
|
|
28
|
+
dialog shows a parent picker above the name field; leave it on **No parent
|
|
29
|
+
pool** for a top-level pool, or pick a pool to create a sub-pool under it.
|
|
30
|
+
|
|
31
|
+
1. Enter the name of your pool.
|
|
32
|
+
|
|
33
|
+
1. Click **Create Pool**
|
|
34
|
+
|
|
35
|
+
</Stepper>
|
|
36
|
+
|
|
37
|
+
You can also create a sub-pool from the parent itself: select it in the tree and
|
|
38
|
+
click **Create Sub-Pool** in the pool's header.
|
|
39
|
+
|
|
40
|
+
After creating a pool, use its tabs to configure it:
|
|
41
|
+
|
|
42
|
+
- **Policy Template**—the pool's [policy template](./policy-templates.mdx), the
|
|
43
|
+
starting policy chain for apps created in the pool.
|
|
44
|
+
- **Usage & Limits**—shared budgets for the pool, its sub-pools, and their apps.
|
|
45
|
+
See [Usage Limits](./usage-limits.mdx).
|
|
46
|
+
- **Members**—who belongs to the pool.
|
|
47
|
+
|
|
48
|
+
## Adding Pool Members
|
|
49
|
+
|
|
50
|
+
Pool members are the users who belong to a pool and can access its apps,
|
|
51
|
+
provided they also have permission to view the Zuplo project. AI provider access
|
|
52
|
+
is separate and depends on the user's Zuplo account or Zuplo project role.
|
|
53
|
+
|
|
54
|
+
<Stepper>
|
|
55
|
+
|
|
56
|
+
1. Open the [Apps](https://portal.zuplo.com/+/account/project/ai/pools) tab of
|
|
57
|
+
your AI Gateway project in the Zuplo Portal.
|
|
58
|
+
|
|
59
|
+
1. Select the pool you want to add members to.
|
|
60
|
+
|
|
61
|
+
1. Click the **Members** tab.
|
|
62
|
+
|
|
63
|
+
1. Click **Add member**.
|
|
64
|
+
|
|
65
|
+
1. Type the email address of the user you want to add. If the user is already a
|
|
66
|
+
member of your Zuplo account, they are added directly to the pool. Otherwise,
|
|
67
|
+
they are invited to join the Zuplo account and then added to the pool.
|
|
68
|
+
|
|
69
|
+
1. Select the role for the user:
|
|
70
|
+
- **Member**: Can access apps owned by the pool.
|
|
71
|
+
- **Admin**: Can manage the pool's settings, members, and apps.
|
|
72
|
+
|
|
73
|
+
:::caution{title="The Member role requires RBAC"}
|
|
74
|
+
|
|
75
|
+
Role-based access control (RBAC) is an optional enterprise add-on on your
|
|
76
|
+
Zuplo account. Without it, every invited user is an **Admin**. For more
|
|
77
|
+
information see the
|
|
78
|
+
[document on roles and permissions](../articles/accounts/roles-and-permissions.mdx).
|
|
79
|
+
|
|
80
|
+
:::
|
|
81
|
+
|
|
82
|
+
1. Click **Add member** to confirm.
|
|
83
|
+
|
|
84
|
+
1. The user will receive an email notification if they're being invited to the
|
|
85
|
+
Zuplo account.
|
|
86
|
+
|
|
87
|
+
</Stepper>
|
|
@@ -18,7 +18,7 @@ project role grants access only to that project. Adding, editing, or deleting
|
|
|
18
18
|
providers and their models requires the **Edit** permission granted to Zuplo
|
|
19
19
|
account and Zuplo project **Admins**. A Zuplo account **Developer**, or a Zuplo
|
|
20
20
|
project **Developer** or **Member**, can view providers but can't manage them.
|
|
21
|
-
|
|
21
|
+
Pool membership doesn't grant provider access. See
|
|
22
22
|
[Role Permissions](../articles/accounts/roles-and-permissions.mdx).
|
|
23
23
|
|
|
24
24
|
:::
|
|
@@ -58,9 +58,9 @@ To add a new AI provider to your Zuplo AI Gateway, follow these steps:
|
|
|
58
58
|
Project ID** and takes a service account JSON key file instead of an API key,
|
|
59
59
|
and [Azure AI](./azure-ai.mdx) asks for an **Azure Resource Name** and a host
|
|
60
60
|
family, plus a mapping for any deployment named differently from the model it
|
|
61
|
-
serves. [OpenRouter](./openrouter.mdx)
|
|
62
|
-
nothing beyond the key, being a single global host
|
|
63
|
-
or project to enter.
|
|
61
|
+
serves. [OpenRouter](./openrouter.mdx) and [Jev](./jev.mdx) are the opposite
|
|
62
|
+
case: they ask for nothing beyond the key, each being a single global host
|
|
63
|
+
with no endpoint, region or project to enter.
|
|
64
64
|
|
|
65
65
|
1. Select the model or models you want to use with this provider. The available
|
|
66
66
|
models will depend on the selected provider. This can be changed later.
|
|
@@ -8,10 +8,10 @@ description:
|
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
Zuplo's AI Gateway acts as an intelligent proxy layer that sits between your
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
11
|
+
apps, your people, and LLM providers like OpenAI, Anthropic, Google, Mistral,
|
|
12
|
+
and xAI. Instead of your apps communicating directly with these providers, all
|
|
13
|
+
requests flow through the Zuplo AI Gateway, which streams responses while
|
|
14
|
+
applying policies, controls, and monitoring.
|
|
15
15
|
|
|
16
16
|
Each piece of software that calls the gateway registers as an **app**—your
|
|
17
17
|
support chatbot is one app; an internal coding agent is another. A single AI
|
|
@@ -23,6 +23,11 @@ you can write your own in TypeScript, and apps add them to their chains like any
|
|
|
23
23
|
other policy. Chain changes apply within about a minute, with no redeploy. See
|
|
24
24
|
[Apps](./apps.mdx).
|
|
25
25
|
|
|
26
|
+
People call the gateway too. Instead of creating an app for each person, every
|
|
27
|
+
AI Gateway project has a **User App**: people create their own personal API key
|
|
28
|
+
and use it from tools such as Claude Code or Codex. Each person draws from their
|
|
29
|
+
own budget, and admins see who spent what. See [User Apps](./user-apps.mdx).
|
|
30
|
+
|
|
26
31
|
## Key Benefits
|
|
27
32
|
|
|
28
33
|
**Provider Independence**: Switch between LLM providers (OpenAI, Anthropic,
|
|
@@ -30,7 +35,7 @@ Google, Mistral, xAI, and more) dynamically without modifying app code.
|
|
|
30
35
|
Configure model access through the gateway rather than hard coding it into your
|
|
31
36
|
apps.
|
|
32
37
|
|
|
33
|
-
**Cost Control**: Set independent spending limits at the gateway,
|
|
38
|
+
**Cost Control**: Set independent spending limits at the gateway, pool, and app
|
|
34
39
|
levels. Configure hourly, daily, weekly, or monthly rules that warn or block
|
|
35
40
|
when usage reaches a limit.
|
|
36
41
|
|
|
@@ -38,9 +43,10 @@ when usage reaches a limit.
|
|
|
38
43
|
attempts and prevent PII leakage in both requests and responses through
|
|
39
44
|
integrated AI firewall policies.
|
|
40
45
|
|
|
41
|
-
**Self-Service Access**: Developers can create apps and
|
|
42
|
-
needing direct access to provider API
|
|
43
|
-
once, and
|
|
46
|
+
**Self-Service Access**: Developers can create apps, and anyone on the project
|
|
47
|
+
can create a personal API key, without needing direct access to provider API
|
|
48
|
+
keys. Administrators configure providers once, and apps and people use them
|
|
49
|
+
securely.
|
|
44
50
|
|
|
45
51
|
**Extensibility**: Write custom policies in TypeScript—content filters, dynamic
|
|
46
52
|
model routing, plan-based limits—and let apps add them to their chains like any
|
|
@@ -57,7 +63,8 @@ time-to-first-byte metrics, and spending patterns across your gateway.
|
|
|
57
63
|
|
|
58
64
|
Your code sends its LLM requests to your AI Gateway instead of directly to a
|
|
59
65
|
provider. Each app has its own URL, API key, and policy chain, so one gateway
|
|
60
|
-
serves
|
|
66
|
+
serves apps with completely different configurations. People send requests to
|
|
67
|
+
the User App's URL with their personal key instead.
|
|
61
68
|
|
|
62
69
|
Each request arrives at the URL of the app it belongs to. The gateway identifies
|
|
63
70
|
that app, runs its policy chain (model access, app-specific cost controls,
|
|
@@ -67,7 +74,7 @@ exposing underlying provider credentials.
|
|
|
67
74
|
|
|
68
75
|
Every app-specific control is opt-in. An app whose policy chain is empty runs no
|
|
69
76
|
app-selected controls: each request names its own model, and no app-specific
|
|
70
|
-
model restrictions, budgets, guardrails, or caching apply. Gateway and
|
|
77
|
+
model restrictions, budgets, guardrails, or caching apply. Gateway and pool
|
|
71
78
|
usage limits still apply.
|
|
72
79
|
|
|
73
80
|
## Core Features
|
|
@@ -78,16 +85,16 @@ Configure multiple LLM providers within a single AI Gateway project. Supported
|
|
|
78
85
|
providers include OpenAI, Anthropic, Google, Mistral, xAI, Microsoft Azure
|
|
79
86
|
(through [Azure AI](./azure-ai.mdx)), Amazon Bedrock (through
|
|
80
87
|
[Bedrock Mantle](./bedrock-mantle.mdx)), Google Cloud
|
|
81
|
-
([Vertex AI](./vertex-ai.mdx)), [OpenRouter](./openrouter.mdx),
|
|
82
|
-
|
|
83
|
-
full list of providers and supported
|
|
84
|
-
`providerName/model`—for example
|
|
85
|
-
models from several providers.
|
|
88
|
+
([Vertex AI](./vertex-ai.mdx)), [OpenRouter](./openrouter.mdx), TypeSafe's
|
|
89
|
+
System One model ([Jev](./jev.mdx)), and OpenAI-compatible custom providers. See
|
|
90
|
+
[AI Providers](./providers.mdx) for the full list of providers and supported
|
|
91
|
+
capabilities. Apps reference models as `providerName/model`—for example
|
|
92
|
+
`openai/gpt-5-mini`—so a single app can use models from several providers.
|
|
86
93
|
|
|
87
94
|
### Source-Controlled Gateway
|
|
88
95
|
|
|
89
96
|
A new AI Gateway project deploys without a Git repository, so you can configure
|
|
90
|
-
providers,
|
|
97
|
+
providers, pools, and apps right away. Connect a
|
|
91
98
|
[Git repository](./source-control.mdx) when you want to write custom policies or
|
|
92
99
|
review gateway changes in Git. After you connect, the default branch is what
|
|
93
100
|
production runs.
|
|
@@ -96,22 +103,24 @@ production runs.
|
|
|
96
103
|
|
|
97
104
|
Each app runs its own ordered chain of policies—model filtering, fallback
|
|
98
105
|
models, app-specific budgets, semantic caching, guardrails, tracing, and any
|
|
99
|
-
custom policies declared in the repository.
|
|
106
|
+
custom policies declared in the repository. Pool policy templates give new apps
|
|
100
107
|
a consistent starting pipeline. See [Policy Chains](./policy-chains.mdx) and
|
|
101
108
|
[Custom Policies](./custom-policies.mdx).
|
|
102
109
|
|
|
103
|
-
###
|
|
110
|
+
### Pool Hierarchy & Budgets
|
|
104
111
|
|
|
105
|
-
Organize
|
|
112
|
+
Organize apps into pools with hierarchical structures. Set a budget at each
|
|
106
113
|
node. Each node totals its own usage and all descendant usage, and any exceeded
|
|
107
114
|
**Block** limit rejects the request:
|
|
108
115
|
|
|
109
116
|
- **Gateway**: Limits across the Zuplo project (for example, $1,000/day).
|
|
110
|
-
- **
|
|
111
|
-
$500/day for the Credit
|
|
117
|
+
- **Pools**: Limits across the pool, its sub-pools, and their apps (for example,
|
|
118
|
+
$500/day for the Credit Pool).
|
|
112
119
|
- **Apps**: Limits for one app's traffic.
|
|
113
120
|
|
|
114
|
-
|
|
121
|
+
The [User App](./user-apps.mdx) has a per-person budget instead, such as $50 per
|
|
122
|
+
day for each person. Gateway limits apply to it as well. See
|
|
123
|
+
[Usage Limits](./usage-limits.mdx).
|
|
115
124
|
|
|
116
125
|
### App Configuration
|
|
117
126
|
|
|
@@ -126,18 +135,21 @@ Each app gets its own:
|
|
|
126
135
|
|
|
127
136
|
A **Zuplo account** is the top-level container for your members and projects.
|
|
128
137
|
Each **Zuplo project** belongs to one account, and an AI Gateway is a type of
|
|
129
|
-
Zuplo project. The providers,
|
|
138
|
+
Zuplo project. The providers, pools, and apps you configure for a gateway all
|
|
130
139
|
belong to that project.
|
|
131
140
|
|
|
132
141
|
Account roles can grant access across the projects in a Zuplo account, while
|
|
133
|
-
project roles grant access to one Zuplo project. AI Gateway
|
|
134
|
-
access to a
|
|
142
|
+
project roles grant access to one Zuplo project. AI Gateway pool roles add
|
|
143
|
+
access to a pool and its apps within that project. The **AI User** project role
|
|
144
|
+
grants only what someone needs to call the User App with their own key. See
|
|
135
145
|
[Role Permissions](../articles/accounts/roles-and-permissions.mdx) for the exact
|
|
136
146
|
permissions each role grants.
|
|
137
147
|
|
|
138
148
|
## Use Cases
|
|
139
149
|
|
|
140
|
-
- **Multi-tenant AI Apps**: Enforce spending limits per customer or
|
|
150
|
+
- **Multi-tenant AI Apps**: Enforce spending limits per customer or pool
|
|
151
|
+
- **Everyday AI for your people**: Give everyone a personal API key with their
|
|
152
|
+
own budget for coding assistants and chat tools
|
|
141
153
|
- **Agent Development**: Build AI agents that can switch providers without code
|
|
142
154
|
changes
|
|
143
155
|
- **Cost Management**: Control and monitor LLM spending across your gateway
|
|
@@ -44,7 +44,7 @@ Request
|
|
|
44
44
|
## Editing a chain
|
|
45
45
|
|
|
46
46
|
The app's **Policies** tab shows the chain in execution order. Drag entries to
|
|
47
|
-
reorder them, and click **Add
|
|
47
|
+
reorder them, and click **Add policy** to add one from the gateway's menu.
|
|
48
48
|
|
|
49
49
|
Each declared policy has a single **Add** button. For a policy with settings,
|
|
50
50
|
Add opens its settings screen, where you confirm or change the values before the
|
|
@@ -54,7 +54,7 @@ straight into the chain with the options declared in `policies.json`.
|
|
|
54
54
|
A policy that's already in the chain shows **Already added** instead of a
|
|
55
55
|
button—edit it in the chain instead.
|
|
56
56
|
|
|
57
|
-
Entries seeded by a [
|
|
57
|
+
Entries seeded by a [pool policy template](./policy-templates.mdx) may be
|
|
58
58
|
locked: a lock icon means the template controls whether this app may edit or
|
|
59
59
|
remove that entry.
|
|
60
60
|
|
|
@@ -149,7 +149,7 @@ fallback. Cache hits return `429` instead of using a fallback.
|
|
|
149
149
|
See the [policies overview](./policies/overview.mdx) for every policy an app can
|
|
150
150
|
run, what each one does, and the identifier it needs in `config/policies.json`.
|
|
151
151
|
Any [custom policy](./custom-policies.mdx) declared in the repository appears in
|
|
152
|
-
the Add
|
|
152
|
+
the Add policy dialog alongside the built-in ones.
|
|
153
153
|
|
|
154
154
|
## Configuration Executor
|
|
155
155
|
|
|
@@ -178,8 +178,8 @@ also affect when a budget change takes effect.
|
|
|
178
178
|
The `ai-gateway-auth-inbound` policy—**API Key Authentication** in the
|
|
179
179
|
portal—requires callers to present the app's API key, and it applies per app:
|
|
180
180
|
add it to an app's chain to require a key for that app alone. A new top-level
|
|
181
|
-
|
|
182
|
-
entry—and sub-
|
|
181
|
+
pool's [policy template](./policy-templates.mdx) includes it as a locked
|
|
182
|
+
entry—and sub-pools inherit that template by default—so apps created in the pool
|
|
183
183
|
require keys from the start.
|
|
184
184
|
|
|
185
185
|
Where the policy applies, clients send the app's API key as a bearer token and
|
|
@@ -206,6 +206,6 @@ isolation, controls who can reach it.
|
|
|
206
206
|
- [Custom Policies](./custom-policies.mdx): write your own policy and add it to
|
|
207
207
|
a chain
|
|
208
208
|
- [Policy Templates](./policy-templates.mdx): seed consistent chains across a
|
|
209
|
-
|
|
210
|
-
- [Usage Limits](./usage-limits.mdx): budgets at the gateway,
|
|
209
|
+
pool's apps
|
|
210
|
+
- [Usage Limits](./usage-limits.mdx): budgets at the gateway, pool, and app
|
|
211
211
|
levels
|
|
@@ -2,46 +2,46 @@
|
|
|
2
2
|
title: AI Gateway Policy Templates
|
|
3
3
|
sidebar_label: Policy Templates
|
|
4
4
|
description:
|
|
5
|
-
|
|
5
|
+
Pool policy templates give every new app in a pool a consistent starting
|
|
6
6
|
policy chain, with per-policy controls over what each app may change or
|
|
7
7
|
remove.
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
A
|
|
11
|
-
start with. Use templates to give every app in a
|
|
10
|
+
A pool's **policy template** is the policy chain that apps created in that pool
|
|
11
|
+
start with. Use templates to give every app in a pool the same app-specific
|
|
12
12
|
controls—such as Model Filtering, Budgets and Costs with an app budget, a
|
|
13
13
|
guardrail, or a custom compliance policy—while controlling what individual apps
|
|
14
|
-
may change.
|
|
14
|
+
may change. Pool and gateway limits don't depend on the template.
|
|
15
15
|
|
|
16
16
|
## How templates apply
|
|
17
17
|
|
|
18
|
-
When an app is created in a
|
|
18
|
+
When an app is created in a pool, the pool's template is copied into the app as
|
|
19
19
|
its starting [policy chain](./policy-chains.mdx). From that point on, the chain
|
|
20
20
|
belongs to the app.
|
|
21
21
|
|
|
22
|
-
A new
|
|
23
|
-
locked entry, so every app created in the
|
|
24
|
-
start. A new sub-
|
|
25
|
-
[inheriting its parent's template](#inheriting-from-a-parent-
|
|
22
|
+
A new pool's template starts with the **API Key Authentication** policy as a
|
|
23
|
+
locked entry, so every app created in the pool requires its API key from the
|
|
24
|
+
start. A new sub-pool starts by
|
|
25
|
+
[inheriting its parent's template](#inheriting-from-a-parent-pool) instead of
|
|
26
26
|
defining its own.
|
|
27
27
|
|
|
28
28
|
:::caution{title="Template edits don't change existing apps"}
|
|
29
29
|
|
|
30
|
-
A template is a starting point, not a live link. Editing a
|
|
31
|
-
affects apps created afterwards—it never changes the chains of the
|
|
30
|
+
A template is a starting point, not a live link. Editing a pool's template
|
|
31
|
+
affects apps created afterwards—it never changes the chains of the pool's
|
|
32
32
|
existing apps. To change a running app's behavior, edit that app's chain on its
|
|
33
33
|
Policies tab.
|
|
34
34
|
|
|
35
35
|
:::
|
|
36
36
|
|
|
37
|
-
New apps inherit their initial budget settings from their
|
|
37
|
+
New apps inherit their initial budget settings from their pool's policy
|
|
38
38
|
template. Configure these settings with **Budgets and Costs** in the template.
|
|
39
|
-
To set a budget shared by all apps in the
|
|
39
|
+
To set a budget shared by all apps in the pool, use **Usage & Limits**. See
|
|
40
40
|
[Usage Limits](./usage-limits.mdx).
|
|
41
41
|
|
|
42
42
|
## Editing a template
|
|
43
43
|
|
|
44
|
-
Open a
|
|
44
|
+
Open a pool and select its **Policy Template** tab. The template editor works
|
|
45
45
|
like the app chain editor: add policies from the gateway's declared menu, order
|
|
46
46
|
them, and configure their options.
|
|
47
47
|
|
|
@@ -55,22 +55,22 @@ you add to a template stays locked until you uncheck the boxes:
|
|
|
55
55
|
| **Cannot delete** | Apps can't remove the entry from their chain |
|
|
56
56
|
|
|
57
57
|
Locked entries appear in the app's chain with a lock icon and the note "Locked
|
|
58
|
-
by the
|
|
58
|
+
by the pool policy template".
|
|
59
59
|
|
|
60
|
-
## Inheriting from a parent
|
|
60
|
+
## Inheriting from a parent pool
|
|
61
61
|
|
|
62
|
-
A sub-
|
|
63
|
-
controlled by the **Inherit from parent** switch on the sub-
|
|
62
|
+
A sub-pool can use its parent pool's template instead of defining its own,
|
|
63
|
+
controlled by the **Inherit from parent** switch on the sub-pool's **Policy
|
|
64
64
|
Template** tab. The switch drafts like the page's other edits and applies when
|
|
65
|
-
you click **Save changes**. While inheriting, the sub-
|
|
65
|
+
you click **Save changes**. While inheriting, the sub-pool's template is
|
|
66
66
|
read-only—it shows the parent's effective template. Turn the switch off to give
|
|
67
|
-
the sub-
|
|
67
|
+
the sub-pool its own template: the editor starts from a copy of the parent's
|
|
68
68
|
effective template, which you then adjust. Turning inheritance back on and
|
|
69
|
-
saving discards the sub-
|
|
69
|
+
saving discards the sub-pool's own template in favor of the parent's—before you
|
|
70
70
|
save, flip the switch again to restore your draft.
|
|
71
71
|
|
|
72
72
|
## Next steps
|
|
73
73
|
|
|
74
74
|
- [Policy Chains](./policy-chains.mdx): how an app's chain executes
|
|
75
|
-
- [Managing
|
|
75
|
+
- [Managing Pools](./managing-pools.mdx): creating pools and sub-pools
|
|
76
76
|
- [Custom Policies](./custom-policies.mdx): put your own policy in a template
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: AI Gateway Pools
|
|
3
|
+
sidebar_label: Overview
|
|
4
|
+
description:
|
|
5
|
+
Pools group AI Gateway apps, carry shared usage limits and policy templates,
|
|
6
|
+
and control who can manage their apps, with nested, hierarchical sub-pools.
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
Pools group AI Gateway apps so they can share budgets, a starting policy chain,
|
|
10
|
+
and access. Each pool can have multiple members and sub-pools. Use pools to
|
|
11
|
+
group apps by department, product, customer, or any other logical grouping.
|
|
12
|
+
|
|
13
|
+
Pools hold [apps](./apps.mdx), not people's personal usage. People who call the
|
|
14
|
+
gateway with their own key use the [User App](./user-apps.mdx), which has a
|
|
15
|
+
per-person budget and isn't part of any pool.
|
|
16
|
+
|
|
17
|
+
## Pools & Apps
|
|
18
|
+
|
|
19
|
+
Every [app](./apps.mdx)—each service or integration that calls the AI
|
|
20
|
+
Gateway—belongs to exactly one pool. What a pool member may do to an app depends
|
|
21
|
+
on that member's role.
|
|
22
|
+
|
|
23
|
+
Pools also carry a [policy template](./policy-templates.mdx)—the policy chain
|
|
24
|
+
that apps created in the pool start with—and [usage limits](./usage-limits.mdx)
|
|
25
|
+
that apply to the pool's own usage and all descendant sub-pools and apps.
|
|
26
|
+
|
|
27
|
+
## Members
|
|
28
|
+
|
|
29
|
+
Members are the users who belong to a pool. Their permissions combine their role
|
|
30
|
+
in the Zuplo account, the Zuplo project that contains the AI Gateway, and the
|
|
31
|
+
pool.
|
|
32
|
+
|
|
33
|
+
There are two roles at the pool level:
|
|
34
|
+
|
|
35
|
+
- **Member**: Can access apps owned by the pool.
|
|
36
|
+
- **Admin**: Can manage the pool's settings, members, and apps.
|
|
37
|
+
|
|
38
|
+
Pool roles don't grant access to AI providers. Providers are configured for the
|
|
39
|
+
Zuplo project, and access to view or manage them depends on the user's Zuplo
|
|
40
|
+
account or Zuplo project role. A user also needs permission to view that project
|
|
41
|
+
before their pool role grants access to the pool's apps.
|
|
42
|
+
|
|
43
|
+
For more information on roles and permissions, see the
|
|
44
|
+
[document on roles and permissions](../articles/accounts/roles-and-permissions.mdx).
|
|
45
|
+
|
|
46
|
+
**Additional Resources**
|
|
47
|
+
|
|
48
|
+
- [Role Permissions](../articles/accounts/roles-and-permissions.mdx) - Details
|
|
49
|
+
on Zuplo account, Zuplo project, and AI Gateway pool roles.
|
|
50
|
+
- [Creating & Editing Pools](./managing-pools.mdx) - How to create and edit
|
|
51
|
+
pools.
|
|
52
|
+
- [Creating & Editing Apps](./managing-apps.mdx) - How to create, configure, and
|
|
53
|
+
delete apps.
|
|
@@ -29,6 +29,8 @@ Zuplo currently supports the following AI providers:
|
|
|
29
29
|
Gemini and Model Garden models on your own Google Cloud project
|
|
30
30
|
- [OpenRouter](./openrouter.mdx)—hundreds of models from every major vendor
|
|
31
31
|
behind a single API key
|
|
32
|
+
- [Jev](./jev.mdx)—TypeSafe's System One model, which answers typed questions
|
|
33
|
+
about text instead of generating it
|
|
32
34
|
- [Zuplo Demo](#zuplo-demo)—a free, keyless provider for trying the gateway
|
|
33
35
|
- OpenAI-compatible [Custom Providers](./custom-providers.mdx) (such as Qwen,
|
|
34
36
|
Kimi, etc)
|
|
@@ -47,6 +49,7 @@ The following capabilities are supported across providers:
|
|
|
47
49
|
| Bedrock Runtime | ❌ | ❌ | ❌ | ❌ |
|
|
48
50
|
| Vertex AI | ✅ | ✅ | ❌ | ✅ |
|
|
49
51
|
| OpenRouter | ✅ | ✅ | ✅ | ✅ |
|
|
52
|
+
| Jev | ❌ | ❌ | ❌ | ❌ |
|
|
50
53
|
| Zuplo Demo | ✅ | ❌ | ❌ | ❌ |
|
|
51
54
|
| OpenAI-compatible (Custom) | ✅ | ✅ | ❌ | ❌ |
|
|
52
55
|
|
|
@@ -60,13 +63,17 @@ Messages (plus chat completions through translation), while its other models
|
|
|
60
63
|
serve chat completions and—per model—Responses. See
|
|
61
64
|
[Using Bedrock Mantle](./bedrock-mantle.mdx#supported-endpoints-by-model-family).
|
|
62
65
|
|
|
63
|
-
Bedrock Runtime
|
|
64
|
-
|
|
66
|
+
Bedrock Runtime and Jev serve none of these capabilities, and the table above
|
|
67
|
+
says so. Bedrock Runtime serves Amazon Bedrock's **native** API instead, at its
|
|
65
68
|
own paths, so your existing AWS SDK code keeps working with the gateway in front
|
|
66
69
|
of it. See [Using Bedrock Runtime](./bedrock-runtime.mdx#supported-operations)
|
|
67
70
|
for the operations it serves, and add Bedrock Mantle alongside it if you also
|
|
68
71
|
want Bedrock models on the Universal API.
|
|
69
72
|
|
|
73
|
+
Jev serves TypeSafe's System One API instead, at `/v1/systemone`. You send it
|
|
74
|
+
text and typed questions, and it returns typed answers rather than generated
|
|
75
|
+
text. See [Using Jev](./jev.mdx#supported-endpoint).
|
|
76
|
+
|
|
70
77
|
Azure AI's capabilities split by model family, and within the OpenAI-compatible
|
|
71
78
|
family they split per deployed model: a chat model serves chat completions and—
|
|
72
79
|
per model and version—Responses, while an embedding model serves embeddings and
|
|
@@ -208,6 +215,29 @@ Three things are specific to it:
|
|
|
208
215
|
For prerequisites, provider steps, code examples, the routing shortcuts, and
|
|
209
216
|
troubleshooting, see [Using OpenRouter](./openrouter.mdx).
|
|
210
217
|
|
|
218
|
+
## Jev
|
|
219
|
+
|
|
220
|
+
**Jev** is TypeSafe's System One model. It doesn't generate text: you send it a
|
|
221
|
+
piece of text and a set of typed questions, and it returns a calibrated answer
|
|
222
|
+
for each one—a probability, a choice from your options, or a score on your
|
|
223
|
+
scale.
|
|
224
|
+
|
|
225
|
+
Three things are specific to it:
|
|
226
|
+
|
|
227
|
+
- **It has its own endpoint.** Requests go to `/v1/systemone`. Jev models are
|
|
228
|
+
refused on chat completions, Responses, Messages, and embeddings, and they
|
|
229
|
+
don't appear in the default `GET /v1/models` list.
|
|
230
|
+
- **Model references are `jev/<model>`.** A provider named `jev` serves
|
|
231
|
+
`jev/jev-latest`, `jev/jev-1.13.0`, and `jev/jev-preview`. The alias names
|
|
232
|
+
move when TypeSafe ships a release, so pin a versioned name if you tune
|
|
233
|
+
thresholds against one.
|
|
234
|
+
- **Only input tokens cost money.** TypeSafe charges for input tokens and not
|
|
235
|
+
for output, and the gateway prices requests from the model catalog.
|
|
236
|
+
|
|
237
|
+
Guardrail policies such as DLP deny System One requests by default, because they
|
|
238
|
+
can't inspect the body. For prerequisites, provider steps, code examples, and
|
|
239
|
+
the policy table, see [Using Jev](./jev.mdx).
|
|
240
|
+
|
|
211
241
|
## Zuplo Demo
|
|
212
242
|
|
|
213
243
|
**Zuplo Demo** is a free provider that Zuplo operates so you can try the AI
|
|
@@ -7,7 +7,7 @@ description:
|
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
An AI Gateway is a Zuplo project configured for AI traffic. A new project
|
|
10
|
-
deploys automatically, so you can configure providers,
|
|
10
|
+
deploys automatically, so you can configure providers, pools, and apps without
|
|
11
11
|
connecting a Git repository. Connect one when you want to write
|
|
12
12
|
[custom policies](./custom-policies.mdx), review gateway changes in Git, or
|
|
13
13
|
deploy from your own CI/CD.
|
|
@@ -86,7 +86,7 @@ Three categories of changes take effect differently:
|
|
|
86
86
|
| Change | Takes effect |
|
|
87
87
|
| ----------------------------------------------------------------- | ---------------------------------------------------- |
|
|
88
88
|
| Repository changes (routes, `policies.json`, custom policy code) | On the next deploy of the default branch |
|
|
89
|
-
| App policy chains,
|
|
89
|
+
| App policy chains, pools, budgets, and templates (portal changes) | Within about a minute, no deploy needed |
|
|
90
90
|
| Provider settings and environment variables | Automatically, via a rebuild and deploy Zuplo starts |
|
|
91
91
|
|
|
92
92
|
:::note
|
|
@@ -70,6 +70,10 @@ clients that can't set a model still work.
|
|
|
70
70
|
| `/v1/responses` | OpenAI Responses API: OpenAI, and the OpenAI-compatible models that serve it on [Azure AI](./azure-ai.mdx), [Bedrock Mantle](./bedrock-mantle.mdx) and [OpenRouter](./openrouter.mdx) |
|
|
71
71
|
| `/v1/messages` | Anthropic Messages API: Anthropic, and the Claude models of [Azure AI](./azure-ai.mdx), [Bedrock Mantle](./bedrock-mantle.mdx), [Vertex AI](./vertex-ai.mdx) and [OpenRouter](./openrouter.mdx) |
|
|
72
72
|
|
|
73
|
+
Jev is the exception: it serves TypeSafe's System One API on `/v1/systemone`,
|
|
74
|
+
which takes a state and typed questions rather than chat messages. That endpoint
|
|
75
|
+
isn't part of the Universal API. See [Jev](./jev.mdx).
|
|
76
|
+
|
|
73
77
|
## Request headers
|
|
74
78
|
|
|
75
79
|
The gateway forwards your request headers to the provider. A header your client
|