zuplo 7.3.2 → 7.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/ai-gateway/apps.mdx +2 -2
- package/docs/ai-gateway/custom-policies.mdx +3 -3
- package/docs/ai-gateway/fallback.mdx +13 -12
- package/docs/ai-gateway/getting-started.mdx +10 -9
- package/docs/ai-gateway/introduction.mdx +19 -18
- package/docs/ai-gateway/managing-apps.mdx +9 -7
- package/docs/ai-gateway/policy-chains.mdx +13 -7
- package/docs/ai-gateway/policy-templates.mdx +4 -3
- package/docs/ai-gateway/usage-limits.mdx +8 -10
- package/docs/policies/_index.md +2 -2
- package/docs/policies/ai-gateway-configuration-executor-v2-inbound/schema.json +1 -1
- package/docs/policies/ai-gateway-metering-v2-inbound/doc.md +13 -13
- package/docs/policies/ai-gateway-metering-v2-inbound/schema.json +1 -1
- package/package.json +5 -5
package/docs/ai-gateway/apps.mdx
CHANGED
|
@@ -27,8 +27,8 @@ Each app has three things of its own:
|
|
|
27
27
|
- **An API key**—authenticates the app's requests when the gateway runs the
|
|
28
28
|
[authentication policy](./policy-chains.mdx#authentication).
|
|
29
29
|
- **A [policy chain](./policy-chains.mdx)**—the ordered policies that run on the
|
|
30
|
-
app's requests: model access, budgets, caching, guardrails, and
|
|
31
|
-
policies. The chain starts out empty unless the app's team has a
|
|
30
|
+
app's requests: model access, app-specific budgets, caching, guardrails, and
|
|
31
|
+
custom policies. The chain starts out empty unless the app's team has a
|
|
32
32
|
[policy template](./policy-templates.mdx).
|
|
33
33
|
|
|
34
34
|
## API Keys
|
|
@@ -7,9 +7,9 @@ description:
|
|
|
7
7
|
push, and add it to an app's policy chain in the portal.
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
The AI Gateway's built-in policies cover model access, budgets,
|
|
11
|
-
guardrails, and tracing—but your gateway can run any policy you can
|
|
12
|
-
TypeScript. A custom policy lives in your gateway's
|
|
10
|
+
The AI Gateway's built-in policies cover model access, app-specific budgets,
|
|
11
|
+
caching, guardrails, and tracing—but your gateway can run any policy you can
|
|
12
|
+
write in TypeScript. A custom policy lives in your gateway's
|
|
13
13
|
[repository](./source-control.mdx), is declared in `config/policies.json`, and
|
|
14
14
|
from then on appears in the portal's **Add Policy** dialog like any built-in
|
|
15
15
|
policy. Apps add it to their [policy chains](./policy-chains.mdx), and it runs
|
|
@@ -7,8 +7,8 @@ description:
|
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
Fallbacks let an app keep serving requests when its primary model fails, times
|
|
10
|
-
out, or
|
|
11
|
-
They're configured with the **Fallback Model** policy in the app's
|
|
10
|
+
out, or an applicable usage limit is exceeded, instead of returning an error to
|
|
11
|
+
the caller. They're configured with the **Fallback Model** policy in the app's
|
|
12
12
|
[policy chain](./policy-chains.mdx).
|
|
13
13
|
|
|
14
14
|
The AI Gateway offers two independent fallback mechanisms, each triggered by a
|
|
@@ -17,7 +17,7 @@ different condition:
|
|
|
17
17
|
| Mechanism | Triggers when… | Without a fallback set… |
|
|
18
18
|
| ---------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
|
|
19
19
|
| **Fallback & Timeout** | The primary fails with a retryable error (`5xx`, `408`, `425`, `429`), a network failure, or a timeout | The gateway returns the error to the caller |
|
|
20
|
-
| **Quota Fallback** |
|
|
20
|
+
| **Quota Fallback** | An app, team, or gateway usage limit is exceeded | The request is blocked with a `429` |
|
|
21
21
|
|
|
22
22
|
Both are configured entirely in the Zuplo Portal, and either can route to _any_
|
|
23
23
|
provider—the fallback doesn't have to share the primary's provider. Fallback
|
|
@@ -57,21 +57,22 @@ is configured, the primary model call runs unbounded.
|
|
|
57
57
|
## Quota fallback
|
|
58
58
|
|
|
59
59
|
The quota fallback routes requests to an alternate, usually cheaper, model when
|
|
60
|
-
|
|
61
|
-
blocking the request with a `429`. This keeps an app available after
|
|
62
|
-
|
|
63
|
-
lower-cost model.
|
|
60
|
+
an app, team, or gateway [usage limit](./usage-limits.mdx) is exceeded, rather
|
|
61
|
+
than blocking the request with a `429`. This keeps an app available after an
|
|
62
|
+
applicable budget, token, or request threshold is crossed, while shifting the
|
|
63
|
+
overflow traffic to a lower-cost model.
|
|
64
64
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
65
|
+
The **Fallback Model** policy supplies the quota fallback selection. Team and
|
|
66
|
+
gateway limits can activate it whether or not the chain includes **Budgets and
|
|
67
|
+
Costs**. Add Budgets and Costs when the app also needs its own limits, and place
|
|
68
|
+
it after Fallback Model. If you leave the quota fallback empty, the gateway
|
|
69
|
+
blocks requests with a `429` once an applicable limit is exceeded.
|
|
69
70
|
|
|
70
71
|
:::tip
|
|
71
72
|
|
|
72
73
|
Point the quota fallback at a smaller, cheaper model so overflow traffic stays
|
|
73
74
|
inexpensive while remaining available. The fallback's own usage still counts
|
|
74
|
-
toward
|
|
75
|
+
toward every applicable limit.
|
|
75
76
|
|
|
76
77
|
:::
|
|
77
78
|
|
|
@@ -107,7 +107,8 @@ account, Zuplo project, and AI Gateway team roles control access.
|
|
|
107
107
|
on later teams, letting you nest them. To set gateway-wide spending limits
|
|
108
108
|
across this Zuplo project, open the gateway's **Settings → Usage Limits**;
|
|
109
109
|
for team-specific limits, use the team's **Usage & Limits** tab. See
|
|
110
|
-
[Usage Limits](./usage-limits.mdx).
|
|
110
|
+
[Usage Limits](./usage-limits.mdx). Gateway and team limits apply to every
|
|
111
|
+
app regardless of its policy chain.
|
|
111
112
|
|
|
112
113
|
5. **Create an app**
|
|
113
114
|
|
|
@@ -117,8 +118,8 @@ account, Zuplo project, and AI Gateway team roles control access.
|
|
|
117
118
|
In the **Apps & Teams** tab, select the team you just created and click
|
|
118
119
|
**Create App** in the team's header. Give the app a descriptive name (for
|
|
119
120
|
example, "Tennis Chat") and confirm the team that owns it. That's the whole
|
|
120
|
-
dialog—models, budgets, and other behavior are configured on the
|
|
121
|
-
chain in the next step.
|
|
121
|
+
dialog—models, app-specific budgets, and other behavior are configured on the
|
|
122
|
+
app's policy chain in the next step.
|
|
122
123
|
|
|
123
124
|
:::tip
|
|
124
125
|
|
|
@@ -148,9 +149,9 @@ account, Zuplo project, and AI Gateway team roles control access.
|
|
|
148
149
|
|
|
149
150
|
:::
|
|
150
151
|
|
|
151
|
-
This is also where you add budgets, semantic caching,
|
|
152
|
-
custom policies. Changes to the chain apply within about
|
|
153
|
-
needed. See [Policy Chains](./policy-chains.mdx).
|
|
152
|
+
This is also where you add app-specific budgets, semantic caching,
|
|
153
|
+
guardrails, and any custom policies. Changes to the chain apply within about
|
|
154
|
+
a minute—no redeploy needed. See [Policy Chains](./policy-chains.mdx).
|
|
154
155
|
|
|
155
156
|
7. **Get the app's URL and API key**
|
|
156
157
|
|
|
@@ -300,8 +301,8 @@ metrics.
|
|
|
300
301
|
|
|
301
302
|
## Next steps
|
|
302
303
|
|
|
303
|
-
- [Policy Chains](./policy-chains.mdx): add budgets, semantic
|
|
304
|
-
guardrails to your app's chain
|
|
304
|
+
- [Policy Chains](./policy-chains.mdx): add app-specific budgets, semantic
|
|
305
|
+
caching, and guardrails to your app's chain
|
|
305
306
|
- [Custom Policies](./custom-policies.mdx): write your own policy and add it to
|
|
306
307
|
an app's chain
|
|
307
308
|
- [Policy Templates](./policy-templates.mdx): give every new app in a team a
|
|
@@ -317,4 +318,4 @@ metrics.
|
|
|
317
318
|
| 404 "Unsupported AI Gateway endpoint" | The path must end with a supported `/v1/...` operation—usually the URL is missing its `/v1` segment. Use the app's API URL plus `/v1/chat/completions`, for example |
|
|
318
319
|
| 400 error asking for `providerName/model` | Prefix the model with the provider name configured in the portal—`openai/gpt-5-mini`, not `gpt-5-mini` |
|
|
319
320
|
| 403 error listing the allowed models | The app's Model Filtering policy doesn't include the requested model; add it on the app's Policies tab or use a listed one |
|
|
320
|
-
| Budget limit reached immediately | Check the
|
|
321
|
+
| Budget limit reached immediately | Check the gateway and team limits, which always apply, plus the app's Budgets and Costs policy; any level's limit can block the request |
|
|
@@ -16,11 +16,12 @@ responses while applying policies, controls, and monitoring.
|
|
|
16
16
|
Each piece of software that calls the gateway registers as an **app**—your
|
|
17
17
|
support chatbot is one app; an internal coding agent is another. A single AI
|
|
18
18
|
Gateway holds as many apps as you need, and you control exactly which policies
|
|
19
|
-
run for each one. Every app has its own ordered policy chain, so
|
|
20
|
-
customer-facing chatbot
|
|
21
|
-
|
|
22
|
-
TypeScript, and apps add them to their chains like any
|
|
23
|
-
changes apply within about a minute, with no redeploy. See
|
|
19
|
+
run for each one. Every app has its own ordered policy chain, so you can give
|
|
20
|
+
the customer-facing chatbot an app-specific budget and guardrails while leaving
|
|
21
|
+
those controls out of the internal agent's chain. Beyond the built-in policies,
|
|
22
|
+
you can write your own in TypeScript, and apps add them to their chains like any
|
|
23
|
+
other policy. Chain changes apply within about a minute, with no redeploy. See
|
|
24
|
+
[Apps](./apps.mdx).
|
|
24
25
|
|
|
25
26
|
## Key Benefits
|
|
26
27
|
|
|
@@ -59,15 +60,15 @@ provider. Each app has its own URL, API key, and policy chain, so one gateway
|
|
|
59
60
|
serves teams with completely different configurations.
|
|
60
61
|
|
|
61
62
|
Each request arrives at the URL of the app it belongs to. The gateway identifies
|
|
62
|
-
that app, runs its policy chain (model access, cost controls,
|
|
63
|
-
guardrails), routes to the selected LLM provider, and streams the
|
|
64
|
-
Throughout this process, the gateway captures metrics without
|
|
65
|
-
underlying provider credentials.
|
|
63
|
+
that app, runs its policy chain (model access, app-specific cost controls,
|
|
64
|
+
security guardrails), routes to the selected LLM provider, and streams the
|
|
65
|
+
response back. Throughout this process, the gateway captures metrics without
|
|
66
|
+
exposing underlying provider credentials.
|
|
66
67
|
|
|
67
|
-
Every control is opt-in. An app whose policy chain is empty
|
|
68
|
-
|
|
69
|
-
budgets, guardrails, or caching apply
|
|
70
|
-
|
|
68
|
+
Every app-specific control is opt-in. An app whose policy chain is empty runs no
|
|
69
|
+
app-selected controls: each request names its own model, and no app-specific
|
|
70
|
+
model restrictions, budgets, guardrails, or caching apply. Gateway and team
|
|
71
|
+
usage limits still apply.
|
|
71
72
|
|
|
72
73
|
## Core Features
|
|
73
74
|
|
|
@@ -90,9 +91,9 @@ runs, so gateway changes go through your normal review workflow. See
|
|
|
90
91
|
### Per-App Policy Chains
|
|
91
92
|
|
|
92
93
|
Each app runs its own ordered chain of policies—model filtering, fallback
|
|
93
|
-
models, budgets, semantic caching, guardrails, tracing, and any
|
|
94
|
-
declared in the repository. Team policy templates give new apps
|
|
95
|
-
starting pipeline. See [Policy Chains](./policy-chains.mdx) and
|
|
94
|
+
models, app-specific budgets, semantic caching, guardrails, tracing, and any
|
|
95
|
+
custom policies declared in the repository. Team policy templates give new apps
|
|
96
|
+
a consistent starting pipeline. See [Policy Chains](./policy-chains.mdx) and
|
|
96
97
|
[Custom Policies](./custom-policies.mdx).
|
|
97
98
|
|
|
98
99
|
### Team Hierarchy & Budgets
|
|
@@ -113,8 +114,8 @@ Each app gets its own:
|
|
|
113
114
|
|
|
114
115
|
- **Unique Gateway URL**: Single endpoint regardless of underlying provider
|
|
115
116
|
- **API Key**: Zuplo-managed key that never exposes provider credentials
|
|
116
|
-
- **Policy Chain**: Model access, budgets, caching, guardrails, and
|
|
117
|
-
policies, applied in the order the app chooses
|
|
117
|
+
- **Policy Chain**: Model access, app-specific budgets, caching, guardrails, and
|
|
118
|
+
custom policies, applied in the order the app chooses
|
|
118
119
|
|
|
119
120
|
### Zuplo accounts and projects
|
|
120
121
|
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
title: Managing Apps
|
|
3
3
|
description:
|
|
4
4
|
Create, edit, and delete AI Gateway apps. Apps are created with just a name
|
|
5
|
-
and a team; models, budgets, and other behavior live on the app's
|
|
6
|
-
chain.
|
|
5
|
+
and a team; models, app-specific budgets, and other behavior live on the app's
|
|
6
|
+
policy chain.
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
An app represents one caller of your AI Gateway—a service, an agent, or a
|
|
@@ -47,7 +47,8 @@ permission to view the Zuplo project that contains the AI Gateway.
|
|
|
47
47
|
That's the whole dialog. The portal then takes you to the app's **Policies**
|
|
48
48
|
tab. If the app's team defines a [policy template](./policy-templates.mdx), the
|
|
49
49
|
new app starts with that chain; otherwise the chain is empty and the app is
|
|
50
|
-
|
|
50
|
+
subject to no app-specific controls until you add policies. Team and gateway
|
|
51
|
+
usage limits still apply.
|
|
51
52
|
|
|
52
53
|
## Restricting models
|
|
53
54
|
|
|
@@ -86,16 +87,17 @@ over—see [Dynamic model routing](./cookbooks/dynamic-model-routing.mdx).
|
|
|
86
87
|
|
|
87
88
|
App budgets live on the **Budgets and Costs** policy in the app's chain. Open
|
|
88
89
|
the app's **Policies** tab and configure it with daily or monthly limits for
|
|
89
|
-
spending, tokens, or requests.
|
|
90
|
-
|
|
90
|
+
spending, tokens, or requests. Team and gateway limits apply to every app
|
|
91
|
+
automatically, even when the app's chain doesn't include Budgets and Costs. See
|
|
92
|
+
[Usage Limits](./usage-limits.mdx).
|
|
91
93
|
|
|
92
94
|
## Editing an App
|
|
93
95
|
|
|
94
96
|
To edit an app, open the
|
|
95
97
|
[Apps & Teams](https://portal.zuplo.com/+/account/project/ai/apps) tab of your
|
|
96
98
|
AI Gateway project in the Zuplo Portal. Select the app you want to edit. The
|
|
97
|
-
app's behavior—models, budgets, caching, guardrails—is edited on
|
|
98
|
-
**Policies** tab; the app's name lives on the **Settings** tab. Make your
|
|
99
|
+
app's behavior—models, app-specific budgets, caching, guardrails—is edited on
|
|
100
|
+
the **Policies** tab; the app's name lives on the **Settings** tab. Make your
|
|
99
101
|
changes and click the **Save** button. Policy chain changes apply within about a
|
|
100
102
|
minute.
|
|
101
103
|
|
|
@@ -8,9 +8,9 @@ description:
|
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
Every request to an AI Gateway app runs down through the app's **policy chain**:
|
|
11
|
-
an ordered list of policies that control model access, budgets,
|
|
12
|
-
guardrails, and tracing for that app. Different apps on the same
|
|
13
|
-
completely different chains.
|
|
11
|
+
an ordered list of policies that control model access, app-specific budgets,
|
|
12
|
+
caching, guardrails, and tracing for that app. Different apps on the same
|
|
13
|
+
gateway can run completely different chains.
|
|
14
14
|
|
|
15
15
|
The chain works in two layers:
|
|
16
16
|
|
|
@@ -24,8 +24,9 @@ The chain works in two layers:
|
|
|
24
24
|
portal. Chain changes apply within about a minute, with no deploy.
|
|
25
25
|
|
|
26
26
|
No policy is required. An app whose chain is empty still serves requests: each
|
|
27
|
-
request names its own model, and no model restrictions, budgets,
|
|
28
|
-
caching apply.
|
|
27
|
+
request names its own model, and no app-specific model restrictions, budgets,
|
|
28
|
+
guardrails, or caching apply. Team and gateway usage limits apply regardless of
|
|
29
|
+
the app's policy chain.
|
|
29
30
|
|
|
30
31
|
A request passes through whatever the app selected before the AI Gateway handler
|
|
31
32
|
calls the provider:
|
|
@@ -105,14 +106,19 @@ Model Filtering → Fallback Model → Budgets and Costs → Semantic Cache
|
|
|
105
106
|
never creates one. A chain with Fallback Model but no Model Filtering does
|
|
106
107
|
nothing at all—the policy logs a warning and passes the request through
|
|
107
108
|
unchanged. That's also why a fallback can never bypass the filter.
|
|
108
|
-
- **Budgets and Costs** third: it
|
|
109
|
-
|
|
109
|
+
- **Budgets and Costs** third: it configures the app's own limits and needs the
|
|
110
|
+
resolved selection so an exceeded app limit can activate the quota fallback.
|
|
110
111
|
- **Semantic Cache** after Budgets and Costs: cache hits still count toward
|
|
111
112
|
request limits.
|
|
112
113
|
|
|
113
114
|
Tracing policies work well at the end of the chain, so traces reflect the
|
|
114
115
|
request the earlier policies produced.
|
|
115
116
|
|
|
117
|
+
Team and gateway limits are enforced outside the app's policy chain. When one is
|
|
118
|
+
exceeded, the app's quota fallback works whether or not its chain includes
|
|
119
|
+
Budgets and Costs, as long as Model Filtering and Fallback Model supply a
|
|
120
|
+
fallback selection.
|
|
121
|
+
|
|
116
122
|
## Built-in policies
|
|
117
123
|
|
|
118
124
|
See the [policies overview](./policies/overview.mdx) for every policy an app can
|
|
@@ -8,9 +8,10 @@ description:
|
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
A team's **policy template** is the policy chain that apps created in that team
|
|
11
|
-
start with. Use templates to
|
|
12
|
-
|
|
13
|
-
custom compliance policy—while controlling what individual apps
|
|
11
|
+
start with. Use templates to give every app in a team the same app-specific
|
|
12
|
+
controls—such as Model Filtering, Budgets and Costs with an app budget, a
|
|
13
|
+
guardrail, or a custom compliance policy—while controlling what individual apps
|
|
14
|
+
may change. Team and gateway limits don't depend on the template.
|
|
14
15
|
|
|
15
16
|
## How templates apply
|
|
16
17
|
|
|
@@ -83,22 +83,20 @@ requests to that model instead of blocking with a `429`—see
|
|
|
83
83
|
[Fallback Models](./fallback.mdx). The fallback's usage still counts toward the
|
|
84
84
|
limits.
|
|
85
85
|
|
|
86
|
-
:::
|
|
86
|
+
:::note{title="App, team, and gateway limits are configured separately"}
|
|
87
87
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
[policy template](./policy-templates.mdx) to make sure every app starts with it.
|
|
88
|
+
The Budgets and Costs policy configures and enforces an app's own limits. Team
|
|
89
|
+
and gateway limits apply to every app automatically, whether or not the app's
|
|
90
|
+
policy chain includes Budgets and Costs.
|
|
92
91
|
|
|
93
92
|
:::
|
|
94
93
|
|
|
95
94
|
:::caution{title="Budgets fail open by default"}
|
|
96
95
|
|
|
97
|
-
The Budgets and Costs policy's `throwOnFailure` option
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
outage rejects the request rather than letting it escape the budget.
|
|
96
|
+
The Budgets and Costs policy's `throwOnFailure` option controls how an error
|
|
97
|
+
while checking the app's own limits is handled. It defaults to `false`, so the
|
|
98
|
+
request continues through the gateway. Set it to `true` to reject the request
|
|
99
|
+
instead. Team and gateway limit checks run independently of this policy.
|
|
102
100
|
|
|
103
101
|
:::
|
|
104
102
|
|
package/docs/policies/_index.md
CHANGED
|
@@ -5,10 +5,10 @@
|
|
|
5
5
|
| set-query-params-inbound | Add or Set Query Parameters | Adds or sets query parameters on the incoming request. | api-gateway |
|
|
6
6
|
| set-headers-inbound | Add or Set Request Headers | Adds or sets headers on the incoming request. | api-gateway |
|
|
7
7
|
| ai-gateway-auth-v2-inbound | AI Gateway Authentication | Authenticates requests to an AI Gateway endpoint with application API keys. Add this policy to an application's `inboundPolicyChain` to require a key for that app only, or place it on the route before the configuration executor to require a key for every application on the route. The policies that follow can read the authenticated application from `request.user` (`sub` is the application name, `data` its metadata), and the application's AI Gateway configuration takes effect for the request. Use `authHeader` and `authScheme` when clients send their app key somewhere other than the default `Authorization: Bearer` header. When the matched route captures an `app_id` path parameter (platform catch-all `/:app_id/(.*)`), this policy also requires `configuration.id === request.params.app_id` and returns 403 on mismatch. | ai-gateway |
|
|
8
|
-
| ai-gateway-configuration-executor-v2-inbound | AI Gateway Configuration Executor | Loads the app configuration for the request (when auth or the configuration loader has not already)
|
|
8
|
+
| ai-gateway-configuration-executor-v2-inbound | AI Gateway Configuration Executor | Loads the app configuration for the request (when auth or the configuration loader has not already), runs the inbound policy chain from that configuration, and enforces limits inherited from parent teams or the gateway root. Place this policy on AI Gateway routes after optional authentication and optional `ai-gateway-configuration-loader-v2-inbound`. When either of those already populated the app-configuration channel, this policy reuses it. Otherwise it loads the configuration with the route's `app_id` path parameter. Applications select from policies pre-declared by the gateway. Applications without a `inboundPolicyChain`, or with an empty chain, run no application-selected policies. Entry options replace the declaration's options as a complete object; omit them to inherit the declaration, including environment-backed credentials. Each occurrence receives a private deep copy of its entry options, so a policy mutating its options cannot corrupt the cached app configuration. | ai-gateway |
|
|
9
9
|
| ai-gateway-configuration-loader-v2-inbound | AI Gateway Configuration Loader | Loads the AI Gateway app configuration for the request into the request-scoped channel and does nothing else. Place this policy on AI Gateway routes before `ai-gateway-configuration-executor-v2-inbound` when you want configuration loading separated from chain execution. When `ai-gateway-auth-v2-inbound` already populated the channel, this policy reuses it. Otherwise it loads the configuration with the route's `app_id` path parameter. If this policy is omitted, the configuration executor still loads configuration itself before running the application chain. | ai-gateway |
|
|
10
10
|
| ai-gateway-fallback-model-v2-inbound | AI Gateway Fallback Model | Adds failure and quota fallbacks to an existing AI Gateway model selection. Place this policy after AI Gateway Model Filtering. It never creates a model selection, so a misplaced policy cannot bypass filtering. | ai-gateway |
|
|
11
|
-
| ai-gateway-metering-v2-inbound | AI Gateway Metering | Meters AI Gateway usage and enforces
|
|
11
|
+
| ai-gateway-metering-v2-inbound | AI Gateway Metering | Meters AI Gateway usage and enforces limits configured by the application. The authentication policy must run before this policy so the app configuration id is available for meter storage and analytics. | ai-gateway |
|
|
12
12
|
| ai-gateway-model-filtering-v2-inbound | AI Gateway Model Filtering | Matches AI Gateway requests against curated allow lists or open block lists, then stores the winning model reference for the route handler. | ai-gateway |
|
|
13
13
|
| ai-gateway-semantic-cache-v2-inbound | AI Gateway Semantic Cache | AI Gateway Semantic Cache policy. This inbound policy looks up the semantic cache on entry and, on a miss, registers a response-sending hook to write the upstream response back into the cache. Caching parameters (semanticTolerance, expirationSecondsTtl, namespace, recentMessageCount) come from the policy options, and presence in the route's chain is what enables it. An id from the authenticated app configuration always provides the cache namespace so application-supplied options cannot cross tenant partitions. The cache key covers the system prompt plus the last `recentMessageCount` messages (default 1) — applied when storing and when matching — so multi-turn conversations can hit entries cached from earlier, shorter ones. Cache outcomes are reported on the response via the RFC 9211 `Cache-Status` header under the cache name `zp-aigw-sem-cache` (hit: `zp-aigw-sem-cache; hit; detail="similarity=0.93"`; miss: `zp-aigw-sem-cache; fwd=miss; stored`). Responses also include the `x-ai-gateway-cache: HIT\|MISS` and `x-ai-gateway-cache-similarity` headers. | ai-gateway |
|
|
14
14
|
| akamai-ai-firewall-v2-inbound | Akamai AI Firewall | Akamai AI Firewall Inbound Policy | ai-gateway |
|
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
"requiresAI": true,
|
|
13
13
|
"policyType": "ai-gateway-configuration-executor-v2",
|
|
14
14
|
"products": ["ai-gateway"],
|
|
15
|
-
"description": "Loads the app configuration for the request (when auth or the configuration loader has not already)
|
|
15
|
+
"description": "Loads the app configuration for the request (when auth or the configuration loader has not already), runs the inbound policy chain from that configuration, and enforces limits inherited from parent teams or the gateway root.\n\nPlace this policy on AI Gateway routes after optional authentication and optional `ai-gateway-configuration-loader-v2-inbound`. When either of those already populated the app-configuration channel, this policy reuses it. Otherwise it loads the configuration with the route's `app_id` path parameter. Applications select from policies pre-declared by the gateway. Applications without a `inboundPolicyChain`, or with an empty chain, run no application-selected policies. Entry options replace the declaration's options as a complete object; omit them to inherit the declaration, including environment-backed credentials. Each occurrence receives a private deep copy of its entry options, so a policy mutating its options cannot corrupt the cached app configuration.",
|
|
16
16
|
"deprecatedMessage": "",
|
|
17
17
|
"required": ["handler"],
|
|
18
18
|
"properties": {
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# AI Gateway Metering
|
|
2
2
|
|
|
3
|
-
AI Gateway Metering records application usage and
|
|
4
|
-
provider request runs. It meters spend, tokens, and requests.
|
|
5
|
-
independent daily and monthly periods and an optional warning
|
|
6
|
-
notifies without blocking.
|
|
3
|
+
AI Gateway Metering records application usage and configures the app's own
|
|
4
|
+
budgets before the provider request runs. It meters spend, tokens, and requests.
|
|
5
|
+
Each meter has independent daily and monthly periods and an optional warning
|
|
6
|
+
threshold that notifies without blocking.
|
|
7
7
|
|
|
8
8
|
When a limit is exceeded, the policy activates the model selection's
|
|
9
9
|
`quotaFallback` when AI Gateway Fallback Model supplied one. Otherwise it
|
|
@@ -52,16 +52,16 @@ The supported meters are `costs`, `tokens`, and `requests`. Each meter can have
|
|
|
52
52
|
|
|
53
53
|
## Team limits
|
|
54
54
|
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
55
|
+
The `options.limits` settings govern only the app running this policy. Limits
|
|
56
|
+
configured on a parent team or the gateway root are enforced centrally after the
|
|
57
|
+
application's policy chain, whether or not this policy appears in that chain. An
|
|
58
|
+
inherited limit activates the selected model's quota fallback when available and
|
|
59
|
+
otherwise returns `429 Too Many Requests`.
|
|
59
60
|
|
|
60
|
-
An
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
request while continuing to meter it, and `true` fails the request.
|
|
61
|
+
An application cannot disable inherited enforcement through its policy chain. If
|
|
62
|
+
the central hierarchical check is unavailable, the request proceeds. The
|
|
63
|
+
policy's `throwOnFailure` option controls failures while checking or recording
|
|
64
|
+
the app's own limits; it does not change inherited-limit behavior.
|
|
65
65
|
|
|
66
66
|
## Set limits from custom code
|
|
67
67
|
|
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
"requiresAI": true,
|
|
13
13
|
"policyType": "ai-gateway-metering-v2",
|
|
14
14
|
"products": ["ai-gateway"],
|
|
15
|
-
"description": "Meters AI Gateway usage and enforces
|
|
15
|
+
"description": "Meters AI Gateway usage and enforces limits configured by the application.\n\nThe authentication policy must run before this policy so the app configuration id is available for meter storage and analytics.",
|
|
16
16
|
"deprecatedMessage": "",
|
|
17
17
|
"required": ["handler"],
|
|
18
18
|
"properties": {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "zuplo",
|
|
3
|
-
"version": "7.
|
|
3
|
+
"version": "7.4.1",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "The programmable API Gateway",
|
|
6
6
|
"author": "Zuplo, Inc.",
|
|
@@ -19,9 +19,9 @@
|
|
|
19
19
|
"zuplo": "zuplo.js"
|
|
20
20
|
},
|
|
21
21
|
"dependencies": {
|
|
22
|
-
"@zuplo/cli": "7.
|
|
23
|
-
"@zuplo/core": "7.
|
|
24
|
-
"@zuplo/runtime": "7.
|
|
25
|
-
"@zuplo/test": "7.
|
|
22
|
+
"@zuplo/cli": "7.4.1",
|
|
23
|
+
"@zuplo/core": "7.4.1",
|
|
24
|
+
"@zuplo/runtime": "7.4.1",
|
|
25
|
+
"@zuplo/test": "7.4.1"
|
|
26
26
|
}
|
|
27
27
|
}
|