zuplo 7.2.3 → 7.2.4

package/docs/ai-gateway/policies/comet-opik-tracing.mdx

@@ -1,177 +0,0 @@
----
-title: Comet Opik tracing
-sidebar_label: Comet Opik tracing
-description:
-  Send LLM traces from the Zuplo AI Gateway to Comet Opik for observability,
-  debugging, and evaluation of your AI applications.
----
-
-The Comet Opik Tracing policy integrates
-[Comet Opik](https://www.comet.com/docs/opik/) with the Zuplo AI Gateway,
-enabling comprehensive observability, tracing, and evaluation of your LLM
-applications in both development and production environments.
-
-Comet Opik is an open-source platform designed to help developers track, view,
-and evaluate Large Language Model (LLM) traces throughout the application
-lifecycle. By integrating Opik with the Zuplo AI Gateway, you gain complete
-visibility into your AI operations, from development debugging to production
-monitoring.
-
-### Key Capabilities
-
-The Comet Opik integration provides powerful observability and evaluation
-features:
-
-- **Comprehensive trace logging**: Automatically capture LLM calls, inputs,
-  outputs, and metadata
-- **Development debugging**: Annotate and label traces through SDK or UI for
-  iterative improvement
-- **LLM evaluation**: Use LLM-as-a-Judge and heuristic evaluators to score trace
-  quality
-- **Production monitoring**: Track feedback scores, trace counts, tokens, and
-  performance metrics at scale
-- **High-volume ingestion**: Support for up to 40 million traces per day
-- **Dataset management**: Store and run evaluations on test datasets
-
-## Adding the policy to an app
-
-The scaffolded gateway declares the Comet Opik Tracing policy in
-`config/policies.json`. Configure it once for the gateway, then add it to each
-app that should send traces:
-
-<Stepper>
-
-1. In your Comet account, create (or pick) a Comet project and workspace and
-   generate an API key.
-
-2. In your gateway's repository, set the policy's settings and push:
-
-   ```json title="config/policies.json"
-   {
-     "name": "comet-opik-tracing-v2-inbound",
-     "policyType": "comet-opik-tracing-v2-inbound",
-     "handler": {
-       "export": "CometOpikTracingV2InboundPolicy",
-       "module": "$import(@zuplo/runtime)",
-       "options": {
-         "apiKey": "your-opik-api-key",
-         "projectName": "your-project-name",
-         "workspace": "your-workspace"
-       }
-     }
-   }
-   ```
-
-3. Open the app's **Policies** tab, click **Add Policy**, and add **Comet Opik
-   Tracing**. Place it at the end of the chain so traces reflect the request the
-   provider actually receives.
-
-</Stepper>
-
-## Benefits with Zuplo AI Gateway
-
-Integrating Comet Opik with the Zuplo AI Gateway provides several advantages:
-
-### Complete Application Observability
-
-Track entire LLM workflows including preprocessing, retrieval steps, model
-calls, and post-processing through your API gateway, providing end-to-end
-visibility.
-
-### Development and Production Parity
-
-Use the same tracing infrastructure in both development and production
-environments, ensuring consistent observability throughout your application
-lifecycle.
-
-### Automatic Trace Capture
-
-The policy automatically logs all AI Gateway requests and responses without
-requiring code changes to your LLM application, simplifying instrumentation.
-
-### Performance Insights
-
-Monitor token usage, latency, error rates, and costs across all your AI
-operations with detailed analytics dashboards.
-
-### Quality Assurance
-
-Evaluate LLM outputs using both automated metrics and LLM-as-a-Judge approaches
-to maintain quality standards as your application evolves.
-
-## How It Works
-
-### Trace Logging
-
-The policy captures comprehensive information about each LLM interaction:
-
-1. **Request data**: User prompts, input parameters, and metadata
-2. **Response data**: Model outputs, token counts, and generation details
-3. **Performance metrics**: Latency, processing time, and resource usage
-4. **Custom metadata**: Tags, conversation IDs, and application-specific data
-
-### Trace Organization
-
-Traces are organized hierarchically to represent complex workflows:
-
-- **Traces**: Top-level records representing complete user interactions
-- **Spans**: Nested operations within a trace (retrieval, generation, etc.)
-- **Thread IDs**: Group related traces by conversation or session
-
-### Evaluation Framework
-
-Opik provides multiple evaluation approaches:
-
-#### Heuristic Metrics
-
-Deterministic evaluation methods including:
-
-- **Exact match**: Verify outputs match expected values
-- **Contains**: Check for presence of specific content
-- **Regex patterns**: Validate output structure and format
-
-#### LLM-as-a-Judge Metrics
-
-AI-powered evaluation for subjective quality assessment:
-
-- **Hallucination detection**: Identify factually incorrect outputs
-- **Relevance scoring**: Measure response appropriateness
-- **Tone and style**: Evaluate alignment with brand guidelines
-- **Safety checks**: Detect harmful or inappropriate content
-
-## Use Cases
-
-### Debugging LLM Applications
-
-Identify and fix issues in LLM applications by examining detailed trace logs,
-including inputs, outputs, and intermediate steps.
-
-### A/B Testing AI Models
-
-Compare performance across different models, prompts, or configurations by
-analyzing traces grouped by experiment variants.
-
-### Cost Optimization
-
-Monitor token usage patterns to identify optimization opportunities and reduce
-AI operation costs.
-
-### Compliance and Auditing
-
-Maintain detailed audit logs of all AI interactions for regulatory compliance
-and security requirements.
-
-### Quality Regression Testing
-
-Track LLM output quality over time using automated evaluations, catching
-degradation before it impacts users.
-
-### Conversation Analytics
-
-Analyze multi-turn conversations using thread IDs to understand user journeys
-and improve conversational AI experiences.
-
-**Additional Resources**
-
-- [Comet Opik Documentation](https://www.comet.com/docs/opik/)
-- [Opik Tracing Guide](https://www.comet.com/docs/opik/tracing/log_traces)

package/docs/ai-gateway/policies/fallback-model.mdx

@@ -1,40 +0,0 @@
----
-title: Fallback Model
-sidebar_label: Fallback Model
-description:
-  Add a backup model for provider errors and timeouts, and a cheaper model for
-  when a budget runs out.
----
-
-`ai-gateway-fallback-model-v2`
-
-Fallback Model adds two kinds of backup to an app's model selection, per
-capability:
-
-- **`fallback`**—the model to try after a provider error that can be retried, or
-  after a timeout.
-- **`quotaFallback`**—the model to use when a usage limit is hit, instead of
-  rejecting the request. [Budgets and Costs](./metering.mdx) is what applies
-  this.
-
-`fallbackTimeoutSeconds` controls how long the gateway waits before giving up on
-the main model and trying the fallback. It defaults to 60 seconds and accepts 1
-to 300. A fallback that names the same model as the main one is skipped.
-
-:::warning
-
-This policy only enriches a model selection that already exists—it never creates
-one. An app whose chain has Fallback Model but no
-[Model Filtering](./model-filtering.mdx) does nothing at all: the policy logs a
-warning and passes the request through unchanged. Place Fallback Model
-immediately after Model Filtering.
-
-:::
-
-That ordering is also why a fallback can never widen what an app may reach—the
-filter has already accepted or rejected the request by the time this policy
-runs.
-
-For the portal workflow and worked examples see
-[Fallback Models](../fallback.mdx); for every option see the
-[generated reference](/policies/ai-gateway-fallback-model-v2-inbound).

package/docs/ai-gateway/policies/galileo-tracing.mdx

@@ -1,161 +0,0 @@
----
-title: Galileo tracing
-sidebar_label: Galileo tracing
-description:
-  Send LLM traces from the Zuplo AI Gateway to Galileo for monitoring,
-  debugging, and quality evaluation of your AI applications.
----
-
-The Galileo Tracing policy integrates [Galileo AI](https://www.galileo.ai/) with
-the Zuplo AI Gateway, providing comprehensive observability, monitoring, and
-evaluation of your LLM applications. This policy automatically captures detailed
-traces of all AI Gateway requests and responses, enabling you to monitor
-performance, debug issues, and optimize your AI operations.
-
-## Key Features
-
-- **Automatic Trace Capture**: Seamlessly logs all LLM requests and responses
-  without code changes
-- **Streaming Support**: Handles both streaming and non-streaming responses
-- **Performance Monitoring**: Tracks token usage, latency, and resource
-  consumption
-- **Hierarchical Tracing**: Organizes traces with workflow and LLM spans for
-  detailed analysis
-
-## How It Works
-
-### Trace Structure
-
-The policy creates a hierarchical trace structure for each AI Gateway request:
-
-1. **Trace**: Top-level record representing the complete user interaction
-2. **Workflow Span**: Contains the entire AI Gateway workflow
-3. **LLM Span**: Captures the specific LLM API call details
-
-### Data Captured
-
-For each request, the policy automatically captures:
-
-**Request Information**
-
-- User prompts and messages
-- Model parameters (temperature, `max_tokens`, etc.)
-- Request metadata (route, request ID)
-- Timestamp and duration
-
-**Response Information**
-
-- Model outputs and completions
-- Token usage (input, output, total tokens)
-- Finish reasons and status
-- Performance metrics
-
-**Metadata**
-
-- Request ID for correlation
-- Route information
-- Custom tags for categorization
-- Duration in nanoseconds for precise timing
-
-## Configuration
-
-<Stepper>
-
-1.  ### Obtain Galileo Credentials
-    1. Sign up for a
-       [Galileo](https://galileo.ai?utm_source=zuplo&utm_medium=web) account
-    2. Create a new Galileo project in your Galileo dashboard
-    3. Generate an API key specifically for use with Zuplo
-
-    :::note
-
-    The `projectId` and `logStreamId` are both found in the URL of the Galileo
-    log stream you want to use. For example:
-    `https://app.galileo.ai/your-app/project/3e71c65e-48b6-4f5d-842d-0851c4704f95/log-streams/f8c71402-1f6b-4f5b-b073-1de999e6a8ea`.
-
-    In this case, the `projectId` is `3e71c65e-48b6-4f5d-842d-0851c4704f95` and
-    the `logStreamId` is `f8c71402-1f6b-4f5b-b073-1de999e6a8ea`.
-
-    :::
-
-2.  ### Configure the declaration in your gateway's repository
-
-    The scaffolded gateway declares the Galileo Tracing policy in
-    `config/policies.json`. Set its settings and push:
-
-    ```json title="config/policies.json"
-    {
-      "name": "galileo-tracing-v2-inbound",
-      "policyType": "galileo-tracing-v2-inbound",
-      "handler": {
-        "export": "GalileoTracingV2InboundPolicy",
-        "module": "$import(@zuplo/runtime)",
-        "options": {
-          "apiKey": "your-galileo-api-key",
-          "projectId": "3e71c65e-48b6-4f5d-842d-0851c4704f95",
-          "logStreamId": "f8c71402-1f6b-4f5b-b073-1de999e6a8ea"
-        }
-      }
-    }
-    ```
-
-3.  ### Add the policy to your app
-
-    Open the app's **Policies** tab, click **Add Policy**, and select **Galileo
-    Tracing**. Place it at the end of the chain so traces reflect the request
-    the provider actually receives. The app inherits the configuration from the
-    declaration.
-
-</Stepper>
-
-### Key Metrics
-
-The policy automatically tracks:
-
-- **Token Usage**: Input, output, and total token counts
-- **Latency**: Request duration in nanoseconds
-- **Throughput**: Requests per second and volume
-- **Error Rates**: Failed requests and error patterns
-- **Model Performance**: Response quality and completion rates
-
-### Custom Metadata
-
-Each trace includes:
-
-- Request ID for correlation with logs
-- Route information for API endpoint analysis
-- Custom tags for categorization
-- User-defined metadata from the request context
-
-## Benefits of using Galileo Tracing
-
-### Development Workflow
-
-- Debug LLM applications with detailed trace inspection
-- Test different models and configurations with comparative analytics
-- Monitor quality regression during development cycles
-
-### Production Monitoring
-
-- Track performance and costs across all AI operations
-- Identify optimization opportunities through usage pattern analysis
-- Maintain audit logs for compliance and security requirements
-
-### Quality Assurance
-
-- Evaluate LLM outputs using Galileo's built-in metrics
-- Monitor response quality trends over time
-- Implement automated quality gates based on trace data
-
-## Troubleshooting
-
-Common issues and solutions:
-
-- **Authentication Errors**: Check that your Galileo API key is valid and has
-  proper permissions
-- **Configuration Issues**: Ensure the policy declaration's environment
-  variables are set in your Zuplo project settings
-
-## Additional Resources
-
-- [Galileo AI Documentation](https://www.galileo.ai/docs)

package/docs/ai-gateway/policies/metering.mdx

@@ -1,47 +0,0 @@
----
-title: Budgets and Costs
-sidebar_label: Budgets and Costs
-description:
-  Record an app's usage and enforce its budgets for spend, tokens, and requests.
----
-
-`ai-gateway-metering-v2`
-
-The **Budgets and Costs** policy records what an app uses and enforces its
-budgets. It meters three things—spend, tokens, and requests—each with
-independent daily and monthly periods, and each with an optional warning
-threshold that notifies without blocking.
-
-When a limit is exceeded, the policy either routes the request to the model
-selection's `quotaFallback` if [Fallback Model](./fallback-model.mdx) supplied
-one, or rejects it with `429`.
-
-An app's own limits aren't the only ones that apply. The policy also checks the
-parent team's and the gateway's limits, which are enforced centrally rather than
-from this policy's options—so a request can be blocked by a limit that isn't
-visible in the app's chain.
-
-Place Budgets and Costs after [Model Filtering](./model-filtering.mdx) and
-[Fallback Model](./fallback-model.mdx), so an exceeded budget can activate the
-quota fallback.
-
-:::caution{title="Budgets fail open by default"}
-
-`throwOnFailure` defaults to `false`, so if the metering service is unavailable
-the request proceeds unmetered—no limit is checked, even for an app that's
-already over budget. Set it to `true` to reject the request instead.
-
-:::
-
-:::note
-
-Each limit period carries an `enabled` flag next to its number, and so does each
-warning threshold. A period enforces nothing until `enabled` is `true` _and_
-`limit` is set. Watch for this when editing options directly, such as in
-`config/policies.json` or a [policy template](../policy-templates.mdx).
-
-:::
-
-For the portal workflow and the gateway and team levels see
-[Usage Limits](../usage-limits.mdx); for every option see the
-[generated reference](/policies/ai-gateway-metering-v2-inbound).

package/docs/ai-gateway/policies/model-filtering.mdx

@@ -1,49 +0,0 @@
----
-title: Model Filtering
-sidebar_label: Model Filtering
-description:
-  Restrict an app to specific models with an allow list or block list, and set
-  the default model.
----
-
-`ai-gateway-model-filtering-v2`
-
-Model Filtering decides which models an app may use and creates the model
-selection the rest of the chain works from. Without it, an app can reach any
-model offered by the providers configured for the Zuplo project.
-
-Configure each capability—completions, embeddings—in one of two modes. The modes
-are mutually exclusive per capability.
-
-- An **allow list** exposes a curated set of models. The first entry is the
-  default when a request names no model.
-- A **block list** permits every available model except the ones named. Each
-  request must name its model, because a block list supplies no default.
-
-Models are written as `providerName/model`, where `providerName` is the provider
-name configured in your gateway—for example `openai/gpt-5`.
-
-## What a rejected request gets
-
-| Request                                          | Response                      |
-| ------------------------------------------------ | ----------------------------- |
-| A model the rules don't allow                    | `403`, listing what's allowed |
-| No `model` named, in block list mode             | `400`                         |
-| A capability the policy doesn't configure at all | `403`                         |
-| A provider name that isn't in the catalog        | `400`                         |
-
-Rejections are returned as problem details, and the `detail` names what the
-caller should have sent—the allowed models for a blocked model, or the gateway's
-configured provider names for an unrecognized one. The
-[Akamai AI Firewall](./akamai-ai-firewall.mdx) is the exception among the
-policies here: it answers in the provider's own error format instead.
-
-Place Model Filtering first in the chain—it creates the model selection that
-[Fallback Model](./fallback-model.mdx) and [Budgets and Costs](./metering.mdx)
-build on.
-
-For the portal workflow see
-[Restricting models](../managing-apps.mdx#restricting-models); for every option
-see the [generated reference](/policies/ai-gateway-model-filtering-v2-inbound).
-When model selection needs logic instead of a list, see
-[Dynamic model routing](../cookbooks/dynamic-model-routing.mdx).

package/docs/ai-gateway/policies/semantic-cache.mdx

@@ -1,48 +0,0 @@
----
-title: Semantic cache
-sidebar_label: Semantic cache
-description:
-  Return a cached response when a new prompt is close enough to one the app has
-  already answered.
----
-
-`ai-gateway-semantic-cache-v2-inbound`
-
-The semantic cache answers a request from a previous response when the new
-prompt is semantically similar to one already seen, skipping the provider
-entirely. One policy handles both halves: it checks the cache on the way in and,
-on a miss, stores the provider's response on the way out. There's no separate
-outbound policy to add.
-
-A cache hit answers the request, so entries after it in the chain don't run.
-Place it after [Budgets and Costs](./metering.mdx) so hits still count toward
-the app's request limits.
-
-## Options
-
-| Option                  | Default                              | What it does                                                                                        |
-| ----------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------- |
-| `semanticTolerance`     | `0.2`                                | How loose a match to accept. A response is reused at similarity ≥ 1 − tolerance, so `0.2` needs 0.8 |
-| `expirationSecondsTtl`  | `3600`                               | How long a cached response stays valid, up to 30 days                                               |
-| `maxConversationLength` | `3`                                  | Longest conversation to cache, in messages                                                          |
-| `endpoints`             | Chat Completions, Anthropic Messages | Which API shapes to cache                                                                           |
-| `namespace`             | The app's ID                         | Cache partition. Only applies when the request has no app configuration                             |
-
-## What isn't cached
-
-- Conversations longer than `maxConversationLength`—three messages by
-  default—aren't cached at all.
-- The Responses API isn't supported. The cache covers Chat Completions and
-  Anthropic Messages.
-- The cache always fails open: if the cache is unavailable, the request goes to
-  the provider as normal.
-
-One app can never read another's cached responses. The namespace is fixed to the
-app's ID whenever the request resolves to an app, so the `namespace` option only
-matters on a gateway that doesn't identify apps.
-
-## Reading cache outcomes
-
-Each response carries the outcome as a `Cache-Status` header under the cache
-name `zp-aigw-sem-cache`, alongside `x-ai-gateway-cache` (`HIT` or `MISS`) and
-`x-ai-gateway-cache-similarity`.