zuplo 6.67.32 → 6.68.0

package/docs/ai-gateway/policies/comet-opik-tracing.mdx

@@ -0,0 +1,139 @@
+---
+title: Comet Opik Tracing
+sidebar_label: Comet Opik Tracing
+---
+
+The Comet Opik Tracing policy integrates
+[Comet Opik](https://www.comet.com/docs/opik/) with the Zuplo AI Gateway,
+enabling comprehensive observability, tracing, and evaluation of your LLM
+applications in both development and production environments.
+
+Comet Opik is an open-source platform designed to help developers track, view,
+and evaluate Large Language Model (LLM) traces throughout the application
+lifecycle. By integrating Opik with the Zuplo AI Gateway, you gain complete
+visibility into your AI operations, from development debugging to production
+monitoring.
+
+### Key Capabilities
+
+The Comet Opik integration provides powerful observability and evaluation
+features:
+
+- **Comprehensive trace logging**: Automatically capture LLM calls, inputs,
+  outputs, and metadata
+- **Development debugging**: Annotate and label traces through SDK or UI for
+  iterative improvement
+- **LLM evaluation**: Use LLM-as-a-Judge and heuristic evaluators to score trace
+  quality
+- **Production monitoring**: Track feedback scores, trace counts, tokens, and
+  performance metrics at scale
+- **High-volume ingestion**: Support for up to 40 million traces per day
+- **Dataset management**: Store and run evaluations on test datasets
+
+## Benefits with Zuplo AI Gateway
+
+Integrating Comet Opik with the Zuplo AI Gateway provides several advantages:
+
+### Complete Application Observability
+
+Track entire LLM workflows including preprocessing, retrieval steps, model
+calls, and post-processing through your API gateway, providing end-to-end
+visibility.
+
+### Development and Production Parity
+
+Use the same tracing infrastructure in both development and production
+environments, ensuring consistent observability throughout your application
+lifecycle.
+
+### Automatic Trace Capture
+
+The policy automatically logs all AI Gateway requests and responses without
+requiring code changes to your LLM application, simplifying instrumentation.
+
+### Performance Insights
+
+Monitor token usage, latency, error rates, and costs across all your AI
+operations with detailed analytics dashboards.
+
+### Quality Assurance
+
+Evaluate LLM outputs using both automated metrics and LLM-as-a-Judge approaches
+to maintain quality standards as your application evolves.
+
+## How It Works
+
+### Trace Logging
+
+The policy captures comprehensive information about each LLM interaction:
+
+1. **Request data**: User prompts, input parameters, and metadata
+2. **Response data**: Model outputs, token counts, and generation details
+3. **Performance metrics**: Latency, processing time, and resource usage
+4. **Custom metadata**: Tags, conversation IDs, and application-specific data
+
+### Trace Organization
+
+Traces are organized hierarchically to represent complex workflows:
+
+- **Traces**: Top-level records representing complete user interactions
+- **Spans**: Nested operations within a trace (retrieval, generation, etc.)
+- **Thread IDs**: Group related traces by conversation or session
+
+### Evaluation Framework
+
+Opik provides multiple evaluation approaches:
+
+#### Heuristic Metrics
+
+Deterministic evaluation methods including:
+
+- **Exact match**: Verify outputs match expected values
+- **Contains**: Check for presence of specific content
+- **Regex patterns**: Validate output structure and format
+
+#### LLM-as-a-Judge Metrics
+
+AI-powered evaluation for subjective quality assessment:
+
+- **Hallucination detection**: Identify factually incorrect outputs
+- **Relevance scoring**: Measure response appropriateness
+- **Tone and style**: Evaluate alignment with brand guidelines
+- **Safety checks**: Detect harmful or inappropriate content
+
+## Use Cases
+
+### Debugging LLM Applications
+
+Identify and fix issues in LLM applications by examining detailed trace logs,
+including inputs, outputs, and intermediate steps.
+
+### A/B Testing AI Models
+
+Compare performance across different models, prompts, or configurations by
+analyzing traces grouped by experiment variants.
+
+### Cost Optimization
+
+Monitor token usage patterns to identify optimization opportunities and reduce
+AI operation costs.
+
+### Compliance and Auditing
+
+Maintain detailed audit logs of all AI interactions for regulatory compliance
+and security requirements.
+
+### Quality Regression Testing
+
+Track LLM output quality over time using automated evaluations, catching
+degradation before it impacts users.
+
+### Conversation Analytics
+
+Analyze multi-turn conversations using thread IDs to understand user journeys
+and improve conversational AI experiences.
+
+**Additional Resources**
+
+- [Comet Opik Documentation](https://www.comet.com/docs/opik/)
+- [Opik Tracing Guide](https://www.comet.com/docs/opik/tracing/log_traces)

package/docs/ai-gateway/policies/galileo-tracing.mdx

@@ -0,0 +1,147 @@
+---
+title: Galileo Tracing
+sidebar_label: Galileo Tracing
+---
+
+The Galileo Tracing policy integrates [Galileo AI](https://www.galileo.ai/) with
+the Zuplo AI Gateway, providing comprehensive observability, monitoring, and
+evaluation of your LLM applications. This policy automatically captures detailed
+traces of all AI Gateway requests and responses, enabling you to monitor
+performance, debug issues, and optimize your AI operations.
+
+## Key Features
+
+- **Automatic Trace Capture**: Seamlessly logs all LLM requests and responses
+  without code changes
+- **Streaming Support**: Handles both streaming and non-streaming responses
+- **Performance Monitoring**: Tracks token usage, latency, and resource
+  consumption
+- **Hierarchical Tracing**: Organizes traces with workflow and LLM spans for
+  detailed analysis
+
+## How It Works
+
+### Trace Structure
+
+The policy creates a hierarchical trace structure for each AI Gateway request:
+
+1. **Trace**: Top-level record representing the complete user interaction
+2. **Workflow Span**: Contains the entire AI Gateway workflow
+3. **LLM Span**: Captures the specific LLM API call details
+
+### Data Captured
+
+For each request, the policy automatically captures:
+
+**Request Information**
+
+- User prompts and messages
+- Model parameters (temperature, max_tokens, etc.)
+- Request metadata (route, request ID)
+- Timestamp and duration
+
+**Response Information**
+
+- Model outputs and completions
+- Token usage (input, output, total tokens)
+- Finish reasons and status
+- Performance metrics
+
+**Metadata**
+
+- Request ID for correlation
+- Route information
+- Custom tags for categorization
+- Duration in nanoseconds for precise timing
+
+## Configuration
+
+<Stepper>
+
+1.  ### Obtain Galileo Credentials
+    1. Sign up for a
+       [Galileo](https://galileo.ai?utm_source=zuplo&utm_medium=web) account
+    2. Create a new project in your Galileo dashboard
+    3. Generate an API key specifically for use with Zuplo
+
+2.  ### Add the policy to your app
+
+    You can add the policy to any AI Gateway app by clicking on **Policies**,
+    then on **Add Policy** and select **Galileo Tracing**.
+
+    ![The AI Gateway policy picker](../../../public/media/galileo-tracing/galileo-tracing-1.png)
+
+3.  ### Configure the policy
+
+    You will need to enter the following information from your Galileo account
+    to configure the policy:
+    - **`apiKey`**: Your Galileo API key for authentication
+    - **`projectId`**: The Galileo project ID to send traces to
+    - **`logStreamId`**: The specific log stream within your project
+    - **`baseUrl`** (optional): Custom Galileo API endpoint (defaults to
+      `https://api.galileo.ai`)
+
+    :::note
+
+    The `projectId` and `logStreamId` are both found in the URL of the Galileo
+    log stream you want to use. For example:
+    `https://app.galileo.ai/your-app/project/3e71c65e-48b6-4f5d-842d-0851c4704f95/log-streams/f8c71402-1f6b-4f5b-b073-1de999e6a8ea`.
+
+    In this case, the `projectId` is `3e71c65e-48b6-4f5d-842d-0851c4704f95` and
+    the `logStreamId` is `f8c71402-1f6b-4f5b-b073-1de999e6a8ea`.
+
+    :::
+
+</Stepper>
+
+### Key Metrics
+
+The policy automatically tracks:
+
+- **Token Usage**: Input, output, and total token counts
+- **Latency**: Request duration in nanoseconds
+- **Throughput**: Requests per second and volume
+- **Error Rates**: Failed requests and error patterns
+- **Model Performance**: Response quality and completion rates
+
+### Custom Metadata
+
+Each trace includes:
+
+- Request ID for correlation with logs
+- Route information for API endpoint analysis
+- Custom tags for categorization
+- User-defined metadata from the request context
+
+## Benefits of using Galileo Tracing
+
+### Development Workflow
+
+- Debug LLM applications with detailed trace inspection
+- Test different models and configurations with comparative analytics
+- Monitor quality regression during development cycles
+
+### Production Monitoring
+
+- Track performance and costs across all AI operations
+- Identify optimization opportunities through usage pattern analysis
+- Maintain audit logs for compliance and security requirements
+
+### Quality Assurance
+
+- Evaluate LLM outputs using Galileo's built-in metrics
+- Monitor response quality trends over time
+- Implement automated quality gates based on trace data
+
+## Troubleshooting
+
+Common issues and solutions:
+
+- **Authentication Errors**: Check that your Galileo API key is valid and has
+  proper permissions
+- **Configuration Issues**: Ensure the user context includes all required
+  Galileo settings
+
+## Additional Resources
+
+- [Galileo AI Documentation](https://www.galileo.ai/docs)

package/docs/ai-gateway/providers.mdx

@@ -0,0 +1,32 @@
+---
+title: AI Providers
+sidebar_label: Overview
+---
+
+Zuplo's AI Gateway supports integration with various AI providers, allowing you
+to leverage different models and services for your AI applications.
+
+## Supported Providers
+
+Zuplo currently supports the following AI providers:
+
+- OpenAI
+- Anthropic
+- Google
+- Mistral
+- OpenAI-compatible [Custom Providers](./custom-providers.mdx) (such as Qwen,
+  Kimi, etc)
+
+The following capabilities are supported across providers:
+
+| Provider                   | Chat Completions | Text Completions | Embeddings | Responses |
+| -------------------------- | ---------------- | ---------------- | ---------- | --------- |
+| OpenAI                     | ✅               | ✅               | ✅         | ✅        |
+| Anthropic                  | ✅               | ✅               | ✅         | ❌        |
+| Google                     | ✅               | ✅               | ✅         | ❌        |
+| Mistral                    | ✅               | ✅               | ✅         | ❌        |
+| OpenAI-compatible (Custom) | ✅               | ✅               | ✅         | ❌        |
+
+If you need support for additional providers or capabilities, please contact us
+at [support@zuplo.com](mailto:support@zuplo.com). We're continually working to
+add support for more providers based on customer demand.

package/docs/ai-gateway/teams.mdx

@@ -0,0 +1,38 @@
+---
+title: AI Gateway Teams
+sidebar_label: Overview
+---
+
+Teams are used to manage access to AI Gateway applications, set usage limits,
+and monitor activity. Each team can have multiple members and sub-teams. Use
+teams to group users by department, project, or any other logical grouping.
+
+## Teams & Apps
+
+Apps in the AI Gateway represent any app or integration that will call the AI
+Gateway. Apps in the AI Gateway are owned by a specific team. Members of the
+team will have permissions to perform various actions on the App depending on
+their permissions.
+
+## Members
+
+Members are the users who belong to a team. The permission of a member will
+depend on their role in the Zuplo account, project, and team.
+
+There are two roles at the team level:
+
+- **Member**: Can access AI Providers and Apps assigned to the team.
+- **Admin**: Can manage team settings, members, and access AI Providers and Apps
+  assigned to the team.
+
+For more information on roles and permissions, see the
+[document on roles and permissions](../articles/accounts/roles-and-permissions.mdx).
+
+**Additional Resources**
+
+- [Role Permissions](../articles/accounts/roles-and-permissions.mdx) - Details
+  on the roles available at the account and project levels.
+- [Creating & Editing Teams](./managing-teams.mdx) - How to create and edit
+  teams.
+- [Creating & Editing Apps](./managing-apps.mdx) - How to add, remove, and set
+  roles for project members.

package/docs/ai-gateway/universal-api.mdx

@@ -0,0 +1,43 @@
+---
+title: AI Gateway Universal API
+sidebar_label: Universal API
+---
+
+Zuplo AI Gateway provides a universal API that standardizes interactions with
+various AI providers. This API follows the
+[OpenAI API specification](https://platform.openai.com/docs/api-reference/introduction),
+making it easy to integrate with existing applications that already use OpenAI's
+API.
+
+## Using the Universal API
+
+The Universal API is automatically enabled for all AI Gateway applications.
+Using this endpoint is as simple as changing your API base URL to point to
+Zuplo. For example, if your Zuplo application is hosted at
+`https://my-ai-gateway.zuplo.app`, you can simply change the base URL in your
+API client to `https://my-ai-gateway.zuplo.app/v1`.
+
+If you are using an SDK or library that supports custom base URLs, you can
+configure it to use your Zuplo application's URL. For example, with the OpenAI
+Node.js SDK, you can set the `baseURL` option:
+
+```ts
+import OpenAI from "openai";
+
+const client = new OpenAI({
+  apiKey: process.env.ZUPLO_API_KEY,
+  baseURL: "https://my-ai-gateway.zuplo.app/v1",
+});
+
+const response = await client.chat.completions.create({
+  model: "gpt-4",
+  messages: [
+    {
+      role: "user",
+      content: "Write a one-sentence bedtime story about a unicorn.",
+    },
+  ],
+});
+
+console.log(response.choices[0].message.content);
+```

package/docs/ai-gateway/usage-limits.mdx

@@ -0,0 +1,89 @@
+---
+title: "Usage Limits & Thresholds"
+sidebar_label: "Usage Limits"
+---
+
+The Zuplo AI Gateway provides hierarchical usage limits and budget controls to
+manage LLM spending across your organization. Limits can be set at the
+organization, team, and application levels.
+
+## Budget Hierarchy
+
+Budget limits cascade down through your organizational structure:
+
+- **Root Team** - Organization-wide limits (for example, $1,000/day)
+- **Sub-Teams** - Team-specific limits that cannot exceed the parent team's
+  budget (for example, $500/day for the Engineering team)
+- **Applications** - Per-app limits for granular control (for example, $10/day
+  for a hackathon project)
+
+A sub-team's budget can never exceed the available budget from its parent team.
+Similarly, an application's budget cannot exceed its owning team's budget.
+
+## Configuring Limits
+
+### Daily Budgets
+
+Set a maximum daily spend for a team or application. When the daily budget is
+reached, requests are either blocked or flagged with a warning depending on your
+enforcement configuration.
+
+To configure daily budgets:
+
+1. Navigate to your AI Gateway project in the
+   [Zuplo Portal](https://portal.zuplo.com)
+2. Select the **Teams** or **Apps** tab
+3. Click on the team or app to edit
+4. Select the **Usage & Limits** tab and configure the **Daily Budget** field
+5. Click **Save Changes**
+
+### Monthly Budgets
+
+Set a maximum monthly spend for applications. Monthly budgets reset on the first
+day of each calendar month.
+
+### Rate Limits
+
+In addition to budget-based limits, you can configure request rate limits to
+control the volume of requests flowing through the gateway.
+
+## Enforcement Modes
+
+When a limit is reached, the AI Gateway can operate in two modes:
+
+- **Enforce** - Requests are blocked and an error response is returned to the
+  caller
+- **Warn** - Requests are allowed through but a warning notification is
+  generated
+
+## Monitoring Usage
+
+Track current usage and spending through the AI Gateway dashboard:
+
+1. Navigate to your AI Gateway project
+2. Click on an app and select **Dashboard**
+3. View real-time metrics including:
+   - Request count
+   - Token usage (input and output)
+   - Current spending against budget
+   - Time to first byte
+
+## Semantic Caching
+
+Enable semantic caching on applications to reduce costs by identifying and
+returning cached responses for similar prompts. This can significantly reduce
+token usage and spending, especially for applications with repeated or similar
+queries.
+
+To enable semantic caching:
+
+1. Navigate to the **Apps** tab and click on the app to edit
+2. Enable the **Semantic Caching** toggle under **Advanced Features**
+3. Save your changes
+
+## Related Resources
+
+- [Getting Started](./getting-started.mdx) - Set up your first AI Gateway
+  project with budget controls
+- [Managing Teams](./managing-teams.mdx) - Configure team-level budgets
+- [Managing Apps](./managing-apps.mdx) - Configure app-level limits

package/docs/api-management/introduction.md

@@ -0,0 +1,127 @@
+---
+title: Zuplo API Management
+sidebar_label: Introduction
+---
+
+The Zuplo API Gateway is a fully-managed, lightweight API management platform
+designed for developers. It offers fast deployment, GitOps-friendly workflows,
+and unlimited environments. Whether you're an individual developer or part of an
+engineering team, Zuplo makes it easy to:
+
+- [Add authentication and access control](../articles/step-3-add-api-key-auth.mdx)
+- [Implement rate limiting](../articles/step-2-add-rate-limiting.mdx)
+- Write custom logic to run at the gateway layer
+- Build a [rich developer portal](../dev-portal/introduction.mdx) with
+  self-serve tools for auth and monetization
+
+Zuplo delivers the core benefits of API management without the overhead of
+legacy platforms. That means no expensive licensing, training requirements, or
+complex setup.
+
+Everything in Zuplo is defined through code and stored in source control.
+Deployments are handled through Git-based workflows and go live globally in
+under 20 seconds.
+
+Explore more by [booking a demo](https://zuplo.com/meeting?utm_source=docs) or
+[signing up](https://portal.zuplo.com/signup?utm_source=docs) for free.
+
+## Zuplo in your stack
+
+Zuplo is a serverless gateway that runs at the edge in over 300 data centers
+worldwide. This edge-first architecture provides:
+
+- Built-in redundancy and high availability
+- Low-latency performance—typically within 50ms of most users
+
+Zuplo is cloud-agnostic. It integrates with backends running on AWS, Azure, GCP,
+or private infrastructure. Multiple
+[secure connectivity options](../articles/securing-your-backend.mdx) are
+available.
+
+In most setups, Zuplo sits between clients and your backend API—whether those
+clients are servers, browsers, mobile apps, or IoT devices. Traffic is routed
+through Zuplo, where you can enforce policies like rate limiting and
+authentication, validate requests, and apply transformations before requests
+reach your backend.
+
+![Zuplo Architecture](../../public/media/what-is-zuplo/zuplo-connect-light.png)
+
+Zuplo also supports global traffic management. Customers with distributed
+backends use Zuplo to route requests to the nearest data center, optimizing for
+speed and reliability.
+
+![Global distribution with Zuplo](../../public/media/what-is-zuplo/zuplo-distributed-light.png)
+
+## Protocols
+
+Zuplo can proxy any HTTP traffic. It supports REST, GraphQL, WebSockets, and
+other HTTP-based protocols (including legacy systems proxying SOAP over HTTP!).
+HTTP/2 is fully supported.
+
+## Languages
+
+Zuplo is configured via JSON and extended using TypeScript or JavaScript. Your
+backend can be written in any language that speaks HTTP, such as Go, Node.js,
+.NET, Java, C, and more.
+
+## Integrations
+
+Zuplo integrates with platforms like Datadog, New Relic and GCP Cloud Logging
+for monitoring and observability. New integrations are continuously added based
+on customer needs. [Reach out](../articles/support.mdx) if you need support for
+a specific tool.
+
+## Runtime
+
+The Zuplo runtime is based on Web Worker technology, supporting JavaScript and
+WebAssembly. It's the same foundational tech used by platforms like Deno Deploy,
+Fastly, Vercel Edge Functions and Cloudflare Workers.
+
+This architecture offers key benefits:
+
+- Near-zero cold start time
+- High throughput
+- Strong developer ergonomics—built on familiar browser APIs like
+  [Response Web API](https://developer.mozilla.org/en-US/docs/Web/API/Response)
+
+## Performance and latency
+
+Zuplo processes billions of requests monthly and has been tested to handle over
+10,000 requests per second, even with policies like API key validation and rate
+limiting enabled.
+
+Typical added latency is in the low milliseconds. Policies are highly optimized
+and can be tuned to meet specific performance goals.
+
+## Multi-cloud
+
+Zuplo is designed to work seamlessly with services across cloud providers
+including AWS, Azure, GCP, and on-premise environments.
+
+The distributed architecture and
+[connectivity options](../articles/securing-your-backend.mdx) ensure secure,
+performant connections to your backend, wherever it runs.
+
+## Security and Compliance
+
+Zuplo is built with a security-first architecture and offers robust tools for
+securing APIs in production environments:
+
+- Support for API key auth, OAuth2, mTLS, IP allowlisting, and custom
+  authentication logic
+- Fine-grained access control and rate limiting applied at the edge
+- Token validation and request enforcement before hitting your backend
+
+Zuplo is SOC 2 Type II compliant and operates in a multi-tenant, zero-trust
+model. All data in transit is encrypted using TLS 1.2+, and backend secrets are
+managed securely.
+
+Deployments are Git-based and fully auditable. All gateway configurations are
+defined as code, making it easy to enforce policy-as-code and maintain
+traceability.
+
+For teams with strict compliance or data residency requirements Zuplo offers
+customizable deployment options, including managed dedicated instances and
+tailored configurations.
+[Book a demo](https://calendly.com/zuplo-api/api-discussion) to discuss your
+needs.