zuplo 6.71.8 → 6.71.10

package/docs/articles/graphql.mdx

@@ -140,6 +140,7 @@ const config = {
     graphqlPlugin({
       schema: "./schema.graphql", // Also accepts a URL, e.g. https://...
       path: graphqlPath,
+      // endpoint: "/my-graphql", // Optional, defaults to "/graphql"
     }),
   ],
 };

package/docs/guides/proxying-between-zuplo-gateways.mdx

@@ -0,0 +1,409 @@
+---
+title: Proxying Between Zuplo Gateways
+sidebar_label: Proxying Between Gateways
+description:
+  Learn how to proxy requests from one Zuplo project to another using the URL
+  Forward Handler or a custom fetch handler, propagate authentication, surface
+  upstream errors, and troubleshoot 522 timeouts.
+tags:
+  - backends
+  - deployment
+  - custom-code
+  - authentication
+---
+
+Some architectures call for one Zuplo gateway to sit in front of one or more
+other Zuplo gateways. A "product" gateway might aggregate several team-owned
+"member" gateways, a BFF might fan out to multiple internal APIs, or a migration
+might route a subset of traffic through a new project while the old one still
+serves the rest.
+
+This guide covers the patterns, auth propagation strategies, error-handling
+pitfalls, and troubleshooting steps you need when the upstream is another Zuplo
+project.
+
+## When this pattern makes sense
+
+- **Product-of-products** — A single public API endpoint forwards different path
+  prefixes to separate Zuplo projects, each owned by a different team.
+- **Backend for frontend (BFF)** — A gateway aggregates data from multiple
+  downstream Zuplo-managed APIs into a single response.
+- **Tenant routing** — Requests are routed to different Zuplo projects based on
+  tenant identity or API key metadata. See
+  [User-Based Backend Routing](./user-based-backend-routing.mdx) for a detailed
+  walkthrough of this approach.
+- **Gradual migration** — During a migration, a new gateway forwards unhandled
+  routes to the old gateway.
+
+If you use a Managed Dedicated deployment with an enterprise plan, consider
+[Federated Gateways](../dedicated/federated-gateways.mdx) instead. Federated
+Gateways is an enterprise add-on that uses the `local://` protocol for
+inter-environment communication within the same dedicated instance, avoiding the
+public internet and providing lower latency.
+
+## Choosing an approach
+
+### Pattern A: URL Forward Handler
+
+The [URL Forward Handler](../handlers/url-forward.mdx) is the simplest option.
+It proxies the request — method, headers, and body — to the downstream Zuplo
+project without writing any code.
+
+```json
+{
+  "handler": {
+    "export": "urlForwardHandler",
+    "module": "$import(@zuplo/runtime)",
+    "options": {
+      "baseUrl": "${env.DOWNSTREAM_GATEWAY_URL}"
+    }
+  }
+}
+```
+
+Store the downstream URL (for example
+`https://member-api-main-abc123.zuplo.app`) in an
+[environment variable](../articles/environment-variables.mdx) so you can change
+it per environment without modifying route configuration.
+
+The URL Forward Handler appends the incoming path to the `baseUrl`. If the outer
+gateway receives `GET /orders/123` and the `baseUrl` is
+`https://member-api-main-abc123.zuplo.app`, the forwarded request goes to
+`https://member-api-main-abc123.zuplo.app/orders/123`.
+
+**When to use this pattern:**
+
+- You want zero-code proxying and are happy forwarding the request as-is.
+- You do not need to inspect or transform the upstream response before returning
+  it to the caller.
+
+### Pattern B: Custom fetch handler
+
+A [Function Handler](../handlers/custom-handler.mdx) gives you full control over
+the outbound request and lets you inspect the upstream response before returning
+it to the caller. This is the recommended pattern when you need to propagate
+authentication credentials, transform the response, or surface upstream error
+details.
+
+```typescript
+import { ZuploContext, ZuploRequest, environment } from "@zuplo/runtime";
+
+export default async function (
+  request: ZuploRequest,
+  context: ZuploContext,
+): Promise<Response> {
+  const url = new URL(request.url);
+  const upstreamUrl = `${environment.DOWNSTREAM_GATEWAY_URL}${url.pathname}${url.search}`;
+
+  const upstreamResponse = await fetch(upstreamUrl, {
+    method: request.method,
+    headers: request.headers,
+    body: request.body,
+  });
+
+  // Return the upstream response directly, preserving status and headers
+  return new Response(upstreamResponse.body, {
+    status: upstreamResponse.status,
+    headers: upstreamResponse.headers,
+  });
+}
+```
+
+**When to use this pattern:**
+
+- You need to add, remove, or transform headers before forwarding.
+- You need to read the upstream response body (for example, to merge responses
+  from multiple downstreams).
+- You want to return the exact upstream status code and body to the caller
+  instead of receiving an opaque 522. See
+  [Surfacing upstream errors](#surfacing-upstream-errors-instead-of-522) below.
+
+### Pattern C: Federated Gateways (Managed Dedicated)
+
+On a [Managed Dedicated](../dedicated/federated-gateways.mdx) plan, use the
+`local://` protocol to call other environments in the same instance:
+
+```json
+{
+  "handler": {
+    "export": "urlForwardHandler",
+    "module": "$import(@zuplo/runtime)",
+    "options": {
+      "baseUrl": "local://member-api-main-abc123"
+    }
+  }
+}
+```
+
+This avoids the public internet entirely. The Lambda handler is not supported
+for federated calls — use URL Forward, URL Rewrite, or a Function Handler
+instead.
+
+## Propagating authentication
+
+When the outer gateway authenticates a request (using
+[API Key Authentication](../concepts/api-keys.md),
+[JWT authentication](../concepts/authentication.mdx), or another method), the
+inner gateway still needs to trust that request. There are several patterns for
+propagating identity between gateways.
+
+### Forward the original credential
+
+The simplest approach is to forward the caller's original `Authorization` header
+(or API key header) to the downstream gateway. Both the URL Forward Handler and
+the custom fetch handler forward request headers by default, so if the
+downstream gateway accepts the same credentials, this works without extra
+configuration.
+
+:::caution
+
+If the outer and inner gateways use different API key buckets or different JWT
+issuers, forwarding the original credential does not work. Use one of the
+patterns below instead.
+
+:::
+
+### Shared secret header
+
+Store a shared secret in an
+[environment variable](../articles/environment-variables.mdx) on both projects.
+On the outer gateway, use a
+[Set Headers policy](../policies/set-headers-inbound.mdx) to add the secret as a
+custom header. On the inner gateway, validate the header in an inbound policy or
+use the same Set Headers policy to check the value.
+
+```json
+{
+  "name": "set-backend-secret",
+  "policyType": "set-headers-inbound",
+  "handler": {
+    "export": "SetHeadersInboundPolicy",
+    "module": "$import(@zuplo/runtime)",
+    "options": {
+      "headers": [
+        {
+          "name": "x-gateway-secret",
+          "value": "$env(DOWNSTREAM_SECRET)"
+        }
+      ]
+    }
+  }
+}
+```
+
+See [Securing your backend](../articles/securing-your-backend.mdx) for a
+complete walkthrough of this approach.
+
+### Upstream Zuplo JWT
+
+The [Upstream Zuplo JWT policy](../policies/upstream-zuplo-jwt-auth-inbound.mdx)
+generates a short-lived, self-signed JWT and attaches it to the outbound
+request. Configure the inner gateway to validate this JWT using the
+[OpenID JWT Authentication policy](../policies/open-id-jwt-auth-inbound.mdx)
+with Zuplo's JWKS endpoint.
+
+This is the most robust option for service-to-service authentication between
+Zuplo projects because it does not require sharing static secrets and the token
+includes claims you can use for authorization on the downstream side.
+
+## Surfacing upstream errors instead of 522
+
+A common problem when proxying between Zuplo gateways: the downstream gateway
+returns a `401 Unauthorized` (or another error), but the caller sees a `522`
+instead.
+
+### Why this happens
+
+Zuplo's managed edge environment uses connection-level timeouts between the
+gateway and the origin server. A `522` status code means a connection-level
+failure occurred between the gateway and the upstream. The
+[Platform Limits](../articles/limits.mdx) documentation lists two scenarios that
+produce a 522: a Complete TCP Connection timeout at 19 seconds and a TCP ACK
+Timeout at 90 seconds.
+
+A `522` can also appear when the upstream closes the connection unexpectedly —
+for example, if the downstream gateway rejects the TLS handshake, returns a
+connection reset, or takes too long to send the response headers.
+
+When the downstream Zuplo project returns an HTTP error like `401` or `500`,
+that is **not** a 522. The 522 means the connection itself failed before an HTTP
+response was received. If you are seeing 522 instead of the expected upstream
+error, the issue is at the network or TLS layer, not the HTTP layer.
+
+### Common causes of 522 between Zuplo projects
+
+- **DNS resolution failure** — The downstream URL is incorrect or the
+  environment no longer exists.
+- **TLS handshake failure** — Misconfigured custom domain or certificate issue
+  on the downstream project.
+- **Connection timeout** — The downstream project takes longer than 19 seconds
+  to accept the TCP connection, usually because it is overloaded or
+  misconfigured.
+- **Egress restrictions** — In some network configurations, outbound connections
+  from one Zuplo project to another may be restricted.
+
+### Returning the actual upstream error
+
+If the TCP connection succeeds but the upstream returns an HTTP error (like 401
+or 500), the URL Forward Handler already returns that status code to the caller.
+You do not need to do anything extra — the upstream's status and body flow
+through.
+
+If you need more control (for example, to log the upstream error or transform it
+before returning), use a custom fetch handler:
+
+```typescript
+import { ZuploContext, ZuploRequest, environment } from "@zuplo/runtime";
+
+export default async function (
+  request: ZuploRequest,
+  context: ZuploContext,
+): Promise<Response> {
+  const url = new URL(request.url);
+  const upstreamUrl = `${environment.DOWNSTREAM_GATEWAY_URL}${url.pathname}${url.search}`;
+
+  try {
+    const upstreamResponse = await fetch(upstreamUrl, {
+      method: request.method,
+      headers: request.headers,
+      body: request.body,
+    });
+
+    if (!upstreamResponse.ok) {
+      context.log.warn(
+        `Upstream returned ${upstreamResponse.status} for ${url.pathname}`,
+      );
+    }
+
+    return new Response(upstreamResponse.body, {
+      status: upstreamResponse.status,
+      headers: upstreamResponse.headers,
+    });
+  } catch (error) {
+    context.log.error(`Failed to reach upstream: ${error}`);
+    return new Response(
+      JSON.stringify({
+        type: "https://httpproblems.com/http-status/502",
+        title: "Bad Gateway",
+        status: 502,
+        detail: "The upstream service is unreachable.",
+      }),
+      {
+        status: 502,
+        headers: { "content-type": "application/problem+json" },
+      },
+    );
+  }
+}
+```
+
+This handler catches connection-level errors (which would otherwise surface as
+a 522) and returns a structured `502 Bad Gateway` response. When the upstream
+does return an HTTP response, the status and body pass through unchanged.
+
+## Custom domains across the fleet
+
+When multiple Zuplo projects form a gateway chain, decide where to attach your
+[custom domain](../articles/custom-domains.mdx):
+
+- **Outer gateway only** — The most common setup. Attach your custom domain (for
+  example, `api.example.com`) to the outer gateway project and let the inner
+  gateways use their default `*.zuplo.app` URLs. Callers only see your custom
+  domain.
+- **Every gateway** — Useful when internal teams also call the inner gateways
+  directly for testing or monitoring. Each project gets its own custom domain.
+
+:::tip
+
+Putting the custom domain only on the outer gateway simplifies DNS management
+and certificate renewal. The inner gateways are implementation details that
+callers do not need to know about.
+
+:::
+
+## Cost considerations
+
+Request traffic is metered at the account level, and every project in the chain
+counts against the same shared allowance. A single client request that fans out
+to three downstream Zuplo projects results in four billed requests: one on the
+outer gateway and one on each downstream project.
+
+Review [Platform Limits](../articles/limits.mdx) and your plan's monthly request
+allowance before designing a fan-out architecture. If the request volume is
+high, consider whether a single Zuplo project with path-based routing can
+replace the multi-project topology.
+
+## Troubleshooting
+
+### 522 with no logs on the downstream project
+
+The outer gateway's runtime could not establish a TCP connection to the
+downstream project. The request never reached the inner gateway, so there are no
+logs there.
+
+**Checklist:**
+
+1. Verify the downstream URL is correct. Check the environment variable value in
+   the outer gateway project. A typo in the environment name (for example,
+   `main` instead of `main-abc123`) produces a DNS failure.
+2. Confirm the downstream project is deployed and its environment is active.
+   Open the downstream project in the [Zuplo Portal](https://portal.zuplo.com)
+   and check the environment status.
+3. If using a custom domain on the downstream project, verify the DNS CNAME
+   record points to `cname.zuplo.app` and the certificate is valid.
+
+### 522 only when forwarding to another Zuplo project
+
+Requests to `httpbin.org` or other external services work fine, but requests to
+`*.zuplo.app` return 522.
+
+**Checklist:**
+
+1. Check that the downstream Zuplo project's environment is not a development
+   environment (ending in `.zuplo.dev`). Development environments have stricter
+   rate limits (1,000 requests per minute) and may reject connections under
+   load.
+2. Verify TLS is working — the outer gateway connects to the downstream over
+   HTTPS. If the downstream has a custom domain with certificate issues, the TLS
+   handshake fails and produces a 522.
+3. Look at the outer gateway's logs for connection error details. The Zuplo
+   runtime logs include the error message when an outbound `fetch` fails.
+
+### Caller receives the upstream 401 directly
+
+This is the expected behavior. The URL Forward Handler and custom fetch handlers
+both return the upstream's HTTP status and body as-is. If the caller sees `401`,
+the downstream project rejected the request at the HTTP level (not a connection
+failure).
+
+If the downstream uses API key authentication and the caller's key is not valid
+on the downstream project, the downstream returns `401`. Review the
+[Propagating authentication](#propagating-authentication) section to choose the
+right credential strategy.
+
+### Mismatched response content types
+
+The downstream project returns JSON but the caller receives an unexpected
+content type or an empty body.
+
+**Checklist:**
+
+1. Verify the downstream route is configured to return the expected content
+   type. Check the handler and outbound policies on the downstream project.
+2. If using a custom fetch handler on the outer gateway, make sure you are
+   forwarding the upstream's `content-type` header. The example handler in
+   [Pattern B](#pattern-b-custom-fetch-handler) preserves all upstream headers.
+3. Check whether an outbound policy on the outer gateway transforms or strips
+   the response body.
+
+## Related resources
+
+- [URL Forward Handler](../handlers/url-forward.mdx)
+- [Function Handler](../handlers/custom-handler.mdx)
+- [Federated Gateways (Managed Dedicated)](../dedicated/federated-gateways.mdx)
+- [Securing your backend](../articles/securing-your-backend.mdx)
+- [Platform Limits](../articles/limits.mdx)
+- [Gateway Timeout error](../errors/gateway-timeout.mdx)
+- [Request Lifecycle](../concepts/request-lifecycle.mdx)
+- [Custom Domains](../articles/custom-domains.mdx)
+- [Environment Variables](../articles/environment-variables.mdx)

package/docs/handlers/mcp-server.mdx

@@ -121,6 +121,17 @@ client, ensure your `outputSchema` is a valid `type: object` JSON Schema and
   included as `structuredContent`. When `false`, only `text` content will be
   returned.
 
+:::caution
+
+Enable `includeOutputSchema` and `includeStructuredContent` together. Turning on
+`includeOutputSchema` alone makes the server advertise an output schema while
+returning only `text` content, which causes MCP clients to fail tool calls with
+`Tool call failed: 500`. See
+[Troubleshooting the MCP Server Handler](../mcp-server/troubleshooting.mdx) for
+how to diagnose and fix this.
+
+:::
+
 ### Operations
 
 Configure MCP tools, prompts, and resources using the `operations`

package/docs/mcp-server/troubleshooting.mdx

@@ -0,0 +1,195 @@
+---
+title: "Troubleshooting the MCP Server Handler"
+sidebar_label: "Troubleshooting"
+description:
+  Diagnose MCP Server handler tool calls that fail in an AI client even though
+  the gateway route returns 200, using debug logging and the structured-content
+  configuration.
+---
+
+When an AI client calls a tool exposed by the
+[MCP Server handler](../handlers/mcp-server.mdx), a failure often surfaces as
+nothing more than `Tool call failed: 500`, with no indication of _why_. The
+underlying gateway route can return `200` while the MCP client still rejects the
+response, which makes these failures hard to diagnose from the client alone.
+
+This page shows how to turn on the handler's debug logging, read the log lines
+that reveal a misconfiguration, and fix the most common cause: a tool that
+advertises an output schema but returns no structured content.
+
+:::tip
+
+Most "200 on the gateway, error in the client" failures come from the
+[`includeOutputSchema` and `includeStructuredContent` options](../handlers/mcp-server.mdx#mcp-2025-06-18-global-options).
+If you enabled one without the other, jump straight to
+[Tool call failed: 500](#tool-call-failed-500--has-an-output-schema-but-did-not-return-structured-content).
+
+:::
+
+## Enable debug logging
+
+The MCP Server handler can emit verbose logs covering server startup, tool
+registration, and each tool call. These logs are the fastest way to confirm what
+the server actually advertised to the client.
+
+<Stepper>
+
+1. **Turn on `debugMode`.** Set `debugMode: true` in the handler options for
+   your MCP route in `routes.oas.json`:
+
+   ```json
+   "post": {
+     "x-zuplo-route": {
+       "handler": {
+         "export": "mcpServerHandler",
+         "module": "$import(@zuplo/runtime)",
+         "options": {
+           "name": "example-mcp-server",
+           "version": "1.0.0",
+           "debugMode": true
+         }
+       }
+     }
+   }
+   ```
+
+2. **Redeploy so the server cold-starts.** Some of the most useful log lines —
+   server startup and tool registration — are written only when the MCP server
+   initializes (a cold start). An already-running worker won't re-emit them.
+   Deploy the working copy or environment again so a fresh worker boots with
+   `debugMode` enabled.
+
+3. **View the logs.** Open your project in the Portal and go to **Observability
+   → Logs**. Trigger a tool call from your MCP client, then read the entries for
+   the MCP route. Filter on the request's `zuplo-request-id` to isolate a single
+   call.
+
+</Stepper>
+
+:::caution
+
+Leave `debugMode` off in production. It logs verbose per-call detail that adds
+noise and overhead. Turn it on to diagnose an issue, then turn it back off and
+redeploy.
+
+:::
+
+### Key log lines to read
+
+With `debugMode: true`, the handler writes lines at each stage of its lifecycle.
+The exact format may change between runtime versions, but the fields below are
+what you're looking for:
+
+| Log line                       | When it's written       | What to check                                                          |
+| ------------------------------ | ----------------------- | ---------------------------------------------------------------------- |
+| `MCP Server cold start`        | A fresh worker boots    | Confirms `debugMode` is active and a new worker started                |
+| `MCP tool registered`          | Each tool is registered | The `includeStructuredContent` and `hasOutputSchema` fields for a tool |
+| `MCP Server response complete` | A tool call finishes    | The response was produced and sent back to the client                  |
+
+The `MCP tool registered` line is the important one. For a tool whose calls fail
+in the client, you'll typically see a line resembling:
+
+```
+MCP tool registered { name: "get_current_weather", hasOutputSchema: true, includeStructuredContent: false }
+```
+
+`hasOutputSchema: true` with `includeStructuredContent: false` is the exact
+misconfiguration described in the next section.
+
+## Tool call failed: 500 / "has an output schema but did not return structured content"
+
+**Symptom.** An AI client (such as Claude) reports `Tool call failed: 500` when
+it invokes a tool. With more verbose client logging, the underlying error reads:
+
+```
+Tool <tool_name> has an output schema but did not return structured content
+```
+
+Meanwhile, the gateway's own logs show the route that backs the tool returned
+`200`.
+
+**Likely cause.** The handler is configured with `includeOutputSchema: true` but
+not `includeStructuredContent: true`. Under the
+[MCP `2025-06-18`](https://modelcontextprotocol.io/specification/2025-06-18)
+structured-content behavior, when a tool advertises an `outputSchema`, the
+client expects every successful result to include a matching `structuredContent`
+object. With `includeOutputSchema` on and `includeStructuredContent` off, the
+server advertises the schema but returns only `text` content, so a
+spec-compliant client rejects an otherwise-successful `200` response.
+
+The `MCP tool registered` debug line confirms it:
+`hasOutputSchema: true, includeStructuredContent: false`.
+
+**Fix.** Enable both options together on the MCP Server handler:
+
+```json
+"options": {
+  "name": "example-mcp-server",
+  "version": "1.0.0",
+  "includeOutputSchema": true,
+  "includeStructuredContent": true
+}
+```
+
+Then redeploy. The next `MCP tool registered` line should read
+`hasOutputSchema: true, includeStructuredContent: true`, and the tool call
+succeeds.
+
+:::note
+
+If you don't need output schemas at all, the alternative fix is to turn both off
+(the defaults). The two options are designed to move together: enable
+`includeOutputSchema` _only_ alongside `includeStructuredContent`.
+
+:::
+
+For the full definitions of both options, see the
+[MCP `2025-06-18` Global Options](../handlers/mcp-server.mdx#mcp-2025-06-18-global-options)
+in the handler reference.
+
+## Client-side debugging tools
+
+When the gateway looks healthy, reproduce the failure against the client to see
+the protocol-level error the AI client hides:
+
+- **MCP Inspector** — the
+  [official inspector](https://github.com/modelcontextprotocol/inspector)
+  (`npx @modelcontextprotocol/inspector`) connects to your remote server, lists
+  tools, and calls them while showing the raw JSON-RPC messages. See
+  [Testing](./testing.mdx) for connection steps.
+- **mcpjam** — the [mcpjam inspector](https://github.com/mcpjam/inspector) is an
+  open-source MCP testing client that's useful for exercising tool calls and
+  inspecting responses outside of an AI client.
+- **Claude Code** — run with the `--debug` flag (for example, `claude --debug`)
+  to print MCP protocol traffic, including the full tool-call error that the
+  chat UI summarizes as `Tool call failed: 500`.
+
+## Checklist: the gateway returns 200 but the client errors
+
+When a tool call fails in the client but the gateway reports success, verify
+each of these in order:
+
+1. **`debugMode` is on and a fresh worker booted.** Confirm a
+   `MCP Server cold start` line appears after your latest deploy. Without a cold
+   start, you're reading stale logs.
+2. **Schema and structured content move together.** In the `MCP tool registered`
+   line, `hasOutputSchema` and `includeStructuredContent` should either both be
+   `true` or both be `false` — never `hasOutputSchema: true` with
+   `includeStructuredContent: false`.
+3. **The output schema is a valid `type: object` JSON Schema.** Some clients
+   reject schemas that aren't `type: object`. The same applies to the
+   `structuredContent` the server returns. See the
+   [compatibility caveat](../handlers/mcp-server.mdx#mcp-2025-06-18-global-options).
+4. **The route really returns JSON.** `includeStructuredContent` parses the
+   response body as JSON to build `structuredContent`. A non-JSON `200` body
+   can't be parsed into the advertised schema.
+5. **Reproduce in a raw client.** Call the tool through the
+   [MCP Inspector](./testing.mdx) or `curl` to read the JSON-RPC error directly,
+   rather than the client's summarized `500`.
+
+## Next steps
+
+- [MCP Server handler reference](../handlers/mcp-server.mdx) — every handler
+  configuration option.
+- [Testing your MCP server](./testing.mdx) — MCP Inspector and `curl` recipes.
+- [MCP Server tools](./tools.mdx) — editing tool definitions and `outputSchema`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "zuplo",
-  "version": "6.71.8",
+  "version": "6.71.10",
   "type": "module",
   "description": "The programmable API Gateway",
   "author": "Zuplo, Inc.",
@@ -19,9 +19,9 @@
     "zuplo": "zuplo.js"
   },
   "dependencies": {
-    "@zuplo/cli": "6.71.8",
-    "@zuplo/core": "6.71.8",
-    "@zuplo/runtime": "6.71.8",
+    "@zuplo/cli": "6.71.10",
+    "@zuplo/core": "6.71.10",
+    "@zuplo/runtime": "6.71.10",
     "@zuplo/test": "1.4.0"
   }
 }