zuplo 6.67.32 → 6.68.0

package/docs/articles/log-plugin-vmware-log-insight.mdx

@@ -0,0 +1,154 @@
+---
+title: VMware Log Insight Log Plugin
+sidebar_label: VMware Log Insight
+---
+
+The VMware Log Insight plugin enables pushing logs to your VMware Log Insights
+via the REST API.
+
+<EnterpriseFeature name="Custom logging" />
+
+## Setup
+
+To add the VMware Log Insight logging plugin to your Zuplo project, add the
+following code to your `zuplo.runtime.ts` file.
+
+```ts title="modules/zuplo.runtime.ts"
+import {
+  RuntimeExtensions,
+  VMwareLogInsightLoggingPlugin,
+  environment,
+} from "@zuplo/runtime";
+
+export function runtimeInit(runtime: RuntimeExtensions) {
+  runtime.addPlugin(
+    new VMwareLogInsightLoggingPlugin({
+      // This is the URL of your VMware Log Insight host
+      url: "https://loginsight.example.com",
+      fields: {
+        appname: "zuplo",
+      },
+    }),
+  );
+}
+```
+
+## Configuration Options
+
+The `VMwareLogInsightLoggingPlugin` constructor accepts an options object with
+the following properties:
+
+- `url` - (required) The URL of your VMware Log Insight host (for example,
+  `https://loginsight.example.com`)
+- `agentId` - (optional) The unique agent identifier of the logger
+- `fields` - (optional) Custom fields to include in each log entry. Can contain
+  string, number, or boolean values
+- `textReplacements` - (optional) An array of string tuples to replace within
+  the text field of a log entry
+- `onMessageSending` - (optional) A callback function to modify log entries
+  before sending
+
+### Custom Fields
+
+Any custom fields you want to include in the log entry can be added to the
+`fields` property. These values will be appended to every log entry.
+
+### Text Replacements
+
+The `textReplacements` option allows you to specify character replacements in
+the log text. For example: `[["'", ""], ['"', ""], ["\\n", ""], [":", "="]]`
+
+## Default Fields
+
+Every log entry will have a `timestamp` and a `text` field. The text field
+contains the log message, which may be JSON encoded for complex data.
+
+Default fields are (in snake_case format):
+
+- `severity` - The log level (for example, `ERROR`, `INFO`, `DEBUG`, `WARN`)
+- `request_id` - The UUID of the request (the value of the `zp-rid` header)
+- `environment_type` - Where the Zuplo API is running. Values are `edge`,
+  `working-copy`, or `local`
+- `environment_stage` - The deployment stage: `working-copy`, `preview`, or
+  `production`
+- `log_source` - The source of the log. Either `user` or `system`
+- `atomic_counter` - An atomic counter used to order logs with identical
+  timestamps
+- `environment` - The environment name of the Zuplo API
+- `request_ray_id` - The network provider identifier (for example, Cloudflare
+  Ray ID) of the request
+
+:::note
+
+VMware Log Insight uses snake_case naming convention for field names.
+
+:::
+
+## Log Format
+
+The shape of the logs sent from Zuplo will be in the following format:
+
+```json
+{
+  "timestamp": 1696596905883,
+  "text": "hello world",
+  "fields": [
+    { "name": "severity", "content": "INFO" },
+    {
+      "name": "request_id",
+      "content": "709d2491-0703-4ea9-86ea-d2af548cd4d9"
+    },
+    { "name": "environment_type", "content": "working-copy" },
+    { "name": "log_source", "content": "request" },
+    { "name": "atomic_counter", "content": 1 }
+  ]
+}
+```
+
+## Example Logs
+
+When objects are logged, they will be converted to a key value string format as
+shown below.
+
+```json
+{
+  "timestamp": 1696603735057,
+  "text": "hello=\"hello world\" foo=1 baz=true",
+  "fields": [
+    { "name": "severity", "content": "INFO" },
+    {
+      "name": "request_id",
+      "content": "709d2491-0703-4ea9-86ea-d2af548cd4d9"
+    },
+    { "name": "environment_type", "content": "working-copy" },
+    { "name": "log_source", "content": "request" },
+    { "name": "atomic_counter", "content": 1 }
+  ]
+}
+```
+
+Errors will be included as fields in the log. The fields are `error_name`,
+`error_message`, and `error_stack`.
+
+```json
+{
+  "timestamp": 1696603735055,
+  "text": "Something bad happened",
+  "fields": [
+    { "name": "severity", "content": "INFO" },
+    {
+      "name": "request_id",
+      "content": "709d2491-0703-4ea9-86ea-d2af548cd4d9"
+    },
+    { "name": "environment_type", "content": "working-copy" },
+    { "name": "log_source", "content": "request" },
+    { "name": "atomic_counter", "content": 1 },
+    { "name": "error_name", "content": "Error" },
+    { "name": "error_message", "content": "This is an error" },
+    {
+      "name": "error_stack",
+      "content": "Error: This is an error\n    at exampleFunction (module/foo.ts:32:21)"
+    }
+  ]
+}
+```

package/docs/articles/log-request-response-data.mdx

@@ -0,0 +1,398 @@
+---
+title: Log Custom Request and Response Data
+sidebar_label: Log Request/Response Data
+---
+
+Logging request and response data is useful for debugging API issues, monitoring
+traffic patterns, and auditing API usage. This guide shows how to create custom
+policies that log various parts of requests and responses while redacting
+sensitive information.
+
+## Logging Request Headers
+
+Create an inbound policy to log headers from incoming requests. This policy
+redacts sensitive headers like `Authorization` and `Cookie` to prevent exposing
+credentials in logs.
+
+```ts title="modules/log-request-headers.ts"
+import type { ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+export default async function policy(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const headers: Record<string, string> = {};
+  for (const [key, value] of request.headers.entries()) {
+    const k = key.toLowerCase();
+    headers[key] =
+      k === "authorization" ||
+      k === "cookie" ||
+      k === "set-cookie" ||
+      k === "x-api-key"
+        ? "[REDACTED]"
+        : value;
+  }
+
+  context.log.info({ headers }, "Incoming request headers");
+
+  return request;
+}
+```
+
+## Logging Query Parameters
+
+Log query parameters from the request URL:
+
+```ts title="modules/log-query-params.ts"
+import type { ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+export default async function policy(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const url = new URL(request.url);
+  const queryParams = Object.fromEntries(url.searchParams);
+
+  context.log.info(
+    {
+      path: url.pathname,
+      query: queryParams,
+    },
+    "Request query parameters",
+  );
+
+  return request;
+}
+```
+
+## Logging Request Body
+
+Log the request body for POST, PUT, or PATCH requests. Clone the request first
+to avoid consuming the body stream.
+
+```ts title="modules/log-request-body.ts"
+import type { ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+export default async function policy(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  if (["POST", "PUT", "PATCH"].includes(request.method)) {
+    const clone = request.clone();
+    const body = await clone.text();
+
+    // Parse JSON if applicable
+    let parsedBody: unknown;
+    try {
+      parsedBody = JSON.parse(body);
+    } catch {
+      parsedBody = body;
+    }
+
+    context.log.info({ body: parsedBody }, "Request body");
+  }
+
+  return request;
+}
+```
+
+:::warning
+
+Logging request bodies can expose sensitive data like passwords, tokens, or
+personal information. Always sanitize or redact sensitive fields before logging.
+
+:::
+
+## Logging Response Headers and Status
+
+Create an outbound policy to log response information:
+
+```ts title="modules/log-response-headers.ts"
+import type { ZuploContext, ZuploRequest, ZuploResponse } from "@zuplo/runtime";
+
+export default async function outboundPolicy(
+  response: ZuploResponse,
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const headers: Record<string, string> = {};
+  for (const [key, value] of response.headers.entries()) {
+    const k = key.toLowerCase();
+    headers[key] = k === "set-cookie" ? "[REDACTED]" : value;
+  }
+
+  context.log.info(
+    { status: response.status, headers },
+    "Outgoing response headers",
+  );
+
+  return response;
+}
+```
+
+## Logging Response Body
+
+Log the response body from your backend. Clone the response first to avoid
+consuming the body stream.
+
+```ts title="modules/log-response-body.ts"
+import type { ZuploContext, ZuploRequest, ZuploResponse } from "@zuplo/runtime";
+
+export default async function outboundPolicy(
+  response: ZuploResponse,
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const clone = response.clone();
+  const body = await clone.text();
+
+  let parsedBody: unknown;
+  try {
+    parsedBody = JSON.parse(body);
+  } catch {
+    parsedBody = body;
+  }
+
+  context.log.info(
+    {
+      status: response.status,
+      body: parsedBody,
+    },
+    "Response body",
+  );
+
+  return response;
+}
+```
+
+## Comprehensive Request Logging
+
+Combine multiple data points into a single log entry:
+
+```ts title="modules/log-request-details.ts"
+import type { ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+export default async function policy(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  const url = new URL(request.url);
+
+  context.log.info(
+    {
+      method: request.method,
+      path: url.pathname,
+      query: Object.fromEntries(url.searchParams),
+      headers: sanitizeHeaders(request.headers),
+      userId: request.user?.sub,
+      params: request.params,
+    },
+    "Incoming request",
+  );
+
+  return request;
+}
+
+function sanitizeHeaders(headers: Headers): Record<string, string> {
+  const sensitiveHeaders = [
+    "authorization",
+    "cookie",
+    "set-cookie",
+    "x-api-key",
+  ];
+  const result: Record<string, string> = {};
+
+  for (const [key, value] of headers.entries()) {
+    result[key] = sensitiveHeaders.includes(key.toLowerCase())
+      ? "[REDACTED]"
+      : value;
+  }
+
+  return result;
+}
+```
+
+## Policy Configuration
+
+Configure the policy in your `policies.json`:
+
+```json title="config/policies.json"
+{
+  "name": "log-request-data",
+  "policyType": "custom-code-inbound",
+  "handler": {
+    "export": "default",
+    "module": "$import(./modules/log-request-details)"
+  }
+}
+```
+
+For outbound policies:
+
+```json title="config/policies.json"
+{
+  "name": "log-response-data",
+  "policyType": "custom-code-outbound",
+  "handler": {
+    "export": "default",
+    "module": "$import(./modules/log-response-headers)"
+  }
+}
+```
+
+## Wiring Up the Policies
+
+Add the policies to your routes in `routes.oas.json`:
+
+```json title="config/routes.oas.json"
+{
+  "paths": {
+    "/my-route": {
+      "get": {
+        "x-zuplo-route": {
+          "handler": {
+            "export": "urlForwardHandler",
+            "module": "$import(@zuplo/runtime)",
+            "options": {
+              "baseUrl": "https://api.example.com"
+            }
+          },
+          "policies": {
+            "inbound": ["log-request-data"],
+            "outbound": ["log-response-data"]
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+## Configurable Options
+
+Make the logging behavior configurable using policy options:
+
+```ts title="modules/log-request-configurable.ts"
+import type { ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+type PolicyOptions = {
+  logHeaders?: boolean;
+  logQuery?: boolean;
+  logBody?: boolean;
+  redactedHeaders?: string[];
+};
+
+const DEFAULT_REDACTED = ["authorization", "cookie", "set-cookie", "x-api-key"];
+
+export default async function policy(
+  request: ZuploRequest,
+  context: ZuploContext,
+  options: PolicyOptions,
+  policyName: string,
+) {
+  const url = new URL(request.url);
+  const logData: Record<string, unknown> = {
+    method: request.method,
+    path: url.pathname,
+  };
+
+  if (options.logQuery !== false) {
+    logData.query = Object.fromEntries(url.searchParams);
+  }
+
+  if (options.logHeaders !== false) {
+    const redacted = (options.redactedHeaders ?? DEFAULT_REDACTED).map((h) =>
+      h.toLowerCase(),
+    );
+    logData.headers = sanitizeHeaders(request.headers, redacted);
+  }
+
+  if (options.logBody && ["POST", "PUT", "PATCH"].includes(request.method)) {
+    const clone = request.clone();
+    const body = await clone.text();
+    try {
+      logData.body = JSON.parse(body);
+    } catch {
+      logData.body = body;
+    }
+  }
+
+  context.log.info(logData, "Incoming request");
+
+  return request;
+}
+
+function sanitizeHeaders(
+  headers: Headers,
+  redacted: string[],
+): Record<string, string> {
+  const result: Record<string, string> = {};
+  for (const [key, value] of headers.entries()) {
+    result[key] = redacted.includes(key.toLowerCase()) ? "[REDACTED]" : value;
+  }
+  return result;
+}
+```
+
+Configure with options:
+
+```json title="config/policies.json"
+{
+  "name": "log-request-data",
+  "policyType": "custom-code-inbound",
+  "handler": {
+    "export": "default",
+    "module": "$import(./modules/log-request-configurable)",
+    "options": {
+      "logHeaders": true,
+      "logQuery": true,
+      "logBody": false,
+      "redactedHeaders": ["authorization", "cookie", "x-api-key", "x-secret"]
+    }
+  }
+}
+```
+
+## Best Practices
+
+:::warning
+
+Always redact sensitive data before logging. Credentials, tokens, passwords, and
+personal information should never appear in logs.
+
+:::
+
+1. **Redact sensitive data** - Always redact `Authorization`, `Cookie`,
+   `Set-Cookie`, API keys, passwords, and any fields containing secrets or
+   personal data.
+
+2. **Use structured logging** - Pass objects to `context.log` instead of string
+   concatenation. This enables better log searching and filtering.
+
+3. **Clone before reading** - Always clone requests and responses before reading
+   their body to avoid consuming the stream.
+
+4. **Consider log volume** - Logging bodies can generate significant log volume
+   and storage costs. Consider enabling body logging only for specific routes or
+   in development environments.
+
+5. **Use appropriate log levels** - Use `debug` for verbose development logging
+   and `info` for production audit trails.
+
+6. **Limit body size** - Consider truncating large bodies to avoid excessive log
+   storage:
+
+   ```ts
+   const body = await clone.text();
+   const truncated = body.length > 1000 ? body.slice(0, 1000) + "..." : body;
+   ```
+
+## See Also
+
+- [Logger](../programmable-api/logger.mdx) - Logger interface documentation
+- [Custom Code Inbound Policy](../policies/custom-code-inbound.mdx) - Writing
+  custom inbound policies
+- [Custom Code Outbound Policy](../policies/custom-code-outbound.mdx) - Writing
+  custom outbound policies
+- [Custom Logging Policy](./custom-logging-example.mdx) - Full request/response
+  logging to external services

package/docs/articles/logging.mdx

@@ -0,0 +1,115 @@
+---
+title: Logging
+sidebar_label: Overview
+---
+
+Zuplo provides real-time logging out of the box. If you would like your logs to
+be sent to your own logging service, you can enable one of Zuplo's logging
+plugins.
+
+To configure your logging, you need to create a `zuplo.runtime.ts` file in the
+`modules`. The examples below show the content of the file with each of the
+different logging plugins.
+
+## Log Integrations
+
+Zuplo offers out-of-the box integrations with many common logging vendors. For
+instructions on how to configure logging, see the documentation for each plugin:
+
+- [AWS CloudWatch](./log-plugin-aws-cloudwatch.mdx)
+- [Datadog](./log-plugin-datadog.mdx)
+- [Dynatrace](./log-plugin-dynatrace.mdx)
+- [New Relic](./log-plugin-new-relic.mdx)
+- [Google Cloud Logging](./log-plugin-gcp.mdx)
+- [Loki](./log-plugin-loki.mdx)
+- [Splunk](./log-plugin-splunk.mdx)
+- [Sumo Logic](./log-plugin-sumo.mdx)
+- [VMware Log Insight](./log-plugin-vmware-log-insight.mdx)
+
+:::info
+
+Not seeing the logging plugin you need? Reach out to
+[support@zuplo.com](mailto:support@zuplo.com) and we'd be happy to work with you
+to add a new logging plugin.
+
+:::
+
+## Custom Logging
+
+In addition to the logging plugins, you can also create your own custom logging
+plugin. For more information, see the
+[Custom Logging Plugin](./custom-logging-example.mdx) documentation.
+
+## Log Fields
+
+Below is a list of the default fields that are sent with log messages. Note that
+the names of these fields may differ depending on your logger as we follow the
+conventions of each log service. So `environmentType` may be `environmentType`,
+`environment_type`, or `environment-type`. See the specific log plugin for
+details.
+
+- `severity`: The log level of the message. Values are `debug`, `info`, `warn`,
+  `error`
+- `requestId`: The value of the `zp-rid` header. This is used for tracing issues
+  across Zuplo systems.
+- `atomicCounter`: This is a counter that indicates log ordering. Because of the
+  shared nature of the edge environments the clock doesn't incriment unless an
+  I/O operation is performed. As such, you may notice that you have several
+  messages with the same timestamp. You can use the value of this counter to
+  determine order. This value will be an integer between 0 and the max integer
+  value. It will cycle back to 0 when the maximum is reached. This number isn't
+  persistent across restarts or shared across environments.
+- `environment`: This is the name of your Zuplo environment. This will be the
+  same as your Zuplo subdomain. for example if your Zuplo URL is
+  `https://silver-lemming-main-b0cef33.zuplo.app`, the environment is
+  `silver-lemming-main-b0cef33`
+- `environmentType`: This indicates where your environment is running. Possible
+  values are:
+  - `edge`: Environments deployed to our 300+ edge locations
+  - `working-copy`: Environments deployed to your single-instance dev server
+  - `local`: When running with Zuplo local development
+- `environmentStage`: This indicates the deployment stage of your environment.
+  Possible values are:
+  - `production`: Environments deployed from your default git branch
+  - `preview`: Environments deployed from any other git branch
+  - `working-copy`: Environments deployed to your single-instance dev server
+  - `local`: When running with Zuplo local development
+- `loggingId`: This string is a unique identifier that combines your environment
+  name, branch name, and project name into one string. We advise against using
+  this value and instead recommend using the `environment` and
+  `environmentStage` values for filtering. This value will likely be deprecated
+  in future releases.
+- `buildId`: A UUID representing the unique build of your API. This value
+  changes every time a new version of your API is built and deployed.
+- `logSource`: Whether the log originated from a request (`request`) or from
+  outside of a request (`runtime`)
+- `rayId`: The value of the network provider request ID (that is Cloudflare Ray
+  ID). This value is used internally to coordinate log event in Zuplo to log
+  events in the environment where your API runs. It's provided in your logs for
+  potential troubleshooting. Normally, it's recommended to rely on the
+  `requestId` for tracing.
+
+## Custom Log Properties
+
+In addition to the default fields, you can also add custom fields to your logs.
+This can be done using the `context.log.setLogProperties!` method.
+
+```ts
+import { ZuploContext, ZuploRequest } from "@zuplo/runtime";
+
+export default async function policy(
+  request: ZuploRequest,
+  context: ZuploContext,
+) {
+  context.log.setLogProperties!({ customProperty: "value" });
+  context.log.info("Hello World");
+  return request;
+}
+```
+
+These properties will be included in all subsequent log messages for that
+request. If the property already exists, it will be overwritten.
+
+The log properties will be exported in a format native to the plugin you are
+using. For example, in the case of the OpenTelemetry plugin, the properties will
+be included in the `attributes` field of the log record.