zilmate 1.8.0 → 1.8.5

package/.agents/skills/upstash/upstash-workflow-js/features/retries-failures-reliability.md

@@ -0,0 +1,239 @@
+# Workflow Reliability Features
+
+This documentation provides a consolidated reference for reliability mechanisms in Upstash Workflow, focusing on retries, failure handling, and the Dead Letter Queue (DLQ). It is optimized for AI‑driven execution but remains clear for developers.
+
+---
+
+## Core Reliability Concepts
+
+Upstash Workflow ensures that workflow runs are durable, recoverable, and traceable. Reliability is achieved through:
+
+- Automatic retries for transient failures
+- Failure handling via `failureFunction` or `failureUrl`
+- DLQ for storing permanently failed runs
+- Manual recovery actions: restart, resume, rerun failure function
+- Mechanisms to prevent retries when failure is intentional
+
+These mechanisms work together to guarantee that no failed execution is lost and that failures can be observed and handled.
+
+---
+
+## Automatic Retries
+
+Workflow steps automatically retry when they fail, using configurable retry count and delay.
+
+### Retry Parameters
+
+- `retries`: How many times the step will retry
+- `retryDelay`: A math expression (string) returning delay in milliseconds; can use `retried` variable
+
+### Common Pitfalls
+
+- The delay expression must be a _string_; raw numbers disable dynamic calculation.
+- A workflow only moves to DLQ after **all** retries fail.
+
+### Example: Configure Retry Count + Delay
+
+```ts
+import { Client, WorkflowNonRetryableError, serve } from "@upstash/workflow";
+
+// Trigger with custom retry settings
+const client = new Client({ token: process.env.WORKFLOW_TOKEN! });
+await client.trigger({
+  url: "https://example.com/workflow",
+  retries: 5,
+  retryDelay: "(1 + retried) * 2000", // 2s, 4s, 6s...
+});
+
+// Workflow with steps that may intentionally skip retries
+export const { POST } = serve(async (context) => {
+  const shouldFail = await context.run("check", () => true);
+
+  if (shouldFail) {
+    // Skips retry cycle and sends directly to failure handling
+    throw new WorkflowNonRetryableError("Bad state");
+  }
+
+  await context.run("process", () => doWork());
+});
+```
+
+---
+
+## Preventing Retries
+
+### When to Stop Retries
+
+Use these when failure is expected or should halt execution immediately:
+
+- `WorkflowNonRetryableError` → Fail fast, trigger failure function, enter DLQ
+- `context.cancel()` → Stop workflow intentionally, no DLQ, no failure function
+- Conditional returns → Graceful early exit without error
+
+### Pitfalls
+
+- `context.cancel()` marks the run as **canceled**, not failed — no failure handling executes.
+- Throwing any error _other than_ `WorkflowNonRetryableError` will still trigger retries.
+
+---
+
+## Failure Handling
+
+You can handle workflow failures using either:
+
+- `failureFunction` (runs in your workflow environment)
+- `failureUrl` (external callback endpoint)
+
+Only **one** can be configured — they are mutually exclusive.
+
+### Failure Function Behavior
+
+- Runs after all retries fail
+- Receives failure metadata and workflow context
+- Cannot create new workflow steps; `context` is read‑only
+- Is retried if it fails
+- Can be manually retried from the DLQ
+
+### Failure Function Parameters
+
+- `context.workflowRunId`
+- `context.url`
+- `context.requestPayload`
+- `context.headers`
+- `context.env`
+- `failStatus`
+- `failResponse`
+- `failHeaders`
+
+### Example: Workflow + Failure Function + Prevent‑Retry Logic
+
+```ts
+export const { POST } = serve(
+  async (context) => {
+    const ok = await context.run("verify", () => verifySomething());
+
+    if (!ok) {
+      throw new WorkflowNonRetryableError("Verification failed");
+    }
+
+    const shouldAbort = await context.run("check", () => false);
+    if (shouldAbort) {
+      await context.cancel();
+      return;
+    }
+
+    await context.run("execute", () => doWork());
+  },
+  {
+    failureFunction: async ({ context, failStatus, failResponse }) => {
+      // Logging / cleanup / alerts
+      await logFailure(context.workflowRunId, failStatus, failResponse);
+    },
+  }
+);
+```
+
+---
+
+## Dead Letter Queue (DLQ)
+
+A workflow enters the DLQ when:
+
+- A step fails _after all retries_
+- A `WorkflowNonRetryableError` is thrown
+- `failureFunction` also fails (visible as failed in DLQ)
+
+### Retention
+
+- Free: 3 days
+- Pay‑as‑you‑go: 1 week
+- Fixed: Up to 3 months
+
+After retention expires, items cannot be recovered.
+
+---
+
+## DLQ Recovery Actions
+
+You may perform these actions from the dashboard or programmatically.
+
+### 1. Resume
+
+Continue from the failed step while keeping previous step results.
+
+### 2. Restart
+
+Start the entire workflow from scratch, discarding previous results.
+
+### 3. Rerun Failure Function
+
+Retry the failure handler **only if it previously failed**.
+
+### Example: All DLQ Actions
+
+```ts
+import { Client } from "@upstash/workflow";
+const client = new Client({ token: process.env.WORKFLOW_TOKEN! });
+
+await client.dlq.resume({ dlqId: "dlq-1", retries: 3 }); // Continue run
+await client.dlq.restart({ dlqId: "dlq-2", retries: 3 }); // Restart
+await client.dlq.retryFailureFunction({ dlqId: "dlq-3" }); // Retry failure handler
+```
+
+### Pitfalls
+
+- `retryFailureFunction` is only allowed when the failure function failed.
+- `resume` cannot be used if workflow code changed before the failure point.
+- `restart` re-executes everything, including expensive steps.
+
+---
+
+## Failure URL (Advanced Option)
+
+Use when failure must be handled by a separate service.
+
+### Notes
+
+- Use **either** `failureFunction` **or** `failureUrl`.
+- Recommended only when your primary app might be offline.
+- Failure URL receives a structured JSON payload about the failed request.
+
+### Example
+
+```ts
+export const { POST } = serve(
+  async (context) => {
+    await context.run("work", () => doWork());
+  },
+  {
+    failureUrl: "https://example.com/failure-handler",
+  }
+);
+```
+
+---
+
+## Debugging Failed Runs
+
+DLQ entries store:
+
+- Request + response headers
+- Payloads
+- Failure metadata
+- Failure function state
+
+Use the dashboard filters (workflow run ID, URL, failure function state) to identify and debug root causes.
+
+---
+
+## Summary
+
+This skill unifies all Workflow reliability controls:
+
+- Retries (automatic, configurable)
+- Failure handling (`failureFunction`, `failureUrl`)
+- DLQ retention + recovery
+- Fast‑fail mechanisms (`WorkflowNonRetryableError`, `cancel`, conditional exits)
+- Manual recovery operations (resume, restart, retry failure function)
+
+Together these features ensure workflows are resilient, debuggable, and recoverable under all failure scenarios.

package/.agents/skills/upstash/upstash-workflow-js/features/wait-for-event.md

@@ -0,0 +1,97 @@
+# Wait for Event
+
+Use this feature to pause workflow execution until an external event arrives. It allows building asynchronous, event‑driven workflows without holding compute resources.
+
+---
+
+## How It Works
+
+`context.waitForEvent()` suspends the workflow and registers a waiter under an event ID. When a matching `notify` call is sent, the workflow resumes with the provided event data.
+
+• Each waiter has a timeout. If the event does not arrive in time, the workflow continues with `{ timeout: true }`.
+• Maximum timeout depends on your pricing tier.
+• Multiple workflow runs may wait on the same event ID; a notify call resumes all of them.
+
+---
+
+## Key Concepts & Pitfalls
+
+### Event IDs
+
+Use unique event IDs to avoid heavy fan‑out notifications.
+
+Example patterns:
+• `order-123-processing-complete`
+• `user-42-email-verified`
+
+### Race Conditions
+
+A notify call sent _before_ the workflow begins waiting is lost.
+
+To avoid this:
+• Always inspect the notify response.
+• Retry if no waiters were found.
+
+---
+
+## Combined Example
+
+Below is a single TypeScript example showing waiting for an event, handling timeouts, and safely notifying:
+
+```ts
+import { serve } from "@upstash/workflow/nextjs";
+import { Client } from "@upstash/workflow";
+
+// Workflow that waits for an event
+export const { POST } = serve(async (context) => {
+  const { orderId } = context.requestPayload;
+  const eventId = `order-${orderId}-processed`;
+
+  // Wait for the event with timeout
+  const { eventData, timeout } = await context.waitForEvent(
+    "wait-for-order-processing", // step name
+    eventId, // event id
+    {
+      timeout: "1d", // optional timeout
+    }
+  );
+
+  if (timeout) {
+    // handle timeout here
+    await context.run("timeout-handler", async () => {
+      console.log("Order processing timed out");
+    });
+    return;
+  }
+
+  // Continue workflow using eventData
+  await context.run("process-completed-order", async () => {
+    console.log("Order processed:", eventData);
+  });
+});
+
+// External notifier (safe retry pattern)
+const client = new Client({ token: "<WORKFLOW_TOKEN>" });
+
+async function notifyProcessingComplete(orderId: string, payload: any) {
+  const eventId = `order-${orderId}-processed`;
+
+  // First attempt - returns array of NotifyResponse
+  const waiters = await client.notify({ eventId, eventData: payload });
+
+  if (waiters > 0) return waiters;
+
+  // Retry if no workflows were waiting
+  await new Promise((r) => setTimeout(r, 3000));
+  return await client.notify({ eventId, eventData: payload });
+}
+```
+
+---
+
+## Best Practices
+
+• Generate event IDs that uniquely identify a specific workflow instance.
+• Always handle the timeout case explicitly.
+• On notification, check the returned array length to detect if any workflows were waiting.
+• Prefer one event ID per workflow run to avoid notifying large groups.

package/.agents/skills/upstash/upstash-workflow-js/features/webhooks.md

@@ -0,0 +1,79 @@
+## Webhooks
+
+This skill teaches agents how to use webhooks inside Upstash Workflow to pause execution, wait for external callbacks, and resume reliably. It covers creation, waiting, validation, and multi-step usage.
+
+---
+
+### Key Concepts
+
+• **Webhook URLs are unique per context.createWebhook() call**.
+• **Workflow execution pauses** when calling `context.waitForWebhook()` until the webhook receives a request or times out.
+• **Lookback protection** ensures the workflow still receives early webhook calls.
+• **Workflows remain dormant** during the waiting period, avoiding compute cost.
+• **Webhooks are ideal for third‑party APIs** that support callback URLs.
+
+---
+
+### Core Workflow: Create → Call External API → Wait → Resume
+
+Below is a consolidated example showing:
+• Creating a webhook
+• Passing it to an external API
+• Handling the callback
+• Waiting for multiple updates
+• Error/timeout handling
+
+```ts
+import { serve } from "@upstash/workflow/nextjs";
+
+export const { POST } = serve(async (context) => {
+  // Create a unique webhook for this workflow
+  const webhook = await context.createWebhook("create webhook");
+
+  // Trigger an external service, sending it the callback URL
+  await context.call("trigger-external", {
+    url: "https://api.example.com/process",
+    method: "POST",
+    body: { webhookUrl: webhook.webhookUrl },
+  });
+
+  // Loop: wait for progress updates until service signals completion
+  let updates = [];
+  while (true) {
+    const res = await context.waitForWebhook(`wait update`, webhook, "5m");
+
+    if (res.timeout) break; // no progress received
+
+    const req = res.request;
+    const payload = await req.json();
+    updates.push(payload);
+
+    if (req.headers.get("x-task-finished") === "true") break;
+  }
+
+  return updates;
+});
+```
+
+---
+
+### Webhook Reliability: Lookback Protection
+
+**Common pitfall:** external systems may call the webhook _before_ the workflow reaches `waitForWebhook()`.
+
+Upstash automatically stores early webhook calls. The next `waitForWebhook()` will immediately resolve with the stored request.
+
+This avoids race conditions and eliminates the need for manual buffering.
+
+---
+
+### When to Use Webhooks vs. Wait for Event
+
+Use **webhooks** when:
+• integrating with third‑party APIs
+• the external system invokes a callback URL
+• you need race‑condition‑safe behavior
+
+Use **Wait for Event** when:
+• you control the notifying system and can use the Workflow Client
+• you don't need lookback semantics

package/.agents/skills/upstash/upstash-workflow-js/how-to/local-dev.md

@@ -0,0 +1,177 @@
+# Local Development for Upstash Workflow
+
+This documentation explains how to run Upstash Workflow locally using the QStash development server and how to expose your local app using a public tunnel such as ngrok. It focuses on practical usage and TypeScript integration.
+
+## Overview
+
+Upstash Workflow uses Upstash QStash under the hood. During development you can:
+
+- Use the **automatic dev server** by setting `QSTASH_DEV=true`. The SDK downloads the QStash CLI binary, spawns the server, and verifies signatures with its dev keys. No tokens or signing keys required.
+- Run a **local QStash development server** manually via the CLI and wire credentials yourself.
+- Optionally expose your local server using **ngrok** if you want to test with the production QStash.
+
+---
+
+## Automatic dev server (recommended)
+
+Set `QSTASH_DEV=true` in your environment and that's it. Workflow's `serve()` endpoint and the `Client` both auto-detect it:
+
+```
+QSTASH_DEV=true
+```
+
+```ts
+// app/api/workflow/route.ts
+import { serve } from "@upstash/workflow/nextjs";
+
+export const { POST } = serve(async (context) => {
+  await context.run("step-1", () => console.log("running locally"));
+});
+```
+
+```ts
+import { Client } from "@upstash/workflow";
+
+const client = new Client({ token: process.env.QSTASH_TOKEN ?? "" });
+
+await client.trigger({
+  url: "http://localhost:3000/api/workflow",
+});
+```
+
+When `QSTASH_DEV=true`:
+
+- The underlying `@upstash/qstash` client downloads the QStash CLI binary on first use, spawns the dev server on port `8080` (override via `QSTASH_DEV_PORT`), and reuses an already-running server on that port instead of spawning a duplicate.
+- `serve()` builds a dev-mode `Receiver` internally, so signature verification works against the dev server's deterministic signing keys with no real credentials.
+- It's a no-op in production (`NODE_ENV=production`), during `next build`, and in browser/edge runtimes.
+
+**Next.js edge routes:** the edge runtime cannot spawn child processes. If your workflow route runs on the Edge Runtime, call `registerQStashDev()` from `instrumentation.ts` so the binary starts at Next.js boot:
+
+```ts
+// instrumentation.ts
+import { registerQStashDev } from "@upstash/qstash/nextjs";
+
+export function register() {
+  registerQStashDev();
+}
+```
+
+See the [QStash Local Development skill](../../upstash-qstash-js/fundamentals/local-development.md) for the full reference, including pitfalls and CLI options.
+
+---
+
+## 1. Start the QStash Local Development Server
+
+If you'd rather manage the dev server yourself instead of using the automatic flow above, run the CLI manually.
+
+Use the QStash CLI:
+
+```
+npx @upstash/qstash-cli dev
+```
+
+The CLI prints:
+
+- QSTASH_TOKEN
+- QSTASH_CURRENT_SIGNING_KEY
+- QSTASH_NEXT_SIGNING_KEY
+- Local server URL (default: `http://127.0.0.1:8080`)
+
+Set these values in your `.env` file so your workflow client uses the local environment.
+
+---
+
+## 2. Set Local Environment Variables
+
+Use the environment values printed by the CLI:
+
+```
+QSTASH_URL="http://127.0.0.1:8080"
+QSTASH_TOKEN="<token-from-cli>"
+QSTASH_CURRENT_SIGNING_KEY="<cur-key>"
+QSTASH_NEXT_SIGNING_KEY="<next-key>"
+```
+
+These ensure all workflow requests are routed locally.
+
+---
+
+## 3. Trigger Workflows Using Local URLs
+
+A common pattern is determining the base URL dynamically based on environment variables. Below is an example that demonstrates:
+
+- Local development (localhost)
+- Production deployments (auto-detected env)
+
+```ts
+import { Client } from "@upstash/workflow";
+
+const client = Client();
+
+// In production, Vercel sets VERCEL_URL. Otherwise use localhost.
+const BASE_URL = process.env.VERCEL_URL
+  ? `https://${process.env.VERCEL_URL}`
+  : `http://localhost:3000`;
+
+// Trigger a workflow with retries
+const { workflowRunId } = await client.trigger({
+  url: `${BASE_URL}/api/workflow`, // Local or production
+  retries: 3, // Optional retry logic
+});
+
+console.log("Workflow run:", workflowRunId);
+```
+
+**Common mistakes:**
+
+- Forgetting to include the full URL including `http://` or `https://`.
+- Using a production URL while the local QStash server is running.
+- Missing environment variables.
+
+---
+
+## 4. Using ngrok (Optional)
+
+If your workflow must be reachable from the managed Upstash servers (not local), expose your local server publicly.
+
+### Install & authenticate
+
+```
+ngrok config add-authtoken <YOUR-AUTH-TOKEN>
+```
+
+### Start a tunnel
+
+```
+ngrok http 3000
+```
+
+ngrok outputs a public URL:
+
+```
+Forwarding  https://1234abcd.ngrok.io -> http://localhost:3000
+```
+
+Use this public URL instead of localhost:
+
+```ts
+const BASE_URL = "https://1234abcd.ngrok.io"; // Public tunnel
+
+await client.trigger({
+  url: `${BASE_URL}/api/workflow`,
+  retries: 3,
+});
+```
+
+**Pitfall:** Your ngrok port must match your dev server port, otherwise all workflow calls will fail.
+
+---
+
+## Summary
+
+- Start QStash locally using `qstash-cli dev`.
+- Copy generated environment variables into `.env`.
+- Use local URLs in TypeScript clients while developing.
+- Optionally expose your server with ngrok if you need remote access.
+
+This setup ensures fast workflow iteration without deploying your app.