memory-journal-mcp 7.7.1 → 8.0.0

package/skills/cloudflare/references/tail-workers/README.md

@@ -0,0 +1,92 @@
+# Cloudflare Tail Workers
+
+Specialized Workers that consume execution events from producer Workers for logging, debugging, analytics, and observability.
+
+## When to Use This Reference
+
+- Implementing observability/logging for Cloudflare Workers
+- Processing Worker execution events, logs, exceptions
+- Building custom analytics or error tracking
+- Configuring real-time event streaming
+- Working with tail handlers or tail consumers
+
+## Core Concepts
+
+### What Are Tail Workers?
+
+Tail Workers automatically process events from producer Workers (the Workers being monitored). They receive:
+
+- HTTP request/response info
+- Console logs (`console.log/error/warn/debug`)
+- Uncaught exceptions
+- Execution outcomes (`ok`, `exception`, `exceededCpu`, etc.)
+- Diagnostic channel events
+
+**Key characteristics:**
+
+- Invoked AFTER producer finishes executing
+- Capture entire request lifecycle including Service Bindings and Dynamic Dispatch sub-requests
+- Billed by CPU time, not request count
+- Available on Workers Paid and Enterprise tiers
+
+### Alternative: OpenTelemetry Export
+
+**Before using Tail Workers, consider OpenTelemetry:**
+
+For batch exports to observability tools (Sentry, Grafana, Honeycomb):
+
+- OTEL export sends logs/traces in batches (more efficient)
+- Built-in integrations with popular platforms
+- Lower overhead than Tail Workers
+- **Use Tail Workers only for custom real-time processing**
+
+## Decision Tree
+
+```
+Need observability for Workers?
+├─ Batch export to known tools (Sentry/Grafana/Honeycomb)?
+│  └─ Use OpenTelemetry export (not Tail Workers)
+├─ Custom real-time processing needed?
+│  ├─ Aggregated metrics?
+│  │  └─ Use Tail Worker + Analytics Engine
+│  ├─ Error tracking?
+│  │  └─ Use Tail Worker + external service
+│  ├─ Custom logging/debugging?
+│  │  └─ Use Tail Worker + KV/HTTP endpoint
+│  └─ Complex event processing?
+│     └─ Use Tail Worker + Durable Objects
+└─ Quick debugging?
+   └─ Use `wrangler tail` (different from Tail Workers)
+```
+
+## Reading Order
+
+1. **[configuration.md](configuration.md)** - Set up Tail Workers
+2. **[api.md](api.md)** - Handler signature, types, redaction
+3. **[patterns.md](patterns.md)** - Common use cases and integrations
+4. **[gotchas.md](gotchas.md)** - Pitfalls and debugging tips
+
+## Quick Example
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    // Process events from producer Worker
+    ctx.waitUntil(
+      fetch(env.LOG_ENDPOINT, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify(events),
+      })
+    )
+  },
+}
+```
+
+## Related Skills
+
+- **observability** - General Workers observability patterns, OTEL export
+- **analytics-engine** - Aggregated metrics storage for tail event data
+- **durable-objects** - Stateful event processing, batching tail events
+- **logpush** - Alternative for batch log export (non-real-time)
+- **workers-for-platforms** - Dynamic dispatch with tail consumers

package/skills/cloudflare/references/tail-workers/api.md

@@ -0,0 +1,203 @@
+# Tail Workers API Reference
+
+## Handler Signature
+
+```typescript
+export default {
+  async tail(events: TraceItem[], env: Env, ctx: ExecutionContext): Promise<void> {
+    // Process events
+  },
+} satisfies ExportedHandler<Env>
+```
+
+**Parameters:**
+
+- `events`: Array of `TraceItem` objects (one per producer invocation)
+- `env`: Bindings (KV, D1, R2, env vars, etc.)
+- `ctx`: Context with `waitUntil()` for async work
+
+**CRITICAL:** Tail handlers don't return values. Use `ctx.waitUntil()` for async operations.
+
+## TraceItem Type
+
+```typescript
+interface TraceItem {
+  scriptName: string // Producer Worker name
+  eventTimestamp: number // Epoch milliseconds
+  outcome:
+    | 'ok'
+    | 'exception'
+    | 'exceededCpu'
+    | 'exceededMemory'
+    | 'canceled'
+    | 'scriptNotFound'
+    | 'responseStreamDisconnected'
+    | 'unknown'
+
+  event?: {
+    request?: {
+      url: string // Redacted by default
+      method: string
+      headers: Record<string, string> // Sensitive headers redacted
+      cf?: IncomingRequestCfProperties
+      getUnredacted(): TraceRequest // Bypass redaction (use carefully)
+    }
+    response?: {
+      status: number
+    }
+  }
+
+  logs: Array<{
+    timestamp: number // Epoch milliseconds
+    level: 'debug' | 'info' | 'log' | 'warn' | 'error'
+    message: unknown[] // Args passed to console function
+  }>
+
+  exceptions: Array<{
+    timestamp: number // Epoch milliseconds
+    name: string // Error type (Error, TypeError, etc.)
+    message: string // Error description
+  }>
+
+  diagnosticsChannelEvents: Array<{
+    channel: string
+    message: unknown
+    timestamp: number // Epoch milliseconds
+  }>
+}
+```
+
+**Note:** Official SDK uses `TraceItem`, not `TailItem`. Use `@cloudflare/workers-types` for accurate types.
+
+## Timestamp Handling
+
+All timestamps are **epoch milliseconds**, not seconds:
+
+```typescript
+// ✅ CORRECT - use directly with Date
+const date = new Date(event.eventTimestamp)
+
+// ❌ WRONG - don't multiply by 1000
+const date = new Date(event.eventTimestamp * 1000)
+```
+
+## Automatic Redaction
+
+By default, sensitive data is redacted from `TraceRequest`:
+
+### Header Redaction
+
+Headers containing these substrings (case-insensitive):
+
+- `auth`, `key`, `secret`, `token`, `jwt`
+- `cookie`, `set-cookie`
+
+Redacted values show as `"REDACTED"`.
+
+### URL Redaction
+
+- **Hex IDs:** 32+ hex digits → `"REDACTED"`
+- **Base-64 IDs:** 21+ chars with 2+ upper, 2+ lower, 2+ digits → `"REDACTED"`
+
+## Bypassing Redaction
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    for (const event of events) {
+      // ⚠️ Use with extreme caution
+      const unredacted = event.event?.request?.getUnredacted()
+      // unredacted.url and unredacted.headers contain raw values
+    }
+  },
+}
+```
+
+**Best practices:**
+
+- Only call `getUnredacted()` when absolutely necessary
+- Never log unredacted sensitive data
+- Implement additional filtering before external transmission
+- Use environment variables for API keys, never hardcode
+
+## Type-Safe Handler
+
+```typescript
+interface Env {
+  LOGS_KV: KVNamespace
+  ANALYTICS: AnalyticsEngineDataset
+  LOG_ENDPOINT: string
+  API_TOKEN: string
+}
+
+export default {
+  async tail(events: TraceItem[], env: Env, ctx: ExecutionContext): Promise<void> {
+    const payload = events.map((event) => ({
+      script: event.scriptName,
+      timestamp: event.eventTimestamp,
+      outcome: event.outcome,
+      url: event.event?.request?.url,
+      status: event.event?.response?.status,
+    }))
+
+    ctx.waitUntil(
+      fetch(env.LOG_ENDPOINT, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify(payload),
+      })
+    )
+  },
+} satisfies ExportedHandler<Env>
+```
+
+## Outcome vs HTTP Status
+
+**IMPORTANT:** `outcome` is script execution status, NOT HTTP status.
+
+- Worker returns 500 → `outcome='ok'` if script completed successfully
+- Uncaught exception → `outcome='exception'` regardless of HTTP status
+- CPU limit exceeded → `outcome='exceededCpu'`
+
+```typescript
+// ✅ Check outcome for script execution status
+if (event.outcome === 'exception') {
+  // Script threw uncaught exception
+}
+
+// ✅ Check HTTP status separately
+if (event.event?.response?.status === 500) {
+  // HTTP 500 returned (script may have handled error)
+}
+```
+
+## Serialization Considerations
+
+`log.message` is `unknown[]` and may contain non-serializable objects:
+
+```typescript
+// ❌ May fail with circular references or BigInt
+JSON.stringify(events)
+
+// ✅ Safe serialization
+const safePayload = events.map((event) => ({
+  ...event,
+  logs: event.logs.map((log) => ({
+    ...log,
+    message: log.message.map((m) => {
+      try {
+        return JSON.parse(JSON.stringify(m))
+      } catch {
+        return String(m)
+      }
+    }),
+  })),
+}))
+```
+
+**Common serialization issues:**
+
+- Circular references in logged objects
+- `BigInt` values (not JSON-serializable)
+- Functions or symbols in console.log arguments
+- Large objects exceeding body size limits

package/skills/cloudflare/references/tail-workers/configuration.md

@@ -0,0 +1,178 @@
+# Tail Workers Configuration
+
+## Setup Steps
+
+### 1. Create Tail Worker
+
+Create a Worker with a `tail()` handler:
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    // Process events from producer Worker
+    ctx.waitUntil(
+      fetch(env.LOG_ENDPOINT, {
+        method: 'POST',
+        body: JSON.stringify(events),
+      })
+    )
+  },
+}
+```
+
+### 2. Configure Producer Worker
+
+In producer's `wrangler.jsonc`:
+
+```jsonc
+{
+  "name": "my-producer-worker",
+  "tail_consumers": [
+    {
+      "service": "my-tail-worker",
+    },
+  ],
+}
+```
+
+### 3. Deploy Both Workers
+
+```bash
+# Deploy Tail Worker first
+cd tail-worker
+wrangler deploy
+
+# Then deploy producer Worker
+cd ../producer-worker
+wrangler deploy
+```
+
+## Wrangler Configuration
+
+### Single Tail Consumer
+
+```jsonc
+{
+  "name": "producer-worker",
+  "tail_consumers": [
+    {
+      "service": "logging-tail-worker",
+    },
+  ],
+}
+```
+
+### Multiple Tail Consumers
+
+```jsonc
+{
+  "name": "producer-worker",
+  "tail_consumers": [
+    {
+      "service": "logging-tail-worker",
+    },
+    {
+      "service": "metrics-tail-worker",
+    },
+  ],
+}
+```
+
+**Note:** Each consumer receives ALL events independently.
+
+### Remove Tail Consumer
+
+```jsonc
+{
+  "tail_consumers": [],
+}
+```
+
+Then redeploy producer Worker.
+
+## Environment Variables
+
+Tail Workers use same binding syntax as regular Workers:
+
+```jsonc
+{
+  "name": "my-tail-worker",
+  "vars": {
+    "LOG_ENDPOINT": "https://logs.example.com/ingest",
+  },
+  "kv_namespaces": [
+    {
+      "binding": "LOGS_KV",
+      "id": "abc123...",
+    },
+  ],
+}
+```
+
+## Testing & Development
+
+### Local Testing
+
+**Tail Workers cannot be fully tested with `wrangler dev`.** Deploy to staging environment for testing.
+
+### Testing Strategy
+
+1. Deploy producer Worker to staging
+2. Deploy Tail Worker to staging
+3. Configure `tail_consumers` in producer
+4. Trigger producer Worker requests
+5. Verify Tail Worker receives events (check destination logs/storage)
+
+### Wrangler Tail Command
+
+```bash
+# Stream logs to terminal (NOT Tail Workers)
+wrangler tail my-producer-worker
+```
+
+**This is different from Tail Workers:**
+
+- `wrangler tail` streams logs to your terminal
+- Tail Workers are Workers that process events programmatically
+
+## Deployment Checklist
+
+- [ ] Tail Worker has `tail()` handler
+- [ ] Tail Worker deployed before producer
+- [ ] Producer's `wrangler.jsonc` has correct `tail_consumers`
+- [ ] Environment variables configured
+- [ ] Tested with staging environment
+- [ ] Monitoring configured for Tail Worker itself
+
+## Limits
+
+| Limit                           | Value                           | Notes                                        |
+| ------------------------------- | ------------------------------- | -------------------------------------------- |
+| Max tail consumers per producer | 10                              | Each receives all events independently       |
+| Events batch size               | Up to 100 events per invocation | Larger batches split across invocations      |
+| Tail Worker CPU time            | Same as regular Workers         | 10ms (free), 30ms (paid), 50ms (paid bundle) |
+| Pricing tier                    | Workers Paid or Enterprise      | Not available on free plan                   |
+| Request body size               | 100 MB max                      | When sending to external endpoints           |
+| Event retention                 | None                            | Events not retried if tail handler fails     |
+
+## Workers for Platforms
+
+For dynamic dispatch Workers, both dispatch and user Worker events sent to tail consumer:
+
+```jsonc
+{
+  "name": "dispatch-worker",
+  "tail_consumers": [
+    {
+      "service": "platform-tail-worker",
+    },
+  ],
+}
+```
+
+Tail Worker receives TWO `TraceItem` elements per request:
+
+1. Dynamic dispatch Worker event
+2. User Worker event
+
+See [patterns.md](patterns.md) for handling.

package/skills/cloudflare/references/tail-workers/gotchas.md

@@ -0,0 +1,206 @@
+# Tail Workers Gotchas & Debugging
+
+## Critical Pitfalls
+
+### 1. Not Using `ctx.waitUntil()`
+
+**Problem:** Async work doesn't complete or tail Worker times out  
+**Cause:** Handlers exit immediately; awaiting blocks processing  
+**Solution:**
+
+```typescript
+// ❌ WRONG - fire and forget
+export default {
+  async tail(events) {
+    fetch(endpoint, { body: JSON.stringify(events) })
+  },
+}
+
+// ❌ WRONG - blocking await
+export default {
+  async tail(events, env, ctx) {
+    await fetch(endpoint, { body: JSON.stringify(events) })
+  },
+}
+
+// ✅ CORRECT
+export default {
+  async tail(events, env, ctx) {
+    ctx.waitUntil(
+      (async () => {
+        await fetch(endpoint, { body: JSON.stringify(events) })
+        await processMore()
+      })()
+    )
+  },
+}
+```
+
+### 2. Missing `tail()` Handler
+
+**Problem:** Producer deployment fails  
+**Cause:** Worker in `tail_consumers` doesn't export `tail()` handler  
+**Solution:** Ensure `export default { async tail(events, env, ctx) { ... } }`
+
+### 3. Outcome vs HTTP Status
+
+**Problem:** Filtering by wrong status  
+**Cause:** `outcome` is script execution status, not HTTP status
+
+```typescript
+// ❌ WRONG
+if (event.outcome === 500) {
+  /* never matches */
+}
+
+// ✅ CORRECT
+if (event.outcome === 'exception') {
+  /* script threw */
+}
+if (event.event?.response?.status === 500) {
+  /* HTTP 500 */
+}
+```
+
+### 4. Timestamp Units
+
+**Problem:** Dates off by 1000x  
+**Cause:** Timestamps are epoch milliseconds, not seconds
+
+```typescript
+// ❌ WRONG: const date = new Date(event.eventTimestamp * 1000);
+// ✅ CORRECT: const date = new Date(event.eventTimestamp);
+```
+
+### 5. Type Name Mismatch
+
+**Problem:** Using `TailItem` type  
+**Cause:** Old docs used `TailItem`, SDK uses `TraceItem`
+
+```typescript
+import type { TraceItem } from '@cloudflare/workers-types'
+export default {
+  async tail(events: TraceItem[], env, ctx) {
+    /* ... */
+  },
+}
+```
+
+### 6. Excessive Logging Volume
+
+**Problem:** Unexpected high costs  
+**Cause:** Invoked on EVERY producer request  
+**Solution:** Sample events
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    if (Math.random() > 0.1) return // 10% sample
+    ctx.waitUntil(sendToEndpoint(events))
+  },
+}
+```
+
+### 7. Serialization Issues
+
+**Problem:** `JSON.stringify()` fails  
+**Cause:** `log.message` is `unknown[]` with non-serializable values  
+**Solution:**
+
+```typescript
+const safePayload = events.map((e) => ({
+  ...e,
+  logs: e.logs.map((log) => ({
+    ...log,
+    message: log.message.map((m) => {
+      try {
+        return JSON.parse(JSON.stringify(m))
+      } catch {
+        return String(m)
+      }
+    }),
+  })),
+}))
+```
+
+### 8. Missing Error Handling
+
+**Problem:** Tail Worker silently fails  
+**Cause:** No try/catch  
+**Solution:**
+
+```typescript
+ctx.waitUntil(
+  (async () => {
+    try {
+      await fetch(env.ENDPOINT, { body: JSON.stringify(events) })
+    } catch (error) {
+      console.error('Tail error:', error)
+      await env.FALLBACK_KV.put(`failed:${Date.now()}`, JSON.stringify(events))
+    }
+  })()
+)
+```
+
+### 9. Deployment Order
+
+**Problem:** Producer deployment fails  
+**Cause:** Tail consumer not deployed yet  
+**Solution:** Deploy tail consumer FIRST
+
+```bash
+cd tail-worker && wrangler deploy
+cd ../producer && wrangler deploy
+```
+
+### 10. No Event Retry
+
+**Problem:** Events lost when handler fails  
+**Cause:** Failed invocations NOT retried  
+**Solution:** Implement fallback storage (see #8)
+
+## Debugging
+
+**View logs:** `wrangler tail my-tail-worker`
+
+**Incremental testing:**
+
+1. Verify receipt: `console.log('Events:', events.length)`
+2. Inspect structure: `console.log(JSON.stringify(events[0], null, 2))`
+3. Add external call with `ctx.waitUntil()`
+
+**Monitor dashboard:** Check invocation count (matches producer?), error rate, CPU time
+
+## Testing
+
+Add test endpoint to producer:
+
+```typescript
+export default {
+  async fetch(request) {
+    if (request.url.includes('/test')) {
+      console.log('Test log')
+      throw new Error('Test error')
+    }
+    return new Response('OK')
+  },
+}
+```
+
+Trigger: `curl https://producer.example.workers.dev/test`
+
+## Common Errors
+
+| Error                         | Cause            | Solution                 |
+| ----------------------------- | ---------------- | ------------------------ |
+| "Tail consumer not found"     | Not deployed     | Deploy tail Worker first |
+| "No tail handler"             | Missing `tail()` | Add to default export    |
+| "waitUntil is not a function" | Missing `ctx`    | Add `ctx` parameter      |
+| Timeout                       | Blocking await   | Use `ctx.waitUntil()`    |
+
+## Performance Notes
+
+- Max 100 events per invocation
+- Each consumer receives all events independently
+- CPU limits same as regular Workers
+- For high volume, use Durable Objects batching