flingit 0.0.63 → 0.0.66

package/node_modules/flingflow/SPEC.md

@@ -0,0 +1,550 @@
+# Durable Execution Engine on Cloudflare
+
+A design for a minimal viable durable execution engine built on Cloudflare Workers, D1, and Workers for Platforms. The goal is to support a subset of Temporal-like functionality, targeting only TypeScript, running entirely within customer projects.
+
+## Goals
+
+- **Durable execution**: Workflows survive worker restarts, timeouts, and failures
+- **Per-project isolation**: All state lives in the project's own D1 database
+- **No platform dependency**: The orchestration runs within the project itself, not in shared infrastructure
+- **Simplicity**: Minimal concepts, minimal schema, easy to reason about
+
+## Alternatives Considered
+
+### Temporal-Style Replay
+
+Temporal maintains an ordered event log and replays the entire workflow history to reconstruct state. The workflow function runs from the beginning, and the SDK intercepts each activity call, matching it against the event history by position.
+
+```typescript
+async function onboard(ctx, input) {
+  const user = await ctx.activity("fetch-user", () => getUser(input.userId));
+  const plan = await ctx.activity("create-plan", () => createPlan(user.id));
+  return { planId: plan.id };
+}
+```
+
+On replay, if `fetch-user` already completed, the SDK returns the stored result without executing. The workflow function stays alive in memory, blocked on the next await.
+
+**Why we rejected it:**
+
+- Workers have CPU time limits (30s on paid plans). Long histories become expensive to replay.
+- Workers can't hold execution state between activity completions. There's no long-lived process—each invocation is ephemeral.
+- Requires strict determinism. Any non-deterministic code (`Math.random`, `Date.now`, etc.) causes replay to diverge.
+
+### Re-Execute with Memoization
+
+This is what Inngest and Cloudflare Workflows do. The workflow function runs from the top on every invocation. Each `step.run()` call checks a key-value store: if the step already completed, return the result; otherwise, execute it, persist the result, and stop.
+
+```typescript
+async function onboard(step, input) {
+  const user = await step.run("fetch-user", () => getUser(input.userId));
+  const plan = await step.run("create-plan", () => createPlan(user.id));
+  return { planId: plan.id };
+}
+```
+
+**Pros:**
+- No long-lived process needed
+- Works with arbitrary control flow
+- Simpler than full replay
+
+**Cons:**
+- Still re-executes all code between steps on every invocation
+- Code between steps must still be deterministic (affects control flow)
+- Step ID collisions if you're not careful with naming
+
+### Compile Workflows to State Machines
+
+Transform the workflow code at build time into an explicit state machine. Each `await step.run()` becomes a case in a switch statement. Local variables that cross step boundaries are lifted into serializable state.
+
+```typescript
+// Compiled output
+function onboard_machine(state, input) {
+  switch (state.phase) {
+    case 0:
+      return { invoke: "fetch-user", fn: () => getUser(input.userId), next: 1 };
+    case 1:
+      const user = state.results["fetch-user"];
+      return { invoke: "create-plan", fn: () => createPlan(user.id), next: 2 };
+    // ...
+  }
+}
+```
+
+**Pros:**
+- O(1) per step—jump directly to the current phase, no replay
+- No determinism constraints
+
+**Cons:**
+- Requires a source-to-source compiler (substantial engineering)
+- Loops and try/catch become complex
+- Non-serializable locals (closures, class instances) fail at runtime
+
+We decided this could be a future optimization but is too complex for v1.
+
+## Chosen Design: Explicit Continuations + Scratchpad
+
+Instead of hiding the state machine inside a framework, make the user write it explicitly. The workflow is a function that takes the current step and returns a continuation describing what to do next.
+
+### The Model
+
+```typescript
+type Continuation =
+  | { step: string }            // next step to run
+  | { done: true; result: any } // workflow complete
+
+type Ctx = {
+  step: string;                           // current step name
+  get: (key: string) => Promise<any>;     // read from scratchpad
+  set: (key: string, value: any) => void; // write to scratchpad (buffered)
+};
+
+type WorkflowFn = (ctx: Ctx) => Promise<Continuation>;
+```
+
+### Example Workflow
+
+```typescript
+async function onboardingWorkflow(ctx: Ctx): Promise<Continuation> {
+  switch (ctx.step) {
+    case "start": {
+      const input = await ctx.get("input");
+      const user = await getUser(input.userId);
+      ctx.set("user", user);
+      return { step: "create-plan" };
+    }
+    case "create-plan": {
+      const user = await ctx.get("user");
+      const plan = await createPlan(user.id);
+      ctx.set("plan", plan);
+      return { step: "send-email" };
+    }
+    case "send-email": {
+      const user = await ctx.get("user");
+      const plan = await ctx.get("plan");
+      await sendWelcomeEmail(user.email, plan);
+      return { done: true, result: { planId: plan.id } };
+    }
+  }
+}
+```
+
+### Why This Design
+
+**Engine simplicity**: The engine just calls the function, persists what it returns, and calls it again for the next step. No parsing, no memoization, no replay.
+
+**No determinism constraints**: Each step is a fresh invocation. You can call `Math.random()`, `Date.now()`, fetch live data—whatever you want.
+
+**State is explicit**: The scratchpad is a key-value store. Each step declares what it reads (`ctx.get`) and writes (`ctx.set`). No magic.
+
+**Debuggable**: Query the database and see exactly where every workflow is and what it knows.
+
+### The Scratchpad
+
+The scratchpad is a per-workflow key-value store.
+
+- `ctx.set(key, value)` buffers a write in memory. It's persisted when the step completes successfully.
+- `ctx.get(key)` is async because it may need to fetch from the database.
+
+If a step fails or times out, buffered writes are discarded. The step is retried with the scratchpad state from before it started.
+
+## Event Sourcing
+
+Rather than storing mutable "current state," we store an append-only event log. The current state is derived from the last event.
+
+### Event Types
+
+| Event Type | Payload |
+|------------|---------|
+| `workflow_created` | `{ writes }` |
+| `step_started` | `{ step, attempt }` |
+| `step_completed` | `{ writes, next_step? }` |
+| `step_failed` | `{ reason, retryable }` |
+| `workflow_completed` | `{ result }` |
+| `workflow_failed` | `{ reason, cause }` |
+
+- `writes` is a plain JSON object: `{ user: {...}, plan: {...} }`
+- `next_step` is omitted when the workflow is done (next event will be `workflow_completed`)
+- `retryable` indicates whether the failure can be retried
+
+### State Machine
+
+The last event determines the workflow's state and what can happen next.
+
+```mermaid
+stateDiagram-v2
+    [*] --> workflow_created
+    workflow_created --> step_started
+    step_started --> step_completed
+    step_started --> step_failed
+    step_completed --> step_started : has next_step
+    step_completed --> workflow_completed : no next_step
+    step_failed --> step_started : retryable, can retry
+    step_failed --> workflow_failed : not retryable or max attempts
+    workflow_completed --> [*]
+    workflow_failed --> [*]
+```
+
+### Valid Transitions
+
+| Last Event | Can Be Followed By |
+|------------|---------------------|
+| `workflow_created` | `step_started` |
+| `step_started` | `step_completed`, `step_failed` |
+| `step_completed` (has `next_step`) | `step_started` |
+| `step_completed` (no `next_step`) | `workflow_completed` |
+| `step_failed` (retryable, under max) | `step_started` |
+| `step_failed` (not retryable or max) | `workflow_failed` |
+| `workflow_completed` | *(terminal)* |
+| `workflow_failed` | *(terminal)* |
+
+## Schema
+
+```sql
+CREATE TABLE workflow_events (
+  workflow_id TEXT NOT NULL,
+  seq INTEGER NOT NULL,
+  event_type TEXT NOT NULL,
+  payload JSON NOT NULL,
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+  PRIMARY KEY (workflow_id, seq)
+);
+
+CREATE INDEX idx_workflow_events_last 
+ON workflow_events(workflow_id, seq DESC);
+```
+
+The composite primary key `(workflow_id, seq)` enforces uniqueness. Two concurrent writers trying to insert `seq = 5` for the same workflow: one wins, one gets a constraint violation.
+
+## Coordination Without Locks
+
+D1 doesn't support interactive transactions—you can't SELECT then UPDATE atomically. We use optimistic concurrency via the unique constraint on `(workflow_id, seq)`.
+
+### Claiming a Step
+
+To start a step:
+
+1. Query the last event for the workflow
+2. Validate that `step_started` is a legal transition from that event
+3. Insert `step_started` with `seq = last_seq + 1`
+
+If the insert fails (unique constraint violation), someone else claimed it. No harm done—the loser just bails.
+
+```typescript
+async function claimStep(db: D1Database, workflowId: string): Promise<ClaimResult | null> {
+  // 1. Get last event
+  const last = await db.prepare(`
+    SELECT * FROM workflow_events
+    WHERE workflow_id = ?
+    ORDER BY seq DESC
+    LIMIT 1
+  `).bind(workflowId).first();
+
+  if (!last) return null;
+
+  // 2. Validate transition
+  const canStart = 
+    last.event_type === "workflow_created" ||
+    (last.event_type === "step_completed" && last.payload.next_step) ||
+    (last.event_type === "step_failed" && last.payload.retryable);
+
+  if (!canStart) return null;
+
+  // 3. Determine step name and attempt
+  const stepName = last.event_type === "workflow_created" 
+    ? "start"
+    : last.event_type === "step_completed"
+      ? last.payload.next_step
+      : last.payload.step; // retry same step
+
+  const attempt = last.event_type === "step_failed" 
+    ? last.payload.attempt + 1 
+    : 1;
+
+  // 4. Try to insert (may fail if someone else got there)
+  try {
+    await db.prepare(`
+      INSERT INTO workflow_events (workflow_id, seq, event_type, payload)
+      VALUES (?, ?, 'step_started', ?)
+    `).bind(
+      workflowId,
+      last.seq + 1,
+      JSON.stringify({ step: stepName, attempt })
+    ).run();
+
+    return { workflowId, step: stepName, attempt, seq: last.seq + 1 };
+  } catch (err) {
+    // Unique constraint violation = someone else claimed it
+    return null;
+  }
+}
+```
+
+## System Components
+
+### Trigger Endpoint
+
+Creates a new workflow:
+
+```typescript
+// POST /workflows
+async function createWorkflow(db: D1Database, name: string, writes: Record<string, unknown>): Promise<string> {
+  const workflowId = generateId();
+
+  await db.prepare(`
+    INSERT INTO workflow_events (workflow_id, seq, event_type, payload)
+    VALUES (?, 0, 'workflow_created', ?)
+  `).bind(workflowId, JSON.stringify({ writes })).run();
+
+  // Kick off first step
+  await fetch(`${INTERNAL_URL}/internal/workflow/step`, {
+    method: "POST",
+    body: JSON.stringify({ workflowId }),
+  });
+
+  return workflowId;
+}
+```
+
+### Step Endpoint
+
+Executes a single step:
+
+```typescript
+// POST /internal/workflow/step
+async function handleStep(request: Request, env: Env) {
+  const { workflowId } = await request.json();
+
+  // 1. Try to claim
+  const claim = await claimStep(env.DB, workflowId);
+  if (!claim) {
+    return new Response("not claimable", { status: 409 });
+  }
+
+  // 2. Build context
+  const ctx = await buildContext(env.DB, workflowId, claim);
+
+  // 3. Execute workflow function
+  const workflowFn = getWorkflowFunction(claim.workflowName);
+  
+  try {
+    const continuation = await workflowFn(ctx);
+
+    // 4. Commit
+    await commitStep(env.DB, workflowId, claim.seq, ctx.pendingWrites, continuation);
+
+    // 5. Continue if there's a next step
+    if (!continuation.done) {
+      env.ctx.waitUntil(
+        fetch(request.url, {
+          method: "POST",
+          body: JSON.stringify({ workflowId }),
+        })
+      );
+    }
+
+    return new Response("ok");
+
+  } catch (err) {
+    // Step failed
+    await insertEvent(env.DB, workflowId, claim.seq + 1, "step_failed", {
+      step: claim.step,
+      attempt: claim.attempt,
+      reason: err.message,
+      retryable: true,
+    });
+
+    return new Response("step failed", { status: 500 });
+  }
+}
+```
+
+### Committing a Step
+
+```typescript
+async function commitStep(
+  db: D1Database,
+  workflowId: string,
+  currentSeq: number,
+  writes: Record<string, any>,
+  continuation: Continuation
+) {
+  if (continuation.done) {
+    // Insert step_completed then workflow_completed
+    await db.batch([
+      db.prepare(`
+        INSERT INTO workflow_events (workflow_id, seq, event_type, payload)
+        VALUES (?, ?, 'step_completed', ?)
+      `).bind(workflowId, currentSeq + 1, JSON.stringify({ writes })),
+      
+      db.prepare(`
+        INSERT INTO workflow_events (workflow_id, seq, event_type, payload)
+        VALUES (?, ?, 'workflow_completed', ?)
+      `).bind(workflowId, currentSeq + 2, JSON.stringify({ result: continuation.result })),
+    ]);
+  } else {
+    // Insert step_completed with next_step
+    await db.prepare(`
+      INSERT INTO workflow_events (workflow_id, seq, event_type, payload)
+      VALUES (?, ?, 'step_completed', ?)
+    `).bind(
+      workflowId, 
+      currentSeq + 1, 
+      JSON.stringify({ writes, next_step: continuation.step })
+    ).run();
+  }
+}
+```
+
+### Cron Job
+
+Runs every 5–10 seconds. Handles:
+
+1. **Timeouts**: Steps that started but never completed
+2. **Stuck workflows**: Workflows that should be running but aren't
+
+```typescript
+async function handleCron(env: Env) {
+  // Get last event for all non-terminal workflows
+  const workflows = await env.DB.prepare(`
+    SELECT 
+      e.workflow_id,
+      e.seq,
+      e.event_type,
+      e.payload,
+      e.created_at
+    FROM workflow_events e
+    INNER JOIN (
+      SELECT workflow_id, MAX(seq) as max_seq
+      FROM workflow_events
+      GROUP BY workflow_id
+    ) latest ON e.workflow_id = latest.workflow_id AND e.seq = latest.max_seq
+    WHERE e.event_type NOT IN ('workflow_completed', 'workflow_failed')
+  `).all();
+
+  for (const wf of workflows.results) {
+    const payload = JSON.parse(wf.payload);
+
+    if (wf.event_type === "step_started") {
+      // Check for timeout
+      const startedAt = new Date(wf.created_at);
+      const timedOut = Date.now() - startedAt.getTime() > STEP_TIMEOUT_MS;
+
+      if (timedOut) {
+        // Insert step_failed
+        await env.DB.prepare(`
+          INSERT INTO workflow_events (workflow_id, seq, event_type, payload)
+          VALUES (?, ?, 'step_failed', ?)
+        `).bind(
+          wf.workflow_id,
+          wf.seq + 1,
+          JSON.stringify({
+            step: payload.step,
+            attempt: payload.attempt,
+            reason: "timeout",
+            retryable: true,
+          })
+        ).run();
+
+        // Kick the workflow
+        env.ctx.waitUntil(triggerStep(wf.workflow_id));
+      }
+    } else {
+      // workflow_created, step_completed, step_failed (retryable)
+      // These should all have a step running, but don't
+      // Kick the workflow to make progress
+      env.ctx.waitUntil(triggerStep(wf.workflow_id));
+    }
+  }
+}
+
+async function triggerStep(workflowId: string) {
+  await fetch(`${INTERNAL_URL}/internal/workflow/step`, {
+    method: "POST",
+    body: JSON.stringify({ workflowId }),
+  });
+}
+```
+
+## Reconstructing the Scratchpad
+
+To get a key's current value, scan `step_completed` events from newest to oldest:
+
+```typescript
+async function getScratchpadValue(db: D1Database, workflowId: string, key: string): Promise<any> {
+  const events = await db.prepare(`
+    SELECT payload FROM workflow_events
+    WHERE workflow_id = ?
+      AND event_type = 'step_completed'
+    ORDER BY seq DESC
+  `).bind(workflowId).all();
+
+  for (const event of events.results) {
+    const payload = JSON.parse(event.payload);
+    if (payload.writes && key in payload.writes) {
+      return payload.writes[key];
+    }
+  }
+
+  return undefined;
+}
+```
+
+For efficiency, the step endpoint can cache reads during execution so multiple `ctx.get()` calls for the same key don't hit the database repeatedly.
+
+## Event Log Example
+
+A successful workflow:
+
+| seq | event_type | payload |
+|-----|------------|---------|
+| 0 | `workflow_created` | `{ writes: { userId: 123 } }` |
+| 1 | `step_started` | `{ step: "start", attempt: 1 }` |
+| 2 | `step_completed` | `{ writes: { user: {...} }, next_step: "create-plan" }` |
+| 3 | `step_started` | `{ step: "create-plan", attempt: 1 }` |
+| 4 | `step_completed` | `{ writes: { plan: {...} }, next_step: "send-email" }` |
+| 5 | `step_started` | `{ step: "send-email", attempt: 1 }` |
+| 6 | `step_completed` | `{ writes: {} }` |
+| 7 | `workflow_completed` | `{ result: { planId: "abc" } }` |
+
+A workflow with a retry:
+
+| seq | event_type | payload |
+|-----|------------|---------|
+| 0 | `workflow_created` | `{ writes: { userId: 123 } }` |
+| 1 | `step_started` | `{ step: "start", attempt: 1 }` |
+| 2 | `step_failed` | `{ step: "start", attempt: 1, reason: "connection timeout", retryable: true }` |
+| 3 | `step_started` | `{ step: "start", attempt: 2 }` |
+| 4 | `step_completed` | `{ writes: { user: {...} }, next_step: "create-plan" }` |
+| ... | | |
+
+## Future Extensions
+
+### Sleep / Delayed Execution
+
+Add `run_at` to `step_completed`:
+
+```typescript
+return { step: "remind-user", runAt: new Date("2025-03-10") };
+```
+
+The cron job would check for `step_completed` events where `run_at <= now` and kick those workflows.
+
+### Signals / External Triggers
+
+Add a `wait_for` field to `step_completed` and a new event type `signal_received`. The workflow pauses until an external HTTP call delivers the signal.
+
+### Child Workflows
+
+A step can spawn a child workflow and wait for its completion. The parent's `step_completed` would reference the child workflow ID, and the step wouldn't complete until the child reaches `workflow_completed`.
+
+## Summary
+
+This design achieves durable execution with:
+
+- **One table**: `workflow_events`
+- **Append-only writes**: Events are immutable
+- **Optimistic concurrency**: The unique constraint on `(workflow_id, seq)` is the lock
+- **No external dependencies**: Everything runs in the project's Workers and D1
+- **Self-healing**: The cron job catches timeouts and stuck workflows
+- **Full audit trail**: The event log is the history