@donkeylabs/server 0.4.8 → 0.5.1

package/docs/external-jobs.md

@@ -258,9 +258,17 @@ router.route("subscribe-job").raw({
 
 ## Wrapper Libraries
 
-### Python Wrapper
+After installing `@donkeylabs/server`, copy the wrapper to your project:
+
+```bash
+# Python
+cp node_modules/@donkeylabs/server/examples/external-jobs/python/donkeylabs_job.py ./workers/
+
+# Shell
+cp node_modules/@donkeylabs/server/examples/external-jobs/shell/donkeylabs-job.sh ./workers/
+```
 
-Located at `examples/external-jobs/python/donkeylabs_job.py`:
+### Python Wrapper
 
 ```python
 from donkeylabs_job import DonkeylabsJob, run_job
@@ -331,19 +339,131 @@ job_complete '{"result": "success"}'
 
 ## Server Restart Resilience
 
-External jobs survive server restarts:
+External jobs automatically survive server restarts through built-in SQLite persistence.
+
+### Default Behavior (SQLite Persistence)
+
+Jobs are automatically persisted to `.donkeylabs/jobs.db` by default:
+
+```typescript
+import { AppServer } from "@donkeylabs/server";
+
+const server = new AppServer({
+  db: createDatabase(),
+  // Jobs automatically use SQLite persistence - no config needed!
+});
+
+server.getCore().jobs.registerExternal("process-video", {
+  command: "python",
+  args: ["-m", "video_processor"],
+});
+```
+
+### Configuration Options
+
+```typescript
+const server = new AppServer({
+  db: createDatabase(),
+  jobs: {
+    // SQLite is used by default (persist: true)
+    persist: true,                    // Set to false for in-memory only
+    dbPath: ".donkeylabs/jobs.db",    // Custom database path
+    external: {
+      socketDir: "/tmp/donkeylabs-jobs",
+    },
+  },
+});
+```
+
+### Custom Adapter
+
+For Postgres, MySQL, or other databases, provide your own adapter:
+
+```typescript
+import { AppServer, SqliteJobAdapter } from "@donkeylabs/server";
+import { MyPostgresJobAdapter } from "./adapters/postgres";
+
+const server = new AppServer({
+  db: createDatabase(),
+  jobs: {
+    adapter: new MyPostgresJobAdapter(db), // Custom adapter
+  },
+});
+```
+
+### What Gets Persisted
+
+The adapter must persist these fields for external jobs:
+
+| Field | Description |
+|-------|-------------|
+| `id` | Unique job ID |
+| `name` | Job name |
+| `data` | Job payload (JSON) |
+| `status` | pending, running, completed, failed |
+| `pid` | External process ID |
+| `socketPath` | Unix socket path |
+| `tcpPort` | TCP port (Windows) |
+| `lastHeartbeat` | Last heartbeat timestamp |
+| `processState` | spawning, running, orphaned |
+
+### How Reconnection Works
+
+1. **On Server Shutdown**: Job state is already persisted in the database
+2. **On Server Restart**:
+   - Server queries for jobs where `status = 'running'` and `external = true`
+   - Checks if the process is still alive (via PID)
+   - Checks if heartbeat hasn't expired
+   - **Reserves** the socket path/port to prevent new jobs from using it
+   - Recreates the socket server on the **same path/port**
+   - External process detects disconnection and retries connecting
+3. **Reconnection**: Once reconnected, the job resumes normal operation
+4. **Cleanup**: When the job completes, fails, or is killed, the reservation is released
+
+### Socket/Port Reservation
+
+The server prevents new jobs from accidentally using socket paths or TCP ports that are reserved for orphaned jobs awaiting reconnection:
+
+- When an orphaned job is detected on startup, its socket path/port is **reserved**
+- New jobs cannot use reserved paths/ports (an error is thrown if attempted)
+- Reservations are automatically released when:
+  - The job completes successfully
+  - The job fails
+  - The job is killed due to stale heartbeat
+  - The process is confirmed dead
+
+This ensures that running external processes can always reconnect to their original socket path/port even if the server restarts multiple times.
+
+### Python Wrapper Reconnection
+
+The Python wrapper automatically handles reconnection:
+
+```python
+# Default reconnection settings
+job = DonkeylabsJob(
+    job_id=job_id,
+    name=name,
+    data=data,
+    socket_path=socket_path,
+    heartbeat_interval=5.0,      # Heartbeat every 5 seconds
+    reconnect_interval=2.0,      # Retry every 2 seconds
+    max_reconnect_attempts=30,   # Try for up to 60 seconds
+)
+```
 
-1. **On Shutdown**: Job state (PID, socket path) is persisted in the database
-2. **On Startup**: Server checks for orphaned jobs:
-   - If process is still alive, attempts reconnection
-   - If process died, marks job as failed
-3. **Reconnection**: External process continues sending heartbeats; server picks them up
+When the connection is lost:
+1. Heartbeat/progress messages fail to send
+2. Background reconnection thread starts
+3. Retries connecting to the same socket path
+4. Once reconnected, sends "started" message to server
+5. Normal operation resumes
 
 ### Best Practices
 
-- External workers should handle reconnection gracefully
-- Use heartbeats to detect server restarts
-- Consider idempotent operations for potential re-execution
+- **Always use a persistent adapter in production**
+- External workers should be idempotent when possible
+- Set `heartbeatTimeout` appropriately (longer = more time to reconnect)
+- Consider longer `max_reconnect_attempts` for critical jobs
 
 ## Error Handling
 

package/docs/workflows.md

@@ -1,6 +1,6 @@
 # Workflows
 
-Workflows provide step function / state machine orchestration for complex multi-step processes. Built on top of the Jobs service, workflows support sequential tasks, parallel execution, conditional branching, retries, and real-time progress via SSE.
+Workflows provide step function / state machine orchestration for complex multi-step processes. Workflows support sequential tasks with inline handlers, parallel execution, conditional branching, retries, and real-time progress via SSE.
 
 ## Overview
 
@@ -17,11 +17,17 @@ Use workflows when you need to:
 
 ```typescript
 import { workflow } from "@donkeylabs/server";
+import { z } from "zod";
 
 const orderWorkflow = workflow("process-order")
+  // First task: inputSchema validates workflow input
   .task("validate", {
-    job: "validate-order",
-    input: (ctx) => ({ orderId: ctx.input.orderId }),
+    inputSchema: z.object({ orderId: z.string() }),
+    outputSchema: z.object({ valid: z.boolean(), inStock: z.boolean(), total: z.number() }),
+    handler: async (input, ctx) => {
+      const order = await ctx.plugins.orders.validate(input.orderId);
+      return { valid: true, inStock: order.inStock, total: order.total };
+    },
   })
   .choice("check-inventory", {
     choices: [
@@ -35,16 +41,35 @@ const orderWorkflow = workflow("process-order")
   .parallel("fulfill", {
     branches: [
       workflow.branch("shipping")
-        .task("ship", { job: "create-shipment" })
+        .task("ship", {
+          // inputSchema as function: maps previous step output to this step's input
+          inputSchema: (prev) => ({ orderId: prev.orderId }),
+          handler: async (input, ctx) => {
+            return await ctx.plugins.shipping.createShipment(input.orderId);
+          },
+        })
         .build(),
       workflow.branch("notification")
-        .task("notify", { job: "send-confirmation" })
+        .task("notify", {
+          inputSchema: (prev, workflowInput) => ({
+            orderId: workflowInput.orderId,
+            total: prev.total,
+          }),
+          handler: async (input, ctx) => {
+            await ctx.plugins.email.sendConfirmation(input);
+            return { sent: true };
+          },
+        })
         .build(),
     ],
     next: "complete",
   })
+  // Subsequent tasks: inputSchema as function receives prev step output
   .task("backorder", {
-    job: "create-backorder",
+    inputSchema: (prev) => ({ orderId: prev.orderId, total: prev.total }),
+    handler: async (input, ctx) => {
+      return await ctx.plugins.orders.createBackorder(input);
+    },
     next: "complete",
   })
   .pass("complete", { end: true })
@@ -82,25 +107,26 @@ ctx.core.events.on("workflow.progress", (data) => {
 
 ### Task
 
-Executes a job (in-process or external) and waits for completion.
+Executes an inline handler function with typed input/output.
 
 ```typescript
 workflow("example")
   .task("step-name", {
-    // Required: job to execute
-    job: "my-job-name",
-
-    // Optional: transform workflow context to job input
-    input: (ctx) => ({
-      orderId: ctx.input.orderId,
-      previousResult: ctx.steps.previousStep,
-    }),
-
-    // Optional: transform job result to step output
-    output: (result, ctx) => ({
-      processed: true,
-      data: result.data,
-    }),
+    // Input: Zod schema (for first step) OR mapper function (for subsequent steps)
+    // First step - validates workflow input:
+    inputSchema: z.object({ orderId: z.string() }),
+    // Subsequent steps - maps previous output to this step's input:
+    // inputSchema: (prev, workflowInput) => ({ orderId: prev.orderId }),
+
+    // Optional: Zod schema for output validation
+    outputSchema: z.object({ success: z.boolean(), data: z.any() }),
+
+    // Required: inline handler function
+    handler: async (input, ctx) => {
+      // input is typed from inputSchema
+      // ctx provides access to plugins, prev, steps, etc.
+      return { success: true, data: await processOrder(input.orderId) };
+    },
 
     // Optional: retry configuration
     retry: {
@@ -119,6 +145,58 @@ workflow("example")
   })
 ```
 
+#### Input Schema Options
+
+**Option 1: Zod Schema (first step or when validating workflow input)**
+```typescript
+.task("validate", {
+  inputSchema: z.object({ orderId: z.string(), userId: z.string() }),
+  handler: async (input, ctx) => {
+    // input: { orderId: string, userId: string } - validated from workflow input
+    return { valid: true };
+  },
+})
+```
+
+**Option 2: Mapper Function (subsequent steps)**
+```typescript
+.task("charge", {
+  // prev = output from previous step, workflowInput = original workflow input
+  inputSchema: (prev, workflowInput) => ({
+    amount: prev.total,
+    userId: workflowInput.userId,
+  }),
+  handler: async (input, ctx) => {
+    // input: { amount: number, userId: string } - inferred from mapper return
+    return { chargeId: "ch_123" };
+  },
+})
+```
+
+#### Legacy API (Job-based)
+
+For backward compatibility, you can still use job references:
+
+```typescript
+workflow("example")
+  .task("step-name", {
+    // Job name to execute
+    job: "my-job-name",
+
+    // Optional: transform workflow context to job input
+    input: (ctx) => ({
+      orderId: ctx.input.orderId,
+      previousResult: ctx.steps.previousStep,
+    }),
+
+    // Optional: transform job result to step output
+    output: (result, ctx) => ({
+      processed: true,
+      data: result.data,
+    }),
+  })
+```
+
 ### Parallel
 
 Runs multiple workflow branches concurrently.
@@ -219,6 +297,9 @@ interface WorkflowContext {
   /** Results from completed steps (keyed by step name) */
   steps: Record<string, any>;
 
+  /** Output from the previous step (undefined for first step) */
+  prev?: any;
+
   /** Current workflow instance */
   instance: WorkflowInstance;
 
@@ -230,18 +311,17 @@ interface WorkflowContext {
 Example usage in step configuration:
 
 ```typescript
+// Using inputSchema mapper function (recommended)
 .task("process", {
-  job: "process-data",
-  input: (ctx) => ({
-    // Access original input
-    orderId: ctx.input.orderId,
-
-    // Access previous step output
-    validationResult: ctx.steps.validate,
-
-    // Type-safe access
-    amount: ctx.getStepResult<{ amount: number }>("calculate")?.amount,
+  inputSchema: (prev, workflowInput) => ({
+    orderId: workflowInput.orderId,
+    validationResult: prev,  // prev = output from previous step
   }),
+  handler: async (input, ctx) => {
+    // Access any step's output
+    const calcResult = ctx.getStepResult<{ amount: number }>("calculate");
+    return { processed: true, amount: calcResult?.amount };
+  },
 })
 ```
 
@@ -421,24 +501,41 @@ For this to work properly:
 
 ```typescript
 import { AppServer, workflow, createDatabase } from "@donkeylabs/server";
+import { z } from "zod";
 
-// Define workflow
+// Define workflow with inline handlers
 const onboardingWorkflow = workflow("user-onboarding")
   .timeout(86400000) // 24 hour max
   .defaultRetry({ maxAttempts: 3 })
 
+  // First step: inputSchema validates workflow input
   .task("create-account", {
-    job: "create-user-account",
-    input: (ctx) => ctx.input,
+    inputSchema: z.object({
+      email: z.string().email(),
+      name: z.string(),
+      plan: z.enum(["free", "pro", "enterprise"]),
+    }),
+    outputSchema: z.object({ userId: z.string() }),
+    handler: async (input, ctx) => {
+      const user = await ctx.plugins.users.create({
+        email: input.email,
+        name: input.name,
+      });
+      return { userId: user.id };
+    },
   })
 
+  // Subsequent steps: inputSchema maps previous output
   .task("send-welcome-email", {
-    job: "send-email",
-    input: (ctx) => ({
-      to: ctx.input.email,
-      template: "welcome",
-      userId: ctx.steps["create-account"].userId,
+    inputSchema: (prev, workflowInput) => ({
+      to: workflowInput.email,
+      template: "welcome" as const,
+      userId: prev.userId,
     }),
+    handler: async (input, ctx) => {
+      await ctx.plugins.email.send(input);
+      return { sent: true };
+    },
   })
 
   .choice("check-plan", {
@@ -452,14 +549,24 @@ const onboardingWorkflow = workflow("user-onboarding")
   })
 
   .task("enterprise-setup", {
-    job: "setup-enterprise",
-    input: (ctx) => ({ userId: ctx.steps["create-account"].userId }),
+    // After a choice step, use handler to access specific step outputs
+    handler: async (input, ctx) => {
+      const userId = ctx.steps["create-account"].userId;
+      await ctx.plugins.accounts.setupEnterprise({
+        userId,
+        features: ["sso", "audit-logs", "dedicated-support"],
+      });
+      return { setup: "enterprise", userId };
+    },
     next: "complete",
   })
 
   .task("standard-setup", {
-    job: "setup-standard",
-    input: (ctx) => ({ userId: ctx.steps["create-account"].userId }),
+    handler: async (input, ctx) => {
+      const userId = ctx.steps["create-account"].userId;
+      await ctx.plugins.accounts.setupStandard({ userId });
+      return { setup: "standard", userId };
+    },
     next: "complete",
   })
 
@@ -476,19 +583,6 @@ const onboardingWorkflow = workflow("user-onboarding")
 // Setup server
 const server = new AppServer({ db: createDatabase() });
 
-// Register jobs that workflows use
-server.getCore().jobs.register("create-user-account", async (data) => {
-  // ... create user
-  return { userId: "user-123" };
-});
-
-server.getCore().jobs.register("send-email", async (data) => {
-  // ... send email
-  return { sent: true };
-});
-
-// ... register other jobs
-
 // Register workflow
 server.getCore().workflows.register(onboardingWorkflow);