@workglow/task-graph 0.0.52

package/src/task/README.md

@@ -0,0 +1,251 @@
+# Task System Documentation
+
+This module provides a flexible task processing system with support for various task types, dependency management, and error handling.
+
+- [Key Components](#key-components)
+  - [Core Classes](#core-classes)
+- [Task Types](#task-types)
+  - [A Simple Task](#a-simple-task)
+  - [GraphAsTask](#graphastask)
+  - [ArrayTask](#arraytask)
+  - [Job Queue Tasks](#job-queue-tasks)
+- [Task Lifecycle](#task-lifecycle)
+- [Event Handling](#event-handling)
+- [Input/Output Schemas](#inputoutput-schemas)
+- [Registry \& Queues](#registry--queues)
+- [Error Handling](#error-handling)
+- [Testing](#testing)
+- [Installation](#installation)
+
+## Key Components
+
+### Core Classes
+
+- `Task`: Base class implementing core task functionality
+- `ArrayTask`: Executes a task or a task with multiple inputs in parallel with a subGraph
+- `JobQueueTask`: Integrates with job queue system for distributed processing
+
+## Task Types
+
+### A Simple Task
+
+```typescript
+interface MyTaskInput {
+  input: number;
+}
+interface MyTaskOutput {
+  result: number;
+}
+class MyTask extends Task {
+  static readonly type = "MyTask"; // Required, unique identifier for the task
+  static readonly category = "Utility"; // Optional, used for grouping tasks in UI
+  static readonly title = "My Task"; // Optional, used for a UI
+  static readonly description = "My Task Description"; // Optional, used for a UI
+  declare runInputData: MyTaskInput;
+  declare runOutputData: MyTaskOutput;
+  static inputSchema = Type.Object({
+    input: Type.Number(),
+  });
+  static outputSchema = Type.Object({
+    result: Type.Number(),
+  });
+
+  // typically you either override execute or executeReactive, but not both
+  async execute(input: MyTaskInput, { signal, updateProgress }: IExecuteContext) {
+    await sleep(1000);
+    if (signal.aborted) {
+      throw new TaskAbortedError("Task aborted");
+    }
+    await updateProgress(0.5, "Processing...");
+    // Do something with the input that takes a long time
+    await sleep(1000);
+    return { result: input.input * 2 };
+  }
+  async executeReactive(input: MyTaskInput, output: MyTaskOutput) {
+    return { result: input.input * 2 };
+  }
+}
+```
+
+### GraphAsTask
+
+- GraphAsTask tasks are tasks that contain other tasks. They are represented as an internal TaskGraph.
+- A ArrayTask is a compound task that can run a task as normal, or if the inputs are an array and the input definition has x-replicate=true defined for that input, then the task will run parallel copies with a subGraph.
+
+### ArrayTask
+
+- ArrayTask is a task that can run a task as normal, or if the inputs are an arryay and the input definition has x-replicate=true, then the task will run parallel copies with a subGraph.
+- The subGraph is a TaskGraph that is created from the inputs of the task.
+- The results of the subGraph are combined such that the outputs are turned into arrays.
+
+### Job Queue Tasks
+
+JobQueueTask is a task that can be used to run a task in a job queue. This is useful for when there might be rate limits or other constraints on the task that make it better to run in a job queue than in the main thread.
+
+```typescript
+class MyJobTask extends JobQueueTask {
+  async createJob() {
+    return new Job({
+      input: this.runInputData,
+      execute: (input) => ({ result: input.value * 3 }),
+    });
+  }
+}
+```
+
+## Task Lifecycle
+
+- **Statuses**: `Pending` → `Processing` → (`Completed`|`Failed`|`Aborted`)
+- **Methods**:
+  - `run()`: Full execution with caching, calls the subclass `execute` method
+  - `runReactive()`: Lightweight execution for UI updates, calls the subclass `executeReactive` method
+  - `abort()`: Cancel running task
+
+## Event Handling
+
+```typescript
+task.on("start", () => console.log("Task started"));
+task.on("progress", (p) => console.log(`Progress: ${p}%`));
+task.on("complete", () => console.log("Task completed"));
+task.on("error", (err) => console.error("Task failed", err));
+task.on("abort", () => console.log("Task aborted"));
+task.on("regenerate", () => console.log("Task regenerated"));
+```
+
+## Input/Output Schemas
+
+The input and output schemas are JSON schemas that are used to validate the input and output of the task. These can be defined using plain JSON Schema objects, TypeBox, or Zod. All schemas must be compatible with `DataPortSchema` from `@workglow/util`.
+
+### Using Plain JSON Schema
+
+```typescript
+import { DataPortSchema } from "@workglow/util";
+
+static inputSchema = () => {
+  return {
+    type: "object",
+    properties: {
+      username: {
+        type: "string",
+        title: "User Name",
+        description: "The name of the user",
+        default: "guest",
+      },
+    },
+    additionalProperties: false,
+  } as const satisfies DataPortSchema;
+};
+
+static outputSchema = () => {
+  return {
+    type: "object",
+    properties: {
+      result: {
+        type: "number",
+        title: "Processing Result",
+        description: "The result of the processing",
+      },
+    },
+    required: ["result"],
+    additionalProperties: false,
+  } as const satisfies DataPortSchema;
+};
+```
+
+### Using TypeBox
+
+TypeBox schemas are JSON Schema compatible and can be used directly:
+
+```typescript
+import { Type } from "@sinclair/typebox";
+import { DataPortSchema } from "@workglow/util";
+
+static inputSchema = () => {
+  return Type.Object({
+    username: Type.String({
+      title: "User Name",
+      description: "The name of the user",
+      default: "guest",
+    }),
+  }) satisfies DataPortSchema;
+};
+
+static outputSchema = () => {
+  return Type.Object({
+    result: Type.Number({
+      title: "Processing Result",
+      description: "The result of the processing",
+    }),
+  }) satisfies DataPortSchema;
+};
+```
+
+### Using Zod
+
+Zod 4 has built-in JSON Schema support using the `.toJSONSchema()` method:
+
+```typescript
+import { z } from "zod";
+import { DataPortSchema } from "@workglow/util";
+
+// Define Zod schemas
+const inputSchemaZod = z.object({
+  username: z.string().default("guest").describe("The name of the user"),
+});
+
+const outputSchemaZod = z.object({
+  result: z.number().describe("The result of the processing"),
+});
+
+// Infer TypeScript types using Zod's built-in type inference
+type MyInput = z.infer<typeof inputSchemaZod>;
+type MyOutput = z.infer<typeof outputSchemaZod>;
+
+static inputSchema = () => {
+  return inputSchemaZod.toJSONSchema() as DataPortSchema;
+};
+
+static outputSchema = () => {
+  return outputSchemaZod.toJSONSchema() as DataPortSchema;
+};
+```
+
+## Registry & Queues
+
+The TaskRegistry is used to register tasks to there is a global registry. This is useful for a node based UI to allow tasks to be dragged and dropped onto the canvas.
+
+```typescript
+TaskRegistry.registerTask(MyTask);
+```
+
+The TaskQueueRegistry is used to get a queue for a given name. This is useful for when you want to run a task in a job queue. A queue can be created for a given task type, and all the tasks of that type will be added to the queue.
+
+```typescript
+// Queue management
+const queue = getTaskQueueRegistry().getQueue("processing");
+queue.add(new MyJobTask());
+```
+
+## Error Handling
+
+```typescript
+try {
+  await task.run();
+} catch (err) {
+  if (err instanceof TaskAbortedError) {
+    console.log("Task was aborted");
+  }
+}
+```
+
+## Testing
+
+```bash
+bun test
+```
+
+## Installation
+
+```bash
+bun add @workglow/task-system
+```

package/src/task-graph/README.md

@@ -0,0 +1,142 @@
+# Task Graph Package
+
+A robust TypeScript library for creating and managing task graphs with dependencies, enabling complex workflow orchestration and execution.
+
+- [Features](#features)
+- [Installation](#installation)
+- [Basic Usage](#basic-usage)
+  - [Creating a Task Graph](#creating-a-task-graph)
+  - [Executing Tasks](#executing-tasks)
+  - [Workflow API](#workflow-api)
+- [Key Components](#key-components)
+  - [TaskGraph](#taskgraph)
+  - [Dataflow](#dataflow)
+  - [TaskGraphRunner](#taskgraphrunner)
+  - [Workflow](#workflow)
+- [Testing](#testing)
+- [API Reference](#api-reference)
+  - [`TaskGraph` Class](#taskgraph-class)
+  - [`Workflow` Class](#workflow-class)
+  - [Testing](#testing-1)
+- [License](#license)
+
+## Features
+
+- Directed Acyclic Graph (DAG) structure for task dependencies
+- Data flow management between task inputs/outputs
+- Workflow builder API with fluent interface
+- Provenance tracking
+- Caching of task results (same run on same input returns cached result)
+- Error handling and abortion support
+- Serial and parallel execution patterns
+- Reactive execution capabilities to drive UI updates
+
+## Installation
+
+```bash
+# Within the monorepo
+npm install @workglow/task-graph
+```
+
+## Basic Usage
+
+### Creating a Task Graph
+
+```typescript
+import { TaskGraph, Dataflow } from "@workglow/task-graph";
+
+const graph = new TaskGraph();
+const task1 = new TestTask({ input: "hello" }, { id: "task1" });
+const task2 = new TestTask({ input: "world" }, { id: "task2" });
+
+graph.addTasks([task1, task2]);
+graph.addDataflow(new Dataflow("task1", "output", "task2", "input"));
+```
+
+### Executing Tasks
+
+```typescript
+const results = await graph.run();
+```
+
+### Workflow API
+
+```typescript
+const workflow = new Workflow()
+  .SimpleTask({ input: "start" })
+  .rename("output", "input")
+  .SimpleProcessingTask()
+  .parallel(
+    (w) => w.TestParallelTask1(),
+    (w) => w.TestParallelTask2()
+  );
+
+const output = await workflow.run();
+```
+
+## Key Components
+
+### TaskGraph
+
+- Manages nodes (tasks) and edges (data flows)
+- Topological sorting
+- Serialization/deserialization
+- Dependency tracking
+
+### Dataflow
+
+- Connects task outputs to inputs
+- Value propagation
+- Provenance tracking
+
+### TaskGraphRunner
+
+- Executes tasks with dependency resolution
+- Multiple scheduler implementations
+- Error handling and recovery
+- Abortion support
+
+### Workflow
+
+- Fluent API for graph construction
+- Automatic input/output matching
+- Parallel task groups
+- Error reporting
+
+## Testing
+
+Tests use Bun test runner. To run tests:
+
+```bash
+bun test
+bun test:watch  # For development
+```
+
+## API Reference
+
+### `TaskGraph` Class
+
+- `addTask(task: Task)`: Add a task
+- `addDataflow(dataflow: Dataflow)`: Connect tasks
+- `getSourceTasks()`/`getTargetTasks()`: Navigate dependencies
+- `toJSON()`: Serialize graph structure
+
+### `Workflow` Class
+
+- `createWorkflow()`: Task-specific builder methods
+- `parallel()`: Create parallel execution branches
+- `rename()`: Customize input/output mappings
+- `pop()`: Pop a task from the workflow (only use in a repl!)
+- `run()`: Execute workflow
+
+### Testing
+
+Tests use Bun test runner. To run tests:
+
+```bash
+bun test
+```
+
+## License
+
+Apache 2.0 - See [LICENSE](../../../LICENSE) file for details