ocpipe 0.2.1 → 0.3.1

package/DESIGN.md

@@ -0,0 +1,257 @@
+# Design
+
+ocpipe separates the **what** (Signatures declare input/output contracts), the **how** (Modules compose predictors), and the **when** (Pipelines orchestrate execution).
+
+## Core Concepts
+
+### Signatures
+
+A Signature declares **what** an LLM interaction does - its inputs, outputs, and purpose.
+
+```typescript
+import { signature, field } from 'ocpipe'
+import { z } from 'zod'
+
+const AnalyzeCode = signature({
+  doc: 'Analyze code for potential issues and improvements.',
+  inputs: {
+    code: field.string('Source code to analyze'),
+    language: field.enum(['typescript', 'python', 'rust'] as const),
+  },
+  outputs: {
+    issues: field.array(
+      z.object({
+        severity: z.enum(['error', 'warning', 'info']),
+        message: z.string(),
+        line: z.number(),
+      }),
+      'List of issues found',
+    ),
+    suggestions: field.array(z.string(), 'Improvement suggestions'),
+    score: field.number('Code quality score 0-100'),
+  },
+})
+```
+
+**Field helpers:**
+
+- `field.string(desc?)` - String field
+- `field.number(desc?)` - Number field
+- `field.boolean(desc?)` - Boolean field
+- `field.array(itemType, desc?)` - Array field
+- `field.object(shape, desc?)` - Object field
+- `field.enum(values, desc?)` - Enum field
+- `field.optional(field)` - Optional wrapper
+- `field.nullable(field)` - Nullable wrapper
+- `field.custom(zodType, desc?)` - Custom Zod type
+
+### Predict
+
+`Predict` bridges a Signature and OpenCode. It handles prompt generation, response parsing, and validation.
+
+```typescript
+import { Predict } from 'ocpipe'
+
+const predict = new Predict(AnalyzeCode)
+const result = await predict.execute(
+  { code: '...', language: 'typescript' },
+  ctx,
+)
+
+// With configuration
+const predict = new Predict(AnalyzeCode, {
+  agent: 'code-reviewer',
+  model: { providerID: 'anthropic', modelID: 'claude-opus-4-5' },
+  newSession: true,
+  template: (inputs) => `...`,
+})
+```
+
+### Module
+
+A Module encapsulates a logical unit of work with one or more Predictors.
+
+**SignatureModule** - For simple modules wrapping a single signature:
+
+```typescript
+import { SignatureModule } from 'ocpipe'
+
+class IntentParser extends SignatureModule<typeof ParseIntent> {
+  constructor() {
+    super(ParseIntent)
+  }
+
+  async forward(input, ctx) {
+    const result = await this.predictor.execute(input, ctx)
+    return result.data
+  }
+}
+```
+
+**Module** - For complex modules with multiple predictors:
+
+```typescript
+import { Module } from 'ocpipe'
+
+class CodeAnalyzer extends Module<
+  { code: string; language: string },
+  { issues: Issue[]; score: number }
+> {
+  private analyze = this.predict(AnalyzeCode)
+  private suggest = this.predict(SuggestFixes, { agent: 'code-fixer' })
+
+  async forward(input, ctx) {
+    const analysis = await this.analyze.execute(input, ctx)
+
+    if (analysis.data.issues.some((i) => i.severity === 'error')) {
+      const fixes = await this.suggest.execute(
+        {
+          code: input.code,
+          issues: analysis.data.issues,
+        },
+        ctx,
+      )
+
+      return {
+        issues: analysis.data.issues,
+        fixes: fixes.data.suggestions,
+        score: analysis.data.score,
+      }
+    }
+
+    return {
+      issues: analysis.data.issues,
+      score: analysis.data.score,
+    }
+  }
+}
+```
+
+### Pipeline
+
+Pipeline orchestrates execution with session management, checkpointing, logging, and retry logic.
+
+```typescript
+import { Pipeline, createBaseState } from 'ocpipe'
+
+const pipeline = new Pipeline(
+  {
+    name: 'code-review',
+    defaultModel: { providerID: 'anthropic', modelID: 'claude-sonnet-4-5' },
+    defaultAgent: 'general',
+    checkpointDir: './ckpt',
+    logDir: './logs',
+    retry: { maxAttempts: 2, onParseError: true },
+    timeoutSec: 300,
+  },
+  createBaseState,
+)
+
+// Run modules
+const result = await pipeline.run(new CodeAnalyzer(), {
+  code: sourceCode,
+  language: 'typescript',
+})
+
+// Run with step options
+const result = await pipeline.run(new CodeAnalyzer(), input, {
+  name: 'analyze-main',
+  model: { providerID: 'anthropic', modelID: 'claude-opus-4-5' },
+  newSession: true,
+  retry: { maxAttempts: 3 },
+})
+
+// Access state
+console.log(pipeline.state.steps)
+console.log(pipeline.getSessionId())
+
+// Resume from checkpoint
+const resumed = await Pipeline.loadCheckpoint(config, sessionId)
+```
+
+### State Management
+
+Automatic checkpointing after each step:
+
+```typescript
+import { createBaseState, extendBaseState } from 'ocpipe'
+
+// Basic state
+const state = createBaseState()
+// { sessionId, startedAt, phase, steps, subPipelines }
+
+// Extended state
+interface MyState extends BaseState {
+  inputPath: string
+  results: AnalysisResult[]
+}
+
+const pipeline = new Pipeline(config, () => ({
+  ...createBaseState(),
+  inputPath: '/path/to/input',
+  results: [],
+}))
+```
+
+## Auto-Correction
+
+Automatically corrects LLM schema mismatches using JSON Patch (RFC 6902):
+
+```typescript
+super(MySignature, {
+  correction: {
+    method: 'json-patch', // or 'jq'
+    maxFields: 5,
+    maxRounds: 3,
+  },
+})
+```
+
+The correction system:
+
+1. Detects schema validation errors
+2. Finds similar field names in the response
+3. Asks the LLM for patches to fix errors
+4. Applies patches and re-validates
+5. Retries up to configured rounds
+
+## Testing
+
+Mock backends for unit testing without real LLM calls:
+
+```typescript
+import {
+  MockAgentBackend,
+  createMockContext,
+  generateMockOutputs,
+} from 'ocpipe'
+import { vi } from 'vitest'
+
+const mock = new MockAgentBackend()
+mock.addJsonResponse({
+  intent: 'greeting',
+  confidence: 0.95,
+  keywords: ['hello', 'world'],
+})
+
+vi.mock('./agent.js', () => ({
+  runAgent: mock.createRunner(),
+}))
+
+const ctx = createMockContext({
+  defaultModel: { providerID: 'anthropic', modelID: 'claude-sonnet-4-5' },
+})
+
+// Auto-generate mock outputs from schema
+const mockData = generateMockOutputs(ParseIntent.outputs)
+```
+
+## Why No ChainOfThought or ReAct?
+
+Unlike DSPy, ocpipe does not provide `ChainOfThought` or `ReAct` variants:
+
+- OpenCode agents already do chain-of-thought reasoning
+- OpenCode agents already have tool access (ReAct)
+- Adding these would duplicate functionality
+
+Configure your OpenCode agent for tool access. The agent handles complexity; ocpipe structures the contract.

package/GETTING_STARTED.md

@@ -0,0 +1,384 @@
+# Getting Started with ocpipe
+
+This guide walks you through building and running a simple "Hello World" application using ocpipe (OpenCode Pipeline).
+
+**Repository:** https://github.com/s4wave/ocpipe
+
+## Prerequisites
+
+- [Bun](https://bun.sh) runtime
+- [OpenCode](https://opencode.ai) CLI installed and configured
+
+## Installation
+
+```bash
+bun add ocpipe zod
+```
+
+## Quick Start with REPL
+
+The fastest way to explore ocpipe is with `bun repl`:
+
+```bash
+bun repl
+```
+
+Then paste this:
+
+```typescript
+import { signature, field, module, Pipeline, createBaseState } from 'ocpipe'
+
+const Greet = signature({
+  doc: 'Generate a friendly greeting for the given name.',
+  inputs: { name: field.string('The name of the person to greet') },
+  outputs: {
+    greeting: field.string('A friendly greeting message'),
+    emoji: field.string('An appropriate emoji for the greeting'),
+  },
+})
+
+const pipeline = new Pipeline(
+  {
+    name: 'repl-demo',
+    defaultModel: { providerID: 'anthropic', modelID: 'claude-haiku-4-5' },
+    defaultAgent: 'code',
+  },
+  createBaseState,
+)
+
+const result = await pipeline.run(module(Greet), { name: 'World' })
+console.log(result.data.greeting, result.data.emoji)
+```
+
+You'll see the pipeline execute and print something like:
+
+```
+Hello, World! It's wonderful to meet you! :wave:
+```
+
+## Running the Example
+
+The `example/` directory contains a complete hello world application. Run it directly:
+
+```bash
+bun run example/index.ts
+```
+
+This will:
+
+1. Create a pipeline with default configuration
+2. Send a greeting request to the LLM
+3. Print the generated greeting and emoji
+
+**Expected output:**
+
+```
+============================================================
+STEP 1: Greeter
+============================================================
+
+>>> OpenCode [code] [anthropic/claude-haiku-4-5] [new session]: Generate a friendly greeting for the given name...
+
+<<< OpenCode done (85 chars) [session:abc123]
+
+=== Result ===
+Greeting: Hello, World! It's wonderful to meet you!
+Emoji: :wave:
+```
+
+**Tip:** You can view what the agent did by running `opencode` to open the OpenCode UI, then typing `/sessions` to see the session list. Find the session ID from the output above and select it to see the full conversation.
+
+## Understanding the Example
+
+The example has three files that demonstrate ocpipe's core concepts:
+
+### 1. Signature (`signature.ts`)
+
+A **Signature** declares the contract between your code and the LLM. It defines:
+
+- `doc`: Instructions for the LLM
+- `inputs`: What data you provide
+- `outputs`: What data you expect back
+
+```typescript
+import { signature, field } from '../index.js'
+
+export const Greet = signature({
+  doc: 'Generate a friendly greeting for the given name.',
+  inputs: {
+    name: field.string('The name of the person to greet'),
+  },
+  outputs: {
+    greeting: field.string('A friendly greeting message'),
+    emoji: field.string('An appropriate emoji for the greeting'),
+  },
+})
+```
+
+### 2. Module (`module.ts`)
+
+A **Module** wraps a signature with execution logic. `SignatureModule` is a convenience class that automatically creates a predictor from your signature:
+
+```typescript
+import { SignatureModule } from '../index.js'
+import { Greet } from './signature.js'
+
+export class Greeter extends SignatureModule<typeof Greet> {
+  constructor() {
+    super(Greet)
+  }
+
+  async forward(input: { name: string }, ctx: ExecutionContext) {
+    const result = await this.predictor.execute(input, ctx)
+    return result.data
+  }
+}
+```
+
+### 3. Pipeline (`index.ts`)
+
+A **Pipeline** orchestrates execution, managing sessions, checkpoints, and retries:
+
+```typescript
+import { Pipeline, createBaseState } from '../index.js'
+import { Greeter } from './module.js'
+
+const pipeline = new Pipeline(
+  {
+    name: 'hello-world',
+    defaultModel: { providerID: 'anthropic', modelID: 'claude-haiku-4-5' },
+    defaultAgent: 'code',
+    checkpointDir: './ckpt',
+    logDir: './logs',
+  },
+  createBaseState,
+)
+
+const result = await pipeline.run(new Greeter(), { name: 'World' })
+console.log(result.data.greeting)
+```
+
+## Modifying the Example
+
+Let's extend the example to generate both a greeting and a farewell.
+
+### Step 1: Add a new signature
+
+Create `farewell-signature.ts`:
+
+```typescript
+import { signature, field } from '../index.js'
+
+export const Farewell = signature({
+  doc: 'Generate a friendly farewell for the given name.',
+  inputs: {
+    name: field.string('The name of the person to bid farewell'),
+    context: field.string(
+      'The context of the farewell (e.g., "end of meeting", "going on vacation")',
+    ),
+  },
+  outputs: {
+    farewell: field.string('A friendly farewell message'),
+    emoji: field.string('An appropriate emoji for the farewell'),
+  },
+})
+```
+
+### Step 2: Add a new module
+
+Create `farewell-module.ts`:
+
+```typescript
+import { SignatureModule } from '../index.js'
+import type { ExecutionContext } from '../types.js'
+import { Farewell } from './farewell-signature.js'
+
+export class Fareweller extends SignatureModule<typeof Farewell> {
+  constructor() {
+    super(Farewell)
+  }
+
+  async forward(
+    input: { name: string; context: string },
+    ctx: ExecutionContext,
+  ) {
+    const result = await this.predictor.execute(input, ctx)
+    return result.data
+  }
+}
+```
+
+### Step 3: Run both modules in sequence
+
+Update `index.ts`:
+
+```typescript
+import { Pipeline, createBaseState } from '../index.js'
+import { Greeter } from './module.js'
+import { Fareweller } from './farewell-module.js'
+
+async function main() {
+  const pipeline = new Pipeline(
+    {
+      name: 'hello-goodbye',
+      defaultModel: { providerID: 'anthropic', modelID: 'claude-haiku-4-5' },
+      defaultAgent: 'code',
+      checkpointDir: './ckpt',
+      logDir: './logs',
+    },
+    createBaseState,
+  )
+
+  // Run greeter
+  const greeting = await pipeline.run(new Greeter(), { name: 'Alice' })
+  console.log(`\nGreeting: ${greeting.data.greeting} ${greeting.data.emoji}`)
+
+  // Run fareweller (reuses the same session for context)
+  const farewell = await pipeline.run(new Fareweller(), {
+    name: 'Alice',
+    context: 'end of meeting',
+  })
+  console.log(`Farewell: ${farewell.data.farewell} ${farewell.data.emoji}`)
+}
+
+main().catch(console.error)
+```
+
+### Step 4: Run it
+
+```bash
+bun run example/index.ts
+```
+
+## Auto-Correction Example
+
+ocpipe automatically corrects schema mismatches using patches when the LLM returns incorrect field names. Run the correction demo:
+
+```bash
+bun run example/correction.ts
+```
+
+This example uses field names that LLMs sometimes get wrong:
+
+- `issue_type` (LLMs may return `type`)
+- `severity` (LLMs may return `priority`)
+- `explanation` (LLMs may return `description` or `reason`)
+- `suggested_tags` (LLMs may return `tags`)
+
+**Note:** Modern LLMs like Claude often follow the schema correctly. The correction system is a safety net for when they don't. You may not see correction rounds if the LLM gets it right the first time.
+
+If the LLM does return incorrect field names, you'll see correction rounds:
+
+```
+>>> Correction round 1/3 [json-patch]: fixing 2 field(s)...
+  JSON Patch: [{"op":"move","from":"/type","path":"/issue_type"},{"op":"move","from":"/priority","path":"/severity"}]
+  Round 1 complete, 0 error(s) remaining
+  Schema correction successful after 1 round(s)!
+```
+
+The correction system:
+
+1. Validates the LLM's response against the output schema
+2. If validation fails, identifies which fields have errors
+3. Asks the LLM to generate patches to fix the errors
+4. Applies patches and re-validates
+5. Retries up to 3 rounds if needed
+
+### Correction Methods
+
+ocpipe supports two correction methods:
+
+| Method                 | Format               | Requirements           |
+| ---------------------- | -------------------- | ---------------------- |
+| `json-patch` (default) | RFC 6902 JSON Patch  | None (pure TypeScript) |
+| `jq`                   | jq-style expressions | `jq` binary installed  |
+
+**JSON Patch** is the default because it requires no external dependencies and uses a standardized format that LLMs are familiar with from API documentation.
+
+To use jq instead:
+
+```typescript
+super(MySignature, {
+  correction: {
+    method: 'jq', // Use jq-style patches (requires jq binary)
+  },
+})
+```
+
+To disable auto-correction:
+
+```typescript
+super(MySignature, { correction: false })
+```
+
+Full configuration options:
+
+```typescript
+super(MySignature, {
+  correction: {
+    method: 'json-patch', // 'json-patch' (default) or 'jq'
+    maxFields: 5, // Max fields to fix per round
+    maxRounds: 3, // Max correction attempts
+  },
+})
+```
+
+## Key Concepts
+
+### Session Continuity
+
+By default, ocpipe reuses the OpenCode session across pipeline steps. This means the LLM maintains context between calls. Use `newSession: true` in run options to start fresh:
+
+```typescript
+await pipeline.run(module, input, { newSession: true })
+```
+
+### Checkpointing
+
+ocpipe automatically saves state after each step to `checkpointDir`. Resume from a checkpoint:
+
+```typescript
+const resumed = await Pipeline.loadCheckpoint(config, sessionId)
+```
+
+### Field Types
+
+ocpipe provides field helpers for common types:
+
+```typescript
+field.string('description') // string
+field.number('description') // number
+field.boolean('description') // boolean
+field.array(z.string(), 'description') // string[]
+field.object({ key: z.string() }) // { key: string }
+field.enum(['a', 'b'] as const) // 'a' | 'b'
+field.optional(field.string()) // string | undefined
+```
+
+### Complex Modules
+
+For modules with multiple predictors or transformed outputs, use the base `Module` class:
+
+```typescript
+import { Module } from '../index.js'
+
+class ComplexModule extends Module<
+  { input: string },
+  { result: string; metadata: object }
+> {
+  private step1 = this.predict(Signature1)
+  private step2 = this.predict(Signature2, { agent: 'specialist' })
+
+  async forward(input, ctx) {
+    const r1 = await this.step1.execute(input, ctx)
+    const r2 = await this.step2.execute({ data: r1.data }, ctx)
+    return { result: r2.data.output, metadata: r1.data }
+  }
+}
+```
+
+## Next Steps
+
+- Read the full [README.md](./README.md) for advanced features
+- Check the test files (`*.test.ts`) for more usage examples
+- Explore `testing.ts` for unit testing without real LLM calls