coaiajs 0.1.0

package/rispecs/03-langfuse-module.spec.md

@@ -0,0 +1,101 @@
+# 03 — Langfuse Module
+
+> Full Langfuse observability integration via REST API.
+
+## Desired Outcome
+
+A comprehensive Langfuse client covering traces, observations, prompts, datasets, scores, score configs, comments, media, and projects — enabling full observability of CoAiA agent sessions through a clean TypeScript API.
+
+## Structural Tension
+
+**Current Reality:**
+- `src/langfuse/` directory exists but is empty
+- Types defined in `src/types.ts`: `ScoreCategory`, `ScoreConfig`
+- coaiapy's `cofuse.py` is 4,480 lines covering: traces, observations (spans/generations/events), prompts, datasets, scores, score configs, comments, media, projects
+- cofuse.py uses raw `requests` library against Langfuse REST API
+- No MCP tools exist for Langfuse in any parent project
+
+**Desired Outcome:**
+TypeScript Langfuse client with:
+- All REST endpoints from cofuse.py ported to `fetch`-based client
+- Lazy client initialization (no connection until first call)
+- Type-safe request/response with Zod validation
+- 8 MCP tools for interactive Langfuse operations
+- Support for both self-hosted and cloud Langfuse instances
+
+## API Surface
+
+### Traces
+```typescript
+listTraces(params?: { page?, limit?, userId?, name?, tags? }): Promise<Trace[]>
+getTrace(traceId: string): Promise<Trace>
+createTrace(input: CreateTraceInput): Promise<Trace>
+```
+
+### Observations
+```typescript
+listObservations(params?: { traceId?, type?, name? }): Promise<Observation[]>
+getObservation(observationId: string): Promise<Observation>
+```
+
+### Prompts
+```typescript
+listPrompts(params?: { name?, label? }): Promise<Prompt[]>
+getPrompt(name: string, version?: number): Promise<Prompt>
+createPrompt(input: CreatePromptInput): Promise<Prompt>
+```
+
+### Datasets
+```typescript
+listDatasets(): Promise<Dataset[]>
+getDataset(name: string): Promise<Dataset>
+createDataset(input: CreateDatasetInput): Promise<Dataset>
+listDatasetItems(datasetName: string): Promise<DatasetItem[]>
+createDatasetItem(input: CreateDatasetItemInput): Promise<DatasetItem>
+listDatasetRuns(datasetName: string): Promise<DatasetRun[]>
+```
+
+### Scores
+```typescript
+listScores(params?: { traceId?, name?, configId? }): Promise<Score[]>
+createScore(input: CreateScoreInput): Promise<Score>
+listScoreConfigs(): Promise<ScoreConfig[]>
+createScoreConfig(input: CreateScoreConfigInput): Promise<ScoreConfig>
+```
+
+### Comments
+```typescript
+listComments(params?: { traceId?, objectType? }): Promise<Comment[]>
+createComment(input: CreateCommentInput): Promise<Comment>
+```
+
+### Media & Projects
+```typescript
+getMedia(mediaId: string): Promise<Media>
+listProjects(): Promise<Project[]>
+```
+
+## MCP Tools (8)
+
+| Tool | Description |
+|------|-------------|
+| `langfuse_list_traces` | List traces with filters |
+| `langfuse_get_trace` | Get trace details with observations |
+| `langfuse_list_prompts` | List prompt templates |
+| `langfuse_get_prompt` | Get specific prompt by name/version |
+| `langfuse_list_datasets` | List available datasets |
+| `langfuse_list_scores` | List scores with filters |
+| `langfuse_create_score` | Score a trace or observation |
+| `langfuse_list_score_configs` | List scoring configurations |
+
+## Authentication
+
+Langfuse REST API uses Basic auth: `base64(publicKey:secretKey)`. Config provides `langfuse_public_key`, `langfuse_secret_key`, and `langfuse_host` (default: `https://cloud.langfuse.com`).
+
+## Quality Criteria
+
+- ✅ Every endpoint in cofuse.py has a TypeScript equivalent
+- ✅ Pagination handled transparently (auto-page through results when needed)
+- ✅ Auth failure returns clear error, not cryptic 401
+- ✅ Missing credentials detected at call time, not import time
+- ✅ Response types validated with Zod before returning

package/rispecs/04-narrative-engine.spec.md

@@ -0,0 +1,185 @@
+# 04 — Narrative Engine
+
+> JSONL knowledge graph with structural tension charts, narrative beats, and MMOT self-evaluation.
+
+This is the most complex module — the heart of CoAiA's creative advancement system.
+
+## Desired Outcome
+
+A KnowledgeGraphManager that provides full CRUD for entities and relations stored in append-only JSONL, implements the complete structural tension chart lifecycle (create → telescope → advance → complete), supports narrative beat creation with multi-universe perspectives, and enables MMOT autonomous self-evaluation — all through a single TypeScript class.
+
+## Structural Tension
+
+**Current Reality:**
+- `src/narrative/` directory exists but is empty
+- Types fully defined in `src/types.ts`: `Entity`, `EntityMetadata`, `Relation`, `RelationMetadata`, `KnowledgeGraph`, `McpToolResult`
+- coaia-narrative v0.12.0 has a working implementation:
+  - `graph-manager.ts` (1,294 lines) — the core engine
+  - `tool-definitions.ts` — 27 MCP tool schemas
+  - `tool-handlers.ts` — tool implementation connecting MCP to graph manager
+  - `tool-groups.ts` — tool filtering by group (STC_TOOLS, KG_TOOLS, CORE_TOOLS)
+  - `types.ts` — type definitions (absorbed into coaiajs `src/types.ts`)
+- coaia-narrative rispecs are the most mature (15 specs) — detailed behavioral specifications exist for every capability
+
+**Desired Outcome:**
+Narrative engine in `src/narrative/` that:
+- Implements `KnowledgeGraphManager` with identical behavior to coaia-narrative
+- Stores all data in append-only JSONL format
+- Manages the full STC lifecycle
+- Supports narrative beats with dramatic typing and multi-universe perspectives
+- Enables MMOT evaluation with Elements of Performance
+- Exposes 28 MCP tools (27 from coaia-narrative + 1 new)
+
+## Core Class: KnowledgeGraphManager
+
+### JSONL Storage
+
+```typescript
+class KnowledgeGraphManager {
+  constructor(memoryFilePath: string)
+  loadGraph(): Promise<KnowledgeGraph>
+  saveGraph(): Promise<void>
+}
+```
+
+The JSONL file is the single source of truth. Each line is a JSON event:
+- `{"type":"entity","data":{...}}` — entity creation/update
+- `{"type":"relation","data":{...}}` — relation creation
+- `{"type":"observation","data":{...}}` — observation added to entity
+
+On load: replay all events to reconstruct in-memory graph. On write: append new events. This append-only design prevents corruption from interrupted writes.
+
+### Knowledge Graph CRUD
+
+```typescript
+// Entities
+createEntities(entities: Entity[]): Promise<Entity[]>
+deleteEntities(entityNames: string[]): Promise<void>
+addObservations(observations: {entityName: string, contents: string[]}[]): Promise<void>
+deleteObservations(deletions: {entityName: string, observations: string[]}[]): Promise<void>
+
+// Relations
+createRelations(relations: Relation[]): Promise<Relation[]>
+deleteRelations(relations: {from: string, to: string, relationType: string}[]): Promise<void>
+
+// Queries
+searchNodes(query: string): Promise<Entity[]>
+openNodes(names: string[]): Promise<Entity[]>
+readGraph(): Promise<KnowledgeGraph>
+```
+
+### Structural Tension Chart Lifecycle
+
+```
+CREATE → TELESCOPE → ADVANCE → COMPLETE
+```
+
+**Create:** Desired outcome + current reality + optional action steps → chart entity + related entities + auto-distributed due dates. Validates creative orientation (no problem-solving language) and delayed resolution (no "ready to begin" in current reality).
+
+**Telescope:** Break an action step into a sub-chart with its own desired outcome, current reality, and action steps. Inherits due date constraints from parent.
+
+**Advance:** Mark action step complete → completion flows into parent current reality as fact. Update progress without completion via progress observations.
+
+**Complete:** All action steps done → chart marked complete. Phase transitions: germination → assimilation → completion.
+
+### STC Tools
+
+| Tool | Purpose |
+|------|---------|
+| `create_structural_tension_chart` | Create master chart |
+| `list_active_charts` | Overview all charts with progress |
+| `get_chart` | Full chart details |
+| `get_chart_progress` | Progress metrics and next action |
+| `manage_action_step` | Add or telescope action steps (unified) |
+| `mark_action_complete` | Complete an action, flow to parent reality |
+| `update_action_progress` | Track progress without completion |
+| `update_current_reality` | Add observations to current reality |
+| `update_desired_outcome` | Modify chart goal |
+| `get_action_step` | Get telescoped action step details |
+| `remove_action_step` | Delete action step from chart |
+
+### Narrative Beat System
+
+Narrative beats document story progression within a structural tension chart. Each beat has:
+- **Act** (1, 2, or 3) — three-act dramatic structure
+- **Dramatic type** — Exposition, Rising Action, Climax, Resolution, Discovery, Crisis, Integration
+- **Universes** — perspectives: Engineer World, Ceremony World, Story Engine World
+- **Prose** — character-level narrative
+- **Lessons** — extracted insights
+
+| Tool | Purpose |
+|------|---------|
+| `create_narrative_beat` | Create dramatic beat within chart |
+| `telescope_narrative_beat` | Break beat into sub-beats |
+| `list_narrative_beats` | List beats for a chart |
+
+### MMOT Evaluation
+
+Managerial Moment of Truth — autonomous self-evaluation against Elements of Performance:
+
+1. **Acknowledge** — Compare expected vs delivered output
+2. **Analyze** — What dynamics caused the gap?
+3. **Update** — Write findings into current reality
+4. **Recommit** — Add corrective action steps if needed
+
+Supports directional perspectives: South (design/structure), East (narrative/execution), West (embodied/execution), North (wisdom/design).
+
+| Tool | Purpose |
+|------|---------|
+| `perform_mmot_evaluation` | Run MMOT self-evaluation on chart |
+| `init_llm_guidance` | Provide methodology guidance to LLMs |
+
+### Knowledge Graph Tools
+
+| Tool | Purpose |
+|------|---------|
+| `create_entities` | Add entities to graph |
+| `create_relations` | Create relationships |
+| `add_observations` | Add facts to entities |
+| `search_nodes` | Full-text search |
+| `open_nodes` | Retrieve specific entities |
+| `read_graph` | Export complete graph |
+| `delete_entities` | Remove entities |
+| `delete_relations` | Remove relationships |
+| `delete_observations` | Remove observations |
+
+## Creative Orientation Validation
+
+Desired outcomes are validated against problem-solving language:
+- **Reject**: fix, solve, eliminate, prevent, stop, avoid, reduce, remove
+- **Accept**: create, build, establish, develop, design, manifest, achieve
+
+Current reality is validated against false readiness:
+- **Reject**: ready to, prepared to, all set, ready for, set to
+- **Accept**: specific factual assessments
+
+Error messages teach the principle, not just reject the input.
+
+## Date Distribution
+
+When action steps are created without explicit due dates:
+```
+totalTime = endDate - startDate
+stepInterval = totalTime / (stepCount + 1)
+dates[i] = startDate + (stepInterval * i)
+```
+
+## Related Rispecs
+
+- `coaia-narrative/rispecs/structural_tension_chart_creation.spec.md` — detailed STC creation behavior
+- `coaia-narrative/rispecs/telescoping_hierarchical_advancement.spec.md` — telescoping rules
+- `coaia-narrative/rispecs/advancing_pattern_tracking.spec.md` — completion flow
+- `coaia-narrative/rispecs/storage_knowledge_graph.spec.md` — JSONL format specification
+- `coaia-narrative/rispecs/mmot_evaluation_loop.spec.md` — MMOT phases
+- `coaia-narrative/rispecs/narrative_beat_creation.spec.md` — beat system
+- `coaia-narrative/rispecs/mcp_api_specification.spec.md` — tool schemas (28 tools)
+
+## Quality Criteria
+
+- ✅ JSONL output is byte-compatible with coaia-narrative output
+- ✅ All 27 coaia-narrative MCP tools produce identical results
+- ✅ Graph loads correctly from existing coaia-narrative JSONL files
+- ✅ Creative orientation validation matches coaia-narrative behavior exactly
+- ✅ STC lifecycle (create → telescope → advance → complete) is complete
+- ✅ MMOT evaluation writes findings into current reality
+- ✅ Narrative beats support all dramatic types and universe perspectives

package/rispecs/05-pde-engine.spec.md

@@ -0,0 +1,112 @@
+# 05 — PDE Engine
+
+> Prompt Decomposition Engine with PDE→STC transformation and session management.
+
+## Desired Outcome
+
+A PDE module that transforms `DecompositionResult` objects into coaia-narrative-compatible JSONL, creating Structural Tension Charts from decomposed prompts — bridging the gap between intent extraction (EAST direction) and strategic action planning (NORTH direction).
+
+## Structural Tension
+
+**Current Reality:**
+- `src/pde/` directory exists but is empty
+- PDE types fully defined in `src/types.ts`: `DecompositionResult`, `PrimaryIntent`, `SecondaryIntent`, `DirectionMap`, `ActionItem`, `AmbiguityFlag`, `StoredDecomposition`, `PdeSession`
+- coaia-pde v0.1.1 has a working implementation:
+  - `stc-mapper.ts` (~300 lines) — transforms DecompositionResult → Entity[]/Relation[]
+  - `session-manager.ts` — persists PDE sessions as JSONL
+  - `mcp-server.ts` — 12 MCP tools
+- coaia-pde rispec exists: `pde-to-stc-transformation.rispec.md` — defines the Four Directions → STC mapping
+
+**Desired Outcome:**
+PDE engine in `src/pde/` that:
+- Accepts a `DecompositionResult` from any PDE implementation
+- Transforms it into coaia-narrative-compatible JSONL entities and relations
+- Manages PDE sessions with JSONL persistence
+- Exposes 12 MCP tools
+
+## Four Directions → STC Mapping
+
+| PDE Direction | STC Component | Rationale |
+|--------------|---------------|-----------|
+| 🌅 EAST (Vision) | **Desired Outcome** | What wants to emerge = what we want to create |
+| 🔥 SOUTH (Analysis) | **Current Reality** | What must be understood = where we are now |
+| 🌊 WEST (Validation) | **Structural Tension** | What must be reflected upon = the tension |
+| ❄️ NORTH (Action) | **Action Steps** | What must be done = steps to resolve tension |
+
+### Field Mapping
+
+```
+primary.target          → STC title / desired outcome summary
+primary.action          → STC action verb
+directions.east[]       → STC desired_outcome (vision statements)
+directions.south[]      → STC current_reality (analysis statements)
+directions.west[]       → STC structural_tension (validation/reflection)
+directions.north[]      → STC action_steps (execution items)
+actionStack[]           → STC action_steps (ordered, with dependencies)
+secondary[].confidence  → STC action step priority weighting
+ambiguities[]           → STC current_reality gaps / tension sources
+```
+
+## Core API
+
+### StcMapper
+
+```typescript
+class StcMapper {
+  mapToEntities(result: DecompositionResult): { entities: Entity[], relations: Relation[] }
+  previewMapping(result: DecompositionResult): StcPreview
+}
+
+interface StcPreview {
+  desiredOutcome: string;
+  currentReality: string;
+  actionSteps: { title: string; priority: number }[];
+  tensions: string[];
+}
+```
+
+### SessionManager
+
+```typescript
+class SessionManager {
+  constructor(pdeDir: string)  // default: '.pde/'
+  createSession(decomposition: DecompositionResult): Promise<PdeSession>
+  getSession(sessionId: string): Promise<PdeSession | null>
+  listSessions(): Promise<PdeSession[]>
+  transformToStc(sessionId: string): Promise<{ chartId: string; jsonlPath: string }>
+}
+```
+
+## MCP Tools (12)
+
+| Tool | Purpose |
+|------|---------|
+| `pde_decompose` | Build prompts for LLM-driven decomposition |
+| `pde_parse_response` | Parse LLM response into DecompositionResult |
+| `pde_get` | Retrieve stored decomposition by ID |
+| `pde_list` | List stored decompositions |
+| `pde_export_markdown` | Export as git-diffable markdown |
+| `pde_to_stc` | Transform decomposition into STC |
+| `pde_preview_stc` | Preview STC mapping without creating |
+| `pde_create_session` | Create new PDE session |
+| `pde_get_session` | Get session details |
+| `pde_list_sessions` | List all sessions |
+| `pde_session_status` | Get session transformation status |
+| `pde_bridge_plan` | Bridge to planning engine |
+
+## Relation to Planning Engine
+
+PDE and planning are parallel input paths converging on narrative JSONL:
+
+```
+mcp-pde → .pde/ → coaiajs pde-engine → narrative JSONL → STC
+Claude Plan mode → plan.md → coaiajs planning-engine → narrative JSONL → STC
+```
+
+## Quality Criteria
+
+- ✅ StcMapper produces entities/relations compatible with narrative engine's JSONL format
+- ✅ All 12 coaia-pde MCP tools produce identical results
+- ✅ PDE sessions are persisted as `.pde/*.json` files
+- ✅ Preview mapping shows the transformation without side effects
+- ✅ Ambiguities from decomposition surface as tension sources in the STC

package/rispecs/06-planning-engine.spec.md

@@ -0,0 +1,128 @@
+# 06 — Planning Engine
+
+> Plan parsing and bidirectional plan↔STC synchronization.
+
+## Desired Outcome
+
+A planning module that parses markdown plans (particularly Claude Plan mode output) into `StructuralTensionPlan` objects, converts them to STC entities, and supports bidirectional sync — so changes to the plan update the STC and vice versa.
+
+## Structural Tension
+
+**Current Reality:**
+- `src/planning/` directory exists but is empty
+- Types defined in `src/types.ts`: `StructuralTensionPlan`, `StructuralElement`
+- coaia-planning v0.1.0 has a working implementation:
+  - `plan-parser.ts` (821 lines) — parses markdown plans into structured objects
+  - `tools/index.ts` — 5 MCP tools + 1 PDE bridge tool
+- No rispecs exist for coaia-planning — this is the first specification
+
+**Desired Outcome:**
+Planning engine in `src/planning/` that:
+- Parses markdown plans into `StructuralTensionPlan` objects
+- Converts plans to narrative-engine-compatible STC entities
+- Supports bidirectional plan↔STC sync
+- Exposes 6 MCP tools
+
+## Plan Format
+
+The parser handles markdown plans with this structure:
+
+```markdown
+# Plan Title
+
+## Goal
+What this plan aims to create.
+
+## Current State
+Where things stand right now.
+
+## Steps
+1. [ ] First action item
+   - Detail about first item
+2. [x] Completed item
+3. [ ] Third item
+   - Sub-detail A
+   - Sub-detail B
+
+## Notes
+Additional context.
+```
+
+## Core API
+
+### PlanParser
+
+```typescript
+class PlanParser {
+  parse(markdown: string): StructuralTensionPlan
+  parseFile(filePath: string): Promise<StructuralTensionPlan>
+}
+
+interface StructuralTensionPlan {
+  title: string;
+  desiredOutcome: string;
+  currentReality: string;
+  elements: StructuralElement[];
+  metadata?: Record<string, unknown>;
+}
+
+interface StructuralElement {
+  title: string;
+  description?: string;
+  completed: boolean;
+  subElements?: StructuralElement[];
+}
+```
+
+### PlanToStcConverter
+
+```typescript
+class PlanToStcConverter {
+  toEntities(plan: StructuralTensionPlan): { entities: Entity[], relations: Relation[] }
+  fromStc(chartId: string, graph: KnowledgeGraph): StructuralTensionPlan
+  sync(plan: StructuralTensionPlan, chartId: string, graph: KnowledgeGraph): SyncResult
+}
+
+interface SyncResult {
+  newEntities: Entity[];
+  updatedEntities: Entity[];
+  removedEntities: string[];
+  conflicts: SyncConflict[];
+}
+```
+
+## Bidirectional Sync
+
+Plan → STC:
+- Plan `Goal` maps to chart desired outcome
+- Plan `Current State` maps to chart current reality
+- Plan `Steps` map to chart action steps
+- Completed steps (`[x]`) map to completed actions
+- Sub-items map to telescoped action steps
+
+STC → Plan:
+- Chart desired outcome maps to plan `Goal`
+- Chart current reality maps to plan `Current State`
+- Action steps map to plan `Steps` with checkbox state
+- Progress observations map to sub-items
+
+Conflict resolution: STC wins by default (it is the source of truth). Conflicts are reported in `SyncResult.conflicts` for human review.
+
+## MCP Tools (6)
+
+| Tool | Purpose |
+|------|---------|
+| `plan_parse` | Parse markdown plan into structured object |
+| `plan_to_stc` | Convert parsed plan to STC entities |
+| `plan_from_stc` | Generate plan markdown from STC |
+| `plan_sync` | Bidirectional sync between plan and STC |
+| `plan_diff` | Show differences between plan and STC |
+| `plan_bridge_pde` | Import PDE decomposition as plan |
+
+## Quality Criteria
+
+- ✅ Parses Claude Plan mode output without modification
+- ✅ Checkbox state (`[ ]` / `[x]`) maps correctly to action completion
+- ✅ Sub-items preserved through parse → STC → plan round-trip
+- ✅ Bidirectional sync detects and reports conflicts
+- ✅ Generated markdown is readable and re-parseable

package/rispecs/07-pipeline-templates.spec.md

@@ -0,0 +1,97 @@
+# 07 — Pipeline Templates
+
+> Pipeline template engine with variable substitution and conditional rendering.
+
+## Desired Outcome
+
+A template engine that loads pipeline definitions, renders them with `{{variable}}` substitution, supports conditionals and built-in functions, and executes multi-step pipelines — enabling reusable agent workflows defined in YAML or JSON.
+
+## Structural Tension
+
+**Current Reality:**
+- `src/pipeline/` directory exists but is empty
+- Types defined in `src/types.ts`: `PipelineVariable`, `PipelineStep`, `PipelineTemplate`
+- coaiapy has two relevant files:
+  - `pipeline.py` — loads and executes pipeline templates
+  - `mobile_template.py` — a specific pipeline template for mobile workflows
+- Pipeline templates use `{{variable}}` Mustache-style substitution
+- No MCP tools exist for pipelines in any parent project
+
+**Desired Outcome:**
+Pipeline engine in `src/pipeline/` that:
+- Loads templates from YAML/JSON files
+- Renders templates with variable substitution
+- Supports conditionals (`{{#if condition}}...{{/if}}`)
+- Provides built-in functions (`{{now}}`, `{{uuid}}`, `{{env.VAR}}`)
+- Executes multi-step pipelines with step dependencies
+- Exposes 3 MCP tools
+
+## Template Format
+
+```yaml
+name: session-init
+description: Initialize a CoAiA agent session
+variables:
+  - name: session_id
+    required: true
+  - name: agent_name
+    default: "unnamed"
+  - name: trace_id
+    default: "{{uuid}}"
+steps:
+  - name: create-trace
+    action: langfuse.createTrace
+    params:
+      name: "{{agent_name}}-session"
+      sessionId: "{{session_id}}"
+      metadata:
+        traceId: "{{trace_id}}"
+  - name: init-redis
+    action: redis.tash
+    params:
+      key: "session:{{session_id}}"
+      value: '{"agent":"{{agent_name}}","started":"{{now}}"}'
+      ttl: 86400
+```
+
+## Core API
+
+```typescript
+class PipelineEngine {
+  loadTemplate(pathOrName: string): Promise<PipelineTemplate>
+  render(template: PipelineTemplate, variables: Record<string, string>): PipelineTemplate
+  execute(template: PipelineTemplate, variables: Record<string, string>): Promise<PipelineResult>
+  listTemplates(directory?: string): Promise<string[]>
+}
+
+interface PipelineResult {
+  success: boolean;
+  steps: { name: string; status: 'success' | 'failed' | 'skipped'; output?: unknown; error?: string }[];
+  duration: number;
+}
+```
+
+## Built-in Functions
+
+| Function | Output |
+|----------|--------|
+| `{{now}}` | ISO 8601 timestamp |
+| `{{uuid}}` | Random UUID v4 |
+| `{{env.VAR_NAME}}` | Environment variable value |
+| `{{date.YYYY-MM-DD}}` | Formatted date |
+
+## MCP Tools (3)
+
+| Tool | Purpose |
+|------|---------|
+| `pipeline_list` | List available pipeline templates |
+| `pipeline_render` | Render template with variables (preview) |
+| `pipeline_execute` | Execute pipeline with variables |
+
+## Quality Criteria
+
+- ✅ Templates load from both YAML and JSON formats
+- ✅ Missing required variables produce clear error with variable name
+- ✅ Default values applied when optional variables are absent
+- ✅ Built-in functions resolve at render time
+- ✅ Step execution is sequential; failure stops pipeline unless step is marked optional