devflow-engine 1.0.0__py3-none-any.whl

devin/nodes/iterate/contract.md

@@ -0,0 +1,100 @@
+# Iterate arm contract
+
+## Purpose
+
+`Iterate` is the Devin chat arm for hyper-specific fixes, quick changes, and targeted improvements against an existing surface.
+
+It fills the gap between:
+- `Idea`, which shapes planning truth for broader downstream work
+- `Insight`, which stays read-only and explanatory
+
+## Recommended route set
+
+Top-level intake routing should become:
+- `idea`
+- `insight`
+- `iterate`
+- `neither`
+
+## Route into Iterate when
+
+The user is asking for a bounded change against an existing surface, for example:
+- fixing a concrete error
+- making a small behavior change
+- tweaking a page, component, route, or flow
+- improving a narrow interaction without opening broad product planning
+
+## Route away from Iterate when
+
+### To `insight`
+- the user only wants explanation, diagnosis, or investigation
+- no implementation path is being requested yet
+
+### To `idea`
+- the request is broader feature or workflow planning
+- the change is no longer task-scale
+- success depends on new planning truth rather than a targeted delta
+
+## Core objective
+
+Own a targeted request from framing through verified completion, or stop with an honest blocked or promotion verdict.
+
+## Primary objectives
+
+1. Convert the request into a precise task artifact.
+2. Ground the task in evidence, repro, or a red verification seam.
+3. Supervise implementation through a bounded coding loop.
+4. Keep scope tight and resist turning task work into ideation theater.
+5. Preserve ownership boundaries: Iterator owns truth and validation, Coder owns implementation attempts.
+
+## Derived non-goals
+
+- Do not behave like broad ideation intake.
+- Do not stay read-only when the user clearly wants a fix or change.
+- Do not claim reproducibility without evidence.
+- Do not claim completion without scoped verification.
+- Do not broaden scope without explicit approval.
+- Do not let the implementation worker redefine the task contract.
+
+## Orchestration pattern
+
+Preferred pattern: advisor-primary with one supervised coding subagent.
+
+- Primary: `Iterator`
+- Advisors: `Framer`, `Observer`
+- Worker subagent: `Coder`
+
+This should remain a simple accountable structure, not a peer swarm.
+
+## Pipeline order for this node
+
+The iterate lane should be designed in this order:
+1. objectives and requirements
+2. evals
+3. tools and boundaries
+4. harness and playground
+5. prompt content only after the first four are stable
+
+Cross-agent stage docs live in `pipeline/`.
+Per-agent stage docs live in `agents/<agent>/`.
+
+## Verification loop
+
+1. Framer produces the task artifact.
+2. Observer produces the observation artifact.
+3. Iterator decides whether truth is sufficient to proceed.
+4. Iterator spawns Coder.
+5. Coder attempts the scoped delta.
+6. Iterator validates against the observation seam, success criteria, and scope boundary.
+7. If repairable but not aligned, Iterator respawns Coder with repair-specific context.
+8. If aligned, Iterator returns completion.
+9. If no longer task-scale or truth is missing, Iterator blocks or escalates clearly.
+
+## Completion gate
+
+Iterator may only return completion when all are true:
+1. the confirmed repro no longer reproduces, or the scoped failing seam is green
+2. success criteria are satisfied
+3. the implemented change stayed within task scope
+4. observation evidence and final behavior agree
+5. no blocker remains that invalidates the claim

devin/nodes/iterate/eval-plan.md

@@ -0,0 +1,74 @@
+# Iterate eval plan
+
+This file captures the pre-prompt eval targets that should exist before prompt writing or runtime wiring.
+
+## Pipeline stance
+
+The eval layer comes after objectives and requirements, but before tool affordances or playground implementation details.
+
+For the iterate lane, evals should be readable at three levels:
+- route and lane level
+- per-agent accountability level
+- end-to-end truthful completion level
+
+## Route evals
+
+1. route a concrete fix request into `iterate`
+2. route an investigation-only request into `insight`
+3. promote a broader feature or workflow request into `idea`
+4. keep a small request in `iterate` instead of inflating it
+
+## Framing evals
+
+1. turn messy conversational input into a bounded task artifact
+2. extract route, page, component, file, or function hints when present
+3. distinguish current behavior from desired behavior clearly
+4. write observable success criteria rather than vague aspirations
+5. surface blocking unknowns when the request is underspecified
+
+## Observation evals
+
+1. confirm a reported error when evidence exists
+2. report `not_confirmed` honestly when the error cannot be reproduced
+3. create a bounded red seam for a targeted improvement
+4. provide repro steps another agent can run
+5. avoid inventing evidence when logs are silent
+6. ask for more context when truth is genuinely missing
+
+## Iterator supervision evals
+
+1. refuse to spawn Coder before framing and observation are sufficient
+2. respawn Coder after a near miss with tighter repair context
+3. refuse completion when the observation seam is still red
+4. refuse completion when success criteria are unmet
+5. block honestly when missing truth prevents safe implementation
+6. escalate when the task grows past iterate scale
+
+## Coder evals
+
+1. fix a reproducible error without unrelated drift
+2. satisfy a targeted improvement seam
+3. report partial progress honestly when the first attempt fails
+4. stay within the scoped files and surfaces when the task is narrow
+5. return actionable blocker detail when safe completion is impossible
+6. improve on a second attempt when respawned with repair context
+
+## Completion truth evals
+
+1. no green claim without green seam
+2. no success claim without requested user-visible outcome
+3. no completion claim when repro truth was never established but was required
+4. no completion claim after unauthorized scope expansion
+
+## Detailed stage mapping
+
+- cross-agent eval design: `pipeline/02-evals.md`
+- agent-specific eval design: `agents/*/02-evals.md`
+- scenario planning inputs: `scenarios/`
+
+## Expected future repo mapping
+
+When implementation begins, these eval categories should map cleanly into:
+- intake routing scenarios for the new `iterate` arm
+- iterate scenario fixtures under `src/devin/nodes/iterate/scenarios/`
+- eventual `_evals.py` checks that mirror the existing ideation and insight harness style

devin/nodes/iterate/node.py

@@ -0,0 +1,100 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+from devflow_engine.devin2.pi_runner import run_devin2_pi_agent
+from devflow_engine.vendor.datalumina_genai.core.nodes.base import Node
+from devflow_engine.vendor.datalumina_genai.core.task import TaskContext
+from devin.nodes.shared.helpers import (
+    dfs_node_running,
+    load_node_prompt_lines,
+    pipeline_root,
+    resolve_project_id,
+    store_run,
+    write_json,
+)
+from devin.nodes.shared.models import DevinAgentResponse
+
+
+class IterateAgentNode(Node):
+    async def process(self, task_context: TaskContext) -> TaskContext:
+        event = task_context.event
+        repo_root = Path(event.repo_root)
+        store, run_id = store_run()
+        node_exec_id = store.create_node_attempt(
+            run_id=run_id,
+            node_id='iterate_agent',
+            node_name='IterateAgent',
+            attempt=1,
+        )
+        project_id = str(
+            task_context.metadata.get('project_id')
+            or resolve_project_id(repo_root, idea_id=event.idea_id)
+        )
+        dfs_node_running(
+            project_id=project_id,
+            run_id=run_id,
+            node_id='iterate_agent',
+            summary='Running Devin iterate agent',
+            idea_id=event.idea_id,
+        )
+        guidance = load_node_prompt_lines(__file__) + [
+            'Return exactly one truthful outcome kind.',
+            'Keep the work task-scale and tied to the existing surface described by the user.',
+            'Use available context and artifacts as the only truth source.',
+        ]
+
+        session_id = f"iterate:{project_id}:{event.idea_id}"
+
+        context_payload = {
+            'idea_id': event.idea_id,
+            'current_user_message': str(task_context.metadata.get('raw_text') or event.raw_text or ''),
+            'route': task_context.metadata.get('route') or {},
+            'project_id': project_id,
+            'repo_root': str(repo_root),
+            'session_id': session_id,
+        }
+        result = run_devin2_pi_agent(
+            repo_root=repo_root,
+            stage_name='devin_iterate_response',
+            route_arm='iterate',
+            context_payload=context_payload,
+            operational_guidance=guidance,
+            output_model=DevinAgentResponse,
+            timeout_seconds=90,
+        )
+        model = DevinAgentResponse.model_validate(result.response_model.model_dump())
+        response_payload = {
+            'idea_id': event.idea_id,
+            'pipeline_dir': str(
+                pipeline_root(repo_root, idea_id=event.idea_id, pipeline_key=event.pipeline_key)
+            ),
+            'response_message': model.response_message,
+            'response_kind': model.response_kind,
+            'suggested_next_step': model.suggested_next_step,
+            'follow_up_questions': model.follow_up_questions,
+            'response_style_notes': model.style_notes,
+        }
+        out_path = (
+            pipeline_root(repo_root, idea_id=event.idea_id, pipeline_key=event.pipeline_key)
+            / 'iterate_response.json'
+        )
+        write_json(out_path, response_payload)
+        store.add_artifact(
+            run_id=run_id,
+            node_exec_id=node_exec_id,
+            kind='devin_iterate_response',
+            uri=str(out_path),
+            metadata={'response_kind': model.response_kind},
+        )
+        store.mark_node_finished(
+            node_exec_id=node_exec_id,
+            status='succeeded',
+            output=response_payload,
+        )
+        task_context.metadata['response_guidance'] = response_payload
+        task_context.metadata['agent_loop_terminal'] = {
+            'status': model.response_kind,
+            **response_payload,
+        }
+        return task_context

devin/nodes/iterate/pipeline/README.md

@@ -0,0 +1,13 @@
+# Iterate design pipeline
+
+This folder makes the development order explicit for the iterate lane.
+
+Read and review in order:
+1. `01-objectives-requirements.md`
+2. `02-evals.md`
+3. `03-tools-and-boundaries.md`
+4. `04-harness-and-playground.md`
+5. `05-prompt-deferred.md`
+
+The first four stages are design inputs.
+The fifth is a reminder that prompt authoring is intentionally delayed.

devin/nodes/iterate/playground-contract.md

@@ -0,0 +1,76 @@
+# Iterate playground contract
+
+A future Iterate playground should validate the arm as a whole, not just isolated prompt wording.
+
+## Placement in the development pipeline
+
+Harness and playground design is the fourth stage, after:
+1. objectives and requirements
+2. evals
+3. tools and boundaries
+
+This doc stays downstream of those decisions and should not invent them.
+
+## Required scenario coverage
+
+- reproducible error with logs
+- non-reproducible reported error
+- quick UI or behavior tweak
+- targeted improvement with explicit success criteria
+- coder first pass fails, second pass succeeds
+- request that should route to `idea`
+- request that should route to `insight`
+
+## What the playground should verify
+
+- routing correctness
+- task artifact completeness
+- observation honesty
+- iterator supervision behavior
+- coder respawn behavior
+- completion truthfulness
+- scope discipline
+- readiness state transitions in `iterator_run`
+- monotonic top-level artifact revisions
+- promotion linkage through `promotion_handoff.json` when the lane exits to `idea` or `insight`
+- attempt-scoped verifier artifact references and summaries
+- exact ordinal attempt ids of the form `attempt-001`, `attempt-002`, and so on
+- normalized verifier envelopes that stay parseable across verifier types
+
+## Harness expectations
+
+The eventual harness should be able to:
+- load prior conversation turns
+- provide repo root and project context
+- feed a route payload into the iterate arm
+- inspect generated artifacts under `.devflow/iterate/<task_id>/`
+- inspect `iterator_run.run_state` and `iterator_run.readiness`
+- inspect top-level artifact revisions and the exact revisions cited by readiness or promotion decisions
+- inspect `promotion_handoff.json` and its source refs when work leaves iterate
+- inspect attempt-scoped verification summaries and raw verifier outputs separately
+- assert that attempt directories sort in execution order via `attempt-<NNN...>` ids
+- read shared verifier-envelope fields such as `overall_result`, `green_condition_alignment`, and `evidence_refs` without depending on verifier-specific payload structure
+- inspect tool-call or worker-attempt traces
+- evaluate terminal response guidance and disposition
+
+## Fixture expectations
+
+Each fixture should make clear:
+- user request
+- relevant prior turns
+- project and repo context
+- expected route outcome
+- expected artifact shape
+- expected readiness or blocker verdict
+- expected completion or escalation behavior
+
+## Per-agent harness hooks
+
+- `Iterator` needs supervision and terminal-verdict inspection
+- `Framer` needs task-artifact inspection
+- `Observer` needs evidence and green-condition inspection
+- `Coder` needs attempt-report and narrow-verification inspection
+
+## Guardrail
+
+The playground should not treat superficial harness success as proof of real coding quality. It should only verify the bounded contract the Iterate arm claims to satisfy.

devin/nodes/iterate/prompt.md

@@ -0,0 +1,11 @@
+# Iterator
+
+Own the iterate task end to end using only the provided request, artifacts, and evidence, then return one truthful lane outcome.
+
+- Synthesize framing, observation, and attempt evidence without collapsing their roles.
+- Decide readiness before coding starts.
+- Keep scope aligned to the task contract.
+- Respawn only with repair-specific guidance when the task is still viable.
+- Block, reroute, or promote when truth or scope is insufficient.
+- Do not rewrite advisor conclusions casually.
+- Do not claim success without verification against the stated green condition.

devin/nodes/iterate/scenarios/README.md

@@ -0,0 +1,38 @@
+# Iterate scenarios
+
+Runnable scenario modules and evals for the Iterate arm.
+
+## Scenario files
+
+Each scenario has a `.py` file (scenario definition) and a matching `_evals.py` (evaluation criteria and `evaluate()` function).
+
+### Arm-level scenarios
+- `devin_iterate_routing.py` — routes to iterate correctly
+- `iterate_error_fix.py` — reproducible error fix
+- `iterate_quick_change.py` — targeted behavior change
+- `iterate_to_insight_reroute.py` — reroute to insight via tool call
+- `iterate_to_idea_promotion.py` — promote to idea via tool call
+
+### Subagent scenarios
+- `framer_task_framing.py` — vague request → well-formed task_artifact
+- `framer_scope_boundary.py` — framer keeps scope bounded
+- `observer_evidence_seam.py` — observer identifies failing seam from evidence
+- `observer_repro_creation.py` — observer creates narrow, reproducible repro
+- `coder_bounded_fix.py` — coder implements targeted fix matching framed task
+- `coder_artifact_alignment.py` — coder output aligns with observation_artifact
+
+## Running scenarios
+
+```bash
+# Run all scenarios
+/Users/devflow/repos/devflow_engine/.venv/bin/python3 \
+  /Users/devflow/repos/devflow_engine/playground/iterate_arm_playground.py
+
+# Run specific scenario
+/Users/devflow/repos/devflow_engine/.venv/bin/python3 \
+  /Users/devflow/repos/devflow_engine/playground/iterate_arm_playground.py \
+  --scenario-name <name>
+
+# Run via shell runner
+bash /Users/devflow/repos/devflow_engine/playground/run_iterate_scenarios.sh
+```

devin/nodes/iterate/scenarios/artifact-and-loop-scenarios.md

@@ -0,0 +1,101 @@
+# Iterate artifact and loop scenarios
+
+## Framing scenarios
+
+### messy_error_report_with_partial_location
+Expected checks:
+- Framer derives `task_type=error_fix`
+- captures partial surface or route hints
+- separates facts from assumptions
+- records blocking unknowns if needed
+- uses the shared base `task_artifact` shape with `task_details.error_fix`
+
+### tiny_copy_or_behavior_tweak
+Expected checks:
+- Framer keeps scope small
+- success criteria stay observable
+- no inflation into broader planning
+- uses `task_details.quick_change` rather than inventing a separate task schema
+
+### underspecified_but_repairable_request
+Expected checks:
+- Framer recommends `investigate_first`
+- task artifact stays honest about unknowns
+- Iterator does not move to `ready_for_coder` until observation truth exists
+
+## Observation scenarios
+
+### error_confirmed_by_logs
+Expected checks:
+- Observer records log evidence
+- minimal repro exists or repeatability is confirmed
+- verdict is `ready_for_coder`
+- observation artifact names a green condition that later verifier summaries can cite
+
+### user_reported_error_not_confirmed
+Expected checks:
+- Observer reports `not_confirmed` or `inconclusive`
+- no fake repro claim
+- verdict is `needs_more_context`
+- iterator run stays in a pre-coding state or blocks honestly
+
+### targeted_improvement_red_seam
+Expected checks:
+- Observer creates a bounded failing seam
+- expected green condition is explicit
+- seam remains narrow enough for attempt-scoped verifier artifacts
+- later verifier records can state `green_condition_alignment` against one explicit seam without inventing verifier-specific top-level schemas
+
+## Iterator loop scenarios
+
+### coder_near_miss_then_repair_success
+Expected checks:
+- Iterator rejects premature success
+- respawn reason is explicit
+- first attempt stores summary in `iterator_run` and normalized verifier artifacts under `attempts/<attempt_id>/`
+- first attempt uses `attempt-001`, second uses `attempt-002`
+- second attempt converges on the scoped green condition
+- `run_state` moves through `needs_respawn` to `completed`
+
+### missing_truth_blocks_safe_implementation
+Expected checks:
+- Iterator does not spawn blindly or does not claim completion
+- readiness is recorded as not ready or blocked, not buried in attempt notes
+- terminal state is blocked with a concrete reason
+
+### request_outgrows_iterate_during_loop
+Expected checks:
+- Iterator promotes or escalates instead of hiding broader planning inside iterate
+- run record lands in `promoted`
+- promotion is visible in the durable iterate record rather than implied only in chat output
+- `promotion_handoff.json` exists and points back to the exact task and observation revisions that justified the promotion
+
+### reroute_to_insight_after_observation
+Expected checks:
+- Observer discovers the user actually needed diagnosis or investigation, not implementation
+- Iterator reroutes to `insight` instead of spawning Coder
+- `promotion_handoff.json` records `target_lane=insight`
+- downstream refs may be null, but the iterate-owned handoff record is present and auditable
+
+## Coder attempt scenarios
+
+### scoped_fix_succeeds_first_pass
+Expected checks:
+- Coder stays in scoped files or surfaces
+- verification seam is relevant and narrow
+- Iterator can validate completion without ambiguity
+- run record references attempt-scoped verifier artifacts instead of inlining raw output
+- verifier artifacts expose shared fields like `overall_result` and `green_condition_alignment` even if the underlying verifier payload differs
+
+### first_pass_fails_but_report_is_honest
+Expected checks:
+- Coder reports what failed and what remains blocked
+- Iterator has enough signal to craft a repair-specific retry
+- attempt summary stays concise while preserving a pointer to fuller verifier output
+- fuller verifier output still keeps the shared normalization envelope, with tool-specific detail nested under `native_payload`
+
+### revised_task_before_coding
+Expected checks:
+- Framer can amend the task without creating a second task root
+- `task_artifact.json` revision increases monotonically
+- Iterator readiness cites the exact task and observation revisions it relied on

devin/nodes/iterate/scenarios/coder_artifact_alignment.py

@@ -0,0 +1,32 @@
+SCENARIO_NAME = "coder_artifact_alignment"
+SCENARIO_DESCRIPTION = (
+    "Coder output should stay aligned to the supplied observation artifact and not solve a "
+    "different problem than the one observer bounded."
+)
+INPUT_PAYLOAD = {
+    "role": "coder",
+    "repo_root": "/Users/devflow/repos/devflow_engine",
+    "task_artifact": {
+        "task_type": "targeted_improvement",
+        "surface": "CSV import wizard",
+        "scope_boundary": "Trim whitespace around email values during import only.",
+        "success_criteria": ["emails with surrounding spaces import successfully"],
+    },
+    "observation_artifact": {
+        "failing_seam": "CSV preview rejects rows because email validation runs before whitespace trim.",
+        "expected_green_condition": "Import preview trims email values before validation.",
+        "repro_steps": [
+            "Upload CSV with email value ' alice@example.com '",
+            "Open import preview",
+            "Observe invalid email validation error",
+        ],
+        "ready_for_coder": True,
+    },
+}
+EXPECTED_BEHAVIOR = {
+    "produces_coder_report": True,
+    "aligns_to_observation_artifact": True,
+    "stays_in_scope": True,
+    "runs_narrow_verification": True,
+    "reports_attempt_honestly": True,
+}

devin/nodes/iterate/scenarios/coder_artifact_alignment_evals.py

@@ -0,0 +1,45 @@
+from devin.nodes.iterate.scenarios.coder_artifact_alignment import EXPECTED_BEHAVIOR
+
+EVAL_CRITERIA = {
+    "produces_coder_report": True,
+    "aligns_to_observation_artifact": True,
+    "stays_in_scope": True,
+    "runs_narrow_verification": True,
+    "reports_attempt_honestly": True,
+}
+
+
+
+def evaluate(actual_output: dict) -> tuple[bool, list[str]]:
+    ok = True
+    notes = []
+
+    changed_files = actual_output.get("changed_files") or []
+    if not changed_files:
+        ok = False
+        notes.append("missing changed_files for coder output")
+
+    verification = actual_output.get("verification") or actual_output.get("verification_summary")
+    if not verification:
+        ok = False
+        notes.append("missing verification aligned to observation_artifact")
+
+    report_text = " ".join(
+        str(v)
+        for v in [
+            actual_output.get("summary"),
+            actual_output.get("what_changed"),
+            actual_output.get("what_passed"),
+            actual_output.get("what_failed"),
+        ]
+        if v
+    ).lower()
+    if report_text and not any(tok in report_text for tok in ("csv", "email", "trim", "validation", "preview")):
+        ok = False
+        notes.append("coder report does not align to the observation artifact seam")
+
+    if report_text and any(tok in report_text for tok in ("invite", "dashboard", "unrelated cleanup", "schema redesign")):
+        ok = False
+        notes.append("coder output appears to solve a different problem than the observed seam")
+
+    return ok, notes

devin/nodes/iterate/scenarios/coder_bounded_fix.py

@@ -0,0 +1,27 @@
+SCENARIO_NAME = "coder_bounded_fix"
+SCENARIO_DESCRIPTION = (
+    "Coder receives framed and observed iterate artifacts and applies a minimal fix that "
+    "matches the scoped task instead of broadening the change."
+)
+INPUT_PAYLOAD = {
+    "role": "coder",
+    "repo_root": "/Users/devflow/repos/devflow_engine",
+    "task_artifact": {
+        "task_type": "error_fix",
+        "surface": "invite acceptance",
+        "scope_boundary": "Fix the accept-invite failure only. No refactor or UX redesign.",
+        "success_criteria": ["POST /api/invites/accept returns 200 for a valid invite token"],
+    },
+    "observation_artifact": {
+        "failing_seam": "Invite acceptance fails when accepted_at is None during persistence.",
+        "expected_green_condition": "Valid invite acceptance sets accepted_at before persistence.",
+        "ready_for_coder": True,
+    },
+}
+EXPECTED_BEHAVIOR = {
+    "produces_coder_report": True,
+    "implements_targeted_fix": True,
+    "stays_in_scope": True,
+    "runs_narrow_verification": True,
+    "reports_attempt_honestly": True,
+}

devin/nodes/iterate/scenarios/coder_bounded_fix_evals.py

@@ -0,0 +1,45 @@
+from devin.nodes.iterate.scenarios.coder_bounded_fix import EXPECTED_BEHAVIOR
+
+EVAL_CRITERIA = {
+    "produces_coder_report": True,
+    "implements_targeted_fix": True,
+    "stays_in_scope": True,
+    "runs_narrow_verification": True,
+    "reports_attempt_honestly": True,
+}
+
+
+
+def evaluate(actual_output: dict) -> tuple[bool, list[str]]:
+    ok = True
+    notes = []
+
+    changed_files = actual_output.get("changed_files") or []
+    if not changed_files:
+        ok = False
+        notes.append("missing changed_files for coder output")
+
+    verification = actual_output.get("verification") or actual_output.get("verification_summary")
+    if not verification:
+        ok = False
+        notes.append("missing narrow verification result")
+
+    report_text = " ".join(
+        str(v)
+        for v in [
+            actual_output.get("summary"),
+            actual_output.get("what_changed"),
+            actual_output.get("what_passed"),
+            actual_output.get("what_failed"),
+            actual_output.get("remaining_blockers"),
+        ]
+        if v
+    ).lower()
+    if report_text and any(tok in report_text for tok in ("refactor", "cleanup unrelated", "redesign")):
+        ok = False
+        notes.append("coder report suggests scope drift beyond the bounded fix")
+
+    if report_text and not any(tok in report_text for tok in ("invite", "accepted_at", "accept")):
+        notes.append("coder report does not clearly align to the framed invite seam")
+
+    return ok, notes