devflow-engine 1.1.16__tar.gz → 1.1.18__tar.gz

devflow_engine-1.1.18/DESIGN_NOTES.md

@@ -0,0 +1,145 @@
+# DevFlow Error-Solver Hardening — Design Notes
+
+Track A landed in `v1.1.18`:
+- `errors/repo_quality_judge.py` — pure-LLM judge (medium tier → MiniMax) scoring
+  repro artifacts on `behavioral_vs_presence` and `fails_for_described_reason`.
+- `BuildObservabilityNode` runs the judge after the observation pass and before
+  `FirstPassFix`. Unhealthy judgments flip `error_tasks.quality_status` to
+  `awaiting_quality_review` and stop the workflow with `status='observed'`.
+- `ResolveNode` enforces the red-before-fix invariant: `verified =
+  green_gate AND observation_was_red`. Closes the "already-green-at-intake" leak
+  (`src/devflow_engine/errors/error_solver_dag.py` ResolveNode).
+- Schema: added `error_tasks.quality_status TEXT` via
+  `ExecutionStore._ensure_error_tasks_schema` (in-code migration, no Alembic).
+
+The rest of this document is the queue for the NEXT vertical slices. Each
+slice is intentionally small enough to land in one PR.
+
+---
+
+## Slice 1 — Playwright observe-time spec generation + visual-artifact capture
+
+**Why:** The repo-quality judge catches presence-only repros at intake, but it
+doesn't yet generate the behavioral repro itself. For UI bugs the observability
+worker should emit a Playwright spec (not just a pytest grep script) plus a
+screenshot/video so the portal can render visual evidence to the submitter and
+the fix worker.
+
+**Where to start:**
+- Sources of truth on Playwright discipline live at:
+  - `/Volumes/devflow/.openclaw/workspace/docs/how_to_playwright_right.md`
+  - `/Volumes/devflow/.openclaw/workspace/docs/PLAYWRITE-E2E-PROCESS.md`
+- Add a sibling to `src/devflow_engine/errors/user_report_observability.py`:
+  `user_report_observability_playwright.py`. Reuse the same agentic pi+tools
+  scaffold but emit `repro.spec.ts` + `artifacts/{screenshot.png,trace.zip}`.
+- BuildObservabilityNode already scopes the sandbox to
+  `.devflow/observability/{etid}/`; mirror that path layout for the new artifact set.
+- Wire selection (pytest vs. Playwright) through `report_context.surface` — when
+  the user-report intake came from a frontend route, prefer Playwright.
+- The repo-quality judge keys off the same payload shape (`repro_artifact.path` +
+  `repro_artifact.source`) — pass the `.spec.ts` source through unchanged.
+
+**Acceptance:**
+- A user-report against a route renders a `.spec.ts` + screenshot + trace.
+- RepoQualityJudge sees `behavioral` for a spec that exercises the page and
+  asserts on visible DOM/text, `presence` for a spec that only asserts a
+  selector exists in the rendered HTML.
+
+---
+
+## Slice 2 — Bug-portal routes `/bugs/{token}` on devflow-host
+
+**Why:** Submitters need to see (and respond to) the analyst's observation +
+repro + quality-judgment without an account. Mirrors the clarify magic-link
+pattern that's already in production for clarification questions.
+
+**Where to start:**
+- Mirror the controller pattern at
+  `controller/api/clarify.py` (in the devflow-host repo, not engine).
+- Token mechanism: reuse `clarification_links.create_link()` —
+  same TTL/rotation semantics already approved.
+- Engine side: add a `create_bug_portal_link(error_task_id)` helper next to
+  the existing clarification-link helper. New rows in the existing
+  `clarification_links` table with a discriminator column, OR a new
+  `bug_portal_links` table — prefer the former to keep the token rotation
+  audit in one place.
+- Portal routes (devflow-host):
+  - `GET /bugs/{token}` — render observation summary, repro source, judgment.
+  - `POST /bugs/{token}/request-changes` — see Slice 3.
+  - `POST /bugs/{token}/approve` — see Slice 4.
+
+**Acceptance:**
+- A user-report row with `quality_status='awaiting_quality_review'` exposes a
+  portal URL in `error_tasks` (new column `portal_url` or derived).
+- Submitter sees the repro and the judge's rationale; can act on it.
+
+---
+
+## Slice 3 — Modification-request → linked-new error_task
+
+**Why:** When a submitter clicks "request changes" on the portal, that's new
+information the analyst doesn't have yet. Spawn a new `error_tasks` row linked
+back to the original so the analyst pool re-claims it cleanly without losing
+the original audit trail.
+
+**Where to start:**
+- New schema column: `error_tasks.parent_error_task_id TEXT` (NULL for roots).
+  Add to `ExecutionStore._ensure_error_tasks_schema` required list. Index on it
+  for `WHERE parent_error_task_id = ?` lookups.
+- Engine helper: `ExecutionStore.create_modification_request(parent_etid,
+  message)` — inserts a child row with `source_kind='user_report'`,
+  `status='open'`, copies `project_id` + `reporter_*` + `source_app` from
+  parent, message = submitter's modification request, fingerprint derived
+  from `parent_etid + nonce` to bypass dedupe.
+- Portal handler `POST /bugs/{token}/request-changes` calls that helper.
+- Worker claim filter (`stores/execution_store.py` ~ line 2461 / 2478) does
+  NOT need changes — child rows are plain `open` user_report rows.
+
+**Acceptance:**
+- Submitter modification creates a sibling row visible to the analyst with a
+  link back to the original via `parent_error_task_id`.
+- Original row's `quality_status` remains; child gets its own DAG run.
+
+---
+
+## Slice 4 — Submitter-approval → staging push integration
+
+**Why:** After RepoQualityJudge + green-gate + observation_was_red all pass,
+the resolution commit lives in the worker's local repo but doesn't reach
+staging until a human approves. The portal `approve` button should be the
+trigger.
+
+**Where to start:**
+- The commit is already created by `_commit_resolution` in
+  `errors/error_solver_dag.py` (lines 179–224). The SHA is persisted to
+  `task_context.metadata["commit_result"]`. Extend ResolveNode to also write
+  the SHA into `error_tasks.resolution_commit_sha TEXT` (new column).
+- Portal handler `POST /bugs/{token}/approve`:
+  1. Look up `resolution_commit_sha` on the row.
+  2. `git push origin HEAD:refs/heads/staging-<etid>` from the worker's repo
+     clone (engine-side helper: `push_resolution_to_staging(etid)`).
+  3. Optionally open a PR via `gh pr create` against the project's main.
+- New `error_tasks.staging_status TEXT` column to track
+  `unpushed | pushed | pr_open | merged | rejected`.
+
+**Acceptance:**
+- Approval pushes exactly the resolution commit (single-commit branch) to
+  staging.
+- Rejection leaves the commit local and flips
+  `quality_status='awaiting_quality_review'` so a fresh analyst pass can
+  iterate.
+
+---
+
+## Anchors quick-reference
+
+- Engine DAG entrypoint: `src/devflow_engine/errors/error_solver_dag.py`
+  - `BuildObservabilityNode.process` — observation + RepoQualityJudge
+  - `ResolveNode.process` — red-before-fix invariant + commit
+  - `_commit_resolution` — atomic per-error commit (existing)
+- Schema migrations: `src/devflow_engine/stores/execution_store.py`
+  `_ensure_error_tasks_schema` (~ line 611)
+- Runner: `ExecutionStore.run_error_solver_dag` (~ line 1953); add new
+  `repo_quality_judge_fn` kwarg already present.
+- Judge: `src/devflow_engine/errors/repo_quality_judge.py`
+- Tests: `tests/area8_errors/test_us85_error_solver_repo_quality.py`

{devflow_engine-1.1.16 → devflow_engine-1.1.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: devflow-engine
-Version: 1.1.16
+Version: 1.1.18
 Summary: DevFlow v2 CLI-first engine (projects, stories, execution store, review packets)
 Author: DevFlow
 License: Proprietary

{devflow_engine-1.1.16 → devflow_engine-1.1.18}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "devflow-engine"
-version = "1.1.16"
+version = "1.1.18"
 description = "DevFlow v2 CLI-first engine (projects, stories, execution store, review packets)"
 readme = "README.md"
 requires-python = ">=3.11"

{devflow_engine-1.1.16 → devflow_engine-1.1.18}/src/devflow_engine/errors/error_solver_dag.py

@@ -130,6 +130,100 @@ def _read_recent_journal(repo_root: Path, *, limit_files: int = 3, max_chars: in
     return {"entries": entries, "summary": summary[-max_chars:]}
 
 
+class WorkspaceDirtyError(RuntimeError):
+    """Raised when a run is asked to start against a non-clean working tree."""
+
+
+_ENGINE_INTERNAL_PREFIXES = (".devflow/",)
+
+
+def _is_source_path(path: str) -> bool:
+    if path.startswith(_ENGINE_INTERNAL_PREFIXES):
+        return False
+    # .pyc and __pycache__/ are regenerated by the running backend
+    # and never belong in a fix commit.
+    if path.endswith(".pyc") or "__pycache__/" in path:
+        return False
+    return True
+
+
+def _git_dirty_paths(repo_root: Path) -> list[str]:
+    cp = subprocess.run(
+        ["git", "-C", str(repo_root), "status", "--porcelain"],
+        text=True,
+        capture_output=True,
+        check=False,
+    )
+    if cp.returncode != 0:
+        return []
+    raw = [line[3:] for line in (cp.stdout or "").splitlines() if line.strip()]
+    return [p for p in raw if _is_source_path(p)]
+
+
+def _assert_clean_workspace(repo_root: Path, *, error_task_id: str) -> None:
+    """Single-worker invariant: each error-solver run must start clean.
+
+    Why: runs commit their own atomic patch at conclusion (see
+    _commit_resolution). If a run starts with leftover edits from a
+    previous run, the resulting commit collapses multiple errors into
+    one and breaks per-error attribution, cherry-pick, and audit.
+    """
+    dirty = _git_dirty_paths(repo_root)
+    if dirty:
+        raise WorkspaceDirtyError(
+            f"error_task {error_task_id}: refusing to start, repo {repo_root} "
+            f"has {len(dirty)} uncommitted path(s): {dirty[:5]}{'…' if len(dirty) > 5 else ''}"
+        )
+
+
+def _commit_resolution(
+    repo_root: Path,
+    *,
+    error_task_id: str,
+    title: str | None,
+    verification: dict[str, Any] | None,
+) -> dict[str, Any]:
+    """Commit working-tree edits made by this run, exclusively.
+
+    Stages explicit paths reported by `git status --porcelain` (never
+    `git add -A` — see project rule). Returns a structured outcome so
+    the journal/portal can link to the commit SHA.
+    """
+    paths = _git_dirty_paths(repo_root)
+    if not paths:
+        return {"committed": False, "reason": "no_changes", "paths": []}
+
+    add = subprocess.run(
+        ["git", "-C", str(repo_root), "add", "--", *paths],
+        text=True, capture_output=True, check=False,
+    )
+    if add.returncode != 0:
+        return {"committed": False, "reason": "add_failed", "stderr": (add.stderr or "")[-1000:]}
+
+    repro_cmd = ""
+    if isinstance(verification, dict):
+        repro_cmd = str(verification.get("repro_run_command") or "")
+    safe_title = (title or "error fix").splitlines()[0][:72]
+    message = (
+        f"fix(error): {safe_title}\n\n"
+        f"error_task_id: {error_task_id}\n"
+        f"resolved_at: {_utc_now_iso()}\n"
+        f"verification: {repro_cmd[:200]}\n"
+    )
+    commit = subprocess.run(
+        ["git", "-C", str(repo_root), "commit", "-m", message, "--no-verify"],
+        text=True, capture_output=True, check=False,
+    )
+    if commit.returncode != 0:
+        return {"committed": False, "reason": "commit_failed", "stderr": (commit.stderr or "")[-1000:]}
+
+    sha = subprocess.run(
+        ["git", "-C", str(repo_root), "rev-parse", "HEAD"],
+        text=True, capture_output=True, check=False,
+    ).stdout.strip()
+    return {"committed": True, "sha": sha, "paths": paths}
+
+
 def _git_changes_since(repo_root: Path, *, unix_ts: int | None) -> dict[str, Any]:
     if not unix_ts:
         return {"summary": "", "commits": []}
@@ -262,6 +356,10 @@ class BuildObservabilityNode(Node):
         repo_root = Path(task_context.event.repo_root)
         error_task_id = task_context.event.error_task_id
 
+        # Single-worker invariant: each run starts clean so its
+        # post-resolve commit isolates this error's changes.
+        _assert_clean_workspace(repo_root, error_task_id=error_task_id)
+
         # Ensure we have a persistor + base context
         persistor: ContextPersistor = task_context.metadata.setdefault(
             "context_persistor", ContextPersistor(repo_root=repo_root, error_task_id=error_task_id)
@@ -423,10 +521,65 @@ class BuildObservabilityNode(Node):
             # - None / legacy → gate-aware stop (pre-1.1.15 behavior).
             if worker_role == "analyst":
                 task_context.stop_workflow()
+                return task_context
             elif worker_role == "fix":
                 pass  # always continue
             elif not gates_passed:
                 task_context.stop_workflow()
+                return task_context
+
+            # Track A: repo-quality LLM judge — pure-LLM gate that runs AFTER
+            # the observation pass and BEFORE FirstPassFix. Closes the leak
+            # where a repro asserts on symbol presence (greppy "found X: True")
+            # rather than observable behavior. The fix worker also runs this
+            # gate — analyst stops above, so we only reach here on roles
+            # {None, "fix"} that are about to advance to FirstPassFix.
+            if gates_passed:
+                judge_fn = funcs.get("repo_quality_judge_fn") if isinstance(funcs, dict) else None
+                if callable(judge_fn):
+                    try:
+                        judgment = judge_fn(context) or {}
+                    except Exception as exc:  # judge transport failure → safe stop
+                        judgment = {
+                            "behavioral_vs_presence": "ambiguous",
+                            "fails_for_described_reason": "unclear",
+                            "rationale": "repo_quality_judge raised",
+                            "judge_ok": False,
+                            "judge_error": str(exc)[:500],
+                        }
+                    persistor.persist(context=context, node_name="RepoQualityJudge", output=judgment)
+
+                    # Persist the judgment to the canonical observability
+                    # sandbox path so the dashboard / portal can render it
+                    # alongside the repro artifact.
+                    quality_dir = repo_root / ".devflow" / "observability" / error_task_id
+                    quality_dir.mkdir(parents=True, exist_ok=True)
+                    (quality_dir / "repo_quality.json").write_text(
+                        json.dumps(judgment, indent=2, sort_keys=True) + "\n",
+                        encoding="utf-8",
+                    )
+
+                    healthy = (
+                        bool(judgment.get("judge_ok", True))
+                        and judgment.get("behavioral_vs_presence") == "behavioral"
+                        and judgment.get("fails_for_described_reason") == "yes"
+                    )
+                    if not healthy:
+                        with store._connect() as conn:
+                            conn.execute(
+                                "UPDATE error_tasks SET quality_status=?, updated_at=? "
+                                "WHERE error_task_id=?",
+                                ("awaiting_quality_review", int(time.time()), error_task_id),
+                            )
+                        task_context.metadata["outcome"] = "awaiting_quality_review"
+                        task_context.metadata["reason"] = (
+                            f"repo_quality: behavioral_vs_presence="
+                            f"{judgment.get('behavioral_vs_presence')!r}, "
+                            f"fails_for_described_reason="
+                            f"{judgment.get('fails_for_described_reason')!r}"
+                        )
+                        task_context.stop_workflow()
+                        return task_context
 
         return task_context
 
@@ -515,7 +668,16 @@ class ResolveNode(Node):
         error_task_id = task_context.event.error_task_id
 
         verification = task_context.metadata.get("verification")
-        verified = bool((verification or {}).get("green_gate"))
+        # Red-before-fix invariant: ResolveNode refuses to mark resolved
+        # unless BuildObservability persisted observable=True at intake.
+        # Closes the "already-green-at-intake" leak — a green-gate True with
+        # no red baseline proves nothing.
+        observation_was_red = False
+        bo = context.get("BuildObservability") if isinstance(context.get("BuildObservability"), dict) else None
+        if isinstance(bo, dict):
+            bo_out = bo.get("output") if isinstance(bo.get("output"), dict) else {}
+            observation_was_red = bool(bo_out.get("observable"))
+        verified = bool((verification or {}).get("green_gate")) and observation_was_red
         runtime_context = context.get("runtimeContext") if isinstance(context.get("runtimeContext"), dict) else None
         prior_runs = context.get("priorRuns") if isinstance(context.get("priorRuns"), dict) else None
         fingerprint = None
@@ -528,6 +690,13 @@ class ResolveNode(Node):
         self.save_output(ResolveNode.OutputType(verified=verified, verification=verification))
 
         if verified:
+            commit_result = _commit_resolution(
+                persistor.repo_root,
+                error_task_id=error_task_id,
+                title=str(title) if title is not None else None,
+                verification=verification,
+            )
+            task_context.metadata["commit_result"] = commit_result
             with store._connect() as conn:
                 conn.execute(
                     "UPDATE error_tasks SET status='resolved', updated_at=? WHERE error_task_id=?",
@@ -543,7 +712,10 @@ class ResolveNode(Node):
                     (int(time.time()), error_task_id),
                 )
             task_context.metadata["outcome"] = "in_progress"
-            task_context.metadata["reason"] = "verification_failed"
+            if not observation_was_red:
+                task_context.metadata["reason"] = "observation_was_not_red_at_intake"
+            else:
+                task_context.metadata["reason"] = "verification_failed"
 
         _append_error_solver_journal(
             repo_root=persistor.repo_root,

devflow_engine-1.1.18/src/devflow_engine/errors/repo_quality_judge.py

@@ -0,0 +1,256 @@
+"""Repro-quality LLM judge for the error-solver DAG (Track A).
+
+Scores a reproduction artifact along two dimensions BEFORE FirstPassFix runs:
+
+  - behavioral_vs_presence: does the repro assert observable behavior, or
+    does it merely check that a symbol exists in the codebase?
+  - fails_for_described_reason: does the failure produced by the repro
+    match the bug-report symptom described by the user?
+
+Failure mode in the wild: LLM-authored repros that grep for a string
+("last_login_at found in User model: True") pass the boolean green-gate
+oracle while proving nothing about the actual bug. This judge catches
+those cases at intake.
+
+The judge is PURE LLM — no regex / heuristic fallback. Per project rule
+(feedback_no_regex_first): never propose regex-first content classification.
+The configured medium-tier provider (MiniMax) is the source of truth
+(feedback_no_provider_swap_suggestions).
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, Literal
+
+from pydantic import BaseModel, Field, ValidationError
+
+from ..llm.invoke import LlmInvocationRequest, invoke_llm
+
+BehavioralVsPresence = Literal["behavioral", "presence", "ambiguous"]
+FailsForDescribedReason = Literal["yes", "no", "unclear"]
+
+
+class RepoQualityJudgment(BaseModel):
+    behavioral_vs_presence: BehavioralVsPresence
+    fails_for_described_reason: FailsForDescribedReason
+    rationale: str = Field(default="")
+    judge_ok: bool = Field(default=True)
+    judge_error: str | None = Field(default=None)
+
+    def is_healthy(self) -> bool:
+        """Both dimensions must be green for the DAG to advance to fix."""
+        return (
+            self.judge_ok
+            and self.behavioral_vs_presence == "behavioral"
+            and self.fails_for_described_reason == "yes"
+        )
+
+
+_PROMPT_INSTRUCTIONS = """\
+You are auditing a bug reproduction (repro) script before any fix is attempted.
+
+You are given:
+  - the user's bug report (title, message, expected_behavior, steps_to_reproduce)
+  - the repro artifact: its source code and the path where it was written
+
+Score the repro on TWO dimensions:
+
+1. behavioral_vs_presence  ∈ {behavioral, presence, ambiguous}
+   - "behavioral"  → the repro executes the system and asserts on observable
+                     behavior (HTTP response, DB row state after an action,
+                     function return value, raised exception, log entry).
+   - "presence"    → the repro merely checks that a symbol / string / file
+                     exists in the codebase. Examples that are "presence":
+                       * grep for a field name in a model file
+                       * `assert "last_login_at" in open("user.py").read()`
+                       * importing a module and asserting an attribute exists
+                     These prove nothing about the bug — they pass on the
+                     string match even when behavior is still broken.
+   - "ambiguous"   → mixed / cannot tell from the artifact alone.
+
+2. fails_for_described_reason ∈ {yes, no, unclear}
+   - "yes"     → the failure mode the repro produces, when run against the
+                 current code, would manifest as the symptom the user described.
+   - "no"      → the repro fails for an unrelated reason (e.g. ImportError,
+                 missing fixture, wrong assertion target).
+   - "unclear" → cannot determine without running it.
+
+Return STRICT JSON with these exact keys:
+{
+  "behavioral_vs_presence": "<behavioral|presence|ambiguous>",
+  "fails_for_described_reason": "<yes|no|unclear>",
+  "rationale": "<one or two sentences citing the specific code patterns you saw>"
+}
+
+No prose outside the JSON. No code fences. No additional keys.
+"""
+
+
+def _read_repro_artifact(repo_root: Path, error_task_id: str) -> tuple[str | None, str | None]:
+    """Look for the repro artifact in the canonical user-report observability sandbox.
+
+    Returns (path_str, source_text) or (None, None) if absent.
+    """
+    base = repo_root / ".devflow" / "observability" / error_task_id
+    for ext in ("py", "test.ts", "test.tsx"):
+        candidate = base / f"repro.{ext}"
+        if candidate.exists():
+            try:
+                return str(candidate), candidate.read_text(encoding="utf-8")
+            except OSError:
+                return str(candidate), None
+    return None, None
+
+
+def _build_prompt_payload(
+    *,
+    bug_report: dict[str, Any],
+    observation: dict[str, Any] | None,
+    repro_path: str | None,
+    repro_source: str | None,
+) -> dict[str, Any]:
+    return {
+        "instructions": _PROMPT_INSTRUCTIONS,
+        "bug_report": {
+            "title": bug_report.get("title"),
+            "message": bug_report.get("message"),
+            "expected_behavior": bug_report.get("expected_behavior"),
+            "steps_to_reproduce": bug_report.get("steps_to_reproduce") or [],
+            "report_context": bug_report.get("report_context") or {},
+        },
+        "observation_summary": {
+            "summary": (observation or {}).get("summary"),
+            "likely_location": (observation or {}).get("likely_location"),
+            "repro_run_command": (observation or {}).get("repro_run_command"),
+        },
+        "repro_artifact": {
+            "path": repro_path,
+            "source": repro_source,
+        },
+    }
+
+
+def _persist_judgment(
+    *,
+    repo_root: Path,
+    error_task_id: str,
+    judgment: RepoQualityJudgment,
+) -> Path:
+    out_dir = repo_root / ".devflow" / "observability" / error_task_id
+    out_dir.mkdir(parents=True, exist_ok=True)
+    out_path = out_dir / "repo_quality.json"
+    out_path.write_text(
+        json.dumps(judgment.model_dump(), indent=2, sort_keys=True) + "\n",
+        encoding="utf-8",
+    )
+    return out_path
+
+
+def judge_repro_quality(
+    *,
+    repo_root: Path,
+    error_task_id: str,
+    bug_report: dict[str, Any],
+    observation: dict[str, Any] | None = None,
+    repro_path: str | None = None,
+    repro_source: str | None = None,
+) -> RepoQualityJudgment:
+    """Run the LLM judge over a repro artifact and persist the judgment.
+
+    Pure LLM — no fallback heuristics. If the LLM call fails or returns
+    unparseable JSON, the judgment is marked judge_ok=False and dimensions
+    fall back to the safe-stop default (ambiguous / unclear) so the DAG
+    holds for human review rather than advancing on weak evidence.
+    """
+    repo_root = Path(repo_root)
+    if repro_path is None and repro_source is None:
+        repro_path, repro_source = _read_repro_artifact(repo_root, error_task_id)
+
+    payload = _build_prompt_payload(
+        bug_report=bug_report,
+        observation=observation,
+        repro_path=repro_path,
+        repro_source=repro_source,
+    )
+
+    request = LlmInvocationRequest(
+        purpose="error_solver.repo_quality_judge",
+        repo_root=repo_root,
+        prompt="",
+        prompt_payload=payload,
+        strength="medium",
+        response_contract="json_only",
+        interaction_model="request_response",
+        timeout_seconds=180,
+    )
+
+    try:
+        result = invoke_llm(request)
+    except Exception as exc:  # pragma: no cover — surface-level transport failures
+        judgment = RepoQualityJudgment(
+            behavioral_vs_presence="ambiguous",
+            fails_for_described_reason="unclear",
+            rationale="judge invocation failed",
+            judge_ok=False,
+            judge_error=str(exc)[:500],
+        )
+        _persist_judgment(repo_root=repo_root, error_task_id=error_task_id, judgment=judgment)
+        return judgment
+
+    parsed: Any = result.parsed_json
+    if parsed is None or not isinstance(parsed, dict):
+        judgment = RepoQualityJudgment(
+            behavioral_vs_presence="ambiguous",
+            fails_for_described_reason="unclear",
+            rationale="judge response was not JSON object",
+            judge_ok=False,
+            judge_error=(result.contract_error or "non-json response")[:500],
+        )
+        _persist_judgment(repo_root=repo_root, error_task_id=error_task_id, judgment=judgment)
+        return judgment
+
+    try:
+        judgment = RepoQualityJudgment(
+            behavioral_vs_presence=parsed.get("behavioral_vs_presence"),
+            fails_for_described_reason=parsed.get("fails_for_described_reason"),
+            rationale=str(parsed.get("rationale") or ""),
+            judge_ok=True,
+            judge_error=None,
+        )
+    except ValidationError as exc:
+        judgment = RepoQualityJudgment(
+            behavioral_vs_presence="ambiguous",
+            fails_for_described_reason="unclear",
+            rationale="judge response failed schema validation",
+            judge_ok=False,
+            judge_error=str(exc)[:500],
+        )
+
+    _persist_judgment(repo_root=repo_root, error_task_id=error_task_id, judgment=judgment)
+    return judgment
+
+
+def build_default_repo_quality_judge_fn(
+    *,
+    repo_root: Path,
+    error_task_id: str,
+) -> "callable":
+    """Factory that closes over repo_root/etid so callers can pass a single
+    (context) → judgment function into the error-solver runtime."""
+
+    def _fn(context: dict[str, Any]) -> dict[str, Any]:
+        runtime_context = context.get("runtimeContext") if isinstance(context.get("runtimeContext"), dict) else {}
+        user_report = runtime_context.get("user_report") if isinstance(runtime_context.get("user_report"), dict) else {}
+        observability = context.get("BuildObservability") if isinstance(context.get("BuildObservability"), dict) else {}
+        out = observability.get("output") if isinstance(observability.get("output"), dict) else {}
+        observation = out.get("observation") if isinstance(out.get("observation"), dict) else None
+        judgment = judge_repro_quality(
+            repo_root=repo_root,
+            error_task_id=error_task_id,
+            bug_report=user_report,
+            observation=observation,
+        )
+        return judgment.model_dump()
+
+    return _fn

{devflow_engine-1.1.16 → devflow_engine-1.1.18}/src/devflow_engine/stores/execution_store.py

@@ -663,6 +663,12 @@ class ExecutionStore:
             # "failed" (observability_error populated) without parsing
             # context.json. Cleared on subsequent success.
             ("observability_error", "TEXT"),
+            # Track A (v1.1.18): RepoQualityJudge verdict on the repro
+            # artifact. NULL while DAG runs; set to
+            # 'awaiting_quality_review' when the judge flags presence-
+            # only or symptom-mismatched repros so the dashboard can
+            # surface them for human review without parsing context.json.
+            ("quality_status", "TEXT"),
             ("created_at", "INTEGER NOT NULL DEFAULT 0"),
             ("updated_at", "INTEGER NOT NULL DEFAULT 0"),
         ]
@@ -1964,6 +1970,7 @@ class ExecutionStore:
         ship_fn: Callable[[dict[str, Any]], dict[str, Any]] | None = None,
         bypass_mode: str | None = None,
         worker_role: str | None = None,
+        repo_quality_judge_fn: Callable[[dict[str, Any]], dict[str, Any]] | None = None,
     ) -> dict[str, Any]:
         """Run the US-8.5 error solver DAG using an artifact-first process contract.
 
@@ -2012,6 +2019,10 @@ class ExecutionStore:
                 "iterative_fix_fn": iterative_fix_fn,
                 "needs_user_input_fn": needs_user_input_fn,
                 "ship_fn": ship_fn,
+                # Track A (v1.1.18): repro-quality judge runs after
+                # BuildObservability persists out and before FirstPassFix.
+                # Pure-LLM gate — see errors/repo_quality_judge.py.
+                "repo_quality_judge_fn": repo_quality_judge_fn,
                 # Role propagation for BuildObservabilityNode:
                 # - analyst → always stop after observability
                 # - fix → skip pi observability (use cached observation_json),