devflow-engine 1.1.20__tar.gz → 1.1.21__tar.gz

devflow_engine-1.1.21/DESIGN_NOTES.md

@@ -0,0 +1,256 @@
+# DevFlow Error-Solver Hardening — Design Notes
+
+Track A landed in `v1.1.18`:
+- `errors/repo_quality_judge.py` — pure-LLM judge (medium tier → MiniMax) scoring
+  repro artifacts on `behavioral_vs_presence` and `fails_for_described_reason`.
+- `BuildObservabilityNode` runs the judge after the observation pass and before
+  `FirstPassFix`. Unhealthy judgments flip `error_tasks.quality_status` to
+  `awaiting_quality_review` and stop the workflow with `status='observed'`.
+- `ResolveNode` enforces the red-before-fix invariant: `verified =
+  green_gate AND observation_was_red`. Closes the "already-green-at-intake" leak
+  (`src/devflow_engine/errors/error_solver_dag.py` ResolveNode).
+- Schema: added `error_tasks.quality_status TEXT` via
+  `ExecutionStore._ensure_error_tasks_schema` (in-code migration, no Alembic).
+
+The rest of this document is the queue for the NEXT vertical slices. Each
+slice is intentionally small enough to land in one PR.
+
+---
+
+## Slice 1 — Playwright proof-artifact pipeline (validated end-to-end 2026-05-25)
+
+**Status:** validated by manual PoC against chi-webapp staging on 2026-05-25.
+The PoC's `manager-invite-error-detail.spec.ts` ran against
+`admin.staging.culturehealth.com`, captured `m2-filled-form.png +
+m4-after-second-submit.png + video.webm + trace.zip`, and the post-submit
+toast displayed the actual backend response detail (`"Not a member of this
+organization"`) instead of the pre-fix fallback (`"Couldn't send invitation."`).
+That validates both the artifact shape AND the underlying fix.
+
+**Why:** The repo-quality judge catches presence-only repros at intake, but
+it doesn't yet generate a visual behavioral artifact. For UI bugs the engine
+should emit a Playwright spec + recorded video + before/after screenshots so
+the portal can render visual proof the fix works.
+
+**Sources of truth on Playwright discipline:**
+- `/Volumes/devflow/.openclaw/workspace/docs/how_to_playwright_right.md`
+- `/Volumes/devflow/.openclaw/workspace/docs/PLAYWRITE-E2E-PROCESS.md`
+- PoC reference implementation: `/Users/marcusswift/proof-poc/` (kept around
+  as the canonical example of fixture wiring + auth setup + spec shape)
+
+### Net-new infrastructure required
+
+**1. Per-project `fixtures.yaml`** committed to repo at `.devflow/fixtures.yaml`.
+Captures the seed knowledge the spec generator needs and is currently
+hand-crafted per PoC. Schema:
+```yaml
+staging:
+  portal_base_url: https://admin.staging.culturehealth.com
+  alt_portal_base_urls:
+    customer_portal: https://portal.staging.culturehealth.com
+auth:
+  strategy: form_login          # form_login | api_token | header_token
+  login_path: /login
+  email_field_label: Email
+  password_field_label: Password
+  submit_button_name: /sign in|log in/i
+  storage_state_path: .auth/state.json
+test_credentials:
+  email_secret: tenant.<uuid>.devflow_host.proof.chi_staging.email
+  password_secret: tenant.<uuid>.devflow_host.proof.chi_staging.password
+  # secrets resolved at run time via clarity_secrets.get_secret
+seed_data:
+  organizations:
+    - id: e480c2a4-0ba2-4955-b8e6-4cf32bb05763  # Demo Corporation
+      groups:
+        - id: 5b1453ab-f22e-4d60-88db-e82576f0aabb
+          name: Sales
+test_email_prefix: proof-test+   # prefix the spec uses for ephemeral test data
+```
+
+**2. One-time test-account seeder** — replaces the ad-hoc procedure I ran on
+2026-05-25 (canonical + role-alias user creation + password into
+`internal.secrets`). Should become a CLI:
+```
+devflow project init-test-account --project chi-webapp --portal admin --role admin
+```
+Implementation: wraps `commands/create_proof_user.py` +
+`create_proof_alias.py` (templates from `/Users/marcusswift/proof-poc/scripts/`
+when this slice lands), generates a 32-char password, stores under
+`tenant.<uuid>.devflow_host.proof.<project>.{email,password}` in
+`internal.secrets`, writes the result into the project's `.devflow/fixtures.yaml`
+auth block.
+
+**3. Playwright observability module** (engine-side, peer of existing pytest one):
+- File: `src/devflow_engine/errors/user_report_observability_playwright.py`
+- Pi-agent-driven generation of `repro.spec.ts` seeded with:
+  - The app surface route from `report_context.page_url`
+  - Source of the relevant React component (already in workspace)
+  - Expected behavior from observability summary
+  - Seed data from `.devflow/fixtures.yaml`
+- Persists to `.devflow/observability/{etid}/repro.spec.ts`
+
+**4. Playwright execution node** (engine-side):
+- New node in `error_solver_dag.py` that runs after the existing repro
+  green-gate passes (i.e. unit/integration test is green) but before
+  ResolveNode commits.
+- Spawns `npx playwright test repro.spec.ts --reporter=null` from inside a
+  bootstrapped `node_modules` (see #7 for image bake).
+- Captures `video.webm`, `before.png`, `after.png`, `trace.zip` to
+  `.devflow/observability/{etid}/proof/`.
+- On failure: persists trace, sets new column `proof_status='failed'` (add to
+  `_ensure_error_tasks_schema`), block ResolveNode from committing.
+
+**5. Portal artifact-serving + template update**:
+- `controller/api/bugs.py`: new `GET /bugs/{token}/artifact/{filename}`
+  (token-authed, returns file from
+  `.devflow/observability/{etid}/proof/`, with mime-type
+  detection). Whitelist filenames against the proof dir to prevent traversal.
+- `controller/templates/bugs/view.html.j2`: when `proof/` exists for the row,
+  render `<video controls src="/bugs/{token}/artifact/session.webm">` plus
+  `<img src=".../before.png">` / `<img src=".../after.png">` thumbnails.
+
+**6. Surface-routing decision** in `BuildObservabilityNode`:
+```python
+def _picks_playwright(report_context):
+    app_surface = report_context.get("app_surface") or ""
+    page_url = report_context.get("page_url") or ""
+    frontend_slugs = {"chi-webapp-admin-frontend", "chi-webapp-customer-portal", ...}
+    if app_surface in frontend_slugs: return True
+    if any(page_url.startswith(p) for p in ("http", "/")): return True
+    return False
+```
+If True → call Playwright module; else → existing pytest module.
+
+**7. Engine image bake** (`images/engine.Dockerfile`):
+```
+RUN apt-get install -y nodejs npm
+RUN npm install -g playwright@1.49 && playwright install chromium --with-deps
+```
+~150 MB image growth; saves 30 s/run vs on-demand install.
+
+### Acceptance
+
+- A user-report against a frontend route renders `.spec.ts` + `video.webm` +
+  `before.png` + `after.png` + `trace.zip` under
+  `.devflow/observability/{etid}/proof/`.
+- The submitter's RESOLVED email's portal link shows the video and
+  screenshots inline.
+- The PoC `manager-invite-error-detail.spec.ts` translates 1:1 — its only
+  hand-coded bits (ORG_ID, group name) come from `.devflow/fixtures.yaml`,
+  its auth comes from the seeded test account.
+
+### Effort estimate
+
+~10-15 hours focused work for one background agent. Gating dependency is
+`.devflow/fixtures.yaml` (#1) — until each project has one, the generator
+must fall back to LLM-guessed seed data, which the PoC showed is risky
+(membership errors leak into the assertion).
+
+### Implementation timing
+
+**Do NOT launch the background agent until the 2026-05-25 rerun drain
+completes** — concurrent engine changes during a 40-row drain caused the
+afternoon's failure cascade. Wait for drain → review the manual proof
+artifacts produced for each resolution (used as live abstraction-validation
+data) → THEN dispatch the implementation agent with the accumulated
+learning from those manual runs.
+
+---
+
+## Slice 2 — Bug-portal routes `/bugs/{token}` on devflow-host
+
+**Why:** Submitters need to see (and respond to) the analyst's observation +
+repro + quality-judgment without an account. Mirrors the clarify magic-link
+pattern that's already in production for clarification questions.
+
+**Where to start:**
+- Mirror the controller pattern at
+  `controller/api/clarify.py` (in the devflow-host repo, not engine).
+- Token mechanism: reuse `clarification_links.create_link()` —
+  same TTL/rotation semantics already approved.
+- Engine side: add a `create_bug_portal_link(error_task_id)` helper next to
+  the existing clarification-link helper. New rows in the existing
+  `clarification_links` table with a discriminator column, OR a new
+  `bug_portal_links` table — prefer the former to keep the token rotation
+  audit in one place.
+- Portal routes (devflow-host):
+  - `GET /bugs/{token}` — render observation summary, repro source, judgment.
+  - `POST /bugs/{token}/request-changes` — see Slice 3.
+  - `POST /bugs/{token}/approve` — see Slice 4.
+
+**Acceptance:**
+- A user-report row with `quality_status='awaiting_quality_review'` exposes a
+  portal URL in `error_tasks` (new column `portal_url` or derived).
+- Submitter sees the repro and the judge's rationale; can act on it.
+
+---
+
+## Slice 3 — Modification-request → linked-new error_task
+
+**Why:** When a submitter clicks "request changes" on the portal, that's new
+information the analyst doesn't have yet. Spawn a new `error_tasks` row linked
+back to the original so the analyst pool re-claims it cleanly without losing
+the original audit trail.
+
+**Where to start:**
+- New schema column: `error_tasks.parent_error_task_id TEXT` (NULL for roots).
+  Add to `ExecutionStore._ensure_error_tasks_schema` required list. Index on it
+  for `WHERE parent_error_task_id = ?` lookups.
+- Engine helper: `ExecutionStore.create_modification_request(parent_etid,
+  message)` — inserts a child row with `source_kind='user_report'`,
+  `status='open'`, copies `project_id` + `reporter_*` + `source_app` from
+  parent, message = submitter's modification request, fingerprint derived
+  from `parent_etid + nonce` to bypass dedupe.
+- Portal handler `POST /bugs/{token}/request-changes` calls that helper.
+- Worker claim filter (`stores/execution_store.py` ~ line 2461 / 2478) does
+  NOT need changes — child rows are plain `open` user_report rows.
+
+**Acceptance:**
+- Submitter modification creates a sibling row visible to the analyst with a
+  link back to the original via `parent_error_task_id`.
+- Original row's `quality_status` remains; child gets its own DAG run.
+
+---
+
+## Slice 4 — Submitter-approval → staging push integration
+
+**Why:** After RepoQualityJudge + green-gate + observation_was_red all pass,
+the resolution commit lives in the worker's local repo but doesn't reach
+staging until a human approves. The portal `approve` button should be the
+trigger.
+
+**Where to start:**
+- The commit is already created by `_commit_resolution` in
+  `errors/error_solver_dag.py` (lines 179–224). The SHA is persisted to
+  `task_context.metadata["commit_result"]`. Extend ResolveNode to also write
+  the SHA into `error_tasks.resolution_commit_sha TEXT` (new column).
+- Portal handler `POST /bugs/{token}/approve`:
+  1. Look up `resolution_commit_sha` on the row.
+  2. `git push origin HEAD:refs/heads/staging-<etid>` from the worker's repo
+     clone (engine-side helper: `push_resolution_to_staging(etid)`).
+  3. Optionally open a PR via `gh pr create` against the project's main.
+- New `error_tasks.staging_status TEXT` column to track
+  `unpushed | pushed | pr_open | merged | rejected`.
+
+**Acceptance:**
+- Approval pushes exactly the resolution commit (single-commit branch) to
+  staging.
+- Rejection leaves the commit local and flips
+  `quality_status='awaiting_quality_review'` so a fresh analyst pass can
+  iterate.
+
+---
+
+## Anchors quick-reference
+
+- Engine DAG entrypoint: `src/devflow_engine/errors/error_solver_dag.py`
+  - `BuildObservabilityNode.process` — observation + RepoQualityJudge
+  - `ResolveNode.process` — red-before-fix invariant + commit
+  - `_commit_resolution` — atomic per-error commit (existing)
+- Schema migrations: `src/devflow_engine/stores/execution_store.py`
+  `_ensure_error_tasks_schema` (~ line 611)
+- Runner: `ExecutionStore.run_error_solver_dag` (~ line 1953); add new
+  `repo_quality_judge_fn` kwarg already present.
+- Judge: `src/devflow_engine/errors/repo_quality_judge.py`
+- Tests: `tests/area8_errors/test_us85_error_solver_repo_quality.py`

{devflow_engine-1.1.20 → devflow_engine-1.1.21}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: devflow-engine
-Version: 1.1.20
+Version: 1.1.21
 Summary: DevFlow v2 CLI-first engine (projects, stories, execution store, review packets)
 Author: DevFlow
 License: Proprietary

{devflow_engine-1.1.20 → devflow_engine-1.1.21}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "devflow-engine"
-version = "1.1.20"
+version = "1.1.21"
 description = "DevFlow v2 CLI-first engine (projects, stories, execution store, review packets)"
 readme = "README.md"
 requires-python = ">=3.11"

{devflow_engine-1.1.20 → devflow_engine-1.1.21}/src/devflow_engine/errors/error_solver_dag.py

@@ -208,8 +208,12 @@ def _is_source_path(path: str) -> bool:
 
 
 def _git_dirty_paths(repo_root: Path) -> list[str]:
+    # -uall expands untracked directories into individual files. Without
+    # it, `git status --porcelain` collapses an untracked dir into the
+    # bare dir entry (e.g. `fastapi_backend/tests/`) and the per-file
+    # preserve/cleanup helper has nothing concrete to write or remove.
     cp = subprocess.run(
-        ["git", "-C", str(repo_root), "status", "--porcelain"],
+        ["git", "-C", str(repo_root), "status", "--porcelain", "-uall"],
         text=True,
         capture_output=True,
         check=False,
@@ -236,6 +240,145 @@ def _assert_clean_workspace(repo_root: Path, *, error_task_id: str) -> None:
         )
 
 
+def _preserve_workspace_on_non_resolve(
+    *,
+    repo_root: Path,
+    error_task_id: str,
+    outcome: str,
+) -> dict[str, Any]:
+    """Preserve pi-agent edits on non-resolve terminals, then reset clean.
+
+    The error-solver DAG only commits on the `resolved` terminal (see
+    `_commit_resolution`). Every other terminal — `blocked`,
+    `observation_failed`, `observation_inconclusive`, the analyst
+    `observed` stop, the non-verified Resolve `in_progress` branch — used
+    to leave any pi-agent edits orphaned in the workspace. The next
+    error-solver run then hit `_assert_clean_workspace` →
+    `WorkspaceDirtyError` and the row silently flipped to `failed`.
+
+    Real example (2026-05-25): rerun row `d6061f8a` ended awaiting_human.
+    The pi agent had created `fastapi_backend/tests/test_auth_last_login_tracking.py`
+    during FirstPassFix. That file orphaned. The next 34 worker calls all
+    failed the clean check before observability could even run.
+
+    Capture-and-reset:
+    1. Detect source-level dirt via `_git_dirty_paths` (already filters
+       `.devflow/` and build/test/runtime noise).
+    2. Build a single combined wip patch at
+       `.devflow/observability/{error_task_id}/wip-{outcome}.patch`
+       containing `git diff` for tracked-modified paths + the raw content
+       of each untracked source file (since `git diff` won't capture
+       untracked). Header records the path list.
+    3. Reset workspace clean:
+       - `git checkout HEAD -- <tracked-modified-paths>`
+       - `rm <untracked-source-paths>`
+
+    No-op when there's no dirt.
+    """
+    paths = _git_dirty_paths(repo_root)
+    if not paths:
+        return {"preserved": False, "paths": [], "patch_path": None}
+
+    # Split into tracked-modified vs untracked using `git ls-files --error-unmatch`.
+    tracked: list[str] = []
+    untracked: list[str] = []
+    for p in paths:
+        cp = subprocess.run(
+            ["git", "-C", str(repo_root), "ls-files", "--error-unmatch", "--", p],
+            text=True, capture_output=True, check=False,
+        )
+        if cp.returncode == 0:
+            tracked.append(p)
+        else:
+            untracked.append(p)
+
+    # tracked diff (modified-against-HEAD)
+    diff_text = ""
+    if tracked:
+        diff_cp = subprocess.run(
+            ["git", "-C", str(repo_root), "diff", "HEAD", "--", *tracked],
+            text=True, capture_output=True, check=False,
+        )
+        diff_text = diff_cp.stdout or ""
+
+    # raw content of untracked files (git diff doesn't capture them)
+    untracked_blocks: list[str] = []
+    for u in untracked:
+        full = repo_root / u
+        try:
+            body = full.read_text(encoding="utf-8")
+        except Exception as exc:
+            body = f"<<unreadable: {exc}>>"
+        untracked_blocks.append(
+            f"\n# === untracked source file: {u} ===\n{body}\n"
+        )
+
+    header_lines = [
+        f"# wip-{outcome}.patch",
+        f"# error_task_id: {error_task_id}",
+        f"# outcome: {outcome}",
+        f"# preserved_at: {_utc_now_iso()}",
+        f"# tracked_modified: {json.dumps(tracked)}",
+        f"# untracked_source: {json.dumps(untracked)}",
+        "",
+    ]
+    patch_body = "\n".join(header_lines) + diff_text + "".join(untracked_blocks)
+
+    out_dir = repo_root / ".devflow" / "observability" / error_task_id
+    out_dir.mkdir(parents=True, exist_ok=True)
+    patch_path = out_dir / f"wip-{outcome}.patch"
+    patch_path.write_text(patch_body, encoding="utf-8")
+
+    # Reset workspace
+    if tracked:
+        subprocess.run(
+            ["git", "-C", str(repo_root), "checkout", "HEAD", "--", *tracked],
+            text=True, capture_output=True, check=False,
+        )
+    for u in untracked:
+        full = repo_root / u
+        try:
+            full.unlink()
+        except FileNotFoundError:
+            pass
+        except Exception:
+            # Best-effort: surface via _git_dirty_paths on next run if it matters.
+            pass
+
+    # Journal: append a short note so the operator can find the wip patch.
+    try:
+        d = repo_root / ".devflow" / "journal"
+        d.mkdir(parents=True, exist_ok=True)
+        day = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+        md = d / f"{day}.md"
+        note = (
+            f"\n## Workspace preserved on non-resolve {uuid.uuid4().hex[:8]}\n"
+            f"- time (UTC): {_utc_now_iso()}\n"
+            f"- error_task_id: `{error_task_id}`\n"
+            f"- outcome: `{outcome}`\n"
+            f"- patch: `{patch_path}`\n"
+            f"- tracked_modified: `{tracked}`\n"
+            f"- untracked_source: `{untracked}`\n"
+        )
+        if md.exists():
+            existing = md.read_text(encoding="utf-8")
+            if not existing.endswith("\n"):
+                existing += "\n"
+            md.write_text(existing + note, encoding="utf-8")
+        else:
+            md.write_text("# LLM Journal\n" + note, encoding="utf-8")
+    except Exception:
+        pass
+
+    return {
+        "preserved": True,
+        "paths": paths,
+        "tracked": tracked,
+        "untracked": untracked,
+        "patch_path": str(patch_path),
+    }
+
+
 def _commit_resolution(
     repo_root: Path,
     *,
@@ -580,11 +723,28 @@ class BuildObservabilityNode(Node):
             #   claim filter requires observability_json IS NOT NULL).
             # - None / legacy → gate-aware stop (pre-1.1.15 behavior).
             if worker_role == "analyst":
+                # v1.1.21: analyst stop is a non-resolve terminal. If the
+                # build_observability_fn (or any sub-step) dirtied source
+                # files, preserve them to a wip patch + reset before the
+                # next claim. outcome here is whatever was set above
+                # (observed / observation_inconclusive / observation_failed).
+                _preserve_workspace_on_non_resolve(
+                    repo_root=repo_root,
+                    error_task_id=error_task_id,
+                    outcome=str(task_context.metadata.get("outcome") or "observed"),
+                )
                 task_context.stop_workflow()
                 return task_context
             elif worker_role == "fix":
                 pass  # always continue
             elif not gates_passed:
+                # v1.1.21: gate-aware stop is a non-resolve terminal —
+                # capture any analyst/observability dirt.
+                _preserve_workspace_on_non_resolve(
+                    repo_root=repo_root,
+                    error_task_id=error_task_id,
+                    outcome=str(task_context.metadata.get("outcome") or "observation_inconclusive"),
+                )
                 task_context.stop_workflow()
                 return task_context
 
@@ -638,6 +798,13 @@ class BuildObservabilityNode(Node):
                             f"fails_for_described_reason="
                             f"{judgment.get('fails_for_described_reason')!r}"
                         )
+                        # v1.1.21: awaiting_quality_review is a non-resolve
+                        # terminal. Preserve any dirt before stopping.
+                        _preserve_workspace_on_non_resolve(
+                            repo_root=repo_root,
+                            error_task_id=error_task_id,
+                            outcome="awaiting_quality_review",
+                        )
                         task_context.stop_workflow()
                         return task_context
 
@@ -802,6 +969,16 @@ class ResolveNode(Node):
             prior_runs=prior_runs,
         )
 
+        # v1.1.21: non-verified Resolve branch is a non-resolve terminal —
+        # preserve any pi-agent edits to a wip patch + reset workspace so
+        # the next error-solver run isn't bounced by _assert_clean_workspace.
+        if not verified:
+            _preserve_workspace_on_non_resolve(
+                repo_root=persistor.repo_root,
+                error_task_id=error_task_id,
+                outcome=str(task_context.metadata.get("outcome") or "in_progress"),
+            )
+
         task_context.stop_workflow()
         return task_context
 
@@ -849,6 +1026,13 @@ class BlockNode(Node):
             runtime_context=runtime_context,
             prior_runs=prior_runs,
         )
+        # v1.1.21: blocked is a non-resolve terminal — capture pi edits
+        # before stop_workflow so the next claim isn't bounced.
+        _preserve_workspace_on_non_resolve(
+            repo_root=persistor.repo_root,
+            error_task_id=error_task_id,
+            outcome="blocked",
+        )
         task_context.stop_workflow()
         return task_context