ouroboros-ai 0.42.2.dev7__tar.gz → 0.42.2.dev9__tar.gz

{ouroboros_ai-0.42.2.dev7 → ouroboros_ai-0.42.2.dev9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ouroboros-ai
-Version: 0.42.2.dev7
+Version: 0.42.2.dev9
 Summary: Specification-first workflow engine for AI coding agents. Works with Claude Code and Codex CLI.
 Author-email: Q00 <jqyu.lee@gmail.com>
 License-File: LICENSE

ouroboros_ai-0.42.2.dev9/docs/rfc/frugality-proof-producers.md

@@ -0,0 +1,84 @@
+# RFC — Frugality-proof producers: wiring the token, grounding, and baseline axes
+
+> Status: **Draft**
+> Epic: [#1465](https://github.com/Q00/ouroboros/issues/1465) ("Frugality you can prove") · Task [#1470](https://github.com/Q00/ouroboros/issues/1470)
+> Depends on: the effort contract + deterministic proof gate (PR stack #1473–#1478).
+
+## Why
+
+The deterministic frugality-proof machine (`orchestrator/frugality_proof.py`) is
+delivered by the proof-gate PR (#1478) this RFC depends on — it is **not yet on
+`main`**; it lands with the #1473–#1478 stack below. There, `assemble_triads()` joins
+per-AC events into a `FrugalityTriadRow`, and `evaluate_proof()` computes the seed's
+PASS/FAIL gate — **grounding regression is a per-AC veto**, then sample sufficiency
+(≥20 triads / ≥3 runs), then aggregate token reduction (≥10%). The **effort axis** is
+produced by the same stack (`execution.ac.effort_routed`, `mode=enforced` on
+Claude/Codex/Copilot).
+
+But a row only `counts_in_proof` when it carries **all** axes, and three are not yet
+emitted. Until they are, the gate honestly returns `INSUFFICIENT_DATA`. This RFC
+specifies the three remaining producers against the event-type contract that gate
+reads — so they have a precise, testable target and the gate stays unchanged.
+
+## The fixed event contract (consumed by the #1478 gate)
+
+The gate (`frugality_proof.py`, shipped in #1478) reads these event types and fields.
+Producers must emit them keyed by the same `ac_id` the effort event uses, and must
+carry the **run anchor** (`seed_run_id`, or `execution_id`) on every event: the proof
+spans runs and the same logical `ac_id` recurs each run, so `assemble_triads()` keys
+rows by `(run, ac_id)`. An axis event without the run anchor cannot be attributed to
+the right run's row.
+
+| Event type | Producer | Required fields | Seed AC |
+|---|---|---|---|
+| `execution.ac.effort_routed` | **done** | `effort_level`, `effort_mode`, `is_decomposed_child`, `ac_id`, `seed_run_id` | (effort contract) |
+| `execution.ac.token_attribution.reported` | **#1 below** | `ac_id`, `seed_run_id`, `token_spend` | AC2 |
+| `execution.ac.deliver_verdict` | **#2 below** | `ac_id`, `seed_run_id`, `traceguard_verdict`, `unsupported_claim_rate`, `grounding_regression` | AC4 |
+| `execution.ac.shadow_replay` | **#3 below** | `ac_id`, `seed_run_id`, `baseline_token_spend`, `baseline_mode`, `decomposition_trustworthy` | AC5 |
+
+## Producer #1 — Per-AC token attribution (AC2)
+
+Emit `execution.ac.token_attribution.reported` carrying the **real** token count an
+AC consumed, from the runtime's usage signals (not estimated from text length). On
+runtimes that surface no usage counters, emit `token_spend: null` honestly rather
+than fabricating — such rows simply will not count toward the proof.
+
+Acceptance: a re-done AC reports a higher spend than a clean one; a clean first-try
+AC is never inflated; a test asserts no value is a hardcoded placeholder.
+
+## Producer #2 — TraceGuard deliver verdict (AC4)
+
+For each AC, run the deliver claim through the deterministic TraceGuard validator
+(`harness/deliver_gate.py`, #978) against the evidence manifest, and emit
+`execution.ac.deliver_verdict` with the accepted/rejected `traceguard_verdict`, the
+`unsupported_claim_rate`, and `grounding_regression` — **true** iff the lowered-effort
+run produced any newly-rejected claim versus its parent-effort baseline.
+`fat_harness` ON is the grounding precondition; under OFF, no verdict is emitted (so
+those rows do not count). This is the axis the per-AC grounding veto reads.
+
+Acceptance: identical inputs yield identical verdicts (deterministic, no noise band);
+a newly-rejected claim at lower effort surfaces `grounding_regression: true`.
+
+## Producer #3 — Shadow-replay paired baseline (AC5)
+
+In an **experiment-harness path only** (never production steady-state), re-execute
+each enforced decomposed-child AC once at its **parent** effort and emit
+`execution.ac.shadow_replay` with `baseline_token_spend`, `baseline_mode:
+"shadow_replay"`, and `decomposition_trustworthy` (false for forced-atomic /
+MECE-repair-failed units, which are quarantined out of the proof). This is the
+paired baseline the frugality bar measures reduction against.
+
+Acceptance: cost bounded to the experiment set (~2× on those rows only); forced-atomic
+units are recorded but excluded; the triad pairs each child's lowered-effort run with
+its parent-effort baseline.
+
+## Out of scope
+
+- Any change to `evaluate_proof()` / the gate — the thresholds and order are fixed.
+- The effort contract (shipped in the #1473–#1478 stack).
+
+## Done = the gate stops returning INSUFFICIENT_DATA
+
+Once all three producers emit their events for an enforced run, `assemble_triads()`
+yields fully-measured rows and `evaluate_proof()` returns a real PASS/FAIL —
+the complete enforced-triad proof the seed set out to produce.

{ouroboros_ai-0.42.2.dev7 → ouroboros_ai-0.42.2.dev9}/src/ouroboros/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.42.2.dev7'
-__version_tuple__ = version_tuple = (0, 42, 2, 'dev7')
+__version__ = version = '0.42.2.dev9'
+__version_tuple__ = version_tuple = (0, 42, 2, 'dev9')
 
 __commit_id__ = commit_id = None

{ouroboros_ai-0.42.2.dev7 → ouroboros_ai-0.42.2.dev9}/src/ouroboros/auto/interview_driver.py

@@ -292,6 +292,8 @@ class AutoInterviewDriver:
         event_type: str,
         aggregate_id: str,
         data: dict[str, object],
+        *,
+        aggregate_type: str = "auto_interview",
     ) -> None:
         """Append a single event with bounded fail-open semantics.
 
@@ -306,7 +308,7 @@ class AutoInterviewDriver:
                 self.event_store.append(
                     BaseEvent(
                         type=event_type,
-                        aggregate_type="auto_interview",
+                        aggregate_type=aggregate_type,
                         aggregate_id=aggregate_id,
                         data=data,
                     )
@@ -339,6 +341,8 @@ class AutoInterviewDriver:
         self,
         event_type: str,
         aggregate_id: str | None,
+        *,
+        aggregate_type: str = "auto_interview",
         **data: object,
     ) -> None:
         """Schedule a typed ``auto.interview.*`` event for background
@@ -366,7 +370,9 @@ class AutoInterviewDriver:
         if not aggregate_id:
             return
         task = asyncio.create_task(
-            self._append_with_fail_open(event_type, aggregate_id, dict(data))
+            self._append_with_fail_open(
+                event_type, aggregate_id, dict(data), aggregate_type=aggregate_type
+            )
         )
         # Hold a strong reference so the task is not garbage-collected
         # mid-flight, and clean it up on completion so the set stays
@@ -606,7 +612,7 @@ class AutoInterviewDriver:
         except TimeoutError as exc:
             self._record_evidence_based_session_id(state, exc, preassigned_id)
             if interview_tool_name == "interview.start":
-                fallback = self._try_close_after_backend_start_failure(state, ledger, exc)
+                fallback = await self._try_close_after_backend_start_failure(state, ledger, exc)
                 if fallback is not None:
                     return fallback
             message = str(exc)
@@ -620,7 +626,7 @@ class AutoInterviewDriver:
             self._record_evidence_based_session_id(state, exc, preassigned_id)
             action = "resume" if interview_tool_name == "interview.resume" else "start"
             if action == "start":
-                fallback = self._try_close_after_backend_start_failure(state, ledger, exc)
+                fallback = await self._try_close_after_backend_start_failure(state, ledger, exc)
                 if fallback is not None:
                     return fallback
             blocker = f"interview {action} failed: {exc}"
@@ -867,14 +873,23 @@ class AutoInterviewDriver:
             )
 
         open_gaps = ledger.open_gaps()
+        entered_payload = {
+            "backend_done": backend_done,
+            "ledger_done": ledger_done,
+            "open_gaps": list(open_gaps),
+            "ambiguity_score": turn.ambiguity_score,
+            "max_rounds": self.max_rounds,
+        }
         log.info(
             "auto.interview.safe_default.entered",
             auto_session_id=state.auto_session_id,
-            backend_done=backend_done,
-            ledger_done=ledger_done,
-            open_gaps=list(open_gaps),
-            ambiguity_score=turn.ambiguity_score,
-            max_rounds=self.max_rounds,
+            **entered_payload,
+        )
+        await self._emit_event(
+            "auto.closure.safe_default_entered",
+            state.auto_session_id,
+            aggregate_type="auto_session",
+            **entered_payload,
         )
 
         finalization = None
@@ -1004,12 +1019,24 @@ class AutoInterviewDriver:
                     return AutoInterviewResult(
                         "blocked", state.interview_session_id, ledger, self.max_rounds, blocker
                     )
+            closed_payload = {
+                "closure_mode": "safe_default",
+                "defaulted_sections": list(finalization.defaulted_sections),
+                "synthesis_pushed": synthesis_pushed,
+                "interview_session_id": state.interview_session_id,
+            }
             log.info(
                 "auto.interview.safe_default.closed",
                 auto_session_id=state.auto_session_id,
                 defaulted_sections=finalization.defaulted_sections,
                 synthesis_pushed=synthesis_pushed,
             )
+            await self._emit_event(
+                "auto.closure.safe_default_closed",
+                state.auto_session_id,
+                aggregate_type="auto_session",
+                **closed_payload,
+            )
             state.ledger = ledger.to_dict()
             state.pending_question = None
             state.interview_completed = True
@@ -1049,6 +1076,12 @@ class AutoInterviewDriver:
             and finalization.defaulted_sections
             and finalization.unsafe_gaps
         ):
+            partial_payload = {
+                "defaulted_sections": list(finalization.defaulted_sections),
+                "unsafe_gaps": list(finalization.unsafe_gaps),
+                "ambiguity_score": turn.ambiguity_score,
+                "interview_session_id": state.interview_session_id,
+            }
             log.info(
                 "auto.interview.safe_default_partial_unsafe_gaps",
                 auto_session_id=state.auto_session_id,
@@ -1057,6 +1090,12 @@ class AutoInterviewDriver:
                 ambiguity_score=turn.ambiguity_score,
                 interview_session_id=state.interview_session_id,
             )
+            await self._emit_event(
+                "auto.closure.safe_default_blocked",
+                state.auto_session_id,
+                aggregate_type="auto_session",
+                **partial_payload,
+            )
             _revert_safe_default_entries(ledger, finalization.defaulted_sections)
             blocker = (
                 f"auto interview reached max_rounds={self.max_rounds} with "
@@ -1119,7 +1158,7 @@ class AutoInterviewDriver:
             "blocked", state.interview_session_id, ledger, self.max_rounds, blocker
         )
 
-    def _try_close_after_backend_start_failure(
+    async def _try_close_after_backend_start_failure(
         self,
         state: AutoPipelineState,
         ledger: SeedDraftLedger,
@@ -1137,6 +1176,8 @@ class AutoInterviewDriver:
         if not _is_authoring_backend_unavailable(exc):
             return None
         closure_mode = "ledger_only_no_backend"
+        finalization = None
+        open_gaps_before_finalization = list(ledger.open_gaps())
         if not ledger.is_seed_ready():
             finalization = finalize_safe_defaultable_gaps(
                 ledger,
@@ -1151,6 +1192,30 @@ class AutoInterviewDriver:
                 return None
             closure_mode = "safe_default_no_backend"
 
+        if closure_mode == "safe_default_no_backend" and finalization is not None:
+            await self._emit_event(
+                "auto.closure.safe_default_entered",
+                state.auto_session_id,
+                aggregate_type="auto_session",
+                backend_done=False,
+                ledger_done=False,
+                open_gaps=open_gaps_before_finalization,
+                ambiguity_score=None,
+                max_rounds=self.max_rounds,
+                closure_mode=closure_mode,
+                error=str(exc),
+            )
+            await self._emit_event(
+                "auto.closure.safe_default_closed",
+                state.auto_session_id,
+                aggregate_type="auto_session",
+                closure_mode=closure_mode,
+                defaulted_sections=list(finalization.defaulted_sections),
+                synthesis_pushed=False,
+                interview_session_id=state.interview_session_id,
+                backend_start_failed=True,
+            )
+
         state.ledger = ledger.to_dict()
         state.pending_question = None
         state.interview_completed = True

{ouroboros_ai-0.42.2.dev7 → ouroboros_ai-0.42.2.dev9}/tests/unit/auto/test_safe_default_observability.py

@@ -64,6 +64,14 @@ def _ledger_all_filled_except(
     return ledger
 
 
+class _RecordingEventStore:
+    def __init__(self) -> None:
+        self.appended = []
+
+    async def append(self, event, **_kwargs) -> None:
+        self.appended.append(event)
+
+
 # ---------------------------------------------------------------------------
 # Test 1: no_gaps_to_default + mutual_agreement_deadlock_at_max_rounds
 # ---------------------------------------------------------------------------
@@ -136,15 +144,18 @@ async def test_safe_default_logs_closed_path(tmp_path) -> None:
         return InterviewTurn("What else should we know?", session_id, seed_ready=False)
 
     state = AutoPipelineState(goal="Build a tiny local CLI", cwd=str(tmp_path))
+    event_store = _RecordingEventStore()
     driver = AutoInterviewDriver(
         FunctionInterviewBackend(start, answer),
         store=AutoStore(tmp_path),
         max_rounds=1,
         timeout_seconds=1,
+        event_store=event_store,
     )
 
     with capture_logs() as captured:
         result = await driver.run(state, ledger)
+    await driver.wait_for_pending_emits()
 
     events = [e["event"] for e in captured]
     assert "auto.interview.safe_default.entered" in events
@@ -156,6 +167,18 @@ async def test_safe_default_logs_closed_path(tmp_path) -> None:
     defaulted = closed_events[0]["defaulted_sections"]
     assert "runtime_context" in defaulted
 
+    closure_events = [
+        event for event in event_store.appended if event.type.startswith("auto.closure.")
+    ]
+    assert [event.type for event in closure_events] == [
+        "auto.closure.safe_default_entered",
+        "auto.closure.safe_default_closed",
+    ]
+    assert all(event.aggregate_type == "auto_session" for event in closure_events)
+    assert all(event.aggregate_id == state.auto_session_id for event in closure_events)
+    assert closure_events[0].data["open_gaps"]
+    assert "runtime_context" in closure_events[1].data["defaulted_sections"]
+
     # Behaviour unchanged: seed ready after safe-default closure.
     assert result.status == "seed_ready"
 
@@ -199,6 +222,50 @@ async def test_safe_default_logs_synthesis_nonclosure(tmp_path) -> None:
     assert "did not close" in (result.blocker or "")
 
 
+@pytest.mark.asyncio
+async def test_no_backend_safe_default_persists_closure_events(tmp_path) -> None:
+    """No-backend safe-default success has queryable closure evidence."""
+    ledger = SeedDraftLedger.from_goal("Build a tiny local CLI")
+
+    async def start(goal: str, cwd: str, *, interview_id: str | None = None) -> InterviewTurn:  # noqa: ARG001
+        raise RuntimeError("codex profile failed before first question")
+
+    async def answer(
+        session_id: str, text: str, *, last_question: str | None = None
+    ) -> InterviewTurn:  # noqa: ARG001
+        raise AssertionError("answer must not run when start fails")
+
+    state = AutoPipelineState(goal="Build a tiny local CLI", cwd=str(tmp_path))
+    event_store = _RecordingEventStore()
+    driver = AutoInterviewDriver(
+        FunctionInterviewBackend(start, answer, is_session_persisted=lambda _id: False),
+        store=AutoStore(tmp_path),
+        max_rounds=1,
+        timeout_seconds=1,
+        event_store=event_store,
+    )
+
+    result = await driver.run(state, ledger)
+    await driver.wait_for_pending_emits()
+
+    closure_events = [
+        event for event in event_store.appended if event.type.startswith("auto.closure.")
+    ]
+    assert [event.type for event in closure_events] == [
+        "auto.closure.safe_default_entered",
+        "auto.closure.safe_default_closed",
+    ]
+    assert all(event.aggregate_type == "auto_session" for event in closure_events)
+    assert all(event.aggregate_id == state.auto_session_id for event in closure_events)
+    assert closure_events[0].data["closure_mode"] == "safe_default_no_backend"
+    assert closure_events[0].data["open_gaps"]
+    assert closure_events[1].data["closure_mode"] == "safe_default_no_backend"
+    assert closure_events[1].data["backend_start_failed"] is True
+    assert closure_events[1].data["defaulted_sections"]
+    assert result.status == "seed_ready"
+    assert state.interview_closure_mode == "safe_default_no_backend"
+
+
 # ---------------------------------------------------------------------------
 # Test 3: unsafe_context_match logs pattern_name and safe metrics
 # ---------------------------------------------------------------------------