superqode 2.3.2__tar.gz → 2.4.0__tar.gz

{superqode-2.3.2 → superqode-2.4.0}/CHANGELOG.md

@@ -5,7 +5,66 @@ All notable changes to SuperQode will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [2.4.0] - 2026-09-18
+
+### Jev integration
+
+SuperQode 2.4.0 adds a general decision harness for Jev, native tool-permission
+checks, labelled decision evaluations, and opt-in rubric grading. Coding models
+continue to generate code; Jev supplies typed decisions that SuperQode validates
+and applies through explicit policies.
+
+### Added
+
+- **General decision harness.** The `systemone` backend evaluates reviewed
+  Choice, Score, and Noul question packs. Input schemas, confidence thresholds,
+  abstention, and pack hashes make each decision inspectable and reproducible.
+  Examples cover ticket triage, factory-route suggestions, and tool permissions.
+- **Native tool-permission checks.** `:systemone live` enables Jev checks in
+  Core/BYOK sessions. Hard policy denials remain authoritative; model ASK uses
+  the approval flow even for otherwise auto-allowed tools. Client failures are
+  visible and fall back to the existing permission policy.
+- **Direct decision sessions.** `:systemone connect <pack>` and
+  `:connect systemone <pack>` connect the TUI directly to Jev without a coding
+  provider. Status distinguishes a decision session from the tool-check sidecar.
+- **Labelled evaluations.** `harness eval` supports exact typed output labels
+  and explicit evaluator results. Scorecards include coverage, accuracy among
+  graded cases, abstentions, errors, dataset hashes, and decision evidence.
+  Synthetic routing and tool-permission starter datasets include held-in and
+  held-out splits.
+- **Rubric grading.** `SUPERQODE_RUBRIC_GRADER=systemone` uses Jev verdicts in
+  the native rubric revision loop. A `jev_rubric` task evaluator judges another
+  harness's response. Headless JSON exposes `rubric_result`.
+- **Transport and observability.** Live decisions report returned model, HTTP
+  status, latency, and available token usage. Configurable compatible endpoints,
+  stub clients, sanitized recordings, and replay support development and testing.
+
+### Changed
+
+- Invalid, uncertain, or unavailable rubric judgments are explicitly `ungraded`.
+  Utility-grader errors no longer count as `satisfied`. Headless runs with an
+  unsatisfied or ungraded rubric report `success: false` and exit with code 2.
+- Legacy evaluation tasks retain their non-empty smoke check, now identified as
+  `smoke_only`. Use labelled or rubric evaluators for stronger assessment.
+
+### Reliability
+
+- Tool-gate pack 1.1.0 requires low destructive and exfiltration risk before
+  auto-approval. Uncertain risk cannot produce ALLOW.
+- Live criteria serialization matches the API contract. Evaluation deadlines
+  include retries, and recording failures do not discard valid decisions.
+- Input redaction covers common credential fields, headers, command flags,
+  URL credentials, and private-key blocks. Offline policies remain respected.
+
+### Scope
+
+Jev integration is opt-in and requires separate TypeSafe credentials for live
+use. The native gate does not intercept external agent runtimes. Route decisions
+are suggestions; automatic factory routing and automatic LLM fallback are not
+included. Starter datasets are examples, not published accuracy benchmarks.
+
+See [the Jev integration guide](docs/advanced/systemone.md) and
+[2.4.0 release notes](docs/advanced/release-2.4.0.md) for setup and migration.
 
 ## [2.3.2] - 2026-09-13
 

{superqode-2.3.2 → superqode-2.4.0}/MANIFEST.in

@@ -6,3 +6,4 @@ include scripts/check_release_metadata.py
 recursive-include examples *.md *.py *.yaml
 recursive-include src/superqode/data/eval_fixtures *.log
 recursive-include src/superqode/data/eval_packs *.yaml
+recursive-include src/superqode/systemone/packs *.yaml

{superqode-2.3.2 → superqode-2.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: superqode
-Version: 2.3.2
+Version: 2.4.0
 Summary: The harness layer for coding agents
 Author-email: Shashi Jagtap <info@super-agentic.ai>
 Maintainer-email: Shashi Jagtap <info@super-agentic.ai>
@@ -34,6 +34,7 @@ Requires-Dist: anyio>=4.0.0
 Requires-Dist: httpx>=0.28.1
 Requires-Dist: httpx-sse>=0.4.3
 Requires-Dist: jsonschema>=4.20.0
+Requires-Dist: pydantic<3,>=2.0
 Requires-Dist: python-frontmatter>=1.1.0
 Requires-Dist: superopt>=0.1.1
 Requires-Dist: prime-agent-python-client<0.3.0,>=0.2.0

superqode-2.4.0/examples/harnesses/systemone-factory-route.yaml

@@ -0,0 +1,12 @@
+version: 1
+name: route-decision
+description: Suggest an existing factory route without executing it.
+flavor: decision
+runtime:
+  backend: systemone
+systemone:
+  enabled: true
+  client: live
+  pack: factory_route
+metadata:
+  example: true

superqode-2.4.0/examples/harnesses/systemone-ticket-triage.yaml

@@ -0,0 +1,13 @@
+version: 1
+name: ticket-triage
+description: Classify support requests with a general System One decision pack.
+flavor: decision
+runtime:
+  backend: systemone
+systemone:
+  enabled: true
+  client: live
+  model: jev-1.13.0
+  pack: ../systemone/ticket-triage-pack.yaml
+metadata:
+  example: true

superqode-2.4.0/examples/harnesses/systemone-tool-gate.yaml

@@ -0,0 +1,16 @@
+version: 1
+name: systemone-tool-gate
+description: Evaluate the frozen tool_gate pack over structured state.
+flavor: decision
+runtime:
+  backend: systemone
+systemone:
+  enabled: true
+  client: live
+  pack: tool_gate
+  model: jev-1.13.0
+  timeout_ms: 5000
+  airplane: skip
+
+metadata:
+  example: true

superqode-2.4.0/examples/systemone/rubric-eval.yaml

@@ -0,0 +1,9 @@
+metadata:
+  description: Example model judgment; does not establish that tests actually ran.
+tasks:
+  - id: explain-approval
+    split: held-out
+    prompt: Explain when a tool needs human approval and what happens when its permission classifier is unavailable.
+    evaluator:
+      type: jev_rubric
+      rubric: The answer explains that uncertain tool decisions can require human approval and distinguishes classifier errors from approval.

superqode-2.4.0/examples/systemone/ticket-triage-pack.yaml

@@ -0,0 +1,30 @@
+id: ticket_triage
+version: "1.0.0"
+description: Classify a support request, score severity, and flag urgency.
+state_schema:
+  type: string
+  minLength: 1
+questions:
+  department:
+    type: choice
+    instructions: Which department should handle this support request?
+    criteria:
+      billing: Payments, invoices, refunds, and subscriptions.
+      technical: Bugs, broken functionality, and outages.
+      sales: Pricing, demos, and purchase enquiries.
+      other: Requests that do not fit the listed departments.
+  severity:
+    type: score
+    instructions: How severe is the reported problem?
+    criteria:
+      - Informational question with no broken functionality.
+      - Minor inconvenience with a workaround.
+      - Important functionality is blocked.
+      - Critical outage affecting the whole service.
+  urgent:
+    type: noul
+    instructions: Does the request explicitly require immediate attention?
+decision_policy:
+  min_confidence: 0.75
+  noul_false_max: 0.2
+  noul_true_min: 0.8

{superqode-2.3.2 → superqode-2.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "superqode"
-version = "2.3.2"
+version = "2.4.0"
 description = "The harness layer for coding agents"
 readme = "README.md"
 license = "Apache-2.0"
@@ -49,6 +49,9 @@ dependencies = [
     # execution, the way pi validates against TypeBox. Already present through
     # mcp/fastmcp; declared because superqode.pipy imports it directly.
     "jsonschema>=4.20.0",
+    # Typed System One packs and answers. Already imported by harness/output.py;
+    # declared because superqode.systemone uses it as the I/O contract.
+    "pydantic>=2.0,<3",
     "python-frontmatter>=1.1.0",
     "superopt>=0.1.1",
     # Native asyncio host for Prime Agent's public RPC mode. Kept as a small,
@@ -270,6 +273,7 @@ superqode = [
     "data/*.yaml",
     "data/eval_fixtures/*.log",
     "data/eval_packs/*.yaml",
+    "systemone/packs/*.yaml",
     "local/data/*.yaml",
     "local/data/model-packs/*.yaml",
     "py.typed",

{superqode-2.3.2 → superqode-2.4.0}/src/superqode/__init__.py

@@ -38,7 +38,7 @@ __all__ = [
     "sidebar",
 ]
 
-__version__ = "2.3.2"
+__version__ = "2.4.0"
 
 # Stable, lightweight public extension surface.  Importing this package does
 # not discover or execute third-party extensions; discovery happens only when

{superqode-2.3.2 → superqode-2.4.0}/src/superqode/agent/loop.py

@@ -658,6 +658,9 @@ class AgentConfig:
     harness_sandbox_backend: str = "local"
     harness_delegation_depth: int = 0
 
+    # Opt-in System One tool gate. None means "read from harness_spec / env".
+    systemone: Optional[Any] = None
+
 
 @dataclass
 class AgentMessage:
@@ -747,6 +750,7 @@ class AgentResponse:
     # and any schema-validation errors (empty list means valid).
     structured_output: Optional[Any] = None
     schema_errors: Optional[List[str]] = None
+    rubric_result: Optional[Dict[str, Any]] = None
     input_tokens: Optional[int] = None
     output_tokens: Optional[int] = None
     total_tokens: Optional[int] = None
@@ -798,6 +802,8 @@ class AgentLoop:
         permission_manager: Optional[PermissionManager] = None,
         hooks: Optional["HookRegistry"] = None,  # Lifecycle hooks (before/after llm/tool/turn)
         allow_peer_agents: bool = True,  # False inside sub/peer agents (no nesting)
+        systemone_client: Optional[Any] = None,
+        on_systemone: Optional[Callable[[Dict[str, Any]], None]] = None,
     ):
         self.gateway = gateway
         self.tools = tools
@@ -891,6 +897,8 @@ class AgentLoop:
         from .hooks import HookRegistry
 
         self.hooks: HookRegistry = hooks if hooks is not None else HookRegistry()
+        self._systemone_client = systemone_client
+        self.on_systemone = on_systemone
         self._current_iteration: int = 0
         # session_start fires once per AgentLoop instance, on the first run().
         self._session_started: bool = False
@@ -999,6 +1007,7 @@ class AgentLoop:
             on_tool_call=self.on_tool_call,
             on_tool_result=self.on_tool_result,
             on_thinking=self.on_thinking,
+            on_systemone=getattr(self, "on_systemone", None),
             parallel_tools=self.parallel_tools,
             mcp_executor=self.mcp_executor,
             mcp_tools=self._mcp_tools,
@@ -1315,6 +1324,7 @@ class AgentLoop:
             on_tool_call=self.on_tool_call,
             on_tool_result=self.on_tool_result,
             on_thinking=self.on_thinking,
+            on_systemone=getattr(self, "on_systemone", None),
             parallel_tools=self.parallel_tools,
             mcp_executor=self.mcp_executor,
             mcp_tools=self._mcp_tools,
@@ -1357,8 +1367,12 @@ class AgentLoop:
            otherwise-ASK command.
         3. The manager's own **deny** (e.g. dangerous-command guards) still wins
            over a permissive hook or rule - hard safety rules are not overridable.
-        4. Manager **allow** passes through; otherwise (ASK) we honour a hook or
-           rule allow, or fall back to the pause/prompt flow.
+        4. Opt-in System One may **deny** an otherwise-allowed call, or
+           **allow** an otherwise-ASK call. Model ASK requires approval.
+           Unavailable evaluations visibly fall back to the existing policy.
+           YAML and manager denials never call the client.
+        5. Manager **allow** passes through; otherwise (ASK) we honour a hook,
+           rule, or System One allow, or fall back to the pause/prompt flow.
         """
         from .hooks import PERMISSION_REQUEST
 
@@ -1405,11 +1419,18 @@ class AgentLoop:
                 error=f"Permission denied for tool: {name}",
                 metadata={"permission": "deny", "tool": name},
             )
-        if permission == Permission.ALLOW and not rule_ask:
+
+        from ..systemone.runtime import apply_systemone_gate
+
+        so_denied, so_allow, so_ask = await apply_systemone_gate(self, name, arguments)
+        if so_denied is not None:
+            return so_denied
+
+        if permission == Permission.ALLOW and not rule_ask and not so_ask:
             return None
         if tool_call_id and tool_call_id in self._approved_tool_call_ids:
             return None
-        if (verdict.allowed or rule_allow) and not rule_ask:
+        if (verdict.allowed or rule_allow or so_allow) and not rule_ask and not so_ask:
             return None
 
         if self.pause_on_approval:
@@ -2275,6 +2296,7 @@ class AgentLoop:
         auto_continues = 0
         length_parts: List[str] = []
         rubric_rounds = 0
+        rubric_result = None
 
         # Emit initial processing log
         if self.on_thinking:
@@ -2626,6 +2648,7 @@ class AgentLoop:
                 if self.config.rubric and rubric_rounds < max(0, self.config.max_rubric_rounds):
                     from .rubric import grade_against_rubric
 
+                    rubric_evidence = {}
                     verdict, feedback = await grade_against_rubric(
                         messages,
                         response_content,
@@ -2633,7 +2656,10 @@ class AgentLoop:
                         self.gateway,
                         self.config.provider,
                         self.config.model,
+                        spec=self.config.harness_spec,
+                        on_result=lambda result: rubric_evidence.update(result),
                     )
+                    rubric_result = {**rubric_evidence, "verdict": verdict, "feedback": feedback}
                     if verdict == "needs_revision" and feedback:
                         rubric_rounds += 1
                         if self.on_thinking:
@@ -2652,9 +2678,11 @@ class AgentLoop:
                             )
                         )
                         continue
-                    if verdict == "failed" and self.on_thinking:
-                        await self.on_thinking(f"Rubric review: failed - {feedback[:120]}")
+                    if verdict in {"failed", "ungraded"} and self.on_thinking:
+                        await self.on_thinking(f"Rubric review: {verdict} - {feedback[:120]}")
 
+                if self.config.rubric and rubric_rounds >= max(0, self.config.max_rubric_rounds):
+                    rubric_result = {"verdict": "ungraded", "reason": "Rubric round limit reached"}
                 if self.on_thinking:
                     await self.on_thinking("Response complete")
                 from .hooks import AFTER_TURN_COMPLETE
@@ -2675,6 +2703,7 @@ class AgentLoop:
                     tool_calls_made=tool_calls_made,
                     iterations=iterations,
                     stopped_reason="complete",
+                    rubric_result=rubric_result,
                 )
                 return await _finish(final)
 
@@ -3123,6 +3152,8 @@ class AgentLoop:
                 # No tool calls - we have the final response
                 # If we had tool calls in previous iterations but no content now,
                 # the model should still provide a summary
+                if self.config.rubric and rubric_rounds >= max(0, self.config.max_rubric_rounds):
+                    rubric_result = {"verdict": "ungraded", "reason": "Rubric round limit reached"}
                 if self.on_thinking:
                     await self.on_thinking("Response complete")
                 if self._session_manager and full_content.strip():

{superqode-2.3.2 → superqode-2.4.0}/src/superqode/agent/rubric.py

@@ -12,6 +12,7 @@ insurance on long unattended runs.
 from __future__ import annotations
 
 import json
+import os
 import re
 from typing import Any, List, Tuple
 
@@ -49,24 +50,22 @@ def _transcript_tail(messages: List[Any], final_content: str) -> str:
 
 
 def parse_grader_response(raw: str) -> Tuple[str, str]:
-    """Parse the grader's JSON; unparseable output counts as satisfied.
-
-    Failing open is deliberate: a flaky grader must never trap a run in
-    revision loops.
-    """
+    """Parse a verdict; malformed output is explicitly ungraded."""
     text = (raw or "").strip()
     text = re.sub(r"^```[a-zA-Z]*\s*\n|\n```\s*$", "", text)
     start = text.find("{")
     end = text.rfind("}")
     if start < 0 or end <= start:
-        return "satisfied", ""
+        return "ungraded", "Rubric grader did not return a valid verdict"
     try:
         data = json.loads(text[start : end + 1])
     except (json.JSONDecodeError, ValueError):
-        return "satisfied", ""
+        return "ungraded", "Rubric grader did not return a valid verdict"
+    if not isinstance(data, dict):
+        return "ungraded", "Rubric grader did not return an object"
     verdict = str(data.get("verdict", "")).strip().lower()
     if verdict not in VERDICTS:
-        return "satisfied", ""
+        return "ungraded", "Rubric grader did not return a valid verdict"
     return verdict, str(data.get("feedback", "")).strip()
 
 
@@ -77,12 +76,28 @@ async def grade_against_rubric(
     gateway: Any,
     provider: str,
     model: str,
+    *,
+    spec: Any = None,
+    on_result: Any = None,
 ) -> Tuple[str, str]:
-    """Run one grader call. Returns (verdict, feedback); fails open to satisfied.
+    """Grade with the utility model or opt-in Jev; uncertainty stays ungraded.
 
     Grading is a utility call: SUPERQODE_UTILITY_PROVIDER can route it to a
     cheaper model (including the on-device apple-fm) instead of the session model.
     """
+    if os.environ.get("SUPERQODE_RUBRIC_GRADER", "").lower() == "systemone":
+        result = await evaluate_jev_rubric(
+            rubric, _transcript_tail(messages, final_content), spec=spec
+        )
+        if on_result:
+            on_result(result)
+        verdict = result["verdict"]
+        if verdict == "needs_revision":
+            return (
+                verdict,
+                f"Re-check each requirement and supply missing evidence. Rubric: {rubric}",
+            )
+        return verdict, result["reason"]
     try:
         from .utility_model import utility_completion
 
@@ -99,7 +114,46 @@ async def grade_against_rubric(
         )
         return parse_grader_response(raw)
     except Exception:
-        return "satisfied", ""
+        return "ungraded", "Rubric grader did not return a valid verdict"
 
 
 __all__ = ["GRADER_SYSTEM_PROMPT", "VERDICTS", "grade_against_rubric", "parse_grader_response"]
+
+
+async def evaluate_jev_rubric(rubric: str, work: str, *, spec=None, client=None) -> dict[str, Any]:
+    """One bounded decision; never turn abstention or client failure into success."""
+    from dataclasses import replace
+
+    from superqode.systemone.config import build_client, resolve_systemone
+    from superqode.systemone.decision import evaluate_decision
+    from superqode.systemone.pack import load_pack
+
+    try:
+        # Inherit endpoint, model, deadline and offline policy. A separate opt-in
+        # enables grading without enabling the permission sidecar.
+        from types import SimpleNamespace
+
+        declared = getattr(spec, "systemone", None)
+        if declared is None or not getattr(declared, "enabled", False):
+            declared = SimpleNamespace(enabled=True, client="live")
+        else:
+            declared = SimpleNamespace(**{**vars(declared), "enabled": True})
+        settings = resolve_systemone(spec=spec, explicit=declared)
+        settings = replace(settings, pack="rubric")
+        active = build_client(settings, injected=client)
+        if active is None:
+            return {
+                "verdict": "ungraded",
+                "reason": "Rubric client unavailable: " + (settings.skip_reason or "disabled"),
+            }
+        decision = await evaluate_decision(
+            active, {"rubric": rubric, "work": work}, load_pack("rubric")
+        )
+        verdict = decision.outputs.get("verdict") if decision.status == "decided" else "ungraded"
+        return {
+            "verdict": verdict or "ungraded",
+            "reason": "Jev rubric: " + str(verdict or "ungraded"),
+            "decision": decision.model_dump(mode="json"),
+        }
+    except Exception:
+        return {"verdict": "ungraded", "reason": "Rubric evaluation unavailable"}

{superqode-2.3.2 → superqode-2.4.0}/src/superqode/app/constants.py

@@ -323,6 +323,12 @@ COMMANDS = [
     ":capabilities",
     ":eval",
     ":evals",
+    ":systemone",
+    ":systemone live",
+    ":systemone off",
+    ":systemone packs",
+    ":systemone connect",
+    ":connect systemone",
     ":tour",
     ":tour next",
     ":connect codex",

{superqode-2.3.2 → superqode-2.4.0}/src/superqode/app/mixins/helper_permissions.py

@@ -41,6 +41,9 @@ class HelperPermissionsMixin:
             return self._request_runtime_permission(tool_name, arguments, log)
 
         pure.on_permission_request = on_permission_request
+        from superqode.systemone.runtime import format_decision
+
+        pure.on_systemone = lambda event: self._call_ui(log.add_system, format_decision(event))
 
     def _announce_pending_approvals(self, source, log) -> None:
         """Surface pending approvals from an active runtime or HarnessSpec session."""