rubricloop 0.1.4__tar.gz → 0.2.0__tar.gz

{rubricloop-0.1.4 → rubricloop-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rubricloop
-Version: 0.1.4
+Version: 0.2.0
 Summary: Bounded, auditable verification loops for AI agent output.
 Project-URL: Homepage, https://rubricloop.com
 Project-URL: Documentation, https://rubricloop.com/docs
@@ -25,6 +25,12 @@ Requires-Dist: build>=1.2; extra == 'dev'
 Requires-Dist: pytest>=8.3; extra == 'dev'
 Requires-Dist: ruff>=0.12; extra == 'dev'
 Requires-Dist: twine<8,>=7; extra == 'dev'
+Provides-Extra: judge
+Requires-Dist: openai<4,>=2; extra == 'judge'
+Provides-Extra: media
+Requires-Dist: openpyxl<4,>=3.1; extra == 'media'
+Requires-Dist: pillow<13,>=11; extra == 'media'
+Requires-Dist: python-pptx<2,>=1.0; extra == 'media'
 Description-Content-Type: text/markdown
 
 # RubricLoop SDK
@@ -32,7 +38,10 @@ Description-Content-Type: text/markdown
 RubricLoop checks AI agent output with ordinary code, returns exact failures to
 the agent, and stops when the work passes or the run hits a clear limit.
 
-The checker never calls an LLM. Your agent can use any model or provider.
+Deterministic checks never call an LLM. An optional, explicitly configured
+hybrid judge can score only the rule IDs you declare and cannot override a
+deterministic failure. Your agent and judge can use any OpenAI-compatible
+provider.
 
 Native registry packages use the deterministic, signed
 [`.rlpack` format](https://rubricloop.com/docs#package-lifecycle).
@@ -47,6 +56,14 @@ export OPENAI_API_KEY="sk-..."
 python -m rubricloop.examples.verify_reply
 ```
 
+Install only the extensions you use:
+
+```bash
+pip install "rubricloop[judge]"  # OpenAI-compatible LLM judge
+pip install "rubricloop[media]"  # image, workbook, and presentation extraction
+pip install "rubricloop[judge,media]"
+```
+
 The example deliberately produces a short first draft containing a forbidden
 promise. RubricLoop measures the failures and passes those diagnostics into the
 next model call. The process exits successfully only when the reply satisfies
@@ -163,6 +180,42 @@ its failed rules to a reviewer.
 - Local in-memory SQLite sandbox
 - Ready-made `engineering/sql-safe` pack
 - `support/refund-policy` action gate
+- Signed, fresh, release-pinned reference feeds
+- Scoped LLM-as-judge rules with token evidence and abstention
+- Decisive streaming checks with cancellation or observation modes
+- Digest-bound artifacts and deterministic media extraction
+
+## Four extension examples
+
+The wheel includes one offline, synthetic example for each extension study.
+They require no API key or customer data and are exercised by the SDK test
+suite:
+
+```bash
+python -m rubricloop.examples.reference_feed_example
+python -m rubricloop.examples.llm_judge_example
+python -m rubricloop.examples.streaming_example
+python -m rubricloop.examples.multimodal_example
+```
+
+The main extension parameters preserve the original text-only behavior:
+
+```python
+run = verify(
+    agent,
+    prompt,
+    rules,
+    judge=judge_config,       # JudgeConfig; credentials are references
+    artifact=artifact,       # Artifact or a sequence of artifacts
+    extract=extract_spec,    # ExtractSpec or a sequence of extractors
+    streaming="cut",         # False, True/"cut", or "observe"
+    constrain=True,          # Pass compiled constraints when supported
+)
+```
+
+Extension evidence is emitted only when used through `run.references`,
+`run.judge`, `run.artifacts`, `run.extractions`, `run.constraints`, and the
+matching keys in `run.to_dict()`.
 
 ## Registry CLI
 

{rubricloop-0.1.4 → rubricloop-0.2.0}/README.md

@@ -3,7 +3,10 @@
 RubricLoop checks AI agent output with ordinary code, returns exact failures to
 the agent, and stops when the work passes or the run hits a clear limit.
 
-The checker never calls an LLM. Your agent can use any model or provider.
+Deterministic checks never call an LLM. An optional, explicitly configured
+hybrid judge can score only the rule IDs you declare and cannot override a
+deterministic failure. Your agent and judge can use any OpenAI-compatible
+provider.
 
 Native registry packages use the deterministic, signed
 [`.rlpack` format](https://rubricloop.com/docs#package-lifecycle).
@@ -18,6 +21,14 @@ export OPENAI_API_KEY="sk-..."
 python -m rubricloop.examples.verify_reply
 ```
 
+Install only the extensions you use:
+
+```bash
+pip install "rubricloop[judge]"  # OpenAI-compatible LLM judge
+pip install "rubricloop[media]"  # image, workbook, and presentation extraction
+pip install "rubricloop[judge,media]"
+```
+
 The example deliberately produces a short first draft containing a forbidden
 promise. RubricLoop measures the failures and passes those diagnostics into the
 next model call. The process exits successfully only when the reply satisfies
@@ -134,6 +145,42 @@ its failed rules to a reviewer.
 - Local in-memory SQLite sandbox
 - Ready-made `engineering/sql-safe` pack
 - `support/refund-policy` action gate
+- Signed, fresh, release-pinned reference feeds
+- Scoped LLM-as-judge rules with token evidence and abstention
+- Decisive streaming checks with cancellation or observation modes
+- Digest-bound artifacts and deterministic media extraction
+
+## Four extension examples
+
+The wheel includes one offline, synthetic example for each extension study.
+They require no API key or customer data and are exercised by the SDK test
+suite:
+
+```bash
+python -m rubricloop.examples.reference_feed_example
+python -m rubricloop.examples.llm_judge_example
+python -m rubricloop.examples.streaming_example
+python -m rubricloop.examples.multimodal_example
+```
+
+The main extension parameters preserve the original text-only behavior:
+
+```python
+run = verify(
+    agent,
+    prompt,
+    rules,
+    judge=judge_config,       # JudgeConfig; credentials are references
+    artifact=artifact,       # Artifact or a sequence of artifacts
+    extract=extract_spec,    # ExtractSpec or a sequence of extractors
+    streaming="cut",         # False, True/"cut", or "observe"
+    constrain=True,          # Pass compiled constraints when supported
+)
+```
+
+Extension evidence is emitted only when used through `run.references`,
+`run.judge`, `run.artifacts`, `run.extractions`, `run.constraints`, and the
+matching keys in `run.to_dict()`.
 
 ## Registry CLI
 

{rubricloop-0.1.4 → rubricloop-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "rubricloop"
-version = "0.1.4"
+version = "0.2.0"
 description = "Bounded, auditable verification loops for AI agent output."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -35,6 +35,14 @@ Documentation = "https://rubricloop.com/docs"
 rubricloop = "rubricloop.cli:main"
 
 [project.optional-dependencies]
+judge = [
+  "openai>=2,<4",
+]
+media = [
+  "openpyxl>=3.1,<4",
+  "Pillow>=11,<13",
+  "python-pptx>=1.0,<2",
+]
 dev = [
   "build>=1.2",
   "pytest>=8.3",

rubricloop-0.2.0/src/rubricloop/__init__.py

@@ -0,0 +1,101 @@
+"""RubricLoop public API."""
+
+from .compile import (
+    BACKEND_SUPPORT,
+    ConstrainedAgentAdapter,
+    HostedConstrainedAgentAdapter,
+    VllmConstrainedAgentAdapter,
+    compile_constraints,
+)
+from .credentials import resolve_credential
+from .loop import build_feedback, verify, verify_artifact
+from .references import resolve_reference
+from .rlpack import (
+    ArchiveLimits,
+    Ed25519Keyring,
+    Ed25519Signer,
+    Rlpack,
+    RlpackError,
+    SignatureError,
+    build_rlpack,
+    canonical_json,
+    create_manifest,
+    read_rlpack,
+    validate_manifest,
+)
+from .rubric import CheckContext, Rubric, Rule, rubric
+from .streaming import StreamGuard, streamability_report
+from .types import (
+    Abstention,
+    AgentRequest,
+    AgentResponse,
+    Artifact,
+    Budget,
+    CheckResult,
+    CompiledConstraints,
+    ExtractionRecord,
+    ExtractionRegion,
+    ExtractSpec,
+    Failure,
+    IterationRecord,
+    JudgeCallRecord,
+    JudgeConfig,
+    ReferenceFailure,
+    ReferenceSnapshot,
+    ReferenceSpec,
+    RunResult,
+    StreamHit,
+    Verdict,
+)
+
+__all__ = [
+    "Abstention",
+    "AgentRequest",
+    "AgentResponse",
+    "Artifact",
+    "ArchiveLimits",
+    "Budget",
+    "BACKEND_SUPPORT",
+    "Ed25519Keyring",
+    "Ed25519Signer",
+    "Rlpack",
+    "RlpackError",
+    "SignatureError",
+    "CheckContext",
+    "CheckResult",
+    "CompiledConstraints",
+    "ConstrainedAgentAdapter",
+    "HostedConstrainedAgentAdapter",
+    "ExtractSpec",
+    "ExtractionRecord",
+    "ExtractionRegion",
+    "Failure",
+    "IterationRecord",
+    "JudgeCallRecord",
+    "JudgeConfig",
+    "ReferenceFailure",
+    "ReferenceSnapshot",
+    "ReferenceSpec",
+    "Rubric",
+    "Rule",
+    "RunResult",
+    "StreamHit",
+    "StreamGuard",
+    "streamability_report",
+    "Verdict",
+    "VllmConstrainedAgentAdapter",
+    "build_rlpack",
+    "build_feedback",
+    "canonical_json",
+    "compile_constraints",
+    "create_manifest",
+    "read_rlpack",
+    "resolve_credential",
+    "resolve_reference",
+    "rubric",
+    "validate_manifest",
+    "verify",
+    "verify_artifact",
+]
+
+__version__ = "0.2.0"

rubricloop-0.2.0/src/rubricloop/artifacts.py

@@ -0,0 +1,149 @@
+"""Artifact integrity and bounded local extraction helpers."""
+
+from __future__ import annotations
+
+import hashlib
+import io
+import json
+from collections.abc import Iterable
+from pathlib import Path
+from typing import Any
+
+from .types import Artifact, ExtractionRecord, ExtractionRegion, ExtractSpec
+
+MAX_ARTIFACT_BYTES = 20 * 1024 * 1024
+
+
+def prepare_artifacts(
+    value: Artifact | Iterable[Artifact] | None,
+) -> tuple[tuple[Artifact, ...], dict[str, bytes]]:
+    if value is None:
+        return (), {}
+    artifacts = (value,) if isinstance(value, Artifact) else tuple(value)
+    names: set[str] = set()
+    payloads: dict[str, bytes] = {}
+    for artifact in artifacts:
+        if artifact.name in names:
+            raise ValueError(f"duplicate artifact name: {artifact.name}")
+        names.add(artifact.name)
+        payload = read_artifact(artifact)
+        payloads[artifact.name] = payload
+    return artifacts, payloads
+
+
+def read_artifact(artifact: Artifact) -> bytes:
+    source = artifact.source
+    if isinstance(source, bytes):
+        payload = source
+    elif isinstance(source, (str, Path)):
+        path = Path(source)
+        if path.stat().st_size > MAX_ARTIFACT_BYTES:
+            raise ValueError(f"artifact {artifact.name} exceeds the size limit")
+        payload = path.read_bytes()
+    elif hasattr(source, "read"):
+        position = source.tell() if hasattr(source, "tell") else None
+        payload = source.read(MAX_ARTIFACT_BYTES + 1)
+        if position is not None and hasattr(source, "seek"):
+            source.seek(position)
+    else:
+        raise TypeError(f"artifact {artifact.name} source must be a path or file-like")
+    if not isinstance(payload, bytes):
+        raise TypeError(f"artifact {artifact.name} source did not return bytes")
+    if len(payload) > MAX_ARTIFACT_BYTES:
+        raise ValueError(f"artifact {artifact.name} exceeds the size limit")
+    if len(payload) != artifact.size_bytes:
+        raise ValueError(f"artifact {artifact.name} size does not match")
+    digest = "sha256:" + hashlib.sha256(payload).hexdigest()
+    if digest != artifact.sha256:
+        raise ValueError(f"artifact {artifact.name} digest does not match")
+    return payload
+
+
+def run_extractions(
+    specs: Iterable[ExtractSpec],
+    artifacts: tuple[Artifact, ...],
+    payloads: dict[str, bytes],
+) -> tuple[dict[str, ExtractionRecord], dict[str, Any]]:
+    records: dict[str, ExtractionRecord] = {}
+    evidence: dict[str, Any] = {}
+    by_name = {artifact.name: artifact for artifact in artifacts}
+    for spec in specs:
+        extraction_id = spec.id or spec.engine
+        try:
+            artifact = _select_artifact(spec, artifacts, by_name)
+            content, regions = _extract(spec, artifact, payloads[artifact.name])
+            record = ExtractionRecord(
+                engine=spec.engine,
+                version=spec.version,
+                config_sha256=_digest(dict(spec.config)),
+                content_sha256=_digest(content),
+                regions=regions,
+                content=content,
+            )
+            records[extraction_id] = record
+            evidence[extraction_id] = {
+                **record.evidence(),
+                "artifact": artifact.name,
+            }
+        except Exception as exc:
+            evidence[extraction_id] = {
+                "status": "unavailable",
+                "reason": f"{type(exc).__name__}: {str(exc)[:180]}",
+            }
+    return records, evidence
+
+
+def _select_artifact(
+    spec: ExtractSpec,
+    artifacts: tuple[Artifact, ...],
+    by_name: dict[str, Artifact],
+) -> Artifact:
+    if spec.artifact is not None:
+        if spec.artifact not in by_name:
+            raise ValueError(f"artifact {spec.artifact} is unavailable")
+        return by_name[spec.artifact]
+    if len(artifacts) != 1:
+        raise ValueError("extraction must name an artifact when the run has multiple")
+    return artifacts[0]
+
+
+def _extract(
+    spec: ExtractSpec, artifact: Artifact, payload: bytes
+) -> tuple[Any, tuple[ExtractionRegion, ...]]:
+    if spec.engine == "provided":
+        if "content" not in spec.config:
+            raise ValueError("provided extraction requires config.content")
+        return spec.config["content"], ()
+    if spec.engine == "image.metadata":
+        try:
+            from PIL import Image
+        except ImportError as exc:
+            raise RuntimeError(
+                "install rubricloop[media] to extract image metadata"
+            ) from exc
+        with Image.open(io.BytesIO(payload)) as image:
+            content = {
+                "width": image.width,
+                "height": image.height,
+                "mode": image.mode,
+                "format": image.format,
+            }
+        return content, (
+            ExtractionRegion(
+                id=f"image:{artifact.name}",
+                content=content,
+                confidence=1.0,
+            ),
+        )
+    raise ValueError(f"unsupported local extraction engine: {spec.engine}")
+
+
+def _digest(value: Any) -> str:
+    encoded = json.dumps(
+        value,
+        sort_keys=True,
+        separators=(",", ":"),
+        ensure_ascii=False,
+        allow_nan=False,
+    ).encode()
+    return "sha256:" + hashlib.sha256(encoded).hexdigest()

{rubricloop-0.1.4 → rubricloop-0.2.0}/src/rubricloop/checks/__init__.py

@@ -2,13 +2,27 @@
 
 from .common import (
     abstain,
+    citation_lookup,
     forbidden_terms,
+    json_field_enum,
     json_schema,
     pii_free,
     required_terms,
     valid_json,
     word_count,
 )
+from .media import (
+    EnvironmentResolver,
+    audio_script_adherence,
+    chart_data_match,
+    cross_modal_match,
+    layout_compliance,
+    media_metric,
+    media_pii_screen,
+    render_check,
+    synthetic_media_flag,
+    table_arithmetic,
+)
 from .sql import (
     executes,
     matches_reference,
@@ -21,10 +35,18 @@ from .sql import (
 
 __all__ = [
     "abstain",
+    "audio_script_adherence",
+    "chart_data_match",
+    "citation_lookup",
     "executes",
     "forbidden_terms",
+    "cross_modal_match",
+    "json_field_enum",
     "json_schema",
     "matches_reference",
+    "layout_compliance",
+    "media_metric",
+    "media_pii_screen",
     "no_select_star",
     "parses",
     "pii_free",
@@ -32,6 +54,10 @@ __all__ = [
     "required_columns",
     "required_terms",
     "schema_conformance",
+    "render_check",
+    "synthetic_media_flag",
+    "table_arithmetic",
     "valid_json",
     "word_count",
+    "EnvironmentResolver",
 ]