rubricloop 0.1.3__tar.gz → 0.2.0__tar.gz

{rubricloop-0.1.3 → rubricloop-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rubricloop
-Version: 0.1.3
+Version: 0.2.0
 Summary: Bounded, auditable verification loops for AI agent output.
 Project-URL: Homepage, https://rubricloop.com
 Project-URL: Documentation, https://rubricloop.com/docs
@@ -25,6 +25,12 @@ Requires-Dist: build>=1.2; extra == 'dev'
 Requires-Dist: pytest>=8.3; extra == 'dev'
 Requires-Dist: ruff>=0.12; extra == 'dev'
 Requires-Dist: twine<8,>=7; extra == 'dev'
+Provides-Extra: judge
+Requires-Dist: openai<4,>=2; extra == 'judge'
+Provides-Extra: media
+Requires-Dist: openpyxl<4,>=3.1; extra == 'media'
+Requires-Dist: pillow<13,>=11; extra == 'media'
+Requires-Dist: python-pptx<2,>=1.0; extra == 'media'
 Description-Content-Type: text/markdown
 
 # RubricLoop SDK
@@ -32,7 +38,10 @@ Description-Content-Type: text/markdown
 RubricLoop checks AI agent output with ordinary code, returns exact failures to
 the agent, and stops when the work passes or the run hits a clear limit.
 
-The checker never calls an LLM. Your agent can use any model or provider.
+Deterministic checks never call an LLM. An optional, explicitly configured
+hybrid judge can score only the rule IDs you declare and cannot override a
+deterministic failure. Your agent and judge can use any OpenAI-compatible
+provider.
 
 Native registry packages use the deterministic, signed
 [`.rlpack` format](https://rubricloop.com/docs#package-lifecycle).
@@ -47,6 +56,14 @@ export OPENAI_API_KEY="sk-..."
 python -m rubricloop.examples.verify_reply
 ```
 
+Install only the extensions you use:
+
+```bash
+pip install "rubricloop[judge]"  # OpenAI-compatible LLM judge
+pip install "rubricloop[media]"  # image, workbook, and presentation extraction
+pip install "rubricloop[judge,media]"
+```
+
 The example deliberately produces a short first draft containing a forbidden
 promise. RubricLoop measures the failures and passes those diagnostics into the
 next model call. The process exits successfully only when the reply satisfies
@@ -163,6 +180,42 @@ its failed rules to a reviewer.
 - Local in-memory SQLite sandbox
 - Ready-made `engineering/sql-safe` pack
 - `support/refund-policy` action gate
+- Signed, fresh, release-pinned reference feeds
+- Scoped LLM-as-judge rules with token evidence and abstention
+- Decisive streaming checks with cancellation or observation modes
+- Digest-bound artifacts and deterministic media extraction
+
+## Four extension examples
+
+The wheel includes one offline, synthetic example for each extension study.
+They require no API key or customer data and are exercised by the SDK test
+suite:
+
+```bash
+python -m rubricloop.examples.reference_feed_example
+python -m rubricloop.examples.llm_judge_example
+python -m rubricloop.examples.streaming_example
+python -m rubricloop.examples.multimodal_example
+```
+
+The main extension parameters preserve the original text-only behavior:
+
+```python
+run = verify(
+    agent,
+    prompt,
+    rules,
+    judge=judge_config,       # JudgeConfig; credentials are references
+    artifact=artifact,       # Artifact or a sequence of artifacts
+    extract=extract_spec,    # ExtractSpec or a sequence of extractors
+    streaming="cut",         # False, True/"cut", or "observe"
+    constrain=True,          # Pass compiled constraints when supported
+)
+```
+
+Extension evidence is emitted only when used through `run.references`,
+`run.judge`, `run.artifacts`, `run.extractions`, `run.constraints`, and the
+matching keys in `run.to_dict()`.
 
 ## Registry CLI
 
@@ -179,11 +232,11 @@ When the package is ready to share, create an API key in the
 hosted registry:
 
 ```bash
-rubricloop --server https://api.rubricloop.dev login --api-key "$RUBRICLOOP_API_KEY"
-rubricloop --server https://api.rubricloop.dev create acme/check --visibility private
-rubricloop --server https://api.rubricloop.dev push dist/acme-check.rlpack --tag latest
-rubricloop --server https://api.rubricloop.dev vet acme/check@latest
-rubricloop --server https://api.rubricloop.dev pull acme/check@latest
+rubricloop login --api-key "$RUBRICLOOP_API_KEY"
+rubricloop create acme/check --visibility private
+rubricloop push dist/acme-check.rlpack --tag latest
+rubricloop vet acme/check@latest
+rubricloop pull acme/check@latest
 ```
 
 Public community packages are free. Private repositories are available only to

{rubricloop-0.1.3 → rubricloop-0.2.0}/README.md

@@ -3,7 +3,10 @@
 RubricLoop checks AI agent output with ordinary code, returns exact failures to
 the agent, and stops when the work passes or the run hits a clear limit.
 
-The checker never calls an LLM. Your agent can use any model or provider.
+Deterministic checks never call an LLM. An optional, explicitly configured
+hybrid judge can score only the rule IDs you declare and cannot override a
+deterministic failure. Your agent and judge can use any OpenAI-compatible
+provider.
 
 Native registry packages use the deterministic, signed
 [`.rlpack` format](https://rubricloop.com/docs#package-lifecycle).
@@ -18,6 +21,14 @@ export OPENAI_API_KEY="sk-..."
 python -m rubricloop.examples.verify_reply
 ```
 
+Install only the extensions you use:
+
+```bash
+pip install "rubricloop[judge]"  # OpenAI-compatible LLM judge
+pip install "rubricloop[media]"  # image, workbook, and presentation extraction
+pip install "rubricloop[judge,media]"
+```
+
 The example deliberately produces a short first draft containing a forbidden
 promise. RubricLoop measures the failures and passes those diagnostics into the
 next model call. The process exits successfully only when the reply satisfies
@@ -134,6 +145,42 @@ its failed rules to a reviewer.
 - Local in-memory SQLite sandbox
 - Ready-made `engineering/sql-safe` pack
 - `support/refund-policy` action gate
+- Signed, fresh, release-pinned reference feeds
+- Scoped LLM-as-judge rules with token evidence and abstention
+- Decisive streaming checks with cancellation or observation modes
+- Digest-bound artifacts and deterministic media extraction
+
+## Four extension examples
+
+The wheel includes one offline, synthetic example for each extension study.
+They require no API key or customer data and are exercised by the SDK test
+suite:
+
+```bash
+python -m rubricloop.examples.reference_feed_example
+python -m rubricloop.examples.llm_judge_example
+python -m rubricloop.examples.streaming_example
+python -m rubricloop.examples.multimodal_example
+```
+
+The main extension parameters preserve the original text-only behavior:
+
+```python
+run = verify(
+    agent,
+    prompt,
+    rules,
+    judge=judge_config,       # JudgeConfig; credentials are references
+    artifact=artifact,       # Artifact or a sequence of artifacts
+    extract=extract_spec,    # ExtractSpec or a sequence of extractors
+    streaming="cut",         # False, True/"cut", or "observe"
+    constrain=True,          # Pass compiled constraints when supported
+)
+```
+
+Extension evidence is emitted only when used through `run.references`,
+`run.judge`, `run.artifacts`, `run.extractions`, `run.constraints`, and the
+matching keys in `run.to_dict()`.
 
 ## Registry CLI
 
@@ -150,11 +197,11 @@ When the package is ready to share, create an API key in the
 hosted registry:
 
 ```bash
-rubricloop --server https://api.rubricloop.dev login --api-key "$RUBRICLOOP_API_KEY"
-rubricloop --server https://api.rubricloop.dev create acme/check --visibility private
-rubricloop --server https://api.rubricloop.dev push dist/acme-check.rlpack --tag latest
-rubricloop --server https://api.rubricloop.dev vet acme/check@latest
-rubricloop --server https://api.rubricloop.dev pull acme/check@latest
+rubricloop login --api-key "$RUBRICLOOP_API_KEY"
+rubricloop create acme/check --visibility private
+rubricloop push dist/acme-check.rlpack --tag latest
+rubricloop vet acme/check@latest
+rubricloop pull acme/check@latest
 ```
 
 Public community packages are free. Private repositories are available only to

{rubricloop-0.1.3 → rubricloop-0.2.0}/examples/verify_reply_registry/README.md

@@ -4,6 +4,24 @@ This package uses the same three deterministic rules as
 `sdk/examples/verify_reply.py`: 40 to 70 words, the word `timeline`, and no
 `guaranteed refund` promise.
 
+You can run the same checks through Hosted Verification before publishing and
+vetting your own package:
+
+```bash
+curl https://api.rubricloop.dev/v1/verify \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: $RUBRICLOOP_API_KEY" \
+  -d '{
+    "pack_id": "support/reply-v1",
+    "output": {
+      "reply": "We apologize for the delay with your order. Our team is currently investigating the issue and working to resolve it as quickly as possible. You can expect an update on the timeline for your order shortly. Thank you for your patience."
+    },
+    "context": {}
+  }'
+```
+
+To publish the package itself:
+
 ```bash
 pip install rubricloop openai
 export OPENAI_API_KEY="sk-..."
@@ -12,24 +30,24 @@ python sdk/examples/verify_reply.py
 python sdk/examples/verify_reply_registry/build_package.py \
   --namespace YOUR_NAMESPACE
 
-rubricloop --server https://api.rubricloop.dev login
-rubricloop --server https://api.rubricloop.dev create \
+rubricloop login
+rubricloop create \
   YOUR_NAMESPACE/support-reply \
   --visibility private \
   --description "Checks support replies before they are sent."
 rubricloop validate dist/support-reply.rlpack
 rubricloop test dist/support-reply.rlpack
-rubricloop --server https://api.rubricloop.dev push \
+rubricloop push \
   dist/support-reply.rlpack \
   --tag latest
-rubricloop --server https://api.rubricloop.dev vet \
+rubricloop vet \
   YOUR_NAMESPACE/support-reply@latest
 ```
 
 Pull the immutable bundle digest returned by `push`, not the mutable tag:
 
 ```bash
-rubricloop --server https://api.rubricloop.dev pull \
+rubricloop pull \
   YOUR_NAMESPACE/support-reply@sha256:BUNDLE_DIGEST \
   --output dist/pulled-support-reply.rlpack
 cmp dist/support-reply.rlpack dist/pulled-support-reply.rlpack
@@ -38,7 +56,7 @@ cmp dist/support-reply.rlpack dist/pulled-support-reply.rlpack
 Once vetting passes and hosted execution is approved for that exact digest:
 
 ```bash
-rubricloop --server https://api.rubricloop.dev run \
+rubricloop run \
   YOUR_NAMESPACE/support-reply@sha256:BUNDLE_DIGEST \
   --input sdk/examples/verify_reply_registry/run-input.json
 ```

{rubricloop-0.1.3 → rubricloop-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "rubricloop"
-version = "0.1.3"
+version = "0.2.0"
 description = "Bounded, auditable verification loops for AI agent output."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -35,6 +35,14 @@ Documentation = "https://rubricloop.com/docs"
 rubricloop = "rubricloop.cli:main"
 
 [project.optional-dependencies]
+judge = [
+  "openai>=2,<4",
+]
+media = [
+  "openpyxl>=3.1,<4",
+  "Pillow>=11,<13",
+  "python-pptx>=1.0,<2",
+]
 dev = [
   "build>=1.2",
   "pytest>=8.3",

rubricloop-0.2.0/src/rubricloop/__init__.py

@@ -0,0 +1,101 @@
+"""RubricLoop public API."""
+
+from .compile import (
+    BACKEND_SUPPORT,
+    ConstrainedAgentAdapter,
+    HostedConstrainedAgentAdapter,
+    VllmConstrainedAgentAdapter,
+    compile_constraints,
+)
+from .credentials import resolve_credential
+from .loop import build_feedback, verify, verify_artifact
+from .references import resolve_reference
+from .rlpack import (
+    ArchiveLimits,
+    Ed25519Keyring,
+    Ed25519Signer,
+    Rlpack,
+    RlpackError,
+    SignatureError,
+    build_rlpack,
+    canonical_json,
+    create_manifest,
+    read_rlpack,
+    validate_manifest,
+)
+from .rubric import CheckContext, Rubric, Rule, rubric
+from .streaming import StreamGuard, streamability_report
+from .types import (
+    Abstention,
+    AgentRequest,
+    AgentResponse,
+    Artifact,
+    Budget,
+    CheckResult,
+    CompiledConstraints,
+    ExtractionRecord,
+    ExtractionRegion,
+    ExtractSpec,
+    Failure,
+    IterationRecord,
+    JudgeCallRecord,
+    JudgeConfig,
+    ReferenceFailure,
+    ReferenceSnapshot,
+    ReferenceSpec,
+    RunResult,
+    StreamHit,
+    Verdict,
+)
+
+__all__ = [
+    "Abstention",
+    "AgentRequest",
+    "AgentResponse",
+    "Artifact",
+    "ArchiveLimits",
+    "Budget",
+    "BACKEND_SUPPORT",
+    "Ed25519Keyring",
+    "Ed25519Signer",
+    "Rlpack",
+    "RlpackError",
+    "SignatureError",
+    "CheckContext",
+    "CheckResult",
+    "CompiledConstraints",
+    "ConstrainedAgentAdapter",
+    "HostedConstrainedAgentAdapter",
+    "ExtractSpec",
+    "ExtractionRecord",
+    "ExtractionRegion",
+    "Failure",
+    "IterationRecord",
+    "JudgeCallRecord",
+    "JudgeConfig",
+    "ReferenceFailure",
+    "ReferenceSnapshot",
+    "ReferenceSpec",
+    "Rubric",
+    "Rule",
+    "RunResult",
+    "StreamHit",
+    "StreamGuard",
+    "streamability_report",
+    "Verdict",
+    "VllmConstrainedAgentAdapter",
+    "build_rlpack",
+    "build_feedback",
+    "canonical_json",
+    "compile_constraints",
+    "create_manifest",
+    "read_rlpack",
+    "resolve_credential",
+    "resolve_reference",
+    "rubric",
+    "validate_manifest",
+    "verify",
+    "verify_artifact",
+]
+
+__version__ = "0.2.0"

rubricloop-0.2.0/src/rubricloop/artifacts.py

@@ -0,0 +1,149 @@
+"""Artifact integrity and bounded local extraction helpers."""
+
+from __future__ import annotations
+
+import hashlib
+import io
+import json
+from collections.abc import Iterable
+from pathlib import Path
+from typing import Any
+
+from .types import Artifact, ExtractionRecord, ExtractionRegion, ExtractSpec
+
+MAX_ARTIFACT_BYTES = 20 * 1024 * 1024
+
+
+def prepare_artifacts(
+    value: Artifact | Iterable[Artifact] | None,
+) -> tuple[tuple[Artifact, ...], dict[str, bytes]]:
+    if value is None:
+        return (), {}
+    artifacts = (value,) if isinstance(value, Artifact) else tuple(value)
+    names: set[str] = set()
+    payloads: dict[str, bytes] = {}
+    for artifact in artifacts:
+        if artifact.name in names:
+            raise ValueError(f"duplicate artifact name: {artifact.name}")
+        names.add(artifact.name)
+        payload = read_artifact(artifact)
+        payloads[artifact.name] = payload
+    return artifacts, payloads
+
+
+def read_artifact(artifact: Artifact) -> bytes:
+    source = artifact.source
+    if isinstance(source, bytes):
+        payload = source
+    elif isinstance(source, (str, Path)):
+        path = Path(source)
+        if path.stat().st_size > MAX_ARTIFACT_BYTES:
+            raise ValueError(f"artifact {artifact.name} exceeds the size limit")
+        payload = path.read_bytes()
+    elif hasattr(source, "read"):
+        position = source.tell() if hasattr(source, "tell") else None
+        payload = source.read(MAX_ARTIFACT_BYTES + 1)
+        if position is not None and hasattr(source, "seek"):
+            source.seek(position)
+    else:
+        raise TypeError(f"artifact {artifact.name} source must be a path or file-like")
+    if not isinstance(payload, bytes):
+        raise TypeError(f"artifact {artifact.name} source did not return bytes")
+    if len(payload) > MAX_ARTIFACT_BYTES:
+        raise ValueError(f"artifact {artifact.name} exceeds the size limit")
+    if len(payload) != artifact.size_bytes:
+        raise ValueError(f"artifact {artifact.name} size does not match")
+    digest = "sha256:" + hashlib.sha256(payload).hexdigest()
+    if digest != artifact.sha256:
+        raise ValueError(f"artifact {artifact.name} digest does not match")
+    return payload
+
+
+def run_extractions(
+    specs: Iterable[ExtractSpec],
+    artifacts: tuple[Artifact, ...],
+    payloads: dict[str, bytes],
+) -> tuple[dict[str, ExtractionRecord], dict[str, Any]]:
+    records: dict[str, ExtractionRecord] = {}
+    evidence: dict[str, Any] = {}
+    by_name = {artifact.name: artifact for artifact in artifacts}
+    for spec in specs:
+        extraction_id = spec.id or spec.engine
+        try:
+            artifact = _select_artifact(spec, artifacts, by_name)
+            content, regions = _extract(spec, artifact, payloads[artifact.name])
+            record = ExtractionRecord(
+                engine=spec.engine,
+                version=spec.version,
+                config_sha256=_digest(dict(spec.config)),
+                content_sha256=_digest(content),
+                regions=regions,
+                content=content,
+            )
+            records[extraction_id] = record
+            evidence[extraction_id] = {
+                **record.evidence(),
+                "artifact": artifact.name,
+            }
+        except Exception as exc:
+            evidence[extraction_id] = {
+                "status": "unavailable",
+                "reason": f"{type(exc).__name__}: {str(exc)[:180]}",
+            }
+    return records, evidence
+
+
+def _select_artifact(
+    spec: ExtractSpec,
+    artifacts: tuple[Artifact, ...],
+    by_name: dict[str, Artifact],
+) -> Artifact:
+    if spec.artifact is not None:
+        if spec.artifact not in by_name:
+            raise ValueError(f"artifact {spec.artifact} is unavailable")
+        return by_name[spec.artifact]
+    if len(artifacts) != 1:
+        raise ValueError("extraction must name an artifact when the run has multiple")
+    return artifacts[0]
+
+
+def _extract(
+    spec: ExtractSpec, artifact: Artifact, payload: bytes
+) -> tuple[Any, tuple[ExtractionRegion, ...]]:
+    if spec.engine == "provided":
+        if "content" not in spec.config:
+            raise ValueError("provided extraction requires config.content")
+        return spec.config["content"], ()
+    if spec.engine == "image.metadata":
+        try:
+            from PIL import Image
+        except ImportError as exc:
+            raise RuntimeError(
+                "install rubricloop[media] to extract image metadata"
+            ) from exc
+        with Image.open(io.BytesIO(payload)) as image:
+            content = {
+                "width": image.width,
+                "height": image.height,
+                "mode": image.mode,
+                "format": image.format,
+            }
+        return content, (
+            ExtractionRegion(
+                id=f"image:{artifact.name}",
+                content=content,
+                confidence=1.0,
+            ),
+        )
+    raise ValueError(f"unsupported local extraction engine: {spec.engine}")
+
+
+def _digest(value: Any) -> str:
+    encoded = json.dumps(
+        value,
+        sort_keys=True,
+        separators=(",", ":"),
+        ensure_ascii=False,
+        allow_nan=False,
+    ).encode()
+    return "sha256:" + hashlib.sha256(encoded).hexdigest()

{rubricloop-0.1.3 → rubricloop-0.2.0}/src/rubricloop/checks/__init__.py

@@ -2,13 +2,27 @@
 
 from .common import (
     abstain,
+    citation_lookup,
     forbidden_terms,
+    json_field_enum,
     json_schema,
     pii_free,
     required_terms,
     valid_json,
     word_count,
 )
+from .media import (
+    EnvironmentResolver,
+    audio_script_adherence,
+    chart_data_match,
+    cross_modal_match,
+    layout_compliance,
+    media_metric,
+    media_pii_screen,
+    render_check,
+    synthetic_media_flag,
+    table_arithmetic,
+)
 from .sql import (
     executes,
     matches_reference,
@@ -21,10 +35,18 @@ from .sql import (
 
 __all__ = [
     "abstain",
+    "audio_script_adherence",
+    "chart_data_match",
+    "citation_lookup",
     "executes",
     "forbidden_terms",
+    "cross_modal_match",
+    "json_field_enum",
     "json_schema",
     "matches_reference",
+    "layout_compliance",
+    "media_metric",
+    "media_pii_screen",
     "no_select_star",
     "parses",
     "pii_free",
@@ -32,6 +54,10 @@ __all__ = [
     "required_columns",
     "required_terms",
     "schema_conformance",
+    "render_check",
+    "synthetic_media_flag",
+    "table_arithmetic",
     "valid_json",
     "word_count",
+    "EnvironmentResolver",
 ]