gateframe 0.1.0__tar.gz → 0.2.0__tar.gz

gateframe-0.2.0/PKG-INFO

@@ -0,0 +1,289 @@
+Metadata-Version: 2.4
+Name: gateframe
+Version: 0.2.0
+Summary: Behavioral validation for LLM outputs in production workflows.
+License-Expression: MIT
+Keywords: llm,validation,behavioral,workflow,ai
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic<3.0,>=2.0
+Requires-Dist: structlog>=23.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.30.0; extra == "anthropic"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0.0; extra == "openai"
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.0.0; extra == "litellm"
+Provides-Extra: otlp
+Requires-Dist: opentelemetry-api>=1.20.0; extra == "otlp"
+Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "otlp"
+Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.20.0; extra == "otlp"
+Provides-Extra: langchain
+Requires-Dist: langchain>=0.1.0; extra == "langchain"
+Dynamic: license-file
+
+# Gateframe
+
+Behavioral validation for LLM outputs in production workflows.
+
+Schema validation, "does this JSON have the right keys?", is a solved problem. Instructor, Pydantic AI, and similar tools handle it well. gateframe solves a different problem: **does this output behave correctly given the context it was generated in?** Does it stay within the decision boundaries this workflow requires? When it fails, does it fail in a way your system can recover from, or does it fail silently?
+
+```python
+from pydantic import BaseModel
+from gateframe import (
+    ValidationContract,
+    StructuralRule,
+    BoundaryRule,
+    ConfidenceRule,
+    AllowedValues,
+    FailureMode,
+)
+
+class TriageDecision(BaseModel):
+    action: str
+    priority: str
+    confidence: float
+    rationale: str
+
+contract = ValidationContract(
+    name="triage_decision",
+    rules=[
+        StructuralRule(schema=TriageDecision),
+        BoundaryRule(
+            check=AllowedValues("action", {"treat", "observe", "refer", "discharge"}),
+            name="action_boundary",
+            failure_message="Action must be one of: treat, observe, refer, discharge.",
+        ),
+        ConfidenceRule(field="confidence", minimum=0.7),
+    ],
+)
+
+result = contract.validate({
+    "action": "prescribe",       # not in allowed set -> HARD_FAIL
+    "priority": "high",
+    "confidence": 0.52,          # below 0.7 -> SOFT_FAIL
+    "rationale": "...",
+})
+
+print(result.passed)             # False
+for failure in result.failures:
+    print(f"[{failure.failure_mode.value}] {failure.rule_name}: {failure.message}")
+# [hard_fail] action_boundary: Action must be one of: treat, observe, refer, discharge.
+# [soft_fail] confidence_check: Confidence 0.52 is below minimum threshold 0.7.
+```
+
+---
+
+## The problem
+
+Most LLM pipelines validate outputs the same way: parse the JSON, check the schema, move on. That catches structural errors. It misses the errors that actually cause production incidents:
+
+- A model recommends an action that is structurally valid but outside its authorized scope
+- Confidence is low but the workflow proceeds as if it weren't
+- A soft failure in step 2 silently degrades the reliability of everything downstream
+- A validation failure gives you `False`, and no context for debugging
+
+gateframe makes these failures explicit, structured, and recoverable.
+
+---
+
+## Failure modes
+
+gateframe distinguishes four failure types instead of binary pass/fail.
+
+**`HARD_FAIL`**, Stop. The output violates a hard constraint that cannot be auto-recovered.
+
+```python
+# Model chose an action outside its authorized scope
+BoundaryRule(
+    check=AllowedValues("action", {"treat", "observe", "refer"}),
+    failure_mode=FailureMode.HARD_FAIL,  # default for BoundaryRule
+)
+```
+
+**`SOFT_FAIL`**, Flag and continue with degraded confidence. Something is off but not critical enough to halt.
+
+```python
+# Model confidence is low, continue but track the degradation
+ConfidenceRule(
+    field="confidence",
+    minimum=0.7,
+    failure_mode=FailureMode.SOFT_FAIL,  # default for ConfidenceRule
+)
+```
+
+**`RETRY`**, Re-prompt with the failure context. The output is likely fixable by trying again.
+
+```python
+# Malformed output that might parse correctly on a second attempt
+StructuralRule(schema=MyOutput, failure_mode=FailureMode.RETRY)
+```
+
+**`SILENT_FAIL`**, The most dangerous kind. The output looks valid but violates a semantic or boundary rule. gateframe makes these visible instead of letting them pass through undetected.
+
+```python
+SemanticRule(
+    check=lambda output, **ctx: output["severity"] != "low" or output["escalated"] is False,
+    failure_mode=FailureMode.SILENT_FAIL,
+    failure_message="Low-severity cases should not be auto-escalated.",
+)
+```
+
+---
+
+## Multi-step workflow validation
+
+Validation state carries forward across steps. A soft failure in step 2 degrades the confidence score that step 4 sees.
+
+```python
+from gateframe import WorkflowContext, ValidationContract, EscalationRouter
+from gateframe.audit.log import AuditLog
+
+ctx = WorkflowContext(workflow_id="incident_response_001", escalation_threshold=0.5)
+router = EscalationRouter()
+audit = AuditLog()
+
+# Step 1
+result1 = contract_step1.validate(output1)
+ctx.update(result1)
+audit.record(result1, workflow_context=ctx)
+
+# Step 2, ctx carries forward degraded confidence from step 1
+result2 = contract_step2.validate(output2)
+ctx.update(result2)
+audit.record(result2, workflow_context=ctx)
+
+print(ctx.confidence)           # degraded from 1.0 by soft failures
+print(ctx.threshold_breached)   # True if confidence < escalation_threshold
+
+if ctx.threshold_breached:
+    escalation = router.route_threshold_breach(ctx)
+    print(escalation.route.value)  # "human_review", "abort", etc.
+```
+
+---
+
+## Provider integrations
+
+gateframe validates outputs from any provider. Integrations are thin wrappers, gateframe does not import any LLM SDK at the core level.
+
+```python
+# OpenAI
+from gateframe.integrations.openai import OpenAIValidator
+validator = OpenAIValidator(contract, parse_json=True)
+result = validator.validate(openai_completion)
+
+# Anthropic
+from gateframe.integrations.anthropic import AnthropicValidator
+validator = AnthropicValidator(contract, parse_json=True)
+result = validator.validate(anthropic_message)
+
+# LiteLLM
+from gateframe.integrations.litellm import LiteLLMValidator
+validator = LiteLLMValidator(contract, parse_json=True)
+result = validator.validate(litellm_response)
+
+# LangChain
+from gateframe.integrations.langchain import LangChainValidator
+validator = LangChainValidator(contract, parse_json=False)
+result = validator.validate(chain_output)
+```
+
+Install the integration you need:
+
+```bash
+pip install "gateframe[openai]"
+pip install "gateframe[anthropic]"
+pip install "gateframe[litellm]"
+pip install "gateframe[langchain]"
+```
+
+---
+
+## Audit trail
+
+Every validation event is logged with structured context. Use the built-in exporters or implement your own.
+
+```python
+from gateframe.audit.log import AuditLog
+from gateframe.audit.exporters import JsonFileExporter
+
+audit = AuditLog(exporters=[JsonFileExporter("audit.jsonl")])
+audit.record(result, workflow_context=ctx)
+audit.flush()
+```
+
+Each entry includes: timestamp, contract name, rules applied, rules failed, failure details, workflow ID, and accumulated confidence score.
+
+---
+
+## When to use gateframe
+
+Use it when:
+- You need to validate LLM output behavior beyond schema checks, decision boundaries, scope enforcement, semantic constraints
+- You need structured, recoverable failure records rather than bare exceptions
+- You're running multi-step workflows where soft failures in early steps should affect confidence downstream
+- You need an audit trail for post-incident debugging
+
+Don't use it when:
+- You only need schema extraction from LLM outputs, use [Instructor](https://github.com/jxnl/instructor) or [Pydantic AI](https://ai.pydantic.dev/)
+- You need offline model evaluation or benchmarking, use [DeepEval](https://github.com/confident-ai/deepeval) or [RAGAS](https://github.com/explodinggradients/ragas)
+- You need content safety filtering, use a dedicated guardrails tool
+
+---
+
+## Installation
+
+```bash
+pip install gateframe
+```
+
+For development:
+
+```bash
+git clone https://github.com/practicalmind-ai/gateframe.git
+cd gateframe
+pip install -e ".[dev]"
+python -m pytest tests/ -v
+```
+
+---
+
+## Examples
+
+**[triage_workflow](examples/triage_workflow/)**, 3-step medical triage pipeline. Demonstrates StructuralRule, BoundaryRule, ConfidenceRule, and WorkflowContext together. Step 2 has confidence below threshold, shows how SOFT_FAIL degrades the workflow score without halting it.
+
+**[rag_output](examples/rag_output/)**, RAG answer validation with two scenarios. Scenario B demonstrates simultaneous soft failures (low confidence + ungrounded answer) and how they accumulate in the workflow context.
+
+**[agent_pipeline](examples/agent_pipeline/)**, 4-step agent workflow with escalation. Demonstrates how multiple soft failures across steps push cumulative confidence below the escalation threshold.
+
+---
+
+## CLI
+
+```bash
+# Inspect a contract file, lists all contracts and their rules
+gateframe inspect contracts.py
+
+# Replay an audit log
+gateframe replay audit.jsonl
+```
+
+---
+
+## License
+
+MIT, see [LICENSE](LICENSE).

gateframe-0.2.0/README.md

@@ -0,0 +1,252 @@
+# Gateframe
+
+Behavioral validation for LLM outputs in production workflows.
+
+Schema validation, "does this JSON have the right keys?", is a solved problem. Instructor, Pydantic AI, and similar tools handle it well. gateframe solves a different problem: **does this output behave correctly given the context it was generated in?** Does it stay within the decision boundaries this workflow requires? When it fails, does it fail in a way your system can recover from, or does it fail silently?
+
+```python
+from pydantic import BaseModel
+from gateframe import (
+    ValidationContract,
+    StructuralRule,
+    BoundaryRule,
+    ConfidenceRule,
+    AllowedValues,
+    FailureMode,
+)
+
+class TriageDecision(BaseModel):
+    action: str
+    priority: str
+    confidence: float
+    rationale: str
+
+contract = ValidationContract(
+    name="triage_decision",
+    rules=[
+        StructuralRule(schema=TriageDecision),
+        BoundaryRule(
+            check=AllowedValues("action", {"treat", "observe", "refer", "discharge"}),
+            name="action_boundary",
+            failure_message="Action must be one of: treat, observe, refer, discharge.",
+        ),
+        ConfidenceRule(field="confidence", minimum=0.7),
+    ],
+)
+
+result = contract.validate({
+    "action": "prescribe",       # not in allowed set -> HARD_FAIL
+    "priority": "high",
+    "confidence": 0.52,          # below 0.7 -> SOFT_FAIL
+    "rationale": "...",
+})
+
+print(result.passed)             # False
+for failure in result.failures:
+    print(f"[{failure.failure_mode.value}] {failure.rule_name}: {failure.message}")
+# [hard_fail] action_boundary: Action must be one of: treat, observe, refer, discharge.
+# [soft_fail] confidence_check: Confidence 0.52 is below minimum threshold 0.7.
+```
+
+---
+
+## The problem
+
+Most LLM pipelines validate outputs the same way: parse the JSON, check the schema, move on. That catches structural errors. It misses the errors that actually cause production incidents:
+
+- A model recommends an action that is structurally valid but outside its authorized scope
+- Confidence is low but the workflow proceeds as if it weren't
+- A soft failure in step 2 silently degrades the reliability of everything downstream
+- A validation failure gives you `False`, and no context for debugging
+
+gateframe makes these failures explicit, structured, and recoverable.
+
+---
+
+## Failure modes
+
+gateframe distinguishes four failure types instead of binary pass/fail.
+
+**`HARD_FAIL`**, Stop. The output violates a hard constraint that cannot be auto-recovered.
+
+```python
+# Model chose an action outside its authorized scope
+BoundaryRule(
+    check=AllowedValues("action", {"treat", "observe", "refer"}),
+    failure_mode=FailureMode.HARD_FAIL,  # default for BoundaryRule
+)
+```
+
+**`SOFT_FAIL`**, Flag and continue with degraded confidence. Something is off but not critical enough to halt.
+
+```python
+# Model confidence is low, continue but track the degradation
+ConfidenceRule(
+    field="confidence",
+    minimum=0.7,
+    failure_mode=FailureMode.SOFT_FAIL,  # default for ConfidenceRule
+)
+```
+
+**`RETRY`**, Re-prompt with the failure context. The output is likely fixable by trying again.
+
+```python
+# Malformed output that might parse correctly on a second attempt
+StructuralRule(schema=MyOutput, failure_mode=FailureMode.RETRY)
+```
+
+**`SILENT_FAIL`**, The most dangerous kind. The output looks valid but violates a semantic or boundary rule. gateframe makes these visible instead of letting them pass through undetected.
+
+```python
+SemanticRule(
+    check=lambda output, **ctx: output["severity"] != "low" or output["escalated"] is False,
+    failure_mode=FailureMode.SILENT_FAIL,
+    failure_message="Low-severity cases should not be auto-escalated.",
+)
+```
+
+---
+
+## Multi-step workflow validation
+
+Validation state carries forward across steps. A soft failure in step 2 degrades the confidence score that step 4 sees.
+
+```python
+from gateframe import WorkflowContext, ValidationContract, EscalationRouter
+from gateframe.audit.log import AuditLog
+
+ctx = WorkflowContext(workflow_id="incident_response_001", escalation_threshold=0.5)
+router = EscalationRouter()
+audit = AuditLog()
+
+# Step 1
+result1 = contract_step1.validate(output1)
+ctx.update(result1)
+audit.record(result1, workflow_context=ctx)
+
+# Step 2, ctx carries forward degraded confidence from step 1
+result2 = contract_step2.validate(output2)
+ctx.update(result2)
+audit.record(result2, workflow_context=ctx)
+
+print(ctx.confidence)           # degraded from 1.0 by soft failures
+print(ctx.threshold_breached)   # True if confidence < escalation_threshold
+
+if ctx.threshold_breached:
+    escalation = router.route_threshold_breach(ctx)
+    print(escalation.route.value)  # "human_review", "abort", etc.
+```
+
+---
+
+## Provider integrations
+
+gateframe validates outputs from any provider. Integrations are thin wrappers, gateframe does not import any LLM SDK at the core level.
+
+```python
+# OpenAI
+from gateframe.integrations.openai import OpenAIValidator
+validator = OpenAIValidator(contract, parse_json=True)
+result = validator.validate(openai_completion)
+
+# Anthropic
+from gateframe.integrations.anthropic import AnthropicValidator
+validator = AnthropicValidator(contract, parse_json=True)
+result = validator.validate(anthropic_message)
+
+# LiteLLM
+from gateframe.integrations.litellm import LiteLLMValidator
+validator = LiteLLMValidator(contract, parse_json=True)
+result = validator.validate(litellm_response)
+
+# LangChain
+from gateframe.integrations.langchain import LangChainValidator
+validator = LangChainValidator(contract, parse_json=False)
+result = validator.validate(chain_output)
+```
+
+Install the integration you need:
+
+```bash
+pip install "gateframe[openai]"
+pip install "gateframe[anthropic]"
+pip install "gateframe[litellm]"
+pip install "gateframe[langchain]"
+```
+
+---
+
+## Audit trail
+
+Every validation event is logged with structured context. Use the built-in exporters or implement your own.
+
+```python
+from gateframe.audit.log import AuditLog
+from gateframe.audit.exporters import JsonFileExporter
+
+audit = AuditLog(exporters=[JsonFileExporter("audit.jsonl")])
+audit.record(result, workflow_context=ctx)
+audit.flush()
+```
+
+Each entry includes: timestamp, contract name, rules applied, rules failed, failure details, workflow ID, and accumulated confidence score.
+
+---
+
+## When to use gateframe
+
+Use it when:
+- You need to validate LLM output behavior beyond schema checks, decision boundaries, scope enforcement, semantic constraints
+- You need structured, recoverable failure records rather than bare exceptions
+- You're running multi-step workflows where soft failures in early steps should affect confidence downstream
+- You need an audit trail for post-incident debugging
+
+Don't use it when:
+- You only need schema extraction from LLM outputs, use [Instructor](https://github.com/jxnl/instructor) or [Pydantic AI](https://ai.pydantic.dev/)
+- You need offline model evaluation or benchmarking, use [DeepEval](https://github.com/confident-ai/deepeval) or [RAGAS](https://github.com/explodinggradients/ragas)
+- You need content safety filtering, use a dedicated guardrails tool
+
+---
+
+## Installation
+
+```bash
+pip install gateframe
+```
+
+For development:
+
+```bash
+git clone https://github.com/practicalmind-ai/gateframe.git
+cd gateframe
+pip install -e ".[dev]"
+python -m pytest tests/ -v
+```
+
+---
+
+## Examples
+
+**[triage_workflow](examples/triage_workflow/)**, 3-step medical triage pipeline. Demonstrates StructuralRule, BoundaryRule, ConfidenceRule, and WorkflowContext together. Step 2 has confidence below threshold, shows how SOFT_FAIL degrades the workflow score without halting it.
+
+**[rag_output](examples/rag_output/)**, RAG answer validation with two scenarios. Scenario B demonstrates simultaneous soft failures (low confidence + ungrounded answer) and how they accumulate in the workflow context.
+
+**[agent_pipeline](examples/agent_pipeline/)**, 4-step agent workflow with escalation. Demonstrates how multiple soft failures across steps push cumulative confidence below the escalation threshold.
+
+---
+
+## CLI
+
+```bash
+# Inspect a contract file, lists all contracts and their rules
+gateframe inspect contracts.py
+
+# Replay an audit log
+gateframe replay audit.jsonl
+```
+
+---
+
+## License
+
+MIT, see [LICENSE](LICENSE).

gateframe-0.2.0/gateframe/__init__.py

@@ -0,0 +1,24 @@
+from gateframe.core.context import WorkflowContext
+from gateframe.core.contract import ValidationContract, ValidationResult
+from gateframe.core.escalation import EscalationRoute, EscalationRouter
+from gateframe.core.failure import FailureMode, FailureResult
+from gateframe.rules.boundary import AllowedValues, BoundaryRule
+from gateframe.rules.confidence import ConfidenceRule
+from gateframe.rules.semantic import LlmJudge, SemanticRule
+from gateframe.rules.structural import StructuralRule
+
+__all__ = [
+    "AllowedValues",
+    "BoundaryRule",
+    "ConfidenceRule",
+    "EscalationRoute",
+    "EscalationRouter",
+    "FailureMode",
+    "FailureResult",
+    "LlmJudge",
+    "SemanticRule",
+    "StructuralRule",
+    "ValidationContract",
+    "ValidationResult",
+    "WorkflowContext",
+]

gateframe-0.2.0/gateframe/audit/exporters.py

@@ -0,0 +1,112 @@
+from __future__ import annotations
+
+import json
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+import structlog
+
+if TYPE_CHECKING:
+    from gateframe.audit.log import AuditEntry
+
+logger = structlog.get_logger()
+
+
+class AuditExporter(ABC):
+    @abstractmethod
+    def export(self, entry: AuditEntry) -> None: ...
+
+    @abstractmethod
+    def flush(self) -> None: ...
+
+    def shutdown(self) -> None:  # noqa: B027
+        pass
+
+
+class JsonFileExporter(AuditExporter):
+    def __init__(self, path: str | Path) -> None:
+        self._path = Path(path)
+        self._handle = open(self._path, "a")  # noqa: SIM115
+
+    def export(self, entry: AuditEntry) -> None:
+        line = json.dumps(entry.to_dict(), default=str)
+        self._handle.write(line + "\n")
+
+    def flush(self) -> None:
+        self._handle.flush()
+
+    def shutdown(self) -> None:
+        self._handle.close()
+
+
+def _require_opentelemetry() -> None:
+    try:
+        import opentelemetry  # noqa: F401
+    except ImportError:
+        raise ImportError(
+            "opentelemetry is required for OtlpExporter. "
+            "Install it with: pip install gateframe[otlp]"
+        ) from None
+
+
+class OtlpExporter(AuditExporter):
+    def __init__(
+        self,
+        endpoint: str = "http://localhost:4317",
+        *,
+        insecure: bool = True,
+        service_name: str = "gateframe",
+    ) -> None:
+        _require_opentelemetry()
+
+        from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
+        from opentelemetry.sdk._logs import LoggerProvider
+        from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
+        from opentelemetry.sdk.resources import Resource
+
+        resource = Resource.create({"service.name": service_name})
+        self._provider = LoggerProvider(resource=resource)
+        exporter = OTLPLogExporter(endpoint=endpoint, insecure=insecure)
+        self._provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
+        self._logger = self._provider.get_logger("gateframe.audit")
+
+    def export(self, entry: AuditEntry) -> None:
+        from opentelemetry.sdk._logs import LogRecord
+
+        severity = self._resolve_severity(entry)
+        attributes: dict[str, Any] = {
+            "contract_name": entry.contract_name,
+            "passed": entry.passed,
+            "rules_applied": entry.rules_applied,
+            "rules_failed": entry.rules_failed,
+        }
+        if entry.workflow_id is not None:
+            attributes["workflow_id"] = entry.workflow_id
+        if entry.confidence is not None:
+            attributes["confidence"] = entry.confidence
+
+        record = LogRecord(
+            body=json.dumps(entry.to_dict(), default=str),
+            severity_number=severity,
+            attributes=attributes,
+        )
+        self._logger.emit(record)
+
+    def flush(self) -> None:
+        self._provider.force_flush()
+
+    def shutdown(self) -> None:
+        self._provider.shutdown()
+
+    @staticmethod
+    def _resolve_severity(entry: AuditEntry) -> int:
+        import logging
+
+        has_hard = any(f.get("failure_mode") == "hard_fail" for f in entry.failures)
+        has_soft = any(f.get("failure_mode") == "soft_fail" for f in entry.failures)
+        if has_hard:
+            return logging.ERROR
+        if has_soft:
+            return logging.WARNING
+        return logging.INFO