gateframe 0.1.0__tar.gz → 0.2.1__tar.gz

{gateframe-0.1.0 → gateframe-0.2.1}/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2025 Practical Mind
+Copyright (c) 2025 Baran Yalcin
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

gateframe-0.2.1/PKG-INFO

@@ -0,0 +1,325 @@
+Metadata-Version: 2.4
+Name: gateframe
+Version: 0.2.1
+Summary: Behavioral validation for LLM outputs in production workflows.
+License-Expression: MIT
+Keywords: llm,validation,behavioral,workflow,ai
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic<3.0,>=2.0
+Requires-Dist: structlog>=23.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.30.0; extra == "anthropic"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0.0; extra == "openai"
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.0.0; extra == "litellm"
+Provides-Extra: otlp
+Requires-Dist: opentelemetry-api>=1.20.0; extra == "otlp"
+Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "otlp"
+Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.20.0; extra == "otlp"
+Provides-Extra: langchain
+Requires-Dist: langchain>=0.1.0; extra == "langchain"
+Dynamic: license-file
+
+# Gateframe
+
+[![CI](https://github.com/baranylcn/gateframe/actions/workflows/ci.yml/badge.svg)](https://github.com/baranylcn/gateframe/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/gateframe)](https://pypi.org/project/gateframe/)
+[![Python](https://img.shields.io/pypi/pyversions/gateframe)](https://pypi.org/project/gateframe/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
+
+Behavioral validation for LLM outputs in production workflows.
+
+Schema validation, "does this JSON have the right keys?", is a solved problem. Instructor, Pydantic AI, and similar tools handle it well. gateframe solves a different problem: **does this output behave correctly given the context it was generated in?** Does it stay within the decision boundaries this workflow requires? When it fails, does it fail in a way your system can recover from, or does it fail silently?
+
+```python
+from pydantic import BaseModel
+from gateframe import (
+    ValidationContract,
+    StructuralRule,
+    BoundaryRule,
+    ConfidenceRule,
+    AllowedValues,
+    FailureMode,
+)
+
+
+class TriageDecision(BaseModel):
+    action: str
+    priority: str
+    confidence: float
+    rationale: str
+
+
+contract = ValidationContract(
+    name="triage_decision",
+    rules=[
+        StructuralRule(schema=TriageDecision),
+        BoundaryRule(
+            check=AllowedValues("action", {"treat", "observe", "refer", "discharge"}),
+            name="action_boundary",
+            failure_message="Action must be one of: treat, observe, refer, discharge.",
+        ),
+        ConfidenceRule(field="confidence", minimum=0.7),
+    ],
+)
+
+result = contract.validate(
+    {
+        "action": "prescribe",  # not in allowed set -> HARD_FAIL
+        "priority": "high",
+        "confidence": 0.52,  # below 0.7 -> SOFT_FAIL
+        "rationale": "...",
+    }
+)
+
+print(result.passed)  # False
+for failure in result.failures:
+    print(f"[{failure.failure_mode.value}] {failure.rule_name}: {failure.message}")
+# [hard_fail] action_boundary: Action must be one of: treat, observe, refer, discharge.
+# [soft_fail] confidence_check: Confidence 0.52 is below minimum threshold 0.7.
+```
+
+---
+
+## The problem
+
+Most LLM pipelines validate outputs the same way: parse the JSON, check the schema, move on. That catches structural errors. It misses the errors that actually cause production incidents:
+
+- A model recommends an action that is structurally valid but outside its authorized scope
+- Confidence is low but the workflow proceeds as if it weren't
+- A soft failure in step 2 silently degrades the reliability of everything downstream
+- A validation failure gives you `False`, and no context for debugging
+
+gateframe makes these failures explicit, structured, and recoverable.
+
+---
+
+## Failure modes
+
+gateframe distinguishes four failure types instead of binary pass/fail.
+
+**`HARD_FAIL`**, Stop. The output violates a hard constraint that cannot be auto-recovered.
+
+```python
+# Model chose an action outside its authorized scope
+BoundaryRule(
+    check=AllowedValues("action", {"treat", "observe", "refer"}),
+    failure_mode=FailureMode.HARD_FAIL,  # default for BoundaryRule
+)
+```
+
+**`SOFT_FAIL`**, Flag and continue with degraded confidence. Something is off but not critical enough to halt.
+
+```python
+# Model confidence is low, continue but track the degradation
+ConfidenceRule(
+    field="confidence",
+    minimum=0.7,
+    failure_mode=FailureMode.SOFT_FAIL,  # default for ConfidenceRule
+)
+```
+
+**`RETRY`**, Re-prompt with the failure context. The output is likely fixable by trying again.
+
+```python
+# Malformed output that might parse correctly on a second attempt
+StructuralRule(schema=MyOutput, failure_mode=FailureMode.RETRY)
+```
+
+**`SILENT_FAIL`**, The most dangerous kind. The output looks valid but violates a semantic or boundary rule. gateframe makes these visible instead of letting them pass through undetected.
+
+```python
+SemanticRule(
+    check=lambda output, **ctx: output["severity"] != "low" or output["escalated"] is False,
+    failure_mode=FailureMode.SILENT_FAIL,
+    failure_message="Low-severity cases should not be auto-escalated.",
+)
+```
+
+---
+
+## Multi-step workflow validation
+
+Validation state carries forward across steps. A soft failure in step 2 degrades the confidence score that step 4 sees.
+
+```python
+from gateframe import WorkflowContext, ValidationContract, EscalationRouter
+from gateframe.audit.log import AuditLog
+
+ctx = WorkflowContext(workflow_id="incident_response_001", escalation_threshold=0.5)
+router = EscalationRouter()
+audit = AuditLog()
+
+# Step 1
+result1 = contract_step1.validate(output1)
+ctx.update(result1)
+audit.record(result1, workflow_context=ctx)
+
+# Step 2, ctx carries forward degraded confidence from step 1
+result2 = contract_step2.validate(output2)
+ctx.update(result2)
+audit.record(result2, workflow_context=ctx)
+
+print(ctx.confidence)  # degraded from 1.0 by soft failures
+print(ctx.threshold_breached)  # True if confidence < escalation_threshold
+
+if ctx.threshold_breached:
+    escalation = router.route_threshold_breach(ctx)
+    print(escalation.route.value)  # "human_review", "abort", etc.
+```
+
+---
+
+## Provider integrations
+
+gateframe validates outputs from any provider. Integrations are thin wrappers, gateframe does not import any LLM SDK at the core level.
+
+```python
+# OpenAI
+from gateframe.integrations.openai import OpenAIValidator
+
+validator = OpenAIValidator(contract, parse_json=True)
+result = validator.validate(openai_completion)
+
+# Anthropic
+from gateframe.integrations.anthropic import AnthropicValidator
+
+validator = AnthropicValidator(contract, parse_json=True)
+result = validator.validate(anthropic_message)
+
+# LiteLLM
+from gateframe.integrations.litellm import LiteLLMValidator
+
+validator = LiteLLMValidator(contract, parse_json=True)
+result = validator.validate(litellm_response)
+
+# LangChain
+from gateframe.integrations.langchain import LangChainValidator
+
+validator = LangChainValidator(contract, parse_json=False)
+result = validator.validate(chain_output)
+```
+
+Install the integration you need:
+
+```bash
+pip install "gateframe[openai]"
+pip install "gateframe[anthropic]"
+pip install "gateframe[litellm]"
+pip install "gateframe[langchain]"
+```
+
+---
+
+## Audit trail
+
+Every validation event is logged with structured context. Use the built-in exporters or implement your own.
+
+```python
+from gateframe.audit.log import AuditLog
+from gateframe.audit.exporters import JsonFileExporter
+
+audit = AuditLog(exporters=[JsonFileExporter("audit.jsonl")])
+audit.record(result, workflow_context=ctx)
+audit.flush()
+```
+
+Each entry includes: timestamp, contract name, rules applied, rules failed, failure details, workflow ID, and accumulated confidence score.
+
+---
+
+## Trend analysis
+
+`ContractTrendAnalyzer` reads an existing JSONL audit log and computes per-contract pass-rate trends across workflow runs. Use it to detect reliability regressions before they become incidents.
+
+```python
+from pathlib import Path
+from gateframe.audit.trend import ContractTrendAnalyzer
+
+analyzer = ContractTrendAnalyzer(Path("audit.jsonl"), window=20)
+report = analyzer.analyze()
+
+if report.any_regression:
+    for ct in report.regressions:
+        print(f"{ct.contract_name}: {ct.direction} (slope={ct.slope:.4f})")
+```
+
+The `window` parameter controls how many most-recent workflow runs are included. A contract is flagged as regressing when its pass-rate slope falls below `-0.02` (two percentage points per run) by default.
+
+---
+
+## When to use gateframe
+
+Use it when:
+- You need to validate LLM output behavior beyond schema checks, decision boundaries, scope enforcement, semantic constraints
+- You need structured, recoverable failure records rather than bare exceptions
+- You're running multi-step workflows where soft failures in early steps should affect confidence downstream
+- You need an audit trail for post-incident debugging
+
+Don't use it when:
+- You only need schema extraction from LLM outputs, use [Instructor](https://github.com/jxnl/instructor) or [Pydantic AI](https://ai.pydantic.dev/)
+- You need offline model evaluation or benchmarking, use [DeepEval](https://github.com/confident-ai/deepeval) or [RAGAS](https://github.com/explodinggradients/ragas)
+- You need content safety filtering, use a dedicated guardrails tool
+
+---
+
+## Installation
+
+```bash
+pip install gateframe
+```
+
+For development:
+
+```bash
+git clone https://github.com/baranylcn/gateframe.git
+cd gateframe
+pip install -e ".[dev]"
+python -m pytest tests/ -v
+```
+
+---
+
+## Examples
+
+**[triage_workflow](examples/triage_workflow/)**, 3-step medical triage pipeline. Demonstrates StructuralRule, BoundaryRule, ConfidenceRule, and WorkflowContext together. Step 2 has confidence below threshold, shows how SOFT_FAIL degrades the workflow score without halting it.
+
+**[rag_output](examples/rag_output/)**, RAG answer validation with two scenarios. Scenario B demonstrates simultaneous soft failures (low confidence + ungrounded answer) and how they accumulate in the workflow context.
+
+**[agent_pipeline](examples/agent_pipeline/)**, 4-step agent workflow with escalation. Demonstrates how multiple soft failures across steps push cumulative confidence below the escalation threshold.
+
+---
+
+## CLI
+
+```bash
+# Inspect a contract file, lists all contracts and their rules
+gateframe inspect contracts.py
+
+# Replay an audit log
+gateframe replay audit.jsonl
+
+# Detect pass-rate regressions across workflow runs
+gateframe trend audit.jsonl --window 20
+```
+
+---
+
+## License
+
+MIT, see [LICENSE](LICENSE).

gateframe-0.2.1/README.md

@@ -0,0 +1,288 @@
+# Gateframe
+
+[![CI](https://github.com/baranylcn/gateframe/actions/workflows/ci.yml/badge.svg)](https://github.com/baranylcn/gateframe/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/gateframe)](https://pypi.org/project/gateframe/)
+[![Python](https://img.shields.io/pypi/pyversions/gateframe)](https://pypi.org/project/gateframe/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
+
+Behavioral validation for LLM outputs in production workflows.
+
+Schema validation, "does this JSON have the right keys?", is a solved problem. Instructor, Pydantic AI, and similar tools handle it well. gateframe solves a different problem: **does this output behave correctly given the context it was generated in?** Does it stay within the decision boundaries this workflow requires? When it fails, does it fail in a way your system can recover from, or does it fail silently?
+
+```python
+from pydantic import BaseModel
+from gateframe import (
+    ValidationContract,
+    StructuralRule,
+    BoundaryRule,
+    ConfidenceRule,
+    AllowedValues,
+    FailureMode,
+)
+
+
+class TriageDecision(BaseModel):
+    action: str
+    priority: str
+    confidence: float
+    rationale: str
+
+
+contract = ValidationContract(
+    name="triage_decision",
+    rules=[
+        StructuralRule(schema=TriageDecision),
+        BoundaryRule(
+            check=AllowedValues("action", {"treat", "observe", "refer", "discharge"}),
+            name="action_boundary",
+            failure_message="Action must be one of: treat, observe, refer, discharge.",
+        ),
+        ConfidenceRule(field="confidence", minimum=0.7),
+    ],
+)
+
+result = contract.validate(
+    {
+        "action": "prescribe",  # not in allowed set -> HARD_FAIL
+        "priority": "high",
+        "confidence": 0.52,  # below 0.7 -> SOFT_FAIL
+        "rationale": "...",
+    }
+)
+
+print(result.passed)  # False
+for failure in result.failures:
+    print(f"[{failure.failure_mode.value}] {failure.rule_name}: {failure.message}")
+# [hard_fail] action_boundary: Action must be one of: treat, observe, refer, discharge.
+# [soft_fail] confidence_check: Confidence 0.52 is below minimum threshold 0.7.
+```
+
+---
+
+## The problem
+
+Most LLM pipelines validate outputs the same way: parse the JSON, check the schema, move on. That catches structural errors. It misses the errors that actually cause production incidents:
+
+- A model recommends an action that is structurally valid but outside its authorized scope
+- Confidence is low but the workflow proceeds as if it weren't
+- A soft failure in step 2 silently degrades the reliability of everything downstream
+- A validation failure gives you `False`, and no context for debugging
+
+gateframe makes these failures explicit, structured, and recoverable.
+
+---
+
+## Failure modes
+
+gateframe distinguishes four failure types instead of binary pass/fail.
+
+**`HARD_FAIL`**, Stop. The output violates a hard constraint that cannot be auto-recovered.
+
+```python
+# Model chose an action outside its authorized scope
+BoundaryRule(
+    check=AllowedValues("action", {"treat", "observe", "refer"}),
+    failure_mode=FailureMode.HARD_FAIL,  # default for BoundaryRule
+)
+```
+
+**`SOFT_FAIL`**, Flag and continue with degraded confidence. Something is off but not critical enough to halt.
+
+```python
+# Model confidence is low, continue but track the degradation
+ConfidenceRule(
+    field="confidence",
+    minimum=0.7,
+    failure_mode=FailureMode.SOFT_FAIL,  # default for ConfidenceRule
+)
+```
+
+**`RETRY`**, Re-prompt with the failure context. The output is likely fixable by trying again.
+
+```python
+# Malformed output that might parse correctly on a second attempt
+StructuralRule(schema=MyOutput, failure_mode=FailureMode.RETRY)
+```
+
+**`SILENT_FAIL`**, The most dangerous kind. The output looks valid but violates a semantic or boundary rule. gateframe makes these visible instead of letting them pass through undetected.
+
+```python
+SemanticRule(
+    check=lambda output, **ctx: output["severity"] != "low" or output["escalated"] is False,
+    failure_mode=FailureMode.SILENT_FAIL,
+    failure_message="Low-severity cases should not be auto-escalated.",
+)
+```
+
+---
+
+## Multi-step workflow validation
+
+Validation state carries forward across steps. A soft failure in step 2 degrades the confidence score that step 4 sees.
+
+```python
+from gateframe import WorkflowContext, ValidationContract, EscalationRouter
+from gateframe.audit.log import AuditLog
+
+ctx = WorkflowContext(workflow_id="incident_response_001", escalation_threshold=0.5)
+router = EscalationRouter()
+audit = AuditLog()
+
+# Step 1
+result1 = contract_step1.validate(output1)
+ctx.update(result1)
+audit.record(result1, workflow_context=ctx)
+
+# Step 2, ctx carries forward degraded confidence from step 1
+result2 = contract_step2.validate(output2)
+ctx.update(result2)
+audit.record(result2, workflow_context=ctx)
+
+print(ctx.confidence)  # degraded from 1.0 by soft failures
+print(ctx.threshold_breached)  # True if confidence < escalation_threshold
+
+if ctx.threshold_breached:
+    escalation = router.route_threshold_breach(ctx)
+    print(escalation.route.value)  # "human_review", "abort", etc.
+```
+
+---
+
+## Provider integrations
+
+gateframe validates outputs from any provider. Integrations are thin wrappers, gateframe does not import any LLM SDK at the core level.
+
+```python
+# OpenAI
+from gateframe.integrations.openai import OpenAIValidator
+
+validator = OpenAIValidator(contract, parse_json=True)
+result = validator.validate(openai_completion)
+
+# Anthropic
+from gateframe.integrations.anthropic import AnthropicValidator
+
+validator = AnthropicValidator(contract, parse_json=True)
+result = validator.validate(anthropic_message)
+
+# LiteLLM
+from gateframe.integrations.litellm import LiteLLMValidator
+
+validator = LiteLLMValidator(contract, parse_json=True)
+result = validator.validate(litellm_response)
+
+# LangChain
+from gateframe.integrations.langchain import LangChainValidator
+
+validator = LangChainValidator(contract, parse_json=False)
+result = validator.validate(chain_output)
+```
+
+Install the integration you need:
+
+```bash
+pip install "gateframe[openai]"
+pip install "gateframe[anthropic]"
+pip install "gateframe[litellm]"
+pip install "gateframe[langchain]"
+```
+
+---
+
+## Audit trail
+
+Every validation event is logged with structured context. Use the built-in exporters or implement your own.
+
+```python
+from gateframe.audit.log import AuditLog
+from gateframe.audit.exporters import JsonFileExporter
+
+audit = AuditLog(exporters=[JsonFileExporter("audit.jsonl")])
+audit.record(result, workflow_context=ctx)
+audit.flush()
+```
+
+Each entry includes: timestamp, contract name, rules applied, rules failed, failure details, workflow ID, and accumulated confidence score.
+
+---
+
+## Trend analysis
+
+`ContractTrendAnalyzer` reads an existing JSONL audit log and computes per-contract pass-rate trends across workflow runs. Use it to detect reliability regressions before they become incidents.
+
+```python
+from pathlib import Path
+from gateframe.audit.trend import ContractTrendAnalyzer
+
+analyzer = ContractTrendAnalyzer(Path("audit.jsonl"), window=20)
+report = analyzer.analyze()
+
+if report.any_regression:
+    for ct in report.regressions:
+        print(f"{ct.contract_name}: {ct.direction} (slope={ct.slope:.4f})")
+```
+
+The `window` parameter controls how many most-recent workflow runs are included. A contract is flagged as regressing when its pass-rate slope falls below `-0.02` (two percentage points per run) by default.
+
+---
+
+## When to use gateframe
+
+Use it when:
+- You need to validate LLM output behavior beyond schema checks, decision boundaries, scope enforcement, semantic constraints
+- You need structured, recoverable failure records rather than bare exceptions
+- You're running multi-step workflows where soft failures in early steps should affect confidence downstream
+- You need an audit trail for post-incident debugging
+
+Don't use it when:
+- You only need schema extraction from LLM outputs, use [Instructor](https://github.com/jxnl/instructor) or [Pydantic AI](https://ai.pydantic.dev/)
+- You need offline model evaluation or benchmarking, use [DeepEval](https://github.com/confident-ai/deepeval) or [RAGAS](https://github.com/explodinggradients/ragas)
+- You need content safety filtering, use a dedicated guardrails tool
+
+---
+
+## Installation
+
+```bash
+pip install gateframe
+```
+
+For development:
+
+```bash
+git clone https://github.com/baranylcn/gateframe.git
+cd gateframe
+pip install -e ".[dev]"
+python -m pytest tests/ -v
+```
+
+---
+
+## Examples
+
+**[triage_workflow](examples/triage_workflow/)**, 3-step medical triage pipeline. Demonstrates StructuralRule, BoundaryRule, ConfidenceRule, and WorkflowContext together. Step 2 has confidence below threshold, shows how SOFT_FAIL degrades the workflow score without halting it.
+
+**[rag_output](examples/rag_output/)**, RAG answer validation with two scenarios. Scenario B demonstrates simultaneous soft failures (low confidence + ungrounded answer) and how they accumulate in the workflow context.
+
+**[agent_pipeline](examples/agent_pipeline/)**, 4-step agent workflow with escalation. Demonstrates how multiple soft failures across steps push cumulative confidence below the escalation threshold.
+
+---
+
+## CLI
+
+```bash
+# Inspect a contract file, lists all contracts and their rules
+gateframe inspect contracts.py
+
+# Replay an audit log
+gateframe replay audit.jsonl
+
+# Detect pass-rate regressions across workflow runs
+gateframe trend audit.jsonl --window 20
+```
+
+---
+
+## License
+
+MIT, see [LICENSE](LICENSE).

gateframe-0.2.1/gateframe/__init__.py

@@ -0,0 +1,27 @@
+from gateframe.core.context import WorkflowContext
+from gateframe.core.contract import ValidationContract, ValidationResult
+from gateframe.core.escalation import EscalationRoute, EscalationRouter
+from gateframe.core.failure import FailureMode, FailureResult
+from gateframe.core.retry import RetryPolicy, RetryResult
+from gateframe.rules.boundary import AllowedValues, BoundaryRule
+from gateframe.rules.confidence import ConfidenceRule
+from gateframe.rules.semantic import LlmJudge, SemanticRule
+from gateframe.rules.structural import StructuralRule
+
+__all__ = [
+    "AllowedValues",
+    "BoundaryRule",
+    "ConfidenceRule",
+    "EscalationRoute",
+    "EscalationRouter",
+    "FailureMode",
+    "FailureResult",
+    "LlmJudge",
+    "SemanticRule",
+    "StructuralRule",
+    "RetryPolicy",
+    "RetryResult",
+    "ValidationContract",
+    "ValidationResult",
+    "WorkflowContext",
+]