applexus-iris-evals 1.0.0__tar.gz

applexus_iris_evals-1.0.0/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: applexus-iris-evals
+Version: 1.0.0
+Summary: Reusable evaluation framework for IRIS SAP Joule A2A agents
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: rich>=13.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: anthropic>=0.30.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+
+# iris-evals
+
+Self-evaluation library for Applexus IRIS SAP Joule agents.
+
+Install once — every agent evaluates itself on startup, runs a background loop every 5 minutes, and exposes a `/health/evals` CI gate endpoint. All golden datasets, rubrics, and thresholds are bundled inside the library. Zero eval data files needed per agent.
+
+---
+
+## Installation
+
+```bash
+pip install git+https://github.com/applexus/iris-evals.git@v1.0.0
+```
+
+Add to your agent's `requirements.txt`:
+
+```
+iris-evals @ git+https://github.com/applexus/iris-evals.git@v1.0.0
+```
+
+---
+
+## Quick Start
+
+Add three blocks to your agent's `app.py`:
+
+```python
+import asyncio, os
+from iris_evals import run_self_eval, self_eval_loop
+
+AGENT_URL = os.getenv("AGENT_URL", "http://localhost:8080")
+
+@app.on_event("startup")
+async def startup_self_eval():
+    report = await run_self_eval(agent_url=AGENT_URL)
+    if not report.overall_passed:
+        logger.warning("[IRIS EVALS] Self-eval FAILED — %s", report.summary)
+
+@app.on_event("startup")
+async def start_eval_loop():
+    asyncio.create_task(
+        self_eval_loop(agent_url=AGENT_URL, interval_seconds=300)
+    )
+
+@app.route("/health/evals", methods=["GET"])
+async def eval_health(request):
+    from starlette.responses import JSONResponse
+    report = await run_self_eval(agent_url=AGENT_URL)
+    return JSONResponse({"passed": report.overall_passed, "summary": report.summary})
+```
+
+Add to your agent's `agent-card.json`:
+
+```json
+"x_evals": {
+  "evaluators": ["deterministic", "safety"],
+  "dataset": "builtin",
+  "threshold_profile": "deterministic_strict"
+}
+```
+
+---
+
+## 7 Eval Types
+
+| # | Type | Needs AI | Use when |
+|---|------|----------|----------|
+| 1 | `deterministic` | No | Rule-based agents — exact verdict match |
+| 2 | `probabilistic` | No | Classifiers — accuracy / F1 ≥ 85% |
+| 3 | `llm_as_judge` | Yes (AI Core) | Free-text output quality — rubric scored |
+| 4 | `agent_as_judge` | Yes (A2A) | Complex SAP domain — peer agent reviews |
+| 5 | `trace` | No | Multi-step agents — correct tool call sequence |
+| 6 | `safety` | No | Every agent — PII + injection checks |
+| 7 | `performance` | No | SLA-critical — latency p95 ≤ 5000ms |
+
+**Quick picks:**
+- Rule-based agent → `1, 6`
+- Triage / classifier → `2, 3, 6`
+- Multi-step → `1, 5, 6`
+- SLA-critical → `1, 5, 6, 7`
+
+---
+
+## How it works
+
+```
+cf push
+  └─ pip installs iris-evals from GitHub
+       └─ startup: run_self_eval() calls agent via A2A with golden records
+            └─ CI polls GET /health/evals → passed: true → promote
+                 └─ background: self_eval_loop() re-runs every 5 minutes
+```
+
+iris-evals runs **inside your agent's process** as an asyncio task — no separate BTP service, no extra cost.
+
+---
+
+## LLM Judge (type 3)
+
+Uses SAP AI Core orchestration. Requires these env vars in your agent's `manifest.yml` (gitignored):
+
+```
+AICORE_CLIENT_ID
+AICORE_CLIENT_SECRET
+AICORE_AUTH_URL
+AICORE_BASE_URL
+AICORE_RESOURCE_GROUP
+ORCHESTRATION_URL
+AICORE_JUDGE_MODEL     (default: gpt-4o)
+```
+
+If your agent already has AI Core credentials, iris-evals picks them up automatically — no extra config.
+
+---
+
+## Skill Integration
+
+The `iris-sap-agent-skill` asks **Q_EVAL** as its last interview question:
+
+```
+Which evaluation types does this agent need?
+→ deterministic / probabilistic / llm_as_judge / agent_as_judge / trace / safety / performance
+```
+
+Answering generates all wiring automatically — `x_evals` block, `app.py` startup/loop/health, and `requirements.txt` entry.
+
+---
+
+## Running Tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+---
+
+## Project Structure
+
+```
+src/iris_evals/
+├── aicore.py          SAP AI Core orchestration client
+├── registry.py        Built-in golden datasets, rubrics, thresholds
+├── schemas.py         EvalType, GoldenRecord, EvalConfig, EvalReport
+├── self_eval.py       run_self_eval() and self_eval_loop()
+├── runner.py          EvalSuite base class
+├── _transport.py      A2A JSON-RPC call helper
+└── evaluators/
+    ├── deterministic.py
+    ├── probabilistic.py
+    ├── llm_judge.py
+    ├── safety.py
+    ├── trace.py
+    └── performance.py
+```
+
+---
+
+## Versioning
+
+| Version | Notes |
+|---------|-------|
+| v1.0.0 | Initial release — 7 eval types, built-in registry, AI Core LLM judge |
+
+Tag a new version and update `requirements.txt` in each agent to upgrade:
+
+```bash
+git tag v1.0.1
+git push --tags
+```

applexus_iris_evals-1.0.0/README.md

@@ -0,0 +1,170 @@
+# iris-evals
+
+Self-evaluation library for Applexus IRIS SAP Joule agents.
+
+Install once — every agent evaluates itself on startup, runs a background loop every 5 minutes, and exposes a `/health/evals` CI gate endpoint. All golden datasets, rubrics, and thresholds are bundled inside the library. Zero eval data files needed per agent.
+
+---
+
+## Installation
+
+```bash
+pip install git+https://github.com/applexus/iris-evals.git@v1.0.0
+```
+
+Add to your agent's `requirements.txt`:
+
+```
+iris-evals @ git+https://github.com/applexus/iris-evals.git@v1.0.0
+```
+
+---
+
+## Quick Start
+
+Add three blocks to your agent's `app.py`:
+
+```python
+import asyncio, os
+from iris_evals import run_self_eval, self_eval_loop
+
+AGENT_URL = os.getenv("AGENT_URL", "http://localhost:8080")
+
+@app.on_event("startup")
+async def startup_self_eval():
+    report = await run_self_eval(agent_url=AGENT_URL)
+    if not report.overall_passed:
+        logger.warning("[IRIS EVALS] Self-eval FAILED — %s", report.summary)
+
+@app.on_event("startup")
+async def start_eval_loop():
+    asyncio.create_task(
+        self_eval_loop(agent_url=AGENT_URL, interval_seconds=300)
+    )
+
+@app.route("/health/evals", methods=["GET"])
+async def eval_health(request):
+    from starlette.responses import JSONResponse
+    report = await run_self_eval(agent_url=AGENT_URL)
+    return JSONResponse({"passed": report.overall_passed, "summary": report.summary})
+```
+
+Add to your agent's `agent-card.json`:
+
+```json
+"x_evals": {
+  "evaluators": ["deterministic", "safety"],
+  "dataset": "builtin",
+  "threshold_profile": "deterministic_strict"
+}
+```
+
+---
+
+## 7 Eval Types
+
+| # | Type | Needs AI | Use when |
+|---|------|----------|----------|
+| 1 | `deterministic` | No | Rule-based agents — exact verdict match |
+| 2 | `probabilistic` | No | Classifiers — accuracy / F1 ≥ 85% |
+| 3 | `llm_as_judge` | Yes (AI Core) | Free-text output quality — rubric scored |
+| 4 | `agent_as_judge` | Yes (A2A) | Complex SAP domain — peer agent reviews |
+| 5 | `trace` | No | Multi-step agents — correct tool call sequence |
+| 6 | `safety` | No | Every agent — PII + injection checks |
+| 7 | `performance` | No | SLA-critical — latency p95 ≤ 5000ms |
+
+**Quick picks:**
+- Rule-based agent → `1, 6`
+- Triage / classifier → `2, 3, 6`
+- Multi-step → `1, 5, 6`
+- SLA-critical → `1, 5, 6, 7`
+
+---
+
+## How it works
+
+```
+cf push
+  └─ pip installs iris-evals from GitHub
+       └─ startup: run_self_eval() calls agent via A2A with golden records
+            └─ CI polls GET /health/evals → passed: true → promote
+                 └─ background: self_eval_loop() re-runs every 5 minutes
+```
+
+iris-evals runs **inside your agent's process** as an asyncio task — no separate BTP service, no extra cost.
+
+---
+
+## LLM Judge (type 3)
+
+Uses SAP AI Core orchestration. Requires these env vars in your agent's `manifest.yml` (gitignored):
+
+```
+AICORE_CLIENT_ID
+AICORE_CLIENT_SECRET
+AICORE_AUTH_URL
+AICORE_BASE_URL
+AICORE_RESOURCE_GROUP
+ORCHESTRATION_URL
+AICORE_JUDGE_MODEL     (default: gpt-4o)
+```
+
+If your agent already has AI Core credentials, iris-evals picks them up automatically — no extra config.
+
+---
+
+## Skill Integration
+
+The `iris-sap-agent-skill` asks **Q_EVAL** as its last interview question:
+
+```
+Which evaluation types does this agent need?
+→ deterministic / probabilistic / llm_as_judge / agent_as_judge / trace / safety / performance
+```
+
+Answering generates all wiring automatically — `x_evals` block, `app.py` startup/loop/health, and `requirements.txt` entry.
+
+---
+
+## Running Tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+---
+
+## Project Structure
+
+```
+src/iris_evals/
+├── aicore.py          SAP AI Core orchestration client
+├── registry.py        Built-in golden datasets, rubrics, thresholds
+├── schemas.py         EvalType, GoldenRecord, EvalConfig, EvalReport
+├── self_eval.py       run_self_eval() and self_eval_loop()
+├── runner.py          EvalSuite base class
+├── _transport.py      A2A JSON-RPC call helper
+└── evaluators/
+    ├── deterministic.py
+    ├── probabilistic.py
+    ├── llm_judge.py
+    ├── safety.py
+    ├── trace.py
+    └── performance.py
+```
+
+---
+
+## Versioning
+
+| Version | Notes |
+|---------|-------|
+| v1.0.0 | Initial release — 7 eval types, built-in registry, AI Core LLM judge |
+
+Tag a new version and update `requirements.txt` in each agent to upgrade:
+
+```bash
+git tag v1.0.1
+git push --tags
+```

applexus_iris_evals-1.0.0/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "applexus-iris-evals"
+version = "1.0.0"
+description = "Reusable evaluation framework for IRIS SAP Joule A2A agents"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "httpx>=0.27.0",
+    "python-dotenv>=1.0.1",
+    "rich>=13.0.0",
+    "pydantic>=2.0.0",
+    "anthropic>=0.30.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-asyncio>=0.23.0",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

applexus_iris_evals-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

applexus_iris_evals-1.0.0/src/applexus_iris_evals.egg-info/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: applexus-iris-evals
+Version: 1.0.0
+Summary: Reusable evaluation framework for IRIS SAP Joule A2A agents
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: rich>=13.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: anthropic>=0.30.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+
+# iris-evals
+
+Self-evaluation library for Applexus IRIS SAP Joule agents.
+
+Install once — every agent evaluates itself on startup, runs a background loop every 5 minutes, and exposes a `/health/evals` CI gate endpoint. All golden datasets, rubrics, and thresholds are bundled inside the library. Zero eval data files needed per agent.
+
+---
+
+## Installation
+
+```bash
+pip install git+https://github.com/applexus/iris-evals.git@v1.0.0
+```
+
+Add to your agent's `requirements.txt`:
+
+```
+iris-evals @ git+https://github.com/applexus/iris-evals.git@v1.0.0
+```
+
+---
+
+## Quick Start
+
+Add three blocks to your agent's `app.py`:
+
+```python
+import asyncio, os
+from iris_evals import run_self_eval, self_eval_loop
+
+AGENT_URL = os.getenv("AGENT_URL", "http://localhost:8080")
+
+@app.on_event("startup")
+async def startup_self_eval():
+    report = await run_self_eval(agent_url=AGENT_URL)
+    if not report.overall_passed:
+        logger.warning("[IRIS EVALS] Self-eval FAILED — %s", report.summary)
+
+@app.on_event("startup")
+async def start_eval_loop():
+    asyncio.create_task(
+        self_eval_loop(agent_url=AGENT_URL, interval_seconds=300)
+    )
+
+@app.route("/health/evals", methods=["GET"])
+async def eval_health(request):
+    from starlette.responses import JSONResponse
+    report = await run_self_eval(agent_url=AGENT_URL)
+    return JSONResponse({"passed": report.overall_passed, "summary": report.summary})
+```
+
+Add to your agent's `agent-card.json`:
+
+```json
+"x_evals": {
+  "evaluators": ["deterministic", "safety"],
+  "dataset": "builtin",
+  "threshold_profile": "deterministic_strict"
+}
+```
+
+---
+
+## 7 Eval Types
+
+| # | Type | Needs AI | Use when |
+|---|------|----------|----------|
+| 1 | `deterministic` | No | Rule-based agents — exact verdict match |
+| 2 | `probabilistic` | No | Classifiers — accuracy / F1 ≥ 85% |
+| 3 | `llm_as_judge` | Yes (AI Core) | Free-text output quality — rubric scored |
+| 4 | `agent_as_judge` | Yes (A2A) | Complex SAP domain — peer agent reviews |
+| 5 | `trace` | No | Multi-step agents — correct tool call sequence |
+| 6 | `safety` | No | Every agent — PII + injection checks |
+| 7 | `performance` | No | SLA-critical — latency p95 ≤ 5000ms |
+
+**Quick picks:**
+- Rule-based agent → `1, 6`
+- Triage / classifier → `2, 3, 6`
+- Multi-step → `1, 5, 6`
+- SLA-critical → `1, 5, 6, 7`
+
+---
+
+## How it works
+
+```
+cf push
+  └─ pip installs iris-evals from GitHub
+       └─ startup: run_self_eval() calls agent via A2A with golden records
+            └─ CI polls GET /health/evals → passed: true → promote
+                 └─ background: self_eval_loop() re-runs every 5 minutes
+```
+
+iris-evals runs **inside your agent's process** as an asyncio task — no separate BTP service, no extra cost.
+
+---
+
+## LLM Judge (type 3)
+
+Uses SAP AI Core orchestration. Requires these env vars in your agent's `manifest.yml` (gitignored):
+
+```
+AICORE_CLIENT_ID
+AICORE_CLIENT_SECRET
+AICORE_AUTH_URL
+AICORE_BASE_URL
+AICORE_RESOURCE_GROUP
+ORCHESTRATION_URL
+AICORE_JUDGE_MODEL     (default: gpt-4o)
+```
+
+If your agent already has AI Core credentials, iris-evals picks them up automatically — no extra config.
+
+---
+
+## Skill Integration
+
+The `iris-sap-agent-skill` asks **Q_EVAL** as its last interview question:
+
+```
+Which evaluation types does this agent need?
+→ deterministic / probabilistic / llm_as_judge / agent_as_judge / trace / safety / performance
+```
+
+Answering generates all wiring automatically — `x_evals` block, `app.py` startup/loop/health, and `requirements.txt` entry.
+
+---
+
+## Running Tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+---
+
+## Project Structure
+
+```
+src/iris_evals/
+├── aicore.py          SAP AI Core orchestration client
+├── registry.py        Built-in golden datasets, rubrics, thresholds
+├── schemas.py         EvalType, GoldenRecord, EvalConfig, EvalReport
+├── self_eval.py       run_self_eval() and self_eval_loop()
+├── runner.py          EvalSuite base class
+├── _transport.py      A2A JSON-RPC call helper
+└── evaluators/
+    ├── deterministic.py
+    ├── probabilistic.py
+    ├── llm_judge.py
+    ├── safety.py
+    ├── trace.py
+    └── performance.py
+```
+
+---
+
+## Versioning
+
+| Version | Notes |
+|---------|-------|
+| v1.0.0 | Initial release — 7 eval types, built-in registry, AI Core LLM judge |
+
+Tag a new version and update `requirements.txt` in each agent to upgrade:
+
+```bash
+git tag v1.0.1
+git push --tags
+```

applexus_iris_evals-1.0.0/src/applexus_iris_evals.egg-info/SOURCES.txt

@@ -0,0 +1,26 @@
+README.md
+pyproject.toml
+src/applexus_iris_evals.egg-info/PKG-INFO
+src/applexus_iris_evals.egg-info/SOURCES.txt
+src/applexus_iris_evals.egg-info/dependency_links.txt
+src/applexus_iris_evals.egg-info/requires.txt
+src/applexus_iris_evals.egg-info/top_level.txt
+src/iris_evals/__init__.py
+src/iris_evals/_transport.py
+src/iris_evals/aicore.py
+src/iris_evals/registry.py
+src/iris_evals/results.py
+src/iris_evals/runner.py
+src/iris_evals/schemas.py
+src/iris_evals/self_eval.py
+src/iris_evals/evaluators/__init__.py
+src/iris_evals/evaluators/deterministic.py
+src/iris_evals/evaluators/llm_judge.py
+src/iris_evals/evaluators/performance.py
+src/iris_evals/evaluators/probabilistic.py
+src/iris_evals/evaluators/safety.py
+src/iris_evals/evaluators/trace.py
+tests/test_deterministic.py
+tests/test_probabilistic.py
+tests/test_safety.py
+tests/test_self_eval.py

applexus_iris_evals-1.0.0/src/applexus_iris_evals.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

applexus_iris_evals-1.0.0/src/applexus_iris_evals.egg-info/requires.txt

@@ -0,0 +1,9 @@
+httpx>=0.27.0
+python-dotenv>=1.0.1
+rich>=13.0.0
+pydantic>=2.0.0
+anthropic>=0.30.0
+
+[dev]
+pytest>=8.0.0
+pytest-asyncio>=0.23.0

applexus_iris_evals-1.0.0/src/applexus_iris_evals.egg-info/top_level.txt

@@ -0,0 +1 @@
+iris_evals

applexus_iris_evals-1.0.0/src/iris_evals/__init__.py

@@ -0,0 +1,64 @@
+"""
+iris-evals — Reusable evaluation framework for IRIS SAP Joule A2A agents.
+
+── Startup self-eval (simplest usage) ──────────────────────────────────────────
+    from iris_evals import run_self_eval
+
+    @app.on_event("startup")
+    async def startup():
+        report = await run_self_eval(agent_url="http://localhost:8080")
+        if not report.overall_passed:
+            logger.warning(f"[IRIS EVALS] {report.summary}")
+
+── Background loop ──────────────────────────────────────────────────────────────
+    from iris_evals import self_eval_loop
+
+    @app.on_event("startup")
+    async def startup():
+        asyncio.create_task(self_eval_loop(agent_url="http://localhost:8080", interval_seconds=300))
+
+── Health endpoint ───────────────────────────────────────────────────────────────
+    @app.get("/health/evals")
+    async def eval_health():
+        report = await run_self_eval(agent_url="http://localhost:8080")
+        return {"passed": report.overall_passed, "summary": report.summary}
+
+── Manual / CLI ─────────────────────────────────────────────────────────────────
+    iris-evals run --agent http://localhost:8080
+    iris-evals run-pipeline --pipeline pipeline.json
+"""
+
+from iris_evals.runner    import EvalSuite
+from iris_evals.self_eval import run_self_eval, self_eval_loop
+from iris_evals.schemas   import (
+    EvalConfig,
+    EvalType,
+    EvalReport,
+    EvalResult,
+    GoldenRecord,
+    JudgeRubric,
+    JudgeCriterion,
+)
+from iris_evals.results   import print_report, save_json
+from iris_evals.registry  import register_agent
+
+__all__ = [
+    # self-eval (primary API)
+    "run_self_eval",
+    "self_eval_loop",
+    # manual suite
+    "EvalSuite",
+    # schemas
+    "EvalConfig",
+    "EvalType",
+    "EvalReport",
+    "EvalResult",
+    "GoldenRecord",
+    "JudgeRubric",
+    "JudgeCriterion",
+    # output
+    "print_report",
+    "save_json",
+    # registry extension
+    "register_agent",
+]