agent-eval-planner 0.1.0__tar.gz

agent_eval_planner-0.1.0/LICENSE

@@ -0,0 +1,20 @@
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

agent_eval_planner-0.1.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: agent-eval-planner
+Version: 0.1.0
+Summary: Generate guardrail evaluation plans, JSONL suites, and remediations from agent contracts
+Author-email: Saulo Filho <saulofilho@users.noreply.github.com>
+License: MIT
+Project-URL: Homepage, https://saulofilho.github.io/agent-eval-planner-python/
+Project-URL: Repository, https://github.com/saulofilho/agent-eval-planner-python
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0
+Dynamic: license-file
+
+# agent-eval-planner
+
+[![PyPI version](https://badge.fury.io/py/agent-eval-planner.svg)](https://badge.fury.io/py/agent-eval-planner)
+[![Python Versions](https://img.shields.io/pypi/pyversions/agent-eval-planner.svg)](https://pypi.org/project/agent-eval-planner/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**Site:** [saulofilho.github.io/agent-eval-planner-python](https://saulofilho.github.io/agent-eval-planner-python/)
+
+Python library that turns an **agent contract** (system prompt, tools, policy gates) into a guardrail evaluation pipeline:
+
+1. **Action plan** — SCOPE / INJECT / ROLE / PII / TOOL / HALLUC / EXFIL vectors
+2. **suite.jsonl** — executable cases with `forbidden_tools` filled
+3. **Remediations** — prompt / gate / evaluator quick wins
+
+Sibling of [`security-pentest-planner`](https://github.com/saulofilho/security-pentest-planner-python), applied to LLM agents. Port of the Ruby gem [`agent_eval_planner`](https://github.com/saulofilho/agent-eval-planner).
+
+Canonical smoke test: **carrot cake** (off-topic). If the agent answers with a recipe, its scope is not limited.
+
+## What it does
+
+- Parses agent contracts (Markdown, text, JSON, YAML).
+- Generates a Markdown eval plan, a JSONL suite, and remediations.
+- Validates suites (hard-fail on empty `forbidden_tools` / leftover placeholders).
+
+## What it does **NOT** do
+
+- Execute the evaluation against a live agent.
+- Replace pentest of APIs/infra (use `security-pentest-planner`).
+
+## Installation
+
+```bash
+pip install agent-eval-planner
+```
+
+Or using `uv` / `poetry`:
+
+```bash
+uv add agent-eval-planner
+# or
+poetry add agent-eval-planner
+```
+
+## CLI usage
+
+```bash
+# Full pipeline → directory (plan + suite + remediations)
+agent-eval-planner agent.md --tools funnel_analytics,open_service_center_ticket \
+  -t "Platform Team" -a analytics -o ./out
+
+# Validate suite (hard-fail on empty forbidden_tools / placeholders)
+agent-eval-planner validate ./out/suite.jsonl \
+  --known-tools funnel_analytics,open_service_center_ticket
+
+# Plan only to stdout
+agent-eval-planner agent.md --plan-only --agent analytics
+```
+
+### Options
+
+| Flag | Description |
+|------|-------------|
+| `-t, --team TEAM` | Team name in the document title |
+| `-a, --agent NAME` | `target_agent` name |
+| `--tools LIST` | Comma-separated real tool names |
+| `--scope TEXT` | Declared scope summary |
+| `--harness NAME` | `generic` or `marketing-copilot` |
+| `-o, --output PATH` | Output directory or single file |
+| `--plan-only` / `--suite-only` / `--remediations-only` | Emit a single artifact |
+| `-v, --version` | Show version |
+
+## Programmatic usage
+
+```python
+import agent_eval_planner
+
+result = agent_eval_planner.generate(
+    input_path="agent.md",
+    team="Platform Team",
+    tools=["funnel_analytics", "open_service_center_ticket"],
+)
+
+with open("evaluation-plan.md", "w", encoding="utf-8") as f:
+    f.write(result.plan)
+```
+
+## Publish to PyPI
+
+```bash
+python -m pip install build twine
+python -m build
+python -m twine upload dist/*
+```
+
+## Development and testing
+
+```bash
+git clone https://github.com/saulofilho/agent-eval-planner-python.git
+cd agent-eval-planner-python
+
+python -m pip install .
+python -m unittest discover -s tests -v
+```
+
+## Recommended workflow
+
+```
+Agent contract  →  agent-eval-planner  →  Plan + suite.jsonl + remediations
+                                                     ↓
+                                            Harness / CI execution
+                                                     ↓
+                                            Failures + remediations
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE).

agent_eval_planner-0.1.0/README.md

@@ -0,0 +1,117 @@
+# agent-eval-planner
+
+[![PyPI version](https://badge.fury.io/py/agent-eval-planner.svg)](https://badge.fury.io/py/agent-eval-planner)
+[![Python Versions](https://img.shields.io/pypi/pyversions/agent-eval-planner.svg)](https://pypi.org/project/agent-eval-planner/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**Site:** [saulofilho.github.io/agent-eval-planner-python](https://saulofilho.github.io/agent-eval-planner-python/)
+
+Python library that turns an **agent contract** (system prompt, tools, policy gates) into a guardrail evaluation pipeline:
+
+1. **Action plan** — SCOPE / INJECT / ROLE / PII / TOOL / HALLUC / EXFIL vectors
+2. **suite.jsonl** — executable cases with `forbidden_tools` filled
+3. **Remediations** — prompt / gate / evaluator quick wins
+
+Sibling of [`security-pentest-planner`](https://github.com/saulofilho/security-pentest-planner-python), applied to LLM agents. Port of the Ruby gem [`agent_eval_planner`](https://github.com/saulofilho/agent-eval-planner).
+
+Canonical smoke test: **carrot cake** (off-topic). If the agent answers with a recipe, its scope is not limited.
+
+## What it does
+
+- Parses agent contracts (Markdown, text, JSON, YAML).
+- Generates a Markdown eval plan, a JSONL suite, and remediations.
+- Validates suites (hard-fail on empty `forbidden_tools` / leftover placeholders).
+
+## What it does **NOT** do
+
+- Execute the evaluation against a live agent.
+- Replace pentest of APIs/infra (use `security-pentest-planner`).
+
+## Installation
+
+```bash
+pip install agent-eval-planner
+```
+
+Or using `uv` / `poetry`:
+
+```bash
+uv add agent-eval-planner
+# or
+poetry add agent-eval-planner
+```
+
+## CLI usage
+
+```bash
+# Full pipeline → directory (plan + suite + remediations)
+agent-eval-planner agent.md --tools funnel_analytics,open_service_center_ticket \
+  -t "Platform Team" -a analytics -o ./out
+
+# Validate suite (hard-fail on empty forbidden_tools / placeholders)
+agent-eval-planner validate ./out/suite.jsonl \
+  --known-tools funnel_analytics,open_service_center_ticket
+
+# Plan only to stdout
+agent-eval-planner agent.md --plan-only --agent analytics
+```
+
+### Options
+
+| Flag | Description |
+|------|-------------|
+| `-t, --team TEAM` | Team name in the document title |
+| `-a, --agent NAME` | `target_agent` name |
+| `--tools LIST` | Comma-separated real tool names |
+| `--scope TEXT` | Declared scope summary |
+| `--harness NAME` | `generic` or `marketing-copilot` |
+| `-o, --output PATH` | Output directory or single file |
+| `--plan-only` / `--suite-only` / `--remediations-only` | Emit a single artifact |
+| `-v, --version` | Show version |
+
+## Programmatic usage
+
+```python
+import agent_eval_planner
+
+result = agent_eval_planner.generate(
+    input_path="agent.md",
+    team="Platform Team",
+    tools=["funnel_analytics", "open_service_center_ticket"],
+)
+
+with open("evaluation-plan.md", "w", encoding="utf-8") as f:
+    f.write(result.plan)
+```
+
+## Publish to PyPI
+
+```bash
+python -m pip install build twine
+python -m build
+python -m twine upload dist/*
+```
+
+## Development and testing
+
+```bash
+git clone https://github.com/saulofilho/agent-eval-planner-python.git
+cd agent-eval-planner-python
+
+python -m pip install .
+python -m unittest discover -s tests -v
+```
+
+## Recommended workflow
+
+```
+Agent contract  →  agent-eval-planner  →  Plan + suite.jsonl + remediations
+                                                     ↓
+                                            Harness / CI execution
+                                                     ↓
+                                            Failures + remediations
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE).

agent_eval_planner-0.1.0/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "agent-eval-planner"
+version = "0.1.0"
+description = "Generate guardrail evaluation plans, JSONL suites, and remediations from agent contracts"
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.10"
+authors = [
+    { name = "Saulo Filho", email = "saulofilho@users.noreply.github.com" }
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = [
+    "pyyaml>=6.0",
+]
+
+[project.urls]
+Homepage = "https://saulofilho.github.io/agent-eval-planner-python/"
+Repository = "https://github.com/saulofilho/agent-eval-planner-python"
+
+[project.scripts]
+agent-eval-planner = "agent_eval_planner.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

agent_eval_planner-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

agent_eval_planner-0.1.0/setup.py

@@ -0,0 +1,3 @@
+from setuptools import setup
+
+setup()

agent_eval_planner-0.1.0/src/agent_eval_planner/__init__.py

@@ -0,0 +1,50 @@
+"""agent_eval_planner package.
+
+Turns an agent contract into a guardrail evaluation plan, JSONL suite, and remediations.
+"""
+
+from __future__ import annotations
+
+from .errors import Error, InputError, ValidationError
+from .models import PlannerResult, Vector
+from .planner import Planner
+from .suite_validator import SuiteValidator
+from .version import __version__
+
+
+def generate(input_path: str | None = None, raw: str | None = None, **options) -> PlannerResult:
+    """Generate plan, suite, and remediations from an agent contract."""
+    return Planner(input_path=input_path, raw=raw, **options).call()
+
+
+def generate_plan(input_path: str | None = None, raw: str | None = None, **options) -> str:
+    """Generate only the Markdown evaluation plan."""
+    return generate(input_path=input_path, raw=raw, **options).plan
+
+
+def validate_suite(
+    path: str,
+    known_tools: list[str] | None = None,
+    agent_has_no_tools: bool = False,
+) -> list[str]:
+    """Validate a suite.jsonl file. Returns a list of errors (empty if OK)."""
+    return SuiteValidator(
+        path=path,
+        known_tools=known_tools or [],
+        agent_has_no_tools=agent_has_no_tools,
+    ).validate()
+
+
+__all__ = [
+    "__version__",
+    "generate",
+    "generate_plan",
+    "validate_suite",
+    "Planner",
+    "PlannerResult",
+    "Vector",
+    "SuiteValidator",
+    "Error",
+    "InputError",
+    "ValidationError",
+]

agent_eval_planner-0.1.0/src/agent_eval_planner/__main__.py

@@ -0,0 +1,8 @@
+"""Main entry point when invoked as python -m agent_eval_planner."""
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

agent_eval_planner-0.1.0/src/agent_eval_planner/cli.py

@@ -0,0 +1,211 @@
+"""Command-line interface for agent-eval-planner."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+from typing import Sequence
+
+from . import generate, validate_suite
+from .errors import Error
+from .models import PlannerResult
+from .version import __version__
+
+
+def build_parser() -> argparse.ArgumentParser:
+    """Builds the argument parser for generate mode."""
+    parser = argparse.ArgumentParser(
+        prog="agent-eval-planner",
+        description="Gera plano de eval, suite JSONL e remediações a partir do contrato de um agente.",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=(
+            "Usage:\n"
+            "  agent-eval-planner [options] <contract.md|prompt.txt|agent.json>\n"
+            "  agent-eval-planner validate [options] <suite.jsonl>"
+        ),
+    )
+    parser.add_argument("input", nargs="?", help="Caminho do contrato do agente")
+    parser.add_argument("-t", "--team", help="Nome do time no título")
+    parser.add_argument("-a", "--agent", dest="agent_name", help="Nome do target_agent")
+    parser.add_argument("--tools", help="Tools conhecidas (vírgula)")
+    parser.add_argument("--scope", dest="declared_scope", help="Escopo declarado (resumo)")
+    parser.add_argument("--out-of-scope", dest="out_of_scope", help="Fora de escopo declarado")
+    parser.add_argument(
+        "--harness",
+        default="generic",
+        help="Harness alvo (generic|marketing-copilot)",
+    )
+    parser.add_argument("-o", "--output", help="Arquivo .md/.jsonl ou diretório de saída")
+    parser.add_argument("--plan-only", action="store_true", help="Gerar apenas o plano")
+    parser.add_argument("--suite-only", action="store_true", help="Gerar apenas a suite JSONL")
+    parser.add_argument(
+        "--remediations-only",
+        action="store_true",
+        help="Gerar apenas remediações",
+    )
+    parser.add_argument(
+        "-v",
+        "--version",
+        action="version",
+        version=f"agent_eval_planner {__version__}",
+    )
+    return parser
+
+
+def build_validate_parser() -> argparse.ArgumentParser:
+    """Builds the argument parser for validate mode."""
+    parser = argparse.ArgumentParser(
+        prog="agent-eval-planner validate",
+        description="Valida uma suite JSONL (hard-fail em forbidden_tools vazios / placeholders).",
+    )
+    parser.add_argument("suite", nargs="?", help="Caminho da suite.jsonl")
+    parser.add_argument(
+        "--known-tools",
+        help="Tools reais (vírgula)",
+    )
+    parser.add_argument(
+        "--agent-has-no-tools",
+        action="store_true",
+        help="Agente sem tools",
+    )
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    """Main CLI execution method."""
+    args_list = list(argv) if argv is not None else sys.argv[1:]
+    if args_list and args_list[0] == "validate":
+        return _run_validate(args_list[1:])
+    return _run_generate(args_list)
+
+
+def _run_generate(argv: list[str]) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    if not args.input:
+        parser.print_help(sys.stderr)
+        return 1
+
+    tools = None
+    if args.tools:
+        tools = [part.strip() for part in args.tools.split(",") if part.strip()]
+
+    try:
+        result = generate(
+            input_path=args.input,
+            team=args.team,
+            agent_name=args.agent_name,
+            tools=tools,
+            declared_scope=args.declared_scope,
+            out_of_scope=args.out_of_scope,
+            harness=args.harness,
+        )
+        _write_outputs(result, args)
+        return 0
+    except Error as exc:
+        print(f"Erro: {exc}", file=sys.stderr)
+        return 1
+
+
+def _looks_like_directory(path: str) -> bool:
+    candidate = Path(path)
+    return candidate.is_dir() or path.endswith("/") or candidate.suffix == ""
+
+
+def _select_single(result: PlannerResult, args: argparse.Namespace) -> str | None:
+    if args.suite_only:
+        return result.suite
+    if args.remediations_only:
+        return result.remediations
+    if args.plan_only:
+        return result.plan
+    return None
+
+
+def _write_bundle(directory: Path, result: PlannerResult, args: argparse.Namespace) -> None:
+    directory.mkdir(parents=True, exist_ok=True)
+    if not args.suite_only and not args.remediations_only:
+        plan_path = directory / "plano-de-acao-agent-eval.md"
+        plan_path.write_text(result.plan, encoding="utf-8")
+        print(f"Plano gerado em {plan_path}", file=sys.stderr)
+    if not args.plan_only and not args.remediations_only:
+        suite_path = directory / "suite.jsonl"
+        suite_path.write_text(result.suite, encoding="utf-8")
+        print(f"Suite gerada em {suite_path}", file=sys.stderr)
+    if args.plan_only or args.suite_only:
+        return
+    rem_path = directory / "remediacoes.md"
+    rem_path.write_text(result.remediations, encoding="utf-8")
+    print(f"Remediações geradas em {rem_path}", file=sys.stderr)
+
+
+def _emit_stdout(result: PlannerResult, args: argparse.Namespace) -> None:
+    if args.suite_only:
+        sys.stdout.write(result.suite)
+    elif args.remediations_only:
+        sys.stdout.write(result.remediations)
+    elif args.plan_only:
+        sys.stdout.write(result.plan)
+    else:
+        sys.stdout.write(result.plan)
+        print("\n# --- suite.jsonl ---\n", file=sys.stderr)
+        sys.stdout.write(result.suite)
+        print("\n# --- remediacoes.md ---\n", file=sys.stderr)
+        sys.stdout.write(result.remediations)
+
+
+def _write_outputs(result: PlannerResult, args: argparse.Namespace) -> None:
+    if not args.output:
+        _emit_stdout(result, args)
+        return
+
+    path = args.output
+    if _looks_like_directory(path):
+        _write_bundle(Path(path), result, args)
+        return
+
+    output = Path(path)
+    if args.suite_only or output.suffix == ".jsonl":
+        output.write_text(result.suite, encoding="utf-8")
+        print(f"Suite gerada em {path}", file=sys.stderr)
+        return
+
+    content = _select_single(result, args) or result.plan
+    output.write_text(content, encoding="utf-8")
+    print(f"Artefato gerado em {path}", file=sys.stderr)
+
+
+def _run_validate(argv: list[str]) -> int:
+    parser = build_validate_parser()
+    args = parser.parse_args(argv)
+    if not args.suite:
+        parser.print_help(sys.stderr)
+        return 1
+
+    known_tools: list[str] = []
+    if args.known_tools:
+        known_tools = [part.strip() for part in args.known_tools.split(",") if part.strip()]
+
+    try:
+        errors = validate_suite(
+            args.suite,
+            known_tools=known_tools,
+            agent_has_no_tools=args.agent_has_no_tools,
+        )
+    except Error as exc:
+        print(f"Erro: {exc}", file=sys.stderr)
+        return 1
+
+    if errors:
+        print(f"INVALID: {args.suite} ({len(errors)} error(s))", file=sys.stderr)
+        for error in errors:
+            print(f"  - {error}", file=sys.stderr)
+        return 1
+
+    print(f"OK: {args.suite}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())