agent-ci-verify 0.1.0__tar.gz

agent_ci_verify-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lewis-404
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_ci_verify-0.1.0/PKG-INFO

@@ -0,0 +1,165 @@
+Metadata-Version: 2.4
+Name: agent-ci-verify
+Version: 0.1.0
+Summary: CI/CD verification pipeline for AI agent outputs — fact check, schema validation, diff verification
+Author: Lewis-404
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Lewis-404/agent-ci
+Project-URL: Repository, https://github.com/Lewis-404/agent-ci.git
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: jsonschema>=4.20
+Requires-Dist: httpx>=0.27
+Requires-Dist: rich>=13.0
+Requires-Dist: click>=8.1
+Provides-Extra: llm
+Requires-Dist: openai>=1.0; extra == "llm"
+Requires-Dist: litellm>=1.0; extra == "llm"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.24; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.8; extra == "dev"
+Dynamic: license-file
+
+# agent-ci
+
+> CI/CD verification pipeline for AI agent outputs.  
+> **Don't trust your agent's output — verify it.**
+
+[![CI](https://github.com/Lewis-404/agent-ci/actions/workflows/ci.yml/badge.svg)](https://github.com/Lewis-404/agent-ci/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/agent-ci.svg)](https://pypi.org/project/agent-ci/)
+[![Python](https://img.shields.io/pypi/pyversions/agent-ci.svg)](https://pypi.org/project/agent-ci/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+---
+
+## Why agent-ci?
+
+AI agents are entering production, but **no one can answer "can I trust this output?"** 
+
+Existing tools are all "eval libraries" — you import them and write tests yourself. That's self-review, not independent verification.
+
+**agent-ci is your agent's CI/CD pipeline** — plug it in, and every agent output goes through an independent verification layer before it reaches your users.
+
+## Quick Start
+
+```bash
+pip install agent-ci-verify
+agent-ci ./agent-output/
+```
+
+```
+agent-ci v0.1.0
+Output dir: ./agent-output/
+Checkers: schema, fact, diff
+
+                               📋 Schema Checker
+┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ ✅   │ json_valid           │                                                ┃
+┃ ✅   │ yaml_valid           │                                                ┃
+┃ ✅   │ security_scan        │ No secrets detected                            ┃
+┗━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
+
+                               🔍 Fact Checker
+┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ ✅   │ fact:file_count      │ 1 files for '*.json'                           ┃
+┃ ✅   │ fact:content_contains│ 'success' found in result.json                 ┃
+┗━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
+
+╭────────────────────────────────── Verdict ────────────────────────────────╮
+│   ✅  PASS                                                                 │
+╰───────────────────────────────────────────────────────────────────────────╯
+```
+
+## Three Verification Layers
+
+| Layer | What it checks | Example |
+|-------|---------------|---------|
+| **Schema** | Format, structure, security | Valid JSON? API key leaked? Required files present? |
+| **Fact** | File existence, API reconciliation, LLM judging | Agent claimed `result.json` exists — does it? API returned 200? |
+| **Diff** | Regression detection, semantic drift | Output changed vs baseline? Similarity below threshold? |
+
+## Configuration
+
+Drop `.agent-ci.yaml` in your agent project root:
+
+```yaml
+pipeline:
+  enabled_checkers: [schema, fact, diff]
+  fail_fast: false
+
+schema:
+  security:
+    enabled: true
+  required_files:
+    - "output/result.json"
+  json_schemas:
+    schemas/output.schema.json: "output/**/*.json"
+
+fact:
+  files:
+    - pattern: "output/**/*.json"
+      expected_count: 1
+      min_size_bytes: 10
+      content_checks:
+        - type: contains
+          value: "success"
+        - type: not_contains
+          value: "error"
+  api:
+    - endpoint: "https://api.example.com/health"
+      expected_status: 200
+  llm_judge:
+    - file: "output/answer.md"
+      rubric: "Is the answer factually correct?"
+      model: "gpt-4o-mini"
+
+diff:
+  baseline: "./baseline-output/"
+  semantic_threshold: 0.7
+  max_changed_files: 5
+```
+
+## Security Scanning
+
+Built-in patterns detect:
+- AWS Access Keys (`AKIA...`)
+- GitHub Tokens (`ghp_...`)
+- OpenAI API Keys (`sk-proj-...`)
+- JWT Tokens
+- Private Keys (RSA, EC, DSA, OpenSSH)
+- Password/Secret assignments
+
+## CI Integration
+
+```yaml
+# .github/workflows/agent-check.yml
+- name: Verify agent output
+  run: |
+    pip install agent-ci
+    agent-ci ./output/
+```
+
+## Development
+
+```bash
+git clone https://github.com/Lewis-404/agent-ci.git
+cd agent-ci
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+## License
+
+MIT — see [LICENSE](./LICENSE)

agent_ci_verify-0.1.0/README.md

@@ -0,0 +1,133 @@
+# agent-ci
+
+> CI/CD verification pipeline for AI agent outputs.  
+> **Don't trust your agent's output — verify it.**
+
+[![CI](https://github.com/Lewis-404/agent-ci/actions/workflows/ci.yml/badge.svg)](https://github.com/Lewis-404/agent-ci/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/agent-ci.svg)](https://pypi.org/project/agent-ci/)
+[![Python](https://img.shields.io/pypi/pyversions/agent-ci.svg)](https://pypi.org/project/agent-ci/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+---
+
+## Why agent-ci?
+
+AI agents are entering production, but **no one can answer "can I trust this output?"** 
+
+Existing tools are all "eval libraries" — you import them and write tests yourself. That's self-review, not independent verification.
+
+**agent-ci is your agent's CI/CD pipeline** — plug it in, and every agent output goes through an independent verification layer before it reaches your users.
+
+## Quick Start
+
+```bash
+pip install agent-ci-verify
+agent-ci ./agent-output/
+```
+
+```
+agent-ci v0.1.0
+Output dir: ./agent-output/
+Checkers: schema, fact, diff
+
+                               📋 Schema Checker
+┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ ✅   │ json_valid           │                                                ┃
+┃ ✅   │ yaml_valid           │                                                ┃
+┃ ✅   │ security_scan        │ No secrets detected                            ┃
+┗━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
+
+                               🔍 Fact Checker
+┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ ✅   │ fact:file_count      │ 1 files for '*.json'                           ┃
+┃ ✅   │ fact:content_contains│ 'success' found in result.json                 ┃
+┗━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
+
+╭────────────────────────────────── Verdict ────────────────────────────────╮
+│   ✅  PASS                                                                 │
+╰───────────────────────────────────────────────────────────────────────────╯
+```
+
+## Three Verification Layers
+
+| Layer | What it checks | Example |
+|-------|---------------|---------|
+| **Schema** | Format, structure, security | Valid JSON? API key leaked? Required files present? |
+| **Fact** | File existence, API reconciliation, LLM judging | Agent claimed `result.json` exists — does it? API returned 200? |
+| **Diff** | Regression detection, semantic drift | Output changed vs baseline? Similarity below threshold? |
+
+## Configuration
+
+Drop `.agent-ci.yaml` in your agent project root:
+
+```yaml
+pipeline:
+  enabled_checkers: [schema, fact, diff]
+  fail_fast: false
+
+schema:
+  security:
+    enabled: true
+  required_files:
+    - "output/result.json"
+  json_schemas:
+    schemas/output.schema.json: "output/**/*.json"
+
+fact:
+  files:
+    - pattern: "output/**/*.json"
+      expected_count: 1
+      min_size_bytes: 10
+      content_checks:
+        - type: contains
+          value: "success"
+        - type: not_contains
+          value: "error"
+  api:
+    - endpoint: "https://api.example.com/health"
+      expected_status: 200
+  llm_judge:
+    - file: "output/answer.md"
+      rubric: "Is the answer factually correct?"
+      model: "gpt-4o-mini"
+
+diff:
+  baseline: "./baseline-output/"
+  semantic_threshold: 0.7
+  max_changed_files: 5
+```
+
+## Security Scanning
+
+Built-in patterns detect:
+- AWS Access Keys (`AKIA...`)
+- GitHub Tokens (`ghp_...`)
+- OpenAI API Keys (`sk-proj-...`)
+- JWT Tokens
+- Private Keys (RSA, EC, DSA, OpenSSH)
+- Password/Secret assignments
+
+## CI Integration
+
+```yaml
+# .github/workflows/agent-check.yml
+- name: Verify agent output
+  run: |
+    pip install agent-ci
+    agent-ci ./output/
+```
+
+## Development
+
+```bash
+git clone https://github.com/Lewis-404/agent-ci.git
+cd agent-ci
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+## License
+
+MIT — see [LICENSE](./LICENSE)

agent_ci_verify-0.1.0/pyproject.toml

@@ -0,0 +1,64 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[project]
+name = "agent-ci-verify"
+version = "0.1.0"
+description = "CI/CD verification pipeline for AI agent outputs — fact check, schema validation, diff verification"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+authors = [
+    { name = "Lewis-404" },
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+
+dependencies = [
+    "pyyaml>=6.0",
+    "jsonschema>=4.20",
+    "httpx>=0.27",
+    "rich>=13.0",
+    "click>=8.1",
+]
+
+[project.optional-dependencies]
+llm = [
+    "openai>=1.0",
+    "litellm>=1.0",
+]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov>=5.0",
+    "pytest-asyncio>=0.24",
+    "ruff>=0.4",
+    "mypy>=1.8",
+]
+
+[project.scripts]
+agent-ci = "agent_ci.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/Lewis-404/agent-ci"
+Repository = "https://github.com/Lewis-404/agent-ci.git"
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "W", "UP", "B", "SIM"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-v --tb=short"

agent_ci_verify-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

agent_ci_verify-0.1.0/src/agent_ci/__init__.py

@@ -0,0 +1,3 @@
+"""agent-ci — CI/CD verification pipeline for AI agent outputs."""
+
+__version__ = "0.1.0"

agent_ci_verify-0.1.0/src/agent_ci/checkers/__init__.py

@@ -0,0 +1,28 @@
+"""Checkers package — verification checkers for agent outputs."""
+
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import Any
+
+from agent_ci.types import CheckerReport
+
+
+class BaseChecker(ABC):
+    """Abstract base for all verification checkers."""
+
+    name: str = "base"
+
+    def __init__(self, config: dict[str, Any] | None = None):
+        self.config = config or {}
+
+    @abstractmethod
+    async def verify(self, output_dir: Path) -> CheckerReport:
+        """Run all checks against the given output directory."""
+        ...
+
+    def _resolve_path(self, output_dir: Path, pattern: str) -> list[Path]:
+        """Glob-resolve a pattern relative to output_dir."""
+        return sorted(output_dir.glob(pattern))
+
+
+__all__ = ["BaseChecker"]

agent_ci_verify-0.1.0/src/agent_ci/checkers/base.py

@@ -0,0 +1,5 @@
+"""Base checker (re-exported from checkers package)."""
+
+from agent_ci.checkers import BaseChecker
+
+__all__ = ["BaseChecker"]

agent_ci_verify-0.1.0/src/agent_ci/checkers/diff.py

@@ -0,0 +1,162 @@
+"""Diff Checker — compares agent output against baseline to detect regressions."""
+
+from pathlib import Path
+from typing import Any
+
+from agent_ci.checkers.base import BaseChecker
+from agent_ci.types import CheckResult, CheckerReport, Severity
+
+
+class DiffChecker(BaseChecker):
+    """Compares current agent output against a baseline for drift, regression, and anomalies."""
+
+    name = "diff"
+
+    async def verify(self, output_dir: Path) -> CheckerReport:
+        report = CheckerReport(checker_name=self.name)
+        config = self.config.get("diff", {})
+        baseline_dir = config.get("baseline")
+        if not baseline_dir:
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff",
+                severity=Severity.WARN,
+                message="No baseline directory configured — skipping diff verification",
+                detail="Set 'diff.baseline' in .agent-ci.yaml to enable diff checks.",
+            ))
+            return report
+
+        baseline = Path(baseline_dir)
+        if not baseline.exists():
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff",
+                severity=Severity.FAIL,
+                message=f"Baseline directory not found: {baseline}",
+            ))
+            return report
+
+        # Collect all text files from both directories
+        text_exts = {".json", ".yaml", ".yml", ".txt", ".md", ".py", ".js",
+                     ".ts", ".go", ".csv", ".html", ".xml", ".toml"}
+
+        current_files = {f.relative_to(output_dir): f
+                         for f in output_dir.rglob("*")
+                         if f.is_file() and f.suffix in text_exts}
+        baseline_files = {f.relative_to(baseline): f
+                          for f in baseline.rglob("*")
+                          if f.is_file() and f.suffix in text_exts}
+
+        max_changed = config.get("max_changed_files")
+        max_added = config.get("max_added_files")
+        max_removed = config.get("max_removed_files")
+        semantic_threshold = config.get("semantic_threshold", 0.7)
+
+        # 1. New files (in current but not baseline)
+        added = set(current_files) - set(baseline_files)
+        for fpath in sorted(added):
+            severity = Severity.FAIL if max_added and len(added) > max_added else Severity.WARN
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff:added",
+                severity=severity,
+                message=f"New file: {fpath}",
+            ))
+        if not added:
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff:added",
+                severity=Severity.PASS,
+                message="No new files detected",
+            ))
+
+        # 2. Removed files (in baseline but not current)
+        removed = set(baseline_files) - set(current_files)
+        for fpath in sorted(removed):
+            severity = Severity.FAIL if max_removed and len(removed) > max_removed else Severity.WARN
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff:removed",
+                severity=severity,
+                message=f"Missing file (was in baseline): {fpath}",
+            ))
+        if not removed:
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff:removed",
+                severity=Severity.PASS,
+                message="No files removed since baseline",
+            ))
+
+        # 3. Changed files
+        common = set(current_files) & set(baseline_files)
+        changed_count = 0
+        for fpath in sorted(common):
+            current_content = current_files[fpath].read_text(encoding="utf-8")
+            baseline_content = baseline_files[fpath].read_text(encoding="utf-8")
+
+            if current_content != baseline_content:
+                changed_count += 1
+                similarity = self._text_similarity(baseline_content, current_content)
+
+                severity = Severity.PASS
+                if similarity < 0.5:
+                    severity = Severity.FAIL
+                elif similarity < semantic_threshold:
+                    severity = Severity.WARN
+
+                report.checks.append(CheckResult(
+                    checker=self.name, check_name="diff:changed",
+                    severity=severity,
+                    message=f"Changed: {fpath} (similarity: {similarity:.1%})",
+                    detail=self._generate_diff(baseline_content, current_content, fpath),
+                    file_path=str(fpath),
+                ))
+
+        if changed_count == 0:
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff:changed",
+                severity=Severity.PASS,
+                message="No files changed since baseline",
+            ))
+
+        # 4. Threshold check
+        if max_changed and changed_count > max_changed:
+            report.checks.append(CheckResult(
+                checker=self.name, check_name="diff:threshold",
+                severity=Severity.FAIL,
+                message=f"Changed files ({changed_count}) exceed max ({max_changed})",
+            ))
+
+        return report
+
+    # ── Similarity ─────────────────────────────────────────────────
+
+    @staticmethod
+    def _text_similarity(text_a: str, text_b: str) -> float:
+        """Simple token-overlap similarity (Jaccard on word tokens)."""
+        if not text_a and not text_b:
+            return 1.0
+        if not text_a or not text_b:
+            return 0.0
+        tokens_a = set(text_a.lower().split())
+        tokens_b = set(text_b.lower().split())
+        intersection = tokens_a & tokens_b
+        union = tokens_a | tokens_b
+        return len(intersection) / len(union) if union else 0.0
+
+    # ── Diff generation ────────────────────────────────────────────
+
+    @staticmethod
+    def _generate_diff(before: str, after: str, relpath: Path,
+                       context_lines: int = 3) -> str:
+        """Generate a unified diff between two strings, capped for report size."""
+        import difflib
+
+        diff = list(difflib.unified_diff(
+            before.splitlines(keepends=True),
+            after.splitlines(keepends=True),
+            fromfile=f"baseline/{relpath}",
+            tofile=f"current/{relpath}",
+            n=context_lines,
+        ))
+        if not diff:
+            return "(binary or identical)"
+        # Cap at 50 lines to avoid huge reports
+        if len(diff) > 50:
+            diff = diff[:47] + ["... (truncated)\n"]
+        return "".join(diff)