agent-audit 0.1.0__tar.gz

agent_audit-0.1.0/PKG-INFO

@@ -0,0 +1,219 @@
+Metadata-Version: 2.1
+Name: agent-audit
+Version: 0.1.0
+Summary: Security scanner for AI agents and MCP configurations - Based on OWASP Agentic Top 10
+Home-page: https://github.com/HeadyZhang/agent-audit
+License: MIT
+Keywords: ai,agent,security,mcp,audit,owasp,vulnerability,scanner
+Author: Agent Security Team
+Author-email: security@example.com
+Requires-Python: >=3.9,<4.0
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Dist: aiofiles (>=23.0,<24.0)
+Requires-Dist: aiohttp (>=3.9,<4.0)
+Requires-Dist: click (>=8.1.0,<9.0.0)
+Requires-Dist: pydantic (>=2.0,<3.0)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: rich (>=13.0.0,<14.0.0)
+Project-URL: Bug Tracker, https://github.com/HeadyZhang/agent-audit/issues
+Project-URL: Changelog, https://github.com/HeadyZhang/agent-audit/releases
+Project-URL: Documentation, https://github.com/HeadyZhang/agent-audit#readme
+Project-URL: Repository, https://github.com/HeadyZhang/agent-audit
+Description-Content-Type: text/markdown
+
+# Agent Audit
+
+[![PyPI version](https://badge.fury.io/py/agent-audit.svg)](https://badge.fury.io/py/agent-audit)
+[![Python](https://img.shields.io/pypi/pyversions/agent-audit.svg)](https://pypi.org/project/agent-audit/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![CI](https://github.com/HeadyZhang/agent-audit/actions/workflows/ci.yml/badge.svg)](https://github.com/HeadyZhang/agent-audit/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/HeadyZhang/agent-audit/branch/master/graph/badge.svg)](https://codecov.io/gh/HeadyZhang/agent-audit)
+
+> 🛡️ Security scanner for AI agents and MCP configurations. Detects vulnerabilities based on the **OWASP Agentic Top 10**.
+
+<p align="center">
+  <img src="docs/demo.gif" alt="Agent Audit Demo" width="800">
+</p>
+
+## ✨ Features
+
+- **🔍 Python AST Scanning** - Detects dangerous patterns like `shell=True`, `eval()`, and tainted input flows
+- **⚙️ MCP Configuration Scanning** - Validates MCP server configurations for security issues
+- **🔐 Secret Detection** - Finds hardcoded credentials (AWS keys, API tokens, private keys)
+- **🌐 Runtime MCP Inspection** - Probes MCP servers without executing tools ("Agent Nmap")
+- **📊 Multiple Output Formats** - Terminal, JSON, SARIF (for GitHub Code Scanning), Markdown
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+pip install agent-audit
+```
+
+### Basic Usage
+
+```bash
+# Scan current directory
+agent-audit scan .
+
+# Scan with JSON output
+agent-audit scan ./my-agent --format json
+
+# Scan with SARIF output for GitHub Code Scanning
+agent-audit scan . --format sarif --output results.sarif
+
+# Fail CI on critical findings only
+agent-audit scan . --fail-on critical
+
+# Inspect an MCP server at runtime
+agent-audit inspect stdio -- npx -y @modelcontextprotocol/server-filesystem /tmp
+```
+
+## 🔗 GitHub Action
+
+Add Agent Audit to your CI/CD pipeline with just a few lines:
+
+```yaml
+name: Security Scan
+on: [push, pull_request]
+
+jobs:
+  agent-audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Run Agent Audit
+        uses: HeadyZhang/agent-audit@v1
+        with:
+          path: '.'
+          fail-on: 'high'
+          upload-sarif: 'true'
+```
+
+### Action Inputs
+
+| Input | Description | Default |
+|-------|-------------|---------|
+| `path` | Path to scan | `.` |
+| `format` | Output format: `terminal`, `json`, `sarif`, `markdown` | `sarif` |
+| `severity` | Minimum severity to report: `info`, `low`, `medium`, `high`, `critical` | `low` |
+| `fail-on` | Exit with error if findings at this severity | `high` |
+| `baseline` | Path to baseline file for incremental scanning | - |
+| `upload-sarif` | Upload SARIF to GitHub Security tab | `true` |
+
+## 🎯 Detected Issues
+
+| Rule ID | Title | Severity |
+|---------|-------|----------|
+| AGENT-001 | Command Injection via Unsanitized Input | 🔴 Critical |
+| AGENT-002 | Excessive Agent Permissions | 🟡 Medium |
+| AGENT-003 | Potential Data Exfiltration Chain | 🟠 High |
+| AGENT-004 | Hardcoded Credentials | 🔴 Critical |
+| AGENT-005 | Unverified MCP Server | 🟠 High |
+
+## ⚙️ Configuration
+
+Create a `.agent-audit.yaml` file to customize scanning:
+
+```yaml
+# Allowed network hosts (reduces AGENT-003 confidence)
+allowed_hosts:
+  - "*.internal.company.com"
+  - "api.openai.com"
+
+# Ignore rules
+ignore:
+  - rule_id: AGENT-003
+    paths:
+      - "auth/**"
+    reason: "Auth module legitimately communicates externally"
+
+# Scan settings
+scan:
+  exclude:
+    - "tests/**"
+    - "venv/**"
+  min_severity: low
+  fail_on: high
+```
+
+## 📈 Baseline Scanning
+
+Track new findings incrementally:
+
+```bash
+# Save current findings as baseline
+agent-audit scan . --save-baseline baseline.json
+
+# Only report new findings
+agent-audit scan . --baseline baseline.json
+```
+
+## 📖 CLI Reference
+
+```
+Usage: agent-audit [OPTIONS] COMMAND [ARGS]...
+
+Commands:
+  scan     Scan agent code and configurations
+  inspect  Inspect an MCP server at runtime
+  init     Initialize configuration file
+
+Options:
+  --version   Show version
+  -v          Enable verbose output
+  -q          Only show errors
+  --help      Show this message
+```
+
+## 🛠️ Development
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.
+
+```bash
+# Clone the repository
+git clone https://github.com/HeadyZhang/agent-audit
+cd agent-security-suite
+
+# Install dependencies
+cd packages/core && poetry install
+cd ../audit && poetry install
+
+# Run tests
+poetry run pytest tests/ -v
+
+# Run the scanner
+poetry run agent-audit scan .
+```
+
+## 📄 License
+
+MIT License - see [LICENSE](LICENSE) for details.
+
+## 🙏 Acknowledgments
+
+- Based on the [OWASP Agentic Security Top 10](https://owasp.org/www-project-agentic-security/)
+- Inspired by the need for better AI agent security tooling
+
+---
+
+<p align="center">
+  Made with ❤️ for the AI agent security community
+</p>
+

agent_audit-0.1.0/README.md

@@ -0,0 +1,181 @@
+# Agent Audit
+
+[![PyPI version](https://badge.fury.io/py/agent-audit.svg)](https://badge.fury.io/py/agent-audit)
+[![Python](https://img.shields.io/pypi/pyversions/agent-audit.svg)](https://pypi.org/project/agent-audit/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![CI](https://github.com/HeadyZhang/agent-audit/actions/workflows/ci.yml/badge.svg)](https://github.com/HeadyZhang/agent-audit/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/HeadyZhang/agent-audit/branch/master/graph/badge.svg)](https://codecov.io/gh/HeadyZhang/agent-audit)
+
+> 🛡️ Security scanner for AI agents and MCP configurations. Detects vulnerabilities based on the **OWASP Agentic Top 10**.
+
+<p align="center">
+  <img src="docs/demo.gif" alt="Agent Audit Demo" width="800">
+</p>
+
+## ✨ Features
+
+- **🔍 Python AST Scanning** - Detects dangerous patterns like `shell=True`, `eval()`, and tainted input flows
+- **⚙️ MCP Configuration Scanning** - Validates MCP server configurations for security issues
+- **🔐 Secret Detection** - Finds hardcoded credentials (AWS keys, API tokens, private keys)
+- **🌐 Runtime MCP Inspection** - Probes MCP servers without executing tools ("Agent Nmap")
+- **📊 Multiple Output Formats** - Terminal, JSON, SARIF (for GitHub Code Scanning), Markdown
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+pip install agent-audit
+```
+
+### Basic Usage
+
+```bash
+# Scan current directory
+agent-audit scan .
+
+# Scan with JSON output
+agent-audit scan ./my-agent --format json
+
+# Scan with SARIF output for GitHub Code Scanning
+agent-audit scan . --format sarif --output results.sarif
+
+# Fail CI on critical findings only
+agent-audit scan . --fail-on critical
+
+# Inspect an MCP server at runtime
+agent-audit inspect stdio -- npx -y @modelcontextprotocol/server-filesystem /tmp
+```
+
+## 🔗 GitHub Action
+
+Add Agent Audit to your CI/CD pipeline with just a few lines:
+
+```yaml
+name: Security Scan
+on: [push, pull_request]
+
+jobs:
+  agent-audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Run Agent Audit
+        uses: HeadyZhang/agent-audit@v1
+        with:
+          path: '.'
+          fail-on: 'high'
+          upload-sarif: 'true'
+```
+
+### Action Inputs
+
+| Input | Description | Default |
+|-------|-------------|---------|
+| `path` | Path to scan | `.` |
+| `format` | Output format: `terminal`, `json`, `sarif`, `markdown` | `sarif` |
+| `severity` | Minimum severity to report: `info`, `low`, `medium`, `high`, `critical` | `low` |
+| `fail-on` | Exit with error if findings at this severity | `high` |
+| `baseline` | Path to baseline file for incremental scanning | - |
+| `upload-sarif` | Upload SARIF to GitHub Security tab | `true` |
+
+## 🎯 Detected Issues
+
+| Rule ID | Title | Severity |
+|---------|-------|----------|
+| AGENT-001 | Command Injection via Unsanitized Input | 🔴 Critical |
+| AGENT-002 | Excessive Agent Permissions | 🟡 Medium |
+| AGENT-003 | Potential Data Exfiltration Chain | 🟠 High |
+| AGENT-004 | Hardcoded Credentials | 🔴 Critical |
+| AGENT-005 | Unverified MCP Server | 🟠 High |
+
+## ⚙️ Configuration
+
+Create a `.agent-audit.yaml` file to customize scanning:
+
+```yaml
+# Allowed network hosts (reduces AGENT-003 confidence)
+allowed_hosts:
+  - "*.internal.company.com"
+  - "api.openai.com"
+
+# Ignore rules
+ignore:
+  - rule_id: AGENT-003
+    paths:
+      - "auth/**"
+    reason: "Auth module legitimately communicates externally"
+
+# Scan settings
+scan:
+  exclude:
+    - "tests/**"
+    - "venv/**"
+  min_severity: low
+  fail_on: high
+```
+
+## 📈 Baseline Scanning
+
+Track new findings incrementally:
+
+```bash
+# Save current findings as baseline
+agent-audit scan . --save-baseline baseline.json
+
+# Only report new findings
+agent-audit scan . --baseline baseline.json
+```
+
+## 📖 CLI Reference
+
+```
+Usage: agent-audit [OPTIONS] COMMAND [ARGS]...
+
+Commands:
+  scan     Scan agent code and configurations
+  inspect  Inspect an MCP server at runtime
+  init     Initialize configuration file
+
+Options:
+  --version   Show version
+  -v          Enable verbose output
+  -q          Only show errors
+  --help      Show this message
+```
+
+## 🛠️ Development
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.
+
+```bash
+# Clone the repository
+git clone https://github.com/HeadyZhang/agent-audit
+cd agent-security-suite
+
+# Install dependencies
+cd packages/core && poetry install
+cd ../audit && poetry install
+
+# Run tests
+poetry run pytest tests/ -v
+
+# Run the scanner
+poetry run agent-audit scan .
+```
+
+## 📄 License
+
+MIT License - see [LICENSE](LICENSE) for details.
+
+## 🙏 Acknowledgments
+
+- Based on the [OWASP Agentic Security Top 10](https://owasp.org/www-project-agentic-security/)
+- Inspired by the need for better AI agent security tooling
+
+---
+
+<p align="center">
+  Made with ❤️ for the AI agent security community
+</p>

agent_audit-0.1.0/agent_audit/__init__.py

@@ -0,0 +1,3 @@
+"""Agent Audit - Security scanner for AI agents and MCP configurations."""
+
+__version__ = "0.1.0"

agent_audit-0.1.0/agent_audit/__main__.py

@@ -0,0 +1,13 @@
+"""Entry point for running agent-audit as a module."""
+
+import sys
+
+# Windows event loop policy fix - must be set before any asyncio imports
+if sys.platform == "win32":
+    import asyncio
+    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
+
+from agent_audit.cli.main import cli
+
+if __name__ == "__main__":
+    cli()

agent_audit-0.1.0/agent_audit/cli/__init__.py

@@ -0,0 +1 @@
+"""CLI module for agent-audit."""

agent_audit-0.1.0/agent_audit/cli/commands/__init__.py

@@ -0,0 +1 @@
+"""CLI commands for agent-audit."""

agent_audit-0.1.0/agent_audit/cli/commands/init.py

@@ -0,0 +1,44 @@
+"""Init command for creating configuration files."""
+
+import sys
+from pathlib import Path
+
+import click
+from rich.console import Console
+
+from agent_audit.config.ignore import create_default_config
+
+console = Console()
+
+
+@click.command()
+@click.option('--force', '-f', is_flag=True, help='Overwrite existing config')
+def init(force: bool):
+    """
+    Initialize agent-audit configuration.
+
+    Creates a .agent-audit.yaml file in the current directory with
+    default settings and example ignore rules.
+
+    Examples:
+
+        agent-audit init
+
+        agent-audit init --force
+    """
+    config_path = Path('.agent-audit.yaml')
+
+    if config_path.exists() and not force:
+        console.print(f"[yellow]Configuration file already exists: {config_path}[/yellow]")
+        console.print("Use --force to overwrite")
+        sys.exit(1)
+
+    config_content = create_default_config()
+    config_path.write_text(config_content, encoding="utf-8")
+
+    console.print(f"[green]Created configuration file: {config_path}[/green]")
+    console.print()
+    console.print("Edit this file to:")
+    console.print("  - Add allowed hosts for network destinations")
+    console.print("  - Configure ignore rules for false positives")
+    console.print("  - Set scan exclusion patterns")

agent_audit-0.1.0/agent_audit/cli/commands/inspect.py

@@ -0,0 +1,236 @@
+"""Inspect command for probing MCP servers."""
+
+import asyncio
+import sys
+from typing import Optional
+
+import click
+from rich.console import Console
+from rich.table import Table
+from rich.panel import Panel
+
+from agent_audit.scanners.mcp_inspector import MCPInspector, MCPInspectionResult
+from agent_audit.utils.mcp_client import TransportType
+
+console = Console()
+
+
+def render_inspection_result(result: MCPInspectionResult, output_format: str = "terminal"):
+    """Render inspection result to the console."""
+    if output_format == "json":
+        _render_json(result)
+    else:
+        _render_terminal(result)
+
+
+def _render_terminal(result: MCPInspectionResult):
+    """Render result as Rich terminal output."""
+    # Status indicator
+    if result.connected:
+        status = "[green]✓ Connected[/green]"
+        border_style = "blue" if result.risk_score < 5 else "red"
+    else:
+        status = "[red]✗ Failed[/red]"
+        border_style = "red"
+
+    # Header panel
+    header_text = (
+        f"[bold]MCP Server Inspection[/bold]\n"
+        f"Server: {result.server_name}"
+    )
+    if result.server_version:
+        header_text += f" v{result.server_version}"
+    header_text += "\n"
+    header_text += f"Status: {status}  |  Response: {result.response_time_ms:.0f}ms\n"
+    header_text += f"Risk Score: {result.risk_score:.1f}/10"
+
+    console.print(Panel.fit(header_text, border_style=border_style))
+
+    if not result.connected:
+        console.print(f"[red]Error: {result.connection_error}[/red]")
+        return
+
+    # Capabilities
+    if result.capabilities_declared:
+        caps = ", ".join(result.capabilities_declared) or "none"
+        console.print(f"\n[dim]Capabilities:[/dim] {caps}")
+
+    # Tools table
+    console.print(f"\n[bold]Tools ({result.tool_count})[/bold]")
+
+    if result.tools:
+        tool_table = Table(show_header=True, header_style="bold cyan")
+        tool_table.add_column("Tool", style="cyan")
+        tool_table.add_column("Permissions", style="yellow")
+        tool_table.add_column("Risk", justify="center")
+        tool_table.add_column("Validation")
+
+        risk_emoji = {
+            1: "🟢",  # SAFE
+            2: "🟢",  # LOW
+            3: "🟡",  # MEDIUM
+            4: "🟠",  # HIGH
+            5: "🔴",  # CRITICAL
+        }
+
+        for tool in result.tools:
+            perms = ", ".join(p.name for p in tool.permissions) or "none"
+            risk_value = tool.risk_level.value if hasattr(tool.risk_level, 'value') else 1
+            risk = risk_emoji.get(risk_value, "⚪")
+            validation = "✅" if tool.has_input_validation else "❌"
+
+            # Truncate description if needed
+            tool_name = tool.name
+            if len(tool_name) > 30:
+                tool_name = tool_name[:27] + "..."
+
+            tool_table.add_row(tool_name, perms, risk, validation)
+
+        console.print(tool_table)
+    else:
+        console.print("[dim]No tools exposed[/dim]")
+
+    # Resources
+    if result.resources:
+        console.print(f"\n[bold]Resources ({result.resource_count})[/bold]")
+        for res in result.resources[:10]:  # Limit display
+            uri = res.get('uri', 'unknown')
+            console.print(f"  📄 {uri}")
+        if result.resource_count > 10:
+            console.print(f"  [dim]... and {result.resource_count - 10} more[/dim]")
+
+    # Prompts
+    if result.prompts:
+        console.print(f"\n[bold]Prompts ({result.prompt_count})[/bold]")
+        for prompt in result.prompts[:10]:
+            name = prompt.get('name', 'unknown')
+            console.print(f"  💬 {name}")
+        if result.prompt_count > 10:
+            console.print(f"  [dim]... and {result.prompt_count - 10} more[/dim]")
+
+    # Security findings
+    if result.findings:
+        console.print(f"\n[bold red]Security Findings ({len(result.findings)})[/bold red]")
+
+        severity_colors = {
+            "critical": "red",
+            "high": "red",
+            "medium": "yellow",
+            "low": "blue"
+        }
+
+        for finding in result.findings:
+            severity = finding.get("severity", "medium")
+            color = severity_colors.get(severity, "white")
+            desc = finding.get("description", "Unknown issue")
+            tool_name = finding.get("tool", "")
+
+            if tool_name:
+                console.print(f"  [{color}]⚠ {severity.upper()}[/{color}]: {desc} (tool: {tool_name})")
+            else:
+                console.print(f"  [{color}]⚠ {severity.upper()}[/{color}]: {desc}")
+
+
+def _render_json(result: MCPInspectionResult):
+    """Render result as JSON."""
+    import json
+
+    output = {
+        "server_name": result.server_name,
+        "server_version": result.server_version,
+        "transport": result.transport.value,
+        "connected": result.connected,
+        "connection_error": result.connection_error,
+        "response_time_ms": result.response_time_ms,
+        "risk_score": result.risk_score,
+        "capabilities": result.capabilities_declared,
+        "tool_count": result.tool_count,
+        "tools": [t.to_dict() for t in result.tools],
+        "resource_count": result.resource_count,
+        "resources": result.resources,
+        "prompt_count": result.prompt_count,
+        "prompts": result.prompts,
+        "findings": result.findings,
+    }
+
+    console.print_json(json.dumps(output, indent=2))
+
+
+async def run_inspect_async(
+    target: str,
+    transport: Optional[str],
+    timeout: int,
+    output_format: str
+) -> int:
+    """Run the inspection asynchronously."""
+    inspector = MCPInspector(timeout=timeout)
+
+    # Determine transport type
+    transport_type: Optional[TransportType] = None
+    if transport:
+        transport_type = TransportType(transport)
+
+    result = await inspector.inspect(target, transport_type)
+    render_inspection_result(result, output_format)
+
+    # Return exit code based on risk
+    if not result.connected:
+        return 2  # Connection failure
+    elif result.risk_score >= 7.0:
+        return 1  # High risk
+    return 0
+
+
+def run_inspect(
+    target: str,
+    transport: Optional[str],
+    timeout: int,
+    output_format: str
+) -> int:
+    """Run the inspection."""
+    return asyncio.run(run_inspect_async(target, transport, timeout, output_format))
+
+
+@click.command()
+@click.argument('transport_type', type=click.Choice(['stdio', 'sse']), required=True)
+@click.argument('target', nargs=-1, required=True)
+@click.option('--timeout', '-t', default=30, help='Connection timeout in seconds')
+@click.option('--format', '-f', 'output_format',
+              type=click.Choice(['terminal', 'json']), default='terminal',
+              help='Output format')
+def inspect(transport_type: str, target: tuple, timeout: int, output_format: str):
+    """
+    Inspect a running MCP server and analyze its tools.
+
+    TRANSPORT_TYPE is either 'stdio' or 'sse'.
+
+    For stdio, TARGET is the command to run the server:
+
+        agent-audit inspect stdio -- python my_mcp_server.py
+
+    For sse, TARGET is the URL:
+
+        agent-audit inspect sse https://example.com/sse
+
+    The inspector connects to the server, retrieves its tool definitions,
+    and analyzes them for security risks WITHOUT executing any tools.
+    """
+    # Join target parts back together
+    target_str = ' '.join(target)
+
+    # Handle the "--" separator for stdio
+    if target_str.startswith('-- '):
+        target_str = target_str[3:]
+
+    if not target_str:
+        console.print("[red]Error: No target specified[/red]")
+        sys.exit(1)
+
+    exit_code = run_inspect(
+        target=target_str,
+        transport=transport_type,
+        timeout=timeout,
+        output_format=output_format
+    )
+
+    sys.exit(exit_code)