@ngocsangairvds/vsaf 3.2.13 → 3.2.15

package/tools/vds-scripts/audit_orchestrator/docs/LLM_AGENT_AUDIT_SPEC.md

@@ -1,721 +0,0 @@
-# LLM Agent-Based Audit System Specification
-
-**Version:** 1.0.0
-**Status:** DRAFT
-**Created:** 2026-01-31
-
----
-
-## 1. Executive Summary
-
-This specification defines the evolution of the VDS Audit Orchestrator to leverage **LLM AI agents** for intelligent, context-aware compliance analysis. Instead of rule-based pattern matching, agents will:
-
-1. **Understand audit requirements** in natural language
-2. **Analyze code and documentation** with semantic understanding
-3. **Reason about compliance** based on context and intent
-4. **Generate actionable recommendations** with explanations
-
-### Evolution Path
-
-| Phase | Approach | Intelligence Level |
-|-------|----------|-------------------|
-| **Current** | Rule-based checks | Pattern matching |
-| **Phase 1** | Content-based scoring | Keyword + semantic similarity |
-| **Phase 2** | Single LLM analysis | GPT/Claude for each check |
-| **Phase 3** | Multi-agent orchestration | Specialized agents collaborate |
-
----
-
-## 2. Industry Research
-
-### 2.1 RepoAudit (State-of-the-Art)
-
-Reference: [RepoAudit - Autonomous LLM-Agent for Repository-Level Code Auditing](https://github.com/PurCL/RepoAudit)
-
-**Architecture:**
-- Multi-agent framework with specialized analyzers
-- Parsing-based analyzers + LLM-driven tools
-- Agent memory for context accumulation
-- Tracks value flow through execution paths
-
-**Key Innovation:** Autonomous repository-level auditing that understands code semantics, not just patterns.
-
-### 2.2 Anthropic Alignment Auditing Agents
-
-Reference: [Building and evaluating alignment auditing agents](https://alignment.anthropic.com/2025/automated-auditing/)
-
-**Three Agent Types:**
-1. **Goal Discovery Agent** - Uncovers hidden goals/behaviors
-2. **Evaluation Builder Agent** - Constructs behavioral tests
-3. **Behavior Surfacing Agent** - Identifies concerning patterns
-
-**Key Innovation:** Agents autonomously design and execute auditing strategies.
-
-### 2.3 AudAgent (Privacy Compliance)
-
-Reference: [AudAgent: Automated Auditing of Privacy Policy Compliance](https://arxiv.org/html/2511.07441v1)
-
-**Architecture:**
-1. **Policy Parsing** - Ensemble of LLMs translates policies to structured format
-2. **Cross-LLM Voting** - Multiple LLMs vote for confidence
-3. **Compliance Verification** - Automated policy checking
-
-**Key Innovation:** Ensemble approach with voting for high-confidence parsing.
-
-### 2.4 Governance-as-a-Service
-
-Reference: [Microsoft Agent Governance](https://microsoft.com)
-
-**Framework:**
-- Runtime policy enforcement layer
-- Trust scores based on compliance history
-- Blocks/redirects high-risk behaviors
-- Logs all agent actions for audit trail
-
-**Key Innovation:** Treating governance as infrastructure-level service.
-
----
-
-## 3. Proposed Multi-Agent Architecture
-
-```
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                    LLM AGENT AUDIT ORCHESTRATOR                             │
-├─────────────────────────────────────────────────────────────────────────────┤
-│                                                                             │
-│  ┌─────────────────────────────────────────────────────────────────────┐   │
-│  │                      ORCHESTRATOR AGENT                              │   │
-│  │  - Receives audit template + project context                        │   │
-│  │  - Decomposes into specialized agent tasks                          │   │
-│  │  - Aggregates results and resolves conflicts                        │   │
-│  │  - Generates final compliance report                                │   │
-│  └─────────────────────────────────────────────────────────────────────┘   │
-│                                    │                                        │
-│         ┌──────────────────────────┼──────────────────────────┐            │
-│         │                          │                          │            │
-│         ▼                          ▼                          ▼            │
-│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐        │
-│  │ CODE ANALYSIS   │    │ DOCUMENTATION   │    │ SECURITY        │        │
-│  │ AGENT           │    │ ANALYSIS AGENT  │    │ ANALYSIS AGENT  │        │
-│  │                 │    │                 │    │                 │        │
-│  │ - Code quality  │    │ - README eval   │    │ - Vuln scanning │        │
-│  │ - Test coverage │    │ - API docs      │    │ - Config review │        │
-│  │ - Architecture  │    │ - Confluence    │    │ - Secrets check │        │
-│  │ - Dependencies  │    │ - Completeness  │    │ - OWASP checks  │        │
-│  └────────┬────────┘    └────────┬────────┘    └────────┬────────┘        │
-│           │                      │                      │                  │
-│           └──────────────────────┼──────────────────────┘                  │
-│                                  ▼                                         │
-│  ┌─────────────────────────────────────────────────────────────────────┐   │
-│  │                      EVIDENCE SYNTHESIS AGENT                        │   │
-│  │  - Cross-references findings from all agents                        │   │
-│  │  - Resolves conflicting assessments                                 │   │
-│  │  - Calculates confidence scores                                     │   │
-│  │  - Generates unified compliance score                               │   │
-│  └─────────────────────────────────────────────────────────────────────┘   │
-│                                  │                                         │
-│                                  ▼                                         │
-│  ┌─────────────────────────────────────────────────────────────────────┐   │
-│  │                      RECOMMENDATION AGENT                            │   │
-│  │  - Prioritizes remediation actions                                  │   │
-│  │  - Generates code fix suggestions                                   │   │
-│  │  - Creates improvement roadmap                                      │   │
-│  │  - Estimates effort for each fix                                    │   │
-│  └─────────────────────────────────────────────────────────────────────┘   │
-│                                                                             │
-└─────────────────────────────────────────────────────────────────────────────┘
-```
-
----
-
-## 4. Agent Definitions
-
-### 4.1 Orchestrator Agent
-
-```python
-class OrchestratorAgent:
-    """Master agent that coordinates the audit workflow."""
-
-    SYSTEM_PROMPT = """
-    You are the Audit Orchestrator Agent. Your role is to:
-    1. Analyze the audit template to understand compliance requirements
-    2. Decompose requirements into tasks for specialized agents
-    3. Coordinate agent execution and handle dependencies
-    4. Aggregate results into a unified compliance assessment
-    5. Resolve conflicts between agent findings
-
-    You have access to these specialized agents:
-    - CodeAnalysisAgent: Analyzes source code quality and structure
-    - DocumentationAgent: Evaluates documentation completeness
-    - SecurityAgent: Assesses security posture and vulnerabilities
-    - EvidenceSynthesisAgent: Cross-references and validates findings
-    - RecommendationAgent: Generates remediation suggestions
-
-    Output format: Structured JSON with scores, findings, and recommendations.
-    """
-
-    async def orchestrate(
-        self,
-        template: AuditTemplate,
-        project: ProjectConfig,
-    ) -> ProjectAuditResult:
-        """Execute full audit workflow."""
-
-        # 1. Parse template requirements
-        requirements = await self._parse_requirements(template)
-
-        # 2. Gather context from all sources
-        context = await self._gather_context(project)
-
-        # 3. Dispatch to specialized agents
-        code_results = await self.code_agent.analyze(context.code, requirements.code)
-        doc_results = await self.doc_agent.analyze(context.docs, requirements.docs)
-        security_results = await self.security_agent.analyze(context, requirements.security)
-
-        # 4. Synthesize evidence
-        synthesis = await self.synthesis_agent.synthesize(
-            code_results, doc_results, security_results
-        )
-
-        # 5. Generate recommendations
-        recommendations = await self.recommendation_agent.generate(synthesis)
-
-        # 6. Build final result
-        return self._build_result(synthesis, recommendations)
-```
-
-### 4.2 Code Analysis Agent
-
-```python
-class CodeAnalysisAgent:
-    """Agent specialized in source code analysis."""
-
-    SYSTEM_PROMPT = """
-    You are a Code Analysis Agent specializing in evaluating code quality,
-    architecture, and compliance with best practices.
-
-    For each code artifact, you should assess:
-    1. Code Quality (0-10): Structure, readability, maintainability
-    2. Test Coverage (0-10): Test presence, quality, coverage percentage
-    3. Architecture (0-10): Design patterns, modularity, dependencies
-    4. Documentation (0-10): Inline comments, docstrings, type hints
-
-    Provide evidence for each assessment with specific file references.
-    Identify issues with severity (Critical, High, Medium, Low, Info).
-    Suggest specific improvements with code examples when helpful.
-
-    Output: Structured analysis with scores, evidence, and recommendations.
-    """
-
-    async def analyze(
-        self,
-        code_context: CodeContext,
-        requirements: list[CodeRequirement],
-    ) -> CodeAnalysisResult:
-        """Analyze code against requirements."""
-
-        analyses = []
-
-        for requirement in requirements:
-            # Build prompt with requirement and relevant code
-            prompt = self._build_analysis_prompt(requirement, code_context)
-
-            # Call LLM for analysis
-            response = await self.llm.analyze(
-                system_prompt=self.SYSTEM_PROMPT,
-                user_prompt=prompt,
-                context=code_context.relevant_files(requirement),
-            )
-
-            # Parse structured response
-            analysis = self._parse_response(response, requirement)
-            analyses.append(analysis)
-
-        return CodeAnalysisResult(analyses=analyses)
-
-    def _build_analysis_prompt(
-        self,
-        requirement: CodeRequirement,
-        context: CodeContext,
-    ) -> str:
-        return f"""
-        ## Requirement
-        {requirement.description}
-
-        ## Expected Evidence
-        {requirement.expected_evidence}
-
-        ## Code to Analyze
-        {context.get_relevant_code(requirement)}
-
-        ## Task
-        Analyze the code above against the requirement.
-        Provide:
-        1. Compliance score (0-10) with justification
-        2. Specific evidence from the code
-        3. Issues found with severity
-        4. Recommendations for improvement
-
-        Respond in JSON format.
-        """
-```
-
-### 4.3 Documentation Analysis Agent
-
-```python
-class DocumentationAnalysisAgent:
-    """Agent specialized in documentation quality assessment."""
-
-    SYSTEM_PROMPT = """
-    You are a Documentation Analysis Agent specializing in evaluating
-    technical documentation quality and completeness.
-
-    For each documentation requirement, assess:
-    1. Existence (0-2): Does the documentation exist?
-    2. Completeness (0-3): Are all required sections present?
-    3. Quality (0-3): Is the content clear, accurate, and helpful?
-    4. Recency (0-2): Is the documentation up-to-date?
-
-    Consider these quality factors:
-    - Clear structure with proper headings
-    - Code examples and usage instructions
-    - API reference with parameters and return values
-    - Troubleshooting and FAQ sections
-    - Diagrams and visual aids where appropriate
-
-    Cross-reference documentation claims with actual code behavior.
-    Flag any discrepancies between docs and implementation.
-    """
-
-    async def analyze(
-        self,
-        doc_context: DocumentationContext,
-        requirements: list[DocRequirement],
-    ) -> DocumentationAnalysisResult:
-        """Analyze documentation against requirements."""
-
-        # Analyze README
-        readme_analysis = await self._analyze_readme(doc_context.readme)
-
-        # Analyze Confluence pages
-        confluence_analyses = []
-        for page in doc_context.confluence_pages:
-            analysis = await self._analyze_confluence_page(page, requirements)
-            confluence_analyses.append(analysis)
-
-        # Cross-reference with code
-        cross_ref = await self._cross_reference(
-            doc_context.documentation,
-            doc_context.code_context,
-        )
-
-        return DocumentationAnalysisResult(
-            readme=readme_analysis,
-            confluence=confluence_analyses,
-            cross_reference=cross_ref,
-        )
-
-    async def _analyze_confluence_page(
-        self,
-        page: ConfluencePage,
-        requirements: list[DocRequirement],
-    ) -> PageAnalysis:
-        """Analyze a Confluence page for quality and completeness."""
-
-        prompt = f"""
-        ## Page: {page.title}
-        ## Content:
-        {page.content[:10000]}  # Truncate for context window
-
-        ## Requirements:
-        {[r.description for r in requirements if r.applies_to(page)]}
-
-        ## Task:
-        Analyze this documentation page for:
-        1. Content quality and clarity
-        2. Completeness against requirements
-        3. Technical accuracy (flag anything that seems outdated)
-        4. Structure and organization
-        5. Missing sections or information
-
-        Score each dimension 0-10 and provide specific evidence.
-        """
-
-        response = await self.llm.analyze(
-            system_prompt=self.SYSTEM_PROMPT,
-            user_prompt=prompt,
-        )
-
-        return self._parse_page_analysis(response)
-```
-
-### 4.4 Security Analysis Agent
-
-```python
-class SecurityAnalysisAgent:
-    """Agent specialized in security assessment."""
-
-    SYSTEM_PROMPT = """
-    You are a Security Analysis Agent specializing in identifying
-    security vulnerabilities and compliance issues.
-
-    Analyze for:
-    1. Code Security (0-10): Injection, XSS, CSRF, auth issues
-    2. Configuration Security (0-10): Secrets, permissions, defaults
-    3. Dependency Security (0-10): Known vulnerabilities, outdated packages
-    4. Infrastructure Security (0-10): Docker, CI/CD, deployment
-
-    Use OWASP Top 10 as a reference framework.
-    Classify findings by:
-    - Severity: Critical, High, Medium, Low, Info
-    - Confidence: High, Medium, Low
-    - CVSS score when applicable
-
-    Provide remediation guidance for each finding.
-    """
-
-    async def analyze(
-        self,
-        context: AuditContext,
-        requirements: list[SecurityRequirement],
-    ) -> SecurityAnalysisResult:
-        """Perform security analysis."""
-
-        findings = []
-
-        # Analyze code for security issues
-        code_findings = await self._analyze_code_security(context.code)
-        findings.extend(code_findings)
-
-        # Analyze configurations
-        config_findings = await self._analyze_configs(context.configs)
-        findings.extend(config_findings)
-
-        # Check dependencies
-        dep_findings = await self._analyze_dependencies(context.dependencies)
-        findings.extend(dep_findings)
-
-        # Analyze CI/CD security
-        cicd_findings = await self._analyze_cicd(context.cicd_config)
-        findings.extend(cicd_findings)
-
-        return SecurityAnalysisResult(
-            findings=findings,
-            overall_score=self._calculate_security_score(findings),
-        )
-```
-
-### 4.5 Evidence Synthesis Agent
-
-```python
-class EvidenceSynthesisAgent:
-    """Agent that cross-references and synthesizes findings."""
-
-    SYSTEM_PROMPT = """
-    You are an Evidence Synthesis Agent responsible for:
-    1. Cross-referencing findings from multiple analysis agents
-    2. Resolving conflicting assessments
-    3. Validating evidence quality and confidence
-    4. Calculating unified compliance scores
-
-    When agents disagree:
-    - Analyze the evidence each agent provides
-    - Consider which evidence is more concrete/verifiable
-    - Flag unresolved conflicts for human review
-
-    Assign confidence levels to final assessments:
-    - High: Multiple agents agree with strong evidence
-    - Medium: Single agent with good evidence or minor conflicts
-    - Low: Conflicting findings or weak evidence
-    """
-
-    async def synthesize(
-        self,
-        code_results: CodeAnalysisResult,
-        doc_results: DocumentationAnalysisResult,
-        security_results: SecurityAnalysisResult,
-    ) -> SynthesisResult:
-        """Synthesize findings from all agents."""
-
-        prompt = f"""
-        ## Code Analysis Findings:
-        {code_results.to_summary()}
-
-        ## Documentation Analysis Findings:
-        {doc_results.to_summary()}
-
-        ## Security Analysis Findings:
-        {security_results.to_summary()}
-
-        ## Task:
-        1. Cross-reference these findings
-        2. Identify any conflicts or inconsistencies
-        3. Calculate overall compliance score for each section
-        4. Assign confidence levels
-        5. Highlight areas needing human review
-
-        Output unified assessment in JSON format.
-        """
-
-        response = await self.llm.analyze(
-            system_prompt=self.SYSTEM_PROMPT,
-            user_prompt=prompt,
-        )
-
-        return self._parse_synthesis(response)
-```
-
----
-
-## 5. Integration with PydanticAI
-
-### 5.1 Agent Definition
-
-```python
-from pydantic import BaseModel
-from pydantic_ai import Agent, ModelRetry, RunContext
-
-class AuditOutput(BaseModel):
-    summary: str
-    confidence: float
-
-agent = Agent(
-    "anthropic:claude-sonnet-4",
-    output_type=AuditOutput,
-    deps_type=AuditDeps,
-)
-
-@agent.output_validator
-def validate_output(ctx: RunContext[AuditDeps], output: AuditOutput) -> AuditOutput:
-    if output.confidence < 0.4:
-        raise ModelRetry("Low confidence; retry with more evidence.")
-    return output
-```
-
-### 5.2 Tool Integration
-
-```python
-# Define tools available to agents
-AUDIT_TOOLS = [
-    {
-        "name": "read_file",
-        "description": "Read a file from the repository",
-        "input_schema": {
-            "type": "object",
-            "properties": {
-                "path": {"type": "string", "description": "File path"},
-            },
-            "required": ["path"],
-        },
-    },
-    {
-        "name": "search_code",
-        "description": "Search for patterns in code",
-        "input_schema": {
-            "type": "object",
-            "properties": {
-                "pattern": {"type": "string"},
-                "file_type": {"type": "string"},
-            },
-            "required": ["pattern"],
-        },
-    },
-    {
-        "name": "get_confluence_page",
-        "description": "Fetch a Confluence documentation page",
-        "input_schema": {
-            "type": "object",
-            "properties": {
-                "space": {"type": "string"},
-                "title": {"type": "string"},
-            },
-            "required": ["space", "title"],
-        },
-    },
-    {
-        "name": "check_sonarqube",
-        "description": "Get SonarQube metrics for a project",
-        "input_schema": {
-            "type": "object",
-            "properties": {
-                "project_key": {"type": "string"},
-                "metrics": {"type": "array", "items": {"type": "string"}},
-            },
-            "required": ["project_key"],
-        },
-    },
-]
-
-
-class ToolEnabledAgent(AuditAgent):
-    """Agent with tool-use capabilities."""
-
-    async def analyze_with_tools(
-        self,
-        system_prompt: str,
-        user_prompt: str,
-    ) -> dict:
-        """Execute analysis with tool access."""
-
-        messages = [{"role": "user", "content": user_prompt}]
-
-        while True:
-            response = await self.client.messages.create(
-                model=self.model,
-                max_tokens=self.max_tokens,
-                system=system_prompt,
-                messages=messages,
-                tools=AUDIT_TOOLS,
-            )
-
-            # Check if agent wants to use a tool
-            if response.stop_reason == "tool_use":
-                tool_use = next(
-                    block for block in response.content
-                    if block.type == "tool_use"
-                )
-
-                # Execute tool
-                tool_result = await self._execute_tool(
-                    tool_use.name,
-                    tool_use.input,
-                )
-
-                # Add tool result to conversation
-                messages.append({"role": "assistant", "content": response.content})
-                messages.append({
-                    "role": "user",
-                    "content": [{
-                        "type": "tool_result",
-                        "tool_use_id": tool_use.id,
-                        "content": tool_result,
-                    }],
-                })
-            else:
-                # Agent is done, return final response
-                return self._parse_json_response(response.content[0].text)
-```
-
----
-
-## 6. Audit Workflow
-
-### 6.1 Full Agent-Based Audit Flow
-
-```python
-async def run_agent_audit(
-    template_path: Path,
-    confluence_space: str,
-    project_key: str,
-) -> ProjectAuditResult:
-    """Execute full agent-based audit."""
-
-    # 1. Initialize agents
-    orchestrator = OrchestratorAgent()
-    code_agent = CodeAnalysisAgent()
-    doc_agent = DocumentationAnalysisAgent()
-    security_agent = SecurityAnalysisAgent()
-    synthesis_agent = EvidenceSynthesisAgent()
-    recommendation_agent = RecommendationAgent()
-
-    # 2. Load inputs
-    template = load_template(template_path)
-    confluence_docs = await fetch_confluence_docs(confluence_space)
-    repositories = await discover_repositories(project_key)
-
-    # 3. Build context
-    context = AuditContext(
-        template=template,
-        confluence=confluence_docs,
-        repositories=repositories,
-    )
-
-    # 4. Orchestrator decomposes work
-    work_plan = await orchestrator.plan_audit(context)
-
-    # 5. Execute specialized agents in parallel where possible
-    results = await asyncio.gather(
-        code_agent.analyze(context, work_plan.code_tasks),
-        doc_agent.analyze(context, work_plan.doc_tasks),
-        security_agent.analyze(context, work_plan.security_tasks),
-    )
-
-    # 6. Synthesize findings
-    synthesis = await synthesis_agent.synthesize(*results)
-
-    # 7. Generate recommendations
-    recommendations = await recommendation_agent.generate(synthesis)
-
-    # 8. Build final report
-    return orchestrator.build_report(synthesis, recommendations)
-```
-
----
-
-## 7. Comparison: Rule-Based vs Agent-Based
-
-| Aspect | Rule-Based (Current) | Agent-Based (Proposed) |
-|--------|---------------------|----------------------|
-| **Pattern Matching** | Regex, keyword search | Semantic understanding |
-| **Context Awareness** | None | Full context across files |
-| **Reasoning** | If-then rules | Chain-of-thought reasoning |
-| **False Positives** | Common | Reduced via context |
-| **Explanations** | Template text | Natural language rationale |
-| **Adaptability** | Requires code changes | Prompt updates only |
-| **Cost** | Low (CPU only) | Higher (API calls) |
-| **Latency** | Fast (ms) | Slower (seconds) |
-
----
-
-## 8. Implementation Phases
-
-| Phase | Sprint | Focus | Deliverables |
-|-------|--------|-------|--------------|
-| **1** | 1-2 | Single Agent MVP | CodeAnalysisAgent for README quality |
-| **2** | 3-4 | Multi-Agent Core | Add DocAgent, SecurityAgent |
-| **3** | 5-6 | Tool Integration | File access, Confluence, SonarQube tools |
-| **4** | 7-8 | Orchestration | OrchestratorAgent, SynthesisAgent |
-| **5** | 9-10 | Production | Caching, rate limiting, cost optimization |
-
----
-
-## 9. Cost Optimization
-
-### 9.1 Strategies
-
-| Strategy | Description | Savings |
-|----------|-------------|---------|
-| **Caching** | Cache LLM responses for identical inputs | 50-70% |
-| **Tiered Models** | Use Haiku for simple, Sonnet for complex | 40% |
-| **Batching** | Combine multiple checks in single prompt | 30% |
-| **Incremental** | Only analyze changed files | 60-80% |
-| **Hybrid** | Use rule-based for simple, LLM for complex | 50% |
-
-### 9.2 Cost Estimation
-
-```
-Per Repository Audit (estimated):
-- Code Analysis: ~5 API calls × $0.01 = $0.05
-- Documentation: ~3 API calls × $0.01 = $0.03
-- Security: ~4 API calls × $0.01 = $0.04
-- Synthesis: ~2 API calls × $0.01 = $0.02
-- Recommendations: ~1 API call × $0.01 = $0.01
-────────────────────────────────────────────
-Total per repo: ~$0.15
-
-Project with 10 repos: ~$1.50 per full audit
-Monthly (weekly audits): ~$6.00
-```
-
----
-
-## 10. References
-
-- [RepoAudit - Autonomous LLM-Agent for Code Auditing](https://github.com/PurCL/RepoAudit)
-- [Anthropic Alignment Auditing Agents](https://alignment.anthropic.com/2025/automated-auditing/)
-- [AudAgent - Privacy Policy Compliance](https://arxiv.org/html/2511.07441v1)
-- [CrewAI Code Review Agent](https://github.com/Ionio-io/LLM-agent-for-code-reviews)
-- [Autonomous Document Verification](https://artificio.ai)
-- [Microsoft Agent Governance](https://microsoft.com)
-- [Databricks Agent Evaluation](https://databricks.com)
-- [CEUR Multi-Agent Verification](https://ceur-ws.org)