@orionscanner/cli 0.1.0

package/src/orion/engine/docs/AUDIT_PROMPT.md

@@ -0,0 +1,1248 @@
+# Orion Engine — Autonomous Ultra-Modern Audit Intelligence Build Specification
+
+## Mission
+
+Transform **Orion Engine** into a production-grade, deeply contextual, future-proof code auditing and security analysis platform.
+
+Orion must not behave like a collection of regex scanners.
+
+It must behave like a **software reasoning system**:
+
+* understand the structure of a repository
+* understand relationships between components
+* trace data and control flow
+* reason about security boundaries
+* correlate evidence across analyzers
+* distinguish exploitable behavior from merely suspicious syntax
+* minimize false positives
+* explain exactly why a finding exists
+* identify uncertainty rather than inventing conclusions
+* remain deterministic and reproducible
+* continuously improve through regression fixtures and evidence-driven heuristics
+
+The target is **LLM-level contextual usefulness implemented primarily through deterministic program analysis, graph reasoning, symbolic reasoning, heuristics, and evidence correlation**.
+
+Do not claim that Orion is equivalent to an LLM. Instead, build an architecture capable of achieving comparable contextual depth for code-audit tasks without requiring an LLM for its core correctness.
+
+---
+
+# 1. Non-Negotiable Principles
+
+## 1.1 No fake implementations
+
+Every registered engine must contain real analysis logic.
+
+Never use:
+
+```python
+return EngineResult(findings=[])
+```
+
+as a placeholder.
+
+Never mark an engine as successful when it did not actually analyze its input.
+
+A failed engine must produce an explicit failure state.
+
+These states must remain distinct:
+
+```text
+SUCCESS_WITH_FINDINGS
+SUCCESS_WITH_NO_FINDINGS
+PARTIAL
+UNAVAILABLE
+FAILED
+TIMEOUT
+RESOURCE_LIMIT
+```
+
+---
+
+# 2. Build a Shared Intelligence Substrate
+
+Do not implement the 38 engines as isolated scanners.
+
+Create a shared Orion analysis substrate.
+
+Conceptually:
+
+```text
+Repository
+    ↓
+File Discovery
+    ↓
+Language Detection
+    ↓
+Parsing
+    ↓
+Canonical Orion IR
+    ↓
+Symbol Graph
+    ↓
+Call Graph
+    ↓
+Control Flow Graph
+    ↓
+Data Flow Graph
+    ↓
+Dependency Graph
+    ↓
+Configuration Graph
+    ↓
+Authorization / Trust-Boundary Graph
+    ↓
+Evidence Graph
+    ↓
+38 Engines
+    ↓
+Cross-Engine Correlation
+    ↓
+Finding Intelligence
+    ↓
+Deduplication / Confidence
+    ↓
+Audit Findings
+```
+
+The shared representation is more important than individual rule count.
+
+---
+
+# 3. Orion IR
+
+Design a versioned canonical IR capable of representing:
+
+* files
+* modules
+* packages
+* functions
+* methods
+* classes
+* variables
+* constants
+* imports
+* exports
+* decorators
+* annotations
+* types
+* calls
+* arguments
+* returns
+* branches
+* loops
+* exceptions
+* assignments
+* expressions
+* control-flow edges
+* data-flow edges
+* taint propagation
+* sinks
+* sources
+* sanitizers
+* authentication boundaries
+* authorization boundaries
+* network boundaries
+* process boundaries
+* filesystem boundaries
+* serialization boundaries
+* database operations
+* external services
+* configuration
+* environment variables
+* dependencies
+* generated code
+* test code
+* build artifacts
+
+Every node and edge must preserve source provenance:
+
+```text
+repository
+file
+line
+column
+AST node
+symbol
+engine
+analysis pass
+```
+
+IR must be versioned so future analyzers can evolve without breaking old findings.
+
+---
+
+# 4. Evidence Graph
+
+Create a first-class Evidence Graph.
+
+Every engine should be able to contribute evidence.
+
+Example:
+
+```text
+HTTP_PARAMETER
+      │
+      ↓
+request.args["id"]
+      │
+      ↓
+Function: get_user()
+      │
+      ↓
+Function: query_user()
+      │
+      ↓
+string concatenation
+      │
+      ↓
+SQL_QUERY
+      │
+      ↓
+database.execute()
+```
+
+Evidence must be structured.
+
+Each evidence item should contain:
+
+```text
+type
+source
+location
+confidence
+provenance
+relationship
+supporting_engine
+```
+
+Never rely solely on textual messages.
+
+---
+
+# 5. Evidence-Based Findings
+
+A finding must contain:
+
+```text
+finding_id
+title
+category
+severity
+confidence
+certainty
+description
+impact
+source_location
+sink_location
+attack_path
+evidence
+related_findings
+related_engines
+remediation
+references
+limitations
+```
+
+The distinction between **severity** and **confidence** is mandatory.
+
+Example:
+
+```text
+severity: CRITICAL
+confidence: 0.98
+```
+
+means:
+
+> If true, this is extremely dangerous, and Orion has very strong evidence that it is real.
+
+Whereas:
+
+```text
+severity: CRITICAL
+confidence: 0.42
+```
+
+means:
+
+> Potentially critical, but insufficient evidence.
+
+Do not report these identically.
+
+---
+
+# 6. Confidence Engine
+
+Build a dedicated confidence/evidence scoring system.
+
+Confidence must increase when independent evidence agrees.
+
+Example:
+
+```text
+SAST pattern                  +0.20
+attacker-controlled source   +0.25
+valid data-flow path          +0.20
+reachable sink               +0.15
+missing sanitizer            +0.10
+missing authorization        +0.05
+cross-engine confirmation    +0.10
+```
+
+Confidence must decrease for:
+
+```text
+sanitization
+validation
+constant input
+unreachable code
+test-only code
+dead code
+known-safe wrapper
+trusted source
+framework guarantee
+false-positive suppression
+```
+
+Do not use arbitrary scores blindly.
+
+Make scoring explainable and configurable.
+
+---
+
+# 7. Cross-Engine Reasoning
+
+The engines must cooperate.
+
+For example:
+
+```text
+SAST
+ +
+Taint
+ +
+Callgraph
+ +
+Dataflow
+ +
+Authorization
+```
+
+should be capable of producing one high-confidence security finding.
+
+Do not produce five duplicate findings for one root cause.
+
+Create a correlation layer:
+
+```text
+Finding A
+Finding B
+Finding C
+Finding D
+        ↓
+Root Cause Analysis
+        ↓
+Unified Finding
+```
+
+Maintain all supporting evidence underneath the unified finding.
+
+---
+
+# 8. The 38 Engines
+
+Implement all 38 as real analyzers:
+
+1. SAST
+2. Secrets
+3. Dependencies
+4. Taint
+5. Callgraph
+6. CFG
+7. Symbol
+8. Dataflow
+9. Authorization
+10. Orphan
+11. Prompt Security
+12. Contract Probe
+13. Wiring Trace
+14. State
+15. License
+16. Complexity
+17. Duplication
+18. Dead Import
+19. Error Handling
+20. Logging
+21. Config Engine
+22. Chaos
+23. Chaos Config
+24. Regression
+25. Performance Guard
+26. Data Residency
+27. Noise
+28. Supply Chain / SBOM
+29. Hunter
+30. Lint
+31. Architecture
+32. Test Coverage
+33. Container
+34. Compliance
+35. Provenance
+36. False-Positive Learning
+37. Sandbox Escape
+38. Config Validation
+
+Every engine must implement the same Orion contract.
+
+---
+
+# 9. SAST Must Be Semantic, Not Regex-Only
+
+SAST may use patterns, but patterns are only candidates.
+
+Implement layered analysis:
+
+```text
+Pattern Candidate
+      ↓
+AST Context
+      ↓
+Type Context
+      ↓
+Symbol Resolution
+      ↓
+Data Flow
+      ↓
+Control Flow
+      ↓
+Reachability
+      ↓
+Sanitizer Analysis
+      ↓
+Framework Knowledge
+      ↓
+Confidence
+```
+
+Example:
+
+Do not report every string concatenation as SQL injection.
+
+Determine:
+
+```text
+Is the value attacker controlled?
+Is it actually used as SQL?
+Is the query dynamically constructed?
+Is a safe parameterization API used?
+Is the code reachable?
+Is there a sanitizer?
+Is this test/mock code?
+```
+
+Only then create a strong finding.
+
+---
+
+# 10. Taint Analysis
+
+Support:
+
+* intra-procedural propagation
+* inter-procedural propagation
+* cross-module propagation
+* object/property propagation
+* container propagation
+* return-value propagation
+* argument propagation
+* framework sources
+* framework sinks
+* sanitizers
+* validators
+* trust boundaries
+
+Support configurable taint policies.
+
+Never silently swallow analysis exceptions.
+
+---
+
+# 11. Callgraph
+
+Build multiple resolution strategies:
+
+```text
+exact resolution
+type-based resolution
+inheritance resolution
+dynamic dispatch approximation
+import resolution
+framework routing
+reflection heuristics
+```
+
+Represent uncertainty explicitly.
+
+Never pretend an approximate call edge is certain.
+
+---
+
+# 12. CFG + Dataflow
+
+Construct real control/data-flow graphs.
+
+Track:
+
+```text
+definitions
+uses
+assignments
+branches
+loops
+returns
+exceptions
+conditions
+aliases
+mutations
+```
+
+Support path-sensitive analysis where practical.
+
+Use bounded exploration to prevent path explosion.
+
+---
+
+# 13. Authorization Intelligence
+
+Model:
+
+```text
+identity
+roles
+permissions
+resources
+actions
+routes
+middleware
+guards
+policies
+ownership checks
+tenant boundaries
+```
+
+Look for:
+
+* missing authorization
+* inconsistent authorization
+* privilege escalation
+* IDOR/BOLA
+* route protection gaps
+* tenant isolation failures
+
+Do not flag an endpoint merely because authorization isn't visible locally.
+
+Search the actual execution path and framework middleware.
+
+---
+
+# 14. Secrets Engine
+
+Do not rely solely on regex.
+
+Combine:
+
+```text
+pattern detection
+entropy
+context
+known prefixes
+assignment semantics
+variable names
+provider formats
+test-fixture detection
+documentation detection
+allowlists
+```
+
+Distinguish:
+
+```text
+real credential
+example credential
+placeholder
+test fixture
+documentation
+revoked/invalid-looking credential
+```
+
+Avoid exposing detected secret values in reports.
+
+Immediately redact sensitive material.
+
+---
+
+# 15. Dependency Intelligence
+
+Support:
+
+* lockfiles
+* manifests
+* transitive dependencies
+* version constraints
+* vulnerability matching
+* dependency reachability
+* runtime usage
+* abandoned packages
+* suspicious packages
+* license issues
+* supply-chain anomalies
+
+Do not report a vulnerable dependency with identical severity whether it is:
+
+```text
+installed but unreachable
+```
+
+or:
+
+```text
+directly imported by internet-facing code
+```
+
+Reachability should influence confidence and prioritization.
+
+---
+
+# 16. Prompt Security
+
+Analyze modern AI applications.
+
+Detect:
+
+* prompt injection
+* indirect prompt injection
+* unsafe tool exposure
+* privilege confusion
+* untrusted content entering prompts
+* secret exposure
+* unsafe tool arguments
+* missing authorization around tools
+* cross-agent trust violations
+* memory poisoning
+* retrieval poisoning
+* instruction/data confusion
+
+Understand the relationship between:
+
+```text
+user
+retrieval
+system prompt
+developer instruction
+tool
+agent
+memory
+external content
+```
+
+Do not flag ordinary prompt text merely because it contains security terminology.
+
+---
+
+# 17. Contract Probe
+
+Understand declared contracts:
+
+```text
+OpenAPI
+JSON Schema
+type hints
+dataclasses
+protobuf
+GraphQL
+function signatures
+configuration schemas
+```
+
+Compare:
+
+```text
+declared behavior
+vs
+actual implementation
+```
+
+Detect mismatches with evidence.
+
+---
+
+# 18. Wiring Trace
+
+Allow Orion to answer:
+
+> Where does this input go?
+
+and:
+
+> What components can reach this operation?
+
+Trace:
+
+```text
+entrypoint
+→ router
+→ middleware
+→ controller
+→ service
+→ repository
+→ external system
+```
+
+Produce explainable paths.
+
+---
+
+# 19. State Engine
+
+Construct state-transition models.
+
+Detect:
+
+* invalid transitions
+* missing transitions
+* inconsistent cleanup
+* race-sensitive states
+* impossible states
+* state-dependent authorization bugs
+
+---
+
+# 20. Quality Engines
+
+Complexity, duplication, dead imports, error handling, and logging must be context-aware.
+
+Do not turn style preferences into security findings.
+
+Separate:
+
+```text
+ERROR
+WARNING
+INFO
+```
+
+from actual vulnerabilities.
+
+---
+
+# 21. Configuration Intelligence
+
+Understand:
+
+* environment variables
+* configuration files
+* defaults
+* overrides
+* deployment configuration
+* secrets
+* security-sensitive flags
+* production/development differences
+
+Detect unsafe combinations rather than isolated settings.
+
+---
+
+# 22. Chaos / Resilience
+
+Do not randomly break systems without control.
+
+Use safe simulation models first.
+
+Analyze:
+
+```text
+timeouts
+dependency failures
+network failures
+resource exhaustion
+partial failures
+retry storms
+circuit breakers
+idempotency
+```
+
+Keep destructive runtime behavior explicitly isolated and opt-in.
+
+---
+
+# 23. Regression Engine
+
+Maintain a baseline.
+
+Compare:
+
+```text
+BASELINE
+    ↓
+CURRENT
+    ↓
+NEW
+FIXED
+UNCHANGED
+REGRESSED
+```
+
+Finding identity must be stable across runs.
+
+---
+
+# 24. Noise / False Positive Intelligence
+
+Create a dedicated layer for reducing noise.
+
+Consider:
+
+```text
+duplicate findings
+same root cause
+test code
+generated code
+dead code
+known-safe wrappers
+framework guarantees
+suppressions
+historical false positives
+confidence
+cross-engine disagreement
+```
+
+Never suppress findings merely because suppression makes the report smaller.
+
+Every suppression must be explainable.
+
+---
+
+# 25. FP Learning
+
+Do not blindly train on developer dismissals.
+
+Represent:
+
+```text
+finding
+reason dismissed
+location
+rule
+context
+historical behavior
+```
+
+Learn contextual suppression patterns while retaining safeguards against poisoning.
+
+Never allow a single dismissal to globally disable a vulnerability class.
+
+---
+
+# 26. Runtime Hunter
+
+Keep runtime analysis isolated and permission-controlled.
+
+Support:
+
+```text
+collector
+probe
+session
+observation
+runtime evidence
+```
+
+Runtime evidence must be clearly distinguished from static evidence.
+
+---
+
+# 27. Security Boundaries
+
+Every engine must respect:
+
+* filesystem boundaries
+* network policy
+* subprocess policy
+* resource limits
+* timeout limits
+* memory limits
+* recursion limits
+* file limits
+* archive extraction limits
+* symlink handling
+* untrusted repository content
+
+Never execute repository code merely to analyze it unless explicitly enabled.
+
+---
+
+# 28. Determinism
+
+The same repository and configuration should produce stable output.
+
+Guarantee:
+
+```text
+stable finding IDs
+stable ordering
+stable serialization
+stable severity
+stable confidence
+```
+
+Avoid dependence on:
+
+* hash iteration order
+* filesystem ordering
+* current time
+* random values
+* machine-specific state
+
+unless explicitly recorded.
+
+---
+
+# 29. Error Handling
+
+This rule is mandatory:
+
+> NEVER silently convert an analysis failure into a clean result.
+
+Forbidden:
+
+```python
+try:
+    analyze()
+except Exception:
+    pass
+```
+
+Instead:
+
+```text
+engine failure
+    ↓
+structured EngineError
+    ↓
+affected analysis scope
+    ↓
+partial/failed status
+    ↓
+visible diagnostic
+```
+
+A broken analyzer must never create a false sense of security.
+
+---
+
+# 30. Evidence Provenance
+
+Every finding must answer:
+
+```text
+Which engine produced this?
+Which version?
+Which source locations?
+Which analysis passes?
+Which assumptions?
+Which evidence?
+Which configuration?
+Which rules?
+```
+
+Use content-addressed IDs where appropriate.
+
+---
+
+# 31. Test Strategy
+
+Create extensive fixtures.
+
+Every engine requires:
+
+```text
+true positives
+true negatives
+edge cases
+framework cases
+cross-file cases
+adversarial cases
+regression cases
+large-project cases
+```
+
+Test false positives as aggressively as true positives.
+
+A detector that catches 100 vulnerabilities but creates 10,000 false positives is not considered successful.
+
+---
+
+# 32. Adversarial Testing
+
+Continuously test Orion against:
+
+* obfuscated code
+* unusual formatting
+* aliases
+* wrappers
+* decorators
+* inheritance
+* dynamic imports
+* generated code
+* nested calls
+* indirect calls
+* framework abstractions
+* dead branches
+* unreachable code
+* misleading variable names
+* intentionally confusing source structures
+
+The objective is not merely benchmark performance.
+
+The objective is **robust reasoning under hostile code structure**.
+
+---
+
+# 33. Performance
+
+Implement:
+
+```text
+incremental analysis
+caching
+parallel analysis
+bounded graph exploration
+memoization
+content-addressed artifacts
+changed-file analysis
+dependency-aware invalidation
+```
+
+Never allow one pathological file to destroy an entire scan.
+
+---
+
+# 34. Finding Deduplication
+
+Use structural fingerprints.
+
+Two findings should be considered related when they share:
+
+```text
+root cause
+source
+sink
+attack path
+symbol
+data-flow path
+```
+
+Preserve all supporting evidence but present a clean unified result.
+
+---
+
+# 35. Explainability
+
+Every high-severity finding must be explainable.
+
+Example:
+
+```text
+CRITICAL — SQL Injection
+
+Confidence: 97%
+
+Attack path:
+
+request.args["id"]
+    ↓
+UserController.get()
+    ↓
+UserService.find()
+    ↓
+query = "... " + id
+    ↓
+db.execute(query)
+
+Why Orion believes this is exploitable:
+✓ attacker-controlled source
+✓ interprocedural propagation confirmed
+✓ SQL sink confirmed
+✓ no recognized sanitizer
+✓ route reachable without required authorization
+
+Supporting engines:
+SAST
+Taint
+Callgraph
+Dataflow
+Authorization
+```
+
+The explanation must be generated from structured evidence, not invented prose.
+
+---
+
+# 36. No Hallucinated Findings
+
+Orion must never claim evidence it did not actually observe.
+
+Forbidden:
+
+```text
+"Authentication appears to be missing"
+```
+
+when the engine never inspected the relevant middleware.
+
+Prefer:
+
+```text
+"Orion could not establish an authentication check on this execution path."
+```
+
+Distinguish:
+
+```text
+NOT FOUND
+NOT OBSERVED
+UNKNOWN
+CONFIRMED
+```
+
+These are not equivalent.
+
+---
+
+# 37. Future-Proof Architecture
+
+Use versioned interfaces:
+
+```text
+IR v1
+Finding Schema v1
+Evidence Schema v1
+Engine Contract v1
+Configuration Schema v1
+```
+
+Keep engine internals replaceable.
+
+Do not hard-code the architecture around one programming language or framework.
+
+Where possible, build language adapters around a common semantic model.
+
+---
+
+# 38. Completion Criteria
+
+Do NOT declare Orion complete because:
+
+```text
+38 directories exist
+```
+
+or:
+
+```text
+38 modules import
+```
+
+Completion requires:
+
+```text
+38 real implementations
++
+shared IR
++
+cross-engine correlation
++
+confidence system
++
+finding deduplication
++
+evidence provenance
++
+deterministic output
++
+resource isolation
++
+failure transparency
++
+positive fixtures
++
+negative fixtures
++
+regression suite
++
+adversarial tests
++
+integration tests
++
+performance tests
+```
+
+Every engine must demonstrate actual analytical behavior.
+
+---
+
+# Final Engineering Objective
+
+Build Orion so that its reasoning pipeline resembles:
+
+```text
+OBSERVE
+   ↓
+UNDERSTAND
+   ↓
+MODEL
+   ↓
+TRACE
+   ↓
+CORRELATE
+   ↓
+CHALLENGE
+   ↓
+VALIDATE
+   ↓
+SCORE
+   ↓
+DEDUPLICATE
+   ↓
+EXPLAIN
+   ↓
+REPORT
+```
+
+Do not optimize for the number of findings.
+
+Optimize for:
+
+**high-confidence findings + strong evidence + low false positives + useful explanations + reproducibility.**
+
+When uncertain, Orion must prefer:
+
+```text
+"I cannot establish this"
+```
+
+over:
+
+```text
+"This is definitely vulnerable"
+```
+
+Do not delete existing analysis infrastructure until its functionality has been understood, migrated, and covered by regression tests.
+
+Do not create stubs merely to satisfy an engine count.
+
+Do not use the name "THANOS" anywhere in the product. The product is **Orion Engine**.
+
+The final system should feel like a serious next-generation software auditing platform rather than a bundle of linters.
+
+Build incrementally, test continuously, preserve working functionality, and only remove legacy components after equivalent or superior Orion functionality has been demonstrated by tests.