cli-jaw 0.1.0

package/skills_ref/static-analysis/skills/sarif-parsing/SKILL.md

@@ -0,0 +1,479 @@
+---
+name: sarif-parsing
+description: Parse, analyze, and process SARIF (Static Analysis Results Interchange Format) files. Use when reading security scan results, aggregating findings from multiple tools, deduplicating alerts, extracting specific vulnerabilities, or integrating SARIF data into CI/CD pipelines.
+allowed-tools:
+  - Bash
+  - Read
+  - Glob
+  - Grep
+---
+
+# SARIF Parsing Best Practices
+
+You are a SARIF parsing expert. Your role is to help users effectively read, analyze, and process SARIF files from static analysis tools.
+
+## When to Use
+
+Use this skill when:
+- Reading or interpreting static analysis scan results in SARIF format
+- Aggregating findings from multiple security tools
+- Deduplicating or filtering security alerts
+- Extracting specific vulnerabilities from SARIF files
+- Integrating SARIF data into CI/CD pipelines
+- Converting SARIF output to other formats
+
+## When NOT to Use
+
+Do NOT use this skill for:
+- Running static analysis scans (use CodeQL or Semgrep skills instead)
+- Writing CodeQL or Semgrep rules (use their respective skills)
+- Analyzing source code directly (SARIF is for processing existing scan results)
+- Triaging findings without SARIF input (use variant-analysis or audit skills)
+
+## SARIF Structure Overview
+
+SARIF 2.1.0 is the current OASIS standard. Every SARIF file has this hierarchical structure:
+
+```
+sarifLog
+├── version: "2.1.0"
+├── $schema: (optional, enables IDE validation)
+└── runs[] (array of analysis runs)
+    ├── tool
+    │   ├── driver
+    │   │   ├── name (required)
+    │   │   ├── version
+    │   │   └── rules[] (rule definitions)
+    │   └── extensions[] (plugins)
+    ├── results[] (findings)
+    │   ├── ruleId
+    │   ├── level (error/warning/note)
+    │   ├── message.text
+    │   ├── locations[]
+    │   │   └── physicalLocation
+    │   │       ├── artifactLocation.uri
+    │   │       └── region (startLine, startColumn, etc.)
+    │   ├── fingerprints{}
+    │   └── partialFingerprints{}
+    └── artifacts[] (scanned files metadata)
+```
+
+### Why Fingerprinting Matters
+
+Without stable fingerprints, you can't track findings across runs:
+
+- **Baseline comparison**: "Is this a new finding or did we see it before?"
+- **Regression detection**: "Did this PR introduce new vulnerabilities?"
+- **Suppression**: "Ignore this known false positive in future runs"
+
+Tools report different paths (`/path/to/project/` vs `/github/workspace/`), so path-based matching fails. Fingerprints hash the *content* (code snippet, rule ID, relative location) to create stable identifiers regardless of environment.
+
+## Tool Selection Guide
+
+| Use Case | Tool | Installation |
+|----------|------|--------------|
+| Quick CLI queries | jq | `brew install jq` / `apt install jq` |
+| Python scripting (simple) | pysarif | `pip install pysarif` |
+| Python scripting (advanced) | sarif-tools | `pip install sarif-tools` |
+| .NET applications | SARIF SDK | NuGet package |
+| JavaScript/Node.js | sarif-js | npm package |
+| Go applications | garif | `go get github.com/chavacava/garif` |
+| Validation | SARIF Validator | sarifweb.azurewebsites.net |
+
+## Strategy 1: Quick Analysis with jq
+
+For rapid exploration and one-off queries:
+
+```bash
+# Pretty print the file
+jq '.' results.sarif
+
+# Count total findings
+jq '[.runs[].results[]] | length' results.sarif
+
+# List all rule IDs triggered
+jq '[.runs[].results[].ruleId] | unique' results.sarif
+
+# Extract errors only
+jq '.runs[].results[] | select(.level == "error")' results.sarif
+
+# Get findings with file locations
+jq '.runs[].results[] | {
+  rule: .ruleId,
+  message: .message.text,
+  file: .locations[0].physicalLocation.artifactLocation.uri,
+  line: .locations[0].physicalLocation.region.startLine
+}' results.sarif
+
+# Filter by severity and get count per rule
+jq '[.runs[].results[] | select(.level == "error")] | group_by(.ruleId) | map({rule: .[0].ruleId, count: length})' results.sarif
+
+# Extract findings for a specific file
+jq --arg file "src/auth.py" '.runs[].results[] | select(.locations[].physicalLocation.artifactLocation.uri | contains($file))' results.sarif
+```
+
+## Strategy 2: Python with pysarif
+
+For programmatic access with full object model:
+
+```python
+from pysarif import load_from_file, save_to_file
+
+# Load SARIF file
+sarif = load_from_file("results.sarif")
+
+# Iterate through runs and results
+for run in sarif.runs:
+    tool_name = run.tool.driver.name
+    print(f"Tool: {tool_name}")
+
+    for result in run.results:
+        print(f"  [{result.level}] {result.rule_id}: {result.message.text}")
+
+        if result.locations:
+            loc = result.locations[0].physical_location
+            if loc and loc.artifact_location:
+                print(f"    File: {loc.artifact_location.uri}")
+                if loc.region:
+                    print(f"    Line: {loc.region.start_line}")
+
+# Save modified SARIF
+save_to_file(sarif, "modified.sarif")
+```
+
+## Strategy 3: Python with sarif-tools
+
+For aggregation, reporting, and CI/CD integration:
+
+```python
+from sarif import loader
+
+# Load single file
+sarif_data = loader.load_sarif_file("results.sarif")
+
+# Or load multiple files
+sarif_set = loader.load_sarif_files(["tool1.sarif", "tool2.sarif"])
+
+# Get summary report
+report = sarif_data.get_report()
+
+# Get histogram by severity
+errors = report.get_issue_type_histogram_for_severity("error")
+warnings = report.get_issue_type_histogram_for_severity("warning")
+
+# Filter results
+high_severity = [r for r in sarif_data.get_results()
+                 if r.get("level") == "error"]
+```
+
+**sarif-tools CLI commands:**
+
+```bash
+# Summary of findings
+sarif summary results.sarif
+
+# List all results with details
+sarif ls results.sarif
+
+# Get results by severity
+sarif ls --level error results.sarif
+
+# Diff two SARIF files (find new/fixed issues)
+sarif diff baseline.sarif current.sarif
+
+# Convert to other formats
+sarif csv results.sarif > results.csv
+sarif html results.sarif > report.html
+```
+
+## Strategy 4: Aggregating Multiple SARIF Files
+
+When combining results from multiple tools:
+
+```python
+import json
+from pathlib import Path
+
+def aggregate_sarif_files(sarif_paths: list[str]) -> dict:
+    """Combine multiple SARIF files into one."""
+    aggregated = {
+        "version": "2.1.0",
+        "$schema": "https://json.schemastore.org/sarif-2.1.0.json",
+        "runs": []
+    }
+
+    for path in sarif_paths:
+        with open(path) as f:
+            sarif = json.load(f)
+            aggregated["runs"].extend(sarif.get("runs", []))
+
+    return aggregated
+
+def deduplicate_results(sarif: dict) -> dict:
+    """Remove duplicate findings based on fingerprints."""
+    seen_fingerprints = set()
+
+    for run in sarif["runs"]:
+        unique_results = []
+        for result in run.get("results", []):
+            # Use partialFingerprints or create key from location
+            fp = None
+            if result.get("partialFingerprints"):
+                fp = tuple(sorted(result["partialFingerprints"].items()))
+            elif result.get("fingerprints"):
+                fp = tuple(sorted(result["fingerprints"].items()))
+            else:
+                # Fallback: create fingerprint from rule + location
+                loc = result.get("locations", [{}])[0]
+                phys = loc.get("physicalLocation", {})
+                fp = (
+                    result.get("ruleId"),
+                    phys.get("artifactLocation", {}).get("uri"),
+                    phys.get("region", {}).get("startLine")
+                )
+
+            if fp not in seen_fingerprints:
+                seen_fingerprints.add(fp)
+                unique_results.append(result)
+
+        run["results"] = unique_results
+
+    return sarif
+```
+
+## Strategy 5: Extracting Actionable Data
+
+```python
+import json
+from dataclasses import dataclass
+from typing import Optional
+
+@dataclass
+class Finding:
+    rule_id: str
+    level: str
+    message: str
+    file_path: Optional[str]
+    start_line: Optional[int]
+    end_line: Optional[int]
+    fingerprint: Optional[str]
+
+def extract_findings(sarif_path: str) -> list[Finding]:
+    """Extract structured findings from SARIF file."""
+    with open(sarif_path) as f:
+        sarif = json.load(f)
+
+    findings = []
+    for run in sarif.get("runs", []):
+        for result in run.get("results", []):
+            loc = result.get("locations", [{}])[0]
+            phys = loc.get("physicalLocation", {})
+            region = phys.get("region", {})
+
+            findings.append(Finding(
+                rule_id=result.get("ruleId", "unknown"),
+                level=result.get("level", "warning"),
+                message=result.get("message", {}).get("text", ""),
+                file_path=phys.get("artifactLocation", {}).get("uri"),
+                start_line=region.get("startLine"),
+                end_line=region.get("endLine"),
+                fingerprint=next(iter(result.get("partialFingerprints", {}).values()), None)
+            ))
+
+    return findings
+
+# Filter and prioritize
+def prioritize_findings(findings: list[Finding]) -> list[Finding]:
+    """Sort findings by severity."""
+    severity_order = {"error": 0, "warning": 1, "note": 2, "none": 3}
+    return sorted(findings, key=lambda f: severity_order.get(f.level, 99))
+```
+
+## Common Pitfalls and Solutions
+
+### 1. Path Normalization Issues
+
+Different tools report paths differently (absolute, relative, URI-encoded):
+
+```python
+from urllib.parse import unquote
+from pathlib import Path
+
+def normalize_path(uri: str, base_path: str = "") -> str:
+    """Normalize SARIF artifact URI to consistent path."""
+    # Remove file:// prefix if present
+    if uri.startswith("file://"):
+        uri = uri[7:]
+
+    # URL decode
+    uri = unquote(uri)
+
+    # Handle relative paths
+    if not Path(uri).is_absolute() and base_path:
+        uri = str(Path(base_path) / uri)
+
+    # Normalize separators
+    return str(Path(uri))
+```
+
+### 2. Fingerprint Mismatch Across Runs
+
+Fingerprints may not match if:
+- File paths differ between environments
+- Tool versions changed fingerprinting algorithm
+- Code was reformatted (changing line numbers)
+
+**Solution:** Use multiple fingerprint strategies:
+
+```python
+def compute_stable_fingerprint(result: dict, file_content: str = None) -> str:
+    """Compute environment-independent fingerprint."""
+    import hashlib
+
+    components = [
+        result.get("ruleId", ""),
+        result.get("message", {}).get("text", "")[:100],  # First 100 chars
+    ]
+
+    # Add code snippet if available
+    if file_content and result.get("locations"):
+        region = result["locations"][0].get("physicalLocation", {}).get("region", {})
+        if region.get("startLine"):
+            lines = file_content.split("\n")
+            line_idx = region["startLine"] - 1
+            if 0 <= line_idx < len(lines):
+                # Normalize whitespace
+                components.append(lines[line_idx].strip())
+
+    return hashlib.sha256("".join(components).encode()).hexdigest()[:16]
+```
+
+### 3. Missing or Incomplete Data
+
+SARIF allows many optional fields. Always use defensive access:
+
+```python
+def safe_get_location(result: dict) -> tuple[str, int]:
+    """Safely extract file and line from result."""
+    try:
+        loc = result.get("locations", [{}])[0]
+        phys = loc.get("physicalLocation", {})
+        file_path = phys.get("artifactLocation", {}).get("uri", "unknown")
+        line = phys.get("region", {}).get("startLine", 0)
+        return file_path, line
+    except (IndexError, KeyError, TypeError):
+        return "unknown", 0
+```
+
+### 4. Large File Performance
+
+For very large SARIF files (100MB+):
+
+```python
+import ijson  # pip install ijson
+
+def stream_results(sarif_path: str):
+    """Stream results without loading entire file."""
+    with open(sarif_path, "rb") as f:
+        # Stream through results arrays
+        for result in ijson.items(f, "runs.item.results.item"):
+            yield result
+```
+
+### 5. Schema Validation
+
+Validate before processing to catch malformed files:
+
+```bash
+# Using ajv-cli
+npm install -g ajv-cli
+ajv validate -s sarif-schema-2.1.0.json -d results.sarif
+
+# Using Python jsonschema
+pip install jsonschema
+```
+
+```python
+from jsonschema import validate, ValidationError
+import json
+
+def validate_sarif(sarif_path: str, schema_path: str) -> bool:
+    """Validate SARIF file against schema."""
+    with open(sarif_path) as f:
+        sarif = json.load(f)
+    with open(schema_path) as f:
+        schema = json.load(f)
+
+    try:
+        validate(sarif, schema)
+        return True
+    except ValidationError as e:
+        print(f"Validation error: {e.message}")
+        return False
+```
+
+## CI/CD Integration Patterns
+
+### GitHub Actions
+
+```yaml
+- name: Upload SARIF
+  uses: github/codeql-action/upload-sarif@v3
+  with:
+    sarif_file: results.sarif
+
+- name: Check for high severity
+  run: |
+    HIGH_COUNT=$(jq '[.runs[].results[] | select(.level == "error")] | length' results.sarif)
+    if [ "$HIGH_COUNT" -gt 0 ]; then
+      echo "Found $HIGH_COUNT high severity issues"
+      exit 1
+    fi
+```
+
+### Fail on New Issues
+
+```python
+from sarif import loader
+
+def check_for_regressions(baseline: str, current: str) -> int:
+    """Return count of new issues not in baseline."""
+    baseline_data = loader.load_sarif_file(baseline)
+    current_data = loader.load_sarif_file(current)
+
+    baseline_fps = {get_fingerprint(r) for r in baseline_data.get_results()}
+    new_issues = [r for r in current_data.get_results()
+                  if get_fingerprint(r) not in baseline_fps]
+
+    return len(new_issues)
+```
+
+## Key Principles
+
+1. **Validate first**: Check SARIF structure before processing
+2. **Handle optionals**: Many fields are optional; use defensive access
+3. **Normalize paths**: Tools report paths differently; normalize early
+4. **Fingerprint wisely**: Combine multiple strategies for stable deduplication
+5. **Stream large files**: Use ijson or similar for 100MB+ files
+6. **Aggregate thoughtfully**: Preserve tool metadata when combining files
+
+## Skill Resources
+
+For ready-to-use query templates, see [{baseDir}/resources/jq-queries.md]({baseDir}/resources/jq-queries.md):
+- 40+ jq queries for common SARIF operations
+- Severity filtering, rule extraction, aggregation patterns
+
+For Python utilities, see [{baseDir}/resources/sarif_helpers.py]({baseDir}/resources/sarif_helpers.py):
+- `normalize_path()` - Handle tool-specific path formats
+- `compute_fingerprint()` - Stable fingerprinting ignoring paths
+- `deduplicate_results()` - Remove duplicates across runs
+
+## Reference Links
+
+- [OASIS SARIF 2.1.0 Specification](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html)
+- [Microsoft SARIF Tutorials](https://github.com/microsoft/sarif-tutorials)
+- [SARIF SDK (.NET)](https://github.com/microsoft/sarif-sdk)
+- [sarif-tools (Python)](https://github.com/microsoft/sarif-tools)
+- [pysarif (Python)](https://github.com/Kjeld-P/pysarif)
+- [GitHub SARIF Support](https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning)
+- [SARIF Validator](https://sarifweb.azurewebsites.net/)

package/skills_ref/static-analysis/skills/sarif-parsing/resources/jq-queries.md

@@ -0,0 +1,162 @@
+# SARIF jq Query Reference
+
+Ready-to-use jq queries for common SARIF parsing tasks.
+
+## Basic Exploration
+
+```bash
+# Pretty print
+jq '.' results.sarif
+
+# Get SARIF version
+jq '.version' results.sarif
+
+# List tool names from all runs
+jq '.runs[].tool.driver.name' results.sarif
+
+# Count runs
+jq '.runs | length' results.sarif
+```
+
+## Result Queries
+
+```bash
+# Total result count
+jq '[.runs[].results[]] | length' results.sarif
+
+# Count by severity level
+jq 'reduce .runs[].results[] as $r ({}; .[$r.level] += 1)' results.sarif
+
+# List unique rule IDs
+jq '[.runs[].results[].ruleId] | unique | sort' results.sarif
+
+# Count per rule
+jq '[.runs[].results[]] | group_by(.ruleId) | map({rule: .[0].ruleId, count: length}) | sort_by(-.count)' results.sarif
+```
+
+## Filtering Results
+
+```bash
+# Only errors
+jq '.runs[].results[] | select(.level == "error")' results.sarif
+
+# Only warnings
+jq '.runs[].results[] | select(.level == "warning")' results.sarif
+
+# By specific rule ID
+jq --arg rule "SQL_INJECTION" '.runs[].results[] | select(.ruleId == $rule)' results.sarif
+
+# By file path (contains)
+jq --arg file "auth" '.runs[].results[] | select(.locations[].physicalLocation.artifactLocation.uri | contains($file))' results.sarif
+
+# By file extension
+jq '.runs[].results[] | select(.locations[].physicalLocation.artifactLocation.uri | test("\\.py$"))' results.sarif
+
+# Multiple conditions
+jq '.runs[].results[] | select(.level == "error" and (.ruleId | startswith("SEC")))' results.sarif
+```
+
+## Extracting Locations
+
+```bash
+# File and line for each result
+jq '.runs[].results[] | {
+  rule: .ruleId,
+  file: .locations[0].physicalLocation.artifactLocation.uri,
+  line: .locations[0].physicalLocation.region.startLine
+}' results.sarif
+
+# Unique affected files
+jq '[.runs[].results[].locations[].physicalLocation.artifactLocation.uri] | unique | sort' results.sarif
+
+# Results grouped by file
+jq '[.runs[].results[] | {file: .locations[0].physicalLocation.artifactLocation.uri, result: .}] | group_by(.file) | map({file: .[0].file, count: length})' results.sarif
+```
+
+## Rule Information
+
+```bash
+# List all rules with severity
+jq '.runs[].tool.driver.rules[] | {id: .id, name: .name, level: .defaultConfiguration.level}' results.sarif
+
+# Get rule description by ID
+jq --arg id "RULE001" '.runs[].tool.driver.rules[] | select(.id == $id)' results.sarif
+
+# Rules with help URLs
+jq '.runs[].tool.driver.rules[] | select(.helpUri) | {id: .id, help: .helpUri}' results.sarif
+```
+
+## Fingerprints
+
+```bash
+# Results with fingerprints
+jq '.runs[].results[] | select(.fingerprints or .partialFingerprints) | {rule: .ruleId, fp: (.fingerprints // .partialFingerprints)}' results.sarif
+
+# Extract all partial fingerprints
+jq '[.runs[].results[].partialFingerprints] | add' results.sarif
+```
+
+## Aggregation and Reporting
+
+```bash
+# Summary by severity and rule
+jq '[.runs[].results[]] | group_by(.level) | map({level: .[0].level, rules: (group_by(.ruleId) | map({rule: .[0].ruleId, count: length}))})' results.sarif
+
+# Top 10 most frequent rules
+jq '[.runs[].results[]] | group_by(.ruleId) | map({rule: .[0].ruleId, count: length}) | sort_by(-.count) | .[0:10]' results.sarif
+
+# Files with most issues
+jq '[.runs[].results[] | .locations[0].physicalLocation.artifactLocation.uri] | group_by(.) | map({file: .[0], count: length}) | sort_by(-.count) | .[0:10]' results.sarif
+```
+
+## Output Formatting
+
+```bash
+# CSV-like output
+jq -r '.runs[].results[] | [.ruleId, .level, .locations[0].physicalLocation.artifactLocation.uri, .locations[0].physicalLocation.region.startLine, .message.text] | @csv' results.sarif
+
+# Tab-separated
+jq -r '.runs[].results[] | [.ruleId, .level, .locations[0].physicalLocation.artifactLocation.uri // "N/A"] | @tsv' results.sarif
+
+# Markdown table
+echo "| Rule | Level | File | Line |"
+echo "|------|-------|------|------|"
+jq -r '.runs[].results[] | "| \(.ruleId) | \(.level) | \(.locations[0].physicalLocation.artifactLocation.uri // "N/A") | \(.locations[0].physicalLocation.region.startLine // "N/A") |"' results.sarif
+```
+
+## Comparison and Diff
+
+```bash
+# Find rules in file1 not in file2
+comm -23 <(jq -r '[.runs[].results[].ruleId] | unique | sort[]' file1.sarif) <(jq -r '[.runs[].results[].ruleId] | unique | sort[]' file2.sarif)
+
+# Compare result counts
+echo "File 1: $(jq '[.runs[].results[]] | length' file1.sarif)"
+echo "File 2: $(jq '[.runs[].results[]] | length' file2.sarif)"
+```
+
+## Transformation
+
+```bash
+# Extract minimal SARIF (results only)
+jq '{version: .version, runs: [.runs[] | {tool: {driver: {name: .tool.driver.name}}, results: .results}]}' results.sarif
+
+# Filter and create new SARIF with only errors
+jq '.runs[].results = [.runs[].results[] | select(.level == "error")]' results.sarif > errors-only.sarif
+
+# Merge multiple SARIF files
+jq -s '{version: "2.1.0", runs: [.[].runs[]]}' file1.sarif file2.sarif > merged.sarif
+```
+
+## Validation Checks
+
+```bash
+# Check if version is 2.1.0
+jq -e '.version == "2.1.0"' results.sarif && echo "Valid version" || echo "Invalid version"
+
+# Check for empty results
+jq -e '[.runs[].results[]] | length > 0' results.sarif && echo "Has results" || echo "No results"
+
+# Verify all results have locations
+jq '[.runs[].results[] | select(.locations | length == 0)] | length' results.sarif
+```