@vigolium/piolium 0.0.1

package/skills/sarif-parsing/SKILL.md

@@ -0,0 +1,483 @@
+---
+name: sarif-parsing
+description: >-
+  Parses and processes SARIF files from static analysis tools like CodeQL, Semgrep, or other
+  scanners. Triggers on "parse sarif", "read scan results", "aggregate findings", "deduplicate
+  alerts", or "process sarif output". Handles filtering, deduplication, format conversion, and
+  CI/CD integration of SARIF data. Does NOT run scans — use the Semgrep or CodeQL skills for that.
+allowed-tools:
+  - Bash
+  - Read
+  - Glob
+  - Grep
+---
+
+# SARIF Parsing Best Practices
+
+You are a SARIF parsing expert. Your role is to help users effectively read, analyze, and process SARIF files from static analysis tools.
+
+## When to Use
+
+Use this skill when:
+- Reading or interpreting static analysis scan results in SARIF format
+- Aggregating findings from multiple security tools
+- Deduplicating or filtering security alerts
+- Extracting specific vulnerabilities from SARIF files
+- Integrating SARIF data into CI/CD pipelines
+- Converting SARIF output to other formats
+
+## When NOT to Use
+
+Do NOT use this skill for:
+- Running static analysis scans (use CodeQL or Semgrep skills instead)
+- Writing CodeQL or Semgrep rules (use their respective skills)
+- Analyzing source code directly (SARIF is for processing existing scan results)
+- Triaging findings without SARIF input (use variant-analysis or audit skills)
+
+## SARIF Structure Overview
+
+SARIF 2.1.0 is the current OASIS standard. Every SARIF file has this hierarchical structure:
+
+```
+sarifLog
+├── version: "2.1.0"
+├── $schema: (optional, enables IDE validation)
+└── runs[] (array of analysis runs)
+    ├── tool
+    │   ├── driver
+    │   │   ├── name (required)
+    │   │   ├── version
+    │   │   └── rules[] (rule definitions)
+    │   └── extensions[] (plugins)
+    ├── results[] (findings)
+    │   ├── ruleId
+    │   ├── level (error/warning/note)
+    │   ├── message.text
+    │   ├── locations[]
+    │   │   └── physicalLocation
+    │   │       ├── artifactLocation.uri
+    │   │       └── region (startLine, startColumn, etc.)
+    │   ├── fingerprints{}
+    │   └── partialFingerprints{}
+    └── artifacts[] (scanned files metadata)
+```
+
+### Why Fingerprinting Matters
+
+Without stable fingerprints, you can't track findings across runs:
+
+- **Baseline comparison**: "Is this a new finding or did we see it before?"
+- **Regression detection**: "Did this PR introduce new vulnerabilities?"
+- **Suppression**: "Ignore this known false positive in future runs"
+
+Tools report different paths (`/path/to/project/` vs `/github/workspace/`), so path-based matching fails. Fingerprints hash the *content* (code snippet, rule ID, relative location) to create stable identifiers regardless of environment.
+
+## Tool Selection Guide
+
+| Use Case | Tool | Installation |
+|----------|------|--------------|
+| Quick CLI queries | jq | `brew install jq` / `apt install jq` |
+| Python scripting (simple) | pysarif | `pip install pysarif` |
+| Python scripting (advanced) | sarif-tools | `pip install sarif-tools` |
+| .NET applications | SARIF SDK | NuGet package |
+| JavaScript/Node.js | sarif-js | npm package |
+| Go applications | garif | `go get github.com/chavacava/garif` |
+| Validation | SARIF Validator | sarifweb.azurewebsites.net |
+
+## Strategy 1: Quick Analysis with jq
+
+For rapid exploration and one-off queries:
+
+```bash
+# Pretty print the file
+jq '.' results.sarif
+
+# Count total findings
+jq '[.runs[].results[]] | length' results.sarif
+
+# List all rule IDs triggered
+jq '[.runs[].results[].ruleId] | unique' results.sarif
+
+# Extract errors only
+jq '.runs[].results[] | select(.level == "error")' results.sarif
+
+# Get findings with file locations
+jq '.runs[].results[] | {
+  rule: .ruleId,
+  message: .message.text,
+  file: .locations[0].physicalLocation.artifactLocation.uri,
+  line: .locations[0].physicalLocation.region.startLine
+}' results.sarif
+
+# Filter by severity and get count per rule
+jq '[.runs[].results[] | select(.level == "error")] | group_by(.ruleId) | map({rule: .[0].ruleId, count: length})' results.sarif
+
+# Extract findings for a specific file
+jq --arg file "src/auth.py" '.runs[].results[] | select(.locations[].physicalLocation.artifactLocation.uri | contains($file))' results.sarif
+```
+
+## Strategy 2: Python with pysarif
+
+For programmatic access with full object model:
+
+```python
+from pysarif import load_from_file, save_to_file
+
+# Load SARIF file
+sarif = load_from_file("results.sarif")
+
+# Iterate through runs and results
+for run in sarif.runs:
+    tool_name = run.tool.driver.name
+    print(f"Tool: {tool_name}")
+
+    for result in run.results:
+        print(f"  [{result.level}] {result.rule_id}: {result.message.text}")
+
+        if result.locations:
+            loc = result.locations[0].physical_location
+            if loc and loc.artifact_location:
+                print(f"    File: {loc.artifact_location.uri}")
+                if loc.region:
+                    print(f"    Line: {loc.region.start_line}")
+
+# Save modified SARIF
+save_to_file(sarif, "modified.sarif")
+```
+
+## Strategy 3: Python with sarif-tools
+
+For aggregation, reporting, and CI/CD integration:
+
+```python
+from sarif import loader
+
+# Load single file
+sarif_data = loader.load_sarif_file("results.sarif")
+
+# Or load multiple files
+sarif_set = loader.load_sarif_files(["tool1.sarif", "tool2.sarif"])
+
+# Get summary report
+report = sarif_data.get_report()
+
+# Get histogram by severity
+errors = report.get_issue_type_histogram_for_severity("error")
+warnings = report.get_issue_type_histogram_for_severity("warning")
+
+# Filter results
+high_severity = [r for r in sarif_data.get_results()
+                 if r.get("level") == "error"]
+```
+
+**sarif-tools CLI commands:**
+
+```bash
+# Summary of findings
+sarif summary results.sarif
+
+# List all results with details
+sarif ls results.sarif
+
+# Get results by severity
+sarif ls --level error results.sarif
+
+# Diff two SARIF files (find new/fixed issues)
+sarif diff baseline.sarif current.sarif
+
+# Convert to other formats
+sarif csv results.sarif > results.csv
+sarif html results.sarif > report.html
+```
+
+## Strategy 4: Aggregating Multiple SARIF Files
+
+When combining results from multiple tools:
+
+```python
+import json
+from pathlib import Path
+
+def aggregate_sarif_files(sarif_paths: list[str]) -> dict:
+    """Combine multiple SARIF files into one."""
+    aggregated = {
+        "version": "2.1.0",
+        "$schema": "https://json.schemastore.org/sarif-2.1.0.json",
+        "runs": []
+    }
+
+    for path in sarif_paths:
+        with open(path) as f:
+            sarif = json.load(f)
+            aggregated["runs"].extend(sarif.get("runs", []))
+
+    return aggregated
+
+def deduplicate_results(sarif: dict) -> dict:
+    """Remove duplicate findings based on fingerprints."""
+    seen_fingerprints = set()
+
+    for run in sarif["runs"]:
+        unique_results = []
+        for result in run.get("results", []):
+            # Use partialFingerprints or create key from location
+            fp = None
+            if result.get("partialFingerprints"):
+                fp = tuple(sorted(result["partialFingerprints"].items()))
+            elif result.get("fingerprints"):
+                fp = tuple(sorted(result["fingerprints"].items()))
+            else:
+                # Fallback: create fingerprint from rule + location
+                loc = result.get("locations", [{}])[0]
+                phys = loc.get("physicalLocation", {})
+                fp = (
+                    result.get("ruleId"),
+                    phys.get("artifactLocation", {}).get("uri"),
+                    phys.get("region", {}).get("startLine")
+                )
+
+            if fp not in seen_fingerprints:
+                seen_fingerprints.add(fp)
+                unique_results.append(result)
+
+        run["results"] = unique_results
+
+    return sarif
+```
+
+## Strategy 5: Extracting Actionable Data
+
+```python
+import json
+from dataclasses import dataclass
+from typing import Optional
+
+@dataclass
+class Finding:
+    rule_id: str
+    level: str
+    message: str
+    file_path: Optional[str]
+    start_line: Optional[int]
+    end_line: Optional[int]
+    fingerprint: Optional[str]
+
+def extract_findings(sarif_path: str) -> list[Finding]:
+    """Extract structured findings from SARIF file."""
+    with open(sarif_path) as f:
+        sarif = json.load(f)
+
+    findings = []
+    for run in sarif.get("runs", []):
+        for result in run.get("results", []):
+            loc = result.get("locations", [{}])[0]
+            phys = loc.get("physicalLocation", {})
+            region = phys.get("region", {})
+
+            findings.append(Finding(
+                rule_id=result.get("ruleId", "unknown"),
+                level=result.get("level", "warning"),
+                message=result.get("message", {}).get("text", ""),
+                file_path=phys.get("artifactLocation", {}).get("uri"),
+                start_line=region.get("startLine"),
+                end_line=region.get("endLine"),
+                fingerprint=next(iter(result.get("partialFingerprints", {}).values()), None)
+            ))
+
+    return findings
+
+# Filter and prioritize
+def prioritize_findings(findings: list[Finding]) -> list[Finding]:
+    """Sort findings by severity."""
+    severity_order = {"error": 0, "warning": 1, "note": 2, "none": 3}
+    return sorted(findings, key=lambda f: severity_order.get(f.level, 99))
+```
+
+## Common Pitfalls and Solutions
+
+### 1. Path Normalization Issues
+
+Different tools report paths differently (absolute, relative, URI-encoded):
+
+```python
+from urllib.parse import unquote
+from pathlib import Path
+
+def normalize_path(uri: str, base_path: str = "") -> str:
+    """Normalize SARIF artifact URI to consistent path."""
+    # Remove file:// prefix if present
+    if uri.startswith("file://"):
+        uri = uri[7:]
+
+    # URL decode
+    uri = unquote(uri)
+
+    # Handle relative paths
+    if not Path(uri).is_absolute() and base_path:
+        uri = str(Path(base_path) / uri)
+
+    # Normalize separators
+    return str(Path(uri))
+```
+
+### 2. Fingerprint Mismatch Across Runs
+
+Fingerprints may not match if:
+- File paths differ between environments
+- Tool versions changed fingerprinting algorithm
+- Code was reformatted (changing line numbers)
+
+**Solution:** Use multiple fingerprint strategies:
+
+```python
+def compute_stable_fingerprint(result: dict, file_content: str = None) -> str:
+    """Compute environment-independent fingerprint."""
+    import hashlib
+
+    components = [
+        result.get("ruleId", ""),
+        result.get("message", {}).get("text", "")[:100],  # First 100 chars
+    ]
+
+    # Add code snippet if available
+    if file_content and result.get("locations"):
+        region = result["locations"][0].get("physicalLocation", {}).get("region", {})
+        if region.get("startLine"):
+            lines = file_content.split("\n")
+            line_idx = region["startLine"] - 1
+            if 0 <= line_idx < len(lines):
+                # Normalize whitespace
+                components.append(lines[line_idx].strip())
+
+    return hashlib.sha256("".join(components).encode()).hexdigest()[:16]
+```
+
+### 3. Missing or Incomplete Data
+
+SARIF allows many optional fields. Always use defensive access:
+
+```python
+def safe_get_location(result: dict) -> tuple[str, int]:
+    """Safely extract file and line from result."""
+    try:
+        loc = result.get("locations", [{}])[0]
+        phys = loc.get("physicalLocation", {})
+        file_path = phys.get("artifactLocation", {}).get("uri", "unknown")
+        line = phys.get("region", {}).get("startLine", 0)
+        return file_path, line
+    except (IndexError, KeyError, TypeError):
+        return "unknown", 0
+```
+
+### 4. Large File Performance
+
+For very large SARIF files (100MB+):
+
+```python
+import ijson  # pip install ijson
+
+def stream_results(sarif_path: str):
+    """Stream results without loading entire file."""
+    with open(sarif_path, "rb") as f:
+        # Stream through results arrays
+        for result in ijson.items(f, "runs.item.results.item"):
+            yield result
+```
+
+### 5. Schema Validation
+
+Validate before processing to catch malformed files:
+
+```bash
+# Using ajv-cli
+npm install -g ajv-cli
+ajv validate -s sarif-schema-2.1.0.json -d results.sarif
+
+# Using Python jsonschema
+pip install jsonschema
+```
+
+```python
+from jsonschema import validate, ValidationError
+import json
+
+def validate_sarif(sarif_path: str, schema_path: str) -> bool:
+    """Validate SARIF file against schema."""
+    with open(sarif_path) as f:
+        sarif = json.load(f)
+    with open(schema_path) as f:
+        schema = json.load(f)
+
+    try:
+        validate(sarif, schema)
+        return True
+    except ValidationError as e:
+        print(f"Validation error: {e.message}")
+        return False
+```
+
+## CI/CD Integration Patterns
+
+### GitHub Actions
+
+```yaml
+- name: Upload SARIF
+  uses: github/codeql-action/upload-sarif@v3
+  with:
+    sarif_file: results.sarif
+
+- name: Check for high severity
+  run: |
+    HIGH_COUNT=$(jq '[.runs[].results[] | select(.level == "error")] | length' results.sarif)
+    if [ "$HIGH_COUNT" -gt 0 ]; then
+      echo "Found $HIGH_COUNT high severity issues"
+      exit 1
+    fi
+```
+
+### Fail on New Issues
+
+```python
+from sarif import loader
+
+def check_for_regressions(baseline: str, current: str) -> int:
+    """Return count of new issues not in baseline."""
+    baseline_data = loader.load_sarif_file(baseline)
+    current_data = loader.load_sarif_file(current)
+
+    baseline_fps = {get_fingerprint(r) for r in baseline_data.get_results()}
+    new_issues = [r for r in current_data.get_results()
+                  if get_fingerprint(r) not in baseline_fps]
+
+    return len(new_issues)
+```
+
+## Key Principles
+
+1. **Validate first**: Check SARIF structure before processing
+2. **Handle optionals**: Many fields are optional; use defensive access
+3. **Normalize paths**: Tools report paths differently; normalize early
+4. **Fingerprint wisely**: Combine multiple strategies for stable deduplication
+5. **Stream large files**: Use ijson or similar for 100MB+ files
+6. **Aggregate thoughtfully**: Preserve tool metadata when combining files
+
+## Skill Resources
+
+For ready-to-use query templates, see [{baseDir}/resources/jq-queries.md]({baseDir}/resources/jq-queries.md):
+- 40+ jq queries for common SARIF operations
+- Severity filtering, rule extraction, aggregation patterns
+
+For Python utilities, see [{baseDir}/resources/sarif_helpers.py]({baseDir}/resources/sarif_helpers.py):
+- `normalize_path()` - Handle tool-specific path formats
+- `compute_fingerprint()` - Stable fingerprinting ignoring paths
+- `deduplicate_results()` - Remove duplicates across runs
+
+## Reference Links
+
+- [OASIS SARIF 2.1.0 Specification](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html)
+- [Microsoft SARIF Tutorials](https://github.com/microsoft/sarif-tutorials)
+- [SARIF SDK (.NET)](https://github.com/microsoft/sarif-sdk)
+- [sarif-tools (Python)](https://github.com/microsoft/sarif-tools)
+- [pysarif (Python)](https://github.com/Kjeld-P/pysarif)
+- [GitHub SARIF Support](https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning)
+- [SARIF Validator](https://sarifweb.azurewebsites.net/)

package/skills/sarif-parsing/resources/jq-queries.md

@@ -0,0 +1,162 @@
+# SARIF jq Query Reference
+
+Ready-to-use jq queries for common SARIF parsing tasks.
+
+## Basic Exploration
+
+```bash
+# Pretty print
+jq '.' results.sarif
+
+# Get SARIF version
+jq '.version' results.sarif
+
+# List tool names from all runs
+jq '.runs[].tool.driver.name' results.sarif
+
+# Count runs
+jq '.runs | length' results.sarif
+```
+
+## Result Queries
+
+```bash
+# Total result count
+jq '[.runs[].results[]] | length' results.sarif
+
+# Count by severity level
+jq 'reduce .runs[].results[] as $r ({}; .[$r.level] += 1)' results.sarif
+
+# List unique rule IDs
+jq '[.runs[].results[].ruleId] | unique | sort' results.sarif
+
+# Count per rule
+jq '[.runs[].results[]] | group_by(.ruleId) | map({rule: .[0].ruleId, count: length}) | sort_by(-.count)' results.sarif
+```
+
+## Filtering Results
+
+```bash
+# Only errors
+jq '.runs[].results[] | select(.level == "error")' results.sarif
+
+# Only warnings
+jq '.runs[].results[] | select(.level == "warning")' results.sarif
+
+# By specific rule ID
+jq --arg rule "SQL_INJECTION" '.runs[].results[] | select(.ruleId == $rule)' results.sarif
+
+# By file path (contains)
+jq --arg file "auth" '.runs[].results[] | select(.locations[].physicalLocation.artifactLocation.uri | contains($file))' results.sarif
+
+# By file extension
+jq '.runs[].results[] | select(.locations[].physicalLocation.artifactLocation.uri | test("\\.py$"))' results.sarif
+
+# Multiple conditions
+jq '.runs[].results[] | select(.level == "error" and (.ruleId | startswith("SEC")))' results.sarif
+```
+
+## Extracting Locations
+
+```bash
+# File and line for each result
+jq '.runs[].results[] | {
+  rule: .ruleId,
+  file: .locations[0].physicalLocation.artifactLocation.uri,
+  line: .locations[0].physicalLocation.region.startLine
+}' results.sarif
+
+# Unique affected files
+jq '[.runs[].results[].locations[].physicalLocation.artifactLocation.uri] | unique | sort' results.sarif
+
+# Results grouped by file
+jq '[.runs[].results[] | {file: .locations[0].physicalLocation.artifactLocation.uri, result: .}] | group_by(.file) | map({file: .[0].file, count: length})' results.sarif
+```
+
+## Rule Information
+
+```bash
+# List all rules with severity
+jq '.runs[].tool.driver.rules[] | {id: .id, name: .name, level: .defaultConfiguration.level}' results.sarif
+
+# Get rule description by ID
+jq --arg id "RULE001" '.runs[].tool.driver.rules[] | select(.id == $id)' results.sarif
+
+# Rules with help URLs
+jq '.runs[].tool.driver.rules[] | select(.helpUri) | {id: .id, help: .helpUri}' results.sarif
+```
+
+## Fingerprints
+
+```bash
+# Results with fingerprints
+jq '.runs[].results[] | select(.fingerprints or .partialFingerprints) | {rule: .ruleId, fp: (.fingerprints // .partialFingerprints)}' results.sarif
+
+# Extract all partial fingerprints
+jq '[.runs[].results[].partialFingerprints] | add' results.sarif
+```
+
+## Aggregation and Reporting
+
+```bash
+# Summary by severity and rule
+jq '[.runs[].results[]] | group_by(.level) | map({level: .[0].level, rules: (group_by(.ruleId) | map({rule: .[0].ruleId, count: length}))})' results.sarif
+
+# Top 10 most frequent rules
+jq '[.runs[].results[]] | group_by(.ruleId) | map({rule: .[0].ruleId, count: length}) | sort_by(-.count) | .[0:10]' results.sarif
+
+# Files with most issues
+jq '[.runs[].results[] | .locations[0].physicalLocation.artifactLocation.uri] | group_by(.) | map({file: .[0], count: length}) | sort_by(-.count) | .[0:10]' results.sarif
+```
+
+## Output Formatting
+
+```bash
+# CSV-like output
+jq -r '.runs[].results[] | [.ruleId, .level, .locations[0].physicalLocation.artifactLocation.uri, .locations[0].physicalLocation.region.startLine, .message.text] | @csv' results.sarif
+
+# Tab-separated
+jq -r '.runs[].results[] | [.ruleId, .level, .locations[0].physicalLocation.artifactLocation.uri // "N/A"] | @tsv' results.sarif
+
+# Markdown table
+echo "| Rule | Level | File | Line |"
+echo "|------|-------|------|------|"
+jq -r '.runs[].results[] | "| \(.ruleId) | \(.level) | \(.locations[0].physicalLocation.artifactLocation.uri // "N/A") | \(.locations[0].physicalLocation.region.startLine // "N/A") |"' results.sarif
+```
+
+## Comparison and Diff
+
+```bash
+# Find rules in file1 not in file2
+comm -23 <(jq -r '[.runs[].results[].ruleId] | unique | sort[]' file1.sarif) <(jq -r '[.runs[].results[].ruleId] | unique | sort[]' file2.sarif)
+
+# Compare result counts
+echo "File 1: $(jq '[.runs[].results[]] | length' file1.sarif)"
+echo "File 2: $(jq '[.runs[].results[]] | length' file2.sarif)"
+```
+
+## Transformation
+
+```bash
+# Extract minimal SARIF (results only)
+jq '{version: .version, runs: [.runs[] | {tool: {driver: {name: .tool.driver.name}}, results: .results}]}' results.sarif
+
+# Filter and create new SARIF with only errors
+jq '.runs[].results = [.runs[].results[] | select(.level == "error")]' results.sarif > errors-only.sarif
+
+# Merge multiple SARIF files
+jq -s '{version: "2.1.0", runs: [.[].runs[]]}' file1.sarif file2.sarif > merged.sarif
+```
+
+## Validation Checks
+
+```bash
+# Check if version is 2.1.0
+jq -e '.version == "2.1.0"' results.sarif && echo "Valid version" || echo "Invalid version"
+
+# Check for empty results
+jq -e '[.runs[].results[]] | length > 0' results.sarif && echo "Has results" || echo "No results"
+
+# Verify all results have locations
+jq '[.runs[].results[] | select(.locations | length == 0)] | length' results.sarif
+```