cli-jaw 0.1.0

package/skills_ref/static-analysis/skills/codeql/references/diagnostic-query-templates.md

@@ -0,0 +1,339 @@
+# Diagnostic Query Templates
+
+Language-specific QL queries for enumerating sources and sinks recognized by CodeQL. Used by the [create-data-extensions workflow](../workflows/create-data-extensions.md).
+
+## Source Enumeration Query
+
+All languages use the class `RemoteFlowSource`. The import differs per language.
+
+### Import Reference
+
+| Language | Imports | Class |
+|----------|---------|-------|
+| Python | `import python` + `import semmle.python.dataflow.new.RemoteFlowSources` | `RemoteFlowSource` |
+| JavaScript | `import javascript` | `RemoteFlowSource` |
+| Java | `import java` + `import semmle.code.java.dataflow.FlowSources` | `RemoteFlowSource` |
+| Go | `import go` | `RemoteFlowSource` |
+| C/C++ | `import cpp` + `import semmle.code.cpp.security.FlowSources` | `RemoteFlowSource` |
+| C# | `import csharp` + `import semmle.code.csharp.security.dataflow.flowsources.Remote` | `RemoteFlowSource` |
+| Ruby | `import ruby` + `import codeql.ruby.dataflow.RemoteFlowSources` | `RemoteFlowSource` |
+
+### Template (Python — swap imports per table above)
+
+```ql
+/**
+ * @name List recognized dataflow sources
+ * @description Enumerates all locations CodeQL recognizes as dataflow sources
+ * @kind problem
+ * @id custom/list-sources
+ */
+import python
+import semmle.python.dataflow.new.RemoteFlowSources
+
+from RemoteFlowSource src
+select src,
+  src.getSourceType()
+    + " | " + src.getLocation().getFile().getRelativePath()
+    + ":" + src.getLocation().getStartLine().toString()
+```
+
+**Note:** `getSourceType()` is available on Python, Java, and C#. For Go, JavaScript, Ruby, and C++ replace the select with:
+```ql
+select src,
+  src.getLocation().getFile().getRelativePath()
+    + ":" + src.getLocation().getStartLine().toString()
+```
+
+---
+
+## Sink Enumeration Queries
+
+The Concepts API differs significantly across languages. Use the correct template.
+
+### Concept Class Reference
+
+| Concept | Python | JavaScript | Go | Ruby |
+|---------|--------|------------|-----|------|
+| SQL | `SqlExecution.getSql()` | `DatabaseAccess.getAQueryArgument()` | `SQL::QueryString` (is-a Node) | `SqlExecution.getSql()` |
+| Command exec | `SystemCommandExecution.getCommand()` | `SystemCommandExecution.getACommandArgument()` | `SystemCommandExecution.getCommandName()` | `SystemCommandExecution.getAnArgument()` |
+| File access | `FileSystemAccess.getAPathArgument()` | `FileSystemAccess.getAPathArgument()` | `FileSystemAccess.getAPathArgument()` | `FileSystemAccess.getAPathArgument()` |
+| HTTP client | `Http::Client::Request.getAUrlPart()` | — | — | — |
+| Decoding | `Decoding.getAnInput()` | — | — | — |
+| XML parsing | — | — | — | `XmlParserCall.getAnInput()` |
+
+### Python
+
+```ql
+/**
+ * @name List recognized dataflow sinks
+ * @description Enumerates security-relevant sinks CodeQL recognizes
+ * @kind problem
+ * @id custom/list-sinks
+ */
+import python
+import semmle.python.Concepts
+
+from DataFlow::Node sink, string kind
+where
+  exists(SqlExecution e | sink = e.getSql() and kind = "sql-execution")
+  or
+  exists(SystemCommandExecution e |
+    sink = e.getCommand() and kind = "command-execution"
+  )
+  or
+  exists(FileSystemAccess e |
+    sink = e.getAPathArgument() and kind = "file-access"
+  )
+  or
+  exists(Http::Client::Request r |
+    sink = r.getAUrlPart() and kind = "http-request"
+  )
+  or
+  exists(Decoding d | sink = d.getAnInput() and kind = "decoding")
+  or
+  exists(CodeExecution e | sink = e.getCode() and kind = "code-execution")
+select sink,
+  kind
+    + " | " + sink.getLocation().getFile().getRelativePath()
+    + ":" + sink.getLocation().getStartLine().toString()
+```
+
+### JavaScript / TypeScript
+
+```ql
+/**
+ * @name List recognized dataflow sinks
+ * @description Enumerates security-relevant sinks CodeQL recognizes
+ * @kind problem
+ * @id custom/list-sinks-js
+ */
+import javascript
+
+from DataFlow::Node sink, string kind
+where
+  exists(DatabaseAccess e |
+    sink = e.getAQueryArgument() and kind = "database-access"
+  )
+  or
+  exists(SystemCommandExecution e |
+    sink = e.getACommandArgument() and kind = "command-execution"
+  )
+  or
+  exists(FileSystemAccess e |
+    sink = e.getAPathArgument() and kind = "file-access"
+  )
+select sink,
+  kind
+    + " | " + sink.getLocation().getFile().getRelativePath()
+    + ":" + sink.getLocation().getStartLine().toString()
+```
+
+### Go
+
+```ql
+/**
+ * @name List recognized dataflow sinks
+ * @description Enumerates security-relevant sinks CodeQL recognizes
+ * @kind problem
+ * @id custom/list-sinks-go
+ */
+import go
+import semmle.go.frameworks.SQL
+
+from DataFlow::Node sink, string kind
+where
+  sink instanceof SQL::QueryString and kind = "sql-query"
+  or
+  exists(SystemCommandExecution e |
+    sink = e.getCommandName() and kind = "command-execution"
+  )
+  or
+  exists(FileSystemAccess e |
+    sink = e.getAPathArgument() and kind = "file-access"
+  )
+select sink,
+  kind
+    + " | " + sink.getLocation().getFile().getRelativePath()
+    + ":" + sink.getLocation().getStartLine().toString()
+```
+
+### Ruby
+
+```ql
+/**
+ * @name List recognized dataflow sinks
+ * @description Enumerates security-relevant sinks CodeQL recognizes
+ * @kind problem
+ * @id custom/list-sinks-ruby
+ */
+import ruby
+import codeql.ruby.Concepts
+
+from DataFlow::Node sink, string kind
+where
+  exists(SqlExecution e | sink = e.getSql() and kind = "sql-execution")
+  or
+  exists(SystemCommandExecution e |
+    sink = e.getAnArgument() and kind = "command-execution"
+  )
+  or
+  exists(FileSystemAccess e |
+    sink = e.getAPathArgument() and kind = "file-access"
+  )
+  or
+  exists(CodeExecution e | sink = e.getCode() and kind = "code-execution")
+select sink,
+  kind
+    + " | " + sink.getLocation().getFile().getRelativePath()
+    + ":" + sink.getLocation().getStartLine().toString()
+```
+
+### Java
+
+Java lacks a unified Concepts module. Use language-specific sink classes. The diagnostics query needs its own `qlpack.yml` with a `codeql/java-all` dependency — create it alongside the `.ql` files:
+
+```yaml
+# $DIAG_DIR/qlpack.yml
+name: custom/diagnostics
+version: 0.0.1
+dependencies:
+  codeql/java-all: "*"
+```
+
+Then run `codeql pack install` in the diagnostics directory before executing queries.
+
+```ql
+/**
+ * @name List recognized dataflow sinks
+ * @description Enumerates security-relevant sinks CodeQL recognizes
+ * @kind problem
+ * @id custom/list-sinks
+ */
+import java
+import semmle.code.java.dataflow.DataFlow
+import semmle.code.java.security.QueryInjection
+import semmle.code.java.security.CommandLineQuery
+import semmle.code.java.security.TaintedPathQuery
+import semmle.code.java.security.XSS
+import semmle.code.java.security.RequestForgery
+import semmle.code.java.security.Xxe
+
+from DataFlow::Node sink, string kind
+where
+  sink instanceof QueryInjectionSink and kind = "sql-injection"
+  or
+  sink instanceof CommandInjectionSink and kind = "command-injection"
+  or
+  sink instanceof TaintedPathSink and kind = "path-injection"
+  or
+  sink instanceof XssSink and kind = "xss"
+  or
+  sink instanceof RequestForgerySink and kind = "ssrf"
+  or
+  sink instanceof XxeSink and kind = "xxe"
+select sink,
+  kind
+    + " | " + sink.getLocation().getFile().getRelativePath()
+    + ":" + sink.getLocation().getStartLine().toString()
+```
+
+### C / C++
+
+C++ uses a similar per-vulnerability-class pattern. Requires a `qlpack.yml` with `codeql/cpp-all` dependency (same approach as Java):
+
+```yaml
+# $DIAG_DIR/qlpack.yml
+name: custom/diagnostics
+version: 0.0.1
+dependencies:
+  codeql/cpp-all: "*"
+```
+
+Then run `codeql pack install` in the diagnostics directory before executing queries.
+
+```ql
+/**
+ * @name List recognized dataflow sinks
+ * @description Enumerates security-relevant sinks CodeQL recognizes
+ * @kind problem
+ * @id custom/list-sinks-cpp
+ */
+import cpp
+import semmle.code.cpp.dataflow.DataFlow
+import semmle.code.cpp.security.CommandExecution
+import semmle.code.cpp.security.FileAccess
+import semmle.code.cpp.security.BufferWrite
+
+from DataFlow::Node sink, string kind
+where
+  exists(FunctionCall call |
+    sink.asExpr() = call.getAnArgument() and
+    call.getTarget().hasGlobalOrStdName("system") and
+    kind = "command-injection"
+  )
+  or
+  exists(FunctionCall call |
+    sink.asExpr() = call.getAnArgument() and
+    call.getTarget().hasGlobalOrStdName(["fopen", "open", "freopen"]) and
+    kind = "file-access"
+  )
+  or
+  exists(FunctionCall call |
+    sink.asExpr() = call.getAnArgument() and
+    call.getTarget().hasGlobalOrStdName(["sprintf", "strcpy", "strcat", "gets"]) and
+    kind = "buffer-write"
+  )
+  or
+  exists(FunctionCall call |
+    sink.asExpr() = call.getAnArgument() and
+    call.getTarget().hasGlobalOrStdName(["execl", "execle", "execlp", "execv", "execvp", "execvpe", "popen"]) and
+    kind = "command-execution"
+  )
+select sink,
+  kind
+    + " | " + sink.getLocation().getFile().getRelativePath()
+    + ":" + sink.getLocation().getStartLine().toString()
+```
+
+### C\#
+
+C# uses per-vulnerability sink classes. Requires a `qlpack.yml` with `codeql/csharp-all` dependency:
+
+```yaml
+# $DIAG_DIR/qlpack.yml
+name: custom/diagnostics
+version: 0.0.1
+dependencies:
+  codeql/csharp-all: "*"
+```
+
+Then run `codeql pack install` in the diagnostics directory before executing queries.
+
+```ql
+/**
+ * @name List recognized dataflow sinks
+ * @description Enumerates security-relevant sinks CodeQL recognizes
+ * @kind problem
+ * @id custom/list-sinks-csharp
+ */
+import csharp
+import semmle.code.csharp.dataflow.DataFlow
+import semmle.code.csharp.security.dataflow.SqlInjectionQuery
+import semmle.code.csharp.security.dataflow.CommandInjectionQuery
+import semmle.code.csharp.security.dataflow.TaintedPathQuery
+import semmle.code.csharp.security.dataflow.XSSQuery
+
+from DataFlow::Node sink, string kind
+where
+  sink instanceof SqlInjection::Sink and kind = "sql-injection"
+  or
+  sink instanceof CommandInjection::Sink and kind = "command-injection"
+  or
+  sink instanceof TaintedPath::Sink and kind = "path-injection"
+  or
+  sink instanceof XSS::Sink and kind = "xss"
+select sink,
+  kind
+    + " | " + sink.getLocation().getFile().getRelativePath()
+    + ":" + sink.getLocation().getStartLine().toString()
+```

package/skills_ref/static-analysis/skills/codeql/references/language-details.md

@@ -0,0 +1,207 @@
+# Language-Specific Guidance
+
+## No Build Required
+
+### Python
+
+```bash
+codeql database create codeql.db --language=python --source-root=.
+```
+
+**Framework Support:**
+- Django, Flask, FastAPI: Built-in models
+- Tornado, Pyramid: Partial support
+- Custom frameworks: May need data extensions
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| Missing Django models | Ensure `settings.py` is at expected location |
+| Virtual env included | Use `paths-ignore` in config |
+| Type stubs missing | Install `types-*` packages before extraction |
+
+### JavaScript/TypeScript
+
+```bash
+codeql database create codeql.db --language=javascript --source-root=.
+```
+
+**Framework Support:**
+- React, Vue, Angular: Built-in models
+- Express, Koa, Fastify: HTTP source/sink models
+- Next.js, Nuxt: Partial SSR support
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| node_modules bloat | Already excluded by default |
+| TypeScript not parsed | Ensure `tsconfig.json` is valid |
+| Monorepo issues | Use `--source-root` for specific package |
+
+### Go
+
+```bash
+codeql database create codeql.db --language=go --source-root=.
+```
+
+**Framework Support:**
+- net/http, Gin, Echo, Chi: Built-in models
+- gRPC: Partial support
+- Custom routers: May need data extensions
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| Missing dependencies | Run `go mod download` first |
+| Vendor directory | CodeQL handles automatically |
+| CGO code | Requires `--command='go build'` with CGO enabled |
+
+### Ruby
+
+```bash
+codeql database create codeql.db --language=ruby --source-root=.
+```
+
+**Framework Support:**
+- Rails: Full support (controllers, models, views)
+- Sinatra: Built-in support
+- Hanami: Partial support
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| Bundler issues | Run `bundle install` first |
+| Rails engines | May need multiple database passes |
+
+## Build Required
+
+### C/C++
+
+```bash
+# Make
+codeql database create codeql.db --language=cpp --command='make -j8'
+
+# CMake
+codeql database create codeql.db --language=cpp \
+  --source-root=/path/to/src \
+  --command='cmake --build build'
+
+# Ninja
+codeql database create codeql.db --language=cpp \
+  --command='ninja -C build'
+```
+
+**Build System Tips:**
+| Build System | Command |
+|--------------|---------|
+| Make | `make clean && make -j$(nproc)` |
+| CMake | `cmake -B build && cmake --build build` |
+| Meson | `meson setup build && ninja -C build` |
+| Bazel | `bazel build //...` |
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| Partial extraction | Ensure `make clean` before CodeQL build |
+| Header-only libraries | Use `--extractor-option cpp_trap_headers=true` |
+| Cross-compilation | Set `CODEQL_EXTRACTOR_CPP_TARGET_ARCH` |
+
+### Java/Kotlin
+
+```bash
+# Gradle
+codeql database create codeql.db --language=java --command='./gradlew build -x test'
+
+# Maven
+codeql database create codeql.db --language=java --command='mvn compile -DskipTests'
+```
+
+**Framework Support:**
+- Spring Boot: Full support
+- Jakarta EE: Built-in models
+- Android: Requires Android SDK
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| Missing dependencies | Run `./gradlew dependencies` first |
+| Kotlin mixed projects | Use `--language=java` (covers both) |
+| Annotation processors | Ensure they run during CodeQL build |
+
+### Rust
+
+```bash
+codeql database create codeql.db --language=rust --command='cargo build'
+```
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| Proc macros | May require special handling |
+| Workspace projects | Use `--source-root` for specific crate |
+| Build script failures | Ensure native dependencies are available |
+
+### C#
+
+```bash
+# .NET Core
+codeql database create codeql.db --language=csharp --command='dotnet build'
+
+# MSBuild
+codeql database create codeql.db --language=csharp --command='msbuild /t:rebuild'
+```
+
+**Framework Support:**
+- ASP.NET Core: Full support
+- Entity Framework: Database query models
+- Blazor: Partial support
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| NuGet restore | Run `dotnet restore` first |
+| Multiple solutions | Specify solution file in command |
+
+### Swift
+
+```bash
+# Xcode project
+codeql database create codeql.db --language=swift \
+  --command='xcodebuild -project MyApp.xcodeproj -scheme MyApp build'
+
+# Swift Package Manager
+codeql database create codeql.db --language=swift --command='swift build'
+```
+
+**Requirements:**
+- macOS only
+- Xcode Command Line Tools
+
+**Common Issues:**
+| Issue | Fix |
+|-------|-----|
+| Code signing | Add `CODE_SIGN_IDENTITY=- CODE_SIGNING_REQUIRED=NO` |
+| Simulator target | Add `-sdk iphonesimulator` |
+
+## Extractor Options
+
+Set via environment variables: `CODEQL_EXTRACTOR_<LANG>_OPTION_<NAME>=<VALUE>`
+
+### C/C++ Options
+
+| Option | Description |
+|--------|-------------|
+| `trap_headers=true` | Include header file analysis |
+| `target_arch=x86_64` | Target architecture |
+
+### Java Options
+
+| Option | Description |
+|--------|-------------|
+| `jdk_version=17` | JDK version for analysis |
+
+### Python Options
+
+| Option | Description |
+|--------|-------------|
+| `python_executable=/path/to/python` | Specific Python interpreter |

package/skills_ref/static-analysis/skills/codeql/references/performance-tuning.md

@@ -0,0 +1,111 @@
+# Performance Tuning
+
+## Memory Configuration
+
+### CODEQL_RAM Environment Variable
+
+Control maximum heap memory (in MB):
+
+```bash
+# 48GB for large codebases
+CODEQL_RAM=48000 codeql database analyze codeql.db ...
+
+# 16GB for medium codebases
+CODEQL_RAM=16000 codeql database analyze codeql.db ...
+```
+
+**Guidelines:**
+| Codebase Size | Recommended RAM |
+|---------------|-----------------|
+| Small (<100K LOC) | 4-8 GB |
+| Medium (100K-1M LOC) | 8-16 GB |
+| Large (1M+ LOC) | 32-64 GB |
+
+## Thread Configuration
+
+### Analysis Threads
+
+```bash
+# Use all available cores
+codeql database analyze codeql.db --threads=0 ...
+
+# Use specific number
+codeql database analyze codeql.db --threads=8 ...
+```
+
+**Note:** `--threads=0` uses all available cores. For shared machines, use explicit count.
+
+## Query-Level Timeouts
+
+Prevent individual queries from running indefinitely:
+
+```bash
+# Set per-query timeout (in milliseconds)
+codeql database analyze codeql.db --timeout=600000 ...
+```
+
+A 10-minute timeout (`600000`) catches runaway queries without killing legitimate complex analysis. Taint-tracking queries on large codebases may need longer.
+
+## Evaluator Diagnostics
+
+When analysis is slow, use `--evaluator-log` to identify which queries consume the most time:
+
+```bash
+codeql database analyze codeql.db \
+  --evaluator-log=evaluator.log \
+  --format=sarif-latest \
+  --output=results.sarif \
+  -- codeql/python-queries:codeql-suites/python-security-extended.qls
+
+# Summarize the log
+codeql generate log-summary evaluator.log --format=text
+```
+
+The summary shows per-query timing and tuple counts. Queries producing millions of tuples are likely the bottleneck.
+
+## Disk Space
+
+| Phase | Typical Size | Notes |
+|-------|-------------|-------|
+| Database creation | 2-10x source size | Compiled languages are larger due to build tracing |
+| Analysis cache | 1-5 GB | Stored in database directory |
+| SARIF output | 1-50 MB | Depends on finding count |
+
+Check available space before starting:
+
+```bash
+df -h .
+du -sh codeql_*.db 2>/dev/null
+```
+
+## Caching Behavior
+
+CodeQL caches query evaluation results inside the database directory. Subsequent runs of the same queries skip re-evaluation.
+
+| Scenario | Cache Effect |
+|----------|-------------|
+| Re-run same packs | Fast — uses cached results |
+| Add new query pack | Only new queries evaluate |
+| `codeql database cleanup` | Clears cache — forces full re-evaluation |
+| `--rerun` flag | Ignores cache for this run |
+
+**When to clear cache:**
+- After deploying new data extensions (cache may hold stale results)
+- When investigating unexpected zero-finding results
+- Before benchmark comparisons (ensures consistent timing)
+
+```bash
+# Clear evaluation cache
+codeql database cleanup codeql_1.db
+```
+
+## Troubleshooting Performance
+
+| Symptom | Likely Cause | Solution |
+|---------|--------------|----------|
+| OOM during analysis | Not enough RAM | Increase `CODEQL_RAM` |
+| Slow database creation | Complex build | Use `--threads`, simplify build |
+| Slow query execution | Large codebase | Reduce query scope, add RAM |
+| Database too large | Too many files | Use exclusion config (see [build-database workflow](../workflows/build-database.md#1b-create-exclusion-config-interpreted-languages-only)) |
+| Single query hangs | Runaway evaluation | Use `--timeout` and check `--evaluator-log` |
+| Repeated runs still slow | Cache not used | Check you're using same database path |

package/skills_ref/static-analysis/skills/codeql/references/ruleset-catalog.md

@@ -0,0 +1,63 @@
+# Ruleset Catalog
+
+## Official CodeQL Suites
+
+| Suite | False Positives | Use Case |
+|-------|-----------------|----------|
+| `security-extended` | Low | **Default** - Security audits |
+| `security-and-quality` | Medium | Comprehensive review |
+| `security-experimental` | Higher | Research, vulnerability hunting |
+
+**Usage:** `codeql/<lang>-queries:codeql-suites/<lang>-security-extended.qls`
+
+**Languages:** `cpp`, `csharp`, `go`, `java`, `javascript`, `python`, `ruby`, `swift`
+
+---
+
+## Trail of Bits Packs
+
+| Pack | Language | Focus |
+|------|----------|-------|
+| `trailofbits/cpp-queries` | C/C++ | Memory safety, integer overflows |
+| `trailofbits/go-queries` | Go | Concurrency, error handling |
+| `trailofbits/java-queries` | Java | Security, code quality |
+
+**Install:**
+```bash
+codeql pack download trailofbits/cpp-queries
+codeql pack download trailofbits/go-queries
+codeql pack download trailofbits/java-queries
+```
+
+---
+
+## CodeQL Community Packs
+
+| Pack | Language |
+|------|----------|
+| `GitHubSecurityLab/CodeQL-Community-Packs-JavaScript` | JavaScript/TypeScript |
+| `GitHubSecurityLab/CodeQL-Community-Packs-Python` | Python |
+| `GitHubSecurityLab/CodeQL-Community-Packs-Go` | Go |
+| `GitHubSecurityLab/CodeQL-Community-Packs-Java` | Java |
+| `GitHubSecurityLab/CodeQL-Community-Packs-CPP` | C/C++ |
+| `GitHubSecurityLab/CodeQL-Community-Packs-CSharp` | C# |
+| `GitHubSecurityLab/CodeQL-Community-Packs-Ruby` | Ruby |
+
+**Install:**
+```bash
+codeql pack download GitHubSecurityLab/CodeQL-Community-Packs-<Lang>
+```
+
+**Source:** [github.com/GitHubSecurityLab/CodeQL-Community-Packs](https://github.com/GitHubSecurityLab/CodeQL-Community-Packs)
+
+---
+
+## Verify Installation
+
+```bash
+# List all installed packs
+codeql resolve qlpacks
+
+# Check specific packs
+codeql resolve qlpacks | grep -E "(trailofbits|GitHubSecurityLab)"
+```