octocode-cli 1.2.6 → 1.2.8

package/skills/octocode-code-engineer/references/improvement-roadmap.md

@@ -0,0 +1,304 @@
+# Improvement Roadmap
+
+Research-backed upgrade plan for the weakest parts of the skill: security analysis, test-quality analysis, semantic analysis, output/reporting, and test-suite quality.
+
+For validation policy, see the Principles section in [SKILL.md](../SKILL.md) and [validate & investigate](./validate-investigate.md).
+
+**Status legend**: Done, Partial, Planned
+
+---
+
+## 1. Security Analysis
+
+### Current weakness
+
+The current security layer is strong on breadth but still depends heavily on single-file heuristics. That creates false positives for patterns that look dangerous but are not proven dangerous in context.
+
+### Target architecture
+
+Move from isolated pattern detection to a lightweight taint model:
+
+`sources -> propagators -> sanitizers -> sinks -> evidence trace`
+
+Keep cheap AST rules for obvious cases:
+- `eval`
+- `new Function`
+- direct `innerHTML`
+- hardcoded secrets
+
+Upgrade the noisier rules to dataflow-backed analysis:
+- `prototype-pollution-risk`
+- `sql-injection-risk`
+- `unsafe-html`
+- `unvalidated-input-sink`
+- `input-passthrough-risk`
+
+### Concrete upgrades
+
+- Add sink-specific rule models for SQL, HTML, command execution, file writes, path joins, and object merge/write sites.
+- Add sanitizer catalogs for common validation and encoding patterns.
+- Add confidence scoring per rule: `high`, `medium`, `low`.
+- Add finding evidence: source parameter, propagation steps, sink call, sanitizer status.
+- Suppress structural false positives where the dynamic key is synthesized locally and never user-controlled.
+
+### P0 work
+
+- Split current security detectors into `pattern` rules and `flow` rules. *(Partial — detectors use evidence/confidence/ruleId)*
+- Add fixture tests for true positive and false positive pairs. *(Partial — test file exists)*
+- Add `confidence` and `evidence` fields to security findings. *(Done — `toSecurityFinding` in security.ts)*
+
+### P1 work
+
+- Build intra-procedural taint tracking inside a function body. *(Planned)*
+- Add reusable source/sink/sanitizer definitions. *(Planned)*
+- Add validation playbooks for each security category using Octocode local tools. *(Partial — playbooks.md covers some)*
+
+---
+
+## 2. Test-Quality Analysis
+
+### Current weakness
+
+The current test-quality pass mostly counts assertions, mocks, and setup hooks. That is helpful, but still shallow for real flakiness and false-confidence detection.
+
+### Target architecture
+
+Extend test analysis from simple counters to behavior-aware checks:
+
+- assertion presence
+- assertion reachability on all paths
+- cleanup and restore behavior
+- deterministic execution
+- framework misuse
+
+### Concrete upgrades
+
+- Detect async tests that neither `await` nor return a promise.
+- Detect `test.only`, `describe.only`, `skip`, and `todo`.
+- Detect fake timers without restore and mock/spy state not reset or restored.
+- Detect time, randomness, environment, and global-state coupling.
+- Detect snapshot-only tests and interaction-only tests with no outcome assertions.
+- Detect cleanup that exists on one path but not all paths.
+
+### P0 work
+
+- Add dedicated detector tests for test-quality rules. *(Done — test-quality.test.ts)*
+- Add rules for focused tests, fake timers without restore, and missing mock restoration. *(Done — all 8 categories implemented)*
+- Add a richer `testProfile` summary for timers, mocks, async patterns, and cleanup hooks. *(Done — TestProfile in interfaces.ts)*
+
+### P1 work
+
+- Add code-path-aware assertion and cleanup checks. *(Planned)*
+- Add framework-specific adapters for Vitest/Jest style APIs. *(Planned)*
+- Add flaky-test tags and recommended remediation steps. *(Planned)*
+
+---
+
+## 3. Semantic Analysis
+
+### Current weakness
+
+Semantic analysis is valuable, but it currently rebuilds a fresh TypeScript language service and uses a constant script version. That limits scale and wastes work on repeated scans.
+
+### Target architecture
+
+Adopt a persistent project-backed semantic engine:
+
+- cache by `tsconfig`
+- track file versions
+- reuse TypeScript project state across scans
+- support project references cleanly
+
+### Concrete upgrades
+
+- Replace ad hoc `LanguageService` creation with a project-service wrapper.
+- Separate semantic fact collection from detector execution.
+- Cache export references, inheritance chains, implementation maps, and symbol relationships.
+- Expose semantic facts to detectors through a stable query surface instead of repeated tree walks.
+
+### P0 work
+
+- Introduce a semantic cache keyed by root + tsconfig + file versions. *(Planned)*
+- Stop hardcoding script version `"1"`. *(Planned)*
+- Benchmark semantic scan cost before and after caching. *(Planned)*
+
+### P1 work
+
+- Move to a Project Service style lifecycle. *(Planned)*
+- Support project references and monorepo workspaces. *(Planned)*
+- Share semantic state between multiple detectors in a single run. *(Partial — `runSemanticDetectors` shares ctx)*
+
+---
+
+## 4. Output & Reporting
+
+### Current weakness
+
+The output is rich, but report generation is currently brittle and the output contract is not explicit enough to protect downstream tooling.
+
+### Target architecture
+
+Treat findings and reports as a versioned API:
+
+- one normalized internal result model
+- multiple emitters from that model
+- stable schema version
+- stable rule IDs
+- optional SARIF output
+
+### Concrete upgrades
+
+- Normalize `summary.json`, `findings.json`, and Markdown generation around one canonical result object.
+- Add `schemaVersion`, `confidence`, `evidence`, and `ruleId`.
+- Add SARIF emission with stable fingerprints.
+- Add diff/baseline mode so teams can adopt the tool incrementally.
+- Add contract tests for output shapes and golden tests for Markdown rendering.
+
+### P0 work
+
+- Fix the report regression first. *(Done)*
+- Add dedicated golden tests for `summary.md`, `summary.json`, and `findings.json`. *(Done — output-contract.test.ts)*
+- Add contract assertions around required keys and nullable fields. *(Done — schemaVersion, REPORT_SCHEMA_VERSION)*
+
+### P1 work
+
+- Add SARIF emitter. *(Planned)*
+- Add baseline and diff output modes. *(Planned)*
+- Add category-level and confidence-level summary slices. *(Planned)*
+
+---
+
+## 5. Test-Suite Quality
+
+### Current weakness
+
+The suite is large, but the failing report tests show that critical output paths can still regress together. Some important detector modules do not have focused test files.
+
+### Target architecture
+
+Use layered testing:
+
+- focused detector unit tests
+- integration tests for orchestration
+- golden tests for reports
+- property-based tests for AST invariants
+- mutation testing for critical rules
+
+### Concrete upgrades
+
+- Add dedicated tests for `security-detectors`, `test-quality-detectors`, and `tree-sitter-analyzer`.
+- Add property-based tests for AST search and report invariants.
+- Add mutation testing for high-risk detectors and output generation.
+- Add smoke tests that run the scanner against its own source and assert key categories.
+
+### P0 work
+
+- Restore a green Vitest run. *(Done)*
+- Add missing focused test files. *(Partial — 34 test files, some detector modules still untested)*
+- Lock down report and findings schema expectations. *(Done — output-contract.test.ts)*
+
+### P1 work
+
+- Add property-based tests with `fast-check`. *(Planned)*
+- Add mutation testing with Stryker for critical modules. *(Planned)*
+- Add self-scan fixture snapshots for detector stability. *(Planned)*
+
+---
+
+## 6. Architecture Analysis Depth
+
+### Current weakness
+
+The current architecture layer is strongest at file-level import analysis and architecture heuristics, but it still underuses graph science and AST/dataflow techniques that would make boundary and coupling defects more explainable.
+
+### Target architecture
+
+Treat architecture analysis as a hybrid of graph evidence and structural evidence:
+
+- graph evidence for dependency shape, chokepoints, layering, and subsystem boundaries
+- AST/semantic evidence for code roles, boundary leaks, side effects, and repeated orchestration
+
+### Graph technique upgrades
+
+- Add SCC condensation graphs so large file-level cycles collapse into interpretable cycle clusters.
+- Add folder/package graphs to surface subsystem-level cycles and cross-boundary chatter.
+- Add articulation-point and bridge-edge detection to identify brittle chokepoints.
+- Add broker or betweenness-centrality scoring to find modules that mediate too many paths.
+- Add change-coupling overlays from git history to catch architecture defects the import graph misses.
+
+### AST and semantic technique upgrades
+
+- Add relational or composite AST rules for architecture motifs, not just single-node patterns.
+- Add symbol-level usage graphs so cohesion and feature-envy checks work below the file level.
+- Add CFG/dataflow checks for boundary leaks, initialization order, and validation-before-sink behavior.
+- Add import-time effect tracing to classify module-scope I/O, registration, and global mutation.
+- Add boundary-role detection so controllers, services, domain modules, and infrastructure code can be checked semantically instead of only by path names.
+
+### P0 work
+
+- Expand the docs and playbooks so agents interpret architecture findings through graph and AST lenses together. *(Done — tool-workflows.md, playbooks.md)*
+- Surface existing hub-node and hotspot signals more explicitly in result reading guidance. *(Done — hotFiles in summary.md)*
+- Add fixture-based tests for graph-hotspot interpretation and architecture-summary rendering. *(Partial)*
+
+### P1 work
+
+- Implement SCC condensation and package-level dependency views. *(Partial — SCC clusters implemented in graph-analytics.ts)*
+- Add broker centrality and articulation-point scoring to hotspot analysis. *(Done — broker-module, bridge-module in graph-analytics.ts)*
+- Add relational AST rules for boundary leaks, split-brain modules, and import-time orchestration. *(Partial — import-side-effect-risk implemented)*
+
+### P2 work
+
+- Add lightweight local dataflow for architecture rules.
+- Combine graph scores with AST evidence into a single architecture-confidence model.
+- Add change-coupling overlays and folder/community clustering for subsystem discovery.
+
+---
+
+## Delivery Phases
+
+### Phase 0: Stabilize
+
+- Fix output/reporting regressions.
+- Make Vitest green.
+- Add missing focused tests.
+- Enforce Octocode local-tool validation in the skill docs and playbooks.
+- Tighten architecture reading guidance around graph and AST signals.
+
+### Phase 1: Improve Precision
+
+- Add security taint modeling inside a function body.
+- Add richer test-quality rules for cleanup, timers, mocks, and async behavior.
+- Add confidence and evidence fields to findings.
+
+### Phase 2: Improve Scale
+
+- Add persistent semantic state and project-backed analysis.
+- Add semantic fact caching.
+- Add baseline/diff mode and SARIF output.
+
+### Phase 3: Deepen Coverage
+
+- Add optional interprocedural summaries.
+- Add property-based and mutation testing.
+- Externalize more AST-only rules into rule packs.
+- Add deeper graph and subsystem analysis for architecture defects.
+
+---
+
+## Research Basis
+
+- TypeScript Compiler API wiki: https://github.com/microsoft/TypeScript/wiki/Using-the-Compiler-API
+- typescript-eslint Project Service docs: https://typescript-eslint.io/packages/project-service/generated/
+- typescript-eslint Project Service blog: https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/website/blog/2025-05-29-project-service.mdx
+- Semgrep taint analysis overview: https://github.com/semgrep/semgrep-docs/blob/main/docs/writing-rules/data-flow/taint-mode/overview.md
+- ast-grep relational rules: https://github.com/ast-grep/ast-grep.github.io/blob/main/website/guide/rule-config/relational-rule.md
+- ESLint code path analysis: https://eslint.org/docs/latest/extend/code-path-analysis
+- Tree-sitter predicates and directives: https://tree-sitter.github.io/tree-sitter/using-parsers/queries/3-predicates-and-directives.html
+- dependency-cruiser rules reference: https://github.com/sverweij/dependency-cruiser/blob/main/doc/rules-reference.md
+- CodeQL data flow analysis: https://github.com/github/codeql/blob/main/docs/codeql/writing-codeql-queries/about-data-flow-analysis.rst
+- CodeQL JS/TS data flow guide: https://github.com/github/codeql/blob/main/docs/codeql/codeql-language-guides/analyzing-data-flow-in-javascript-and-typescript.rst
+- Vitest coverage reporters: https://github.com/vitest-dev/vitest/blob/main/docs/config/coverage.md
+- Vitest timers guide: https://vitest.dev/guide/mocking/timers
+- Stryker JS usage: https://github.com/stryker-mutator/stryker-js/blob/master/docs/usage.md
+- fast-check getting started: https://fast-check.dev/docs/introduction/getting-started/
+- GitHub SARIF fingerprints: https://docs.github.com/en/code-security/reference/code-scanning/sarif-files/sarif-support-for-code-scanning

package/skills/octocode-code-engineer/references/output-files.md

@@ -0,0 +1,144 @@
+# Output Files
+
+Each scan writes to `.octocode/scan/<timestamp>/`:
+
+| File | Contents | When to Read |
+|------|----------|-------------|
+| `summary.md` | Health scores, tags, severity, per-pillar counts, top recs, change risk hotspots | **Always first** |
+| `summary.json` | Machine-readable scan metadata, `agentOutput`, `analysisSummary`, `investigationPrompts`, `parseErrors[]` | Programmatic access |
+| `architecture.json` | Dep graph, arch findings, `hotFiles[]`, `graphSignals[]`, chokepoints, optional advanced graph overlays | Cycles, coupling, SDP, D metric, test gaps, side-effect risk |
+| `code-quality.json` | Up to 28 quality findings, severity/category breakdowns | Duplicates, complexity, perf |
+| `dead-code.json` | Up to 10 hygiene findings, severity/category breakdowns | Dead code cleanup |
+| `file-inventory.json` | Per-file: functions, flows, metrics, `issueIds[]` | Deep-diving a specific file |
+| `findings.json` | ALL findings sorted by severity with `ruleId`, `analysisLens`, `confidence`, `impact`, `correlatedSignals[]`, `recommendedValidation`, and optional `flowTrace[]` | Complete sorted list |
+| `ast-trees.txt` | `Kind[startLine:endLine]` per file (on by default, disable with `--no-tree`) | Structural overview |
+| `graph.md` | Mermaid dependency graph (only with `--graph`) | Visual architecture |
+
+---
+
+## JSON Key Reference
+
+### `summary.json`
+
+```
+schemaVersion, generatedAt, repoRoot, options, parser,
+summary { totalPackages, totalFiles, totalNodes, totalFunctions, totalFlows, totalDependencyFiles, byPackage },
+agentOutput { totalFindings, highPriority, mediumPriority, lowPriority,
+              topRecommendations[] { id, file, severity, category, title, reason, suggestedFix },
+              filesWithIssues[] { file, issueCount, issueIds } },
+analysisSummary { graphSignals[], astSignals[], strongestGraphSignal, strongestAstSignal, combinedSignals[], recommendedValidation },
+strongestGraphSignal, strongestAstSignal, combinedSignals[], recommendedValidation, investigationPrompts[],
+parseErrors[] { file, message },
+outputFiles { summary, architecture, codeQuality, deadCode, fileInventory, findings, ... }
+```
+
+Use `summary.json` to drive the first decision:
+
+- Use `agentOutput.topRecommendations[]` and `filesWithIssues[]` to decide where to drill in first
+- Use `summary.md` or `architecture.json` for graph-specific detail such as `cycles`, `criticalPaths`, and hotspots
+- If top recommendations are mostly complexity, duplication, or side-effect findings, switch to AST-first investigation
+- If graph-heavy recommendations and AST-heavy recommendations appear together, plan a combined investigation before proposing refactors
+
+### `findings.json`
+
+```
+generatedAt,
+optimizationFindings[] { id, ruleId, severity, category, analysisLens, confidence,
+                         file, lineStart, lineEnd, title, reason,
+                         files[], suggestedFix { strategy, steps[] }, impact, tags[],
+                         correlatedSignals[], recommendedValidation, flowTrace[], lspHints[] },
+totalFindings
+```
+
+Filter: `jq '.optimizationFindings[] | select(.tags | contains(["coupling"]))' findings.json`
+
+Use `findings.json` to correlate categories:
+
+- `feature-envy` + `low-cohesion` = likely boundary error
+- `layer-violation` + `feature-envy` = likely dependency leak
+- `import-side-effect-risk` + hotspot tags = likely startup risk
+- `dependency-critical-path` + complexity tags = likely change chokepoint
+
+### `architecture.json`
+
+```
+schemaVersion, generatedAt,
+dependencyGraph { totalModules, totalEdges, criticalModules[], cycles[], criticalPaths[], ... },
+dependencyFindings[], findings[], findingsCount,
+severityBreakdown { critical, high, medium, low },
+categoryBreakdown { "dependency-cycle": N, ... },
+hotFiles[] { file, riskScore, fanIn, fanOut, complexityScore, exportCount, inCycle, onCriticalPath },
+graphSignals[], chokepoints[], criticalHubCandidates[],
+sccClusters[] (with `--graph-advanced`), packageGraphSummary (with `--graph-advanced`), packageHotspots[] (with `--graph-advanced`)
+```
+
+Use `architecture.json` as the graph lens:
+
+- `criticalModules[]` = hub nodes already surfaced by the dependency summary
+- `cycles[]` = immediate structural loops
+- `criticalPaths[]` = long change propagation chains
+- `hotFiles[]` = current approximation of graph chokepoints
+- `graphSignals[]` = already-interpreted graph narratives for triage
+- `chokepoints[]` = broker and articulation-style structural pressure points
+- `categoryBreakdown` = whether the repo’s architecture risk is mostly cycles, layering, cohesion, or side effects
+
+Good investigation prompts:
+
+- "Do critical hub modules also appear in hotFiles or critical paths?"
+- "Which files are both hot and on a critical path?"
+- "Which layer violations cluster around the same folder?"
+- "Do side-effectful modules also have high fan-in?"
+
+### `code-quality.json`
+
+```
+generatedAt, duplicateFlows { duplicateFunctions[], redundantFlows[] },
+optimizationOpportunities[] { type, message, file, lineStart, lineEnd, details },
+findings[], findingsCount, severityBreakdown, categoryBreakdown
+```
+
+### `dead-code.json`
+
+```
+generatedAt, findings[], findingsCount, severityBreakdown, categoryBreakdown
+```
+
+### `file-inventory.json`
+
+```
+generatedAt, fileCount,
+fileInventory[] { package, file, parseEngine, nodeCount, kindCounts,
+                  functions[] { name, lineStart, lineEnd, complexity, cognitiveComplexity, ... },
+                  flows[], dependencyProfile { internalDependencies[], externalDependencies[],
+                  declaredExports[], importedSymbols[], reExports[] },
+                  emptyCatches[], switchesWithoutDefault[], anyCount, magicNumbers[],
+                  topLevelEffects[], effectProfile, symbolUsageSummary, boundaryRoleHints[], cfgFlags,
+                  prototypePollutionSites[], issueIds[] }
+```
+
+Use `file-inventory.json` as the AST lens:
+
+- `functions[]` = shape and complexity of orchestration
+- `flows[]` = repeated control structures
+- `dependencyProfile` = exported/imported symbol detail for cohesion and feature-envy follow-up
+- `topLevelEffects[]` = hidden initialization / import-time work
+- `effectProfile` = summarized import-time risk
+- `symbolUsageSummary` = compact symbol/import/export shape for boundary follow-up
+- `boundaryRoleHints[]` = lightweight role inference for the file
+- `cfgFlags` = lightweight flow clues for validation, cleanup, exit behavior, and async boundaries (with `--flow`)
+
+If `architecture.json` names a hotspot, use `file-inventory.json` to explain why that hotspot is structurally hard to change.
+
+---
+
+## Reading `ast-trees.txt`
+
+For format, navigation commands, and usage guide, see [ast-tree-search.md](./ast-tree-search.md).
+
+On by default (`--emit-tree`). Suppress with `--no-tree`. Tree depth: `--tree-depth N` (default: 4).
+
+---
+
+## Legacy Single-File Mode (`--out path/to/file.json`)
+
+Keys: `summary`, `fileInventory[]`, `duplicateFlows`, `dependencyGraph`, `dependencyFindings[]`, `optimizationFindings[]`, `agentOutput`, `parseErrors[]`.