@dailephd/my-dev-kit-lab 0.3.0 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -6
- package/dist/scripts/audits/runAudit.js +9 -2
- package/dist/scripts/security/validate.js +8 -29
- package/dist/src/audits/codeRot/detectors/deadCodeCandidateDetector.js +138 -11
- package/dist/src/audits/codeRot/detectors/duplicateImplementationDetector.js +93 -1
- package/dist/src/audits/codeRot/detectors/testRotDetector.js +176 -0
- package/dist/src/audits/codeRot/utils/filePatternUtils.js +50 -0
- package/dist/src/audits/codeRot/utils/sourceFactsLookup.js +53 -0
- package/dist/src/audits/core/auditConfig.js +2 -2
- package/dist/src/audits/core/auditRunner.js +23 -1
- package/dist/src/audits/core/auditTypes.js +14 -7
- package/dist/src/audits/core/collectSourceFacts.js +142 -0
- package/dist/src/audits/core/languageAnalyzerRegistry.js +36 -0
- package/dist/src/audits/core/projectInventory.js +185 -3
- package/dist/src/audits/core/pythonAnalyzer.js +328 -0
- package/dist/src/audits/core/pythonProjectMetadata.js +80 -0
- package/dist/src/audits/core/sourceFacts.js +17 -0
- package/dist/src/audits/core/typescriptJavaScriptAnalyzer.js +299 -0
- package/dist/src/audits/report/auditReportModel.js +15 -0
- package/dist/src/audits/report/renderAuditJsonReport.js +3 -0
- package/dist/src/audits/report/renderAuditTextReport.js +59 -3
- package/dist/src/audits/security/mapSecurityFindingToAuditIssue.js +70 -0
- package/dist/src/audits/security/securityAuditAdapter.js +67 -0
- package/dist/src/audits/security/securityAuditTypes.js +17 -0
- package/dist/src/securityValidation/report/buildSecurityReport.js +31 -0
- package/dist/src/securityValidation/validate/resolveTarget.js +9 -0
- package/package.json +97 -96
package/README.md
CHANGED
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
# my-dev-kit-lab
|
|
2
2
|
|
|
3
|
-
my-dev-kit-lab is the experiment, evidence, reporting, security-validation, and
|
|
3
|
+
my-dev-kit-lab is the experiment, evidence, reporting, security-validation, and audit companion for my-dev-kit. It runs reproducible experiments that test whether my-dev-kit's graph-guided retrieval helps coding-agent workflows, collects metrics, renders reports, generates plots, captures screenshots, builds gallery outputs, performs automated CLI/package security validation, and runs the generic audit framework.
|
|
4
4
|
|
|
5
|
-
The current
|
|
5
|
+
The current published baseline is `v0.3.1`. `v0.3.0` introduced the generic audit framework and the first implemented code-rot detector family; `v0.3.1` added the language-aware code-rot substrate plus TypeScript/JavaScript support. The generic experiment-plugin framework was introduced in `v0.2.0`. The first plugin is `context-strategy-comparison`, which preserves the existing raw-full-file vs my-dev-kit-guided workflow through the plugin runner.
|
|
6
|
+
|
|
7
|
+
The checked-out package state is `v0.3.2` release-prepared on top of the published `v0.3.1` baseline: Python-aware code-rot support (a Python source-facts analyzer, Python project metadata, and Python-aware dead-code/duplicate-implementation/test-rot detector signals) and a security-validation audit adapter (`npm run audit -- --types security`, mapped security findings, and a `securitySummary` audit report field). Package metadata is bumped for release preparation, but `v0.3.2` is not published yet; Java, Kotlin, cross-language stability, Android validation, framework-aware profiles, and the `quality`/`project`/`all` audit types remain future roadmap work.
|
|
6
8
|
|
|
7
9
|
my-dev-kit is most useful when the repository is larger than the task. It helps coding agents work with large codebases through reusable structural indexing, graph-guided retrieval, targeted source slices, and auditable context selection. Results are scoped evidence; the project does not claim that my-dev-kit always saves tokens.
|
|
8
10
|
|
|
@@ -31,8 +33,10 @@ my-dev-kit is most useful when the repository is larger than the task. It helps
|
|
|
31
33
|
- First experiment plugin: `context-strategy-comparison`
|
|
32
34
|
- Target-aware experiment execution for local projects via `experiment:run -- --target <path>`
|
|
33
35
|
- Plugin-aware JSON and HTML reports with plugin, target, variant, metric, artifact, warning, skip, and failure metadata
|
|
34
|
-
- Security validation framework: dependency audit, package tarball inspection, CLI adversarial tests, static scans (CodeQL/Semgrep), bounded fuzz smoke, attack-scenario checks, and structured verdict/report output — runnable against any local project via `security:validate --target <path
|
|
35
|
-
- Generic audit framework
|
|
36
|
+
- Security validation framework: dependency audit, package tarball inspection, CLI adversarial tests, static scans (CodeQL/Semgrep), bounded fuzz smoke, attack-scenario checks, and structured verdict/report output — runnable against any local project via `security:validate --target <path>`. `security:validate` remains the standalone, focused security-validation command; it is not replaced by the audit adapter below.
|
|
37
|
+
- Generic audit framework, current published baseline: `npm run audit` runs heuristic, conservative code-rot detectors and writes text/JSON reports under `reports/audits/code-rot/` by default. `code-rot` and `security` are the currently implemented audit types (`--types code-rot`, `--types security`, or combined `--types code-rot,security`); `quality`, `project`, and `all` audit types are planned and fail cleanly instead of running. `npm run audit` supports `--target`, `--types`, and other flags — see [docs/COMMANDS.md](docs/COMMANDS.md). Audit does not modify target files and does not auto-fix issues.
|
|
38
|
+
- Language-aware code-rot support, current checked-out package state: normalized language/file-role inventory, a source-facts model and collector, a TypeScript/JavaScript analyzer, and a Python analyzer. Source facts feed the `dead-code-candidate`, `duplicate-implementation-candidate`, and `test-rot` detectors with parsed-import/export/declaration evidence for both languages. Python project metadata (presence of `pyproject.toml`, `requirements.txt`, `setup.py`/`setup.cfg`, `tox.ini`, `pytest.ini`, and a best-effort project name) is collected and reported as its own audit report field. Java and Kotlin remain inventory-classified but fallback-only; no analyzer is registered for them yet.
|
|
39
|
+
- Security-validation audit adapter, current checked-out package state: `npm run audit -- --types security` runs security validation through the shared audit/report surface by calling the same `runSecurityValidation()` internals `security:validate` uses, maps security findings into audit-issue-shaped entries, and adds a `securitySummary` field (verdict, check counts, finding counts, and links to the original `reports/security/*.txt`/`*.json` report) to audit JSON/text reports. `npm run audit -- --types code-rot,security` runs both audit types together. The original `security:validate` command and its `reports/security/` output are unchanged.
|
|
36
40
|
|
|
37
41
|
---
|
|
38
42
|
|
|
@@ -203,7 +207,14 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
|
|
|
203
207
|
|
|
204
208
|
## Current baseline release positioning
|
|
205
209
|
|
|
206
|
-
my-dev-kit-lab is at a working baseline. The raw-vs-indexed experiment pipeline is fully implemented and produces reproducible artifacts. Real-agent campaign support exists for Codex and Claude. The
|
|
210
|
+
my-dev-kit-lab is at a working baseline. The raw-vs-indexed experiment pipeline is fully implemented and produces reproducible artifacts. Real-agent campaign support exists for Codex and Claude. The current published npm baseline is `v0.3.1`, which includes the generic audit framework, the code-rot detector family, and the language-aware code-rot substrate with TypeScript/JavaScript support, on top of the earlier experiment and automated security-validation work.
|
|
211
|
+
|
|
212
|
+
Planned roadmap direction after the published baseline:
|
|
213
|
+
|
|
214
|
+
- `v0.3.2`: Python code-rot support plus a first security-validation audit adapter, release-prepared and not yet published
|
|
215
|
+
- `v0.3.3` through `v0.3.4`: Java/Kotlin and cross-language stability for the language-aware code-rot track
|
|
216
|
+
- `v0.4.0` through `v0.4.2`: Android automated security validation, including an Android-specific extension of the security audit adapter
|
|
217
|
+
- manual pentest: post-v1 / version TBD
|
|
207
218
|
|
|
208
219
|
---
|
|
209
220
|
|
|
@@ -211,6 +222,8 @@ my-dev-kit-lab is at a working baseline. The raw-vs-indexed experiment pipeline
|
|
|
211
222
|
|
|
212
223
|
my-dev-kit-lab owns a release-security validation track for **my-dev-kit**. This work is separate from the experiment pipeline and does not replace the generic experiment-plugin roadmap. Its purpose is to generate release-validation evidence for the local CLI/package before release preparation.
|
|
213
224
|
|
|
225
|
+
`security:validate` remains the standalone, focused command for this validation and is unaffected by the current checked-out state. `npm run audit -- --types security` additionally adapts the same `security:validate` internals into the shared audit/report surface (mapped issues, a `securitySummary` field, and links to the original `reports/security/` output) — it does not replace or duplicate `security:validate`'s own checks or reports.
|
|
226
|
+
|
|
214
227
|
This is not a web application pentest framework. **my-dev-kit** is a local CLI/package, so the validation model is CLI/package adversarial testing focused on whether it remains:
|
|
215
228
|
|
|
216
229
|
- local-first
|
|
@@ -241,7 +254,7 @@ Each security command can validate my-dev-kit-lab itself or another local projec
|
|
|
241
254
|
|
|
242
255
|
`security:validate` supports `--checks`, `--profile`, `--format`, `--fail-on`, `--out`, and `--report-prefix`. The no-flag path remains backward compatible and still runs the classic implemented check groups: `deps`, `package`, `static`, `cli-adversarial`, and `fuzz`. Supplying `--profile` without `--checks` swaps in that profile's default checks; supplying explicit `--checks` always wins over profile defaults.
|
|
243
256
|
|
|
244
|
-
The current attack-scenario checks cover boundary, subprocess, secrets, and network assumptions. They are automated adversarial checks, not a manual pentest. `verdictImpact` metadata from each registered scenario drives blocker categorization in report reasoning, and narrowed `--checks` runs are labeled as scoped rather than described as a full release gate.
|
|
257
|
+
The current attack-scenario checks cover boundary, subprocess, secrets, and network assumptions. They are automated adversarial checks, not a manual pentest. Manual pentest is deferred until after `v1.0.0`. `verdictImpact` metadata from each registered scenario drives blocker categorization in report reasoning, and narrowed `--checks` runs are labeled as scoped rather than described as a full release gate.
|
|
245
258
|
|
|
246
259
|
JSON report poisoning/config-injection checks use a baseline-diff schema guard so legitimate additive JSON fields do not fail the guard while payload-created trusted top-level fields still do. Text reports are sanitized to strip ANSI/control-byte payloads before rendering.
|
|
247
260
|
|
|
@@ -18,7 +18,7 @@ import { writeAuditReports } from "../../src/audits/report/writeAuditReports.js"
|
|
|
18
18
|
// renderAuditJsonReport.ts, renderAuditTextReport.ts, writeAuditReports.ts.
|
|
19
19
|
// Mirrors the structure of scripts/security/validate.ts.
|
|
20
20
|
// ---------------------------------------------------------------------------
|
|
21
|
-
const USAGE = "Usage: npm run audit -- [--target <path>] [--types code-rot] " +
|
|
21
|
+
const USAGE = "Usage: npm run audit -- [--target <path>] [--types code-rot,security] " +
|
|
22
22
|
"[--include docs,tests,package,architecture,cli] [--format text,json] " +
|
|
23
23
|
"[--fail-on blocker|high|medium|low|none] [--out <path>]";
|
|
24
24
|
const rawArgs = process.argv.slice(2);
|
|
@@ -80,12 +80,19 @@ if (model.summary.detectorErrorCount > 0) {
|
|
|
80
80
|
console.log(` [${e.id}] ${e.message}`);
|
|
81
81
|
}
|
|
82
82
|
}
|
|
83
|
-
if (model.summary.noDetectorsRegistered) {
|
|
83
|
+
if (model.summary.noDetectorsRegistered && !model.securitySummary.ran) {
|
|
84
84
|
console.log(`Note: no code-rot detectors are registered for this run's --types/--include selection -- this run collected inventory/source-of-truth data only.`);
|
|
85
85
|
}
|
|
86
|
+
if (model.securitySummary.ran) {
|
|
87
|
+
console.log(`\nSecurity validation: verdict=${model.securitySummary.verdictLabel} checks=${model.securitySummary.totalChecks} findings(blocker=${model.securitySummary.findingCounts.blocker} major=${model.securitySummary.findingCounts.major} minor=${model.securitySummary.findingCounts.minor} info=${model.securitySummary.findingCounts.informational})`);
|
|
88
|
+
if (model.securitySummary.reportPaths.text) {
|
|
89
|
+
console.log(` Full report: ${model.securitySummary.reportPaths.text}`);
|
|
90
|
+
}
|
|
91
|
+
}
|
|
86
92
|
console.log(`\nInventory: ${model.inventory.totalScannedFileCount} file(s) scanned, ${model.inventory.skippedFileCount} skipped`);
|
|
87
93
|
console.log(` source=${model.inventory.filesByCategory.source} tests=${model.inventory.filesByCategory.tests} docs=${model.inventory.filesByCategory.docs} package=${model.inventory.filesByCategory.package} config=${model.inventory.filesByCategory.config} scripts=${model.inventory.filesByCategory.scripts} ci=${model.inventory.filesByCategory.ci}`);
|
|
88
94
|
console.log(`Source of truth: package=${model.sourceOfTruth.packageName ?? "(none)"}@${model.sourceOfTruth.packageVersion ?? "?"} readme=${model.sourceOfTruth.hasReadme} changelog=${model.sourceOfTruth.hasChangelog} ciWorkflows=${model.sourceOfTruth.ciWorkflowCount}`);
|
|
95
|
+
console.log(`Source facts: ${model.sourceFacts.totalFilesAnalyzed} file(s) analyzed (parsed=${model.sourceFacts.filesByParseStatus.parsed} file-level-only=${model.sourceFacts.filesByParseStatus["file-level-only"]} unsupported=${model.sourceFacts.filesByParseStatus.unsupported})`);
|
|
89
96
|
if (writtenPaths.length > 0) {
|
|
90
97
|
console.log(`\nReports written:`);
|
|
91
98
|
for (const p of writtenPaths) {
|
|
@@ -4,6 +4,7 @@ import fs from "node:fs";
|
|
|
4
4
|
import { runSecurityValidation } from "../../src/securityValidation/validate/runSecurityValidation.js";
|
|
5
5
|
import { resolveValidationTarget, reportFilenamePrefix } from "../../src/securityValidation/validate/resolveTarget.js";
|
|
6
6
|
import { renderTextReport, renderJsonReport } from "../../src/securityValidation/report/renderSecurityReport.js";
|
|
7
|
+
import { buildSecurityReportFromSummary } from "../../src/securityValidation/report/buildSecurityReport.js";
|
|
7
8
|
import { parseSecurityValidateArgs, normalizeSecurityValidateConfig, applyProfileDefaultChecksIfApplicable, } from "../../src/securityValidation/validate/cliOptions.js";
|
|
8
9
|
import { resolveAttackProfile } from "../../src/securityValidation/attackScenarios/attackProfile.js";
|
|
9
10
|
import { findingsBreachFailOnThreshold } from "../../src/securityValidation/validate/verdict.js";
|
|
@@ -81,35 +82,13 @@ const summary = await runSecurityValidation({
|
|
|
81
82
|
// downgrade an existing blocker.
|
|
82
83
|
const failOnBreached = findingsBreachFailOnThreshold(summary.findings, config.failOnThreshold);
|
|
83
84
|
// Build report object
|
|
84
|
-
const report = {
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
packageName: summary.packageName,
|
|
92
|
-
packageVersion: summary.packageVersion,
|
|
93
|
-
branch: summary.auditedBranch,
|
|
94
|
-
commit: summary.auditedCommit,
|
|
95
|
-
isSelf: summary.isSelf,
|
|
96
|
-
generatedAt: summary.finishedAt,
|
|
97
|
-
totalDurationMs: new Date(summary.finishedAt).getTime() - new Date(summary.startedAt).getTime(),
|
|
98
|
-
profile: config.profile,
|
|
99
|
-
selectedChecks: config.checks,
|
|
100
|
-
failOnThreshold: config.failOnThreshold,
|
|
101
|
-
formats: config.formats,
|
|
102
|
-
failOnBreached,
|
|
103
|
-
isFullReleaseGate: summary.isFullReleaseGate,
|
|
104
|
-
},
|
|
105
|
-
sections: [],
|
|
106
|
-
allChecks: summary.checks,
|
|
107
|
-
allFindings: summary.findings,
|
|
108
|
-
verdict: summary.verdict,
|
|
109
|
-
recommendedNextStep: summary.recommendedNextStep,
|
|
110
|
-
attackResults: summary.attackResults,
|
|
111
|
-
verdictReasonSummary: summary.verdictReasonSummary,
|
|
112
|
-
};
|
|
85
|
+
const report = buildSecurityReportFromSummary(summary, {
|
|
86
|
+
profile: config.profile,
|
|
87
|
+
selectedChecks: config.checks,
|
|
88
|
+
failOnThreshold: config.failOnThreshold,
|
|
89
|
+
formats: config.formats,
|
|
90
|
+
failOnBreached,
|
|
91
|
+
});
|
|
113
92
|
const textReport = renderTextReport(report);
|
|
114
93
|
const jsonReport = renderJsonReport(report);
|
|
115
94
|
// Determine output directory
|
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
import path from "node:path";
|
|
2
2
|
import { readBoundedFileText } from "../utils/boundedRead.js";
|
|
3
|
-
import { baseNameNoExt } from "../utils/filePatternUtils.js";
|
|
3
|
+
import { baseNameNoExt, isGenericOrTestPrefixedDeclarationName } from "../utils/filePatternUtils.js";
|
|
4
4
|
import { deduplicateIssuesById, makeCodeRotIssue } from "../utils/issueFactories.js";
|
|
5
|
+
import { collectRelativeImportBasenames, getParsedSourceFacts, indexSourceFactsByPath } from "../utils/sourceFactsLookup.js";
|
|
5
6
|
// ---------------------------------------------------------------------------
|
|
6
7
|
// v0.3.0 Batch 4 — dead-code candidate detector.
|
|
7
8
|
//
|
|
@@ -49,6 +50,9 @@ export const DEAD_CODE_CANDIDATE_DETECTOR = {
|
|
|
49
50
|
if (ctx.config.include.includes("tests")) {
|
|
50
51
|
issues.push(...findUnreferencedFixtures(ctx));
|
|
51
52
|
}
|
|
53
|
+
if (ctx.config.include.includes("cli") || ctx.config.include.includes("architecture")) {
|
|
54
|
+
issues.push(...findPossiblyUnreferencedPythonDeclarations(ctx));
|
|
55
|
+
}
|
|
52
56
|
return deduplicateIssuesById(issues);
|
|
53
57
|
},
|
|
54
58
|
};
|
|
@@ -140,7 +144,13 @@ function findUnreferencedScriptFiles(ctx, pkg) {
|
|
|
140
144
|
return issues;
|
|
141
145
|
}
|
|
142
146
|
function buildReverseReferenceIndex(ctx) {
|
|
143
|
-
|
|
147
|
+
// v0.3.1 Batch 4 -- seeded with basenames the TypeScript/JavaScript
|
|
148
|
+
// analyzer already resolved from real import declarations (parsed files
|
|
149
|
+
// only). This only ever adds references, never removes them, so it can
|
|
150
|
+
// only reduce false positives relative to the regex-only scan below --
|
|
151
|
+
// safe even though ctx.sourceFacts is optional (returns an empty set when
|
|
152
|
+
// absent, e.g. in existing detector-only unit tests).
|
|
153
|
+
const referencedBasenames = collectRelativeImportBasenames(ctx.sourceFacts);
|
|
144
154
|
const allCandidateFiles = [
|
|
145
155
|
...ctx.inventory.sourceFiles,
|
|
146
156
|
...ctx.inventory.testFiles,
|
|
@@ -164,6 +174,7 @@ function buildReverseReferenceIndex(ctx) {
|
|
|
164
174
|
function findUnreferencedSourceFiles(ctx, pkg, docsConcat) {
|
|
165
175
|
const referencedBasenames = buildReverseReferenceIndex(ctx);
|
|
166
176
|
const binCandidates = pkg ? binTargetsAsSourceCandidates(pkg) : new Set();
|
|
177
|
+
const sourceFactsIndex = indexSourceFactsByPath(ctx.sourceFacts);
|
|
167
178
|
const issues = [];
|
|
168
179
|
for (const file of ctx.inventory.sourceFiles) {
|
|
169
180
|
if (file.sizeBytes > MAX_READ_BYTES)
|
|
@@ -187,6 +198,26 @@ function findUnreferencedSourceFiles(ctx, pkg, docsConcat) {
|
|
|
187
198
|
continue;
|
|
188
199
|
if (referencedBasenames.has(base.toLowerCase()))
|
|
189
200
|
continue;
|
|
201
|
+
const evidence = [
|
|
202
|
+
{
|
|
203
|
+
kind: "observation",
|
|
204
|
+
message: "No relative import/require specifier in the scanned source/test/script set resolves to this file's basename.",
|
|
205
|
+
filePath: file.relativePath,
|
|
206
|
+
source: DETECTOR_ID,
|
|
207
|
+
confidence: "low",
|
|
208
|
+
},
|
|
209
|
+
];
|
|
210
|
+
// v0.3.1 Batch 4 -- when the TypeScript/JavaScript analyzer actually
|
|
211
|
+
// parsed this file, surface its export count as extra candidate
|
|
212
|
+
// evidence (an exported symbol with zero detected external references
|
|
213
|
+
// is more suspicious than an unexported one). This never changes
|
|
214
|
+
// whether the issue is emitted, only its evidence -- the flagging
|
|
215
|
+
// decision above is unchanged from the existing basename-reference
|
|
216
|
+
// heuristic.
|
|
217
|
+
const facts = getParsedSourceFacts(sourceFactsIndex, file.relativePath);
|
|
218
|
+
if (facts) {
|
|
219
|
+
evidence.push(buildSourceFactsEvidence(facts));
|
|
220
|
+
}
|
|
190
221
|
issues.push(makeCodeRotIssue({
|
|
191
222
|
auditType: "code-rot",
|
|
192
223
|
detectorId: DETECTOR_ID,
|
|
@@ -203,20 +234,116 @@ function findUnreferencedSourceFiles(ctx, pkg, docsConcat) {
|
|
|
203
234
|
releaseBlocking: false,
|
|
204
235
|
implementationBlocking: false,
|
|
205
236
|
autoFixEligible: false,
|
|
206
|
-
evidence
|
|
207
|
-
{
|
|
208
|
-
kind: "observation",
|
|
209
|
-
message: "No relative import/require specifier in the scanned source/test/script set resolves to this file's basename.",
|
|
210
|
-
filePath: file.relativePath,
|
|
211
|
-
source: DETECTOR_ID,
|
|
212
|
-
confidence: "low",
|
|
213
|
-
},
|
|
214
|
-
],
|
|
237
|
+
evidence,
|
|
215
238
|
affectedFiles: [file.relativePath],
|
|
216
239
|
}));
|
|
217
240
|
}
|
|
218
241
|
return issues;
|
|
219
242
|
}
|
|
243
|
+
// v0.3.1 Batch 4 -- describes a parsed file's source-facts shape as an
|
|
244
|
+
// evidence entry. Only called for parseStatus "parsed" facts (see callers),
|
|
245
|
+
// since file-level-only/parse-error facts carry empty fact arrays that
|
|
246
|
+
// would misleadingly read as "zero exports" rather than "unknown".
|
|
247
|
+
function buildSourceFactsEvidence(facts) {
|
|
248
|
+
return {
|
|
249
|
+
kind: "reference",
|
|
250
|
+
message: `Source facts: the TypeScript/JavaScript analyzer parsed this file and recorded ${facts.exports.length} export(s), ${facts.declarations.length} declaration(s), and ${facts.imports.length} import(s).`,
|
|
251
|
+
filePath: facts.relativePath,
|
|
252
|
+
source: DETECTOR_ID,
|
|
253
|
+
confidence: "medium",
|
|
254
|
+
};
|
|
255
|
+
}
|
|
256
|
+
// v0.3.2 Batch 2 -- conservative, declaration-level Python dead-code
|
|
257
|
+
// candidate check. TS/JS module-level "unused exported symbol" detection is
|
|
258
|
+
// explicitly SKIPPED in this file's own header comment (no safe/cheap way
|
|
259
|
+
// without a real type-checker) -- this Python-only check is narrower and
|
|
260
|
+
// safer than that skipped TS/JS idea would be, because Python source facts
|
|
261
|
+
// already carry declarations *and* a repo-wide cross-file "imported name"
|
|
262
|
+
// index (unlike anything comparable for TS/JS bare specifiers), so the
|
|
263
|
+
// false-positive surface is much smaller. Every one of the skip conditions
|
|
264
|
+
// below is a deliberate, spec-required conservatism -- see the requirements
|
|
265
|
+
// list in this batch's task description. Findings stay at the lowest
|
|
266
|
+
// severity/confidence tier in this detector's own schema (matches
|
|
267
|
+
// findUnreferencedFixtures()'s tier) and use "candidate"/"may indicate"
|
|
268
|
+
// wording throughout, never "unused" or "dead".
|
|
269
|
+
function findPossiblyUnreferencedPythonDeclarations(ctx) {
|
|
270
|
+
if (!ctx.sourceFacts)
|
|
271
|
+
return [];
|
|
272
|
+
const index = indexSourceFactsByPath(ctx.sourceFacts);
|
|
273
|
+
// Cross-file "this name was imported somewhere" signal -- a declaration
|
|
274
|
+
// named here is referenced by at least one `from module import name`
|
|
275
|
+
// elsewhere in the scanned Python file set, which is the strongest signal
|
|
276
|
+
// this conservative scan can offer (Python source facts carry no
|
|
277
|
+
// call/identifier reference tracking -- see pythonAnalyzer.ts's header
|
|
278
|
+
// comment on why `references` stays empty by design).
|
|
279
|
+
const importedNamesAcrossProject = new Set();
|
|
280
|
+
for (const facts of index.values()) {
|
|
281
|
+
if (facts.language !== "python" || facts.parseStatus !== "parsed")
|
|
282
|
+
continue;
|
|
283
|
+
for (const imp of facts.imports) {
|
|
284
|
+
for (const name of imp.importedNames ?? [])
|
|
285
|
+
importedNamesAcrossProject.add(name);
|
|
286
|
+
}
|
|
287
|
+
}
|
|
288
|
+
const issues = [];
|
|
289
|
+
for (const facts of index.values()) {
|
|
290
|
+
if (facts.language !== "python" || facts.role !== "source" || facts.parseStatus !== "parsed")
|
|
291
|
+
continue;
|
|
292
|
+
// __init__.py commonly exists only to re-export a package's public
|
|
293
|
+
// surface or to mark a directory as a package -- its declarations (if
|
|
294
|
+
// any) are conservatively excluded entirely rather than guessed at.
|
|
295
|
+
if (path.basename(facts.relativePath) === "__init__.py")
|
|
296
|
+
continue;
|
|
297
|
+
for (const decl of facts.declarations) {
|
|
298
|
+
// Methods are excluded entirely -- this detector has no existing
|
|
299
|
+
// method-level dead-code evidence model for any language (TS/JS
|
|
300
|
+
// methods are never flagged either), and Python method visibility is
|
|
301
|
+
// even harder to infer safely (inherited/overridden/dispatched via
|
|
302
|
+
// duck typing).
|
|
303
|
+
if (decl.kind === "method")
|
|
304
|
+
continue;
|
|
305
|
+
if (!decl.exported)
|
|
306
|
+
continue; // leading-underscore convention already applied by pythonAnalyzer.ts
|
|
307
|
+
if (isGenericOrTestPrefixedDeclarationName(decl.name))
|
|
308
|
+
continue; // main/run/setup/teardown/__init__/__call__/test_*
|
|
309
|
+
// A name explicitly listed in a real `__all__` is the file's own
|
|
310
|
+
// strongest "this is public API" declaration -- skip rather than
|
|
311
|
+
// second-guess it.
|
|
312
|
+
if (facts.exports.some((e) => e.name === decl.name))
|
|
313
|
+
continue;
|
|
314
|
+
if (importedNamesAcrossProject.has(decl.name))
|
|
315
|
+
continue;
|
|
316
|
+
issues.push(makeCodeRotIssue({
|
|
317
|
+
auditType: "code-rot",
|
|
318
|
+
detectorId: DETECTOR_ID,
|
|
319
|
+
idCues: ["unreferenced-python-declaration", facts.relativePath, decl.kind, decl.name],
|
|
320
|
+
title: `Python ${decl.kind} "${decl.name}" in "${facts.relativePath}" has no detected cross-file import (weak signal)`,
|
|
321
|
+
description: `"${decl.name}" (${decl.kind}) is declared in "${facts.relativePath}" and does not appear as an imported name in any other scanned Python file. This is a deterministic source-facts signal from a conservative, non-semantic scan -- it does not resolve dynamic imports, framework/plugin discovery, string-based dispatch, or usage from outside the scanned project, so it may indicate dead code but is not proof. Confirm manually before removing.`,
|
|
322
|
+
severity: "info",
|
|
323
|
+
confidence: "low",
|
|
324
|
+
falsePositiveRisk: "high",
|
|
325
|
+
category: "dead-code-candidate",
|
|
326
|
+
recommendedAction: "Confirm the declaration is genuinely unused before removing it.",
|
|
327
|
+
suggestedFixStrategy: `Remove "${decl.name}" from "${facts.relativePath}" if confirmed unused, or reference it from another module.`,
|
|
328
|
+
validationCommands: ["npm run audit -- --types code-rot --include cli"],
|
|
329
|
+
releaseBlocking: false,
|
|
330
|
+
implementationBlocking: false,
|
|
331
|
+
autoFixEligible: false,
|
|
332
|
+
evidence: [
|
|
333
|
+
{
|
|
334
|
+
kind: "reference",
|
|
335
|
+
message: `Source facts: the Python analyzer parsed this file and found no other scanned Python file importing the name "${decl.name}". Static-analysis limitation -- confidence reduced because this scan cannot see dynamic imports or usage outside the scanned file set.`,
|
|
336
|
+
filePath: facts.relativePath,
|
|
337
|
+
source: DETECTOR_ID,
|
|
338
|
+
confidence: "low",
|
|
339
|
+
},
|
|
340
|
+
],
|
|
341
|
+
affectedFiles: [facts.relativePath],
|
|
342
|
+
}));
|
|
343
|
+
}
|
|
344
|
+
}
|
|
345
|
+
return issues;
|
|
346
|
+
}
|
|
220
347
|
function findOldOrDeprecatedFilesNotInDocs(ctx, docsConcat) {
|
|
221
348
|
const issues = [];
|
|
222
349
|
for (const file of ctx.inventory.files) {
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { readBoundedFileText } from "../utils/boundedRead.js";
|
|
2
|
-
import { firstPathSegment, GENERIC_INFRA_BASENAMES, baseNameNoExt } from "../utils/filePatternUtils.js";
|
|
2
|
+
import { firstPathSegment, GENERIC_INFRA_BASENAMES, baseNameNoExt, isGenericOrTestPrefixedDeclarationName, } from "../utils/filePatternUtils.js";
|
|
3
3
|
import { deduplicateIssuesById, makeCodeRotIssue } from "../utils/issueFactories.js";
|
|
4
|
+
import { indexSourceFactsByPath } from "../utils/sourceFactsLookup.js";
|
|
4
5
|
// ---------------------------------------------------------------------------
|
|
5
6
|
// v0.3.0 Batch 4 — duplicate/parallel implementation candidate detector.
|
|
6
7
|
//
|
|
@@ -60,6 +61,7 @@ export const DUPLICATE_IMPLEMENTATION_DETECTOR = {
|
|
|
60
61
|
if (ctx.config.include.includes("architecture")) {
|
|
61
62
|
issues.push(...findParallelDirectories(ctx.inventory.files.map((f) => f.relativePath)));
|
|
62
63
|
issues.push(...findDuplicateBasenamesAcrossRoots(ctx.inventory.sourceFiles.map((f) => f.relativePath)));
|
|
64
|
+
issues.push(...findDuplicateDeclarationCandidates(ctx));
|
|
63
65
|
}
|
|
64
66
|
issues.push(...findDuplicateDetectorRegistryIds(ctx));
|
|
65
67
|
return deduplicateIssuesById(issues);
|
|
@@ -214,6 +216,96 @@ function firstTopLevelUnderSrc(relativePath) {
|
|
|
214
216
|
return null;
|
|
215
217
|
return `${parts[0]}/${parts[1]}`;
|
|
216
218
|
}
|
|
219
|
+
// v0.3.1 Batch 4 -- source-facts-aware duplicate-declaration candidate
|
|
220
|
+
// check. Weak/low-confidence signal, same spirit as
|
|
221
|
+
// findDuplicateBasenamesAcrossRoots(): the same exported declaration name
|
|
222
|
+
// and kind (e.g. two exported `class Logger`) appears in 2-4 distinct
|
|
223
|
+
// "source"-role files that a registered analyzer actually parsed. Bounded
|
|
224
|
+
// to declaration kinds that carry real implementation weight
|
|
225
|
+
// (function/class/interface/type/enum) -- "variable"/"constant"/"method"/
|
|
226
|
+
// "unknown" are excluded as too noisy (methods repeat across unrelated
|
|
227
|
+
// classes by convention; module-scope variables/constants repeat even more
|
|
228
|
+
// often). Generic/lifecycle declaration names (run, init, handler, __init__,
|
|
229
|
+
// test_*, ...) are excluded via isGenericOrTestPrefixedDeclarationName() for
|
|
230
|
+
// the same reason GENERIC_INFRA_BASENAMES excludes generic file basenames
|
|
231
|
+
// above. Requires ctx.sourceFacts -- returns no issues when it is absent
|
|
232
|
+
// (e.g. existing detector unit tests that build a literal
|
|
233
|
+
// AuditDetectorContext without it), so this check is purely additive.
|
|
234
|
+
const NOISY_DECLARATION_KINDS = new Set(["variable", "constant", "method", "unknown"]);
|
|
235
|
+
function findDuplicateDeclarationCandidates(ctx) {
|
|
236
|
+
if (!ctx.sourceFacts)
|
|
237
|
+
return [];
|
|
238
|
+
const index = indexSourceFactsByPath(ctx.sourceFacts);
|
|
239
|
+
const byKey = new Map();
|
|
240
|
+
for (const facts of index.values()) {
|
|
241
|
+
if (facts.role !== "source")
|
|
242
|
+
continue;
|
|
243
|
+
if (facts.parseStatus !== "parsed")
|
|
244
|
+
continue;
|
|
245
|
+
for (const decl of facts.declarations) {
|
|
246
|
+
if (!decl.exported)
|
|
247
|
+
continue;
|
|
248
|
+
if (NOISY_DECLARATION_KINDS.has(decl.kind))
|
|
249
|
+
continue;
|
|
250
|
+
const nameLower = decl.name.toLowerCase();
|
|
251
|
+
if (isGenericOrTestPrefixedDeclarationName(nameLower))
|
|
252
|
+
continue;
|
|
253
|
+
// v0.3.2 Batch 2 -- the key is scoped by `facts.analyzerId` (not
|
|
254
|
+
// `facts.language`) so a Python declaration is only ever compared
|
|
255
|
+
// against other Python declarations, never against a same-named
|
|
256
|
+
// TypeScript/JavaScript one. `analyzerId`, not `language`, is the
|
|
257
|
+
// right scope here: TypeScript and JavaScript are two different
|
|
258
|
+
// NormalizedLanguage values but share one analyzer
|
|
259
|
+
// (typescript-javascript-analyzer) and were already compared together
|
|
260
|
+
// before this batch -- scoping by `language` instead would have
|
|
261
|
+
// silently split that existing, still-desired TS/JS pairing in two
|
|
262
|
+
// (each side then below the "2+ files" threshold, so both would be
|
|
263
|
+
// dropped entirely), which is exactly the "preserve existing
|
|
264
|
+
// TypeScript/JavaScript duplicate behavior" regression this batch
|
|
265
|
+
// must not introduce. Scoping by `analyzerId` fixes the real
|
|
266
|
+
// cross-language bug (Python vs TS/JS) without touching TS/JS's own
|
|
267
|
+
// existing behavior at all.
|
|
268
|
+
const key = `${facts.analyzerId}:${decl.kind}:${nameLower}`;
|
|
269
|
+
const entry = byKey.get(key) ?? { name: decl.name, kind: decl.kind, paths: new Set() };
|
|
270
|
+
entry.paths.add(facts.relativePath);
|
|
271
|
+
byKey.set(key, entry);
|
|
272
|
+
}
|
|
273
|
+
}
|
|
274
|
+
const issues = [];
|
|
275
|
+
for (const entry of byKey.values()) {
|
|
276
|
+
if (entry.paths.size < 2 || entry.paths.size > 4)
|
|
277
|
+
continue;
|
|
278
|
+
const sortedPaths = [...entry.paths].sort((a, b) => a.localeCompare(b));
|
|
279
|
+
issues.push(makeCodeRotIssue({
|
|
280
|
+
auditType: "code-rot",
|
|
281
|
+
detectorId: DETECTOR_ID,
|
|
282
|
+
idCues: ["duplicate-declaration-candidate", entry.kind, entry.name.toLowerCase(), ...sortedPaths],
|
|
283
|
+
title: `Exported ${entry.kind} "${entry.name}" is declared in ${sortedPaths.length} unrelated files (weak signal)`,
|
|
284
|
+
description: `An exported ${entry.kind} named "${entry.name}" is declared in: ${sortedPaths.join(", ")}. This is a source-facts-derived candidate signal (same declaration name and kind, not a semantic-equivalence check) for parallel implementations -- inspect manually before consolidating.`,
|
|
285
|
+
severity: "info",
|
|
286
|
+
confidence: "low",
|
|
287
|
+
falsePositiveRisk: "high",
|
|
288
|
+
category: "duplicate-implementation-candidate",
|
|
289
|
+
recommendedAction: "Manually confirm whether these declarations serve genuinely different purposes before taking any action.",
|
|
290
|
+
suggestedFixStrategy: "No automatic action -- rename or consolidate only after manual review.",
|
|
291
|
+
validationCommands: ["npm run audit -- --types code-rot --include architecture"],
|
|
292
|
+
releaseBlocking: false,
|
|
293
|
+
implementationBlocking: false,
|
|
294
|
+
autoFixEligible: false,
|
|
295
|
+
evidence: [
|
|
296
|
+
{
|
|
297
|
+
kind: "reference",
|
|
298
|
+
message: `Source facts: an exported ${entry.kind} named "${entry.name}" was parsed in ${sortedPaths.length} distinct files.`,
|
|
299
|
+
excerpt: sortedPaths.join(", "),
|
|
300
|
+
source: DETECTOR_ID,
|
|
301
|
+
confidence: "low",
|
|
302
|
+
},
|
|
303
|
+
],
|
|
304
|
+
affectedFiles: sortedPaths,
|
|
305
|
+
}));
|
|
306
|
+
}
|
|
307
|
+
return issues;
|
|
308
|
+
}
|
|
217
309
|
// Cheap regex scan (not a semantic parse) of already-small detector source
|
|
218
310
|
// files for exported `id: "..."` string literals. Flags duplicates only if
|
|
219
311
|
// genuinely found via this file-based check -- per the spec's own allowed
|