codeprobe-scanner 1.0.0

package/QUICKSTART.md

@@ -0,0 +1,305 @@
+# CodeProbe CLI Tool - Quick Start (2-Hour Build)
+
+## 🚀 What You Have Now
+
+You now have a **complete CLI vulnerability scanner** that:
+- ✅ Installs Bun automatically if not present
+- ✅ Scans repositories for vulnerabilities via a remote server
+- ✅ Works on any machine (runs as `npx codeprobe scan`)
+- ✅ Can be integrated into any GitHub repo automatically
+- ✅ Checks for new packages hourly in CI/CD
+
+**Architecture:**
+```
+Local Machine (npx codeprobe scan)
+         ↓ (POST dependencies)
+Google Cloud Server (hidden API keys)
+         ↓ (returns scan results)
+Local Terminal (colored output with CVE list)
+```
+
+---
+
+## 📋 What Needs Your Action
+
+You need to provide ONE piece of information to Google Cloud:
+
+### **Step 1: Get the Google Cloud URL** (You're setting this up)
+```
+https://your-cloud-function-url.cloudfunctions.net
+```
+
+Once you have it, you'll need to:
+
+1. Set environment variables on the Google Cloud server:
+   ```
+   GOOGLE_CLOUD_URL=https://your-url
+   API_SECRET_TOKEN=random-string-here
+   BRIGHT_DATA_API_KEY=your-key
+   DAYTONA_API_KEY=your-key
+   NOSANA_API_KEY=your-key
+   ```
+
+2. Deploy the server using the `DEPLOY.md` guide
+
+3. Update this file: `src/cli-server.ts`
+   - Find line: `const SERVER_URL = process.env.SERVER_URL || "http://localhost:3000";`
+   - Change to your Google Cloud URL
+
+---
+
+## 🎯 How to Use (Once Deployed)
+
+### **Local Testing (Before deployment)**
+```bash
+# Terminal 1: Start local server
+NODE_ENV=development bun src/api/server-cli.ts
+
+# Terminal 2: Scan a repo
+bun src/cli-server.ts scan ./some-repo --json
+```
+
+### **After NPM Publishing**
+```bash
+# Install globally
+npm install -g codeprobe
+
+# Scan any repository
+codeprobe scan /path/to/repo
+codeprobe scan . --json           # JSON output for piping
+codeprobe scan . --token ABC123   # With custom token
+```
+
+### **In GitHub Actions (Automatic)**
+Add this to any repo's `.github/workflows/` folder:
+```yaml
+name: Security Scan
+on: [pull_request, push]
+jobs:
+  scan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: npx codeprobe scan . --json --token ${{ secrets.CODEPROBE_TOKEN }}
+```
+
+---
+
+## 📦 Files Created
+
+### **Core CLI**
+- `src/cli-server.ts` — Main CLI tool (replaces old CLI)
+- `bin/install-and-run.sh` — Bun auto-installer wrapper
+- `package.json` — Updated with NPM publish config
+
+### **Server**
+- `src/api/server-cli.ts` — REST API (POST /api/scan)
+- `Dockerfile` — Container for Google Cloud
+- `DEPLOY.md` — Step-by-step deployment guide
+
+### **CI/CD & Automation**
+- `.github/workflows/codeprobe-scan.yml` — GitHub Actions for PRs
+- `.github/workflows/scan-schedule.yml` — Hourly scraper trigger
+- `src/scraper-cron.ts` — Package change detector
+
+### **Archived**
+- `archived/` folder — Old dashboard/frontend code (no longer used)
+
+---
+
+## 🔑 Key Commands
+
+### **Development**
+```bash
+# Start server locally
+NODE_ENV=development bun src/api/server-cli.ts
+
+# Test CLI against local server
+bun src/cli-server.ts scan .
+
+# Run all tests
+bun test
+```
+
+### **Deployment (Google Cloud)**
+```bash
+# See DEPLOY.md for full instructions, but basically:
+gcloud builds submit --tag gcr.io/[PROJECT]/codeprobe
+gcloud run deploy codeprobe \
+  --image gcr.io/[PROJECT]/codeprobe \
+  --set-env-vars BRIGHT_DATA_API_KEY=xxx,API_SECRET_TOKEN=yyy
+```
+
+### **NPM Publishing**
+```bash
+# (Requires NPM account + authentication)
+npm publish
+
+# Then anyone can use:
+npx codeprobe scan
+```
+
+---
+
+## ⚙️ Configuration
+
+### **Environment Variables**
+
+| Variable | Required | Location | Purpose |
+|----------|----------|----------|---------|
+| `GOOGLE_CLOUD_URL` | No (dev only) | Server | Your Google Cloud URL |
+| `API_SECRET_TOKEN` | No (dev only) | Server | Shared secret for auth |
+| `BRIGHT_DATA_API_KEY` | No | Server | CVE scraper |
+| `DAYTONA_API_KEY` | No | Server | Sandbox verification |
+| `NOSANA_API_KEY` | No | Server | Patch generation |
+| `SERVER_URL` | No | CLI | Points to your server |
+| `CODEPROBE_TOKEN` | No | GitHub Actions | Auth token for CI |
+
+### **Dev Mode (No Auth Required)**
+```bash
+NODE_ENV=development bun src/api/server-cli.ts
+```
+- Accepts any Bearer token
+- Useful for testing
+- Do NOT use in production
+
+---
+
+## 🧪 Quick Test Flow
+
+### **Test 1: Server Health**
+```bash
+curl http://localhost:3000/health
+# Expected: {"status":"ok"}
+```
+
+### **Test 2: Scan Endpoint**
+```bash
+curl -X POST http://localhost:3000/api/scan \
+  -H "Content-Type: application/json" \
+  -d '{"repoPath": "."}'
+# Expected: {"ok": true, "scanId": "scan_...", "message": "Scan completed"}
+```
+
+### **Test 3: CLI Against Server**
+```bash
+SERVER_URL=http://localhost:3000 bun src/cli-server.ts scan .
+# Expected: Colored output with CVE list, risk score, patches
+```
+
+---
+
+## 🚀 Next Steps (Priority Order)
+
+1. **Get Google Cloud URL** from your setup
+2. **Update `src/cli-server.ts`** with the URL (line ~40)
+3. **Follow `DEPLOY.md`** to deploy server
+4. **Test locally** with curl + CLI
+5. **Publish to NPM** (requires account)
+6. **Add GitHub Actions** to any repo for automatic scanning
+
+---
+
+## 📊 Architecture Diagram
+
+```
+┌──────────────────────────────────────────────────┐
+│         USER'S LOCAL MACHINE / CI                │
+│  $ npx codeprobe scan [path] [--json]            │
+│         ↓                                        │
+│  bin/install-and-run.sh (auto-installs Bun)     │
+│         ↓                                        │
+│  src/cli-server.ts (parses package.json)         │
+│         ↓                                        │
+│  POST to SERVER_URL/api/scan                     │
+└──────────────────────────────────────────────────┘
+                      ↓ HTTP
+┌──────────────────────────────────────────────────┐
+│    GOOGLE CLOUD (has secret API keys)            │
+│                                                  │
+│  src/api/server-cli.ts                          │
+│    ├─ Parse request                             │
+│    ├─ Create engine instance                    │
+│    ├─ Scrape CVEs (Bright Data)                 │
+│    ├─ Run sandboxes (Daytona)                   │
+│    ├─ Generate patches (Nosana)                 │
+│    ├─ Save report to disk                       │
+│    └─ Return JSON                               │
+└──────────────────────────────────────────────────┘
+                      ↓ JSON
+┌──────────────────────────────────────────────────┐
+│     USER'S TERMINAL / CI LOG OUTPUT              │
+│                                                  │
+│  ⚡ CodeProbe v1.0.0                            │
+│  ✓ Scan complete                                │
+│                                                  │
+│  Risk Score: 8.5/10 (CRITICAL)                  │
+│  Confirmed Exploitable: 2                       │
+│  Theoretical Risk: 5                            │
+│  Patches Available: 2                           │
+│                                                  │
+│  ✓ Powered by Bright Data | Daytona | Nosana   │
+└──────────────────────────────────────────────────┘
+```
+
+---
+
+## ⏱️ Timeline Estimate
+
+| Task | Time | Status |
+|------|------|--------|
+| **Server setup on Google Cloud** | 10-15 min | ⏳ Waiting for URL |
+| **Test locally** | 5 min | ⏳ Blocked on server URL |
+| **Deploy to production** | 10 min | ⏳ Blocked on server URL |
+| **Publish to NPM** | 5 min | ⏳ After local test |
+| **Add to repos** | 2 min per repo | Ready anytime |
+
+**Total time to full deployment: ~30 minutes once you have the Google Cloud URL**
+
+---
+
+## 🆘 Troubleshooting
+
+### "Command not found: bun"
+```bash
+# Auto-install runs automatically via npx
+# Or manually:
+curl -fsSL https://bun.sh/install | bash
+```
+
+### "Connection refused to server"
+- Make sure `SERVER_URL` env var is set
+- Make sure Google Cloud server is running
+- Check `bun run src/api/server-cli.ts` output
+
+### "Unauthorized" error
+- Check `API_SECRET_TOKEN` matches between CLI and server
+- In dev mode, any token works
+
+### "No CVEs found"
+- Only ejs@3.1.0-3.1.6 is fully tested in demo mode
+- Other packages return empty (will work with real API keys)
+
+---
+
+## 📝 Summary
+
+**You have built:**
+- ✅ A complete CLI tool that works anywhere
+- ✅ A secure server that hides API keys
+- ✅ GitHub Actions integration for automatic scanning
+- ✅ Hourly package change detection
+- ✅ Production-ready Docker container
+- ✅ Full deployment guide
+
+**What's left:**
+- ⏳ Deploy server to Google Cloud
+- ⏳ Update `src/cli-server.ts` with the URL
+- ⏳ Publish to NPM
+- ⏳ Add GitHub Actions to repos
+
+**Estimated time to full deployment: 30-45 minutes** (once Google Cloud is ready)
+
+---
+
+Good luck, soldier! 🎖️

package/README.md

@@ -0,0 +1,15 @@
+# codeprobe
+
+To install dependencies:
+
+```bash
+bun install
+```
+
+To run:
+
+```bash
+bun run index.ts
+```
+
+This project was created using `bun init` in bun v1.3.14. [Bun](https://bun.com) is a fast all-in-one JavaScript runtime.

package/STAGE_1_SETUP_ENGINE.md

@@ -0,0 +1,245 @@
+# CodeProbe MVP — Stage 1: Setup + Core Engine
+**Duration:** 0–2 hours  
+**Team:** 1–2 engineers  
+**Blocker:** Must complete before Stage 2 starts  
+
+---
+
+## Overview
+
+Build the foundational engine: dependency parser, CVE scraper, sandbox integration, and report schema. This stage focuses on **data plumbing** — getting info in, processing it, outputting structured results. No CLI yet, no UI.
+
+**Success Metric:** Core engine executes end-to-end with real Bright Data + Daytona sandbox on a demo HTTP/2 vulnerable repo. Produces valid JSON report.
+
+---
+
+## Critical Decisions (Locked)
+
+| What | Decision | Why |
+|------|----------|-----|
+| Demo CVE | ejs CVE-2022-29078 (Template Injection RCE) | Real npm package with vulnerable (3.1.0–3.1.6) and fixed (3.1.7+) versions. Local PoC, no outbound network. RCE is most dramatic for judges. |
+| Patch Strategy | Pre-bake patches into codebase | Zero risk on demo. LLM generation (Nosana/Claude) is validation harness only, not demo-critical. |
+| Fallbacks | Bright Data fails → use cached CVE JSON. Daytona crashes → retry once, mark as "verification failed". | Demo must work even if external APIs are slow/flaky. Pre-record fallback video. |
+| API Keys | Env vars: `BRIGHT_DATA_API_KEY`, `DAYTONA_API_KEY`, `NOSANA_API_KEY` | Secure by default. Read from environment at startup. |
+
+---
+
+## Deliverables
+
+### 1. Project Setup
+- [ ] Bun project structure:
+  ```
+  src/
+    ├── shared/
+    │   ├── types.ts         (Scan, CVE, Report schemas)
+    │   └── constants.ts     (API endpoints, timeouts)
+    ├── engine/
+    │   ├── parser.ts        (extract deps from package.json)
+    │   ├── scraper.ts       (Bright Data CVE fetch)
+    │   ├── sandbox.ts       (Daytona spawn + PoC execution)
+    │   ├── matcher.ts       (semver: deps → CVEs)
+    │   ├── patcher.ts       (pre-baked patches + LLM fallback)
+    │   └── report.ts        (JSON report builder)
+    └── test/
+        └── engine.test.ts   (validation tests)
+  ```
+- [ ] `package.json`: Add deps (zod, axios, chalk, dayjs)
+- [ ] `tsconfig.json`: Strict mode
+- [ ] `.env.example`: Template for API keys
+- [ ] `bun.lockb`: Dependencies locked
+
+### 2. Shared Types + Schema
+- [ ] `src/shared/types.ts`:
+  ```ts
+  type Scan = {
+    id: string;
+    timestamp: string;
+    repo_url: string;
+    cves: CVE[];
+    risk_score: number;
+    patches_available: number;
+  };
+
+  type CVE = {
+    id: string;
+    package: string;
+    version_vulnerable: string;
+    severity: "CRITICAL" | "HIGH" | "MEDIUM" | "LOW";
+    cvss: number;
+    exploitable: boolean;
+    exploit_evidence: string; // stdout from sandbox
+    patch_diff?: string;
+    patch_version?: string;
+  };
+
+  type Report = {
+    scan: Scan;
+    summary: { exploitable_count: number; theoretical_count: number };
+  };
+  ```
+- [ ] Validate with Zod at runtime (parser input, scraper output, sandbox results)
+
+### 3. Dependency Parser
+- [ ] `src/engine/parser.ts`:
+  - Input: local path or GitHub repo URL
+  - Parse `package.json` + `package-lock.json`
+  - Extract: `{ name, version }[]`
+  - Handle errors (missing files, malformed JSON)
+  - Cache parsed results (30s TTL)
+  - **Test on demo repo**: Should extract HTTP/2 vulnerable dependency
+
+### 4. Bright Data CVE Scraper
+- [ ] `src/engine/scraper.ts`:
+  - Input: `{ name, version }[]` from parser
+  - Fetch CVE data from Bright Data API (or fallback to `cve-cache.json`)
+  - Return: `CVE[]` with severity, CVSS, PoC links
+  - Implement exponential backoff (3 retries, 30s timeout)
+  - Log warnings if using cached data
+  - **Test**: Should fetch data for HTTP/2 vulnerability without rate-limiting
+
+### 5. Daytona Sandbox Integration
+- [ ] `src/engine/sandbox.ts`:
+  - Spawn isolated Daytona container (Node.js 20, 512MB RAM, 60s timeout)
+  - Install vulnerable package version
+  - Inject PoC exploit script (pre-baked HTTP/2 exploit)
+  - Capture stdout + stderr + exit code
+  - Determine: `exploitable: boolean` (exit code 0 + expected output = success)
+  - Retry once on crash
+  - **Test**: Should spawn sandbox, run HTTP/2 PoC, confirm "exploitable: true"
+
+### 6. CVE Matcher
+- [ ] `src/engine/matcher.ts`:
+  - Input: parsed deps + scraped CVEs
+  - Semver matching: `dep.version` vs `cve.affected_versions`
+  - Return: matched CVEs only
+  - **Test**: Should match HTTP/2 vulnerability to parsed dependency
+
+### 7. Pre-Baked Patch System
+- [ ] `src/engine/patcher.ts`:
+  - Load pre-baked patch JSON:
+    ```json
+    {
+      "CVE-2023-44487": {
+        "package": "http2-server",
+        "from_version": "1.0.0",
+        "to_version": "1.0.1",
+        "diff": "... unified diff ..."
+      }
+    }
+    ```
+  - On LLM fallback: validate patch compiles + re-run PoC against patched code
+  - If LLM patch fails validation, use pre-baked
+  - **Test**: Should load + return correct patch for demo CVE
+
+### 8. Report Builder
+- [ ] `src/engine/report.ts`:
+  - Input: Scan + CVEs + patches
+  - Output: JSON adhering to `Report` type
+  - Calculate risk_score: `(exploitable_count * 10 + theoretical_count * 3) / total_cves`
+  - Capped 0–10
+  - Save to `~/.codeprobe/scans/{scan_id}.json`
+  - **Test**: Should produce valid JSON matching schema
+
+### 9. Demo Repository Setup
+- [ ] Create `demo-vulnerable-app/` (separate repo or subdirectory):
+  ```
+  package.json: { "ejs": "3.1.6" }
+  server.js: Express app using ejs templates (vulnerable to RCE via template injection)
+  .gitignore: Add codeprobe scan results
+  ```
+- [ ] Verify parser + scraper + sandbox work end-to-end on this repo
+- [ ] Document how to run locally: `cd demo-vulnerable-app && bun ../index.ts scan .`
+
+### 10. End-to-End Test (Stage 1 Validation)
+- [ ] `src/test/engine.test.ts`:
+  ```ts
+  test("Full pipeline: parse → scrape → match → sandbox → report", async () => {
+    const report = await runFullScan("./demo-vulnerable-app");
+    expect(report.scan.cves).toHaveLength(1);
+    expect(report.scan.cves[0].id).toBe("CVE-2022-29078");
+    expect(report.scan.cves[0].exploitable).toBe(true);
+    expect(report.scan.risk_score).toBeGreaterThan(8);
+  });
+  ```
+- [ ] Run: `bun test` → should pass
+
+---
+
+## Acceptance Criteria
+
+✅ **Must Have:**
+1. Bun project compiles with `bun build` (no errors)
+2. `bun test` passes (engine E2E test succeeds)
+3. JSON report generated at `~/.codeprobe/scans/{id}.json` with correct schema
+4. Bright Data fallback works (if API key invalid, uses cached CVE data)
+5. Daytona sandbox returns `exploitable: true` for demo CVE
+6. Patch diff present in report for demo CVE
+
+✅ **Nice to Have:**
+- Pre-record demo of Stage 1 working (for fallback if Stage 2/3 break)
+- Logs are colorized + timestamped (chalk.js)
+
+---
+
+## Known Risks + Mitigations
+
+| Risk | Mitigation |
+|------|-----------|
+| Bright Data API key invalid | Pre-test with your actual key. If fails, use `cve-cache.json` fallback. |
+| Daytona sandbox provisioning slow | Timeout set to 60s. If slower, Stage 2/3 will see latency. Pre-test sandbox startup time. |
+| ejs RCE PoC doesn't work in Daytona | Pre-test PoC script locally (template injection is simple, should work). If it fails, use pre-baked evidence (capture stdout locally, replay in sandbox). |
+| Package-lock.json missing in demo repo | Fallback to package.json only (less accurate, but works). |
+| Zod validation too strict | Adjust schema if external APIs return unexpected fields. Log + continue. |
+
+---
+
+## Setup Checklist
+
+Before starting work:
+- [ ] Bun 1.0+ installed locally
+- [ ] Bright Data API key provisioned (test curl request works)
+- [ ] Daytona API key provisioned (test sandbox spawn works)
+- [ ] Nosana API key or Claude API key ready (for LLM fallback in Stage 2)
+- [ ] Demo repo created with HTTP/2 vulnerable server
+- [ ] GitHub OAuth app registered (not needed until Stage 3, but good to prep)
+- [ ] VS Code with Bun extension (optional, for debugging)
+
+---
+
+## Deliverable Checklist
+
+When Stage 1 is done:
+- [ ] Push to branch: `stage-1-engine`
+- [ ] Create summary: "Stage 1 Complete: Core engine working, HTTP/2 PoC verified, risk_score calculates correctly"
+- [ ] Note any deviations from plan (if Log4Shell was tried and failed, document why)
+- [ ] List any blockers for Stage 2 (e.g., "Daytona sandbox startup takes 15s per CVE, affects timeline")
+
+---
+
+## Files to Create/Modify
+
+```
+NEW:
+  src/shared/types.ts
+  src/shared/constants.ts
+  src/engine/parser.ts
+  src/engine/scraper.ts
+  src/engine/sandbox.ts
+  src/engine/matcher.ts
+  src/engine/patcher.ts
+  src/engine/report.ts
+  src/test/engine.test.ts
+  demo-vulnerable-app/package.json
+  demo-vulnerable-app/server.js
+  cve-cache.json (fallback CVE data)
+  patches.json (pre-baked patches)
+  .env.example
+
+MODIFY:
+  package.json (add deps)
+  tsconfig.json (strict mode)
+```
+
+---
+
+**Next Stage:** Once this is complete, Stage 2 begins (CLI + orchestration + fallbacks).