tewip-ai 0.7.0 → 0.7.1

package/README.md

@@ -169,9 +169,10 @@ own metric is still open.
 ## Security: which findings actually matter (Phase 9, new)
 
 ```bash
-tewip secure                 # secrets + npm audit / pip-audit / osv-scanner, triaged
-tewip secure --secrets-only  # a fast check before you commit
-tewip secure --gate          # fails a build only on what needs a person now
+tewip secure                              # secrets + npm audit / pip-audit / osv-scanner, triaged
+tewip secure --url http://localhost:3000  # also check the running app
+tewip secure --zap-report zap.json        # and triage an OWASP ZAP baseline report
+tewip secure --gate                       # fails a build only on what needs a person now
 ```
 
 Scanners are good at finding things and bad at saying which of them matter. Tewip runs the ones your
@@ -181,8 +182,16 @@ from a README), or **accepted** (a person already judged it, in a file only peop
 verdict carries the evidence. No model is called, so it costs nothing, and **a credential is
 located, never printed**.
 
+With `--url` it also checks the running app the way a browser would: security headers, cookie
+flags, and the files that must never be served (.env, .git). No payloads, no fuzzing, no login
+bypass - and **an address that is not your machine is refused** unless you say `--i-am-authorised`.
+For the deep scan it prints the OWASP ZAP command and triages ZAP's report rather than pretending
+to be a scanner.
+
 Measured on the gitleaks project's own labelled samples, for the credential kinds Tewip covers:
-**50 of 54 real credentials found**. Four defects that corpus exposed are fixed; the four still
+**50 of 54 real credentials found**. The web check found **8 of 8 planted weaknesses** in two local
+apps and reported **nothing** about the carefully served one; on four real public homepages its
+report matched the headers those sites actually send, every time. Four defects that corpus exposed are fixed; the four still
 missed are things that are not credentials (an unsigned token, a Stripe *test* key). It never claims
 a project is secure - it prints what it checked and what it did not.
 

package/TEAM-GUIDE.md

@@ -121,7 +121,7 @@ collects environment failures in one issue, and (optionally) opens a fix pull re
 | `tewip write` | Turns the plan's scenarios into real tests in your own framework, and keeps only the ones that pass repeatedly **and** fail when their check is broken on purpose | When you know what to test but not how to write it |
 | `tewip plan --issue PROJ-123` | Plans from the Jira ticket itself: its description and acceptance criteria | "Here's the story, what should we test?" |
 | `tewip jira plan.json --project PROJ` | Files the plan's scenarios as test-case tickets, linked to the story | Handing work to the team |
-| `tewip secure` | Runs your dependency and secret scanners, then says which findings need someone now and which are noise or unreachable | Before a release, or in CI |
+| `tewip secure [--url http://localhost:3000]` | Runs your dependency and secret scanners, checks the running app's headers, cookies and exposed files, then says which findings need someone now and which are noise or unreachable | Before a release, or in CI |
 | `tewip spec "<feature>"` | An epic, its user stories and acceptance criteria, grounded in your code; files them in Jira with `--project PROJ --create` | Before the work is broken down |
 | `tewip report report.xml --gate` | The CI command for GitLab, Jenkins, CircleCI, Azure, Bitbucket and the rest: diagnoses the run, comments on the GitLab merge request, and fails the build only on failures that need a person | Your CI is not GitHub Actions |
 | `tewip watch --cmd "npm test"` | Runs your tests when you save a file and explains each failure straight away | While you are writing code |

package/dist/cli/tewip.js

@@ -38,11 +38,16 @@ Usage:
                                     File the plan's scenarios as test-case issues, linked to the
                                     story. Shows what it would create; --create actually files
                                     them, and the same scenario is never filed twice.
-  tewip secure [--secrets-only] [--gate] [--out security.md]
+  tewip secure [--url http://localhost:3000] [--secrets-only] [--gate] [--out security.md]
                                     Runs the scanners this project already has (npm audit,
                                     pip-audit, osv-scanner) plus Tewip's own secret scan, then says
                                     which findings need someone now and which are noise or not
                                     reachable. No model, no cost, and it never prints a credential.
+      --url <page>                  Also check the running app: security headers, cookie flags, and
+                                    files that must never be served (.env, .git). Ordinary requests
+                                    only. An address that is not this machine needs
+                                    --i-am-authorised.
+      --zap-report zap.json         Triage the findings of an OWASP ZAP baseline scan as well.
   tewip spec "<feature>" [--diff main...HEAD] [--url <page> ...] [--project PROJ] [--create]
                                     An epic, its user stories and their acceptance criteria, from
                                     this project's code, tests and failure history. Anything it
@@ -163,6 +168,9 @@ async function main() {
                 out: flag(args, 'out'),
                 secretsOnly: args.includes('--secrets-only'),
                 gate: args.includes('--gate'),
+                url: flag(args, 'url'),
+                authorised: args.includes('--i-am-authorised'),
+                zapReport: flag(args, 'zap-report'),
             });
             // The gate fails a build only for what Tewip decided a person must act on now.
             if (args.includes('--gate') && items.some((i) => i.verdict === 'real-risk'))

package/dist/engine/secure/secure-command.js

@@ -7,7 +7,9 @@ import { mkdirSync, writeFileSync } from 'node:fs';
 import { dirname, join, resolve } from 'node:path';
 import { auditDependencies } from './deps.js';
 import { scanSecrets, sourceFiles } from './secrets.js';
-import { readAccepted, triageDependency, triageSecret, worstFirst } from './triage.js';
+import { readAccepted, triageDependency, triageSecret, triageWeb, worstFirst } from './triage.js';
+import { isLocal, scanWeb } from './web.js';
+import { readZapReport, zapAvailable, zapInstructions } from './zap.js';
 const CODE = /\.(?:[cm]?[jt]sx?|py|rb|go|java|kt|cs|php|rs|vue|svelte)$/;
 export function renderSecurity(items, checked, notChecked, root) {
     const real = items.filter((i) => i.verdict === 'real-risk');
@@ -65,8 +67,36 @@ export async function runSecure(cmd) {
     else {
         notChecked.push('dependencies (--secrets-only was asked for)');
     }
-    // A web scan is a different kind of check and needs a running app; it is not pretended here.
-    notChecked.push('a running application (no web scan is part of this command yet)');
+    if (cmd.url) {
+        // Checking someone else's site is their decision, not Tewip's. A local address is the person's
+        // own machine; anything else has to be said out loud.
+        if (!isLocal(cmd.url) && !cmd.authorised) {
+            throw new Error([
+                `${cmd.url} is not this machine. Tewip only checks a running site when you say you are allowed to:`,
+                `  tewip secure --url ${cmd.url} --i-am-authorised`,
+                'It sends ordinary requests - no payloads, no fuzzing, no attempt to get past a login - but',
+                'scanning a system you do not run is not Tewip\'s decision to make.',
+            ].join('\n'));
+        }
+        const scan = await scanWeb(cmd.url);
+        checked.push(`${cmd.url} (${scan.requests} request(s): headers, cookies, files that must not be served)`);
+        if (scan.note)
+            notChecked.push(scan.note);
+        for (const finding of scan.findings)
+            items.push(triageWeb(finding, accepted));
+    }
+    else {
+        notChecked.push('a running application (pass --url http://localhost:3000 to check one)');
+    }
+    if (cmd.zapReport) {
+        const fromZap = readZapReport(resolve(cmd.zapReport));
+        checked.push(`an OWASP ZAP report (${fromZap.length} finding(s))`);
+        for (const finding of fromZap)
+            items.push(triageWeb(finding, accepted));
+    }
+    else if (cmd.url) {
+        notChecked.push(`a full web scan. Tewip's own check is passive; for the deep one: ${zapInstructions(cmd.url, zapAvailable(root))}`);
+    }
     items.sort(worstFirst);
     const report = renderSecurity(items, checked, notChecked, root);
     console.log(report);

package/dist/engine/secure/triage.js

@@ -128,6 +128,26 @@ export function triageDependency(finding, root, sources, accepted) {
     }
     return { ...base, verdict: 'real-risk', why: `this project imports ${finding.package} directly, and the advisory applies to the installed version`, next: upgrade };
 }
+/**
+ * A finding about a running page. There is no "not reachable" here: the check was made against the
+ * running system, so what it saw is what a visitor sees. A person can still accept one, and a page
+ * that is only ever served inside a network is judged by severity, not by pretending it is safe.
+ */
+export function triageWeb(finding, accepted) {
+    const where = `${finding.url} (${finding.kind})`;
+    const base = { kind: 'web', title: finding.what, severity: finding.severity, where, source: finding.kind.startsWith('zap-') ? 'OWASP ZAP' : 'tewip baseline web check' };
+    if (accepted.has(`web ${finding.url} ${finding.kind}`) || accepted.has(`web ${finding.kind}`)) {
+        return { ...base, verdict: 'accepted', why: 'a person marked this one accepted in .tewip/security-accepted.txt', next: 'nothing: it was reviewed' };
+    }
+    // Everything here was observed in a real response, so it is real; only how much it matters varies.
+    const verdict = finding.severity === 'low' ? 'not-reachable' : 'real-risk';
+    return {
+        ...base,
+        verdict,
+        why: verdict === 'real-risk' ? finding.evidence : `${finding.evidence} - worth fixing, but on its own it is not a way in`,
+        next: finding.next,
+    };
+}
 const ORDER = { 'real-risk': 0, 'not-reachable': 1, noise: 2, accepted: 3 };
 const SEV = { critical: 0, high: 1, medium: 2, low: 3 };
 export const worstFirst = (a, b) => ORDER[a.verdict] - ORDER[b.verdict] || SEV[a.severity] - SEV[b.severity];

package/dist/engine/secure/web.js

@@ -0,0 +1,180 @@
+// Phase 9: the baseline check of a running application.
+//
+// What this is: the safe half of a web scan. Tewip asks for pages the way a browser does, reads
+// what comes back, and reports what is verifiably missing or exposed - a cookie without Secure, a
+// missing Content-Security-Policy, a .env file that answers with its contents. Every finding is a
+// fact about a response, quotable in the report.
+//
+// What this is NOT, on purpose: no injection payloads, no fuzzing, no brute force, no attempt to
+// get past a login, no scanning of hosts that were not named. Those need authorisation, a scope and
+// a person - and for teams that have those, OWASP ZAP already exists, so Tewip reads ZAP's report
+// instead of pretending to be it (see zap.ts).
+import { setTimeout as delay } from 'node:timers/promises';
+/** A header check: what must be there, and what it protects against. */
+const HEADERS = [
+    {
+        name: 'content-security-policy',
+        kind: 'missing-csp',
+        what: 'no Content-Security-Policy, so any injected script the page ends up with will run',
+        severity: 'high',
+        next: "add a Content-Security-Policy. Start in report-only mode with default-src 'self' and tighten it from the reports.",
+    },
+    {
+        name: 'strict-transport-security',
+        kind: 'missing-hsts',
+        what: 'no Strict-Transport-Security, so a first visit over http can be intercepted',
+        severity: 'medium',
+        next: 'add Strict-Transport-Security: max-age=31536000; includeSubDomains once every subdomain is on https.',
+        httpsOnly: true,
+    },
+    {
+        name: 'x-content-type-options',
+        kind: 'missing-nosniff',
+        what: 'no X-Content-Type-Options, so a browser may guess a file is script when it is not',
+        severity: 'medium',
+        next: 'add X-Content-Type-Options: nosniff.',
+    },
+    {
+        name: 'referrer-policy',
+        kind: 'missing-referrer-policy',
+        what: 'no Referrer-Policy, so full URLs (with any ids in them) are sent to other sites',
+        severity: 'low',
+        next: 'add Referrer-Policy: strict-origin-when-cross-origin.',
+    },
+];
+/** Headers that tell an attacker what to attack. */
+const TELLS = ['server', 'x-powered-by', 'x-aspnet-version', 'x-generator'];
+const VERSIONED = /\d+\.\d+/;
+/** Paths that must never answer with their contents. Each is a plain GET, nothing more. */
+const EXPOSED = [
+    { path: '/.env', kind: 'exposed-env', what: 'the .env file is served, so its settings and keys are public', severity: 'critical', looksReal: /^[A-Z][A-Z0-9_]*=/m },
+    { path: '/.git/config', kind: 'exposed-git', what: 'the .git folder is served, so the whole source history can be downloaded', severity: 'critical', looksReal: /\[core\]|\[remote/ },
+    { path: '/.aws/credentials', kind: 'exposed-aws', what: 'AWS credentials are served', severity: 'critical', looksReal: /aws_access_key_id/i },
+    { path: '/config.json', kind: 'exposed-config', what: 'a configuration file is served', severity: 'medium', looksReal: /"(?:password|secret|token|apiKey)"\s*:/i },
+];
+const ok = (response) => response.status >= 200 && response.status < 300;
+/** One request, with a short timeout, never following more than a couple of redirects. */
+async function get(url, timeoutMs) {
+    try {
+        const controller = new AbortController();
+        const timer = globalThis.setTimeout(() => controller.abort(), timeoutMs);
+        const res = await fetch(url, { redirect: 'follow', signal: controller.signal, headers: { 'User-Agent': 'tewip-secure/1.0 (baseline check)' } });
+        globalThis.clearTimeout(timer);
+        const body = (await res.text()).slice(0, 4000);
+        return { status: res.status, headers: res.headers, body };
+    }
+    catch {
+        return null;
+    }
+}
+/** Cookie names that usually carry a session or an identity. */
+const SESSION_COOKIE = /sess|auth|token|login|sid$|^sid|jwt|remember|csrf|identity|account/i;
+/** Cookies a page sets, and what protects them. */
+function cookieFindings(headers, url) {
+    const raw = typeof headers.getSetCookie === 'function' ? headers.getSetCookie() : [headers.get('set-cookie')].filter(Boolean);
+    const findings = [];
+    for (const cookie of raw) {
+        const name = cookie.split('=')[0].trim();
+        const flags = cookie.toLowerCase();
+        const missing = [];
+        if (!flags.includes('httponly'))
+            missing.push('HttpOnly');
+        if (!flags.includes('secure') && url.startsWith('https'))
+            missing.push('Secure');
+        if (!flags.includes('samesite'))
+            missing.push('SameSite');
+        if (missing.length === 0)
+            continue;
+        // A cookie that carries a session is worth someone's evening; an analytics cookie without
+        // HttpOnly is a deliberate choice on plenty of real sites, and calling it "high" is how a
+        // security report gets ignored. The name is the only honest signal available here.
+        const session = SESSION_COOKIE.test(name);
+        findings.push({
+            kind: 'weak-cookie',
+            what: `the cookie "${name}" is set without ${missing.join(', ')}`,
+            severity: session && missing.includes('HttpOnly') ? 'high' : 'medium',
+            evidence: `Set-Cookie: ${cookie.slice(0, 120)}`,
+            url,
+            next: session
+                ? `set ${missing.join(', ')} on ${name}. HttpOnly keeps script from reading it; SameSite=Lax stops it riding along on other sites' requests.`
+                : `set ${missing.join(', ')} on ${name} if it carries anything private. If it is only analytics, script may need to read it, and this is a decision rather than a fault.`,
+        });
+    }
+    return findings;
+}
+/**
+ * The baseline check of one page. Nothing is sent that a browser would not send, and at most a
+ * dozen requests are made.
+ */
+export async function scanWeb(url, { timeoutMs = 15_000, probeExposed = true, pauseMs = 150 } = {}) {
+    const findings = [];
+    let requests = 0;
+    const page = await get(url, timeoutMs);
+    requests++;
+    if (!page)
+        return { findings, requests, note: `${url} did not answer, so nothing about it was checked.` };
+    const https = url.startsWith('https://');
+    for (const header of HEADERS) {
+        if (header.httpsOnly && !https)
+            continue;
+        if (page.headers.get(header.name))
+            continue;
+        findings.push({ kind: header.kind, what: header.what, severity: header.severity, evidence: `the response to ${url} has no ${header.name} header`, url, next: header.next });
+    }
+    // Clickjacking: either header will do, so it is one finding, not two.
+    const csp = page.headers.get('content-security-policy') ?? '';
+    if (!page.headers.get('x-frame-options') && !/frame-ancestors/i.test(csp)) {
+        findings.push({
+            kind: 'framable',
+            what: 'the page can be put in a frame on another site, which is how clickjacking works',
+            severity: 'medium',
+            evidence: `no X-Frame-Options and no frame-ancestors in the policy for ${url}`,
+            url,
+            next: "add X-Frame-Options: DENY, or frame-ancestors 'none' to the Content-Security-Policy.",
+        });
+    }
+    for (const name of TELLS) {
+        const value = page.headers.get(name);
+        if (value && VERSIONED.test(value)) {
+            findings.push({
+                kind: 'version-header',
+                what: `the ${name} header names the exact software version, which tells an attacker which holes to try`,
+                severity: 'low',
+                evidence: `${name}: ${value}`,
+                url,
+                next: `stop sending ${name}, or remove the version from it.`,
+            });
+        }
+    }
+    findings.push(...cookieFindings(page.headers, url));
+    if (probeExposed) {
+        const base = new URL(url);
+        for (const target of EXPOSED) {
+            await delay(pauseMs);
+            const at = new URL(target.path, base.origin).href;
+            const res = await get(at, timeoutMs);
+            requests++;
+            if (!res || !ok(res) || !target.looksReal.test(res.body))
+                continue;
+            findings.push({
+                kind: target.kind,
+                what: target.what,
+                severity: target.severity,
+                evidence: `GET ${at} answered ${res.status} with the file's real contents`,
+                url: at,
+                next: `stop serving ${target.path}: it belongs outside the web root, and anything in it must be treated as leaked.`,
+            });
+        }
+    }
+    return { findings, requests, note: null };
+}
+/** Localhost, a private address, or a name only this machine resolves. */
+export function isLocal(url) {
+    try {
+        const host = new URL(url).hostname;
+        return /^(?:localhost|127\.|0\.0\.0\.0|\[::1\]|::1|10\.|192\.168\.|172\.(?:1[6-9]|2\d|3[01])\.)/.test(host) || host.endsWith('.local') || host.endsWith('.localhost');
+    }
+    catch {
+        return false;
+    }
+}

package/dist/engine/secure/zap.js

@@ -0,0 +1,56 @@
+// Phase 9: OWASP ZAP. For teams that already run a real web scanner, Tewip does not compete with
+// it - it reads its report and triages the findings, the same way it does for npm audit.
+//
+// ZAP's baseline scan is the one teams run in CI: it crawls passively and reports what it saw. Its
+// JSON report is read here; Tewip does not start a scan on a host nobody named.
+import { existsSync, readFileSync } from 'node:fs';
+import { spawnSync } from 'node:child_process';
+/** ZAP's risk names, as its report writes them. */
+const RISK = { High: 'high', Medium: 'medium', Low: 'low', Informational: 'low' };
+/** Findings from a ZAP JSON report, in Tewip's own shape. */
+export function readZapReport(path) {
+    if (!existsSync(path))
+        return [];
+    let report;
+    try {
+        report = JSON.parse(readFileSync(path, 'utf8'));
+    }
+    catch {
+        return [];
+    }
+    const alerts = (report.site ?? []).flatMap((site) => site.alerts ?? []);
+    const strip = (html) => html.replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim();
+    return alerts.map((alert) => {
+        const risk = (alert.riskdesc ?? '').split(' ')[0];
+        const first = (alert.instances ?? [])[0] ?? {};
+        return {
+            kind: `zap-${(alert.alert ?? alert.name ?? 'finding').toLowerCase().replace(/[^a-z0-9]+/g, '-').slice(0, 40)}`,
+            what: strip(alert.alert ?? alert.name ?? 'a finding from ZAP').slice(0, 200),
+            severity: RISK[risk] ?? 'low',
+            evidence: `${first.uri ?? ''}${first.evidence ? ` - ${strip(first.evidence).slice(0, 120)}` : ''}`.trim() || strip(alert.desc ?? '').slice(0, 200),
+            url: first.uri ?? '',
+            next: strip(alert.solution ?? '').slice(0, 300) || 'see the ZAP report for the recommended fix',
+        };
+    });
+}
+/** Whether a ZAP baseline scan can be run from here at all. */
+export function zapAvailable(cwd) {
+    const has = (command) => spawnSync(command, { cwd, shell: true, encoding: 'utf8', timeout: 30_000 }).status === 0;
+    if (has('zap-baseline.py --help'))
+        return 'script';
+    if (has('docker --version'))
+        return 'docker';
+    return null;
+}
+/**
+ * How to run ZAP against this address, for the person to copy. Tewip prints it rather than running
+ * it: a scan of a running system is the person's decision, and its report is read afterwards with
+ * `--zap-report`.
+ */
+export function zapInstructions(url, how) {
+    if (how === 'script')
+        return `zap-baseline.py -t ${url} -J zap.json   # then: tewip secure --zap-report zap.json`;
+    if (how === 'docker')
+        return `docker run --rm -v "$PWD:/zap/wrk" -t ghcr.io/zaproxy/zaproxy zap-baseline.py -t ${url} -J zap.json   # then: tewip secure --zap-report zap.json`;
+    return `OWASP ZAP is not installed here. With Docker: docker run --rm -v "$PWD:/zap/wrk" -t ghcr.io/zaproxy/zaproxy zap-baseline.py -t ${url} -J zap.json`;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tewip-ai",
-  "version": "0.7.0",
+  "version": "0.7.1",
   "description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
   "keywords": [
     "test-failures",