shipped-secrets 1.0.0 → 1.0.2

package/README.md

@@ -22,7 +22,7 @@ Secrets are always masked in the output. Nothing is sent anywhere unless you ask
 
 ## What it checks
 
-1. **Your real secret values.** It reads your `.env*` files (not `.env.example`), Cloudflare's `.dev.vars`, and in CI the environment variables whose names look secret. Then it searches every file the browser gets, source maps included, for each server value. This catches any secret, whatever it looks like, including your own tokens that no pattern knows.
+1. **Your real secret values.** It reads your `.env*` files (not `.env.example`), Cloudflare's `.dev.vars`, and in CI the environment variables whose names look secret. Then it searches every file the browser gets, source maps included, for each server value. This catches any secret, whatever it looks like, including your own tokens that no pattern knows. It also looks for each value the common ways it gets encoded on its way into a bundle: base64 (standard and URL-safe, with or without padding), hex, URL-encoded, and with `/` escaped as `\/` or `/` the way some JSON and RSC payloads write it. Each of these is an exact transform of your real value, so none of them can raise a false alarm.
 2. **Known key shapes**, for keys that are not in your local files: Stripe secret and restricted keys, Supabase service_role keys and `sb_secret_` keys, OpenAI, Anthropic, AWS, GitHub, SendGrid, Mailgun, Resend, Twilio, Slack, private keys, and database URLs with a password. Google API keys are a warning, because Maps and Firebase browser keys are meant to be public.
 3. **Supabase keys are decoded.** The anon key and the service_role key look the same. `shipped-secrets` reads the role inside the key, so the anon key is never reported and the service_role key always is.
 4. **Risky names.** A public prefix on a name that sounds secret (`VITE_STRIPE_SECRET`, `NEXT_PUBLIC_SUPABASE_SERVICE_ROLE_KEY`) is a warning. Names meant for browsers (anon key, publishable key, client ID, client token, site key) are not.
@@ -113,8 +113,18 @@ Checked: Stripe (`/v1/balance`), OpenAI and Anthropic (list models), GitHub (`/u
 
 `--all` is `--history` and `--verify` together.
 
+## A secret split into pieces
+
+```sh
+npx shipped-secrets --deep
+```
+
+A plain split like `"sk_live_" + "..."` is usually caught already, because minifiers join string pieces back together. `--deep` is for a split the minifier cannot undo, such as pieces in an array joined when the code runs. It cuts each `.env` value into pieces of 10 characters, keeps only the ones that look random (a digit, a letter, eight different characters, so never a common start like `sk_live_` or `https://`), and reports a file that holds two or more of them but never the whole value. It is off by default to keep the default run free of false alarms. The Vite plugin takes it too: `shippedSecrets({ deep: true })`.
+
 ## What it does not do
 
+- It does not find an encrypted secret, or one encoded in a way not listed above. If the browser can decrypt it, so can anyone, so it belongs on the server anyway.
+- `--deep` does not find a secret cut into pieces shorter than 10 characters, and it is not used on git history.
 - It does not fix anything. A shipped or committed secret must be rotated at the provider: deleting it from the code, or rewriting history, does not take it back from anyone who already has it.
 - `--verify` does not check AWS, Twilio, private keys or database URLs: checking those safely needs more than one harmless call.
 

package/bin/shipped-secrets.js

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-// shipped-secrets [folder or URL] [--dir <path>] [--allow NAME] [--history] [--verify] [--all] [--strict] [--json]
+// shipped-secrets [folder or URL] [--dir <path>] [--allow NAME] [--history] [--verify] [--deep] [--all] [--strict] [--json]
 //
 // With no argument: finds the browser output in this project (dist, .next/static,
 // build, out, ...), reads this project's .env files, and looks for your real
@@ -25,6 +25,7 @@ names that look secret, and source maps left public.
 
   --history       also search every commit in git history, and flag any .env file ever committed
   --verify        ask each found key's own provider whether it still works (sends the key to that provider only)
+  --deep          also look for a secret split into pieces (two or more long random pieces of it in one file)
   --all           --history and --verify
   --dir <path>    scan this folder instead of finding the build output (repeatable)
   --allow NAME    a variable that is fine to ship (repeatable). Or add "# shipped-ok" to its .env line
@@ -37,14 +38,15 @@ const target = positional[0] ?? '.'
 const allow = values('--allow')
 const history = flag('--history') || flag('--all')
 const verify = flag('--verify') || flag('--all')
+const deep = flag('--deep')
 let result
 if (/^https?:\/\//.test(target)) {
   const { secrets } = loadSecrets(process.cwd(), { env: process.env, allow })
-  result = await scanUrl(target, secrets)
+  result = await scanUrl(target, secrets, { deep })
 } else {
   const root = resolve(target)
   const dirs = values('--dir')
-  result = scanProject(root, { dirs: dirs.length ? dirs.map((d) => resolve(root, d)) : undefined, allow })
+  result = scanProject(root, { dirs: dirs.length ? dirs.map((d) => resolve(root, d)) : undefined, allow, deep })
   if (!dirs.length && !result.dirs.length && !history) {
     console.error('No build output found (dist, build, out, .next/static, ...). Build first, pass --dir <path>, or use --history.')
     process.exit(2)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "shipped-secrets",
-  "version": "1.0.0",
+  "version": "1.0.2",
   "description": "Find secrets that ship to the browser or sit in git history, and check whether they still work: your real .env values, known key shapes and risky public env names, in any framework's build output, source maps included.",
   "type": "module",
   "exports": {

package/src/history.js

@@ -7,7 +7,7 @@ import { spawn } from 'node:child_process'
 import { createInterface } from 'node:readline'
 import { findShapes, withKey } from './shapes.js'
 import { isSecretLine, mask, parseDotenv } from './env.js'
-import { keyOf } from './scan.js'
+import { forms, keyOf } from './scan.js'
 
 const ENV_FILE = /(^|\/)(\.env(\.[^/]*)?|\.dev\.vars)$/
 const SAFE_ENV = /\.(example|sample|template|dist|defaults)$/i
@@ -70,8 +70,10 @@ export function scanHistory(root, secrets = []) {
         return
       }
       for (const s of secrets) {
-        if (added.includes(s.value)) {
-          add(keyOf({ level: 'error', kind: 'history-value', file, line, commit: commit.slice(0, 8), date, what: `${s.name} was committed in ${file} (${commit.slice(0, 8)}, ${date})`, sample: mask(s.value) }, s.value))
+        for (const { text: form, how } of forms(s.value)) {
+          if (added.includes(form)) {
+            add(keyOf({ level: 'error', kind: 'history-value', file, line, commit: commit.slice(0, 8), date, what: `${s.name}${how ? ` (${how.slice(2)})` : ''} was committed in ${file} (${commit.slice(0, 8)}, ${date})`, sample: mask(s.value) }, s.value))
+          }
         }
       }
       for (const f of findShapes(added)) {

package/src/index.d.ts

@@ -1,6 +1,6 @@
 export type Finding = {
   level: 'error' | 'warn'
-  kind: 'value' | 'shape' | 'name' | 'sourcemap' | 'history-value' | 'history-shape' | 'history-env'
+  kind: 'value' | 'pieces' | 'shape' | 'name' | 'sourcemap' | 'history-value' | 'history-shape' | 'history-env'
   file: string
   line: number
   what: string
@@ -15,13 +15,13 @@ export type Finding = {
 export type Result = { findings: Finding[]; files: number; dirs?: string[]; secretsChecked?: number; commits?: number; verified?: boolean }
 export type Secret = { name: string; value: string; source: string; line: number; ok: boolean }
 
-export declare function scanProject(root: string, opts?: { dirs?: string[]; allow?: string[]; env?: Record<string, string | undefined>; envFiles?: string[] }): Result
-export declare function scanUrl(url: string, secrets?: Secret[]): Promise<Result>
+export declare function scanProject(root: string, opts?: { dirs?: string[]; allow?: string[]; env?: Record<string, string | undefined>; envFiles?: string[]; deep?: boolean }): Result
+export declare function scanUrl(url: string, secrets?: Secret[], opts?: { deep?: boolean }): Promise<Result>
 export declare function scanHistory(root: string, secrets?: Secret[]): Promise<{ findings: Finding[]; commits: number } | null>
 export declare function verifyKey(shapeId: string, key: string, fetch?: typeof globalThis.fetch): Promise<'live' | 'dead' | 'unknown'>
 export declare function verifyFindings(findings: Finding[], fetch?: typeof globalThis.fetch): Promise<Finding[]>
 export declare const VERIFIABLE: string[]
-export declare function scanText(text: string, secrets: Secret[]): Omit<Finding, 'file'>[]
+export declare function scanText(text: string, secrets: Secret[], opts?: { deep?: boolean }): Omit<Finding, 'file'>[]
 export declare function findOutputDirs(root: string): string[]
 export declare function loadSecrets(dir: string, opts?: { env?: Record<string, string | undefined>; files?: string[]; allow?: string[] }): { secrets: Secret[]; all: Secret[] }
 export declare function report(result: Result): string

package/src/scan.js

@@ -73,14 +73,75 @@ export function keyOf(finding, value) {
   return shape && shape.match === value ? withKey(Object.assign(finding, { shapeId: shape.shape.id }), value) : finding
 }
 
-export function scanText(text, secrets) {
+/**
+ * The value as written and the common ways it gets encoded on its way into a
+ * bundle: base64 (padding dropped, so it matches with or without the "="), hex,
+ * URL-encoded, and with "/" escaped the way some JSON and RSC payloads write it.
+ * Each is an exact transform of the real value, so none of them can raise a
+ * false alarm.
+ */
+export function forms(value) {
+  const out = [{ text: value, how: '' }]
+  const add = (text, how) => { if (text !== value && !out.some((f) => f.text === text)) out.push({ text, how }) }
+  const buf = Buffer.from(value, 'utf8')
+  const b64 = buf.toString('base64').replace(/=+$/, '')
+  add(b64, ', base64')
+  add(b64.replace(/\+/g, '-').replace(/\//g, '_'), ', base64url')
+  add(buf.toString('hex'), ', hex')
+  add(buf.toString('hex').toUpperCase(), ', hex')
+  add(encodeURIComponent(value), ', URL-encoded')
+  if (value.includes('/')) {
+    add(value.replace(/\//g, '\\/'), ', escaped')
+    add(value.replace(/\//g, '\\u002f'), ', escaped')
+  }
+  return out
+}
+
+const PIECE = 10
+
+/**
+ * For --deep: pieces of the value, to find a secret split up so that the whole
+ * of it never appears. Only from values long enough to have a random part, and
+ * only pieces that look random (a digit, a letter, eight different characters),
+ * so a common start like "sk_live_" or "https://" is never one of them.
+ */
+export function pieces(value) {
+  if (value.length < 24) return []
+  const out = []
+  for (let i = 0; i + PIECE <= value.length; i += PIECE) {
+    const p = value.slice(i, i + PIECE)
+    if (/\d/.test(p) && /[A-Za-z]/.test(p) && new Set(p).size >= 8) out.push(p)
+  }
+  return out
+}
+
+/**
+ * @param {string} text
+ * @param {{ name: string, value: string, source: string }[]} secrets
+ * @param {{ deep?: boolean }} [opts]
+ */
+export function scanText(text, secrets, opts = {}) {
   const found = []
   for (const s of secrets) {
-    let at = text.indexOf(s.value)
-    while (at !== -1) {
-      found.push(keyOf({ level: 'error', kind: 'value', what: `${s.name} from ${s.source.split(/[\\/]/).pop()}`, sample: mask(s.value), line: lineAt(text, at) }, s.value))
-      at = text.indexOf(s.value, at + s.value.length)
-      if (found.length > 50) break
+    const from = s.source.split(/[\\/]/).pop()
+    let whole = false
+    for (const { text: form, how } of forms(s.value)) {
+      let at = text.indexOf(form)
+      while (at !== -1) {
+        whole = true
+        found.push(keyOf({ level: 'error', kind: 'value', what: `${s.name} from ${from}${how}`, sample: mask(s.value), line: lineAt(text, at) }, s.value))
+        at = text.indexOf(form, at + form.length)
+        if (found.length > 50) break
+      }
+    }
+    // Two or more random pieces of one secret in the same file, and never the
+    // whole of it: split on purpose. A single piece is left alone.
+    if (opts.deep && !whole) {
+      const all = pieces(s.value)
+      const hits = all.map((p) => text.indexOf(p)).filter((at) => at !== -1)
+      if (hits.length >= 2) {
+        found.push(keyOf({ level: 'error', kind: 'pieces', what: `${s.name} from ${from}, in pieces (${hits.length} of ${all.length} found)`, sample: mask(s.value), line: lineAt(text, Math.min(...hits)) }, s.value))
+      }
     }
   }
   for (const f of findShapes(text)) {
@@ -95,7 +156,7 @@ export function scanText(text, secrets) {
  * Scans a project folder: its env files for risky names, and its browser output
  * for real secret values and known key shapes.
  * @param {string} root
- * @param {{ dirs?: string[], allow?: string[], env?: Record<string, string | undefined>, envFiles?: string[] }} [opts]
+ * @param {{ dirs?: string[], allow?: string[], env?: Record<string, string | undefined>, envFiles?: string[], deep?: boolean }} [opts]
  */
 export function scanProject(root, opts = {}) {
   const { secrets, all } = loadSecrets(root, { env: opts.env ?? process.env, files: opts.envFiles, allow: opts.allow })
@@ -116,7 +177,7 @@ export function scanProject(root, opts = {}) {
       files++
       if (file.endsWith('.map')) maps++
       const text = readFileSync(file, 'utf8')
-      for (const f of scanText(text, secrets)) findings.push(Object.assign(f, { file: relative(root, file) }))
+      for (const f of scanText(text, secrets, { deep: opts.deep })) findings.push(Object.assign(f, { file: relative(root, file) }))
     }
   }
   if (maps) {

package/src/url.js

@@ -1,67 +1,67 @@
-// Scans a live site the way a stranger would: the page, the scripts it loads,
-// the chunks those scripts load, and any source maps they point to.
-
-import { scanText } from './scan.js'
-
-const MAX_FILES = 200
-
-async function get(url) {
-  try {
-    const res = await fetch(url, { redirect: 'follow', headers: { 'user-agent': 'shipped-secrets' } })
-    if (!res.ok) return null
-    const type = res.headers.get('content-type') || ''
-    if (!/text|javascript|json|html|xml|svg/.test(type) && !/\.(m?js|map|json|css|html?)(\?|$)/.test(url)) return null
-    return await res.text()
-  } catch {
-    return null
-  }
-}
-
-function assetsIn(text, base, isHtml) {
-  const urls = new Set()
-  const add = (u) => {
-    try {
-      const abs = new URL(u, base)
-      if (abs.origin === new URL(base).origin) urls.add(abs.href.split('#')[0])
-    } catch {
-      /* not a URL */
-    }
-  }
-  if (isHtml) {
-    for (const m of text.matchAll(/<script[^>]*\bsrc=["']([^"']+)["']/gi)) add(m[1])
-    for (const m of text.matchAll(/<link[^>]*\brel=["'](?:modulepreload|preload|stylesheet)["'][^>]*\bhref=["']([^"']+)["']/gi)) add(m[1])
-    for (const m of text.matchAll(/<link[^>]*\bhref=["']([^"']+)["'][^>]*\brel=["'](?:modulepreload|preload|stylesheet)["']/gi)) add(m[1])
-  } else {
-    // Chunks a bundle loads later, and its source map.
-    for (const m of text.matchAll(/["'`]((?:\.{0,2}\/)?[\w./-]+\.(?:m?js|css))["'`]/g)) add(m[1])
-    for (const m of text.matchAll(/\/[/*]# sourceMappingURL=([^\s*]+)/g)) if (!m[1].startsWith('data:')) add(m[1])
-  }
-  return urls
-}
-
-/**
- * Scans a deployed site by URL. Only files on the same site are fetched.
- * @param {string} url
- * @param {{ name: string, value: string, source: string }[]} [secrets] local values to look for too
- */
-export async function scanUrl(url, secrets = []) {
-  const findings = []
-  const seen = new Set()
-  const queue = [url]
-  let maps = 0
-  while (queue.length && seen.size < MAX_FILES) {
-    const u = queue.shift()
-    if (seen.has(u)) continue
-    seen.add(u)
-    const text = await get(u)
-    if (text === null) continue
-    const isHtml = u === url || /<html|<!doctype/i.test(text.slice(0, 500))
-    if (/\.map(\?|$)/.test(u)) maps++
-    for (const f of scanText(text, secrets)) findings.push(Object.assign(f, { file: u }))
-    for (const next of assetsIn(text, u, isHtml)) if (!seen.has(next)) queue.push(next)
-  }
-  if (maps) {
-    findings.push({ level: 'warn', kind: 'sourcemap', file: url, line: 0, what: `${maps} source map${maps === 1 ? '' : 's'} can be downloaded from this site. Anyone can read your original source code from them.` })
-  }
-  return { findings, files: seen.size }
-}
+// Scans a live site the way a stranger would: the page, the scripts it loads,
+// the chunks those scripts load, and any source maps they point to.
+
+import { scanText } from './scan.js'
+
+const MAX_FILES = 200
+
+async function get(url) {
+  try {
+    const res = await fetch(url, { redirect: 'follow', headers: { 'user-agent': 'shipped-secrets' } })
+    if (!res.ok) return null
+    const type = res.headers.get('content-type') || ''
+    if (!/text|javascript|json|html|xml|svg/.test(type) && !/\.(m?js|map|json|css|html?)(\?|$)/.test(url)) return null
+    return await res.text()
+  } catch {
+    return null
+  }
+}
+
+function assetsIn(text, base, isHtml) {
+  const urls = new Set()
+  const add = (u) => {
+    try {
+      const abs = new URL(u, base)
+      if (abs.origin === new URL(base).origin) urls.add(abs.href.split('#')[0])
+    } catch {
+      /* not a URL */
+    }
+  }
+  if (isHtml) {
+    for (const m of text.matchAll(/<script[^>]*\bsrc=["']([^"']+)["']/gi)) add(m[1])
+    for (const m of text.matchAll(/<link[^>]*\brel=["'](?:modulepreload|preload|stylesheet)["'][^>]*\bhref=["']([^"']+)["']/gi)) add(m[1])
+    for (const m of text.matchAll(/<link[^>]*\bhref=["']([^"']+)["'][^>]*\brel=["'](?:modulepreload|preload|stylesheet)["']/gi)) add(m[1])
+  } else {
+    // Chunks a bundle loads later, and its source map.
+    for (const m of text.matchAll(/["'`]((?:\.{0,2}\/)?[\w./-]+\.(?:m?js|css))["'`]/g)) add(m[1])
+    for (const m of text.matchAll(/\/[/*]# sourceMappingURL=([^\s*]+)/g)) if (!m[1].startsWith('data:')) add(m[1])
+  }
+  return urls
+}
+
+/**
+ * Scans a deployed site by URL. Only files on the same site are fetched.
+ * @param {string} url
+ * @param {{ name: string, value: string, source: string }[]} [secrets] local values to look for too
+ */
+export async function scanUrl(url, secrets = [], opts = {}) {
+  const findings = []
+  const seen = new Set()
+  const queue = [url]
+  let maps = 0
+  while (queue.length && seen.size < MAX_FILES) {
+    const u = queue.shift()
+    if (seen.has(u)) continue
+    seen.add(u)
+    const text = await get(u)
+    if (text === null) continue
+    const isHtml = u === url || /<html|<!doctype/i.test(text.slice(0, 500))
+    if (/\.map(\?|$)/.test(u)) maps++
+    for (const f of scanText(text, secrets, { deep: opts.deep })) findings.push(Object.assign(f, { file: u }))
+    for (const next of assetsIn(text, u, isHtml)) if (!seen.has(next)) queue.push(next)
+  }
+  if (maps) {
+    findings.push({ level: 'warn', kind: 'sourcemap', file: url, line: 0, what: `${maps} source map${maps === 1 ? '' : 's'} can be downloaded from this site. Anyone can read your original source code from them.` })
+  }
+  return { findings, files: seen.size }
+}

package/src/vite.d.ts

@@ -1 +1 @@
-export default function shippedSecrets(opts?: { allow?: string[]; failOnWarn?: boolean }): { name: string; apply: 'build' }
+export default function shippedSecrets(opts?: { allow?: string[]; failOnWarn?: boolean; deep?: boolean }): { name: string; apply: 'build' }

package/src/vite.js

@@ -7,7 +7,7 @@
 import { report, scanProject } from './index.js'
 
 /**
- * @param {{ allow?: string[], failOnWarn?: boolean }} [opts]
+ * @param {{ allow?: string[], failOnWarn?: boolean, deep?: boolean }} [opts]
  */
 export default function shippedSecrets(opts = {}) {
   let root = process.cwd()
@@ -23,7 +23,7 @@ export default function shippedSecrets(opts = {}) {
     },
     closeBundle() {
       if (!isBuild) return
-      const result = scanProject(root, { dirs: [outDir], allow: opts.allow })
+      const result = scanProject(root, { dirs: [outDir], allow: opts.allow, deep: opts.deep })
       if (!result.findings.length) return
       const text = report(result)
       const fail = result.findings.some((f) => f.level === 'error') || (opts.failOnWarn && result.findings.length)