@sanity-labs/secret-scan 1.0.0 → 1.1.0

package/README.md

@@ -1,8 +1,11 @@
 # @sanity-labs/secret-scan
 
-Detect and redact secrets in strings. Works in browser and Node.js. Zero runtime dependencies.
+Detect and redact secrets in strings. Designed for **chat and paste contexts** where secrets appear without surrounding code context.
 
-Rules derived from [gitleaks](https://github.com/gitleaks/gitleaks) (MIT licensed) — 221 rules covering API keys, tokens, passwords, and credentials from 100+ providers.
+- **1,100+ detection rules** extracted from [TruffleHog](https://github.com/trufflesecurity/trufflehog) detectors
+- **Zero runtime dependencies** — works in browser and Node.js
+- **Fast** — keyword pre-filtering means most rules are skipped for any given input (~0.15ms for short messages)
+- **Two functions** — `scan(input)` finds secrets, `redact(input, replacer)` replaces them
 
 ## Install
 
@@ -12,114 +15,81 @@ npm install @sanity-labs/secret-scan
 
 ## Usage
 
-### `scan` — find secrets
-
 ```typescript
-import { scan } from '@sanity-labs/secret-scan'
-
-const secrets = scan('OPENAI_API_KEY=sk-proj-abc123...\nMODE=production')
-// [
-//   {
-//     rule: 'openai-api-key',
-//     label: 'OpenAI API Key',
-//     text: 'sk-proj-abc123...',
-//     confidence: 'high',
-//     start: 15,
-//     end: 32
-//   }
-// ]
+import { scan, redact } from '@sanity-labs/secret-scan'
+
+// Find secrets
+const secrets = scan('my key is ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh1234')
+// [{ rule: 'github-v2', label: 'Github V2', text: 'ghp_...', confidence: 'high', start: 10, end: 50 }]
+
+// Redact secrets
+const safe = redact(input, (secret, index) => `[secret:${index}]`)
+// 'my key is [secret:0]'
 ```
 
-### `redact` — find and replace secrets
+## What it detects
+
+Bare paste (no surrounding context needed):
+
+| Provider | Prefix/Pattern | Rule ID |
+|----------|---------------|---------|
+| OpenAI | `sk-proj-...T3BlbkFJ...` | `openai` |
+| Anthropic | `sk-ant-api03-...` | `anthropic` |
+| AWS | `AKIA...` | `aws-access_keys` |
+| GitHub | `ghp_`, `gho_`, `github_pat_` | `github-v2` |
+| Stripe | `sk_live_`, `rk_live_` | `stripe` |
+| Slack | `xoxb-`, `xoxp-` | `slack` |
+| Groq | `gsk_` | `groq` |
+| Replicate | `r8_` | `replicate` |
+| SendGrid | `SG.` | `sendgrid` |
+| JWT | `eyJ...` | `jwt` |
+| GitLab | `glpat-` | `gitlab-v2` |
+| NPM | `npm_` | `npmtokenv2` |
+| Linear | `lin_api_` | `linearapi` |
+| Supabase | `sbp_` | `supabasetoken` |
+| Postman | `PMAK-` | `postman` |
+
+Plus 850+ more providers. Connection strings (postgres://, mongodb://, redis://) and Bearer tokens are also detected.
 
-```typescript
-import { redact } from '@sanity-labs/secret-scan'
+## How it works
 
-const secrets = new Map()
-let nextId = 0
+Rules are extracted from [TruffleHog's Go detectors](https://github.com/trufflesecurity/trufflehog/tree/main/pkg/detectors) and compiled to JavaScript RegExp. TruffleHog's keyword pre-filter uses strings from the **secret itself** (prefixes like `gsk_`, `T3BlbkFJ`), not surrounding context — this is why it works for bare paste in chat.
 
-const redacted = redact(pastedText, (secret) => {
-  const key = `[secret:${nextId++}]`
-  secrets.set(key, secret)
-  return key
-})
+A keyword index maps each keyword to its rules. For any input, only rules whose keywords appear in the input are tested — typically <10 rules out of 1,100+.
 
-// redacted: "OPENAI_API_KEY=[secret:0]\nSTRIPE_KEY=[secret:1]"
-// secrets: Map { '[secret:0]' => { text: 'sk-proj-...' }, ... }
+### Updating rules
+
+```bash
+npm run update-rules
 ```
 
+This clones/updates TruffleHog, parses all detector Go files, converts Go regex to JS, and regenerates `src/rules.ts`.
+
 ## API
 
 ### `scan(input: string): Secret[]`
 
-Returns an array of every secret found in the input.
-
-### `redact(input: string, replacer: (secret: Secret) => string): string`
-
-Calls `replacer` for each detected secret. The return value replaces the secret in the output string. The caller owns all state — `redact` just does string replacement.
-
-### `Secret`
+Returns all secrets found in the input string.
 
 ```typescript
 interface Secret {
-  rule: string                   // gitleaks rule ID, e.g. 'openai-api-key'
-  label: string                  // human-readable, e.g. 'OpenAI API Key'
-  text: string                   // the matched secret value
-  confidence: 'high' | 'medium'  // provider pattern vs entropy-based
-  start: number                  // start index in input
-  end: number                    // end index (exclusive) in input
+  rule: string        // Rule ID (e.g., 'openai')
+  label: string       // Human-readable label
+  text: string        // The matched secret value
+  confidence: 'high' | 'medium'
+  start: number       // Start index in input
+  end: number         // End index (exclusive)
 }
 ```
 
-### `shannonEntropy(s: string): number`
-
-Shannon entropy calculation. Exported for advanced use cases.
-
-## How it works
-
-1. **Keyword pre-filter** — Each rule has keywords. Before running a regex, we check if the input contains any of its keywords (case-insensitive). This keeps scanning fast with 221 rules — most regexes are skipped for any given input.
-
-2. **Regex matching** — Rules are compiled from [gitleaks.toml](https://github.com/gitleaks/gitleaks/blob/master/config/gitleaks.toml) with Go→JS regex conversion (named groups, inline flags, dotall).
-
-3. **Entropy filtering** — Many rules have Shannon entropy thresholds. Low-entropy matches (like `KEY=aaaaaaa`) are filtered out.
-
-4. **Allowlist filtering** — Global and per-rule allowlists filter false positives. Includes 1,446 stopwords for the generic-api-key rule.
-
-## Updating rules
-
-```bash
-npm run update-rules
-```
-
-Fetches the latest `gitleaks.toml` from GitHub, converts Go regex → JS regex, and writes `src/rules.ts`. Run this whenever gitleaks updates their rules.
-
-### Go → JS regex conversion
-
-| Go pattern | JS equivalent | Notes |
-|---|---|---|
-| `(?P<name>...)` | `(?<name>...)` | Named groups |
-| `(?i)` at start | `i` flag | Case-insensitive |
-| `(?i:...)` mid-pattern | `(?:...)` + `i` flag | Promoted to global flag |
-| `(?-i:...)` | `(?:...)` | Groups already enumerate cases |
-| `(?s:.)` | `[\s\S]` | Dotall |
-| `\z` | `$` | End of string |
-
-## Rules coverage
+### `redact(input: string, replacer: (secret: Secret) => string): string`
 
-221 rules from gitleaks covering:
+Finds and replaces all secrets. Replacements applied right-to-left to preserve indices.
 
-- **Cloud providers**: AWS, GCP, Azure, DigitalOcean, Heroku, Fly.io, etc.
-- **AI/ML**: OpenAI, Anthropic, Cohere, HuggingFace, Perplexity
-- **Payment**: Stripe, Square, Plaid, Coinbase, Flutterwave
-- **DevOps**: GitHub, GitLab, Bitbucket, CircleCI, Travis CI, Jenkins
-- **Communication**: Slack, Discord, Telegram, Twilio, SendGrid
-- **Databases**: PlanetScale, MongoDB Atlas, ClickHouse
-- **And 80+ more providers**
+### `shannonEntropy(s: string): number`
 
-Plus the `generic-api-key` rule which catches `KEY=value` patterns with entropy thresholds and 1,446 stopwords.
+Calculate Shannon entropy of a string. Used internally for entropy-based filtering.
 
 ## License
 
-MIT — includes gitleaks copyright notice per their license terms.
-
-This package uses rules derived from [gitleaks](https://github.com/gitleaks/gitleaks), which is also MIT licensed. Copyright (c) 2019 Zachary Rice.
+MIT. Rules derived from [TruffleHog](https://github.com/trufflesecurity/trufflehog) (Apache 2.0).

package/dist/index.cjs

@@ -6712,6 +6712,61 @@ var customRules = [
     ),
     keywords: ["bearer"],
     entropy: 3.5
+  },
+  // ─── Inngest signing keys ─────────────────────────────────────────
+  // Format: signkey-prod-<hex> or signkey-test-<hex>
+  // No TruffleHog detector exists. Distinctive prefix makes this safe.
+  {
+    id: "inngest-signing-key",
+    label: "Inngest Signing Key",
+    regex: new RegExp(
+      "\\b(signkey-(?:prod|test|staging)-[a-f0-9]{32,})\\b",
+      ""
+    ),
+    keywords: ["signkey-"]
+  },
+  // ─── Inngest event keys ───────────────────────────────────────────
+  // Format varies, but commonly used with INNGEST_EVENT_KEY= context.
+  // Event keys are hex strings. We detect them when the env var name
+  // provides context.
+  {
+    id: "inngest-event-key",
+    label: "Inngest Event Key",
+    regex: new RegExp(
+      `(?:INNGEST_EVENT_KEY\\s*[=:]\\s*)["']?([a-f0-9A-F]{16,})["']?`,
+      ""
+    ),
+    keywords: ["inngest_event_key"]
+  },
+  // ─── ElevenLabs bare sk_ keys ─────────────────────────────────────
+  // TruffleHog's elevenlabs-v2 rule matches sk_ + 48 hex chars but
+  // requires "elevenlabs" keyword in the input. This catches bare paste
+  // of sk_ + 48 hex chars without context. The sk_ prefix + hex-only
+  // suffix is distinctive enough.
+  {
+    id: "elevenlabs-bare",
+    label: "ElevenLabs API Key",
+    regex: new RegExp(
+      "\\b(sk_[a-f0-9]{48})\\b",
+      ""
+    ),
+    keywords: ["sk_"]
+  },
+  // ─── Sanity bare tokens ───────────────────────────────────────────
+  // TruffleHog's sanity rule matches sk + 79 alphanumeric chars but
+  // requires "sanity" keyword in the input. Sanity tokens start with
+  // sk followed by a capital letter (skE..., skR..., skS...) which
+  // distinguishes them from other sk-prefixed keys. Real tokens are
+  // ~80 chars total (sk + 78). Entropy threshold prevents false positives.
+  {
+    id: "sanity-bare",
+    label: "Sanity API Token",
+    regex: new RegExp(
+      "\\b(sk[A-Z][A-Za-z0-9]{58,})\\b",
+      ""
+    ),
+    keywords: ["sk"],
+    entropy: 4
   }
 ];
 
@@ -6750,6 +6805,59 @@ for (const rule of rules2) {
     }
   }
 }
+var MIN_SUFFIX_ENTROPY = 1.5;
+var KNOWN_PREFIXES = [
+  "sk-proj-",
+  "sk-svcacct-",
+  "sk-admin-",
+  "sk-ant-api03-",
+  "sk_live_",
+  "sk_test_",
+  "pk_live_",
+  "pk_test_",
+  "rk_live_",
+  "rk_test_",
+  "ghp_",
+  "gho_",
+  "ghu_",
+  "ghs_",
+  "ghr_",
+  "github_pat_",
+  "xoxb-",
+  "xoxp-",
+  "xoxa-",
+  "xoxo-",
+  "xapp-",
+  "glpat-",
+  "glsa_",
+  "npm_",
+  "lin_api_",
+  "gsk_",
+  "r8_",
+  "sbp_",
+  "sk_",
+  "SG.",
+  "dp.pt.",
+  "PMAK-",
+  "signkey-prod-",
+  "signkey-test-",
+  "signkey-staging-",
+  "sk-",
+  "AKIA"
+];
+function isPlaceholder(secret) {
+  let longestPrefix = "";
+  for (const prefix of KNOWN_PREFIXES) {
+    if (secret.startsWith(prefix) && prefix.length > longestPrefix.length) {
+      longestPrefix = prefix;
+    }
+  }
+  if (!longestPrefix) return false;
+  const suffix = secret.slice(longestPrefix.length);
+  if (suffix.length === 0) return true;
+  const entropy = shannonEntropy(suffix);
+  return entropy < MIN_SUFFIX_ENTROPY;
+}
 function isGlobalAllowlisted(secret) {
   for (const { regex } of globalAllowlist.regexes) {
     if (regex.test(secret)) return true;
@@ -6804,13 +6912,12 @@ function extractSecret(match, rule) {
   }
   return match[0];
 }
-var GENERIC_RULE_IDS = /* @__PURE__ */ new Set(["generic-sk-secret", "bearer-token"]);
+var GENERIC_RULE_IDS = /* @__PURE__ */ new Set(["generic-sk-secret", "bearer-token", "sanity-bare"]);
 function scan(input) {
   if (!input) return [];
   const inputLower = input.toLowerCase();
   const candidates = getCandidateRules(inputLower);
-  const secrets = [];
-  const matchedRanges = [];
+  const allMatches = [];
   for (const rule of candidates) {
     const regex = new RegExp(rule.regex.source, rule.regex.flags.replace("g", "") + "g");
     let match;
@@ -6820,19 +6927,16 @@ function scan(input) {
       const secretStart = input.indexOf(secret, match.index);
       const start = secretStart >= 0 ? secretStart : match.index;
       const end = start + secret.length;
-      const overlaps = matchedRanges.some(
-        (r) => start < r.end && end > r.start
-      );
-      if (overlaps) continue;
       if (rule.entropy !== void 0) {
         const entropy = shannonEntropy(secret);
         if (entropy < rule.entropy) continue;
       }
+      if (isPlaceholder(secret)) continue;
       if (isGlobalAllowlisted(secret)) continue;
       const line = getLineForIndex(input, match.index);
       if (isRuleAllowlisted(rule, secret, match[0], line)) continue;
       const confidence = GENERIC_RULE_IDS.has(rule.id) ? "medium" : "high";
-      secrets.push({
+      allMatches.push({
         rule: rule.id,
         label: rule.label,
         text: secret,
@@ -6840,10 +6944,24 @@ function scan(input) {
         start,
         end
       });
-      matchedRanges.push({ start, end });
       if (match[0].length === 0) regex.lastIndex++;
     }
   }
+  allMatches.sort((a, b) => {
+    const lenDiff = b.end - b.start - (a.end - a.start);
+    if (lenDiff !== 0) return lenDiff;
+    return a.start - b.start;
+  });
+  const secrets = [];
+  const matchedRanges = [];
+  for (const candidate of allMatches) {
+    const overlaps = matchedRanges.some(
+      (r) => candidate.start < r.end && candidate.end > r.start
+    );
+    if (overlaps) continue;
+    secrets.push(candidate);
+    matchedRanges.push({ start: candidate.start, end: candidate.end });
+  }
   secrets.sort((a, b) => a.start - b.start);
   return secrets;
 }