builtwith-official-cli 1.0.0

package/README.md

@@ -0,0 +1,396 @@
+# BuiltWith CLI 🔍
+
+> Non-interactive, scriptable CLI for the [BuiltWith API](https://api.builtwith.com) — designed for automation, CI/CD pipelines, and AI agent consumption.
+
+```bash
+bw domain lookup shopify.com --format table
+bw domain lookup shopify.com --nopii | jq '.Results[0].Technologies[].Name'
+bw live feed --duration 60 > events.ndjson
+bw mcp   # start MCP server for Claude Desktop, VS Code, etc.
+```
+
+## 🤔 Why this exists
+
+The [BuiltWith TUI](https://github.com/builtwith/builtwith-tui) is great for interactive exploration. This CLI is intentionally different:
+
+- **stdout = data only** (JSON/table/CSV) — safe to pipe anywhere
+- **stderr = human output** (spinners, errors, debug info)
+- **Structured exit codes** — scripts can distinguish auth failures from rate limits from network errors
+- **Multiple auth paths** — works in CI with env vars, locally with rc files, or inline with `--key`
+
+---
+
+## 📦 Installation
+
+```bash
+npm install -g builtwith-official-cli
+```
+
+Or run directly without installing:
+
+```bash
+npx builtwith-official-cli domain lookup example.com --key YOUR_KEY
+```
+
+Registers as both `bw` (short) and `builtwith` (discoverable).
+
+---
+
+## 🔑 Authentication
+
+API key is resolved in priority order:
+
+| Priority | Method |
+|---|---|
+| 1 | `--key <value>` CLI flag |
+| 2 | `BUILTWITH_API_KEY` environment variable |
+| 3 | `.builtwithrc` in current directory |
+| 4 | `.builtwithrc` in home directory |
+
+`.env` files in the current directory are loaded automatically, so `BUILTWITH_API_KEY=xxx` in `.env` works too.
+
+**`.builtwithrc` format:**
+```json
+{"key": "YOUR_API_KEY"}
+```
+
+Copy `.builtwithrc.example` to get started:
+```bash
+cp .builtwithrc.example ~/.builtwithrc
+# then edit with your key
+```
+
+---
+
+## 💻 Commands
+
+### 🌐 Domain
+
+```bash
+bw domain lookup <domain> [flags]
+```
+
+| Flag | Description |
+|---|---|
+| `--nopii` | Exclude PII data |
+| `--nometa` | Exclude meta data |
+| `--noattr` | Exclude attribution data |
+| `--liveonly` | Only currently-live technologies |
+| `--fdrange <YYYYMMDD-YYYYMMDD>` | First-detected date range |
+| `--ldrange <YYYYMMDD-YYYYMMDD>` | Last-detected date range |
+
+```bash
+bw domain lookup shopify.com
+bw domain lookup shopify.com --format table
+bw domain lookup shopify.com --nopii --liveonly | jq '.Results[0].Technologies[].Name'
+bw domain lookup shopify.com --fdrange 20240101-20241231
+```
+
+### 📋 Lists
+
+```bash
+bw lists tech <tech> [--offset <n>] [--limit <n>]
+```
+
+```bash
+bw lists tech WordPress
+bw lists tech Shopify --limit 50 --offset 100
+```
+
+### 🔗 Relationships
+
+```bash
+bw relationships lookup <domain>
+```
+
+### 🆓 Free
+
+```bash
+bw free lookup <domain>
+```
+
+### 🏢 Company
+
+```bash
+bw company find <name>
+```
+
+```bash
+bw company find "Shopify"
+```
+
+### 🏷️ Tags
+
+```bash
+bw tags lookup <lookup>
+```
+
+### 💡 Recommendations
+
+```bash
+bw recommendations lookup <domain>
+```
+
+### ↪️ Redirects
+
+```bash
+bw redirects lookup <domain>
+```
+
+### 🔤 Keywords
+
+```bash
+bw keywords lookup <domain>
+```
+
+### 📈 Trends
+
+```bash
+bw trends tech <tech>
+```
+
+```bash
+bw trends tech React
+```
+
+### 🛍️ Products
+
+```bash
+bw products search <query> [--page <n>] [--limit <n>]
+```
+
+```bash
+bw products search "coffee maker"
+bw products search "running shoes" --page 2 --limit 50
+```
+
+### 🛡️ Trust
+
+```bash
+bw trust lookup <domain>
+```
+
+### 👤 Account
+
+```bash
+bw account whoami
+bw account usage
+```
+
+### 📡 Live Feed
+
+Stream live technology detection events as [NDJSON](https://jsonlines.org/), one event per line.
+
+```bash
+bw live feed [--duration <seconds>]
+```
+
+```bash
+# Stream indefinitely (Ctrl+C to stop)
+bw live feed
+
+# Capture 60 seconds of events
+bw live feed --duration 60 > events.ndjson
+
+# Pipe to jq in real time
+bw live feed | jq --unbuffered '.domain'
+```
+
+---
+
+## 🚩 Global Flags
+
+Available on every command:
+
+| Flag | Description |
+|---|---|
+| `--key <apikey>` | API key (highest priority) |
+| `--format <fmt>` | `json` (default) \| `table` \| `csv` |
+| `--no-color` | Disable color on stderr |
+| `--dry-run` | Print request URL (key masked) and exit |
+| `--debug` | Print HTTP metadata to stderr |
+| `--quiet` | Suppress spinner/info stderr output |
+
+---
+
+## 🖨️ Output Formats
+
+### JSON (default)
+
+```bash
+bw domain lookup example.com | jq '.Results[0].Technologies[].Name'
+```
+
+### Table
+
+```bash
+bw domain lookup example.com --format table
+```
+
+### CSV
+
+```bash
+bw domain lookup example.com --format csv > results.csv
+```
+
+---
+
+## 🚦 Exit Codes
+
+Scripts can use exit codes to handle different failure modes:
+
+| Code | Meaning |
+|---|---|
+| `0` | ✅ Success |
+| `1` | 💥 Unexpected error |
+| `2` | 🔐 Auth failure (missing key, 401, 403) |
+| `3` | 🔍 Not found (404) |
+| `4` | ⏱️ Rate limit (429) |
+| `5` | ⚠️ Other API error |
+| `6` | 🌐 Network failure |
+| `7` | ❌ Invalid input |
+| `8` | 🛑 Interrupted (SIGINT) |
+
+```bash
+bw domain lookup example.com
+case $? in
+  0) echo "success" ;;
+  2) echo "check your API key" ;;
+  4) echo "rate limited — slow down" ;;
+  6) echo "network error" ;;
+esac
+```
+
+---
+
+## 🔧 Pipeline Examples
+
+```bash
+# Get all live tech names for a domain
+bw domain lookup shopify.com --liveonly | \
+  jq -r '.Results[0].Technologies[].Name' | sort
+
+# Check if a domain uses WordPress
+bw domain lookup example.com --quiet --liveonly | \
+  jq -e '.Results[0].Technologies[] | select(.Name == "WordPress")' > /dev/null \
+  && echo "uses WordPress"
+
+# Export tech stack to CSV
+bw domain lookup shopify.com --format csv > shopify-tech.csv
+
+# Capture 5 minutes of live events
+bw live feed --duration 300 --quiet > feed.ndjson
+
+# Find all sites using a technology (paginated)
+for offset in 0 20 40 60 80; do
+  bw domain lists tech React --offset $offset --limit 20 --quiet
+done | jq -s 'add'
+
+# CI/CD: fail build if domain check fails
+bw domain lookup mysite.com --key "$BUILTWITH_API_KEY" --quiet || exit 1
+```
+
+---
+
+## 🐛 Dry Run & Debugging
+
+```bash
+# Preview the URL that would be called (key is masked)
+bw domain lookup example.com --key MYKEY --dry-run
+# → https://api.builtwith.com/v22/api.json?KEY=REDACTED&LOOKUP=example.com
+
+# See HTTP response metadata
+bw domain lookup example.com --debug
+```
+
+---
+
+## 🤖 MCP Server
+
+`bw mcp` starts a [Model Context Protocol](https://modelcontextprotocol.io) server over stdio, exposing all BuiltWith API endpoints as structured tools that any MCP-compatible client can call — Claude Desktop, VS Code, Cursor, Zed, and more.
+
+```bash
+bw mcp
+bw mcp --key YOUR_API_KEY   # pass key inline instead of env/rc file
+bw mcp --debug              # log JSON-RPC traffic to stderr
+```
+
+### ⚙️ Client configuration
+
+Add to your MCP client config (e.g. `claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "builtwith": {
+      "command": "bw",
+      "args": ["mcp"]
+    }
+  }
+}
+```
+
+If your API key isn't in an env var or `.builtwithrc`, pass it inline:
+
+```json
+{
+  "mcpServers": {
+    "builtwith": {
+      "command": "bw",
+      "args": ["mcp", "--key", "YOUR_API_KEY"]
+    }
+  }
+}
+```
+
+### 🧰 Available tools
+
+| Tool | Description |
+|---|---|
+| `domain_lookup` | 🌐 Technology stack for a domain (supports `nopii`, `liveonly`, date ranges) |
+| `lists_tech` | 📋 Domains currently using a technology |
+| `relationships_lookup` | 🔗 Related domains (shared infra, ownership) |
+| `free_lookup` | 🆓 Free-tier category counts for a domain |
+| `company_find` | 🏢 Domains associated with a company name |
+| `tags_lookup` | 🏷️ Domains related to an IP or tag attribute |
+| `recommendations_lookup` | 💡 Technology recommendations for a domain |
+| `redirects_lookup` | ↪️ Live and historical redirect chains |
+| `keywords_lookup` | 🔤 Keyword data for a domain |
+| `trends_tech` | 📈 Historical adoption trend for a technology |
+| `products_search` | 🛍️ Search ecommerce products across indexed stores |
+| `trust_lookup` | 🛡️ Trust/quality score for a domain |
+| `account_whoami` | 👤 Authenticated account identity |
+| `account_usage` | 📊 API usage statistics |
+
+### 🔬 Implementation note
+
+The MCP server is implemented as a pure JSON-RPC 2.0 stdio server with no additional dependencies — auth, HTTP calls, and error handling all use the same code paths as the regular CLI commands.
+
+---
+
+## 🛠️ Development
+
+```bash
+git clone https://github.com/builtwith/builtwith-cli
+cd builtwith-cli
+npm install
+npm test        # 24 tests, node:test built-in (no extra framework)
+```
+
+```bash
+# Run without installing globally
+node bin/bw.js domain lookup example.com --key YOUR_KEY
+```
+
+---
+
+## 🔗 Related
+
+- [BuiltWith TUI](https://github.com/builtwith/builtwith-tui) — interactive terminal UI for the BuiltWith API
+- [BuiltWith API Docs](https://api.builtwith.com) — full API reference
+
+---
+
+## 📄 License
+
+MIT

package/bin/bw.js

@@ -0,0 +1,4 @@
+#!/usr/bin/env node
+'use strict';
+
+require('../lib/cli').run();

package/lib/cli.js

@@ -0,0 +1,59 @@
+'use strict';
+
+const { Command } = require('commander');
+const { EXIT_CODES, BuiltWithError } = require('./errors');
+const output = require('./output');
+
+function run() {
+  const program = new Command();
+
+  program
+    .name('bw')
+    .description('Non-interactive, scriptable CLI for the BuiltWith API')
+    .version('1.0.0')
+    .exitOverride()
+    .addHelpCommand()
+    .option('--key <apikey>', 'API key (overrides env and rc file)')
+    .option('--format <fmt>', 'Output format: json (default) | table | csv', 'json')
+    .option('--no-color', 'Disable color on stderr')
+    .option('--dry-run', 'Print request URL (key masked) and exit')
+    .option('--debug', 'Print HTTP metadata to stderr')
+    .option('--quiet', 'Suppress spinner/info stderr output');
+
+  // Register all command modules
+  require('./commands/domain')(program);
+  require('./commands/lists')(program);
+  require('./commands/relationships')(program);
+  require('./commands/free')(program);
+  require('./commands/company')(program);
+  require('./commands/tags')(program);
+  require('./commands/recommendations')(program);
+  require('./commands/redirects')(program);
+  require('./commands/keywords')(program);
+  require('./commands/trends')(program);
+  require('./commands/products')(program);
+  require('./commands/trust')(program);
+  require('./commands/account')(program);
+  require('./commands/live')(program);
+  require('./commands/mcp')(program);
+
+  program.parseAsync(process.argv).catch(err => {
+    if (err.code === 'commander.helpDisplayed' || err.code === 'commander.version') {
+      process.exit(EXIT_CODES.SUCCESS);
+    }
+    if (err instanceof BuiltWithError) {
+      output.error(err.message);
+      process.exit(err.exitCode);
+    }
+    // Commander parse errors (missing args, unknown options)
+    if (err.code && err.code.startsWith('commander.')) {
+      output.error(err.message);
+      process.exit(EXIT_CODES.INVALID_INPUT);
+    }
+    output.error(err.message || String(err));
+    if (process.env.DEBUG) console.error(err.stack);
+    process.exit(EXIT_CODES.UNEXPECTED);
+  });
+}
+
+module.exports = { run };

package/lib/client.js

@@ -0,0 +1,101 @@
+'use strict';
+
+const fetch = require('node-fetch');
+const { ApiError, NetworkError } = require('./errors');
+
+const BASE_URLS = {
+  domain:          'https://api.builtwith.com/v22/api.json',
+  lists:           'https://api.builtwith.com/lists12/api.json',
+  relationships:   'https://api.builtwith.com/rv4/api.json',
+  free:            'https://api.builtwith.com/free1/api.json',
+  company:         'https://api.builtwith.com/ctu3/api.json',
+  tags:            'https://api.builtwith.com/tag1/api.json',
+  recommendations: 'https://api.builtwith.com/rec1/api.json',
+  redirects:       'https://api.builtwith.com/redirect1/api.json',
+  keywords:        'https://api.builtwith.com/kw2/api.json',
+  trends:          'https://api.builtwith.com/trends/v6/api.json',
+  products:        'https://api.builtwith.com/productv1/api.json',
+  trust:           'https://api.builtwith.com/trustv1/api.json',
+  whoami:          'https://api.builtwith.com/whoamiv1/api.json',
+  usage:           'https://api.builtwith.com/usagev2/api.json',
+};
+
+/**
+ * Build a URL with query params.
+ * Boolean true values become empty string (e.g., NOPII=) which BuiltWith treats as flag present.
+ */
+function buildUrl(endpoint, params) {
+  const base = BASE_URLS[endpoint];
+  if (!base) throw new Error(`Unknown endpoint: ${endpoint}`);
+  const url = new URL(base);
+  for (const [k, v] of Object.entries(params)) {
+    if (v === null || v === undefined || v === false) continue;
+    url.searchParams.set(k, v === true ? '' : String(v));
+  }
+  return url.toString();
+}
+
+function maskKey(url, key) {
+  if (!key) return url;
+  return url.replace(encodeURIComponent(key), 'REDACTED').replace(key, 'REDACTED');
+}
+
+async function request(endpoint, params, opts = {}) {
+  const { dryRun, debug, spinner } = opts;
+  const url = buildUrl(endpoint, params);
+
+  if (dryRun) {
+    const masked = maskKey(url, params.KEY || params.key);
+    process.stdout.write(masked + '\n');
+    process.exit(0);
+  }
+
+  if (debug) {
+    process.stderr.write(`[debug] GET ${maskKey(url, params.KEY || params.key)}\n`);
+  }
+
+  if (spinner) spinner.start();
+
+  let res;
+  try {
+    res = await fetch(url, {
+      headers: { 'User-Agent': 'builtwith-cli/1.0.0' },
+      timeout: 30000,
+    });
+  } catch (err) {
+    if (spinner) spinner.stop();
+    throw new NetworkError(`Network request failed: ${err.message}`);
+  }
+
+  if (debug) {
+    process.stderr.write(`[debug] HTTP ${res.status} ${res.statusText}\n`);
+  }
+
+  if (spinner) spinner.stop();
+
+  let body;
+  try {
+    body = await res.json();
+  } catch (_) {
+    throw new ApiError(`Invalid JSON response (HTTP ${res.status})`, res.status);
+  }
+
+  if (!res.ok) {
+    const msg = body && body.Errors
+      ? body.Errors.map(e => e.Message || e).join('; ')
+      : `HTTP ${res.status} ${res.statusText}`;
+    throw new ApiError(msg, res.status);
+  }
+
+  // BuiltWith API sometimes returns error info in the body with HTTP 200
+  if (body && body.Errors && body.Errors.length > 0) {
+    const msg = body.Errors.map(e => e.Message || e).join('; ');
+    // Treat auth-related messages as auth errors
+    const isAuth = /key|auth|unauthori/i.test(msg);
+    throw new ApiError(msg, isAuth ? 403 : 400);
+  }
+
+  return body;
+}
+
+module.exports = { buildUrl, maskKey, request, BASE_URLS };

package/lib/commands/account.js

@@ -0,0 +1,45 @@
+'use strict';
+
+const { Command } = require('commander');
+const { requireKey } = require('../config');
+const { request } = require('../client');
+const output = require('../output');
+const ora = require('ora');
+
+module.exports = function registerAccount(program) {
+  const account = program.command('account').description('Account information');
+
+  account
+    .command('whoami')
+    .description('Show account identity')
+    .action(async () => {
+      const opts = program.opts();
+      if (opts.noColor) output.setNoColor(true);
+      const key = requireKey(opts.key);
+      const spinner = opts.quiet ? null : ora({ text: 'Fetching account...', stream: process.stderr }).start();
+      try {
+        const data = await request('whoami', { KEY: key }, { dryRun: opts.dryRun, debug: opts.debug, spinner });
+        output.print(data, { format: opts.format });
+      } catch (err) {
+        if (spinner) spinner.stop();
+        throw err;
+      }
+    });
+
+  account
+    .command('usage')
+    .description('Show API usage')
+    .action(async () => {
+      const opts = program.opts();
+      if (opts.noColor) output.setNoColor(true);
+      const key = requireKey(opts.key);
+      const spinner = opts.quiet ? null : ora({ text: 'Fetching usage...', stream: process.stderr }).start();
+      try {
+        const data = await request('usage', { KEY: key }, { dryRun: opts.dryRun, debug: opts.debug, spinner });
+        output.print(data, { format: opts.format });
+      } catch (err) {
+        if (spinner) spinner.stop();
+        throw err;
+      }
+    });
+};

package/lib/commands/company.js

@@ -0,0 +1,28 @@
+'use strict';
+
+const { requireKey } = require('../config');
+const { request } = require('../client');
+const output = require('../output');
+const ora = require('ora');
+
+module.exports = function registerCompany(program) {
+  const company = program.command('company').description('Company to URL lookup');
+
+  company
+    .command('find <name>')
+    .description('Find domains associated with a company name')
+    .action(async (name) => {
+      const opts = program.opts();
+      if (opts.noColor) output.setNoColor(true);
+      const key = requireKey(opts.key);
+      const params = { KEY: key, COMPANY: name };
+      const spinner = opts.quiet ? null : ora({ text: `Finding company "${name}"...`, stream: process.stderr }).start();
+      try {
+        const data = await request('company', params, { dryRun: opts.dryRun, debug: opts.debug, spinner });
+        output.print(data, { format: opts.format });
+      } catch (err) {
+        if (spinner) spinner.stop();
+        throw err;
+      }
+    });
+};

package/lib/commands/domain.js

@@ -0,0 +1,45 @@
+'use strict';
+
+const { requireKey } = require('../config');
+const { request } = require('../client');
+const { InputError } = require('../errors');
+const output = require('../output');
+const ora = require('ora');
+
+module.exports = function registerDomain(program) {
+  const domain = program.command('domain').description('Domain technology lookup');
+
+  domain
+    .command('lookup <domain>')
+    .description('Look up technologies used by a domain')
+    .option('--nopii', 'Exclude PII data')
+    .option('--nometa', 'Exclude meta data')
+    .option('--noattr', 'Exclude attribution data')
+    .option('--liveonly', 'Only return currently-live technologies')
+    .option('--fdrange <YYYYMMDD-YYYYMMDD>', 'First-detected date range')
+    .option('--ldrange <YYYYMMDD-YYYYMMDD>', 'Last-detected date range')
+    .action(async (domainArg, cmdOpts) => {
+      const opts = program.opts();
+      if (opts.noColor) output.setNoColor(true);
+      if (!domainArg) throw new InputError('Domain argument is required');
+      const key = requireKey(opts.key);
+
+      const params = { KEY: key, LOOKUP: domainArg };
+      // Boolean flags: true → '' so URL gets ?NOPII= which BuiltWith treats as present
+      if (cmdOpts.nopii)   params.NOPII   = true;
+      if (cmdOpts.nometa)  params.NOMETA  = true;
+      if (cmdOpts.noattr)  params.NOATTR  = true;
+      if (cmdOpts.liveonly) params.LIVEONLY = true;
+      if (cmdOpts.fdrange) params.FDRANGE = cmdOpts.fdrange;
+      if (cmdOpts.ldrange) params.LDRANGE = cmdOpts.ldrange;
+
+      const spinner = opts.quiet ? null : ora({ text: `Looking up ${domainArg}...`, stream: process.stderr }).start();
+      try {
+        const data = await request('domain', params, { dryRun: opts.dryRun, debug: opts.debug, spinner });
+        output.print(data, { format: opts.format });
+      } catch (err) {
+        if (spinner) spinner.stop();
+        throw err;
+      }
+    });
+};