crowbar-cli 0.1.0__tar.gz

crowbar_cli-0.1.0/.gitignore

@@ -0,0 +1,39 @@
+# Corpus and build output - never committed (see docs/DATASETS.md)
+data/
+index/
+*.db
+*.db.*
+*.sha256
+
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+*.egg-info/
+dist/
+build/
+
+# Agent/editor local config
+opencode.json
+.opencode/
+.crowbar
+.DS_Store
+
+# Experiments / non-public working dirs
+ganith/
+Exploits/
+notes/
+skills/hacker-skill/
+_publish/
+
+# Docusaurus
+website/node_modules/
+website/build/
+website/.docusaurus/
+assets/demo.mp4
+assets/demo.webm
+
+# nix build output
+result
+result-*

crowbar_cli-0.1.0/AGENTS.md

@@ -0,0 +1,386 @@
+# AGENTS.md - crowbar traversal manual
+
+Offline vulnerability intelligence at the repo root (or `.crowbar/` in a linked
+project). One SQLite+FTS5 database, one CLI, one MCP server.
+
+**Three ways in, in order of preference:**
+
+1. MCP tools (`cb_*`) when the server is connected.
+2. `bin/crowbar <cmd> --json` - stable JSON, no parsing needed.
+3. Read-only SQL on `index/crowbar.db` via `crowbar sql` (never write).
+
+Never answer CVE facts from memory. If a fact is not in the DB, say so.
+
+## Product and authority
+
+crowbar is a local, read-only vulnerability-intelligence index with shared
+CLI and MCP workflows. It grounds answers in indexed upstream data - it is not
+a scanner and not authorization. Only test systems the user owns or may test.
+Rank threat by **KEV / SSVC active > EPSS > CVSS**, never CVSS alone. KEV and
+SSVC describe exploitation observed elsewhere; EPSS is a model estimate; a
+missing row is not proof of "not affected".
+
+## Performance (agent tokens first)
+
+Return the smallest sufficient answer. `cb_cve` caps the large per-CVE arrays
+(`affects`, `references`, `products`, `exploits`) at 50 rows each and reports an
+`affects_total` / `references_total` / `products_total` count; use `crowbar sql` or
+the CLI for the full list. Any tool response over the hard ceiling comes back as
+a valid-JSON `result_too_large` notice, never a sliced payload - so narrow the
+query or lower `limit` rather than parsing a truncated blob.
+
+Contributing to the repo itself: see [CONTRIBUTING.md](CONTRIBUTING.md).
+
+---
+
+## 0. Orient (do this first, every session)
+
+```bash
+.crowbar/bin/crowbar doctor          # built? schema? stale?  (exit 1 = needs work)
+.crowbar/bin/crowbar stats --json    # row counts + freshness
+.crowbar/bin/crowbar map             # dataset -> path -> tables -> command
+.crowbar/bin/crowbar map --tables    # what every table means
+.crowbar/bin/crowbar where --json    # absolute paths
+```
+
+If the index is missing: `crowbar sync && crowbar build` (minutes - tell the user).
+`crowbar update` is the cheap refresh (≈10 s). `crowbar update --deep` reindexes the
+intel tables.
+
+Freshness: the CLI and the MCP server start a background `crowbar update` once
+the index is a day old (at most hourly; `CROWBAR_NO_AUTO_UPDATE=1` turns it off,
+log in `index/update.log`). `cb_stats` → `index.age_days` / `index.stale`; stale
+means older than NVD's 8-day change window, so say recent CVEs may be missing.
+An installed package keeps its index in `~/.crowbar` (`CROWBAR_HOME` overrides).
+
+---
+
+## 1. Repo map
+
+```
+crowbar/
+├── bin/crowbar                  CLI shim (works through symlinks)
+├── mcp/server.py           MCP stdio shim -> crowbar.mcp
+├── crowbar/
+│   ├── paths.py            root resolution, YEARS, SCHEMA_VERSION
+│   ├── datasets.py         registry + downloader + manifest/freshness
+│   ├── indexer.py          schema, NVD/KEV/intel loaders, build(), update()
+│   ├── loaders.py          bulk loaders: OSV, Debian, MSRC, Apple, toolkit,
+│   │                       Exploit-DB source code, Metasploit sources
+│   ├── catalog.py          static dataset -> table -> command map (crowbar map)
+│   ├── query.py            read layer; every function returns dicts
+│   ├── cli.py              terminal commands (single source of truth)
+│   └── mcp.py              MCP tool definitions (calls query.py directly)
+├── scripts/
+│   ├── selftest.py         end-to-end corpus checks
+│   ├── check.py            static checks (schema/catalog/docs/style)
+│   └── make_fixture.py     tiny synthetic index for CI
+├── tests/
+│   ├── test_unit.py        111 fixture tests (CI, no corpus needed)
+│   └── test_corpus.py      generated tests against index/crowbar.db
+├── docs/
+│   ├── DATASETS.md         per-source license, shape, caveats
+│   ├── MAP.md              full tree + data flow
+│   └── NIX.md              scheduling via home-manager (no launchctl)
+├── data/                   raw upstream corpora (git-ignored, ~3.3 GB)
+├── index/
+│   ├── crowbar.db               the index (query this)
+│   ├── manifest.json       url/bytes/sha256/fetched_at per dataset
+│   ├── state.json          schema, built_at, updated_at
+│   └── stats.json          last build/update summary
+├── AGENTS.md               this file
+├── README.md               user docs
+├── TODO.md                 known gaps
+└── Makefile                sync / build / update / test / doctor
+```
+
+Raw data is grouped by source; `crowbar map` is authoritative for the mapping:
+
+```
+data/nvd/          NVD year + recent/modified feeds (gzip)
+data/kev/          CISA KEV
+data/epss/         EPSS scores
+data/cwe/ capec/   MITRE taxonomies
+data/attack/       ATT&CK enterprise/mobile/ics
+data/vulnrichment/ CISA SSVC (per-CVE JSON)
+data/exploitdb/    index CSVs + full/ source tree
+data/nuclei/       template YAML
+data/metasploit/   tree.json + source/modules/
+data/osv/          per-ecosystem advisory zips
+data/debian/       tracker.json
+data/msrc/         Microsoft Security Update Guide (sug.json)
+data/apple/        security release index + advisory pages
+data/toolkit/      gtfobins, lolbas, payloads
+data/models/       models.dev catalog
+```
+
+---
+
+## 2. Tables
+
+```sql
+-- core
+cve(id PK, published, modified, status, description,
+    cvss, cvss_version, cvss_vector, severity,
+    kev INT, epss REAL, epss_pct REAL,
+    has_exploit INT, has_nuclei INT, has_msf INT, year INT)
+cve_cwe(cve_id, cwe)
+cve_product(cve_id, vendor, product)
+cpe_match(cve_id, part, vendor, product, version,
+          v_start_incl, v_start_excl, v_end_incl, v_end_excl)
+cve_ref(cve_id, url, tags)
+cve_exploit(cve_id, exploit_id)
+cve_nuclei(cve_id, template_id)
+cve_msf(cve_id, module_path)
+
+-- intel
+kev(cve PK, vendor, product, name, date_added, ransomware, action, due, notes,
+    epss, epss_pct)   -- EPSS on every KEV row, even CVEs outside the tier's NVD years
+exploit(id PK, file, description, date, author, type, platform,
+        verified, codes, tags, url, raw)
+shellcode(id PK, file, description, date, platform, url)
+nuclei(id PK, name, severity, tags, cve, path, description, domain)
+msf(path PK, type, name, platform, privileged, disclosure_date,
+    description, cves)
+attack(id, domain, name, tactics, platforms, description, url)  -- PK(id,domain)
+cwe(id PK, name, abstraction, status, description)
+capec(id PK, name, severity, likelihood, cwe, description)
+model(provider, provider_name, id, name, context, output_limit,
+      input_cost, output_cost, released, reasoning, tool_call, vision)
+ssvc(cve_id PK, exploitation, automatable, technical_impact, timestamp, source)
+
+-- bulk / ecosystem / vendor
+advisory(source, id, ecosystem, package, cve, summary, severity,
+         published, modified, url, aliases, affected_count)
+package_range(advisory_id, ecosystem, package, introduced, fixed,
+              last_affected, versions)
+debian(package, release, cve, status, fixed, urgency, description)
+os_advisory(source, cve, platform, product, version, title, severity,
+            impact, exploited, released, url)
+toolkit(source, name, path, description, tags)
+
+-- full-text
+cve_fts(id, description, products)
+exploit_fts(id, description, tags, codes)
+nuclei_fts(id, name, tags, cve)
+model_fts(provider, id, name)
+exploit_code_fts(exploit_id, path, content)   -- Exploit-DB source code
+toolkit_fts(source, name, description, tags)
+
+-- schema v3: CNA + public PoC + provenance
+cve_cna(cve_id PK, assigner, state, published, updated, title, description,
+        cvss, cvss_version, cvss_vector, severity, record_path, source,
+        source_url, source_sha256, fetched_at)
+cve_cna_ref(cve_id, url, tags)            -- CNA references, kept apart from NVD
+cve_cna_product(cve_id, vendor, product)  -- CNA affected products
+cve_meta(cve_id PK, published_year, cve_year, has_poc, poc_count)
+poc(source, cve_id, repo, url, repo_id, stars, language,
+    created_at, pushed_at, description, fetched_at)
+poc_orphan(cve_id PK, count)   -- PoCs for CVEs the index does not carry
+cve_alias(cve_id, alias, source)
+source_state(source PK, url, path, license, sha256, fetched_at, loaded_at,
+             record_count, status, note)
+
+meta(key, value)
+```
+
+`cve_cna` holds the **CNA-authored** record (CVE List v5) beside the NVD row;
+it never replaces `cve.description` or `cve.cvss`. `cve_meta.published_year` is
+the publication year, `cve_meta.cve_year` the year in the CVE ID - they differ
+for ~85k records. `poc` holds repository metadata only; no PoC is ever cloned
+or executed.
+
+`os_advisory.platform` is `windows` (MSRC) or `ios` / `macos` / `apple`
+(Apple). `impact` is normalized: `RCE`, `EoP`, `DoS`, `InfoDisclosure`,
+`Spoofing`, `Bypass`, `Tampering`.
+
+---
+
+## 3. Which path answers which question
+
+| question | use | not this |
+|---|---|---|
+| is product X version Y affected? | `crowbar product X --version Y` (`cpe_match`) | `crowbar search --product` |
+| what does CVE-X mean? | `crowbar cve CVE-X` | search |
+| CVE in an npm/PyPI/Go package | `crowbar advisory --package pkg` or `crowbar cve` → `advisories` | CVE search |
+| Windows / iOS / macOS advisory | `crowbar os` (`--platform`, `--impact RCE`) | vendor CVE title grep |
+| Debian fixed version | `crowbar debian pkg` | NVD |
+| public PoC repository for a CVE | `crowbar poc --cve CVE-X` | `crowbar cve` → `pocs` |
+| CNA-authored description / refs | `crowbar cve CVE-X` → `cna` | NVD description |
+| which source filled a table | `crowbar sources` | - |
+| PoC code, not just a title | `crowbar code "<snippet>"` | `crowbar exploit` |
+| living-off-the-land binary | `crowbar bin <name>` | ATT&CK |
+| technique for an actor/behavior | `crowbar attack <q>` | CWE |
+| weakness class | `crowbar cwe <n>` / `crowbar capec <q>` | CVE |
+| model pricing/context | `crowbar models` | - |
+
+---
+
+## 4. Recipes
+
+### Triage one CVE
+
+```bash
+.crowbar/bin/crowbar cve CVE-2021-44228 --json | jq \
+  '{id,severity,cvss,epss,kev,cwe,ssvc,
+    exploits:(.exploits|length), nuclei:(.nuclei|length),
+    os:.os, advisories:.advisories, ranges:[.affects[].label]}'
+```
+
+Interpretation order:
+
+1. `kev == 1` → confirmed exploitation somewhere in the wild.
+2. `ssvc.exploitation` → `active` / `poc` / `none`.
+3. `epss` (30-day probability) and `epss_pct`.
+4. `cvss` / `severity` → impact if triggered.
+5. `exploits` / `nuclei` / `metasploit` → working or near-working PoC.
+6. `os` / `advisories` / `debian` → vendor and distro applicability.
+7. `cwe` / `capec` → bug class → mitigations and detection.
+
+Never present CVSS alone as "risk". KEV ≫ SSVC ≫ EPSS ≫ CVSS.
+
+### "We run <product> <version> - what's exploitable?"
+
+```bash
+crowbar product nginx --version 1.18.0 --json
+```
+
+Tests every `cpe_match` row against the version (exact or start/end bounds) and
+returns only real hits plus the matched range. Rank by `kev`, `epss`,
+`has_exploit`. NVD CPE data is incomplete for some vendors (notably Apple);
+cross-check `crowbar os`.
+
+### Ecosystem package
+
+```bash
+crowbar advisory --package lodash --ecosystem npm --json
+crowbar advisory "prototype pollution" --limit 20
+```
+
+Each result carries `ranges[]` with `introduced` / `fixed` / `last_affected`.
+`crowbar cve` also lists advisory ids and Debian fixed versions.
+
+### Windows / iOS / macOS
+
+```bash
+crowbar os --platform windows --impact RCE --since 2025-01-01
+crowbar os "Windows 11" --json
+crowbar os --platform ios
+crowbar os --platform macos --limit 100
+```
+
+MSRC rows carry `severity` and normalized `impact`; Apple rows carry the
+product/version title and the CVEs extracted from each advisory page.
+
+### PoC / exploit code
+
+```bash
+crowbar exploit "log4j jndi" --json     # Exploit-DB index
+crowbar exploit 50592                   # download source, prints local path
+crowbar code "jndi lookup ldap"         # search inside the sources, returns snippet
+crowbar nuclei "cve-2021-44228" --json  # template path
+crowbar msf ms17 --json
+```
+
+Nuclei YAML is at `data/nuclei/templates/<path>`; Exploit-DB sources at
+`data/exploitdb/full/<path>`. Neither is executed.
+
+### Search
+
+```bash
+crowbar search "remote code execution" --limit 50 --json
+```
+
+AND over FTS5 tokens (prefix ≥4 chars), falling back to the longest term.
+Filters: `--kev --exploit --epss N --cvss N --severity A,B --year Y --since DATE
+--cwe CWE-N --product P --vendor V --ssvc active`.
+
+Ranking: `10·normalized_bm25 + 4·KEV + 5·EPSS + 1.5·min(exploits,3)`.
+`--sort newest|cvss|epss` overrides.
+
+### SQL escape hatch
+
+```bash
+crowbar sql "select cve, vendor, product, name, date_added from kev
+        where date_added >= date('now','-90 day') order by date_added desc" --json
+
+crowbar sql "select c.id, c.epss, c.cvss from cve c
+        where c.epss > 0.5 and c.kev = 0 and c.has_exploit = 1
+        order by c.epss desc limit 50" --json
+
+crowbar sql "select advisory_id, ecosystem, package, fixed from package_range
+        where package='log4j-core'" --json
+
+crowbar sql "select cve, product, version, impact from os_advisory
+        where platform='windows' and impact='RCE'
+        and released >= '2026-01-01' order by released desc" --json
+```
+
+`crowbar sql` is `PRAGMA query_only=ON`: no writes, no DDL.
+
+### ATT&CK / CWE / CAPEC / models
+
+```bash
+crowbar attack T1059 --json
+crowbar attack "credential dumping" --domain enterprise --json
+crowbar attack plc --domain ics --json
+crowbar cwe 79
+crowbar capec 66
+crowbar models --json | jq '[.[]|{provider,id,context,input_cost}]|sort_by(.input_cost)[:10]'
+```
+
+---
+
+## 5. Project linking
+
+```bash
+crowbar link /path/to/project     # .crowbar -> corpus, .opencode/skills/crowbar -> skill
+crowbar unlink /path/to/project
+```
+
+Inside a linked project use `.crowbar/bin/crowbar`. Unlinking removes only the
+symlinks. The MCP server works in every project whether or not it is linked.
+
+---
+
+## 6. Guardrails
+
+- **Authorization first.** Only test hosts you own or are permitted to test.
+- Exploits are untrusted code. Run in a disposable VM/container.
+  `crowbar exploit <id>` downloads source; it never executes it.
+- KEV/SSVC describe observed exploitation elsewhere, not your host.
+- EPSS is model output, not a verdict.
+- `cpe_match` keeps NVD's `vulnerable=true` rows; wildcard vendors/products are
+  dropped. If a CVE has no CPE data, read the raw NVD JSON.
+- OSV/Apple/MSRC coverage is advisory-driven, not exhaustive. Absence of a row
+  is not proof of "not affected".
+
+---
+
+## 7. Maintenance
+
+```bash
+crowbar sync                 # fetch changed datasets (sha256 + HTTP 304)
+crowbar sync --bulk          # + OSV zips, Debian tracker, full Exploit-DB, toolkit
+crowbar build                # full rebuild, atomic swap (heavy with bulk data)
+crowbar update               # in-place incremental (~10 s)
+crowbar update --deep        # + Nuclei / MSF / ATT&CK / vendor + toolkit tables
+crowbar verify               # SQLite integrity + sha256 of every dataset
+crowbar stats                # coverage + freshness
+python3 scripts/selftest.py    # corpus health checks
+python3 scripts/check.py       # schema/catalog/docs/style consistency
+python3 -m tests.test_unit     # 111 fixture tests (also runs in CI)
+python3 -m tests.test_corpus   # generated corpus tests
+pre-commit run --all-files     # check + unit + corpus + gitleaks
+```
+
+CI (`.github/workflows/`): `ci.yml` runs lint (ruff, shellcheck, actionlint),
+check + fixture tests on Linux/macOS × Python 3.11–3.14, wheel/sdist/uvx/nix/docker
+installs through `scripts/smoke.py`, npm pack contents, and plugin/skill/MCP
+manifest validation. `upstream.yml` builds a real lite index nightly, checks what
+lite promises, and runs the auto-update end to end. Plus CodeQL, scorecard, PR title,
+dependency review and link checks.
+
+Build needs ~2× the DB size in free space (writes `crowbar.db.tmp`, then swaps).
+Do not kill an update mid-write on removable media; `crowbar verify` repairs a hot
+journal.

crowbar_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 kushvinth
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

crowbar_cli-0.1.0/Makefile

@@ -0,0 +1,62 @@
+SHELL := /bin/bash
+PY ?= python3
+HV := ./bin/crowbar
+
+.PHONY: all sync build update test test-unit test-corpus check selftest precommit \
+        doctor stats map verify link unlink clean demo
+
+all: sync build
+
+sync:
+	$(PY) -m crowbar.cli sync
+
+build:
+	$(PY) -m crowbar.cli build
+
+update:
+	$(PY) -m crowbar.cli update --deep
+
+test: test-unit test-corpus
+
+test-unit:
+	$(PY) -m tests.test_unit
+
+test-corpus:
+	$(PY) -m tests.test_corpus
+
+check:
+	$(PY) scripts/check.py
+
+selftest:
+	$(PY) scripts/selftest.py
+
+precommit:
+	pre-commit run --all-files
+
+doctor:
+	$(HV) doctor
+
+stats:
+	$(HV) stats
+
+map:
+	$(HV) map --tables
+
+verify:
+	$(HV) verify
+
+link:
+	$(HV) link $(PWD)
+
+unlink:
+	$(HV) unlink $(PWD)
+
+clean:
+	rm -rf data index/crowbar.db index/crowbar.db.tmp index/manifest.json index/state.json
+	find . -name __pycache__ -type d -prune -exec rm -rf {} +
+
+# README GIFs from the asciinema casts (needs agg: nix run nixpkgs#asciinema-agg)
+AGG ?= agg --theme monokai --font-size 16 --idle-time-limit 2 --last-frame-duration 4
+demo:
+	$(AGG) website/demo.cast docs/assets/demo.gif
+	$(AGG) assets/agent-session.cast docs/assets/agent-session.gif