vulnmirror 0.1.0__tar.gz

vulnmirror-0.1.0/LICENSE

@@ -0,0 +1,5 @@
+Copyright (C) 2026 by Nemo Xiong
+
+Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

vulnmirror-0.1.0/PKG-INFO

@@ -0,0 +1,218 @@
+Metadata-Version: 2.4
+Name: vulnmirror
+Version: 0.1.0
+Summary: A local, incrementally updated SQLite mirror of CVE Program, NVD, CISA KEV and GitHub Advisory Database metadata.
+Keywords: cve,nvd,kev,ghsa,osv,vulnerability,sqlite,mirror
+Author: Nemo Xiong
+Author-email: Nemo Xiong <nemo@anzupop.com>
+License-Expression: 0BSD
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Security
+Requires-Dist: certifi>=2024.2.2
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# vulnmirror
+
+A local, incrementally updated SQLite mirror of public vulnerability metadata:
+
+| Source | What it contributes | Incremental channel |
+|---|---|---|
+| CVE Program [`cvelistV5`](https://github.com/CVEProject/cvelistV5) | the authoritative CVE record: state, dates, CNA, description, CNA/ADP affected vendor + product, references with tags, CWE, CVSS | `cves/deltaLog.json` (about 30 days) |
+| [NVD](https://nvd.nist.gov) CVE API 2.0 feeds | NVD analysis status, CPE configurations, NVD reference tags, NVD CWE and CVSS | `modified` + `recent` feeds (8 days) |
+| [CISA KEV](https://www.cisa.gov/known-exploited-vulnerabilities-catalog) | known-exploited flag, date added | full reload (small) |
+| [GitHub Advisory Database](https://github.com/github/advisory-database) (OSV format) | package ecosystem and name, affected and fixed versions, CVE aliases, CWE, severity | `git diff` between synced commits |
+
+Everything lands in one SQLite file you can query with plain SQL. Every row records which
+provider supplied it (`cna`, `adp:<name>`, `nvd`), so CNA-reported and NVD-enriched facts
+are never silently mixed.
+
+## Requirements
+
+- Python 3.11 or newer, `curl` and `git` on `PATH`
+- Disk: about 3.2 GB of raw downloads and a 2.8 GB database (measured 2026-09-24)
+
+## Install
+
+```sh
+uv tool install vulnmirror          # or: pipx install vulnmirror
+# from a checkout:
+uv tool install --editable .
+```
+
+Or run it without installing:
+
+```sh
+uvx vulnmirror status                                 # from PyPI, in a throwaway environment
+uv run vulnmirror status                              # inside a checkout
+uv run --project /path/to/vulnmirror vulnmirror status  # a checkout, from any directory
+```
+
+Every form reads the same data directory (next section), so they can be mixed freely.
+
+## Quick start
+
+```sh
+vulnmirror init        # download everything, clone the advisory database, build, update
+vulnmirror status      # sync markers, their age, row counts
+vulnmirror update      # afterwards: incremental update, about a minute
+```
+
+`init` retries through network outages, and every download resumes where it stopped, so
+it is safe to interrupt. A first run took roughly half an hour on 2026-09-23: the NVD feeds
+took about 13 minutes (NVD serves them slowly; they are fetched in parallel), the rest went
+to cloning the advisory database and a build of under 3 minutes.
+
+## Where the data lives
+
+The data directory ("home") is resolved in this order:
+
+1. `--home PATH`
+2. `$VULNMIRROR_HOME`
+3. `home = "..."` in `$XDG_CONFIG_HOME/vulnmirror/config.toml` (default `~/.config/vulnmirror/config.toml`)
+4. `$XDG_DATA_HOME/vulnmirror` (default `~/.local/share/vulnmirror`)
+
+```sh
+vulnmirror config set-home ~/datasets/vulnmirror
+vulnmirror config show
+```
+
+Layout inside the home: `vulnmirror.sqlite`, `MANIFEST.json` (the raw snapshot), and
+`raw/` (`cvelistV5/`, `nvd/`, `kev/`, `ghsa/advisory-database/`, `cases/`).
+
+## Keeping it current
+
+```sh
+vulnmirror update                   # all sources
+vulnmirror update --only nvd,ghsa   # a subset
+```
+
+Each source is applied in its own transaction. A source that fails is rolled back and keeps
+its sync marker, so running `update` again retries the same window; running it twice in a
+row changes nothing. Run it at least weekly:
+
+- NVD older than 7 days: `update` reloads the NVD tables from the yearly feeds by itself.
+- cvelistV5 older than the delta log (about 30 days): `update` refuses; run `vulnmirror fetch`
+  and `vulnmirror build`, then `update`.
+- The advisory-database clone missing or re-cloned: `update` clones it again and reloads the
+  GHSA tables when the recorded commit is gone.
+
+A full rebuild (`vulnmirror fetch && vulnmirror build`) writes to `vulnmirror.sqlite.building`
+and renames it into place when done, so queries keep working meanwhile.
+
+## Individual records: `vulnmirror get`
+
+Fetch single records by identifier or URL, one at a time or from lists:
+
+```sh
+vulnmirror get CVE-2024-3094                                  # CVE record + NVD record
+vulnmirror get --type nvd CVE-2024-3094                       # NVD record only
+vulnmirror get https://nvd.nist.gov/vuln/detail/CVE-2021-44228
+vulnmirror get https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
+vulnmirror get -f refs.txt                                    # one reference per line, '#' comments
+cat refs.txt | vulnmirror get -f -                            # from stdin
+vulnmirror get -f refs.txt --ingest                           # also upsert into the database
+```
+
+| Reference | Record type |
+|---|---|
+| `CVE-YYYY-NNNN` | `cve` and `nvd` (narrow with `--type`) |
+| `GHSA-xxxx-xxxx-xxxx` | `ghsa` |
+| `cve.org`, `cve.mitre.org`, cvelistV5 file URLs | `cve` |
+| `nvd.nist.gov` pages and NVD API URLs | `nvd` |
+| GitHub advisory pages (global or repository), `osv.dev`, `api.osv.dev`, advisory-database file URLs | `ghsa` |
+| any other URL that contains exactly one CVE or GHSA identifier | inferred from the identifier |
+
+Records are stored as `raw/cases/<type>/<ID>.json`. An existing file is kept unless you pass
+`--force`, so re-running a long list only fetches what is missing. Every download is logged
+with its source URL in `raw/cases/index.jsonl`.
+
+- `cve` comes from cvelistV5 on GitHub.
+- `nvd` comes from the NVD CVE API 2.0. Requests are spaced 6.5 s apart; set `NVD_API_KEY` to
+  use the higher limit (0.7 s).
+- `ghsa` is the original file from github/advisory-database, located through the OSV API. If
+  the original cannot be fetched, the OSV copy is stored instead and the log says so.
+
+## Querying
+
+```sh
+vulnmirror sql "SELECT kind, count(*) FROM reference GROUP BY kind"
+vulnmirror sql "SELECT * FROM kev LIMIT 5" --format json
+vulnmirror filter --vendor examplevendor --from 2024-06-01 --to 2025-05-31 --format csv -o hits.csv
+vulnmirror filter --cpe-part h --cpe-scope any --from 2024-06-01 --to 2025-05-31 --count
+```
+
+`filter` searches the CNA/ADP `affected` table and the NVD `nvd_cpe` table together; the
+`sources` column says which one matched. Output formats: `tsv` (default), `csv`, `json`, `jsonl`.
+
+### Tables
+
+| Table | Grain | Notes |
+|---|---|---|
+| `cve` | one row per CVE ID | from cvelistV5; `state` is `PUBLISHED` or `REJECTED` |
+| `nvd` | one row per CVE ID known to NVD | `status` is NVD's analysis status |
+| `affected` | CVE × provider × product | vendor and product are free text as supplied |
+| `reference` | CVE × provider × URL | `kind` is derived from the URL; `tags` are the provider's own vocabulary |
+| `weakness`, `metric` | CVE × provider × CWE / CVSS version | |
+| `nvd_cpe` | CVE × CPE match | `part` is `a` / `o` / `h`; `vulnerable` = 1 for vulnerable matches |
+| `kev` | one row per KEV entry | |
+| `ghsa` | one row per advisory | `reviewed` = 1 for GitHub-reviewed advisories |
+| `ghsa_alias` | advisory × alias | join to `cve.id` on CVE aliases |
+| `ghsa_affected` | advisory × package | first `introduced` / `fixed` / `last_affected` event; full ranges as JSON |
+| `ghsa_reference`, `ghsa_cwe` | advisory × URL / CWE | |
+| `snapshot` | key / value | source snapshots and sync markers |
+
+### Before you count
+
+- **NVD marks device hardware as a non-vulnerable platform.** A device CVE is usually modelled as
+  vulnerable firmware (`o:*_firmware`, vulnerable = 1) running on hardware (`h`, vulnerable = 0).
+  Use `--cpe-scope any` for hardware queries.
+- **NVD CPE coverage is incomplete** for recent CVEs, so CPE-based counts are lower bounds; the
+  CNA `affected` table covers records NVD has not analysed yet.
+- **CNA vendor fields are often `n/a`**, with the vendor only in NVD's CPEs. Query both tables
+  (`filter` does).
+- **There is no device-type field.** Selecting CVEs for one kind of device means choosing vendor
+  and product criteria and checking a sample by hand.
+- **Dates differ by source.** `cve.date_published`, `nvd.published` and `ghsa.published` can be
+  hours to days apart; say which one a count uses.
+- **Only GitHub-reviewed advisories** reliably carry package ecosystems and version ranges.
+
+## Data terms
+
+vulnmirror downloads data; it does not redistribute it. Each source has its own terms:
+
+- cvelistV5: "You may search, download, and use the content hosted in this repository, per the
+  [CVE Program Terms of Use](https://www.cve.org/Legal/TermsOfUse)" (cvelistV5 README).
+- GitHub Advisory Database: [CC BY 4.0](https://github.com/github/advisory-database/blob/main/LICENSE.md);
+  attribute GitHub when you publish derived data.
+- NVD and CISA KEV: see the terms published on their sites.
+
+## License
+
+vulnmirror is released under the [BSD Zero Clause License](LICENSE) (SPDX `0BSD`). The data it
+downloads is not covered by this license; see [Data terms](#data-terms).
+
+## Development
+
+```sh
+uv sync                                        # environment with dev dependencies
+uv run pytest                                  # unit and integration tests on synthetic data, no network
+VULNMIRROR_REGRESSION=1 uv run pytest -m regression   # checks a real mirror against reference counts
+uv run ruff check src tests && uv run ruff format --check src tests
+uv build                                       # sdist and wheel in dist/
+```
+
+The regression tests compare a real mirror with counts computed independently from the NVD API
+on 2026-09-23 (for example 42,612 non-rejected CVEs published 2024-06-01 to 2025-05-31). NVD
+re-analyses records over time, so these counts carry a tolerance.

vulnmirror-0.1.0/README.md

@@ -0,0 +1,193 @@
+# vulnmirror
+
+A local, incrementally updated SQLite mirror of public vulnerability metadata:
+
+| Source | What it contributes | Incremental channel |
+|---|---|---|
+| CVE Program [`cvelistV5`](https://github.com/CVEProject/cvelistV5) | the authoritative CVE record: state, dates, CNA, description, CNA/ADP affected vendor + product, references with tags, CWE, CVSS | `cves/deltaLog.json` (about 30 days) |
+| [NVD](https://nvd.nist.gov) CVE API 2.0 feeds | NVD analysis status, CPE configurations, NVD reference tags, NVD CWE and CVSS | `modified` + `recent` feeds (8 days) |
+| [CISA KEV](https://www.cisa.gov/known-exploited-vulnerabilities-catalog) | known-exploited flag, date added | full reload (small) |
+| [GitHub Advisory Database](https://github.com/github/advisory-database) (OSV format) | package ecosystem and name, affected and fixed versions, CVE aliases, CWE, severity | `git diff` between synced commits |
+
+Everything lands in one SQLite file you can query with plain SQL. Every row records which
+provider supplied it (`cna`, `adp:<name>`, `nvd`), so CNA-reported and NVD-enriched facts
+are never silently mixed.
+
+## Requirements
+
+- Python 3.11 or newer, `curl` and `git` on `PATH`
+- Disk: about 3.2 GB of raw downloads and a 2.8 GB database (measured 2026-09-24)
+
+## Install
+
+```sh
+uv tool install vulnmirror          # or: pipx install vulnmirror
+# from a checkout:
+uv tool install --editable .
+```
+
+Or run it without installing:
+
+```sh
+uvx vulnmirror status                                 # from PyPI, in a throwaway environment
+uv run vulnmirror status                              # inside a checkout
+uv run --project /path/to/vulnmirror vulnmirror status  # a checkout, from any directory
+```
+
+Every form reads the same data directory (next section), so they can be mixed freely.
+
+## Quick start
+
+```sh
+vulnmirror init        # download everything, clone the advisory database, build, update
+vulnmirror status      # sync markers, their age, row counts
+vulnmirror update      # afterwards: incremental update, about a minute
+```
+
+`init` retries through network outages, and every download resumes where it stopped, so
+it is safe to interrupt. A first run took roughly half an hour on 2026-09-23: the NVD feeds
+took about 13 minutes (NVD serves them slowly; they are fetched in parallel), the rest went
+to cloning the advisory database and a build of under 3 minutes.
+
+## Where the data lives
+
+The data directory ("home") is resolved in this order:
+
+1. `--home PATH`
+2. `$VULNMIRROR_HOME`
+3. `home = "..."` in `$XDG_CONFIG_HOME/vulnmirror/config.toml` (default `~/.config/vulnmirror/config.toml`)
+4. `$XDG_DATA_HOME/vulnmirror` (default `~/.local/share/vulnmirror`)
+
+```sh
+vulnmirror config set-home ~/datasets/vulnmirror
+vulnmirror config show
+```
+
+Layout inside the home: `vulnmirror.sqlite`, `MANIFEST.json` (the raw snapshot), and
+`raw/` (`cvelistV5/`, `nvd/`, `kev/`, `ghsa/advisory-database/`, `cases/`).
+
+## Keeping it current
+
+```sh
+vulnmirror update                   # all sources
+vulnmirror update --only nvd,ghsa   # a subset
+```
+
+Each source is applied in its own transaction. A source that fails is rolled back and keeps
+its sync marker, so running `update` again retries the same window; running it twice in a
+row changes nothing. Run it at least weekly:
+
+- NVD older than 7 days: `update` reloads the NVD tables from the yearly feeds by itself.
+- cvelistV5 older than the delta log (about 30 days): `update` refuses; run `vulnmirror fetch`
+  and `vulnmirror build`, then `update`.
+- The advisory-database clone missing or re-cloned: `update` clones it again and reloads the
+  GHSA tables when the recorded commit is gone.
+
+A full rebuild (`vulnmirror fetch && vulnmirror build`) writes to `vulnmirror.sqlite.building`
+and renames it into place when done, so queries keep working meanwhile.
+
+## Individual records: `vulnmirror get`
+
+Fetch single records by identifier or URL, one at a time or from lists:
+
+```sh
+vulnmirror get CVE-2024-3094                                  # CVE record + NVD record
+vulnmirror get --type nvd CVE-2024-3094                       # NVD record only
+vulnmirror get https://nvd.nist.gov/vuln/detail/CVE-2021-44228
+vulnmirror get https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
+vulnmirror get -f refs.txt                                    # one reference per line, '#' comments
+cat refs.txt | vulnmirror get -f -                            # from stdin
+vulnmirror get -f refs.txt --ingest                           # also upsert into the database
+```
+
+| Reference | Record type |
+|---|---|
+| `CVE-YYYY-NNNN` | `cve` and `nvd` (narrow with `--type`) |
+| `GHSA-xxxx-xxxx-xxxx` | `ghsa` |
+| `cve.org`, `cve.mitre.org`, cvelistV5 file URLs | `cve` |
+| `nvd.nist.gov` pages and NVD API URLs | `nvd` |
+| GitHub advisory pages (global or repository), `osv.dev`, `api.osv.dev`, advisory-database file URLs | `ghsa` |
+| any other URL that contains exactly one CVE or GHSA identifier | inferred from the identifier |
+
+Records are stored as `raw/cases/<type>/<ID>.json`. An existing file is kept unless you pass
+`--force`, so re-running a long list only fetches what is missing. Every download is logged
+with its source URL in `raw/cases/index.jsonl`.
+
+- `cve` comes from cvelistV5 on GitHub.
+- `nvd` comes from the NVD CVE API 2.0. Requests are spaced 6.5 s apart; set `NVD_API_KEY` to
+  use the higher limit (0.7 s).
+- `ghsa` is the original file from github/advisory-database, located through the OSV API. If
+  the original cannot be fetched, the OSV copy is stored instead and the log says so.
+
+## Querying
+
+```sh
+vulnmirror sql "SELECT kind, count(*) FROM reference GROUP BY kind"
+vulnmirror sql "SELECT * FROM kev LIMIT 5" --format json
+vulnmirror filter --vendor examplevendor --from 2024-06-01 --to 2025-05-31 --format csv -o hits.csv
+vulnmirror filter --cpe-part h --cpe-scope any --from 2024-06-01 --to 2025-05-31 --count
+```
+
+`filter` searches the CNA/ADP `affected` table and the NVD `nvd_cpe` table together; the
+`sources` column says which one matched. Output formats: `tsv` (default), `csv`, `json`, `jsonl`.
+
+### Tables
+
+| Table | Grain | Notes |
+|---|---|---|
+| `cve` | one row per CVE ID | from cvelistV5; `state` is `PUBLISHED` or `REJECTED` |
+| `nvd` | one row per CVE ID known to NVD | `status` is NVD's analysis status |
+| `affected` | CVE × provider × product | vendor and product are free text as supplied |
+| `reference` | CVE × provider × URL | `kind` is derived from the URL; `tags` are the provider's own vocabulary |
+| `weakness`, `metric` | CVE × provider × CWE / CVSS version | |
+| `nvd_cpe` | CVE × CPE match | `part` is `a` / `o` / `h`; `vulnerable` = 1 for vulnerable matches |
+| `kev` | one row per KEV entry | |
+| `ghsa` | one row per advisory | `reviewed` = 1 for GitHub-reviewed advisories |
+| `ghsa_alias` | advisory × alias | join to `cve.id` on CVE aliases |
+| `ghsa_affected` | advisory × package | first `introduced` / `fixed` / `last_affected` event; full ranges as JSON |
+| `ghsa_reference`, `ghsa_cwe` | advisory × URL / CWE | |
+| `snapshot` | key / value | source snapshots and sync markers |
+
+### Before you count
+
+- **NVD marks device hardware as a non-vulnerable platform.** A device CVE is usually modelled as
+  vulnerable firmware (`o:*_firmware`, vulnerable = 1) running on hardware (`h`, vulnerable = 0).
+  Use `--cpe-scope any` for hardware queries.
+- **NVD CPE coverage is incomplete** for recent CVEs, so CPE-based counts are lower bounds; the
+  CNA `affected` table covers records NVD has not analysed yet.
+- **CNA vendor fields are often `n/a`**, with the vendor only in NVD's CPEs. Query both tables
+  (`filter` does).
+- **There is no device-type field.** Selecting CVEs for one kind of device means choosing vendor
+  and product criteria and checking a sample by hand.
+- **Dates differ by source.** `cve.date_published`, `nvd.published` and `ghsa.published` can be
+  hours to days apart; say which one a count uses.
+- **Only GitHub-reviewed advisories** reliably carry package ecosystems and version ranges.
+
+## Data terms
+
+vulnmirror downloads data; it does not redistribute it. Each source has its own terms:
+
+- cvelistV5: "You may search, download, and use the content hosted in this repository, per the
+  [CVE Program Terms of Use](https://www.cve.org/Legal/TermsOfUse)" (cvelistV5 README).
+- GitHub Advisory Database: [CC BY 4.0](https://github.com/github/advisory-database/blob/main/LICENSE.md);
+  attribute GitHub when you publish derived data.
+- NVD and CISA KEV: see the terms published on their sites.
+
+## License
+
+vulnmirror is released under the [BSD Zero Clause License](LICENSE) (SPDX `0BSD`). The data it
+downloads is not covered by this license; see [Data terms](#data-terms).
+
+## Development
+
+```sh
+uv sync                                        # environment with dev dependencies
+uv run pytest                                  # unit and integration tests on synthetic data, no network
+VULNMIRROR_REGRESSION=1 uv run pytest -m regression   # checks a real mirror against reference counts
+uv run ruff check src tests && uv run ruff format --check src tests
+uv build                                       # sdist and wheel in dist/
+```
+
+The regression tests compare a real mirror with counts computed independently from the NVD API
+on 2026-09-23 (for example 42,612 non-rejected CVEs published 2024-06-01 to 2025-05-31). NVD
+re-analyses records over time, so these counts carry a tolerance.

vulnmirror-0.1.0/pyproject.toml

@@ -0,0 +1,71 @@
+[project]
+name = "vulnmirror"
+version = "0.1.0"
+description = "A local, incrementally updated SQLite mirror of CVE Program, NVD, CISA KEV and GitHub Advisory Database metadata."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "0BSD"
+license-files = ["LICENSE"]
+keywords = [
+    "cve",
+    "nvd",
+    "kev",
+    "ghsa",
+    "osv",
+    "vulnerability",
+    "sqlite",
+    "mirror",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "Intended Audience :: Information Technology",
+    "Operating System :: MacOS",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Database",
+    "Topic :: Security",
+]
+dependencies = ["certifi>=2024.2.2"]
+
+[[project.authors]]
+name = "Nemo Xiong"
+email = "nemo@anzupop.com"
+
+[project.scripts]
+vulnmirror = "vulnmirror.cli:main"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "ruff>=0.6",
+]
+
+[build-system]
+requires = ["uv_build>=0.12.10,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+source-include = ["tests/**"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = ["regression: checks a real, fully built mirror against reference counts (opt in with VULNMIRROR_REGRESSION=1)"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py311"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "W",
+    "I",
+    "B",
+    "UP",
+]

vulnmirror-0.1.0/pyproject.toml.orig

@@ -0,0 +1,52 @@
+[project]
+name = "vulnmirror"
+version = "0.1.0"
+description = "A local, incrementally updated SQLite mirror of CVE Program, NVD, CISA KEV and GitHub Advisory Database metadata."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "0BSD"
+license-files = ["LICENSE"]
+authors = [{ name = "Nemo Xiong", email = "nemo@anzupop.com" }]
+keywords = ["cve", "nvd", "kev", "ghsa", "osv", "vulnerability", "sqlite", "mirror"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "Intended Audience :: Information Technology",
+    "Operating System :: MacOS",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Database",
+    "Topic :: Security",
+]
+dependencies = ["certifi>=2024.2.2"]
+
+[project.scripts]
+vulnmirror = "vulnmirror.cli:main"
+
+[dependency-groups]
+dev = ["pytest>=8.0", "ruff>=0.6"]
+
+[build-system]
+requires = ["uv_build>=0.12.10,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+# Ship the test suite in the sdist so downstream packagers can run it.
+source-include = ["tests/**"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+    "regression: checks a real, fully built mirror against reference counts (opt in with VULNMIRROR_REGRESSION=1)",
+]
+
+[tool.ruff]
+line-length = 120
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP"]

vulnmirror-0.1.0/src/vulnmirror/__init__.py

@@ -0,0 +1 @@
+"""vulnmirror: a local, incrementally updated SQLite mirror of CVE, NVD, KEV and GHSA metadata."""

vulnmirror-0.1.0/src/vulnmirror/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from vulnmirror.cli import main
+
+sys.exit(main())

vulnmirror-0.1.0/src/vulnmirror/build.py

@@ -0,0 +1,105 @@
+"""Full build of <home>/vulnmirror.sqlite from the raw mirrors listed in <home>/MANIFEST.json.
+
+The database is written to vulnmirror.sqlite.building and renamed into place only when
+complete, so queries against the previous database keep working during a rebuild.
+"""
+
+import json
+import sqlite3
+import zipfile
+from pathlib import Path
+
+from vulnmirror import ghsa, records
+from vulnmirror.config import Paths
+
+
+def iter_cvelist_export(paths: Paths, asset: str):
+    """Yield CVE JSON 5 records from the daily export (a zip that wraps a zip)."""
+    with zipfile.ZipFile(paths.cvelist_dir / asset) as outer:
+        inner_names = [n for n in outer.namelist() if n.endswith(".zip")]
+        if inner_names:
+            inner_path = paths.cvelist_dir / "_inner.zip"
+            if not inner_path.exists() or inner_path.stat().st_size != outer.getinfo(inner_names[0]).file_size:
+                with outer.open(inner_names[0]) as src, open(inner_path, "wb") as dst:
+                    while chunk := src.read(1 << 20):
+                        dst.write(chunk)
+            z = zipfile.ZipFile(inner_path)
+        else:
+            z = outer
+        with z:
+            for name in z.namelist():
+                base = name.rsplit("/", 1)[-1]
+                if base.startswith("CVE-") and base.endswith(".json"):
+                    yield json.loads(z.read(name))
+
+
+def export_cutoff(asset_name: str) -> str:
+    """'2026-09-23_all_CVEs_at_midnight.zip.zip' -> '2026-09-23T00:00:00Z'."""
+    return asset_name[:10] + "T00:00:00Z"
+
+
+def build(paths: Paths, log=print) -> dict:
+    if not paths.manifest.exists():
+        raise FileNotFoundError(f"no {paths.manifest}; run `vulnmirror fetch` first")
+    manifest = json.loads(paths.manifest.read_text())
+    tmp = Path(str(paths.db) + ".building")
+    tmp.unlink(missing_ok=True)
+    db = sqlite3.connect(tmp)
+    db.executescript(records.sql_text("schema.sql"))
+    db.execute("PRAGMA journal_mode=OFF")
+    db.execute("PRAGMA synchronous=OFF")
+
+    cl = manifest["cvelistV5"]
+    log(f"cvelistV5 {cl['release_tag']}")
+    n_cl = 0
+    for rec in iter_cvelist_export(paths, cl["asset"]):
+        if records.insert_cvelist_record(db, rec):
+            n_cl += 1
+            if n_cl % 100000 == 0:
+                log(f"  cvelistV5: {n_cl} records")
+    db.commit()
+
+    # 'modified' and 'recent' first: their copy of a CVE is the newest, and the first copy wins.
+    feeds = [paths.nvd_dir / f"nvdcve-2.0-{n}.json.gz" for n in ("modified", "recent")]
+    feeds = [f for f in feeds if f.exists()] + sorted(paths.nvd_dir.glob("nvdcve-2.0-[0-9]*.json.gz"))
+    seen, n_nvd = set(), 0
+    for f in feeds:
+        for c in records.iter_nvd_feed(f):
+            if c["id"] in seen:
+                continue
+            seen.add(c["id"])
+            records.insert_nvd_record(db, c)
+            n_nvd += 1
+        db.commit()
+        log(f"  nvd: {f.name} done ({n_nvd} total)")
+
+    n_kev = records.load_kev(db, paths.kev_file) if paths.kev_file.exists() else 0
+
+    n_ghsa, commit = 0, ""
+    if paths.ghsa_repo.exists():
+        n_ghsa, commit = ghsa.load_all(db, paths.ghsa_repo, log=log)
+        log(f"  ghsa: {n_ghsa} advisories @ {commit[:12]}")
+
+    nvd_feeds = {f["feed"]: f["lastModifiedDate"] for f in manifest.get("nvd", [])}
+    state = dict(
+        cvelistV5_release=cl["release_tag"],
+        cvelist_synced_through=export_cutoff(cl["asset"]),
+        nvd_synced_at=nvd_feeds.get("modified", ""),
+        nvd_feeds=json.dumps(nvd_feeds),
+        kev_catalog=manifest.get("kev", {}).get("catalogVersion", ""),
+        fetched_at=manifest["fetched_at"],
+        n_cvelist=n_cl,
+        n_nvd=n_nvd,
+        n_kev=n_kev,
+        n_ghsa=n_ghsa,
+    )
+    if commit:
+        state["ghsa_commit"] = commit
+    records.set_state(db, **state)
+    db.executescript(records.sql_text("indexes.sql"))
+    db.commit()
+    db.close()
+    tmp.replace(paths.db)
+    summary = {"cvelist": n_cl, "nvd": n_nvd, "kev": n_kev, "ghsa": n_ghsa}
+    log(f"built {paths.db}: " + " ".join(f"{k}={v}" for k, v in summary.items()))
+    return summary