threatwatch 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- threatwatch-1.0.0/CHANGELOG.md +46 -0
- threatwatch-1.0.0/LICENSE +21 -0
- threatwatch-1.0.0/MANIFEST.in +6 -0
- threatwatch-1.0.0/PKG-INFO +496 -0
- threatwatch-1.0.0/README.md +473 -0
- threatwatch-1.0.0/SECURITY.md +77 -0
- threatwatch-1.0.0/config.example.json +211 -0
- threatwatch-1.0.0/pyproject.toml +37 -0
- threatwatch-1.0.0/setup.cfg +4 -0
- threatwatch-1.0.0/tests/test_core.py +534 -0
- threatwatch-1.0.0/threatwatch/__init__.py +8 -0
- threatwatch-1.0.0/threatwatch/__main__.py +6 -0
- threatwatch-1.0.0/threatwatch/cli.py +592 -0
- threatwatch-1.0.0/threatwatch/config.py +228 -0
- threatwatch-1.0.0/threatwatch/correlate.py +478 -0
- threatwatch-1.0.0/threatwatch/extract.py +160 -0
- threatwatch-1.0.0/threatwatch/http.py +352 -0
- threatwatch-1.0.0/threatwatch/models.py +181 -0
- threatwatch-1.0.0/threatwatch/opsec.py +718 -0
- threatwatch-1.0.0/threatwatch/progress.py +323 -0
- threatwatch-1.0.0/threatwatch/report.py +971 -0
- threatwatch-1.0.0/threatwatch/scoring.py +264 -0
- threatwatch-1.0.0/threatwatch/sources.py +633 -0
- threatwatch-1.0.0/threatwatch/state.py +99 -0
- threatwatch-1.0.0/threatwatch/summarize.py +386 -0
- threatwatch-1.0.0/threatwatch/taxonomy.py +591 -0
- threatwatch-1.0.0/threatwatch/utils.py +193 -0
- threatwatch-1.0.0/threatwatch.egg-info/PKG-INFO +496 -0
- threatwatch-1.0.0/threatwatch.egg-info/SOURCES.txt +30 -0
- threatwatch-1.0.0/threatwatch.egg-info/dependency_links.txt +1 -0
- threatwatch-1.0.0/threatwatch.egg-info/entry_points.txt +2 -0
- threatwatch-1.0.0/threatwatch.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to ThreatWatch are documented here.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [1.0.0] - 2026-11-14
|
|
9
|
+
|
|
10
|
+
Initial public release.
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- Live collection from CISA KEV, NVD 2.0, FIRST EPSS, GitHub Security Advisories,
|
|
15
|
+
and curated RSS/Atom feeds.
|
|
16
|
+
- Defensive-only scope: no exploit code, scanning, credential use, or offensive
|
|
17
|
+
capability. Extracted indicators are defanged and treated as unvalidated.
|
|
18
|
+
- Explainable additive risk scoring with CVSS, EPSS, KEV, ransomware, recency,
|
|
19
|
+
AI relevance, and asset exposure. KEV floors and tier-based remediation SLAs.
|
|
20
|
+
- Audience profiles for baseline, CISO, SOC, incident response, and AI platform
|
|
21
|
+
teams.
|
|
22
|
+
- Markdown, single-file HTML, JSON, and CSV reports with KPI dashboards, priority
|
|
23
|
+
actions, vulnerability detail, KEV watch, AI/LLM threat mapping, campaign
|
|
24
|
+
intelligence, OPSEC controls, detection priorities, and IOC appendices.
|
|
25
|
+
- AI/LLM taxonomy mapped to OWASP Top 10 for LLM Applications (2025) and MITRE
|
|
26
|
+
ATLAS.
|
|
27
|
+
- "Recently Patched / Fix Available" view with NVD patch references, explicit
|
|
28
|
+
fix language, and GitHub fixed-version metadata. Includes `--patched-only`
|
|
29
|
+
filtering and patch fields in JSON/CSV exports.
|
|
30
|
+
- Live weighted progress bar for interactive terminals, with plain milestone
|
|
31
|
+
output for cron/CI, `--quiet`, `--no-progress`, and `--no-color` controls.
|
|
32
|
+
- Persistent HTTP cache with offline fallback, retry/backoff, and polite rate
|
|
33
|
+
limiting for unauthenticated NVD and GitHub API access.
|
|
34
|
+
- Watch state and `--diff-only` reporting for new items since the previous run.
|
|
35
|
+
- CI gating with `--fail-on p1|p2|p3` and strict source-health checks.
|
|
36
|
+
- Optional LLM narrative polish with configurable endpoint and privacy controls.
|
|
37
|
+
- `enrich` command for single-CVE deep dives, and `opsec` command for a defensive
|
|
38
|
+
control baseline.
|
|
39
|
+
- Standard-library-only implementation for Python 3.11+.
|
|
40
|
+
|
|
41
|
+
### Security
|
|
42
|
+
|
|
43
|
+
- API keys and tokens are read from environment variables and redacted by
|
|
44
|
+
`config show`.
|
|
45
|
+
- Generated reports may contain sensitive threat intelligence; the tool includes
|
|
46
|
+
classification markings and defanged indicators to support safe handling.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 ghoste12125
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,496 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: threatwatch
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Live defensive cyber and AI threat intelligence reports from public sources
|
|
5
|
+
Author-email: ghoste12125 <anonymosmauros@gmail.com>
|
|
6
|
+
Maintainer-email: ghoste12125 <anonymosmauros@gmail.com>
|
|
7
|
+
License-Expression: MIT
|
|
8
|
+
Keywords: threat-intelligence,cybersecurity,ai-security,kev,epss,opsec
|
|
9
|
+
Classifier: Environment :: Console
|
|
10
|
+
Classifier: Intended Audience :: Information Technology
|
|
11
|
+
Classifier: Intended Audience :: System Administrators
|
|
12
|
+
Classifier: Operating System :: OS Independent
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
18
|
+
Classifier: Topic :: Security
|
|
19
|
+
Requires-Python: >=3.11
|
|
20
|
+
Description-Content-Type: text/markdown
|
|
21
|
+
License-File: LICENSE
|
|
22
|
+
Dynamic: license-file
|
|
23
|
+
|
|
24
|
+
# ThreatWatch
|
|
25
|
+
|
|
26
|
+
Live, defensive cyber and AI threat intelligence from public sources, rendered as a professional report for executives, SOC, incident response, and AI platform owners.
|
|
27
|
+
|
|
28
|
+
ThreatWatch collects exploitation evidence and reporting, scores what matters, explains every score, and produces Markdown, single-file HTML, JSON, and CSV. It is designed for repeatable use in a terminal, cron job, systemd timer, or CI pipeline.
|
|
29
|
+
|
|
30
|
+
> **Scope:** defensive intelligence only. ThreatWatch does not include exploit code, scanning, credential use, or offensive capability. All indicators are defanged and treated as unvalidated public reporting.
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## Contents
|
|
35
|
+
|
|
36
|
+
- [Quickstart](#quickstart)
|
|
37
|
+
- [Commands](#commands)
|
|
38
|
+
- [Live progress](#live-progress)
|
|
39
|
+
- [Profiles](#profiles)
|
|
40
|
+
- [Report sections](#report-sections)
|
|
41
|
+
- [Recently patched / fix available](#recently-patched--fix-available)
|
|
42
|
+
- [How prioritisation works](#how-prioritisation-works)
|
|
43
|
+
- [AI and LLM taxonomy](#ai-and-llm-taxonomy)
|
|
44
|
+
- [Sources](#sources)
|
|
45
|
+
- [Configuration](#configuration)
|
|
46
|
+
- [CI gating](#ci-gating)
|
|
47
|
+
- [Scheduling](#scheduling)
|
|
48
|
+
- [LLM polish and privacy](#llm-polish-and-privacy)
|
|
49
|
+
- [Caching, state, and offline use](#caching-state-and-offline-use)
|
|
50
|
+
- [Limitations and responsible use](#limitations-and-responsible-use)
|
|
51
|
+
- [Security](#security)
|
|
52
|
+
- [Changelog](#changelog)
|
|
53
|
+
- [Development](#development)
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
## Quickstart
|
|
58
|
+
|
|
59
|
+
Requirements: Python **3.11+** (standard library only, no third-party packages).
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
# Install the command from PyPI.
|
|
63
|
+
python3 -m pip install threatwatch
|
|
64
|
+
|
|
65
|
+
# Optional: create a local config and edit assets, thresholds, and org details.
|
|
66
|
+
threatwatch config init
|
|
67
|
+
|
|
68
|
+
# Collect, score, and write the default Markdown report.
|
|
69
|
+
threatwatch run
|
|
70
|
+
|
|
71
|
+
# Or write every format at once.
|
|
72
|
+
threatwatch run --format all --outdir reports
|
|
73
|
+
|
|
74
|
+
# See what was produced.
|
|
75
|
+
ls reports/
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Reports are written as `threatwatch-YYYYMMDD-HHMMSSZ.<ext>` plus `latest.<ext>` unless `report.write_latest` is disabled.
|
|
79
|
+
|
|
80
|
+
From a source checkout, use `python3 -m threatwatch` instead of `threatwatch`, or install the checkout in editable mode:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
cd threatwatch
|
|
84
|
+
python3 -m pip install -e .
|
|
85
|
+
threatwatch run
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
No network access is required for `--offline` runs after a cache has been populated.
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## Commands
|
|
93
|
+
|
|
94
|
+
| Command | Purpose |
|
|
95
|
+
|---|---|
|
|
96
|
+
| `run` (alias `report`) | Collect, correlate, score, and write a full report. |
|
|
97
|
+
| `watch` | Report only items new since the previous run; exits non-zero on new P1/P2 by default. |
|
|
98
|
+
| `feeds` | List configured feeds or check feed/API health. |
|
|
99
|
+
| `opsec` | Print the defensive OPSEC control baseline in Markdown, JSON, or CSV. |
|
|
100
|
+
| `enrich CVE-YYYY-NNNN` | Deep-dive one CVE with KEV, EPSS, AI mapping, and remediation context. |
|
|
101
|
+
| `config init\|show\|validate` | Create, inspect, or validate configuration. |
|
|
102
|
+
| `version` | Print the version. |
|
|
103
|
+
|
|
104
|
+
Running `python3 -m threatwatch` with no arguments is equivalent to `run`.
|
|
105
|
+
|
|
106
|
+
Common examples:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
# All formats, named output directory.
|
|
110
|
+
python3 -m threatwatch run --format all --outdir reports
|
|
111
|
+
|
|
112
|
+
# Executive view for the last 14 days.
|
|
113
|
+
python3 -m threatwatch run --profile ciso --since-days 14 --format md --format html
|
|
114
|
+
|
|
115
|
+
# Only AI/LLM-relevant items.
|
|
116
|
+
python3 -m threatwatch run --ai-only --profile ai-platform
|
|
117
|
+
|
|
118
|
+
# Only vulnerabilities with patch or fixed-version evidence.
|
|
119
|
+
python3 -m threatwatch run --patched-only --format md --format html
|
|
120
|
+
|
|
121
|
+
# New items only (the watch command does this automatically).
|
|
122
|
+
python3 -m threatwatch run --diff-only --format json --output -
|
|
123
|
+
|
|
124
|
+
# Single CVE deep dive.
|
|
125
|
+
python3 -m threatwatch enrich CVE-2024-3400
|
|
126
|
+
|
|
127
|
+
# OPSEC controls for a SOC.
|
|
128
|
+
python3 -m threatwatch opsec --profile soc --format md --output opsec-soc.md
|
|
129
|
+
|
|
130
|
+
# Check sources before relying on a scheduled run.
|
|
131
|
+
python3 -m threatwatch feeds --check-apis
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Useful run flags:
|
|
135
|
+
|
|
136
|
+
| Flag | Effect |
|
|
137
|
+
|---|---|
|
|
138
|
+
| `--format md,html,json,csv,all` | Output formats; repeatable. Default `md`. |
|
|
139
|
+
| `--outdir DIR` | Output directory. Default `reports`. |
|
|
140
|
+
| `--output FILE` / `--output -` | Single file, or stdout. |
|
|
141
|
+
| `--since-days N` | CVE window. |
|
|
142
|
+
| `--kev-days N` | KEV addition window. |
|
|
143
|
+
| `--news-hours N` | Reporting window. |
|
|
144
|
+
| `--top N` | Rows in vulnerability tables. |
|
|
145
|
+
| `--min-tier P1..P4` | Show this tier or higher. |
|
|
146
|
+
| `--ai-only` | Keep only AI/LLM-relevant vulnerabilities and reporting. |
|
|
147
|
+
| `--patched-only` | Keep only vulnerabilities with detected patch or fixed-version evidence. |
|
|
148
|
+
| `--diff-only` | Only items new since the previous run. |
|
|
149
|
+
| `--all-kev` | Include the full KEV catalog, not just recent/relevant entries. |
|
|
150
|
+
| `--no-feeds`, `--no-nvd`, `--no-ghsa` | Skip a collection source. |
|
|
151
|
+
| `--profile baseline\|ciso\|soc\|ir\|ai-platform` | Audience and OPSEC profile. |
|
|
152
|
+
| `--fail-on p1\|p2\|p3\|none` | CI gate; exits `2` when the threshold is met. |
|
|
153
|
+
| `--no-state`, `--state-file PATH` | Disable or relocate watch state. |
|
|
154
|
+
| `--offline`, `--refresh`, `--no-cache` | Cache behaviour. |
|
|
155
|
+
| `--org`, `--classification` | Override report metadata. |
|
|
156
|
+
| `--llm` / `--no-llm` | Opt in/out of configured LLM polish. |
|
|
157
|
+
| `--print` | Also print the report to stdout. |
|
|
158
|
+
| `--verbose` | Show fetch retries and diagnostics. |
|
|
159
|
+
| `--quiet` | Suppress all progress and status output (ideal for cron/CI logs). |
|
|
160
|
+
| `--no-progress` | Disable the live progress bar but keep normal status lines. |
|
|
161
|
+
| `--no-color` | Disable ANSI colour. `NO_COLOR=1` is also honoured. |
|
|
162
|
+
|
|
163
|
+
Exit codes: `0` success, `1` usage/runtime error, `2` CI gate or strict source-health failure.
|
|
164
|
+
|
|
165
|
+
### Live progress
|
|
166
|
+
|
|
167
|
+
On an interactive terminal, collection shows a single animated bar with per-stage and sub-stage updates: KEV catalog, NVD pages, GitHub advisories, news feeds, EPSS batches, and scoring. The bar is drawn on `stderr`, so report output (`--output -`) stays clean.
|
|
168
|
+
|
|
169
|
+
When output is redirected (cron, CI, logs), ThreatWatch automatically falls back to plain stage and milestone lines instead of ANSI animation. Use `--quiet` to silence it entirely, or `--no-progress` to keep status messages but skip the bar.
|
|
170
|
+
|
|
171
|
+
```bash
|
|
172
|
+
# Interactive: animated bar.
|
|
173
|
+
python3 -m threatwatch run --format all
|
|
174
|
+
|
|
175
|
+
# CI: plain, timestamp-friendly lines; no ANSI codes.
|
|
176
|
+
NO_COLOR=1 python3 -m threatwatch watch
|
|
177
|
+
|
|
178
|
+
# Cron: no progress at all.
|
|
179
|
+
python3 -m threatwatch watch --quiet
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## Profiles
|
|
185
|
+
|
|
186
|
+
Profiles change emphasis, narrative, and which OPSEC controls are included. They do not change the underlying scoring.
|
|
187
|
+
|
|
188
|
+
| Profile | Audience and emphasis |
|
|
189
|
+
|---|---|
|
|
190
|
+
| `baseline` | Balanced technical and management view. |
|
|
191
|
+
| `ciso` | Exposure, deadlines, accountability, and decision-ready summaries. |
|
|
192
|
+
| `soc` | Detection, KEV/EPSS triage, hunting leads, and operational queues. |
|
|
193
|
+
| `ir` | Exposure verification, evidence preservation, and containment for confirmed exploitation. |
|
|
194
|
+
| `ai-platform` | Model serving, agent permissions, retrieval boundaries, and artifact integrity. |
|
|
195
|
+
|
|
196
|
+
OPSEC controls are tagged `Baseline`, `Enhanced`, or `Advanced`. Profiles select the relevant subset; `--advanced` / `--no-advanced` override inclusion.
|
|
197
|
+
|
|
198
|
+
---
|
|
199
|
+
|
|
200
|
+
## Report sections
|
|
201
|
+
|
|
202
|
+
- Posture KPIs and bottom line
|
|
203
|
+
- Executive summary and key findings
|
|
204
|
+
- Priority actions with owners, tiers, and due dates
|
|
205
|
+
- Vulnerability priorities with explainable risk drivers
|
|
206
|
+
- Recently patched / fix available, sorted by most recent source modification
|
|
207
|
+
- CISA KEV watch, including ransomware and asset matches
|
|
208
|
+
- AI/LLM threat landscape mapped to OWASP and MITRE ATLAS
|
|
209
|
+
- Exploit and campaign intelligence
|
|
210
|
+
- OPSEC controls, quick wins, assurance questions, and metrics
|
|
211
|
+
- Detection priorities
|
|
212
|
+
- Defanged indicator appendix
|
|
213
|
+
- SLA and metrics table
|
|
214
|
+
- Methodology, confidence, and limitations
|
|
215
|
+
- Source health and collection warnings
|
|
216
|
+
- New-since-last-run delta when state is enabled
|
|
217
|
+
|
|
218
|
+
The HTML export is a single self-contained file with inline CSS/JS, filtering, a theme toggle, and print/PDF support.
|
|
219
|
+
|
|
220
|
+
### Recently patched / fix available
|
|
221
|
+
|
|
222
|
+
ThreatWatch infers patch evidence conservatively from public metadata:
|
|
223
|
+
|
|
224
|
+
- NVD references tagged `Patch`.
|
|
225
|
+
- Explicit fix language in NVD descriptions, such as `fixed in`, `patched in`, `security update`, or `upgrade to`.
|
|
226
|
+
- GitHub Security Advisory `first_patched_version` and `patched_versions` fields.
|
|
227
|
+
|
|
228
|
+
The **Recently Patched / Fix Available** section shows these records sorted by the most recent source modification date (NVD `lastModified` or GHSA `updated_at`), with the fixed version, tier, risk, and KEV status. Use `--patched-only` to restrict the main vulnerability table and AI sections to the same records. CSV and JSON exports include `patch_available`, `patched_versions`, and `patch_urls`.
|
|
229
|
+
|
|
230
|
+
Patch detection is a metadata heuristic, not confirmation that a safe update exists for your deployment. Verify with the vendor advisory before relying on it.
|
|
231
|
+
|
|
232
|
+
---
|
|
233
|
+
|
|
234
|
+
## How prioritisation works
|
|
235
|
+
|
|
236
|
+
Every vulnerability gets an additive, explainable risk score (0–100):
|
|
237
|
+
|
|
238
|
+
```text
|
|
239
|
+
risk = clamp(3.0 × CVSS
|
|
240
|
+
+ 20 × EPSS
|
|
241
|
+
+ 22 × KEV
|
|
242
|
+
+ 6 × ransomware
|
|
243
|
+
+ 8 × recency
|
|
244
|
+
+ 12 × AI relevance
|
|
245
|
+
+ 8 × asset exposure)
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
- **Recency** decays with a 21-day half-life.
|
|
249
|
+
- **KEV floors** ensure confirmed exploitation is never buried: `72` baseline, `82` with known ransomware, `86` with strong AI relevance, `88` when a declared asset matches.
|
|
250
|
+
- **Tiers** come from configurable thresholds: P1 ≥ 80, P2 ≥ 62, P3 ≥ 42, otherwise P4.
|
|
251
|
+
- **SLA hours** are P1 24h, P2 72h, P3 14 days, P4 30 days, capped by the CISA KEV due date where one exists.
|
|
252
|
+
- **Explanations** are stored on each record and shown in tables, cards, JSON, and `enrich`.
|
|
253
|
+
|
|
254
|
+
Asset matching is keyword-based against the `assets` list in configuration. It is intentionally simple and should be tuned to your real inventory.
|
|
255
|
+
|
|
256
|
+
---
|
|
257
|
+
|
|
258
|
+
## AI and LLM taxonomy
|
|
259
|
+
|
|
260
|
+
AI relevance requires strong indicators, not generic words such as “AI”, “model”, or “agent”. Categories map to:
|
|
261
|
+
|
|
262
|
+
- **OWASP Top 10 for LLM Applications (2025)** — e.g. LLM01 Prompt Injection, LLM03 Supply Chain, LLM06 Excessive Agency, LLM07 System Prompt Leakage.
|
|
263
|
+
- **MITRE ATLAS** — verified technique IDs such as AML.T0051 (LLM Prompt Injection), AML.T0010 (AI Supply Chain Compromise), AML.T0040 (AI Model Inference API Access), and AML.T0110 (AI Agent Tool Poisoning).
|
|
264
|
+
|
|
265
|
+
The classifier combines indicator phrases, context guards (for example, avoiding “RAG” false positives), and AI-stack inventory terms.
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
## Sources
|
|
270
|
+
|
|
271
|
+
| Source | Role |
|
|
272
|
+
|---|---|
|
|
273
|
+
| CISA Known Exploited Vulnerabilities (KEV) | Authoritative confirmed exploitation and due dates. |
|
|
274
|
+
| NIST NVD 2.0 | CVSS, CWE, CPE, descriptions, references. |
|
|
275
|
+
| FIRST EPSS | 30-day exploitation probability and percentile. |
|
|
276
|
+
| GitHub Security Advisories | Open-source package advisories and GHSA identifiers. |
|
|
277
|
+
| Curated RSS/Atom feeds | Vendor research, incident reporting, and AI security commentary. |
|
|
278
|
+
|
|
279
|
+
`python3 -m threatwatch feeds --list` shows built-in feed IDs. Add your own under `feeds.custom`. The default enabled set favours established security publishers and AI-security sources; tune it to your sector.
|
|
280
|
+
|
|
281
|
+
NVD unauthenticated access is rate-limited. Set `NVD_API_KEY` to raise limits. Set `GITHUB_TOKEN` for higher GitHub API limits. Both are optional.
|
|
282
|
+
|
|
283
|
+
---
|
|
284
|
+
|
|
285
|
+
## Configuration
|
|
286
|
+
|
|
287
|
+
Configuration is JSON, merged over built-in defaults. Discovery order:
|
|
288
|
+
|
|
289
|
+
1. `--config PATH`
|
|
290
|
+
2. `THREATWATCH_CONFIG` environment variable
|
|
291
|
+
3. `.threatwatch/config.json`
|
|
292
|
+
4. `config.json` in the current directory
|
|
293
|
+
|
|
294
|
+
Start from the documented example:
|
|
295
|
+
|
|
296
|
+
```bash
|
|
297
|
+
python3 -m threatwatch config init # writes .threatwatch/config.json
|
|
298
|
+
python3 -m threatwatch config show # effective config, secrets redacted
|
|
299
|
+
python3 -m threatwatch config validate # warnings and feed checks
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
`config.example.json` in this repository mirrors the default configuration and can be copied manually.
|
|
303
|
+
|
|
304
|
+
Key sections:
|
|
305
|
+
|
|
306
|
+
| Section | Purpose |
|
|
307
|
+
|---|---|
|
|
308
|
+
| `org` | Name, sector, classification marking, timezone. |
|
|
309
|
+
| `assets` | Keyword-based inventory with criticality and exposure. |
|
|
310
|
+
| `ai_stack` | Providers, frameworks, servers, vector DBs, and AI infrastructure terms. |
|
|
311
|
+
| `feeds` | Enabled built-in feeds, custom feeds, per-feed item cap. |
|
|
312
|
+
| `windows` | Default CVE, KEV, and news windows. |
|
|
313
|
+
| `thresholds`, `sla_hours` | Tier boundaries and remediation SLAs. |
|
|
314
|
+
| `report` | Top-N, patched-section cap, IOC/OPSEC inclusion, theme, profile, latest-file behaviour. |
|
|
315
|
+
| `cache`, `state` | Cache TTLs, cache age, and watch-state location. |
|
|
316
|
+
| `nvd`, `epss`, `ghsa`, `http` | Rate limits, tokens, retries, timeout, user agent. |
|
|
317
|
+
| `llm` | Optional local/remote LLM endpoint and privacy controls. |
|
|
318
|
+
| `opsec` | Advanced-control inclusion and default profile. |
|
|
319
|
+
|
|
320
|
+
---
|
|
321
|
+
|
|
322
|
+
## CI gating
|
|
323
|
+
|
|
324
|
+
Use `--fail-on` to turn intelligence into a release or deployment gate. A gate failure exits `2`.
|
|
325
|
+
|
|
326
|
+
```bash
|
|
327
|
+
# Fail if any P1 is present.
|
|
328
|
+
python3 -m threatwatch run --format json --output threatwatch.json --fail-on p1
|
|
329
|
+
|
|
330
|
+
# Watch mode defaults to failing on newly seen P1 or P2 items.
|
|
331
|
+
python3 -m threatwatch watch --format md --outdir reports
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
Example GitHub Actions step:
|
|
335
|
+
|
|
336
|
+
```yaml
|
|
337
|
+
- name: Threat intelligence gate
|
|
338
|
+
env:
|
|
339
|
+
NVD_API_KEY: ${{ secrets.NVD_API_KEY }}
|
|
340
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
341
|
+
run: |
|
|
342
|
+
python -m pip install .
|
|
343
|
+
threatwatch run --format json --output threatwatch.json --fail-on p1
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
Recommended pattern: run `watch` on a schedule, archive JSON as an artifact, and keep the hard gate on P1. Avoid failing builds on P3/P4 noise unless your risk appetite justifies it.
|
|
347
|
+
|
|
348
|
+
---
|
|
349
|
+
|
|
350
|
+
## Scheduling
|
|
351
|
+
|
|
352
|
+
Cron example (every four hours, new items only):
|
|
353
|
+
|
|
354
|
+
```cron
|
|
355
|
+
0 */4 * * * cd /opt/threatwatch && /usr/bin/python3 -m threatwatch watch --quiet --outdir /var/lib/threatwatch/reports >> /var/log/threatwatch.log 2>&1
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
A systemd timer is preferable for jitter, logging, and restart behaviour:
|
|
359
|
+
|
|
360
|
+
```ini
|
|
361
|
+
# /etc/systemd/system/threatwatch.service
|
|
362
|
+
[Unit]
|
|
363
|
+
Description=ThreatWatch intelligence report
|
|
364
|
+
After=network-online.target
|
|
365
|
+
|
|
366
|
+
[Service]
|
|
367
|
+
Type=oneshot
|
|
368
|
+
WorkingDirectory=/opt/threatwatch
|
|
369
|
+
Environment=NVD_API_KEY=...
|
|
370
|
+
ExecStart=/usr/bin/python3 -m threatwatch watch --quiet --outdir /var/lib/threatwatch/reports
|
|
371
|
+
User=threatwatch
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
```ini
|
|
375
|
+
# /etc/systemd/system/threatwatch.timer
|
|
376
|
+
[Unit]
|
|
377
|
+
Description=Run ThreatWatch every four hours
|
|
378
|
+
|
|
379
|
+
[Timer]
|
|
380
|
+
OnCalendar=00/4:00
|
|
381
|
+
RandomizedDelaySec=10m
|
|
382
|
+
Persistent=true
|
|
383
|
+
|
|
384
|
+
[Install]
|
|
385
|
+
WantedBy=timers.target
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
```bash
|
|
389
|
+
sudo systemctl daemon-reload
|
|
390
|
+
sudo systemctl enable --now threatwatch.timer
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
For a daily full report plus a frequent watch, use two units with different `--outdir` values.
|
|
394
|
+
|
|
395
|
+
---
|
|
396
|
+
|
|
397
|
+
## LLM polish and privacy
|
|
398
|
+
|
|
399
|
+
LLM polish is **off by default**. When enabled with `--llm`, ThreatWatch sends **aggregate metrics only** — counts, tiers, category names, and short risk summaries — not raw reporting text, indicators, or asset details. Organisation name can be redacted with `llm.redact_org`.
|
|
400
|
+
|
|
401
|
+
The default endpoint is local Ollama (`http://127.0.0.1:11434`). If you point it at a remote provider, review data-handling and residency requirements first. The report always has a deterministic fallback, so LLM failure does not block generation.
|
|
402
|
+
|
|
403
|
+
For sensitive environments, leave LLM polish disabled and use the deterministic narrative.
|
|
404
|
+
|
|
405
|
+
---
|
|
406
|
+
|
|
407
|
+
## Caching, state, and offline use
|
|
408
|
+
|
|
409
|
+
- HTTP responses are cached under `.threatwatch/cache` with per-source TTLs.
|
|
410
|
+
- `--refresh` bypasses TTLs; `--no-cache` disables reads and writes; `--offline` uses cache only.
|
|
411
|
+
- For time-windowed APIs (NVD, KEV, EPSS, GHSA), offline mode falls back to the most recent cached response for the same endpoint and query shape, and labels it stale rather than dropping the source.
|
|
412
|
+
- Stale cache is labelled in Source Health rather than silently presented as current.
|
|
413
|
+
- Watch state lives in `.threatwatch/state.json`. `--diff-only` / `watch` report new items; `--no-state` leaves state untouched.
|
|
414
|
+
- Failed fetches are retried with rate limiting and recorded as collection warnings.
|
|
415
|
+
|
|
416
|
+
The cache is not a substitute for a controlled archive. For audit purposes, retain generated JSON reports alongside their timestamps.
|
|
417
|
+
|
|
418
|
+
---
|
|
419
|
+
|
|
420
|
+
## Limitations and responsible use
|
|
421
|
+
|
|
422
|
+
- Public reporting can be incomplete, delayed, duplicated, or wrong. Validate before emergency change.
|
|
423
|
+
- KEV is authoritative for known exploitation; EPSS is probabilistic; NVD enrichment can lag.
|
|
424
|
+
- Asset matching is keyword-based and may miss or over-match. Tune `assets` and review results.
|
|
425
|
+
- AI tagging is a triage aid, not proof of exploitability or attribution. Attribution is not assessed.
|
|
426
|
+
- Indicators are automatically extracted, defanged, and unvalidated. Do not block on a single low-confidence indicator.
|
|
427
|
+
- The tool does not replace vendor advisories, EDR telemetry, vulnerability scanners, or incident response.
|
|
428
|
+
- Reports may contain security-sensitive content. Apply the classification marking, restrict distribution, and defang before external sharing.
|
|
429
|
+
|
|
430
|
+
ThreatWatch is intended for defensive security teams operating within their authorisation and applicable law.
|
|
431
|
+
|
|
432
|
+
---
|
|
433
|
+
|
|
434
|
+
## Security
|
|
435
|
+
|
|
436
|
+
Report suspected vulnerabilities privately to **anonymosmauros@gmail.com** with the subject `[SECURITY] ThreatWatch — <short summary>`. Do not open a public issue before coordinated disclosure.
|
|
437
|
+
|
|
438
|
+
Supported release: **1.0.x**. The full policy, scope, and reporting guidance are in `SECURITY.md`.
|
|
439
|
+
|
|
440
|
+
ThreatWatch is defensive-only, reads API credentials from environment variables, sends no telemetry, and redacts configured secrets in `config show`. Generated reports may still contain sensitive threat intelligence; store and share them according to your classification policy.
|
|
441
|
+
|
|
442
|
+
---
|
|
443
|
+
|
|
444
|
+
## Changelog
|
|
445
|
+
|
|
446
|
+
Release notes are maintained in `CHANGELOG.md`. This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
447
|
+
|
|
448
|
+
---
|
|
449
|
+
|
|
450
|
+
## Development
|
|
451
|
+
|
|
452
|
+
```bash
|
|
453
|
+
cd threatwatch
|
|
454
|
+
|
|
455
|
+
# Offline test suite (stdlib unittest).
|
|
456
|
+
python3 -m unittest discover -s tests -v
|
|
457
|
+
|
|
458
|
+
# Byte-compile all modules.
|
|
459
|
+
python3 -m compileall -q threatwatch
|
|
460
|
+
|
|
461
|
+
# Live smoke test without GitHub advisories.
|
|
462
|
+
python3 -m threatwatch run --no-ghsa --since-days 3 --format md --format html --format json --outdir reports
|
|
463
|
+
```
|
|
464
|
+
|
|
465
|
+
Project layout:
|
|
466
|
+
|
|
467
|
+
```text
|
|
468
|
+
threatwatch/
|
|
469
|
+
├── pyproject.toml
|
|
470
|
+
├── MANIFEST.in
|
|
471
|
+
├── config.example.json
|
|
472
|
+
├── README.md
|
|
473
|
+
├── CHANGELOG.md
|
|
474
|
+
├── SECURITY.md
|
|
475
|
+
├── LICENSE
|
|
476
|
+
├── tests/
|
|
477
|
+
│ └── test_core.py
|
|
478
|
+
└── threatwatch/
|
|
479
|
+
├── __main__.py
|
|
480
|
+
├── cli.py # commands, argument wiring, output
|
|
481
|
+
├── config.py # defaults, loading, validation
|
|
482
|
+
├── models.py # data structures and serialisation
|
|
483
|
+
├── taxonomy.py # OWASP LLM Top 10 + MITRE ATLAS mapping
|
|
484
|
+
├── scoring.py # risk model, tiers, SLAs, asset matching
|
|
485
|
+
├── sources.py # KEV, NVD, EPSS, GHSA, RSS/Atom collectors
|
|
486
|
+
├── http.py # caching, retries, rate limiting
|
|
487
|
+
├── extract.py # conservative IOC extraction
|
|
488
|
+
├── state.py # watch state and diffs
|
|
489
|
+
├── correlate.py # collection orchestration and KPIs
|
|
490
|
+
├── summarize.py # deterministic narrative + optional LLM polish
|
|
491
|
+
├── opsec.py # defensive control library
|
|
492
|
+
├── report.py # Markdown, HTML, JSON, CSV renderers
|
|
493
|
+
└── utils.py # shared formatting helpers
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
Contributions should preserve three properties: standard library only, defensive scope only, and explainable scoring.
|