jmo-security 0.3.2__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of jmo-security might be problematic. Click here for more details.
- {jmo_security-0.3.2 → jmo_security-0.4.0}/PKG-INFO +263 -149
- {jmo_security-0.3.2 → jmo_security-0.4.0}/README.md +262 -148
- {jmo_security-0.3.2 → jmo_security-0.4.0}/jmo_security.egg-info/PKG-INFO +263 -149
- {jmo_security-0.3.2 → jmo_security-0.4.0}/jmo_security.egg-info/SOURCES.txt +1 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/pyproject.toml +25 -1
- jmo_security-0.4.0/scripts/__init__.py +1 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/cli/clone_from_tsv.py +47 -13
- jmo_security-0.4.0/scripts/cli/jmo.py +1270 -0
- jmo_security-0.4.0/scripts/cli/jmotools.py +383 -0
- jmo_security-0.4.0/scripts/cli/wizard.py +857 -0
- jmo_security-0.4.0/scripts/core/__init__.py +1 -0
- jmo_security-0.4.0/scripts/core/adapters/__init__.py +1 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/bandit_adapter.py +14 -2
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/checkov_adapter.py +5 -1
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/gitleaks_adapter.py +18 -3
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/hadolint_adapter.py +1 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/noseyparker_adapter.py +9 -3
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/osv_adapter.py +15 -5
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/semgrep_adapter.py +22 -5
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/syft_adapter.py +9 -2
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/tfsec_adapter.py +14 -4
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/trivy_adapter.py +27 -5
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/adapters/trufflehog_adapter.py +5 -1
- jmo_security-0.4.0/scripts/core/common_finding.py +126 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/config.py +29 -9
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/generate_dashboard.py +248 -206
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/normalize_and_report.py +29 -10
- jmo_security-0.4.0/scripts/core/reporters/__init__.py +1 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/reporters/basic_reporter.py +4 -1
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/reporters/html_reporter.py +23 -9
- jmo_security-0.4.0/scripts/core/reporters/sarif_reporter.py +181 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/reporters/suppression_reporter.py +8 -2
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/reporters/yaml_reporter.py +2 -2
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/schema_utils.py +3 -1
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/core/suppress.py +25 -8
- jmo_security-0.3.2/scripts/__init__.py +0 -1
- jmo_security-0.3.2/scripts/cli/jmo.py +0 -801
- jmo_security-0.3.2/scripts/cli/jmotools.py +0 -252
- jmo_security-0.3.2/scripts/core/__init__.py +0 -1
- jmo_security-0.3.2/scripts/core/adapters/__init__.py +0 -1
- jmo_security-0.3.2/scripts/core/common_finding.py +0 -41
- jmo_security-0.3.2/scripts/core/reporters/__init__.py +0 -1
- jmo_security-0.3.2/scripts/core/reporters/sarif_reporter.py +0 -64
- {jmo_security-0.3.2 → jmo_security-0.4.0}/LICENSE +0 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/jmo_security.egg-info/dependency_links.txt +0 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/jmo_security.egg-info/entry_points.txt +0 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/jmo_security.egg-info/requires.txt +0 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/jmo_security.egg-info/top_level.txt +0 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/scripts/cli/__init__.py +0 -0
- {jmo_security-0.3.2 → jmo_security-0.4.0}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: jmo-security
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.4.0
|
|
4
4
|
Summary: JMo Security Audit Suite (terminal-first, multi-tool, unified outputs)
|
|
5
5
|
Author-email: James Moceri <general@jmogaming.com>
|
|
6
6
|
License: MIT
|
|
@@ -22,9 +22,12 @@ Dynamic: license-file
|
|
|
22
22
|
</p>
|
|
23
23
|
|
|
24
24
|
[](https://github.com/jimmy058910/jmo-security-repo/actions/workflows/tests.yml?query=branch%3Amain)
|
|
25
|
+
[](https://app.codecov.io/gh/jimmy058910/jmo-security-repo)
|
|
26
|
+
[](https://pypi.org/project/jmo-security/)
|
|
25
27
|
[](https://jmotools.com)
|
|
26
28
|
[](LICENSE)
|
|
27
|
-
[](
|
|
29
|
+
[](https://www.python.org/)
|
|
30
|
+
|
|
28
31
|
[](https://github.com/jimmy058910/jmo-security-repo/issues)
|
|
29
32
|
[](https://ko-fi.com/jmogaming)
|
|
30
33
|
|
|
@@ -48,65 +51,173 @@ Project homepage: [jmotools.com](https://jmotools.com)
|
|
|
48
51
|
|
|
49
52
|
> Origin story: This started as part of a Cybersecurity Capstone Project. It has since grown into a general-purpose toolkit. I’d love for folks with deeper expertise to jump in—issues and PRs are welcome!
|
|
50
53
|
|
|
51
|
-
Thinking about contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup
|
|
54
|
+
Thinking about contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and coding standards. For publishing, see [docs/RELEASE.md](docs/RELEASE.md).
|
|
52
55
|
|
|
53
56
|
Roadmap & history:
|
|
54
|
-
|
|
57
|
+
|
|
58
|
+
- **Latest:** ROADMAP #2 (Interactive Wizard) ✅ Complete - see [docs/examples/wizard-examples.md](docs/examples/wizard-examples.md)
|
|
59
|
+
- Completed steps (summary): see [CHANGELOG.md](CHANGELOG.md) → ROADMAP Items #1-2 and Steps 1–13
|
|
55
60
|
- Active/planned work: see [ROADMAP.md](ROADMAP.md)
|
|
56
61
|
|
|
57
62
|
For scanning a list of repos from a TSV end-to-end (clone + unshallow + full toolchain), see: [docs/examples/scan_from_tsv.md](docs/examples/scan_from_tsv.md)
|
|
58
63
|
|
|
59
|
-
##
|
|
64
|
+
## ✅ CI and release at a glance
|
|
65
|
+
|
|
66
|
+
- Tests run on a matrix of operating systems and Python versions:
|
|
67
|
+
- OS: ubuntu-latest, macos-latest
|
|
68
|
+
- Python: 3.10, 3.11, 3.12
|
|
69
|
+
- CI uses concurrency to cancel redundant runs on rapid pushes and sets a 20-minute job timeout.
|
|
70
|
+
- Coverage is uploaded to Codecov without a token (OIDC/tokenless on public repos) using `codecov/codecov-action@v5`.
|
|
71
|
+
- Releases to PyPI use Trusted Publishers (OIDC) via `pypa/gh-action-pypi-publish@v1`; no PyPI API token is required once the repo is authorized in PyPI.
|
|
72
|
+
|
|
73
|
+
See `.github/workflows/tests.yml` and `.github/workflows/release.yml` for details.
|
|
74
|
+
|
|
75
|
+
Quick link: CI Troubleshooting → [Interpreting CI failures](docs/USER_GUIDE.md#interpreting-ci-failures-deeper-guide)
|
|
76
|
+
|
|
77
|
+
## 🎉 Recent Improvements (Phase 1 - October 2025)
|
|
78
|
+
|
|
79
|
+
**Security & Bug Fixes:**
|
|
80
|
+
|
|
81
|
+
- ✅ **XSS vulnerability patched** in HTML dashboard with comprehensive input escaping
|
|
82
|
+
- ✅ **OSV scanner fully integrated** for open-source vulnerability detection
|
|
83
|
+
- ✅ **Type-safe severity enum** with comparison operators for cleaner code
|
|
84
|
+
- ✅ **Backward-compatible suppression keys** (`suppressions` and legacy `suppress`)
|
|
85
|
+
|
|
86
|
+
**Enhanced Features:**
|
|
87
|
+
|
|
88
|
+
- 🚀 **Enriched SARIF output** with CWE/OWASP/CVE taxonomies, code snippets, and CVSS scores
|
|
89
|
+
- ⚙️ **Configurable thread recommendations** via `jmo.yml` profiling section
|
|
90
|
+
- 📝 **Magic numbers extracted** to named constants for better maintainability
|
|
91
|
+
- 📚 **9 new roadmap enhancements** including Policy-as-Code (OPA), SLSA attestation, GitHub App, and more
|
|
92
|
+
|
|
93
|
+
**Quality Metrics:**
|
|
94
|
+
|
|
95
|
+
- ✅ 100/100 tests passing
|
|
96
|
+
- ✅ 88% code coverage (exceeds 85% requirement)
|
|
97
|
+
- ✅ No breaking changes to existing workflows
|
|
98
|
+
|
|
99
|
+
See [CHANGELOG.md](CHANGELOG.md) for complete details.
|
|
100
|
+
|
|
101
|
+
## 🚀 Three Ways to Get Started
|
|
102
|
+
|
|
103
|
+
### Option 1: 🧙 Interactive Wizard (Recommended for Beginners)
|
|
104
|
+
|
|
105
|
+
**Never used security scanners before?** Start with the guided wizard:
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
jmotools wizard
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
The wizard provides:
|
|
112
|
+
|
|
113
|
+
- ✅ **Step-by-step guidance** through all configuration options
|
|
114
|
+
- ✅ **Profile selection** (fast/balanced/deep) with time estimates
|
|
115
|
+
- ✅ **Docker vs native mode** - zero-installation Docker option!
|
|
116
|
+
- ✅ **Smart target detection** - auto-discovers repositories
|
|
117
|
+
- ✅ **Command preview** - see what will run before executing
|
|
118
|
+
- ✅ **Auto-open results** - dashboard and summary automatically displayed
|
|
119
|
+
|
|
120
|
+
**Non-interactive mode for automation:**
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
jmotools wizard --yes # Use smart defaults
|
|
124
|
+
jmotools wizard --docker # Force Docker mode
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
**Generate reusable artifacts:**
|
|
128
|
+
|
|
129
|
+
```bash
|
|
130
|
+
jmotools wizard --emit-make-target Makefile.security # Team Makefile
|
|
131
|
+
jmotools wizard --emit-script scan.sh # Shell script
|
|
132
|
+
jmotools wizard --emit-gha .github/workflows/security.yml # GitHub Actions
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
📖 **Full wizard guide:** [docs/examples/wizard-examples.md](docs/examples/wizard-examples.md)
|
|
60
136
|
|
|
61
|
-
|
|
137
|
+
---
|
|
62
138
|
|
|
63
|
-
|
|
64
|
-
- `jmotools balanced` — default balanced scan
|
|
65
|
-
- `jmotools full` — deep scan with more tools
|
|
139
|
+
### Option 2: 🐳 Docker (Zero Installation)
|
|
66
140
|
|
|
67
|
-
|
|
141
|
+
**Want to start scanning immediately with no tool installation?**
|
|
68
142
|
|
|
69
143
|
```bash
|
|
70
|
-
#
|
|
71
|
-
|
|
144
|
+
# Pull the image (one-time, ~500MB)
|
|
145
|
+
docker pull ghcr.io/jimmy058910/jmo-security:latest
|
|
146
|
+
|
|
147
|
+
# Scan current directory
|
|
148
|
+
docker run --rm -v $(pwd):/scan ghcr.io/jimmy058910/jmo-security:latest \
|
|
149
|
+
scan --repo /scan --results /scan/results --profile balanced --human-logs
|
|
72
150
|
|
|
73
|
-
#
|
|
74
|
-
|
|
151
|
+
# View results
|
|
152
|
+
open results/summaries/dashboard.html # macOS
|
|
153
|
+
xdg-open results/summaries/dashboard.html # Linux
|
|
75
154
|
```
|
|
76
155
|
|
|
77
|
-
|
|
156
|
+
**Three image variants available:**
|
|
157
|
+
|
|
158
|
+
- `latest` (~500MB) - All 11+ scanners included
|
|
159
|
+
- `slim` (~200MB) - Core 6 scanners for CI/CD
|
|
160
|
+
- `alpine` (~150MB) - Minimal footprint
|
|
161
|
+
|
|
162
|
+
📖 **Complete Docker guide:** [docs/DOCKER_README.md](docs/DOCKER_README.md)
|
|
163
|
+
📖 **Beginner Docker tutorial:** [docs/DOCKER_QUICKSTART_BEGINNERS.md](docs/DOCKER_QUICKSTART_BEGINNERS.md)
|
|
164
|
+
|
|
165
|
+
---
|
|
166
|
+
|
|
167
|
+
### Option 3: 🧪 CLI Wrapper Commands (Local Install)
|
|
168
|
+
|
|
169
|
+
**Already have tools installed? Use our simple wrapper commands:**
|
|
170
|
+
|
|
171
|
+
```bash
|
|
172
|
+
# Quick fast scan (auto-opens results)
|
|
173
|
+
jmotools fast --repos-dir ~/repos
|
|
174
|
+
|
|
175
|
+
# Balanced scan (recommended default)
|
|
176
|
+
jmotools balanced --repos-dir ~/repos
|
|
177
|
+
|
|
178
|
+
# Deep scan with all tools
|
|
179
|
+
jmotools full --repos-dir ~/repos
|
|
180
|
+
```
|
|
78
181
|
|
|
79
|
-
|
|
182
|
+
**Clone from TSV and scan:**
|
|
80
183
|
|
|
81
184
|
```bash
|
|
82
|
-
|
|
83
|
-
jmotools setup --check
|
|
84
|
-
jmotools setup --auto-install # or --print-commands, --force-reinstall
|
|
185
|
+
jmotools balanced --tsv ./repositories.tsv --dest ./cloned-repos
|
|
85
186
|
```
|
|
86
187
|
|
|
87
|
-
|
|
188
|
+
**Setup tools quickly:**
|
|
88
189
|
|
|
89
190
|
```bash
|
|
90
|
-
|
|
91
|
-
|
|
191
|
+
jmotools setup --check # Verify tool installation
|
|
192
|
+
jmotools setup --auto-install # Auto-install on Linux/WSL/macOS
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
**Makefile shortcuts:**
|
|
196
|
+
|
|
197
|
+
```bash
|
|
198
|
+
make setup # Verify tools (installs package if needed)
|
|
199
|
+
make fast DIR=~/repos # Run fast profile
|
|
92
200
|
make balanced DIR=~/repos
|
|
93
201
|
make full DIR=~/repos
|
|
94
202
|
```
|
|
95
203
|
|
|
204
|
+
Note: Under the hood, wrapper commands verify your OS/tools, optionally clone from TSV, run `jmo ci` with the appropriate profile, and auto-open results.
|
|
205
|
+
|
|
96
206
|
## 🎯 Overview
|
|
97
207
|
|
|
98
208
|
This project provides an automated framework for conducting thorough security audits on code repositories. It orchestrates multiple industry-standard security tools to detect secrets, vulnerabilities, and security issues.
|
|
99
209
|
|
|
100
210
|
### Key Features
|
|
101
211
|
|
|
102
|
-
- ✅ **Multi-Tool Scanning**: Curated set covering secrets (gitleaks, noseyparker), SAST (semgrep, bandit), SBOM+vuln/misconfig (syft+trivy), IaC (checkov, tfsec), Dockerfile (hadolint)
|
|
103
|
-
- 📊 **Comprehensive Reporting**: Unified findings (JSON/YAML), SARIF, Markdown summary, and an interactive HTML dashboard
|
|
104
|
-
- 🎨 **Easy-to-Read Outputs**: Well-formatted reports with severity categorization
|
|
212
|
+
- ✅ **Multi-Tool Scanning**: Curated set covering secrets (gitleaks, noseyparker, trufflehog), SAST (semgrep, bandit), SBOM+vuln/misconfig (syft+trivy), IaC (checkov, tfsec), Dockerfile (hadolint), and open-source vulnerabilities (osv-scanner)
|
|
213
|
+
- 📊 **Comprehensive Reporting**: Unified findings (JSON/YAML), enriched SARIF 2.1.0 with taxonomies, Markdown summary, and an interactive HTML dashboard with XSS protection
|
|
214
|
+
- 🎨 **Easy-to-Read Outputs**: Well-formatted reports with severity categorization using type-safe enums
|
|
105
215
|
- 🔄 **Automated Workflows**: One CLI to scan, aggregate, and gate on severity (scan/report/ci)
|
|
106
|
-
- 🧭 **Profiles and Overrides**: Named profiles, per-tool flags/timeouts, include/exclude patterns
|
|
216
|
+
- 🧭 **Profiles and Overrides**: Named profiles, per-tool flags/timeouts, include/exclude patterns, configurable thread recommendations
|
|
107
217
|
- 🔁 **Resilience**: Timeouts, retries with per-tool success codes, human-friendly logs, graceful cancel
|
|
218
|
+
- 🔒 **Security-First**: XSS vulnerability patched, comprehensive input escaping, secure-by-default configurations
|
|
108
219
|
|
|
109
|
-
## 🚀 Quick Start
|
|
220
|
+
## 🚀 Quick Start (Local Installation)
|
|
110
221
|
|
|
111
222
|
### Install or Update (curated tools)
|
|
112
223
|
|
|
@@ -125,21 +236,46 @@ Optional: install the package locally to get `jmo` and `jmotools` commands on yo
|
|
|
125
236
|
pip install -e .
|
|
126
237
|
```
|
|
127
238
|
|
|
239
|
+
#### Pre-commit hooks (YAML & Actions validation)
|
|
240
|
+
|
|
241
|
+
We ship pre-commit hooks for YAML linting and GitHub Actions validation (among other basic checks):
|
|
242
|
+
|
|
243
|
+
```bash
|
|
244
|
+
make pre-commit-install # installs the git hooks
|
|
245
|
+
make pre-commit-run # run checks on all files
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
These run locally via pre-commit and are also enforced in CI.
|
|
249
|
+
|
|
250
|
+
We ship a `.yamllint.yaml` and validate GitHub Actions workflows via `actionlint`. The same checks are executed in CI.
|
|
251
|
+
|
|
252
|
+
#### Reproducible dev dependencies (optional)
|
|
253
|
+
|
|
254
|
+
This repo ships a `requirements-dev.in` with a compiled `requirements-dev.txt`. Use pip-tools or uv to pin/sync your dev environment:
|
|
255
|
+
|
|
256
|
+
```bash
|
|
257
|
+
make upgrade-pip
|
|
258
|
+
make deps-compile # compile dev deps
|
|
259
|
+
make deps-sync # sync env to compiled lock
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
CI verifies that `requirements-dev.txt` is up to date on PRs. If it fails, run `make deps-compile` and commit the diff.
|
|
263
|
+
|
|
128
264
|
### Quick Start (Unified CLI)
|
|
129
265
|
|
|
130
|
-
1
|
|
266
|
+
1. Verify your environment (Linux/WSL/macOS) and see install hints for optional tools:
|
|
131
267
|
|
|
132
268
|
```bash
|
|
133
269
|
make verify-env
|
|
134
270
|
```
|
|
135
271
|
|
|
136
|
-
|
|
272
|
+
1. Install Python dev dependencies (for running tests and reporters):
|
|
137
273
|
|
|
138
274
|
```bash
|
|
139
275
|
make dev-deps
|
|
140
276
|
```
|
|
141
277
|
|
|
142
|
-
|
|
278
|
+
1. Scan repositories using a profile, then aggregate reports:
|
|
143
279
|
|
|
144
280
|
```bash
|
|
145
281
|
# Scan immediate subfolders under ~/repos with the 'balanced' profile (default)
|
|
@@ -150,8 +286,11 @@ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name balanced --hu
|
|
|
150
286
|
python3 scripts/cli/jmo.py report ./results --profile --human-logs
|
|
151
287
|
# or
|
|
152
288
|
python3 scripts/cli/jmo.py report --results-dir ./results --profile --human-logs
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
#### Or do both in one step for CI with a failure threshold
|
|
153
292
|
|
|
154
|
-
|
|
293
|
+
```bash
|
|
155
294
|
python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name fast --fail-on HIGH --profile --human-logs
|
|
156
295
|
```
|
|
157
296
|
|
|
@@ -196,9 +335,9 @@ Prefer the Python CLI above. For legacy flows, you can still use the shell wrapp
|
|
|
196
335
|
python3 scripts/cli/jmo.py ci --repos-dir ~/test-repos --fail-on HIGH --profile --human-logs
|
|
197
336
|
|
|
198
337
|
# 3. View results
|
|
199
|
-
cat
|
|
200
|
-
# macOS: open
|
|
201
|
-
# Linux: xdg-open
|
|
338
|
+
cat results/summaries/SUMMARY.md
|
|
339
|
+
# macOS: open results/summaries/dashboard.html
|
|
340
|
+
# Linux: xdg-open results/summaries/dashboard.html
|
|
202
341
|
```
|
|
203
342
|
|
|
204
343
|
Looking for screenshots and how to capture them? See: [docs/screenshots/README.md](docs/screenshots/README.md)
|
|
@@ -215,53 +354,44 @@ The security audit follows this workflow:
|
|
|
215
354
|
4. **Report Generation**: JSON/MD/YAML/HTML/SARIF and suppression summary
|
|
216
355
|
5. **Dashboard Creation**: Self-contained HTML dashboard with an optional profiling panel
|
|
217
356
|
|
|
218
|
-
### Output Structure
|
|
219
|
-
|
|
220
|
-
```
|
|
221
|
-
|
|
222
|
-
├──
|
|
223
|
-
|
|
224
|
-
├──
|
|
225
|
-
│
|
|
226
|
-
│ ├──
|
|
227
|
-
│ ├──
|
|
228
|
-
│ ├──
|
|
229
|
-
│ ├──
|
|
230
|
-
│ ├──
|
|
231
|
-
│
|
|
232
|
-
├──
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
├──
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
- Tool-by-tool breakdown
|
|
257
|
-
- Detailed issue listings
|
|
258
|
-
- Severity classifications
|
|
259
|
-
|
|
260
|
-
4. **Tool Comparison** (`tool-comparisons/comparison.md`)
|
|
261
|
-
- Detection metrics
|
|
262
|
-
- Tool capabilities matrix
|
|
263
|
-
- Implementation strategy guide
|
|
264
|
-
- Tool selection recommendations
|
|
357
|
+
### Output Structure (Default)
|
|
358
|
+
|
|
359
|
+
```text
|
|
360
|
+
results/
|
|
361
|
+
├── individual-repos/
|
|
362
|
+
│ └── <repo-name>/
|
|
363
|
+
│ ├── gitleaks.json
|
|
364
|
+
│ ├── trufflehog.json
|
|
365
|
+
│ ├── semgrep.json
|
|
366
|
+
│ ├── noseyparker.json
|
|
367
|
+
│ ├── syft.json
|
|
368
|
+
│ ├── trivy.json
|
|
369
|
+
│ ├── hadolint.json
|
|
370
|
+
│ ├── checkov.json
|
|
371
|
+
│ ├── tfsec.json
|
|
372
|
+
│ └── bandit.json
|
|
373
|
+
└── summaries/
|
|
374
|
+
├── findings.json
|
|
375
|
+
├── findings.yaml # requires PyYAML
|
|
376
|
+
├── findings.sarif # SARIF 2.1.0
|
|
377
|
+
├── SUMMARY.md
|
|
378
|
+
├── dashboard.html
|
|
379
|
+
├── SUPPRESSIONS.md # written when suppressions apply
|
|
380
|
+
└── timings.json # written when --profile is used
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
### Reporters
|
|
384
|
+
|
|
385
|
+
The aggregator writes unified outputs under `results/summaries/`:
|
|
386
|
+
|
|
387
|
+
- JSON (`findings.json`) — complete, machine-readable findings list
|
|
388
|
+
- Markdown (`SUMMARY.md`) — human-readable overview with severity counts and top rules
|
|
389
|
+
- YAML (`findings.yaml`) — optional; requires PyYAML
|
|
390
|
+
- HTML (`dashboard.html`) — interactive dashboard with filters, sorting, exports, and theme toggle
|
|
391
|
+
- SARIF (`findings.sarif`) — 2.1.0 for code scanning integrations
|
|
392
|
+
- Suppression summary (`SUPPRESSIONS.md`) — appears when suppression rules filter findings
|
|
393
|
+
|
|
394
|
+
See `SAMPLE_OUTPUTS.md` for real examples produced from the `infra-demo` fixture.
|
|
265
395
|
|
|
266
396
|
### How we normalize findings
|
|
267
397
|
|
|
@@ -271,8 +401,6 @@ All tool outputs are converted into a single CommonFinding schema during aggrega
|
|
|
271
401
|
- Required fields include: schemaVersion (1.0.0), id, ruleId, severity, tool (name/version), location (path/lines), and message. Optional fields include title, description, remediation, references, tags, cvss, and raw (original tool payload).
|
|
272
402
|
- Fingerprint (id): deterministically derived from a stable subset of attributes (tool | ruleId | path | startLine | message snippet) to support cross-tool dedupe. The aggregation step deduplicates by this id.
|
|
273
403
|
|
|
274
|
-
|
|
275
|
-
|
|
276
404
|
## 🛠️ Tool Installation
|
|
277
405
|
|
|
278
406
|
### macOS (Homebrew)
|
|
@@ -319,10 +447,10 @@ curl -sSfL https://raw.githubusercontent.com/trufflesecurity/trufflehog/main/scr
|
|
|
319
447
|
|
|
320
448
|
Nosey Parker doesn’t ship via apt/brew universally. Install the release binary and put it on your PATH:
|
|
321
449
|
|
|
322
|
-
1
|
|
323
|
-
https://github.com/praetorian-inc/noseyparker/releases
|
|
450
|
+
1. Download the latest release for your OS/arch from:
|
|
451
|
+
<https://github.com/praetorian-inc/noseyparker/releases>
|
|
324
452
|
|
|
325
|
-
2
|
|
453
|
+
2. Unpack and move the binary onto PATH (example for Linux x86_64):
|
|
326
454
|
|
|
327
455
|
```bash
|
|
328
456
|
tar -xzf noseyparker-*.tar.gz
|
|
@@ -339,11 +467,12 @@ On WSL Ubuntu, installing Nosey Parker natively is the most reliable path (prebu
|
|
|
339
467
|
|
|
340
468
|
The CLI automatically falls back to a Docker-based Nosey Parker runner when the local binary is missing or not runnable (common on older WSL/glibc). When enabled via profiles, scans will transparently produce the expected JSON here:
|
|
341
469
|
|
|
342
|
-
```
|
|
470
|
+
```text
|
|
343
471
|
results/individual-repos/<repo-name>/noseyparker.json
|
|
344
472
|
```
|
|
345
473
|
|
|
346
474
|
Requirements for the fallback:
|
|
475
|
+
|
|
347
476
|
- Docker installed and running
|
|
348
477
|
- Ability to pull or use `ghcr.io/praetorian-inc/noseyparker:latest`
|
|
349
478
|
|
|
@@ -380,6 +509,7 @@ Note: we recommend isolating CLI tools via pipx or OS packages for stability. Th
|
|
|
380
509
|
This helper script streamlines the process of cloning multiple repositories for security scanning, with performance optimizations for WSL environments.
|
|
381
510
|
|
|
382
511
|
**Features:**
|
|
512
|
+
|
|
383
513
|
- 🚀 Shallow clones (depth=1) for faster cloning
|
|
384
514
|
- ⚡ Parallel cloning for improved performance
|
|
385
515
|
- 🔄 Unshallow option for secret scanners requiring full history
|
|
@@ -405,7 +535,8 @@ This helper script streamlines the process of cloning multiple repositories for
|
|
|
405
535
|
```
|
|
406
536
|
|
|
407
537
|
**Repository List Format (`samples/repos.txt`):**
|
|
408
|
-
|
|
538
|
+
|
|
539
|
+
```text
|
|
409
540
|
# One GitHub repository URL per line
|
|
410
541
|
# Lines starting with # are comments
|
|
411
542
|
https://github.com/user/repo1.git
|
|
@@ -413,37 +544,29 @@ https://github.com/user/repo2.git
|
|
|
413
544
|
```
|
|
414
545
|
|
|
415
546
|
**Performance Tips for WSL:**
|
|
547
|
+
|
|
416
548
|
1. Use shallow clones initially for 10x faster cloning
|
|
417
549
|
2. Adjust `--parallel` based on network speed (default: 4)
|
|
418
550
|
3. Use `--unshallow` only if secret scanners need full git history
|
|
419
551
|
4. Clone to WSL filesystem (not Windows mount) for better performance
|
|
420
552
|
|
|
421
|
-
###
|
|
553
|
+
### CLI-first usage
|
|
422
554
|
|
|
423
|
-
|
|
424
|
-
```bash
|
|
425
|
-
./scripts/core/check_tools.sh
|
|
426
|
-
```
|
|
555
|
+
Prefer the Python CLI for report generation from existing results:
|
|
427
556
|
|
|
428
|
-
2. **Main Audit Script**:
|
|
429
557
|
```bash
|
|
430
|
-
|
|
431
|
-
|
|
558
|
+
# Default reporters (formats controlled by jmo.yml)
|
|
559
|
+
python3 scripts/cli/jmo.py report /path/to/results
|
|
432
560
|
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
# Generate dashboard with default output (results_dir/dashboard.html)
|
|
436
|
-
python3 scripts/core/generate_dashboard.py /path/to/results
|
|
561
|
+
# Set thread workers explicitly for aggregation
|
|
562
|
+
python3 scripts/cli/jmo.py report /path/to/results --threads 6
|
|
437
563
|
|
|
438
|
-
#
|
|
439
|
-
python3 scripts/
|
|
440
|
-
```
|
|
564
|
+
# Record profiling timings (writes summaries/timings.json)
|
|
565
|
+
python3 scripts/cli/jmo.py report /path/to/results --profile
|
|
441
566
|
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
- Graceful handling of missing or empty scan results
|
|
446
|
-
- UTF-8 safe file I/O for international characters
|
|
567
|
+
# Human-friendly colored logs (stderr)
|
|
568
|
+
python3 scripts/cli/jmo.py report /path/to/results --human-logs
|
|
569
|
+
```
|
|
447
570
|
|
|
448
571
|
### Unified CLI: report-only
|
|
449
572
|
|
|
@@ -486,7 +609,7 @@ python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name balanced --fail
|
|
|
486
609
|
|
|
487
610
|
The `summaries/` folder also contains unified outputs:
|
|
488
611
|
|
|
489
|
-
```
|
|
612
|
+
```text
|
|
490
613
|
summaries/
|
|
491
614
|
├── findings.json # Unified normalized findings (machine-readable)
|
|
492
615
|
├── SUMMARY.md # Human-readable summary
|
|
@@ -503,7 +626,7 @@ You can define named profiles in `jmo.yml` to control which tools run, include/e
|
|
|
503
626
|
|
|
504
627
|
Example `jmo.yml` snippet:
|
|
505
628
|
|
|
506
|
-
```
|
|
629
|
+
```yaml
|
|
507
630
|
default_profile: fast
|
|
508
631
|
retries: 1
|
|
509
632
|
profiles:
|
|
@@ -530,7 +653,7 @@ per_tool:
|
|
|
530
653
|
|
|
531
654
|
Using a profile from CLI:
|
|
532
655
|
|
|
533
|
-
```
|
|
656
|
+
```bash
|
|
534
657
|
# Scan using profile 'fast' with human-friendly logs
|
|
535
658
|
python3 scripts/cli/jmo.py scan --repos-dir ~/repos --profile-name fast --human-logs
|
|
536
659
|
|
|
@@ -539,16 +662,12 @@ python3 scripts/cli/jmo.py ci --repos-dir ~/repos --profile-name deep --fail-on
|
|
|
539
662
|
```
|
|
540
663
|
|
|
541
664
|
Retries behavior:
|
|
665
|
+
|
|
542
666
|
- Global `retries` (or per-profile) retries failed tool commands a limited number of times
|
|
543
667
|
- Some tools use non-zero exit to indicate “findings”; we treat those as success codes to avoid useless retries
|
|
544
668
|
|
|
545
669
|
Human logs show per-tool retry attempts when > 1, e.g.: `attempts={'semgrep': 2}`
|
|
546
670
|
|
|
547
|
-
4. **Generate Comparison Report**:
|
|
548
|
-
```bash
|
|
549
|
-
./scripts/core/generate_comparison_report.sh /path/to/results
|
|
550
|
-
```
|
|
551
|
-
|
|
552
671
|
### Customizing Tool Execution
|
|
553
672
|
|
|
554
673
|
Prefer jmo.yml profiles and per_tool overrides. For one-off local tweaks, use:
|
|
@@ -567,10 +686,13 @@ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --tools gitleaks semgrep --t
|
|
|
567
686
|
|
|
568
687
|
### Severity Levels
|
|
569
688
|
|
|
689
|
+
The toolkit uses a type-safe severity enum with comparison operators for consistent filtering and sorting:
|
|
690
|
+
|
|
570
691
|
- **CRITICAL**: Verified secrets requiring immediate action
|
|
571
692
|
- **HIGH**: Likely secrets or serious vulnerabilities
|
|
572
693
|
- **MEDIUM**: Potential issues requiring review
|
|
573
|
-
- **LOW
|
|
694
|
+
- **LOW**: Minor issues for regular maintenance
|
|
695
|
+
- **INFO**: Informational findings
|
|
574
696
|
|
|
575
697
|
### Key Metrics
|
|
576
698
|
|
|
@@ -589,46 +711,26 @@ python3 scripts/cli/jmo.py scan --repos-dir ~/repos --tools gitleaks semgrep --t
|
|
|
589
711
|
## 🎯 Three-Stage Implementation Strategy
|
|
590
712
|
|
|
591
713
|
### Stage 1: Pre-commit Hooks
|
|
714
|
+
|
|
592
715
|
- **Tool**: Gitleaks
|
|
593
716
|
- **Purpose**: Prevent secrets before commit
|
|
594
717
|
- **Speed**: Fast (suitable for developer workflow)
|
|
595
718
|
|
|
596
719
|
### Stage 2: CI/CD Pipeline
|
|
720
|
+
|
|
597
721
|
- **Tools**: Gitleaks + Semgrep
|
|
598
722
|
- **Purpose**: Automated PR/commit scanning
|
|
599
723
|
- **Coverage**: Secrets + vulnerabilities
|
|
600
724
|
|
|
601
725
|
### Stage 3: Deep Periodic Audits
|
|
726
|
+
|
|
602
727
|
- **Tools**: All tools
|
|
603
728
|
- **Purpose**: Comprehensive security assessment
|
|
604
729
|
- **Frequency**: Weekly/monthly
|
|
605
730
|
|
|
606
|
-
## 📊 Sample
|
|
607
|
-
|
|
608
|
-
### Dashboard Preview
|
|
609
|
-
The HTML dashboard provides:
|
|
610
|
-
- Visual metric cards with key statistics
|
|
611
|
-
- Severity breakdown tables
|
|
612
|
-
- Repository-by-repository comparison
|
|
613
|
-
- Tool performance analysis
|
|
614
|
-
- Actionable recommendations
|
|
615
|
-
|
|
616
|
-
### Sample Summary Report
|
|
617
|
-
```markdown
|
|
618
|
-
## Aggregate Results
|
|
731
|
+
## 📊 Sample Outputs
|
|
619
732
|
|
|
620
|
-
|
|
621
|
-
- Total Issues Found: 1562
|
|
622
|
-
- Critical Issues: 5
|
|
623
|
-
- High Severity Issues: 579
|
|
624
|
-
- Medium Severity Issues: 61
|
|
625
|
-
- Verified Secrets: 5
|
|
626
|
-
|
|
627
|
-
### Recommendations
|
|
628
|
-
- Rotate all 5 verified secrets immediately
|
|
629
|
-
- Prioritize remediation of 584 critical and high severity issues
|
|
630
|
-
- Schedule follow-up review for the remaining medium findings
|
|
631
|
-
```
|
|
733
|
+
For a current snapshot produced from the `infra-demo` fixture, see: [SAMPLE_OUTPUTS.md](SAMPLE_OUTPUTS.md).
|
|
632
734
|
|
|
633
735
|
## 🤝 Contributing
|
|
634
736
|
|
|
@@ -638,7 +740,7 @@ Contributions are welcome! Please feel free to submit pull requests or open issu
|
|
|
638
740
|
|
|
639
741
|
If this toolkit saves you time, consider fueling development with an energy drink.
|
|
640
742
|
|
|
641
|
-
- Prefer one-time tips? Ko‑fi: https://ko-fi.com/jmogaming
|
|
743
|
+
- Prefer one-time tips? Ko‑fi: <https://ko-fi.com/jmogaming>
|
|
642
744
|
- When you’re ready, replace the badge target with your preferred platform: GitHub Sponsors (industry standard), Open Collective, Ko-fi, or Stripe Checkout.
|
|
643
745
|
- GitHub Sponsors integrates directly with your GitHub profile and repository sidebar once enabled.
|
|
644
746
|
|
|
@@ -651,9 +753,6 @@ MIT License. See LICENSE.
|
|
|
651
753
|
- [Gitleaks Documentation](https://github.com/zricethezav/gitleaks)
|
|
652
754
|
- [TruffleHog Documentation](https://github.com/trufflesecurity/trufflehog)
|
|
653
755
|
- [Semgrep Documentation](https://semgrep.dev)
|
|
654
|
-
- [Nosey Parker Documentation](https://github.com/praetorian-inc/noseyparker)
|
|
655
|
-
|
|
656
|
-
## 💡 Tips
|
|
657
756
|
|
|
658
757
|
1. **Start Small**: Test on a single repository first
|
|
659
758
|
2. **Review Regularly**: Schedule periodic audits
|
|
@@ -666,22 +765,36 @@ MIT License. See LICENSE.
|
|
|
666
765
|
### Common Issues
|
|
667
766
|
|
|
668
767
|
**Problem**: Tools not found
|
|
669
|
-
|
|
768
|
+
|
|
769
|
+
- **Solution**: Run `make verify-env` (or `jmotools setup --check`) to verify installation and get platform-specific hints
|
|
670
770
|
|
|
671
771
|
**Problem**: JSON parsing errors
|
|
772
|
+
|
|
672
773
|
- **Solution**: Ensure jq is installed and tools are outputting valid JSON
|
|
673
774
|
|
|
674
775
|
**Problem**: Permission denied
|
|
675
|
-
|
|
776
|
+
|
|
777
|
+
- **Solution**: Ensure scripts are executable:
|
|
778
|
+
|
|
779
|
+
```bash
|
|
780
|
+
find scripts -type f -name "*.sh" -exec chmod +x {} +
|
|
781
|
+
```
|
|
676
782
|
|
|
677
783
|
**Problem**: Out of memory
|
|
784
|
+
|
|
678
785
|
- **Solution**: Scan repositories in smaller batches
|
|
679
786
|
|
|
787
|
+
```bash
|
|
788
|
+
./scripts/core/populate_targets.sh --unshallow
|
|
789
|
+
```
|
|
790
|
+
|
|
680
791
|
**Problem**: Path errors (e.g., "//run_security_audit.sh not found")
|
|
792
|
+
|
|
681
793
|
- **Solution**: This issue has been fixed in the latest version. Update to the latest main branch.
|
|
682
794
|
- The wrapper scripts now use absolute paths computed from the script's real path location.
|
|
683
795
|
|
|
684
796
|
**Problem**: AttributeError when generating dashboard with TruffleHog results
|
|
797
|
+
|
|
685
798
|
- **Solution**: This has been fixed. The dashboard generator now handles all TruffleHog output formats:
|
|
686
799
|
- JSON arrays: `[{...}, {...}]`
|
|
687
800
|
- Single objects: `{...}`
|
|
@@ -705,11 +818,12 @@ python3 scripts/core/generate_dashboard.py ~/security-results-20251010-120000 ~/
|
|
|
705
818
|
```
|
|
706
819
|
|
|
707
820
|
This is useful when you want to:
|
|
821
|
+
|
|
708
822
|
- Update the dashboard after manually editing JSON files
|
|
709
823
|
- Generate multiple dashboards with different configurations
|
|
710
824
|
- Share results by exporting to a specific location
|
|
711
825
|
|
|
712
826
|
---
|
|
713
827
|
|
|
714
|
-
**Last Updated**: October
|
|
828
|
+
**Last Updated**: October 13th, 2025
|
|
715
829
|
**Author**: James Moceri
|